Investigate eflomal

[alex-rawlings-yyc]

https://github.com/robertostling/eflomal

[alex-rawlings-yyc]

As per Opus 4.8:

eflomal Integration Feasibility — 2026-06-15

Assessment of integrating the eflomal word aligner (../eflomal, pinned at
commit 1fe2a43, 2025-01-15) into the interlinearizer extension to produce automatic
source↔target alignment suggestions.

TL;DR: Feasible and partly anticipated by the data model. The only sanctioned runtime
path is spawning a native binary via PAPI's createProcess elevated privilege.
Recommended runtime: bundle prebuilt per-platform C binaries and reimplement eflomal's
small preprocessing/parsing layer in TypeScript (no Python). Two non-technical blockers
to resolve first: a GPLv3-vs-MIT license conflict and per-platform binary build/ship
infrastructure.

---

1. What eflomal is and how it runs

• Word alignment tool (IBM1 / HMM / HMM+fertility models). Input: tokenized parallel
  sentences. Output: Pharaoh-style alignment pairs.
• No JS or WASM path exists in the repo — searched exhaustively. The only invocation
  routes are the Python eflomal.Aligner API or the eflomal-align CLI, both of which
  ultimately spawn a compiled C binary.
• The C binary (../eflomal/src/eflomal.c, $(CC) + OpenMP via src/Makefile,
  -Ofast -march=native --std=gnu99 -fopenmp) does not read raw text. It reads a
  preprocessed numeric format. The Python/Cython layer
  (read_text/write_text/align in eflomal.pyx)
  does the input glue:
  • tokens → integer vocabulary IDs (read_text),
  • writes eflomal's binary input files: a "<n_sents> <voc_size>" header line, then per
    sentence a length-prefixed list of space-separated token IDs (write_text),
  • computes per-model iteration counts from the sentence count when not supplied
    (align: iters = max(2, round(rel_iterations*5000 / sqrt(n_sentences)))),
  • spawns the binary, passing the source/target input files and the output link paths
    (-s/-t/-f/-r, plus -m/-n/-N/-1/-2/-3).
  • The binary then writes the link files directly to the -f/-r paths; the .pyx
    glue does not read them back. Output parsing lives in the caller (e.g.
    calculate_priors in __init__.py) and is a
    trivial split of a plain-text file — so a TS port owns the input encoding plus a one-line
    output reader.
• Output format: space-separated src-tgt index pairs, 0-based, one line per
  sentence, forward (-f) and/or reverse (-r). NULL alignments are omitted
  (NULL_LINK 0xffffU in eflomal.c). Directly consumable; symmetrization (the atools
  program from fast_align) is optional and is a separate tool not bundled with eflomal.

2. How it fits our data model — already half-anticipated

AlignmentLink in interlinearizer.d.ts was designed with
this in mind:

• origin?: string documents 'eflomal' as an expected value.
• status already includes 'suggested' / 'candidate' — the right states for
  machine-generated links awaiting human review.
• AlignmentEndpoint references tokens by tokenRef + a surfaceText snapshot for drift
  detection. The tokenizer (bookTokenizer.ts) already
  emits stable refs (SID:charStart) and word/punctuation typing — exactly the token list
  eflomal needs (we'd feed it type === 'word' surface texts).
• InterlinearProject.links exists and is initialized to [] for bilateral projects
  (projectStorage.ts:135).

But nothing populates links today. No Redux actions, no UI, no compute path. The type
layer is pre-designed; the implementation is greenfield.

3. The decisive enabler: PAPI allows subprocesses

PAPI exposes context.elevatedPrivileges.createProcess.spawn(executionToken, command, args, options)
and .fork(...) — a thin layer over Node child_process, with cwd forced to the
extension root and children auto-killed in packaged builds
(papi.d.ts:6183-6228; the CreateProcess
object type is at papi.d.ts:6272). It requires adding
"createProcess" to the currently-empty elevatedPrivileges array in
manifest.json (the string token is ElevatedPrivilegeNames.CreateProcess === 'createProcess'). This is the intended mechanism for exactly this use case —
so a binary-spawn integration is architecturally sanctioned, not a hack.

The extension currently shells out to nothing — no subprocesses, workers, or native
deps. This would be the first.

4. Runtime options

Option	Integration code	Footprint	OS fragility	Verdict
A. Prebuilt C binary + TS preprocessing	Reimplement the input encoder (read_text/write_text + iteration-count math) in TS; output reader is a trivial text split	~1 small binary per OS	Low (no interpreter)	Recommended
B. Bundle Python + eflomal pkg	Minimal — Python does the glue	~50MB+ Python per OS	High	Avoid
C. WASM port of eflomal.c	No subprocess/privilege; in-process	Medium (.wasm)	Lowest at runtime	Future ideal; real porting effort (OpenMP, RNG, FS via Emscripten); doesn't exist today

Recommendation: Option A

Reimplementing the preprocessing in TypeScript is modest, bounded work — the formats are
small and fully readable in eflomal.pyx. It avoids shipping a Python interpreter (a poor
fit for a Platform.Bible extension's size/portability expectations) and keeps the compute
contract entirely inside our codebase. Option C is the long-term ideal (no elevated
privilege, no per-OS binaries) but is a separate research project; revisit if binary
build/ship infra proves painful or if privilege-free execution becomes a requirement.

5. Where the implementation work lands (Option A)

1. Manifest: add "createProcess" to elevatedPrivileges in manifest.json.
2. Binary build/ship: compile eflomal for win32/linux/darwin and bundle under the
   extension (the createProcess examples assume per-platform subdirs). This is the
   biggest non-code lift — needs CI build infra (a C compiler + OpenMP; the Makefile uses
   $(CC), GCC in practice) and a way to obtain Windows/macOS binaries. Note: the
   stock Makefile compiles with -march=native, which targets the build machine's CPU; a
   redistributable binary must drop that for a portable baseline (e.g. -march=x86-64 /
   -mtune=generic) or it may fault on older/different CPUs.
3. New backend service (under src/services/): gather word tokens for a
   verse/range from source + target books → vocab-map and write eflomal input → spawn
   binary → parse src-tgt pairs → build AlignmentLink[] with origin: 'eflomal',
   status: 'suggested'. Per AGENTS.md this needs 100% test coverage; the subprocess will
   need a mock alongside the existing PAPI mocks in __mocks__/.
4. Command in main.ts, e.g. interlinearizer.suggestAlignments.
5. Persistence gap: updateAnalysis currently writes only the analysis field, not
   links (projectStorage.ts). Saving suggestions
   requires a links write path (new method or extend the existing one).
6. Redux + UI: actions to hold links, and UI to render suggestions over the
   source/target token rows with approve/reject/refine. (No alignment-link UI exists yet —
   only PhraseAnalysisLink grouping, which is unrelated.)

6. Blockers / decisions to resolve before coding

• License conflict (must resolve first). eflomal is GPLv3
  (../eflomal/LICENSE); this extension is MIT (manifest.json).
  Bundling/redistributing a GPLv3 binary with an MIT extension has real implications.
  Spawning it as a separate process (vs. linking) is the lighter-touch form of use, but
  redistribution still needs legal/licensing sign-off from SIL.
• Binary provenance for all 3 OSes. Building Windows/macOS binaries with OpenMP (and a
  portable -march, per §5.2), and keeping them in sync with the pinned eflomal commit,
  needs a real build pipeline.
• Quality / UX. eflomal needs reasonable sentence counts to align well; single-verse
  alignment in isolation may be weak. Consider aligning a whole book (or using --priors
  trained on a larger corpus) and whether suggestions are book-batch or on-demand. Worth a
  user-questions.md entry per AGENTS.md (UX decision affecting people outside dev).
• Symmetrization. Forward+reverse only; if we want grow-diag-final-and we either bundle
  atools (another dependency) or implement symmetrization ourselves.

7. Bottom line

Technically very feasible and the data model is ready for it. The engineering is bounded
(Option A). The gating items are licensing and per-platform binary infrastructure —
both organizational rather than code problems, and both should be answered before
investing in the implementation.