Project inside the batch, not at the dataset flush (implement ADR 13)

[ddeboer]

Implement ADR 13 – Project inside the batch, per root type (docs/decisions/0013-project-inside-the-batch-per-root-type.md).

The design, the reasoning, and every decision it touches live in the ADR. This issue is the work.

Problem

searchIndexWriter buffers every quad of a dataset (search-index-writer.ts:150-152), then every projected document (byType, 119-130), before writing any of them. Memory is O(dataset), not O(batch) – ADR 12’s known violation.

That is a real bound for the catalog grain, where a “dataset” is a ~15-quad description. For the object grain one dataset is the whole input: millions of objects. Same code, same sentence, and the bound evaporates because the unit changed meaning underneath it.

Both ends of the pipe already stream – AsyncQueue is bounded and back-pressured, BatchImporter holds only batchSize documents – and searchIndexWriter materializes between them, handing the streaming importer stream(documents): a simulated stream over a fully materialized array.

Slices

Land in this order. Each ships alone.

1. @lde/search: projectRoots + explicit roots. Additive; projectGraph stays for now. projectRoots(quads, roots, schema, searchType) calling assertTypeInSchema; buildSubjectIndex(quads) loses rootTypes; frameSubjects(index, roots); delete rootsByType and frameByType. Covered by test/frame-by-type.test.ts and test/project.test.ts. No pipeline change.
2. @lde/pipeline: Stage<Out = Quad> + project, and Pipeline<Out = Quad>. The seam is ~40 lines at stage.ts:353-357. project requires itemSelector, forbids stages, and excludes the stage from beforeStageWrite; a pipeline that projects must also reject beforeDatasetWrite at construction, since that one wraps the run writer (pipeline.ts:476), not a stage. AsyncQueue’s capacity becomes a StageOptions knob (it is hard-coded to 128 today and never passed – stage.ts:273). Threading is narrow: PipelineOptions.stages/writers, FanOutWriter/FanOutRunWriter, stageWriter, runStages/runStage. The QuadTransform surface stays Quad. Provable against a fake Writer<string> with no search dependency. This is the slice that needs careful review.
3. @lde/pipeline: correct RunWriter’s JSDoc. writer.ts:59 claims a run is “bracketed by exactly one commit or abort” and writer.ts:95 that commit is “called exactly once”. pipeline.ts:479-487 already aborts a run whose commit throws, so both are false today. Docs only; independently correct.
4. @lde/search-pipeline: searchStages(). Move TypedSearchDocument here (it exists because a terminal routes – glue, not projection). searchIndexWriter becomes Writer<TypedSearchDocument>: keeps ADR 9’s routing and the run lifecycle, loses projection and buffering. Delete projectGraph. Breaking, but a small diff once 1–2 have landed.
5. The counting test – see Acceptance.
6. Dataset Register migration – separate repo, separate issue. Four selectors, one per root type; its rdf:type injection and minted a dcat:Dataset become unnecessary.

Acceptance

Memory is bounded by batchSize – asserted by counting, not measuring. Run a synthetic input at 10 roots and 10,000 roots, batchSize: 10, against a recording fake Writer, and assert:

1. project is invoked ceil(roots / batchSize) times;
2. the largest quads.length handed to project is identical at both input sizes;
3. the writer’s peak live document count is identical at both input sizes.

(2) and (3) are assertions about a bound: they fail deterministically the moment a structure starts growing with input, and they fail against today’s searchIndexWriter, whose peak is every document. A process.memoryUsage() assertion is both flakier and weaker – GC timing can hide a leak that a count cannot.

Plus: no behavioural regression in LDE’s own tests and fixtures – the same documents land in the same collections.

This does not mean “the Dataset Register keeps working”. The change is breaking by construction (searchIndexWriter stops projecting), and DR is updated separately. LDE is pre-release: do not contort the design to preserve an API we have decided to break.

A contract the code cannot enforce

Root-boundedness is a promise @lde/pipeline cannot check. A selector that does not bind the CONSTRUCT’s subject yields a batch that is not root-complete, and projection will then silently emit partial documents. Document it alongside expectsOutput.

Not in scope

• Close the derive hole: make path the whole statement of what the projection reads #607 (the derive hole) and Implement inline references and class-less reference types #608 (inline references). Internal fields will look necessary the moment you are inside projection. They are not needed for this, and they are separately breaking.
• Generate extraction CONSTRUCTs from the search schema #548 (the extraction generator). Independent both ways: this needs no generator, and the generator does not fix this.

Relates to

Design and reasoning: ADR 13. Invariant: ADR 12. Object grain and the epic: #534. ADR 9’s fan-out is untouched.