feat(search)!: generate extraction CONSTRUCTs from the search schema

AGENTS.md

@@ -82,11 +82,11 @@ Each package uses conditional exports with a `development` condition for local d
 - Test files use `.test.ts` suffix in `test/` directory
 - Fixtures in `test/fixtures/`
 - HTTP mocking with Nock
-- Tests that start a local SPARQL endpoint (`@lde/local-sparql-endpoint`) must use unique ports across packages to avoid conflicts when Nx runs tests in parallel. Current port allocations: `dataset-registry-client` (3002), `pipeline` sparqlQuery (3001), `pipeline` executor (3003), `pipeline` provenance store (3004), `pipeline-void` namespace-normalization (3005–3006)
+- Tests that start a local SPARQL endpoint (`@lde/local-sparql-endpoint`) must use unique ports across packages to avoid conflicts when Nx runs tests in parallel. Current port allocations: `dataset-registry-client` (3002), `pipeline` sparqlQuery (3001), `pipeline` executor (3003), `pipeline` provenance store (3004), `pipeline-void` namespace-normalization (3005–3006), `search-pipeline` extraction round-trip (3007)
 
 ### Key Dependencies
 
-- RDF: `n3`, `sparqljs`, `jsonld`
+- RDF: `n3`, `jsonld`
 - Query engines: `@comunica/query-sparql-file`, `ldkit`
 - CLI packages use Commander
 

packages/search-pipeline/README.md

@@ -21,7 +21,12 @@ The division of labour ([ADR 6](../../docs/decisions/0006-make-the-writer-transa
     **root-complete batch** into documents paired with their `SearchType`
     (`TypedSearchDocument`) – so projection happens inside the batch and memory
     is bounded by `batchSize` roots, not the dataset
-    ([ADR 13](../../docs/decisions/0013-project-inside-the-batch-per-root-type.md));
+    ([ADR 13](../../docs/decisions/0013-project-inside-the-batch-per-root-type.md)).
+    Extraction is **generated from the schema**: `extractionQuery` mints a
+    CONSTRUCT that reads each field’s source `path` (a SPARQL property path) and
+    emits it under the field’s IR Alias (`urn:lde:‹Type›/‹field›`) – the same key
+    the projection reads back – so the reader and the projection cannot drift. A
+    stage without an explicit `readers` defaults to this generated reader;
   - `searchIndexWriter` is the pipeline’s single terminal: an engine-agnostic
     router that dispatches each document to the engine writer for **its**
     type’s collection ([ADR 9](../../docs/decisions/0009-route-a-whole-schema-projection-to-per-type-collections.md)).
@@ -32,7 +37,7 @@ So a search pipeline is **one terminal, N stages**: `new Pipeline<TypedSearchDoc
 ## Usage
 
 ```typescript
-import { Pipeline, SparqlConstructReader } from '@lde/pipeline';
+import { Pipeline } from '@lde/pipeline';
 import { searchSchema } from '@lde/search';
 import { BlueGreenRebuild } from '@lde/search-typesense';
 import {
@@ -83,22 +88,23 @@ const pipeline = new Pipeline<TypedSearchDocument>({
   // One stage per root type: each selects its own roots (here by class –
   // `selectByClass` is a convenience for the object grain; a deployment writes
   // its own selector where the entry point is a domain fact) and extracts each
-  // root’s quads. `rootVariable` couples the selector’s projected variable to
-  // what the projection reads; it must not be `dataset` (reserved by the reader).
+  // root’s quads. Omitting `readers` defaults each stage to the Extraction
+  // CONSTRUCT generated from the schema (pass one only for a non-SPARQL source or
+  // to merge readers). `rootVariable` couples the selector’s projected variable
+  // to the CONSTRUCT’s free subject; it must not be `dataset` (reserved by the
+  // reader).
   stages: searchStages({
     schema,
     types: [
       {
         searchType: dataset,
         rootVariable: 'root',
         itemSelector: selectByClass(dataset),
-        readers: new SparqlConstructReader({ query: '…' }),
       },
       {
         searchType: organization,
         rootVariable: 'root',
         itemSelector: selectByClass(organization),
-        readers: new SparqlConstructReader({ query: '…' }),
       },
     ],
   }),
@@ -180,3 +186,37 @@ across a dataset or across datasets, so the same pipeline indexes a 15-quad
 catalog entry and a dataset of millions of objects. The one irreducible atom is a
 single root’s quads: unbounded for a pathological root, and nothing here changes
 that.
+
+## Extraction queries and non-deduplicating engines
+
+The generated extraction CONSTRUCTs are deliberately shaped to be well-formed for
+**non-deduplicating** SPARQL engines such as
+[QLever](https://github.com/ad-freiburg/qlever), which emit one copy of the
+CONSTRUCT template per solution row rather than a deduplicated set. The generator
+guarantees one output triple per genuine value:
+
+- **UNION per field, never conjunction.** Each field is its own `UNION` branch
+  (`?root <path> ?value`), so two independent multi-valued fields never form a
+  cross-product – the failure mode that inflates a non-deduplicating engine’s
+  output by the product of their cardinalities (roughly 18-fold on a measured
+  QLever workload).
+- **No projected-away template triple.** Every template triple binds its own
+  variables (`?root <alias> ?value`); there is no constant triple (e.g.
+  `?root a <Type>`) to be re-emitted once per solution.
+- **Given roots.** Roots arrive as a `VALUES` set, not a pattern that fans out.
+- **Inline references keep the discipline recursively:** a reference type’s
+  fields are UNION’d off the referent variable, so even a multi-hop nested
+  template never conjoins independent multi-valued fields – only the intermediate
+  link triple repeats, and only linearly.
+
+Because the queries are duplicate-free by construction, correctness and bounded
+output volume do **not** depend on a client-side **post-processing deduplication
+step** – buffering a whole result to deduplicate it would defeat the
+batch-bounded streaming memory model above. The only residual duplication is
+linear and comes from duplicate _input_ – a non-`DISTINCT` selector feeding a
+multi-typed root more than once – which the streaming per-quad subject index
+(`buildSubjectIndex`) collapses as a cheap backstop, not as a compensator for
+query-shape inflation. Validated on a real QLever index over the full source
+dump: raw CONSTRUCT output equalled the distinct set for a type with distinct
+roots, and the projected documents were byte-identical to those from a
+deduplicating engine.

packages/search-pipeline/package.json

@@ -26,6 +26,9 @@
   ],
   "dependencies": {
     "@lde/search": "^0.11.0",
+    "@traqula/generator-sparql-1-1": "^1.1.8",
+    "@traqula/parser-sparql-1-1": "^1.1.5",
+    "@traqula/rules-sparql-1-1": "^1.1.0",
     "tslib": "^2.3.0"
   },
   "devDependencies": {

packages/search-pipeline/src/extraction.ts

@@ -0,0 +1,189 @@
+import {
+  AstFactory,
+  type Pattern,
+  type PatternBgp,
+  type PatternGroup,
+  type QueryConstruct,
+  type QuerySelect,
+  type TermVariable,
+  type TripleNesting,
+} from '@traqula/rules-sparql-1-1';
+import { Parser } from '@traqula/parser-sparql-1-1';
+import { Generator } from '@traqula/generator-sparql-1-1';
+import {
+  irAlias,
+  isInlineReference,
+  referenceTypeNamed,
+} from '@lde/search/adapter';
+import type { SearchSchema, SearchType } from '@lde/search';
+
+const factory = new AstFactory();
+const parser = new Parser();
+const generator = new Generator();
+
+/** Options for {@link extractionQuery}. */
+export interface ExtractionOptions {
+  /**
+   * The variable the roots bind to. Left free in the generated query for the
+   * pipeline’s VALUES injection (`injectValues`), so it must match the stage’s
+   * `rootVariable` – the variable the item selector projects. Defaults to
+   * `root`, matching `selectByClass`. Must not be `dataset` (the SPARQL reader
+   * substitutes `?dataset` with the dataset IRI).
+   * @default 'root'
+   */
+  readonly subjectVariable?: string;
+}
+
+/**
+ * Generate a type’s **Extraction** CONSTRUCT from its {@link SearchType}
+ * declaration: the query whose output the projection frames and reads. Pure –
+ * `SearchType → QueryConstruct` – with no engine or deployment knowledge.
+ *
+ * - **template**: one `?root <`{@link irAlias IR Alias}`> ?value` triple per
+ *   path-bearing field – IRIs only, a single subject. A property path cannot be
+ *   a CONSTRUCT verb, so the flattened value must be minted onto its subject
+ *   under the field’s alias; the projection reads it back under the same alias.
+ * - **WHERE**: one UNION branch per field (`?root <path> ?value`), the field’s
+ *   `path` embedded as a literal SPARQL property path (sequence / alternative /
+ *   inverse). UNION per branch, never a conjunction – conjoining every path in
+ *   one BGP cross-multiplies the multi-valued ones (~4× inflation).
+ * - **roots**: the subject variable is left free, to be bound by the pipeline’s
+ *   `injectValues` (`VALUES ?root { … }`). Root *selection* is a separate,
+ *   deployment concern (`selectByClass` / the stage’s `itemSelector`); this
+ *   generator emits IR Aliases, the selector queries the source class.
+ * - **inline references** ({@link isInlineReference}): one CONSTRUCT with a
+ *   nested template (`{ ?root <…/ref> ?r . ?r <…/field> ?v }`), recursing into
+ *   the reference type to the schema’s declared depth. The referent-binding hop
+ *   uses the source `path`; the emitted triples use the minted aliases.
+ *
+ * Wire the result into a `SparqlConstructReader` (see `searchStages`), which
+ * runs it per batch with the roots injected as VALUES.
+ */
+export function extractionQuery(
+  searchType: SearchType,
+  schema: SearchSchema,
+  options: ExtractionOptions = {},
+): QueryConstruct {
+  const subjectVariable = options.subjectVariable ?? 'root';
+  const subject = factory.termVariable(subjectVariable, factory.gen());
+  const built = buildFor(searchType, subject, schema, { next: 0 });
+  if (built.branches.length === 0) {
+    throw new Error(
+      `Cannot generate an extraction CONSTRUCT for “${searchType.name}”: it declares no path-bearing field, so there is nothing to extract.`,
+    );
+  }
+  const where = factory.patternGroup(
+    [factory.patternUnion(built.branches, factory.gen())],
+    factory.gen(),
+  );
+  return factory.queryConstruct(
+    factory.gen(),
+    [],
+    factory.patternBgp(built.template, factory.gen()),
+    where,
+    {},
+    factory.datasetClauses([], factory.gen()),
+  );
+}
+
+/** {@link extractionQuery}, serialised to a SPARQL string a reader can run. */
+export function extractionQueryString(
+  searchType: SearchType,
+  schema: SearchSchema,
+  options?: ExtractionOptions,
+): string {
+  return generator.generate(extractionQuery(searchType, schema, options));
+}
+
+/** A per-query counter minting distinct value/referent variable names. */
+interface VariableCounter {
+  next: number;
+}
+
+/** The template triples and WHERE branches a type contributes off `subject`. */
+interface Built {
+  readonly template: TripleNesting[];
+  readonly branches: PatternGroup[];
+}
+
+/**
+ * Walk a type’s path-bearing fields off a subject variable, collecting the
+ * template triples (minted aliases) and one WHERE branch per field (source
+ * paths). Recurses into an inline reference’s type off a fresh referent
+ * variable, so the nested template keeps the `subject → referent → value` link
+ * in one CONSTRUCT. The recursion terminates because `searchSchema` rejects
+ * inline reference cycles.
+ */
+function buildFor(
+  searchType: SearchType,
+  subject: TermVariable,
+  schema: SearchSchema,
+  counter: VariableCounter,
+): Built {
+  const template: TripleNesting[] = [];
+  const branches: PatternGroup[] = [];
+  for (const field of searchType.fields) {
+    if (field.path === undefined) {
+      continue;
+    }
+    const alias = factory.termNamed(factory.gen(), irAlias(searchType, field));
+    const sourcePath = liftPath(field.path);
+    if (isInlineReference(field)) {
+      const referenceType = referenceTypeNamed(schema, field.ref.typeName);
+      if (referenceType === undefined) {
+        continue;
+      }
+      const referent = factory.termVariable(
+        `r${counter.next++}`,
+        factory.gen(),
+      );
+      template.push(factory.triple(subject, alias, referent));
+      const nested = buildFor(referenceType, referent, schema, counter);
+      template.push(...nested.template);
+      const patterns: Pattern[] = [
+        factory.patternBgp(
+          [factory.triple(subject, sourcePath, referent)],
+          factory.gen(),
+        ),
+      ];
+      // A single-branch union serialises to just its group, so only add the
+      // nested union when the reference type has fields of its own.
+      if (nested.branches.length > 0) {
+        patterns.push(factory.patternUnion(nested.branches, factory.gen()));
+      }
+      branches.push(factory.patternGroup(patterns, factory.gen()));
+    } else {
+      const value = factory.termVariable(`v${counter.next++}`, factory.gen());
+      template.push(factory.triple(subject, alias, value));
+      branches.push(
+        factory.patternGroup(
+          [
+            factory.patternBgp(
+              [factory.triple(subject, sourcePath, value)],
+              factory.gen(),
+            ),
+          ],
+          factory.gen(),
+        ),
+      );
+    }
+  }
+  return { template, branches };
+}
+
+/**
+ * Lift a field’s `path` – written in the SPARQL reader adapter’s grammar (a
+ * property path) – into a predicate AST node, by parsing it inside a throwaway
+ * query and taking the verb. A single IRI yields a plain named node; a
+ * sequence / alternative / inverse yields a path node. The WHERE consumes it;
+ * the CONSTRUCT template never does (a path cannot be a template verb).
+ */
+function liftPath(path: string): TripleNesting['predicate'] {
+  // Always a single-triple SELECT by construction, so the verb is at a fixed
+  // spot; a malformed `path` makes the parser itself throw, which is the right
+  // failure for invalid reader-adapter grammar.
+  const ast = parser.parse(`SELECT * WHERE { ?s ${path} ?o }`) as QuerySelect;
+  const bgp = ast.where.patterns[0] as PatternBgp;
+  const triple = bgp.triples[0] as TripleNesting;
+  return triple.predicate;
+}

packages/search-pipeline/src/index.ts

@@ -1,5 +1,7 @@
 export { searchStages, selectByClass } from './search-stages.js';
 export type { SearchStagesOptions, SearchStageType } from './search-stages.js';
+export { extractionQuery, extractionQueryString } from './extraction.js';
+export type { ExtractionOptions } from './extraction.js';
 export { searchIndexWriter } from './search-index-writer.js';
 export type { SearchIndexWriterOptions } from './search-index-writer.js';
 export type { TypedSearchDocument } from './typed-search-document.js';

packages/search-pipeline/src/search-stages.ts

@@ -1,10 +1,12 @@
 import {
+  SparqlConstructReader,
   SparqlItemSelector,
   Stage,
   type ItemSelector,
   type StageReaders,
 } from '@lde/pipeline';
 import { projectRoots, type RootType, type SearchSchema } from '@lde/search';
+import { extractionQueryString } from './extraction.js';
 import type { TypedSearchDocument } from './typed-search-document.js';
 
 /** One root type’s stage in a search pipeline. */
@@ -30,8 +32,16 @@ export interface SearchStageType {
    * convenience for the object grain, not a default.
    */
   itemSelector: ItemSelector;
-  /** Reader(s) that extract each selected root’s quads. */
-  readers: StageReaders;
+  /**
+   * Reader(s) that extract each selected root’s quads. Defaults to a single
+   * {@link SparqlConstructReader} running the **Extraction** CONSTRUCT generated
+   * from `searchType` ({@link extractionQuery}), with `rootVariable` as the free
+   * subject the batch’s roots bind to – the schema-derived reader the projection
+   * is guaranteed to agree with (they share the {@link irAlias} convention).
+   * Supply your own only to read from a non-SPARQL source, merge several
+   * readers, or attach transforms.
+   */
+  readers?: StageReaders;
   /**
    * Roots (and so documents) per batch – the memory bound. Under a root-bound
    * selector it moves memory and request count, never output.
@@ -88,9 +98,19 @@ export function searchStages(
       );
     }
     const { rootVariable } = type;
+    // Default to the Extraction CONSTRUCT generated from the schema, its subject
+    // left free for the batch’s VALUES injection. The reader and the projection
+    // then agree by construction: both key off the same IR Aliases.
+    const readers =
+      type.readers ??
+      new SparqlConstructReader({
+        query: extractionQueryString(searchType, schema, {
+          subjectVariable: rootVariable,
+        }),
+      });
     return new Stage<TypedSearchDocument>({
       name: searchType.name,
-      readers: type.readers,
+      readers,
       itemSelector: type.itemSelector,
       batchSize: type.batchSize,
       maxConcurrency: type.maxConcurrency,