Generate extraction CONSTRUCTs from the search schema

[ddeboer]

Retire the hand-written dr:* CONSTRUCTs by generating extraction from the search schema. Epic: #534.

Problem

The Dataset Register’s search extraction uses hand-written dr:* CONSTRUCTs (register-source.ts / dkg-source.ts) that the projection frames against. Every new record type needs another hand-written CONSTRUCT, and the CONSTRUCTs can silently drift from the SearchSchema. They already do: register-source.ts CONSTRUCTs dr:dateRead, which no field reads, and nothing catches it.

The object grain (#534) is the forcing function – without this, indexing SCHEMA-AP objects means writing a fresh pile of dr:-style CONSTRUCTs.

What the generator reads

The TypeScript SearchSchema, not a SHACL shapes graph. SearchField.path is a plain path, not sh:path; SHACL stays one possible upstream source of a SearchSchema (#325), never a dependency. (This issue previously said “SHACL-driven”; that was aspirational.)

The dr: namespace cannot be removed – and that was the wrong target

SPARQL forbids a property path as a CONSTRUCT verb (a ConstructTemplate’s verb is VarOrIri), so flattening a multi-hop value onto its subject must mint a predicate for it – there is no legal triple that says ?dataset dct:publisher/foaf:name ?name. And a field that projects flat needs the value flat: the projection reads a flat IR keyed by alias, so a nested dct:publisher node would not answer publisherName.

Both reasons are properties of SPARQL and of the projection. Neither depends on an engine quirk.

So dr: is not a workaround – it is the inevitable output alias of a flattening CONSTRUCT. What this issue retires is its hand-authorship: the alias becomes a mechanical function of the field name (urn:lde:‹Type›/‹field›), minted by the generator, read back by the projection, never written by a human, never a public vocabulary. Field names are already unique per type and already restricted to [A-Za-z_][A-Za-z0-9_]*, so the alias needs no escaping; qualifying by type name matters because one subject can be a root of two types (frame-by-type.ts).

The function that mints it belongs in @lde/search/adapter beside physicalFields, for the reason that JSDoc already gives: one home for a naming convention that two sides must agree on.

Scope

A read-side adapter (@lde/search-sparql-style) generating a type’s extraction from its declaration:

• template: ?s <urn:lde:‹Type›/‹field›> ?v per field – IRIs only, single subject.
• WHERE: one UNION branch per field (?s <path> ?v), path fed in literally as a SPARQL property path. UNION per branch, not conjunction – register-source.ts measured ~4× result inflation from the cross-product.
• roots: VALUES ?s { … } via the pipeline’s existing injectValues.
• inline references: one CONSTRUCT with a nested template ({ ?s <…/registration> ?r . ?r <…/datePosted> ?v }) – the obvious shape, and the default. A two-subject template was reported to lose the ?s → ?r link on QLever (register-source.ts), which would force one single-subject CONSTRUCT per hop level instead, each re-walking the reference path in its WHERE to bind its subject. That report does not reproduce (see Acceptance), so it is a contingency, not the design: build the nested template, and split only if the claim survives verification at scale.

Plus:

• The projection reads node[alias] instead of node[field.path]; path becomes purely source-side and opaque to @lde/search (the reader adapter owns its grammar).
• A schema↔CONSTRUCT contract test: the alias set the generator mints is exactly the field set the projection reads.
• Retire register-source.ts / dkg-source.ts once the generator covers them.

Path expressiveness: A ∧ B, not A vs B

The previous “Option A (refs are enough) vs Option B (sequence paths, deferred)” framing was a false choice. Both are needed, and together they are total:

• Property paths flatten simple deep values – publisherName is dct:publisher/foaf:name, format is dcat:distribution/dcat:mediaType, organization is dct:publisher|dct:creator. Sequence, alternative and inverse cover 8 of the 14 dr: predicates. (isIRI is free: kind: 'reference' already reads only IRIs.)
• Inline references (Implement inline references and class-less reference types #608) reach values behind a qualified hop, which no path can express – inline the intermediate node with its discriminator and let a derive select. Covers the DQV block (five fields, one path), the IIIF subsets, and the terminology-source linkset.
• Aggregate selection is not a path problem either: newest-registration-wins is a derive over an inlined ^schema:about array.

With both, no extraction needs an escape hatch and no path grammar grows into a query language.

Not in scope

• Root selection. Which subjects are roots is a deployment concern, supplied as Stage.itemSelector and bound through VALUES, defaulting to ?x a <class> – which the object grain gets for free. It is not derivable from the schema: the register does not type its datasets at all (register-source.ts mints a dcat:Dataset), and the three label-source types have no source type either (label-collections.ts injects one). SearchType.class is the IR/API class a type’s documents are instances of, not a selector into the source.
• Memory (Project inside the batch, not at the dataset flush (implement ADR 13) #606). Independent: the generator neither fixes it nor needs it.

Acceptance

• Object and catalog extraction generated from SearchField.path – no hand-written intermediate predicate anywhere in the pipeline.
• Every IR key the projection reads is minted from the schema; a contract test fails on drift.
• The QLever two-subject-template claim is settled before the query shape is built, and the outcome is pinned by a test rather than a comment. register-source.ts (committed on DR’s main, written 2026-06-19/21) asserts that a template spanning two subjects drops the link, and that a UNION mixing named graphs silently drops branches. Neither reproduces – tested on adfreiburg/qlever:commit-dbce463 (the build DR pinned when the claims were written) and on commit-a14e0a0 (LDE’s pin), with the two-subject template returning all links intact, including inside GRAPH ?dataset. A third claim in the same JSDoc, that QLever does not deduplicate CONSTRUCT output, does reproduce on both builds. Notably the commit that introduced all three is titled “drop the cross-product”, and only the cross-product carries a measurement.
  • What remains untested is scale and query complexity: the fixture is 10 quads, while DR’s real propertiesQuery has 9 UNION branches and factsQuery a MAX sub-SELECT with FILTER NOT EXISTS over the full register. Reproduce at that shape and scale.
  • If it does not reproduce: build the nested template, and correct register-source.ts’s JSDoc so the claim stops propagating.
  • If it does: split per hop level and pin it with a regression test naming the QLever version, so the workaround is not inherited on faith the way it nearly was here.
  • The confirmed dedup quirk stays: keep deduplication, and give it one owner (Project inside the batch, not at the dataset flush (implement ADR 13) #606 makes both copies batch-scoped, so the framer’s can go and the reader’s stays). Document the survivors in @lde/sparql-qlever, the one package that names QLever.
  • But the generator may retire the manual dedup entirely – verify. QLever duplicates a CONSTRUCT result only when the template projects away WHERE variables (a constant/shared triple like ?s a <Type> instantiated once per WHERE solution). This generator removes exactly that trigger: roots are given (from the itemSelector), so the CONSTRUCT no longer emits ?s a <Type>, and UNION-per-property, single-subject templates are ?s <alias> ?v with ?v bound per solution – every emitted triple is distinct. So a well-formed generated CONSTRUCT should produce duplicate-free output even on QLever. Measure it (at real scale/complexity, per above); if it holds, the framer’s dedup can go. The one source it does not cover, and which must be handled separately, is multi-reader merge: a batch that merges two readers (DR’s register + DKG) can carry the same triple from both, independent of any single query’s shape – that dedup stays wherever the merge happens. Count-style derives (e.g. irisOf(node, …).length) are the correctness canary: a leaked duplicate is a silently wrong number, not just wasted work, so any decision to drop dedup must be proven, not assumed. Interlocks with Research: can @lde/search drop JSON-LD framing for a group-by-subject builder? #626 (a group-by-subject framer wants to know its input is already duplicate-free).

Depends on

• Close the derive hole: make path the whole statement of what the projection reads #607 – path must be the whole statement of what the projection reads. 13 of the Dataset Register’s 23 Dataset predicates are currently readable only from inside a derive closure, so a generator would miss them and no contract test could see them.
• Implement inline references and class-less reference types #608 – inline references and class-less reference types.

Design context and the object grain: #534. Upstream: #325.

Tracking note. #540 (the LDE tracker for the DR consumer side) was closed as completed on 2026-07-09, though its body says the work “lands after the @lde/* packages publish” and that “the implementation issue lives in the DR repo”; no successor is currently open in either repo. Do not read #540 as live work.