[NIDX-01D] Cut over exact-term schema walks
Parent workstream: #14002
Blocked by: #14010
Summary
Extend the native stored-document visitor with exact term and exact-term OR selection, then move metadata/schema/reader.WalkShard and WalkDocs to it. This slice adds the ICE term dictionary and postings intersection needed by schema kinds; it does not add ranges, prefix/wildcard expansion, analyzers, or sorting.
Blocked by
NIDX-01C (#14010) must be merged before this work starts. Before applying Backlog, update identifier names in this body if NIDX-01C's merged visitor uses different names; do not change the behavioral seam.
Pre-agreed test seam
The product seam is the existing:
func reader.WalkShard(shardPath string, visit func(reader.Doc) error, kinds ...schema.Kind) error
func reader.WalkDocs(schemaRoot string, visit func(reader.Doc) error, kinds ...schema.Kind) error
The native visitor may gain exactly one bounded “field equals any of these literal terms” input. Do not expose a general query language, FST type, posting iterator, or analyzer. Boundary tests observe WalkShard/WalkDocs documents and errors, not dictionary internals.
Independent fixture
The legacy schema corpus contains:
p1, kind Stream, revisions 1 and 2;
p2, kind Measure, revision 1;
p3, kind Stream, revision 1 followed by a tombstone at revision 2;
p4, kind Group, revision 1; and
p5, kind Measure, with deliberately malformed stored _source bytes.
For kinds {Stream}, the result is only p1@2. For {Stream, Group}, results are p1@2 and p4@1. p3 is absent. A Stream-only walk must not decode p5; a Measure walk must return the native typed stored-value error for it.
Requirements
R1. Exact lookup resolves only the requested field/term. Exact-term OR unions the requested postings, applies the selected snapshot's deletions, and visits each live document once.
R2. WalkShard and WalkDocs return exactly the fixture results above, preserving _source bytes, latest-revision selection, and tombstone suppression.
R3. Filtering happens before stored-field decode: Stream-only load succeeds despite corrupt Measure p5; selecting Measure reports the typed error.
R4. Term count, term length, dictionary enumeration, posting decode, and cancellation are bounded. CRC32 remains ignored and no reader lock or directory mutation is introduced.
RED and end-to-end contract
- Boundary RED: exercise exact term and two-term OR through the bounded native visitor inherited from NIDX-01C; the compile-time or unsupported-operation failure proves the new slice is RED.
- E2E RED: run
WalkDocs/schema loaders for Stream, Stream+Group, and corrupt Measure against the checked-in legacy corpus and assert the declared results and typed failure. The command must fail before the production caller is switched.
Acceptance criteria
go test ./pkg/index/inverted/... ./banyand/metadata/schema/reader -run 'Test(NativeExactTerms|E2ESchemaWalkNative)' -count=1 passes.
- A call-graph/dependency assertion proves schema walks no longer build or execute Bluge queries.
- Exact, OR, deletion, corrupt-unselected-field, cancellation, and resource-limit cases pass under
-race.
make test-ci PKG='./pkg/index/inverted/... ./banyand/metadata/schema/reader', make build, and make pre-push pass.
Scope
Packages: pkg/index/inverted, its private ICE dictionary/posting reader, and banyand/metadata/schema/reader.
Out of scope: numeric/date range, prefix, wildcard, MATCH, NOT, sorting, doc values, Property repair, and every writer.
Compatibility and rollback
This slice reads historical bytes and emits none. Roll back schema readers to the retained Bluge path without changing any directory.
Lexical non-regression
Repository changes for every implementation leaf before the final removal may delete existing references but must add zero new case-insensitive bluge tokens and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral legacy oracle or compatibility writer labels plus an immutable revision or content hash rather than adding a retired module name.
This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.
Design
BDB-NIDX-SPEC-001 revision 0.2 — NIDX-01
[NIDX-01D] Cut over exact-term schema walks
Parent workstream: #14002
Blocked by: #14010
Summary
Extend the native stored-document visitor with exact term and exact-term OR selection, then move
metadata/schema/reader.WalkShardandWalkDocsto it. This slice adds the ICE term dictionary and postings intersection needed by schema kinds; it does not add ranges, prefix/wildcard expansion, analyzers, or sorting.Blocked by
NIDX-01C (#14010) must be merged before this work starts. Before applying
Backlog, update identifier names in this body if NIDX-01C's merged visitor uses different names; do not change the behavioral seam.Pre-agreed test seam
The product seam is the existing:
The native visitor may gain exactly one bounded “field equals any of these literal terms” input. Do not expose a general query language, FST type, posting iterator, or analyzer. Boundary tests observe
WalkShard/WalkDocsdocuments and errors, not dictionary internals.Independent fixture
The legacy schema corpus contains:
p1, kind Stream, revisions1and2;p2, kind Measure, revision1;p3, kind Stream, revision1followed by a tombstone at revision2;p4, kind Group, revision1; andp5, kind Measure, with deliberately malformed stored_sourcebytes.For kinds
{Stream}, the result is onlyp1@2. For{Stream, Group}, results arep1@2andp4@1.p3is absent. A Stream-only walk must not decodep5; a Measure walk must return the native typed stored-value error for it.Requirements
R1. Exact lookup resolves only the requested field/term. Exact-term OR unions the requested postings, applies the selected snapshot's deletions, and visits each live document once.
R2.
WalkShardandWalkDocsreturn exactly the fixture results above, preserving_sourcebytes, latest-revision selection, and tombstone suppression.R3. Filtering happens before stored-field decode: Stream-only load succeeds despite corrupt Measure
p5; selecting Measure reports the typed error.R4. Term count, term length, dictionary enumeration, posting decode, and cancellation are bounded. CRC32 remains ignored and no reader lock or directory mutation is introduced.
RED and end-to-end contract
WalkDocs/schema loaders for Stream, Stream+Group, and corrupt Measure against the checked-in legacy corpus and assert the declared results and typed failure. The command must fail before the production caller is switched.Acceptance criteria
go test ./pkg/index/inverted/... ./banyand/metadata/schema/reader -run 'Test(NativeExactTerms|E2ESchemaWalkNative)' -count=1passes.-race.make test-ci PKG='./pkg/index/inverted/... ./banyand/metadata/schema/reader',make build, andmake pre-pushpass.Scope
Packages:
pkg/index/inverted, its private ICE dictionary/posting reader, andbanyand/metadata/schema/reader.Out of scope: numeric/date range, prefix, wildcard, MATCH, NOT, sorting, doc values, Property repair, and every writer.
Compatibility and rollback
This slice reads historical bytes and emits none. Roll back schema readers to the retained Bluge path without changing any directory.
Lexical non-regression
Repository changes for every implementation leaf before the final removal may delete existing references but must add zero new case-insensitive
blugetokens and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutrallegacy oracleorcompatibility writerlabels plus an immutable revision or content hash rather than adding a retired module name.This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.
Design
BDB-NIDX-SPEC-001 revision 0.2 — NIDX-01