Skip to content

[NIDX-01D] Cut over exact-term schema walks #14011

Description

@hanahmily

[NIDX-01D] Cut over exact-term schema walks

Parent workstream: #14002
Blocked by: #14010

Summary

Extend the native stored-document visitor with exact term and exact-term OR selection, then move metadata/schema/reader.WalkShard and WalkDocs to it. This slice adds the ICE term dictionary and postings intersection needed by schema kinds; it does not add ranges, prefix/wildcard expansion, analyzers, or sorting.

Blocked by

NIDX-01C (#14010) must be merged before this work starts. Before applying Backlog, update identifier names in this body if NIDX-01C's merged visitor uses different names; do not change the behavioral seam.

Pre-agreed test seam

The product seam is the existing:

func reader.WalkShard(shardPath string, visit func(reader.Doc) error, kinds ...schema.Kind) error
func reader.WalkDocs(schemaRoot string, visit func(reader.Doc) error, kinds ...schema.Kind) error

The native visitor may gain exactly one bounded “field equals any of these literal terms” input. Do not expose a general query language, FST type, posting iterator, or analyzer. Boundary tests observe WalkShard/WalkDocs documents and errors, not dictionary internals.

Independent fixture

The legacy schema corpus contains:

  • p1, kind Stream, revisions 1 and 2;
  • p2, kind Measure, revision 1;
  • p3, kind Stream, revision 1 followed by a tombstone at revision 2;
  • p4, kind Group, revision 1; and
  • p5, kind Measure, with deliberately malformed stored _source bytes.

For kinds {Stream}, the result is only p1@2. For {Stream, Group}, results are p1@2 and p4@1. p3 is absent. A Stream-only walk must not decode p5; a Measure walk must return the native typed stored-value error for it.

Requirements

R1. Exact lookup resolves only the requested field/term. Exact-term OR unions the requested postings, applies the selected snapshot's deletions, and visits each live document once.

R2. WalkShard and WalkDocs return exactly the fixture results above, preserving _source bytes, latest-revision selection, and tombstone suppression.

R3. Filtering happens before stored-field decode: Stream-only load succeeds despite corrupt Measure p5; selecting Measure reports the typed error.

R4. Term count, term length, dictionary enumeration, posting decode, and cancellation are bounded. CRC32 remains ignored and no reader lock or directory mutation is introduced.

RED and end-to-end contract

  • Boundary RED: exercise exact term and two-term OR through the bounded native visitor inherited from NIDX-01C; the compile-time or unsupported-operation failure proves the new slice is RED.
  • E2E RED: run WalkDocs/schema loaders for Stream, Stream+Group, and corrupt Measure against the checked-in legacy corpus and assert the declared results and typed failure. The command must fail before the production caller is switched.

Acceptance criteria

  • go test ./pkg/index/inverted/... ./banyand/metadata/schema/reader -run 'Test(NativeExactTerms|E2ESchemaWalkNative)' -count=1 passes.
  • A call-graph/dependency assertion proves schema walks no longer build or execute Bluge queries.
  • Exact, OR, deletion, corrupt-unselected-field, cancellation, and resource-limit cases pass under -race.
  • make test-ci PKG='./pkg/index/inverted/... ./banyand/metadata/schema/reader', make build, and make pre-push pass.

Scope

Packages: pkg/index/inverted, its private ICE dictionary/posting reader, and banyand/metadata/schema/reader.

Out of scope: numeric/date range, prefix, wildcard, MATCH, NOT, sorting, doc values, Property repair, and every writer.

Compatibility and rollback

This slice reads historical bytes and emits none. Roll back schema readers to the retained Bluge path without changing any directory.

Lexical non-regression

Repository changes for every implementation leaf before the final removal may delete existing references but must add zero new case-insensitive bluge tokens and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral legacy oracle or compatibility writer labels plus an immutable revision or content hash rather than adding a retired module name.

This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.

Design

BDB-NIDX-SPEC-001 revision 0.2 — NIDX-01

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions