[NIDX-01C] Walk stored documents for migration source reads

[hanahmily]

[NIDX-01C] Walk stored documents for migration source reads

Parent workstream: #14002
Blocked by: #14009

Summary

Add one bounded native match-all/stored-field seam and use it in the source-read sides of Measure index-mode copy and series-index union. Destination publication remains on the retained legacy writer. This live slice adds stored-chunk decoding without adding dictionaries, term queries, doc values, or sorting.

Blocked by

NIDX-01B (#14009) must be merged before this work starts.

Pre-agreed test seam

Expose one BanyanDB-owned read-only document visitor from pkg/index/inverted. It must:

• stream one live document at a time from a pinned committed generation;
• expose repeated raw (field name, value bytes) visits;
• document that borrowed values expire when the document callback returns; and
• expose no ICE, Bluge, mmap, FST, or roaring type.

The contract author chooses the smallest concrete Go signature satisfying that shape. Boundary tests target only that visitor. End-to-end tests observe existing migration copy/union outputs, not private stored-chunk functions.

Independent fixture

Extend the legacy corpus with live series documents:

• series 101: _timestamp=100, _version=2, repeated color=[blue, green], and binary identity 0x010203;
• series 202: _timestamp=200, _version=1, color=[red]; and
• a deleted series 303, which must not be visited.

A second source repeats series 101 for the union case. The declared walk is exactly 101, 202; repeated values stay in the listed order; union output contains 101 once and 202 once.

Requirements

R1. The visitor streams only live documents and preserves every repeated field name/value byte sequence exactly. It retains at most one document plus configured decode buffers.

R2. Measure index-mode copy rebuilds series 101 and 202 with the declared timestamps, versions, fields, and identities through the native source visitor, while its destination remains legacy-written.

R3. Series union reads both sources natively and publishes one 101 and one 202; the existing dedup winner rule is unchanged.

R4. Cancellation stops between documents/chunks, releases files, and publishes no partial destination. Truncated stored chunks, invalid lengths, and decode-limit overflow return the native typed corruption error. CRC32 remains ignored.

RED and end-to-end contract

• Boundary RED: visit the checked-in fixture through the new BanyanDB-owned visitor and assert the literal documents and repeated values above.
• E2E RED: run index-mode copy and series union from legacy-created sources, verify their legacy-written destinations contain exactly 101 and 202, then run a corrupt-source case and assert no destination is published.

Acceptance criteria

• go test ./pkg/index/inverted/... ./banyand/measure ./banyand/internal/migration -run 'Test(NativeStoredDocumentWalk|E2E.*NativeSource)' -count=1 passes.
• A call-graph/dependency assertion proves both named source readers use the native visitor and no longer directly open Bluge readers.
• Cancellation, corrupt-chunk, repeated-value, deletion, and bounded-memory cases pass under -race.
• make test-ci PKG='./pkg/index/inverted/... ./banyand/measure ./banyand/internal/migration', make build, and make pre-push pass.

Scope

Packages: pkg/index/inverted, banyand/measure migration index-mode source reading, and banyand/internal/migration union source reading.

Out of scope: schema kind filtering, term dictionary/posting queries, Property repair, native destination writers, and all online writers.

Compatibility and rollback

The native path reads legacy ICE v3/snapshot v3 and emits no index bytes. Roll back only the two source-reader selections; existing destination directories require no conversion.

Lexical non-regression

Repository changes for every implementation leaf before the final removal may delete existing references but must add zero new case-insensitive bluge tokens and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral legacy oracle or compatibility writer labels plus an immutable revision or content hash rather than adding a retired module name.

This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.

Design

BDB-NIDX-SPEC-001 revision 0.2 — NIDX-01