feat: add plan-verified streaming decode

CHANGELOG.md

@@ -13,6 +13,27 @@ _No unreleased changes._
 
 ### Added
 
+- **Forward-only, exactly-sized decoders.** `RankQuant::read_from_sized` and
+  `SignBitmap::read_from_sized` accept a `Read` plus the exact encoded bytes
+  remaining at its current position. Existing seek-capable readers delegate to
+  them; truncation and trailing data remain fail-closed without a seek or read
+  beyond the declared record.
+- **Plan-verified artifact decoding.** `ordvec-manifest` now exposes
+  `VerifiedLoadPlan::decode_primary_with` and
+  `VerifiedAuxiliaryArtifactPlan::decode_verified_with`. They open one
+  no-follow regular-file descriptor, cap and hash the decoder's reads, drain
+  unread bytes through a 64 KiB buffer, compare initial/final descriptor size,
+  and return typed access, type, stale, decoder, or incomplete-consumption
+  errors with documented precedence. Optional-absent auxiliaries are typed.
+- **Composable current-schema loading.** Public
+  `read_manifest_bytes_bounded` and `parse_current_manifest_bytes` let policy
+  layers dispatch over one bounded read while retaining strict v2 parsing.
+  Manifest reads now reject a final symlink/reparse point or nonregular file.
+- **Stable JSON-number feature profile.** The production manifest dependency
+  now enables `serde_json/float_roundtrip` rather than
+  `serde_json/arbitrary_precision`. Current-schema parsing, plus a public
+  validator for compatibility schemas, rejects lossy number tokens from the
+  original bounded bytes before nested `Value` fields can alias.
 - **`ordvec-manifest`: typed verification classification.** Every
   verification issue code is now a named `pub const` in a `codes` module
   (zero bare literals at emit sites), with a `#[non_exhaustive]`
@@ -146,12 +167,13 @@ _No unreleased changes._
 
 ### Fixed
 
-- **`ordvec-manifest`: canonical content addresses are independent of
-  `serde_json` feature unification.** Nested extension and attestation maps are
-  recursively key-sorted and JSON numbers are normalized before serialization,
-  so downstream `preserve_order` / `arbitrary_precision` features cannot change
-  the same logical manifest's bytes. Manifest creation now rejects non-UTF-8
-  artifact paths instead of embedding the lossy replacement character.
+- **`ordvec-manifest`: canonical content addresses normalize nested JSON.**
+  Nested extension and attestation maps are recursively key-sorted and supplied
+  JSON numeric values are normalized before serialization. Release consumers
+  enforce one reviewed serde_json feature profile; current parsing rejects
+  lossy original number tokens before they become `Value`s. Manifest creation
+  now rejects non-UTF-8 artifact paths instead of embedding the lossy
+  replacement character.
 - **`ordvec-manifest`: manifest writes are atomic and fail before replacement.**
   `write_manifest_file` writes and syncs a same-directory temporary file before
   atomically replacing the destination, and rejects non-finite distortion
@@ -164,6 +186,10 @@ _No unreleased changes._
 
 ### Security
 
+- Plan-verified decoding closes the verify/path-reopen gap for the bytes it
+  consumes on controlled local storage. It does not authenticate a producer,
+  confine hostile ancestor replacement, detect a mutation perfectly restored
+  between observations, or make a later file-backed mapping immutable.
 - The release CI gate now waits, with a 30-minute fail-closed deadline, for
   the latest exact-HEAD push runs instead of treating normal Actions
   visibility/in-progress delay as an immediate failure. It rejects terminal

THREAT_MODEL.md

@@ -414,8 +414,14 @@ CAN still cause I/O and CPU proportional to the byte size it declares and
 actually supplies on disk. The flat `ResourceLimits` byte caps are opt-in
 ceilings (unbounded by default) for deployments that must bound worst-case
 verification time on attacker-supplied bundles. A `VerifiedLoadPlan` remains
-a verification snapshot, not a byte pin: bytes can change between
-verification and use by a local actor with write access (see scope).
+a verification snapshot, not a byte pin. Its plan-verified decode methods open
+the final component without following it, require a regular file (and, on
+Windows, a disk handle), and
+return decoded data only after delivered bytes and initial/final descriptor
+size match the plan. This closes the path-reopen race within the documented
+trusted-parent-directory boundary. It does not confine hostile ancestor
+replacement, authenticate the producer, detect a transient mutation perfectly
+restored between observations, or make a long-lived file-backed mmap immutable.
 
 **THREAT-QUERY-002 (P3): Panic on contract violation in Rust server contexts.**
 Rust APIs fail fast on invalid contract input (non-finite floats, dimension /

docs/INDEX_PROVENANCE.md

@@ -65,19 +65,20 @@ required artifacts and `--optional-aux NAME=PATH` for optional artifacts.
 Rust callers can use `verify_for_load(manifest_path, VerifyOptions)` to get a
 `VerifiedLoadPlan` containing the canonical artifact path, probed metadata,
 row-identity summary, auxiliary artifact states, and the full verification
-report, then call `require_auxiliary(name)` for sidecars that must be present
-before loading. Callers that already hold a `ManifestDocument` can use
+report, then call `decode_primary_with` and an auxiliary plan's
+`decode_verified_with` to decode through one no-follow, bounded, hashing file
+descriptor. Callers that already hold a `ManifestDocument` can use
 `verify_document_for_load(&document, VerifyOptions)` without re-reading the
-manifest file. The plan helpers do not call an ordvec loader, pin file
-descriptors, or make mutable shared storage immutable; callers still own the
-final policy decision and should load from the returned paths only while the
-verified files remain under their control.
+manifest file. The caller supplies the concrete decoder. The plan methods
+establish that the bytes delivered during that call match the verified plan,
+but do not make mutable shared storage or a later file-backed mmap immutable;
+callers still own the final policy decision.
 `ordvec-manifest/README.md` shows the intended verify-then-immediate-load
 pattern, a concrete `manifest.json + index.ovrq + ids.bin` sidecar-backed
 bundle, and the stable report fields/codes for sidecar audit logs. If another
 process can mutate the manifest, index, row map, or sidecar between
-verification and load, re-run `verify_for_load` at the load boundary or load
-from immutable storage or a caller-owned loading path that pins bytes.
+verification and use, decode through the plan at the load boundary or load
+from immutable storage.
 
 The manifest verifier checks:
 

ordvec-manifest/Cargo.toml

@@ -32,11 +32,20 @@ hex = "0.4.3"
 ordvec = { version = "0.6.0", path = ".." }
 rusqlite = { version = "0.39.0", optional = true }
 serde = { version = "1.0", features = ["derive"] }
-serde_json = { version = "1.0", features = ["arbitrary_precision"] }
+serde_json = { version = "1.0", features = ["float_roundtrip"] }
 sha2 = "0.11.0"
 tempfile = "3.27.0"
 uuid = { version = "1.23.2", features = ["v4"] }
 
+[target.'cfg(unix)'.dependencies]
+libc = "0.2.183"
+
+[target.'cfg(windows)'.dependencies]
+windows-sys = { version = "0.61.2", features = ["Win32_Foundation", "Win32_Storage_FileSystem"] }
+
+[target.'cfg(target_os = "wasi")'.dependencies]
+rustix = { version = "1.1.4", features = ["fs"] }
+
 [dev-dependencies]
 serde_json = { version = "1.0", features = ["preserve_order"] }
 

ordvec-manifest/README.md

@@ -94,20 +94,28 @@ The helper verifies the manifest with the supplied options, fails closed by
 returning `VerifiedLoadPlanError::VerificationFailed(report)` when report
 errors exist, and otherwise returns the canonical primary artifact path, probed
 metadata, row-identity summary, declared auxiliary artifact states, and the full
-report. Callers that already hold a `ManifestDocument` can use
+report. `decode_primary_with` and an auxiliary plan's `decode_verified_with`
+then consume one no-follow regular-file descriptor through a forward-only,
+size-capped hashing reader. Callers that already hold a `ManifestDocument` can use
 `verify_document_for_load(&document, VerifyOptions)` without re-reading the
-manifest file. These helpers do not call `Rank::load`, `RankQuant::load`,
-`Bitmap::load`, or `SignBitmap::load`, and they do not pin file descriptors or
-lock mutable storage. Callers should load from the returned paths immediately on
-storage they control, or re-verify if the backing files can change between
-verification and load.
+manifest file. The verification helpers do not choose a core index decoder.
+The plan-verified decode methods call the decoder supplied by the application,
+then return its value only after the descriptor's size and delivered-byte
+digest still match the plan. They trust caller-selected ancestor directories
+and do not lock mutable storage or authenticate the manifest producer.
 
 Controlled-storage load pattern:
 
 ```rust
 let plan = ordvec_manifest::verify_for_load(&manifest_path, options)?;
-let _app_ids = plan.require_auxiliary("app.ids")?;
-let index = ordvec::RankQuant::load(plan.artifact_path())?;
+let index = plan.decode_primary_with(
+    ordvec_manifest::ManifestIndexKind::RankQuant,
+    |reader, encoded_len| ordvec::RankQuant::read_from_sized(reader, encoded_len),
+)?;
+let app_ids = plan
+    .auxiliary_by_name("app.ids")
+    .ok_or("app.ids is not declared")?
+    .decode_verified_with(parse_caller_owned_ids_streaming)?;
 ```
 
 Concrete sidecar-backed bundle pattern:
@@ -167,10 +175,14 @@ let plan = ordvec_manifest::verify_for_load(
 let metadata = plan.metadata();
 assert_eq!(metadata.vector_count, expected_rows);
 
-let index = ordvec::RankQuant::load(plan.artifact_path())?;
-let ids_path = plan.require_auxiliary("app.ids")?;
-let ids_bytes = std::fs::read(ids_path)?;
-let doc_ids = parse_caller_owned_ids(&ids_bytes)?;
+let index = plan.decode_primary_with(
+    ordvec_manifest::ManifestIndexKind::RankQuant,
+    |reader, encoded_len| ordvec::RankQuant::read_from_sized(reader, encoded_len),
+)?;
+let doc_ids = plan
+    .auxiliary_by_name("app.ids")
+    .ok_or("app.ids is not declared")?
+    .decode_verified_with(parse_caller_owned_ids_streaming)?;
 ```
 
 `ordvec-manifest` owns the path, size, SHA-256, and index-metadata checks.
@@ -189,8 +201,12 @@ queue_for_later(plan);
 
 If a manifest or artifact lives on shared, writable, or otherwise mutable
 storage, re-run `verify_for_load` immediately before loading, load from
-immutable storage, or use a caller-owned loading path that pins bytes.
-`VerifiedLoadPlan` is not a byte pin.
+immutable storage, or use the plan-verified decode methods. Those methods
+detect mutations reflected in delivered bytes or descriptor sizes observed
+during the call. They cannot detect a transient mutation perfectly restored
+between those observations, and they do not make a long-lived file-backed mmap
+immutable after open. `VerifiedLoadPlan` itself remains a snapshot, not a byte
+pin.
 
 Verification uses bounded parser/report defaults on both CLI and library paths.
 Stable limit codes are part of the contract: