diff --git a/packages/kits/CorpusKit/docs/AGENT_MAP.md b/packages/kits/CorpusKit/docs/AGENT_MAP.md
index cba0ea4..822fd43 100644
--- a/packages/kits/CorpusKit/docs/AGENT_MAP.md
+++ b/packages/kits/CorpusKit/docs/AGENT_MAP.md
@@ -16,9 +16,9 @@ sources:
- path: Sources/CorpusKit/Chunker.swift
blob: a2718e06d1715f539ff633e7037c70e10ecb7a2d
- path: Sources/CorpusKit/CorpusIngestQueue.swift
- blob: 2c32133701ce728bc017d4ddad51b052cae990db
+ blob: 4dd3bd8eefb8e0e7dd474793d35acbe03b452df3
- path: Sources/CorpusKit/CorpusKit.swift
- blob: 4518f15fdb798c3a203c4a9db949f4d6172f540d
+ blob: bd4433bb5a7dc347df5e24c04d5f76a533261e54
- path: Sources/CorpusKit/CorpusKitError.swift
blob: 68ac8d0a248bc9c2dd1885b0bc531ac4ed9cb91d
- path: Sources/CorpusKit/CorpusProviderCountsStore.swift
@@ -30,7 +30,7 @@ sources:
- path: Sources/CorpusKit/Engine/InvertedIndex.swift
blob: 1273adcb3794b1997c93488182fc5ed95b21f9ec
- path: Sources/CorpusKit/Engine/InvertedIndexStore.swift
- blob: 242baf05e5c846c719c403599a9a407bba646f5b
+ blob: 14f9de0b8e7ce0eae638605f24e43493afa0ac4e
- path: Sources/CorpusKit/Engine/SparseTypes.swift
blob: 54654e2c49b09d31f60c06503216a2b281939f87
- path: Sources/CorpusKit/HybridRecall.swift
@@ -82,47 +82,55 @@ PURPOSE: standalone on-device RAG kit. Text → chunks (content-addressed UUID)
DEPS: CorpusKit imports SubstrateTypes, SubstrateLib (MerkleHash), SubstrateML, EngramLib, EideticLib (sentence segmentation), IntellectusLib (telemetry, off-by-default), PersistenceKit (+InMemory, +SQLite), ConvergenceKit (manifest only), VectorKit, QueueKit, Crypto. CorpusKitProviders additionally imports SubstrateKernel (FloatVecOps), LatticeLib (FDC runtime; FDC math NOT reimplemented). Imported by: GeniusLocusKit (orchestrator tier). Rust ports: `rust/` (core, crate corpus-kit) + `rust-providers/` (crate corpus-kit-providers); shared fixtures `Tests/SharedVectors/*.json` read by BOTH legs gate bit-identity. NL providers are Swift-only (ADR-019, no Rust twin).
ENTRY POINTS (most callers need only these):
-- CorpusKit.swift:702 `Corpus.init(storage:models:)` : open estate corpus; `models[0]` = default signal (:674 single-model convenience)
-- CorpusKit.swift:1001 `Corpus.ingest(_ text:sourceID:now:)` : synchronous chunk+index+embed
-- CorpusIngestQueue.swift:157 `Corpus.enqueueIngest(_:sourceID:now:)` : async queued ingest (production path)
-- CorpusKit.swift:1629 `Corpus.recall(_ query:limit:now:) -> [ScoredChunk]` : hybrid RRF recall on default signal
+- CorpusKit.swift:725 `Corpus.init(storage:models:)` : open estate corpus; `models[0]` = default signal (:697 single-model convenience)
+- CorpusKit.swift:1026 `Corpus.ingest(_ text:sourceID:now:)` : synchronous chunk+index+embed
+- CorpusIngestQueue.swift:184 `Corpus.enqueueIngest(_:sourceID:now:)` : async queued ingest, "encode" stream (production path)
+- CorpusIngestQueue.swift:230 `Corpus.enqueueIngestBatchImport(_:)` : async queued BULK import, "import" stream (chunk+BM25 only; caller reindexes once at the end)
+- CorpusKit.swift:1990 `Corpus.recall(_ query:limit:now:) -> [ScoredChunk]` : hybrid RRF recall on default signal
- DefaultEnsemble.swift:62 `CorpusEnsemble.defaultEnsemble() -> [EmbeddingModel]` : the five production signals, fresh per estate
## Symbol Table
### Facade : CorpusKit.swift
-- :48 `enum FloatLaneOutcome` : `.hits`/`.unavailableProviderOptOut`/`.unavailableNoVocabHit`/`.unavailableNoFloatRows`/`.emptyQuery`/`.storeError`; dark lanes are typed outcomes, NEVER errors
-- :134 `enum EmbeddingModel` : `.deterministic` (:147, seed 0xC05B_D15C_A15D_1B00, federation baseline), `.miniLM/.mpNet/.embeddingGemma(inference:)` (:157/:167/:177, host closure), `.randomIndexing/.ppmi/.lsa/.nmf(provider:)` (:194–:233, trainable), `.fdc(provider:)` (:254, stateless), `.nlEmbedding/.nlContextualEmbedding` (:275/:293, Apple-only); `.default = .deterministic` (:297)
-- :338 `EmbeddingModel.isTrainable` : true iff carried provider conforms to TrainableEmbeddingBasis
-- :362 `EmbeddingModel.reconstruct(from: Data)` : routes blob to concrete type; throws `.notTrainable`
-- :417 `enum EncodeSpeed` : `.foreground` (all cores) / `.background` (cores/4, floor 1)
-- :426 `public actor Corpus` : composition root; sealed-vector principle (no VectorKit type in public API except :649 `sharedVectorStore`)
-- :491 `setEncodeSpeed(_:)`; :641 `onEncoded` callback (set via CorpusIngestQueue.swift:134 `setOnEncoded`)
-- :1001 `ingest(_:sourceID:now:)` : idempotent; re-ingest clears tombstone; first-ingest auto-train (gate = no persisted basis, NOT factory-blob presence)
-- :1181 `ingestBatch(_:)` : identical output to per-item; commit windows 512 items / 4096 rows (:436–:437); slice-parallel embed; batch-aware first-basis bootstrap (train once on full batch, never per item)
-- :1416 `maintainedVocabAnchor() -> Int` : governor's vocab-growth retrain trigger read
-- :1488 `reindex(now:)` : THE explicit retrain: reconstruct-fresh → trainOnCorpus(active chunks) → upsert basis → re-embed all under every slot; excludes removed sources
-- :1629 `recall(_:limit:now:)`; :1657 `remove(sourceID:)` (BM25 rows + ALL models' vectors + tombstone; chunks kept); :1711 `expunge(sourceID:)` (scrubText FIRST, then remove); :1738 `destroyRecallIndex()` (all derived state; chunks survive)
-- :1779 `bm25TopKBySource(query:limit:)` : pure keyword lane, max-chunk-score per source, frontierK ≤ 256
-- :1833 `embed(_:) -> Engram`; :1846 `modelID`; :1863 `embedFloat(_:)` (throws on provider opt-out); :2238 `supportsFloat`
-- :1895 `floatNearest(query:limit:) -> FloatLaneOutcome` : never throws; sim = 1 − distance/10_000; source aggregation = MAX chunk cosine
-- :2141 `floatNearestPerSignal(query:limit:)` : per-slot dense lanes in slot order, NO fusion (caller fuses); :2214 `floatFarthestPerSignal` : anti-similarity, MIN chunk cosine per source, ascending
-- :2249 `count()` (excludes removed sources); :2266 `indexedSourceIDs()`; :2274/:2280 `corpusMerkleRoot(for:)`/`globalCorpusMerkleRoot()`
-- :2300 `EmbeddingModel.makeProvider()` (private) : pinned seeds: miniLM 0x4D49_4E4C_4D5F_7631 "MINLM_v1", mpNet 0x4D50_4E45_545F_7631 "MPNET_v1", embeddingGemma 0x454D_4247_4D5F_7631 "EMBGM_v1"; model IDs corpus-deterministic-v1 / minilm-v6 / mpnet-base-v2 / embedding-gemma-300m
-- :2413 `CorpusDefaultTokenizer` (internal) : FNV-1a fold, duplicated from providers to avoid circular dep; :2445 `CorpusTextProvider` (private) : tokenize→inference→FloatSimHash; :2498 `embedPair` computes pooled vector ONCE for both lanes
-- Test seams (never production): :914 init(storage:provider:), :980 `_testForceFloatStoreError`, :656 `_ingestFailureHook`
+- :50 `enum FloatLaneOutcome` : `.hits`/`.unavailableProviderOptOut`/`.unavailableNoVocabHit`/`.unavailableNoFloatRows`/`.emptyQuery`/`.storeError`; dark lanes are typed outcomes, NEVER errors
+- :136 `enum EmbeddingModel` : `.deterministic` (:149, seed 0xC05B_D15C_A15D_1B00, federation baseline), `.miniLM/.mpNet/.embeddingGemma(inference:)` (:159/:169/:179, host closure), `.randomIndexing/.ppmi/.lsa/.nmf(provider:)` (:196–:235, trainable), `.fdc(provider:)` (:256, stateless), `.nlEmbedding/.nlContextualEmbedding` (:277/:295, Apple-only); `.default = .deterministic` (:299)
+- :340 `EmbeddingModel.isTrainable` : true iff carried provider conforms to TrainableEmbeddingBasis
+- :364 `EmbeddingModel.reconstruct(from: Data)` : routes blob to concrete type; throws `.notTrainable`
+- :419 `enum EncodeSpeed` : `.foreground` (all cores) / `.background` (cores/4, floor 1)
+- :428 `public actor Corpus` : composition root; sealed-vector principle (no VectorKit type in public API except :672 `sharedVectorStore`)
+- :493 `setEncodeSpeed(_:)`; :664 `onEncoded` callback (set via CorpusIngestQueue.swift:161 `setOnEncoded`)
+- :622/:630 `ingestDrainWorker`/`drainLease` (encode stream); :638/:641/:651 `importDrainWorker`/`importDrainLease`/`importQueue` (discrete bulk-import stream, own DrainLease, own QueueKit facade over its own connection on SQLite estates; nil on in-memory estates, where the import worker shares `ingestQueue`)
+- :1026 `ingest(_:sourceID:now:)` : idempotent; re-ingest clears tombstone; first-ingest auto-train (gate = no persisted basis, NOT factory-blob presence)
+- :1432 `ingestBatch(_:)` : identical output to per-item; commit windows 512 items / 4096 rows (:438); slice-parallel embed; batch-aware first-basis bootstrap (train once on full batch, never per item)
+- :1200 `ingestBatchImport(_:)` : DISCRETE bulk-import path used by the import drain worker; chunk + bundle + BM25 + source-map + counts fold ONLY, NO train, NO embed; SQLite estates take :1224 `ingestBatchImportSharded` (EXT-4: per-core shard files chunk+tokenize+write BM25 postings off the actor, one writer bundles rows and merges shards via `SQLiteStorage.mergeShard`, :1221 `importShardItems = 2500` items per shard); non-SQLite estates take :1382 `ingestBatchImportSerial` (same steps, one item at a time on the actor)
+- :1667 `maintainedVocabAnchor() -> Int` : governor's vocab-growth retrain trigger read
+- :1739 `reindex(now:)` : THE explicit retrain, now two phases. Phase 1 (:1846 `trainAndPersistBasis` inlined as a concurrent task group): every trainable slot reconstructs fresh and trains CONCURRENTLY, then installs + persists serially; Phase 2 (:1887 `reembedChunks`): re-embeds each slot's chunks, with the embed compute itself fanned across bounded batches (3000 chunks each) and the old per-chunk delete replaced by one bulk `vectorStore.replaceModelVectors(modelID:)` clear-then-add; excludes removed sources; logs phase progress since a large reindex runs minutes
+- :1990 `recall(_:limit:now:)`; :2018 `remove(sourceID:)` (BM25 rows + ALL models' vectors + tombstone; chunks kept); :2072 `expunge(sourceID:)` (scrubText FIRST, then remove); :2099 `destroyRecallIndex()` (all derived state; chunks survive)
+- :2140 `bm25TopKBySource(query:limit:)` : pure keyword lane, max-chunk-score per source, frontierK ≤ 256
+- :2194 `embed(_:) -> Engram`; :2207 `modelID`; :2224 `embedFloat(_:)` (throws on provider opt-out); :2599 `supportsFloat`
+- :2256 `floatNearest(query:limit:) -> FloatLaneOutcome` : never throws; sim = 1 − distance/10_000; source aggregation = MAX chunk cosine
+- :2502 `floatNearestPerSignal(query:limit:)` : per-slot dense lanes in slot order, NO fusion (caller fuses); :2575 `floatFarthestPerSignal` : anti-similarity, MIN chunk cosine per source, ascending
+- :2610 `count()` (excludes removed sources); :2627 `indexedSourceIDs()`; :2635/:2641 `corpusMerkleRoot(for:)`/`globalCorpusMerkleRoot()`
+- :2661 `EmbeddingModel.makeProvider()` (private) : pinned seeds: miniLM 0x4D49_4E4C_4D5F_7631 "MINLM_v1", mpNet 0x4D50_4E45_545F_7631 "MPNET_v1", embeddingGemma 0x454D_4247_4D5F_7631 "EMBGM_v1"; model IDs corpus-deterministic-v1 / minilm-v6 / mpnet-base-v2 / embedding-gemma-300m
+- :2774 `CorpusDefaultTokenizer` (internal) : FNV-1a fold, duplicated from providers to avoid circular dep; :2806 `CorpusTextProvider` (private) : tokenize→inference→FloatSimHash; :2859 `embedPair` computes pooled vector ONCE for both lanes
+- :2869 `extension BackendConfiguration`; :2874 `sqliteURLForShards` : SQLite URL for import-shard placement beside the estate; nil for non-SQLite backends (takes the serial import path)
+- Test seams (never production): :939 init(storage:provider:), :1005 `_testForceFloatStoreError`, :679 `_ingestFailureHook`
### Ingest queue : CorpusIngestQueue.swift (extension Corpus)
-- :63 `mountIngestQueue()` : idempotent; SQLite estate → encrypted sibling `queue.sqlite` via `EstateConfiguration.queueSibling` (ADR-021 D7/T4; replaced plaintext maildir hole); InMemory estate → fixed :486 `ingestQueueStoreID` (no UUID() nondeterminism)
-- :120 `dropIngestQueue()`; :134 `setOnEncoded(_:)` : the ONLY CorpusKit→orchestrator callback
-- :157 `enqueueIngest(...)`; :179 `enqueueIngestBatch(...)` : one transaction for all jobs (bulk-import bottleneck fix; caller bounds batch size)
-- :205 `awaitIngestDrain(timeout: 30s)` : barrier: drained AND vector index republished; throws drainTimeout
-- :229 `ingestQueueDepth() -> (pending, inFlight)` : pending IS stream-scoped, inFlight is NOT (all streams)
-- :248 `drainIngestQueueOnce()` : claims whole batch; undecodable → `.blocked` (terminal), empty text → `.done`; batch `ingestBatch` + bulk session reply; falls back serial on batch throw
-- :341 `runIngestDrainLoop` (private) : DrainLease single drainer; first-acquire crash recovery `reclaimInFlight`; standby poll 3 s; lease TTL 15 s (failover ≈ 15–18 s, not instant); spin-while-draining, 15 ms idle sleep; vector index published once per burst (O(N) not O(N²))
-- :430 `ingestOneAndReply` (private) : retry in place ≤ :476 `ingestMaxAttempts = 8`, then `.blocked`; sound ONLY because ingest is idempotent
-- :480 `encodeStreamID = "encode"` : EVERY queue op must be scoped to it (shared queue.sqlite may carry other streams; unscoped awaitDrain deadlocks)
-- :515 `IngestJob` : wire fields `sourceID`/`text`/`capturedAtISO8601` = pinned cross-port JSON contract; :551 `toJob`, :563 `from(job:)`
+- :63 `mountIngestQueue()` : idempotent; SQLite estate → encrypted sibling `queue.sqlite` via `EstateConfiguration.queueSibling` (ADR-021 D7/T4; replaced plaintext maildir hole); InMemory estate → fixed :698 `ingestQueueStoreID` (no UUID() nondeterminism). Also mounts a SECOND drain worker for the `"import"` stream, with its own `DrainLease` and, on SQLite estates, its own `QueueKit` connection (`importQueue`); in-memory estates share `ingestQueue` for the import worker too
+- :142 `dropIngestQueue()` : cancels BOTH drain workers and releases BOTH leases; :161 `setOnEncoded(_:)` : the ONLY CorpusKit→orchestrator callback
+- :184 `enqueueIngest(...)`; :206 `enqueueIngestBatch(...)` : one transaction for all jobs (bulk-import bottleneck fix; caller bounds batch size); "encode" stream, daily-driving captures
+- :230 `enqueueIngestBatchImport(...)` : same durable `queue.sqlite`, "import" stream; claimed only by the import drain worker, chunk+BM25 only (no embed/train per item)
+- :257 `awaitIngestDrain(timeout: 30s)` : barrier scoped to the "encode" stream: drained AND vector index republished; throws drainTimeout
+- :281 `ingestQueueDepth() -> (pending, inFlight)` : BOTH fields now stream-scoped to "encode" (fixed; inFlight used to count all streams, which let a concurrent import inflate the encode probe)
+- :298 `importQueueDepth() -> (pending, inFlight)` : the "import"-stream twin of `ingestQueueDepth`; both zero means every enqueued import job has been chunk+BM25-ingested and replied
+- :317 `drainIngestQueueOnce()` : claims whole "encode" batch; undecodable → `.blocked` (terminal), empty text → `.done`; batch `ingestBatch` + bulk session reply; falls back serial on batch throw
+- :410 `runIngestDrainLoop` (private) : DrainLease single drainer for "encode"; first-acquire crash recovery `reclaimInFlight`; standby poll 3 s; lease TTL 15 s (failover ≈ 15–18 s, not instant); spin-while-draining, 15 ms idle sleep; vector index published once per burst (O(N) not O(N²))
+- :499 `ingestOneAndReply` (private) : retry in place ≤ :683 `ingestMaxAttempts = 8`, then `.blocked`; sound ONLY because ingest is idempotent
+- :538 `drainImportQueueOnce()` : the "import"-stream twin of `drainIngestQueueOnce`; claims through `importQueue` (falls back to `ingestQueue` on in-memory estates); batch `ingestBatchImport` + bulk session reply; falls back to :645 `ingestOneImportAndReply` per job on batch throw; NO `onEncoded` callback (per-batch rollups deferred to the import cycle's tail)
+- :596 `runImportDrainLoop` (private) : the "import" twin of `runIngestDrainLoop`; own lease, own on-mount `reclaimInFlight`, same poll/TTL/idle-sleep constants; NO vector-index publish step (the import drain writes no vectors, `reindex` does that once at the import tail)
+- :687 `encodeStreamID = "encode"`; :692 `importStreamID = "import"` : EVERY queue op must be scoped to one or the other (shared queue.sqlite carries both streams; an unscoped op deadlocks or double-counts)
+- :727 `IngestJob` : wire fields `sourceID`/`text`/`capturedAtISO8601` = pinned cross-port JSON contract, shared by both streams; :763 `toJob(streamID:...)`, :775 `from(job:)`
### Chunks : Chunk.swift / Chunker.swift
- Chunk.swift:35 `struct Chunk: Sendable, Equatable, Codable` : immutable, content-addressed
@@ -168,7 +176,7 @@ ENTRY POINTS (most callers need only these):
### Persistent keyword index : Engine/InvertedIndexStore.swift
- :47 `public actor InvertedIndexStore`; :55 schemaDeclaration (kitID "InvertedIndexStore", tables iix_termfreqs/iix_doclens : RAW statistics only, weighted index derived+cached, so k1/b changes need no migration); :105 init (storage pre-opened/migrated); :114 `open()` (load mirrors, O(terms+docs), no chunk bodies)
-- :159 `index(itemID:tokens:now:)` : atomic replace, idempotent; empty tokens = removal; `now` unused (determinism discipline); :199 `remove(itemID:)`; :230 `buildIndex(parameters:)` (cached; invalidated per write); :253 `topK(queryTerms:k:parameters:algorithm:)`; :273 `deleteAll()`; :295 `documentCount`
+- :159 `index(itemID:tokens:now:)` : atomic replace, idempotent; empty tokens = removal; `now` unused (determinism discipline); :203 `foldPostings(_:)` : the EXT-4 sharded-import twin of `index`, folds worker-computed `(itemID, tf, docLen)` triples straight into the in-memory maps (nothing re-read from disk), idempotent under queue retry, invalidates the cached BM25 build once per batch; :216 `remove(itemID:)`; :247 `buildIndex(parameters:)` (cached; invalidated per write); :270 `topK(queryTerms:k:parameters:algorithm:)`; :290 `deleteAll()`; :312 `documentCount`
- Rust twin owns a PRIVATE connection with begin/commit/rollback_batch; Swift shares estate storage : hence Corpus-managed transaction windows in ingestBatch
### Legacy keyword index : BM25Index.swift
@@ -227,13 +235,13 @@ ENTRY POINTS (most callers need only these):
- DETERMINISM DISCIPLINE: engines never read `Date()`/`UUID()`; `now` is always caller-supplied. Sanctioned exceptions: RemovedSourceStore audit stamp, telemetry timestamps. Training is a pure function of (texts, pinned seeds).
- UNIVERSAL JOIN KEY: chunk.id.uuidString == VectorKit item_id == inverted-index item_id. Swift canonical uuidString is UPPERCASE; HybridRecall re-canonicalizes vector-lane ids (P3-secfix). Universal tie-break everywhere: score DESC, then id ASC ("smaller id wins").
-- PINNED CONSTANTS (change = new conformance vectors + possible fleet re-key): chunker 800/100; BM25 k1 1.5 / b 0.75; QUANT_SCALE 100 with round-HALF-TO-EVEN; BMW block 128; RRF k 60, weights 0.6/0.4; candidate over-fetch max(limit×4, 32); float-lane sim quantization ×10_000; commit windows 512/4096; ingestMaxAttempts 8; idle drain sleep 15 ms; lease TTL 15 s / standby poll 3 s; RI/PPMI D 2048 K 10 window 4; LSA rank 64 sweeps 30; NMF rank 32 iterations 100 tolerance 0 factorization seed 0xDEADBEEFCAFEBABE; reduced-vocab cap 512; Chunk.namespaceBytes; basisFormatVersion 1.
+- PINNED CONSTANTS (change = new conformance vectors + possible fleet re-key): chunker 800/100; BM25 k1 1.5 / b 0.75; QUANT_SCALE 100 with round-HALF-TO-EVEN; BMW block 128; RRF k 60, weights 0.6/0.4; candidate over-fetch max(limit×4, 32); float-lane sim quantization ×10_000; commit windows 512/4096; ingestMaxAttempts 8; idle drain sleep 15 ms; lease TTL 15 s / standby poll 3 s; RI/PPMI D 2048 K 10 window 4; LSA rank 64 sweeps 30; NMF rank 32 iterations 100 tolerance 0 factorization seed 0xDEADBEEFCAFEBABE; reduced-vocab cap 512; Chunk.namespaceBytes; basisFormatVersion 1; importShardItems 2500; reindex reembed batch 3000; reembed progress log stride 5000 (these last three are throughput knobs, not conformance-pinned).
- PROJECTION SEEDS partition vector storage by model and must be unique + frozen: deterministic 0xC05B_D15C_A15D_1B00, MINLM_v1, MPNET_v1, EMBGM_v1, RI_V1_MX, PPMI_V1M, LSA_V1_M, NMF_V1_M, FDC_V1_P, APNLEMB1, APNLCTX1. All projection goes through SubstrateML.FloatSimHash : ad-hoc projections are banned from the kit graph.
-- ONLY TWO TRAIN TRIGGERS: first-ingest auto-train (gate = no persisted basis) and explicit `reindex(now:)`. trainOnCorpus is ADDITIVE : always reconstruct fresh from the slot's freshBasisBlob before retraining; growth-threshold auto-retrain is future policy (anchors persisted, path unwired : "HALF A").
+- ONLY TWO TRAIN TRIGGERS: first-ingest auto-train (gate = no persisted basis) and explicit `reindex(now:)`. `ingestBatchImport` adds no third trigger : it only chunks and BM25-indexes, and the caller is expected to run one `reindex(now:)` after the whole import completes. trainOnCorpus is ADDITIVE : always reconstruct fresh from the slot's freshBasisBlob before retraining; growth-threshold auto-retrain is future policy (anchors persisted, path unwired : "HALF A"). `reindex` now trains every trainable slot CONCURRENTLY (one task group) before installing and persisting each result serially; per-slot output stays byte-identical to the old serial loop.
- TOMBSTONE CONSULTATION IS UNENFORCED: every chunk-replay path (reindex, first-ingest train, count, any future rebuild) MUST subtract RemovedSourceStore.removedIDs() or removed sources resurrect on the governor's auto-reindex.
- BundleStore.insert returns ONLY newly-inserted chunks; fold derived state over the returned subset. count(asOf:) ignores asOf.
- DUAL TypedValue DECODE (BundleStore, BasisStore): SQLite round-trips UUID/HLC/timestamp as primitives, InMemory keeps semantic forms; decoders must accept both; InMemory-only tests cannot catch the regression (proven bug class: silent total data loss on reopen).
-- STREAM SCOPING: every queue op in CorpusIngestQueue uses stream "encode"; the shared queue.sqlite may carry other streams; an unscoped awaitDrain deadlocks. IngestJob JSON field names are a frozen cross-port wire contract.
+- STREAM SCOPING: CorpusIngestQueue now runs TWO streams on one `queue.sqlite`, "encode" (daily-driving, `enqueueIngest`/`enqueueIngestBatch`) and "import" (discrete bulk import, `enqueueIngestBatchImport`); every queue op must scope to the right one, or an unscoped op deadlocks (`awaitIngestDrain`) or double-counts (`ingestQueueDepth`/`importQueueDepth`, both now correctly stream-scoped on both fields). Each stream has its own DrainLease and its own drain worker. IngestJob JSON field names are a frozen cross-port wire contract shared by both streams.
- appendOnly means two different things: sync conflict policy (SyncManifest, `.appendOnly`, safe via content addressing) vs BundleStore schema flag (`appendOnly: false`, required so scrubText can UPDATE). Do not conflate.
- Chunks are immutable BY CONVENTION (no update API), not by DB trigger; edit = delete + reinsert with new id; metadata is the only legal per-chunk side-data slot (doctrine §2).
- keywordTokens override breaks the single-tokenizer guarantee shared by BM25 + distributional providers (convention, not compiler-enforced).
diff --git a/packages/kits/CorpusKit/docs/DETAILS.md b/packages/kits/CorpusKit/docs/DETAILS.md
index 696b0df..03a0b3b 100644
--- a/packages/kits/CorpusKit/docs/DETAILS.md
+++ b/packages/kits/CorpusKit/docs/DETAILS.md
@@ -16,9 +16,9 @@ sources:
- path: Sources/CorpusKit/Chunker.swift
blob: a2718e06d1715f539ff633e7037c70e10ecb7a2d
- path: Sources/CorpusKit/CorpusIngestQueue.swift
- blob: 2c32133701ce728bc017d4ddad51b052cae990db
+ blob: 4dd3bd8eefb8e0e7dd474793d35acbe03b452df3
- path: Sources/CorpusKit/CorpusKit.swift
- blob: 4518f15fdb798c3a203c4a9db949f4d6172f540d
+ blob: bd4433bb5a7dc347df5e24c04d5f76a533261e54
- path: Sources/CorpusKit/CorpusKitError.swift
blob: 68ac8d0a248bc9c2dd1885b0bc531ac4ed9cb91d
- path: Sources/CorpusKit/CorpusProviderCountsStore.swift
@@ -30,7 +30,7 @@ sources:
- path: Sources/CorpusKit/Engine/InvertedIndex.swift
blob: 1273adcb3794b1997c93488182fc5ed95b21f9ec
- path: Sources/CorpusKit/Engine/InvertedIndexStore.swift
- blob: 242baf05e5c846c719c403599a9a407bba646f5b
+ blob: 14f9de0b8e7ce0eae638605f24e43493afa0ac4e
- path: Sources/CorpusKit/Engine/SparseTypes.swift
blob: 54654e2c49b09d31f60c06503216a2b281939f87
- path: Sources/CorpusKit/HybridRecall.swift
@@ -364,9 +364,15 @@ only requires an in-memory rebuild. `open()` loads all rows once, at a
cost proportional to terms plus documents, never to chunk text.
`index(itemID:tokens:now:)` replaces a document's terms atomically, and
-it is idempotent. Empty tokens remove the item. `remove(itemID:)` and
-`deleteAll()` complete the mutation surface. `topK(queryTerms:k:parameters:algorithm:)`
-is the one-call query path. The actor serializes all mutation. The Rust
+it is idempotent. Empty tokens remove the item. `foldPostings(_:)` is a
+second write path for a bulk import. A sharded import worker computes
+term frequencies and document lengths off the actor, in parallel. This
+method then folds those already-computed rows straight into the same
+in-memory maps `index` writes. Nothing gets re-read from disk. A
+re-delivered item just overwrites its prior entry, so retries under the
+import queue stay safe. `remove(itemID:)` and `deleteAll()` complete the
+mutation surface. `topK(queryTerms:k:parameters:algorithm:)` is the
+one-call query path. The actor serializes all mutation. The Rust
twin owns a private database connection with explicit batch methods.
The Swift store instead shares the estate's storage, which is why the
facade manages transaction windows around it during bulk ingest.
@@ -545,12 +551,35 @@ commits in windows of 512 items or 4,096 rows, long enough to amortize
disk syncs and short enough not to starve concurrent captures. It also
fans embedding work out in contiguous slices per core.
-`reindex(now:)` is the explicit retrain trigger. It performs five steps
-in order. It reconstructs fresh providers. It trains on all active
-chunks, excluding tombstoned sources. It installs the result. It
-persists the basis. It re-embeds each active chunk under each slot.
-Only two train triggers exist in the whole kit: first ingest, and
-explicit reindex.
+`ingestBatchImport(_:)` is a separate bulk-import path. The import
+drain worker calls it instead of `ingestBatch`. It chunks each item,
+bundles the chunks, and BM25-indexes them. It also folds counts. It
+never trains, and it never embeds. A SQLite estate takes a sharded
+route. Worker tasks chunk and tokenize their own slice off the actor.
+Each writes its BM25 postings into a private shard file. One writer
+then bundles the chunk rows as usual, merges each shard's postings with
+one attach and one sorted insert, and folds the postings into the
+in-memory index. A non-SQLite estate instead runs the same steps one
+item at a time, on the actor. Neither route trains a basis or writes a
+vector. The caller reindexes once, after the whole import completes, to
+do both.
+
+`reindex(now:)` is the explicit retrain trigger. It runs in two phases
+now, instead of one loop per slot. Phase one trains every trainable
+slot at once. Each slot reconstructs a fresh provider from its stored
+empty basis, and trains it on all active chunks, excluding tombstoned
+sources. That reconstruct-and-train step runs as a concurrent task per
+slot, since it is pure computation over a shared chunk snapshot. The
+actor then installs each trained provider, and persists its basis, one
+slot at a time. Phase two re-embeds every active chunk under every
+slot. The embed compute itself now fans out over bounded batches of
+three thousand chunks, run concurrently up to the embed concurrency
+cap. A single bulk clear-and-write then replaces that slot's whole
+vector set in one pass, rather than deleting and re-adding vectors one
+chunk at a time. The old per-chunk delete scanned every resident vector
+on every chunk, so a large reindex used to cost quadratic time. The
+bulk replace costs linear time instead. Only two train triggers exist
+in the whole kit: first ingest, and explicit reindex.
### Recall, Removal, Observation
@@ -599,10 +628,12 @@ both the engram and the float row.
## CorpusIngestQueue.swift
This file holds the asynchronous ingest pipeline. It arrives as an
-extension on `Corpus`. Three pieces make up the pipeline: a durable
-queue, a background drain worker, and a single-drainer lease. It exists
-so CorpusKit is a complete standalone substrate. Any consumer gets
-queued, multi-core encoding with no orchestrator.
+extension on `Corpus`. The pipeline runs two parallel drain paths over
+one durable queue: a daily-driving encode path, and a discrete
+bulk-import path. Each path gets its own background drain worker and
+its own single-drainer lease. The file exists so CorpusKit is a
+complete standalone substrate. Any consumer gets queued, multi-core
+encoding with no orchestrator.
`mountIngestQueue()` picks the backend by estate durability. A SQLite
estate gets a sibling `queue.sqlite` file, derived deterministically
@@ -610,15 +641,22 @@ from the estate configuration. That file is encrypted with the same key
as the estate. It replaces an earlier plaintext directory queue, which
was a real security hole beside an encrypted estate. An in-memory estate
gets a transient store instead, under a fixed constant UUID, which
-avoids random-id nondeterminism. Since the physical queue can
-carry other streams, each operation here is scoped to the `"encode"`
-stream. An unscoped wait would deadlock on jobs this drainer never
+avoids random-id nondeterminism. Since the physical queue can carry
+other streams, each operation here is scoped to one stream, `"encode"`
+or `"import"`. An unscoped wait would deadlock on jobs a drainer never
claims.
-The drain loop coordinates through a `DrainLease`. Only one live drainer
-exists per estate, with crash recovery on first acquisition that resets
-orphaned in-flight jobs. That recovery is safe precisely since the
-lease guarantees no other live drainer holds them. A losing process
+The method mounts both drain workers at once. A SQLite estate opens a
+second connection for the import worker. Two live connections give each
+worker its own file-level arbitration under the database, rather than
+one shared connection whose transactions could collide. An in-memory
+estate has no real transactions to arbitrate, so the import worker
+there just shares the encode worker's queue facade.
+
+The encode drain loop coordinates through a `DrainLease`. Only one live
+drainer exists per estate, with crash recovery on first acquisition that
+resets orphaned in-flight jobs. That recovery is safe precisely since
+the lease guarantees no other live drainer holds them. A losing process
becomes a warm standby instead. It re-checks each three seconds,
bounded by the lease's fifteen-second staleness window. Each drain pass
claims the whole available batch. It decodes jobs: undecodable ones are
@@ -632,14 +670,35 @@ milliseconds, the near-realtime latency floor. A failing item retries in
place, up to eight attempts, before a terminal blocked reply. In-place
retry is sound only since ingest is idempotent.
+The import drain loop is the discrete twin of the loop above. It claims
+only the `"import"` stream, and it holds its own lease, so the two loops
+run and fail over independently. Each pass hands its batch to
+`ingestBatchImport` instead of `ingestBatch`. That path chunks and
+BM25-indexes only. It never trains a basis, and it never embeds a
+vector. The import loop publishes no vector index of its own. A bulk
+import instead calls an explicit reindex once, at the very end, to train
+every trainable slot and embed every chunk. On a SQLite estate, the
+import path also shards its chunking and tokenizing work across cores,
+then merges the shards back through the estate's single writer.
+
`enqueueIngest(_:sourceID:now:)` and `enqueueIngestBatch(_:)` stamp jobs
-with caller-supplied instants, never the wall clock. The batch variant
-wraps all inserts in one transaction, which removed the last full-core
+with caller-supplied instants, never the wall clock. Both write to the
+`"encode"` stream. `enqueueIngestBatchImport(_:)` is the bulk-import
+twin. It writes the same durable rows to the `"import"` stream instead,
+where only the import drain worker claims them. The batch variants each
+wrap all inserts in one transaction, which removed the last full-core
bottleneck of bulk imports on encrypted SQLite.
+
`awaitIngestDrain(timeout:)` is the barrier importers use to know writes
-are searchable. `setOnEncoded(_:)` installs the one callback CorpusKit
-ever makes toward an orchestrator. The `IngestJob` wire format's JSON
-field names form a pinned cross-port contract with the Rust twin.
+are searchable on the `"encode"` stream. `ingestQueueDepth()` reports
+pending and in-flight counts for that same stream. Both counts are now
+stream-scoped. The in-flight count once counted jobs across every
+stream, so a concurrent bulk import could inflate the number an encode
+caller read. `importQueueDepth()` is the same read, scoped to the
+`"import"` stream instead. `setOnEncoded(_:)` installs the one callback
+CorpusKit ever makes toward an orchestrator. The `IngestJob` wire
+format's JSON field names form a pinned cross-port contract with the
+Rust twin, shared by both streams.
## BasisCodec.swift
diff --git a/packages/kits/CorpusKit/docs/OVERVIEW.md b/packages/kits/CorpusKit/docs/OVERVIEW.md
index b84e656..1c665f6 100644
--- a/packages/kits/CorpusKit/docs/OVERVIEW.md
+++ b/packages/kits/CorpusKit/docs/OVERVIEW.md
@@ -16,9 +16,9 @@ sources:
- path: Sources/CorpusKit/Chunker.swift
blob: a2718e06d1715f539ff633e7037c70e10ecb7a2d
- path: Sources/CorpusKit/CorpusIngestQueue.swift
- blob: 2c32133701ce728bc017d4ddad51b052cae990db
+ blob: 4dd3bd8eefb8e0e7dd474793d35acbe03b452df3
- path: Sources/CorpusKit/CorpusKit.swift
- blob: 4518f15fdb798c3a203c4a9db949f4d6172f540d
+ blob: bd4433bb5a7dc347df5e24c04d5f76a533261e54
- path: Sources/CorpusKit/CorpusKitError.swift
blob: 68ac8d0a248bc9c2dd1885b0bc531ac4ed9cb91d
- path: Sources/CorpusKit/CorpusProviderCountsStore.swift
@@ -30,7 +30,7 @@ sources:
- path: Sources/CorpusKit/Engine/InvertedIndex.swift
blob: 1273adcb3794b1997c93488182fc5ed95b21f9ec
- path: Sources/CorpusKit/Engine/InvertedIndexStore.swift
- blob: 242baf05e5c846c719c403599a9a407bba646f5b
+ blob: 14f9de0b8e7ce0eae638605f24e43493afa0ac4e
- path: Sources/CorpusKit/Engine/SparseTypes.swift
blob: 54654e2c49b09d31f60c06503216a2b281939f87
- path: Sources/CorpusKit/HybridRecall.swift
@@ -146,6 +146,16 @@ identity. A repeat ingest of one document is therefore a harmless
no-op. Two federated devices that ingest the same content converge on
identical rows.
+A second drain worker handles bulk import. It claims jobs from its own
+queue stream and never touches the daily-driving stream above. It
+chunks each item and updates the keyword index, but it skips training
+and skips embedding. A bulk import instead trains its basis once. It
+then embeds every chunk once, at the end, through an explicit reindex.
+Skipping per-item training and embedding turns a large import from
+repeated wasted work into one pass. For a SQLite estate, the import
+worker also shards the chunking and tokenizing work across cores, then
+merges the results back through one writer.
+
Each stored chunk is indexed twice. The keyword side tokenizes the
chunk. It records term frequencies in a persistent inverted index. An
inverted index is a table that maps each word to the chunks that contain
diff --git a/packages/kits/CorpusKit/docs/topology.svg b/packages/kits/CorpusKit/docs/topology.svg
index 228e080..38ee800 100644
--- a/packages/kits/CorpusKit/docs/topology.svg
+++ b/packages/kits/CorpusKit/docs/topology.svg
@@ -7,15 +7,15 @@ authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7
authored_date: 2026-07-04
sources:
- path: Sources/CorpusKit/CorpusKit.swift
- blob: 4518f15fdb798c3a203c4a9db949f4d6172f540d
+ blob: bd4433bb5a7dc347df5e24c04d5f76a533261e54
- path: Sources/CorpusKit/CorpusIngestQueue.swift
- blob: 2c32133701ce728bc017d4ddad51b052cae990db
+ blob: 4dd3bd8eefb8e0e7dd474793d35acbe03b452df3
- path: Sources/CorpusKit/Chunker.swift
blob: a2718e06d1715f539ff633e7037c70e10ecb7a2d
- path: Sources/CorpusKit/BundleStore.swift
blob: 419b1c0609597cdd68bf623ed37bd40a0171597b
- path: Sources/CorpusKit/Engine/InvertedIndexStore.swift
- blob: 242baf05e5c846c719c403599a9a407bba646f5b
+ blob: 14f9de0b8e7ce0eae638605f24e43493afa0ac4e
- path: Sources/CorpusKit/BasisStore.swift
blob: 48850906faa7c2fe4aac2859a1c4e892cff32cab
- path: Sources/CorpusKit/HybridRecall.swift
@@ -51,7 +51,7 @@ sources:
Ingest queue
- QueueKit · encode stream
+ QueueKit · encode + import streams
Corpus actor
diff --git a/packages/kits/LocusKit/docs/AGENT_MAP.md b/packages/kits/LocusKit/docs/AGENT_MAP.md
index e1cac03..72377e7 100644
--- a/packages/kits/LocusKit/docs/AGENT_MAP.md
+++ b/packages/kits/LocusKit/docs/AGENT_MAP.md
@@ -98,7 +98,7 @@ sources:
- path: Sources/LocusKit/Summaries.swift
blob: fa624f6b00d8bf40f1ccc02821e3e2b94bf94f6e
- path: Sources/LocusKit/Tunnel.swift
- blob: 8da22988147aab0a9451f06ec681d10891da1e95
+ blob: 8b1590668c930fb0e03eb2e89d2c31cd3d66c34b
- path: Sources/LocusKit/TunnelOperational.swift
blob: a5b9b8d3b5b7990ebcb9d9c6209564a7307aa13e
---
@@ -322,10 +322,11 @@ ENTRY POINTS (most callers need only these):
- :654 `sort(_:ordering:nodeNames:)` (private)
### Tunnel : Tunnel.swift
-- :24 `struct Tunnel: Equatable, Hashable, Codable, Sendable`
-- :29 `id` (conventionally SHA-256 of canonicalised endpoint pair : NOT enforced at this layer, LOCI-5)
+- :25 `struct Tunnel: Equatable, Hashable, Codable, Sendable`
+- :30 `id` (conventionally SHA-256 of canonicalised endpoint pair : NOT enforced at this layer, LOCI-5)
- source/target: wing+room+optional drawerId (nil = "the room itself")
-- :63 `kind: TunnelKind` (default .references) / :69/:75/:80 adjective/operational/provenance bitmaps / :99 `orderKey: Double?` (fractional-index sibling order, .parent kind only)
+- :64 `kind: TunnelKind` (default .references) / :70/:76/:81 adjective/operational/provenance bitmaps / :100 `orderKey: Double?` (fractional-index sibling order, .parent kind only)
+- :153 `adjectiveSensitivity: AdjectiveSensitivity` (extension, imports SubstrateKernel for BitField) : bits 6-11 of adjectiveBitmap, fallback .normal; parity peer of Rust Tunnel::adjective_sensitivity
### Tunnel operational : TunnelOperational.swift
- :38 `enum TunnelKind: Int` : supersedes=0…parent=9 (outline containment, ADR-017 §11; source=child, target=parent, one-parent-per-child kit-level constraint)
diff --git a/packages/kits/LocusKit/docs/DETAILS.md b/packages/kits/LocusKit/docs/DETAILS.md
index d60564c..5f2ef43 100644
--- a/packages/kits/LocusKit/docs/DETAILS.md
+++ b/packages/kits/LocusKit/docs/DETAILS.md
@@ -98,7 +98,7 @@ sources:
- path: Sources/LocusKit/Summaries.swift
blob: fa624f6b00d8bf40f1ccc02821e3e2b94bf94f6e
- path: Sources/LocusKit/Tunnel.swift
- blob: 8da22988147aab0a9451f06ec681d10891da1e95
+ blob: 8b1590668c930fb0e03eb2e89d2c31cd3d66c34b
- path: Sources/LocusKit/TunnelOperational.swift
blob: a5b9b8d3b5b7990ebcb9d9c6209564a7307aa13e
---
@@ -887,6 +887,15 @@ the same parent sort by ascending `orderKey`. So no sibling needs to
be rewritten when a new one is inserted between two existing
siblings.
+This file also provides `Tunnel.adjectiveSensitivity`. This computed
+accessor decodes bits six through eleven of `adjectiveBitmap` as an
+`AdjectiveSensitivity`. An unrecognised raw value falls back to
+`.normal`. This fail-closed default matches `KGFact.adjectiveSensitivity`
+and `Drawer.adjectiveSensitivity`. The accessor is named
+`adjectiveSensitivity`, not `sensitivity`, so it does not collide with
+the provenance-bitmap `sensitivity` accessor pattern used elsewhere in
+LocusKit.
+
## TunnelOperational.swift
This file provides `TunnelKind`, the ten-case closed relationship
diff --git a/packages/kits/LocusKit/docs/OVERVIEW.md b/packages/kits/LocusKit/docs/OVERVIEW.md
index 063b643..a575d3f 100644
--- a/packages/kits/LocusKit/docs/OVERVIEW.md
+++ b/packages/kits/LocusKit/docs/OVERVIEW.md
@@ -98,7 +98,7 @@ sources:
- path: Sources/LocusKit/Summaries.swift
blob: fa624f6b00d8bf40f1ccc02821e3e2b94bf94f6e
- path: Sources/LocusKit/Tunnel.swift
- blob: 8da22988147aab0a9451f06ec681d10891da1e95
+ blob: 8b1590668c930fb0e03eb2e89d2c31cd3d66c34b
- path: Sources/LocusKit/TunnelOperational.swift
blob: a5b9b8d3b5b7990ebcb9d9c6209564a7307aa13e
---
diff --git a/packages/kits/VectorKit/docs/AGENT_MAP.md b/packages/kits/VectorKit/docs/AGENT_MAP.md
index 8ff7873..1dcec4e 100644
--- a/packages/kits/VectorKit/docs/AGENT_MAP.md
+++ b/packages/kits/VectorKit/docs/AGENT_MAP.md
@@ -40,14 +40,14 @@ sources:
- path: Sources/VectorKit/VectorMatch.swift
blob: 24cc2c1bd25f71a7cef60a043c3a640df2368a23
- path: Sources/VectorKit/VectorStore.swift
- blob: 3c7fe4a19eba1142ac82a993cee0e7660a4ffdce
+ blob: 45a4a6290398c27b383bc06ac5999cfaa0ca3422
---
# AGENT_MAP | VectorKit
PURPOSE: on-device embedding generation (`EmbeddingProvider` seam) + model-tagged vector storage (`VectorStore`, PersistenceKit-backed) + dual-lane nearest-neighbour search: binary Hamming (Lane A `BruteForceIndex` oracle / Lane B `MIHIndex` sub-linear exact, promoted at `mihThreshold`) and float cosine/l2/dot (`FloatBruteForceIndex`, one index per modelID) + ColBERT MaxSim late-interaction scorer (`MaxSimScorer`, standalone, not wired into VectorStore).
-DEPS: imports EngramLib (Engram type, Hamming kernel via EngramLib.distances/Session | I-7 absolute), SubstrateML (FloatSimHash.project), SubstrateTypes, PersistenceKit (Storage/RowStore/BlobStore, product "PersistenceKit"), IntellectusLib (Intellectus.report telemetry, no-op when disabled). Test target additionally depends on PersistenceKitInMemory, PersistenceKitSQLite. Imported by: CorpusKit / CorpusKitProviders (concrete text embedding providers built on FloatSimHashEmbeddingProvider), GeniusLocusKit (destroyAllVectors as part of estate teardown). Rust port in rust/ mirrors every file (vector_store.rs, engine/{brute_force,mih,float_brute_force,max_sim,resident,resident_store,key,payload,hit,metric,seam}.rs, embedding_provider.rs, simhash_embedding_provider.rs, error.rs); no shared cross-language fixture file | conformance rests on both ports implementing the documented algorithms identically (colex enumeration, sidecar byte layout, budget arithmetic). Float lane is explicitly NOT four-way bit-identical (documented, not a gap).
+DEPS: imports EngramLib (Engram type, Hamming kernel via EngramLib.distances/Session | I-7 absolute), SubstrateML (FloatSimHash.project), SubstrateTypes, PersistenceKit (Storage/RowStore/BlobStore, product "PersistenceKit"), IntellectusLib (Intellectus.report telemetry, no-op when disabled). Test target additionally depends on PersistenceKitInMemory, PersistenceKitSQLite. Imported by: CorpusKit / CorpusKitProviders (concrete text embedding providers built on FloatSimHashEmbeddingProvider; CorpusKit's reindex job also calls replaceModelVectors for a full-model re-embed), GeniusLocusKit (destroyAllVectors as part of estate teardown). Rust port in rust/ mirrors every file (vector_store.rs, engine/{brute_force,mih,float_brute_force,max_sim,resident,resident_store,key,payload,hit,metric,seam}.rs, embedding_provider.rs, simhash_embedding_provider.rs, error.rs); no shared cross-language fixture file | conformance rests on both ports implementing the documented algorithms identically (colex enumeration, sidecar byte layout, budget arithmetic). Float lane is explicitly NOT four-way bit-identical (documented, not a gap).
ENTRY POINTS (most callers need only these):
- VectorStore.swift:451 `VectorStore.addVector(itemID:engram:modelID:modelVersion:filedAt:)` | write one binary vector
@@ -175,7 +175,9 @@ ENTRY POINTS (most callers need only these):
- :1294 `findFarthestFloat(probe:modelID:limit:)` | same as findNearestFloat but calls searchFarthest; anti-similarity
- :1337 `findByKeyword(_:limit:)` | substring LIKE on item_id; NOT full BM25 (CorpusKit's job); emits vectorkit.search.keyword_result_count
- :1377 `deleteVector(itemID:modelID:)` / :1386 `deleteAllVectors(itemID:modelID:)` | both flush pending deferred burst FIRST if dirty, then delete+tombstone; deleteAllVectors invalidates that model's float index (lazy rebuild)
+- :1437 `replaceModelVectors(modelID:_:)` | BATCH reindex re-embed path, deliberately separate from addPayloads/deleteAllVectors (per-key mutation is O(n²) over a full re-embed); bulk-deletes + plain-inserts the model's rows in ONE transaction (single fsync, no per-row existence SELECT), drops the model's Lane D float index, then calls _rebuildBinaryIndexFromTable ONCE; caller is CorpusKit's reindex job
- :1442 `destroyAllVectors()` | full wipe: table+both binary indexes+sidecar+ALL float indices; used by GeniusLocusKit estate teardown
+- :1485 `_rebuildBinaryIndexFromTable()` | private; rebuilds sidecar array + BruteForceIndex + MIHIndex from the durable table in ONE pass; used only by replaceModelVectors; unlike _ensureIndexBuilt it does NOT trust the sidecar live-count (a re-embed can keep the SAME row count, so a live-count match would wrongly keep a stale sidecar) | always rereads the table
- :1487 `_ensureIndexBuilt()` | idempotent; sidecar trusted iff `snap.liveCount == tableCount` (live-vs-live, NOT snap.count vs table | avoids spurious rebuild after deletes, "C5 fix")
- :1558 `_ensureFloatIndexBuilt(modelID:)` | nil return (no cache) when model has zero float rows, so a later first-ingest can still build a real index
- :1736 `_deleteAndTombstone(itemID:vectorIndex:modelID:)` | scans ALL slots matching (itemID,vectorIndex,modelID) across modelVersions, tombstones every match (no break-after-first) | hard-delete contract
@@ -200,6 +202,8 @@ ENTRY POINTS (most callers need only these):
- FloatBruteForceIndex establishes its stride from the FIRST vector added via add(key:vector:); a later vector of different byte count throws rather than corrupting the flat storage buffer. One index = one dimension, always.
- FloatBruteForceIndex is labeled "Lane C" in its own file header but "Lane D" in VectorStore.swift's comments | same type, inconsistent label, not a functional divergence. Do not "fix" one file to match the other without checking whether a Lane-lettering convention doc elsewhere disambiguates it first.
- MaxSimScorer (Lane E1) is NOT wired into VectorStore's public search API in this package version | it is a standalone scorer callers invoke directly with pre-fetched token-Engram arrays. Do not assume `findNearest` performs late interaction.
+- replaceModelVectors is deliberately NOT a thin wrapper over addPayloads+deleteAllVectors | those two mutate the resident index per key, and every BruteForceIndex add/remove rebuilds all partitions (O(n)), so N of them (a full re-embed) is O(n²); replaceModelVectors instead does one durable-table transaction + one resident-index rebuild via _rebuildBinaryIndexFromTable. Do not "simplify" replaceModelVectors to call the per-key paths in a loop.
+- _rebuildBinaryIndexFromTable (replaceModelVectors' private helper) intentionally does NOT use the `_ensureIndexBuilt` live-count trust check | a re-embed can replace every vector for a model while the row count stays identical, so a live-count match would wrongly keep the pre-re-embed sidecar. It always rereads the table.
- Telemetry (Intellectus.report) is off by default; the emitted metrics (vectorkit.index.insert_latency_ms, .batch_insert_latency_ms, vectorkit.search.latency_ms, .result_count, .keyword_result_count, vectorkit.mih.enumeration_fallback) are named constants used by dashboards | renaming any of them is a breaking change to monitoring, not just to code.
- Pinned/default constants | changing any requires updating dependent conformance tests: mihThreshold 50,000, mihBandCount .m16 (sub_bits=16), deferredPendingLimit 50,000, compactionThreshold 0.25, poolSubmitThreshold-equivalent N/A (not used here), sidecar format version 0x0002, MIHBandCount ∈ {4,8,16,32}, cosine-distance quantization scale ×10,000.
- Actor boundaries: VectorStore, BruteForceIndex, MIHIndex, FloatBruteForceIndex, ResidentArrayStore are all actors | all mutation and reads are serialized per actor. ResidentVectorArray, VectorPayload, VectorRecordKey, DenseHit, VectorMatch, StoredVector are plain Sendable value types safe to pass across actor boundaries once constructed.
diff --git a/packages/kits/VectorKit/docs/DETAILS.md b/packages/kits/VectorKit/docs/DETAILS.md
index 95ce027..3d51cf1 100644
--- a/packages/kits/VectorKit/docs/DETAILS.md
+++ b/packages/kits/VectorKit/docs/DETAILS.md
@@ -40,7 +40,7 @@ sources:
- path: Sources/VectorKit/VectorMatch.swift
blob: 24cc2c1bd25f71a7cef60a043c3a640df2368a23
- path: Sources/VectorKit/VectorStore.swift
- blob: 3c7fe4a19eba1142ac82a993cee0e7660a4ffdce
+ blob: 45a4a6290398c27b383bc06ac5999cfaa0ca3422
---
# VectorKit Details
@@ -660,6 +660,30 @@ store: every row, both resident indexes, the sidecar, and the per-model
float indexes. This runs as part of a coordinated estate teardown. An
estate is one user's complete memory store in MOOTx01.
+### Batch reindex path
+
+`replaceModelVectors(modelID:_:)` replaces an entire model's vectors in
+one re-embed pass. It exists apart from `addPayloads` and
+`deleteAllVectors`, which live captures keep using unchanged. Those two
+paths mutate the resident index one key at a time. Every add or remove
+on `BruteForceIndex` rebuilds every partition. That cost is fine for the
+small, frequent writes those two paths serve. It is not fine for a full
+re-embed of tens of thousands of rows, an operation that would cost
+`O(n²)` through that path. `replaceModelVectors` instead deletes and
+inserts the whole model inside one durable-table transaction. This is
+one fsync for the whole batch, not one per row. The bulk delete runs
+first, so the following inserts never conflict, and each insert skips
+the usual per-row existence check. After the transaction commits, the
+resident binary index is rebuilt once, by the private
+`_rebuildBinaryIndexFromTable()`. This function does not trust the
+sidecar's live-count check that `_ensureIndexBuilt()` uses elsewhere. A
+re-embed can replace every vector while the row count for the model
+stays the same. A live-count match in that case would wrongly keep a
+now-stale sidecar. `_rebuildBinaryIndexFromTable()` always rereads the
+table instead, so this case cannot fool it. The rebuild also drops the
+model's Lane D float index, so the next float search for that model
+rebuilds it from the fresh rows.
+
### Coherence helpers
`_ensureIndexBuilt()` is the one-time, per process, function that
diff --git a/packages/kits/VectorKit/docs/OVERVIEW.md b/packages/kits/VectorKit/docs/OVERVIEW.md
index 4aaccfc..a3488c7 100644
--- a/packages/kits/VectorKit/docs/OVERVIEW.md
+++ b/packages/kits/VectorKit/docs/OVERVIEW.md
@@ -40,7 +40,7 @@ sources:
- path: Sources/VectorKit/VectorMatch.swift
blob: 24cc2c1bd25f71a7cef60a043c3a640df2368a23
- path: Sources/VectorKit/VectorStore.swift
- blob: 3c7fe4a19eba1142ac82a993cee0e7660a4ffdce
+ blob: 45a4a6290398c27b383bc06ac5999cfaa0ca3422
---
# VectorKit Overview
@@ -128,6 +128,15 @@ to re-read the database. The in-memory array can optionally be backed
by an on-disk cache file, called a sidecar. This lets the store reopen
without rebuilding the array from every database row.
+A whole model can also be replaced in one batch. This path is
+`VectorStore.replaceModelVectors`. It deletes every vector for one model
+and inserts the fresh batch. It does this inside one transaction, so it
+costs one disk sync for the whole batch. It then rebuilds the in-memory
+array once, rather than once per vector. A model reindex changes every
+vector at once. Updating the in-memory array one vector at a time would
+be far slower for that case. This batch path keeps a full model
+re-embed fast.
+
Searching has two independent paths, one per vector kind. A binary
search, `findNearest`, compares the query fingerprint against the
in-memory array using Hamming distance. Below a configurable threshold
diff --git a/packages/kits/VectorKit/docs/topology.svg b/packages/kits/VectorKit/docs/topology.svg
index 6541f48..6d1d20f 100644
--- a/packages/kits/VectorKit/docs/topology.svg
+++ b/packages/kits/VectorKit/docs/topology.svg
@@ -11,7 +11,7 @@ sources:
- path: Sources/VectorKit/FloatSimHashEmbeddingProvider.swift
blob: efcb85396ceace1373e3017c0f799371a9a5c3bf
- path: Sources/VectorKit/VectorStore.swift
- blob: 3c7fe4a19eba1142ac82a993cee0e7660a4ffdce
+ blob: 45a4a6290398c27b383bc06ac5999cfaa0ca3422
- path: Sources/VectorKit/VectorMatch.swift
blob: 24cc2c1bd25f71a7cef60a043c3a640df2368a23
- path: Sources/VectorKit/Engine/ResidentArrayStore.swift