From 4434063bed8c365c24d4ca048911a4a59f97ea45 Mon Sep 17 00:00:00 2001 From: Fable Date: Sat, 4 Jul 2026 16:06:53 -0500 Subject: [PATCH 1/2] =?UTF-8?q?docs(sdk):=20package=20docs=20fan-out=20?= =?UTF-8?q?=E2=80=94=20OVERVIEW,=20DETAILS,=20AGENT=5FMAP,=20topology=20pe?= =?UTF-8?q?r=20package?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3-4 of the Moot-SDK documentation initiative. Voice: Turabian at 8th grade per the LatticeLib pilot exemplar. Front-matter pins per-repo authored_commit and per-source git blob hashes for docs-scrub. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Pzkqg3bpbbhfuWRsYomJ3o --- packages/kits/CorpusKit/docs/AGENT_MAP.md | 243 +++ packages/kits/CorpusKit/docs/DETAILS.md | 931 ++++++++++++ packages/kits/CorpusKit/docs/OVERVIEW.md | 202 +++ packages/kits/CorpusKit/docs/topology.svg | 126 ++ packages/kits/LocusKit/docs/AGENT_MAP.md | 481 ++++++ packages/kits/LocusKit/docs/DETAILS.md | 1315 +++++++++++++++++ packages/kits/LocusKit/docs/OVERVIEW.md | 304 ++++ packages/kits/LocusKit/docs/topology.svg | 129 ++ packages/kits/VectorKit/docs/AGENT_MAP.md | 205 +++ packages/kits/VectorKit/docs/DETAILS.md | 658 +++++++++ .../kits/VectorKit/docs/INTERFACE_DOCTRINE.md | 79 - packages/kits/VectorKit/docs/OVERVIEW.md | 202 +++ packages/kits/VectorKit/docs/topology.svg | 124 ++ 13 files changed, 4920 insertions(+), 79 deletions(-) create mode 100644 packages/kits/CorpusKit/docs/AGENT_MAP.md create mode 100644 packages/kits/CorpusKit/docs/DETAILS.md create mode 100644 packages/kits/CorpusKit/docs/OVERVIEW.md create mode 100644 packages/kits/CorpusKit/docs/topology.svg create mode 100644 packages/kits/LocusKit/docs/AGENT_MAP.md create mode 100644 packages/kits/LocusKit/docs/DETAILS.md create mode 100644 packages/kits/LocusKit/docs/OVERVIEW.md create mode 100644 packages/kits/LocusKit/docs/topology.svg create mode 100644 packages/kits/VectorKit/docs/AGENT_MAP.md create mode 100644 packages/kits/VectorKit/docs/DETAILS.md delete mode 100644 packages/kits/VectorKit/docs/INTERFACE_DOCTRINE.md create mode 100644 packages/kits/VectorKit/docs/OVERVIEW.md create mode 100644 packages/kits/VectorKit/docs/topology.svg diff --git a/packages/kits/CorpusKit/docs/AGENT_MAP.md b/packages/kits/CorpusKit/docs/AGENT_MAP.md new file mode 100644 index 0000000..075f5ab --- /dev/null +++ b/packages/kits/CorpusKit/docs/AGENT_MAP.md @@ -0,0 +1,243 @@ +--- +doc: AGENT_MAP +package: CorpusKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/CorpusKit/BasisStore.swift + blob: 48850906faa7c2fe4aac2859a1c4e892cff32cab + - path: Sources/CorpusKit/BM25Index.swift + blob: 06fb90cd40e81f013e01a8a6c4c6f94e71bf33f3 + - path: Sources/CorpusKit/BundleStore.swift + blob: 419b1c0609597cdd68bf623ed37bd40a0171597b + - path: Sources/CorpusKit/Chunk.swift + blob: d5a1be1bb08858f5f7bd59bb141a8a0ba6f1dfbe + - path: Sources/CorpusKit/Chunker.swift + blob: a2718e06d1715f539ff633e7037c70e10ecb7a2d + - path: Sources/CorpusKit/CorpusIngestQueue.swift + blob: 2c32133701ce728bc017d4ddad51b052cae990db + - path: Sources/CorpusKit/CorpusKit.swift + blob: 4518f15fdb798c3a203c4a9db949f4d6172f540d + - path: Sources/CorpusKit/CorpusKitError.swift + blob: 68ac8d0a248bc9c2dd1885b0bc531ac4ed9cb91d + - path: Sources/CorpusKit/CorpusProviderCountsStore.swift + blob: c92160041765cc8546501c5ac4d8a2b769656e93 + - path: Sources/CorpusKit/Engine/BM25Weighting.swift + blob: 622f45870ab1118d7590cce1b379a063619b4714 + - path: Sources/CorpusKit/Engine/Fusion.swift + blob: d128ed9bc206612fc7c2e849a2e77d03d8a6cafa + - path: Sources/CorpusKit/Engine/InvertedIndex.swift + blob: 1273adcb3794b1997c93488182fc5ed95b21f9ec + - path: Sources/CorpusKit/Engine/InvertedIndexStore.swift + blob: 242baf05e5c846c719c403599a9a407bba646f5b + - path: Sources/CorpusKit/Engine/SparseTypes.swift + blob: 54654e2c49b09d31f60c06503216a2b281939f87 + - path: Sources/CorpusKit/HybridRecall.swift + blob: 21d9fb3415b699c6469a1e80d5e84da1e43981ca + - path: Sources/CorpusKit/RemovedSourceStore.swift + blob: 138a2a094ff369ee69eafa04eeba77206a58e5f4 + - path: Sources/CorpusKit/SyncManifest.swift + blob: 39af591d1fbf1f213c93eb143c213258e41c6c4b + - path: Sources/CorpusKit/Tokenizer.swift + blob: 603028510f91b3c6d75cdda1cb0a1db1c59eee28 + - path: Sources/CorpusKit/TrainableEmbeddingBasis.swift + blob: 4722def84980b3a8987adb090a1a702ac789f8ab + - path: Sources/CorpusKitProviders/BasisCodec.swift + blob: d107e1efd6341648fd8f717c7956a15b98c1b29f + - path: Sources/CorpusKitProviders/DefaultEnsemble.swift + blob: c58168f0991cc4e4c3ec490e2272bdc1a5a17be1 + - path: Sources/CorpusKitProviders/DeterministicTokenizer.swift + blob: 0586b3a4ae93dc0b58ef8f62c0d104a81dcfefe3 + - path: Sources/CorpusKitProviders/EmbeddingGemmaProvider.swift + blob: 593cd5952aad04fe0390e133e68cc07cad983d86 + - path: Sources/CorpusKitProviders/FdcProvider.swift + blob: 96f3ffaf64c7b17e2f617f49ce06cd649e70a027 + - path: Sources/CorpusKitProviders/LsaProvider.swift + blob: 3870f0d24659cb27ba13dc2cba9f94debb6b5c07 + - path: Sources/CorpusKitProviders/MiniLMTextProvider.swift + blob: 35c739a37e9ef1a92098458b48c5f5a06f11050f + - path: Sources/CorpusKitProviders/MPNetTextProvider.swift + blob: 6f13fc6dcd4733460cad366e78273f542c65a844 + - path: Sources/CorpusKitProviders/NLContextualEmbeddingProvider.swift + blob: 17d1acc363bab9c5d0bb807041e0b4c3d66fa0ee + - path: Sources/CorpusKitProviders/NLEmbeddingProvider.swift + blob: 2d714a051a1b6bd4dbde1ecd0e182a81f94b7008 + - path: Sources/CorpusKitProviders/NmfProvider.swift + blob: 43f0e339a42426d68788a03e8a216890c73eb05a + - path: Sources/CorpusKitProviders/PpmiProvider.swift + blob: 282e2185cb7e3d066979ea23b74e096ee337545b + - path: Sources/CorpusKitProviders/RandomIndexingProvider.swift + blob: 552b55b1be93fb57b9e7daf123ecf3a73df7abef + - path: Sources/CorpusKitProviders/ReducedVocab.swift + blob: fb50a8566f9ef3b2a9c650102274b20894d6542d + - path: Sources/CorpusKitProviders/TermDocumentCounts.swift + blob: e72231cf3e50799b6bbeac6a165b80410dc40317 +--- + +# AGENT_MAP — CorpusKit + +PURPOSE: standalone on-device RAG kit. Text → chunks (content-addressed UUID) → BundleStore (PersistenceKit) + persistent BM25 inverted index + per-provider vectors (VectorKit) → hybrid recall (Hamming kNN + BM25, weighted RRF) → [ScoredChunk]. Ships two targets: CorpusKit (core: stores, engines, protocols) and CorpusKitProviders (concrete embedding providers/tokenizers). Default production ensemble = five honest deterministic signals (RI/PPMI/LSA/NMF/FDC). + +DEPS: CorpusKit imports SubstrateTypes, SubstrateLib (MerkleHash), SubstrateML, EngramLib, EideticLib (sentence segmentation), IntellectusLib (telemetry, off-by-default), PersistenceKit (+InMemory, +SQLite), ConvergenceKit (manifest only), VectorKit, QueueKit, Crypto. CorpusKitProviders additionally imports SubstrateKernel (FloatVecOps), LatticeLib (FDC runtime; FDC math NOT reimplemented). Imported by: GeniusLocusKit (orchestrator tier). Rust ports: `rust/` (core, crate corpus-kit) + `rust-providers/` (crate corpus-kit-providers); shared fixtures `Tests/SharedVectors/*.json` read by BOTH legs gate bit-identity. NL providers are Swift-only (ADR-019, no Rust twin). + +ENTRY POINTS (most callers need only these): +- CorpusKit.swift:702 `Corpus.init(storage:models:)` — open estate corpus; `models[0]` = default signal (:674 single-model convenience) +- CorpusKit.swift:1001 `Corpus.ingest(_ text:sourceID:now:)` — synchronous chunk+index+embed +- CorpusIngestQueue.swift:157 `Corpus.enqueueIngest(_:sourceID:now:)` — async queued ingest (production path) +- CorpusKit.swift:1629 `Corpus.recall(_ query:limit:now:) -> [ScoredChunk]` — hybrid RRF recall on default signal +- DefaultEnsemble.swift:62 `CorpusEnsemble.defaultEnsemble() -> [EmbeddingModel]` — the five production signals, fresh per estate + +## Symbol Table + +### Facade — CorpusKit.swift +- :48 `enum FloatLaneOutcome` — `.hits`/`.unavailableProviderOptOut`/`.unavailableNoVocabHit`/`.unavailableNoFloatRows`/`.emptyQuery`/`.storeError`; dark lanes are typed outcomes, NEVER errors +- :134 `enum EmbeddingModel` — `.deterministic` (:147, seed 0xC05B_D15C_A15D_1B00, federation baseline), `.miniLM/.mpNet/.embeddingGemma(inference:)` (:157/:167/:177, host closure), `.randomIndexing/.ppmi/.lsa/.nmf(provider:)` (:194–:233, trainable), `.fdc(provider:)` (:254, stateless), `.nlEmbedding/.nlContextualEmbedding` (:275/:293, Apple-only); `.default = .deterministic` (:297) +- :338 `EmbeddingModel.isTrainable` — true iff carried provider conforms to TrainableEmbeddingBasis +- :362 `EmbeddingModel.reconstruct(from: Data)` — routes blob to concrete type; throws `.notTrainable` +- :417 `enum EncodeSpeed` — `.foreground` (all cores) / `.background` (cores/4, floor 1) +- :426 `public actor Corpus` — composition root; sealed-vector principle (no VectorKit type in public API except :649 `sharedVectorStore`) +- :491 `setEncodeSpeed(_:)`; :641 `onEncoded` callback (set via CorpusIngestQueue.swift:134 `setOnEncoded`) +- :1001 `ingest(_:sourceID:now:)` — idempotent; re-ingest clears tombstone; first-ingest auto-train (gate = no persisted basis, NOT factory-blob presence) +- :1181 `ingestBatch(_:)` — identical output to per-item; commit windows 512 items / 4096 rows (:436–:437); slice-parallel embed; batch-aware first-basis bootstrap (train once on full batch, never per item) +- :1416 `maintainedVocabAnchor() -> Int` — governor's vocab-growth retrain trigger read +- :1488 `reindex(now:)` — THE explicit retrain: reconstruct-fresh → trainOnCorpus(active chunks) → upsert basis → re-embed all under every slot; excludes removed sources +- :1629 `recall(_:limit:now:)`; :1657 `remove(sourceID:)` (BM25 rows + ALL models' vectors + tombstone; chunks kept); :1711 `expunge(sourceID:)` (scrubText FIRST, then remove); :1738 `destroyRecallIndex()` (all derived state; chunks survive) +- :1779 `bm25TopKBySource(query:limit:)` — pure keyword lane, max-chunk-score per source, frontierK ≤ 256 +- :1833 `embed(_:) -> Engram`; :1846 `modelID`; :1863 `embedFloat(_:)` (throws on provider opt-out); :2238 `supportsFloat` +- :1895 `floatNearest(query:limit:) -> FloatLaneOutcome` — never throws; sim = 1 − distance/10_000; source aggregation = MAX chunk cosine +- :2141 `floatNearestPerSignal(query:limit:)` — per-slot dense lanes in slot order, NO fusion (caller fuses); :2214 `floatFarthestPerSignal` — anti-similarity, MIN chunk cosine per source, ascending +- :2249 `count()` (excludes removed sources); :2266 `indexedSourceIDs()`; :2274/:2280 `corpusMerkleRoot(for:)`/`globalCorpusMerkleRoot()` +- :2300 `EmbeddingModel.makeProvider()` (private) — pinned seeds: miniLM 0x4D49_4E4C_4D5F_7631 "MINLM_v1", mpNet 0x4D50_4E45_545F_7631 "MPNET_v1", embeddingGemma 0x454D_4247_4D5F_7631 "EMBGM_v1"; model IDs corpus-deterministic-v1 / minilm-v6 / mpnet-base-v2 / embedding-gemma-300m +- :2413 `CorpusDefaultTokenizer` (internal) — FNV-1a fold, duplicated from providers to avoid circular dep; :2445 `CorpusTextProvider` (private) — tokenize→inference→FloatSimHash; :2498 `embedPair` computes pooled vector ONCE for both lanes +- Test seams (never production): :914 init(storage:provider:), :980 `_testForceFloatStoreError`, :656 `_ingestFailureHook` + +### Ingest queue — CorpusIngestQueue.swift (extension Corpus) +- :63 `mountIngestQueue()` — idempotent; SQLite estate → encrypted sibling `queue.sqlite` via `EstateConfiguration.queueSibling` (ADR-021 D7/T4; replaced plaintext maildir hole); InMemory estate → fixed :486 `ingestQueueStoreID` (no UUID() nondeterminism) +- :120 `dropIngestQueue()`; :134 `setOnEncoded(_:)` — the ONLY CorpusKit→orchestrator callback +- :157 `enqueueIngest(...)`; :179 `enqueueIngestBatch(...)` — one transaction for all jobs (bulk-import bottleneck fix; caller bounds batch size) +- :205 `awaitIngestDrain(timeout: 30s)` — barrier: drained AND vector index republished; throws drainTimeout +- :229 `ingestQueueDepth() -> (pending, inFlight)` — pending IS stream-scoped, inFlight is NOT (all streams) +- :248 `drainIngestQueueOnce()` — claims whole batch; undecodable → `.blocked` (terminal), empty text → `.done`; batch `ingestBatch` + bulk session reply; falls back serial on batch throw +- :341 `runIngestDrainLoop` (private) — DrainLease single drainer; first-acquire crash recovery `reclaimInFlight`; standby poll 3 s; lease TTL 15 s (failover ≈ 15–18 s, not instant); spin-while-draining, 15 ms idle sleep; vector index published once per burst (O(N) not O(N²)) +- :430 `ingestOneAndReply` (private) — retry in place ≤ :476 `ingestMaxAttempts = 8`, then `.blocked`; sound ONLY because ingest is idempotent +- :480 `encodeStreamID = "encode"` — EVERY queue op must be scoped to it (shared queue.sqlite may carry other streams; unscoped awaitDrain deadlocks) +- :515 `IngestJob` — wire fields `sourceID`/`text`/`capturedAtISO8601` = pinned cross-port JSON contract; :551 `toJob`, :563 `from(job:)` + +### Chunks — Chunk.swift / Chunker.swift +- Chunk.swift:35 `struct Chunk: Sendable, Equatable, Codable` — immutable, content-addressed +- Chunk.swift:68 content-addressed init (normal path); :90 explicit-id init (reconstruction; caller must guarantee id matches content) +- Chunk.swift:129 `Chunk.deriveID(sourceID:startOffset:text:)` — RFC 4122 v5 (SHA-1) over fields joined by \u{1F}; :114 `namespaceBytes` PERMANENT (change re-keys fleet + breaks vector join) +- Chunk.swift:149 `struct ScoredChunk` — chunk + score/vectorScore/keywordScore (per-lane preserved) +- Chunker.swift:28 `ChunkerConfiguration` — targetChars 800 / overlapChars 100 / respectSentences true; init clamps (overlap < target) +- Chunker.swift:49 `Chunker.chunk(text:sourceID:configuration:hlcGenerator:)` — EideticLib.sentences segmentation; greedy fill + tail overlap; hlcGenerator `inout`, stamps emission order; offsets are Character counts + +### Tokenization — Tokenizer.swift +- :10 `protocol Tokenizer: Sendable` — vocabID/maxTokens/padTokenID/unknownTokenID; :27 `tokenize(_:) -> [Int32]` (truncation is implementer's job); :33 `keywordTokens(_:)` (default :62) +- :44 `defaultKeywordTokens(_:)` — lowercase + alphabetic/ASCII-digit runs; THE single keyword tokenizer for BM25 + all distributional providers; parity-critical with Rust; overriding keywordTokens breaks the guarantee (convention, not compiler) + +### Errors — CorpusKitError.swift +- :5 `enum CorpusKitError: Error, Sendable, Equatable` — encodingFailure/decodingFailure/tokenizerUnavailable/modelUnavailable/embeddingFailed/storeUnavailable/:18 notTrainable; Equatable on exact message strings + +### Chunk store — BundleStore.swift +- :94 `public actor BundleStore`; :139 `schemaDeclaration` v3 — `chunks` (10 cols incl. content_hash BLOB, ext JSON) + `corpus_metadata`; indices source_id, hlc +- :199 `init(storage:dirtyChainSink:)` — wraps HashingRowStore (MerkleHash.leaf per insert); :70 `ParentChainCache` bridges sync hash callback +- :268 `insert(_:)` — idempotent (duplicateKey = no-op, first write wins); RETURNS ONLY NEWLY-INSERTED chunks — derived-state callers must fold the returned subset, never the input +- :348 `get(id:asOf:)`; :361 `getMany(ids:asOf:)`; :375 `chunksForSource(_:asOf:)` (start_offset ASC); :399 `allSourceIDs` (full scan, maintenance only); :427 `chunkSourcePairs()` (body-free warm-load projection); :457 `count(asOf:)` — asOf accepted but IGNORED; :461 `allChunks(asOf:)` (hlc ASC) +- :500 `scrubText(sourceID:)` — hard-delete text zeroing via direct UPDATE (why schema appendOnly: false); leaves content_hash stale intentionally +- :564 `corpusMerkleRoot(for:)`; :583 `globalCorpusMerkleRoot()` — corpus/root UUIDs derived from fixed SHA-256 namespace strings (cross-port) +- :632 `decodeChunk(_:)` — MUST accept both TypedValue forms (SQLite primitive .text/.int AND InMemory semantic .uuid/.hlc); historical bug: semantic-only decoder dropped all chunks on reopen; InMemory-only tests cannot catch regressions here + +### Tombstones — RemovedSourceStore.swift +- :44 `public actor RemovedSourceStore`; :52 schemaDeclaration (own kitID "CorpusKitRemovedSources") +- :76 `markRemoved(_:now:)` (idempotent upsert; row presence IS the state — no Bool column); :90 `clearRemoved(_:)` (re-ingest = the undo); :99 `removedIDs() -> Set` — EVERY rebuild path (reindex, first-ingest train, count) must subtract this set (unenforced convention; resurrection bug class); :118 `deleteAll()` + +### Provider counts — CorpusProviderCountsStore.swift +- :71 `PersistedCounts` (modelID/modelVersion/counts blob/documentCount/vocabSize/updatedAt); :101 `CountsGrowthAnchor` (cheap pair, no blob) +- :112 `public actor CorpusProviderCountsStore`; :121 schema (kitID "CorpusKitCounts", PK (model_id, model_version), ext slot) +- :156 `upsert(_:)` — full-row replace (caller folds blob first, no atomic increment); :173 `load(...)`; :191 `growthAnchor(...)` — the staleness-check read; :210 `deleteAll()` +- STATUS: "HALF A" — counts persisted/restored, but `Corpus.reindex` still retrains from raw chunk text; counts-backed retrain + vector re-projection (HALF B) not wired + +### Sync — SyncManifest.swift +- :11 `enum CorpusKitSync`; :17 `manifest(zoneIdentifier:)` — chunks table, bidirectional, PK id, conflictPolicy `.appendOnly` (safe because content-addressed); kitID "CorpusKit", schemaVersion 1. Declarative only. Sync-layer appendOnly ≠ BundleStore schema `appendOnly: false` — different systems, same word + +### Sparse engine — Engine/SparseTypes.swift, Engine/BM25Weighting.swift, Engine/InvertedIndex.swift +- SparseTypes.swift:39 `typealias LaneTag = VectorKit.LaneTag` (single owner, avoids ambiguity); :56 `ImpactPosting` (impact Int32, quantized ONCE at build); :94 `SparseHit` (impact Float = int/100); :136 `FusedHit` (fusedScore + perLane raw scores; absent key = no hit in lane) +- BM25Weighting.swift:29 `BM25Parameters` — k1 1.5 / b 0.75 pinned defaults; :50 `quantizeImpact(_:)` — round HALF-TO-EVEN × 100 (Swift default rounding differs at .5 — do not "simplify"); :82 `buildTermIDMap` (sorted term-id assignment); :110 `build(termFreqs:docLengths:parameters:)` — float BM25 math exactly once; IDF = ln((N−df+0.5)/(df+0.5)+1); :164 `queryPairs` — OOV dropped, duplicates deduped, weight = 100 +- InvertedIndex.swift:37 `invertedIndexQuantScale = 100`; :42 `invertedIndexBlockSize = 128` (pinned for conformance traces); :116 `struct InvertedIndex: Sendable` — immutable after init (init sorts postings itemID ASC); :195 `enum Algorithm` .wand/.blockMaxWand (result-identical); :211 `topK(query:k:algorithm:)` — EXACT top-k, integer-only path, tie-break smaller itemID wins; :246 `exhaustiveScan(query:k:)` — DAAT conformance oracle, not production. Item IDs compare as STRINGS (uuidString lexicographic ≠ numeric UUID order, but consistent cross-port) + +### Persistent keyword index — Engine/InvertedIndexStore.swift +- :47 `public actor InvertedIndexStore`; :55 schemaDeclaration (kitID "InvertedIndexStore", tables iix_termfreqs/iix_doclens — RAW statistics only, weighted index derived+cached, so k1/b changes need no migration); :105 init (storage pre-opened/migrated); :114 `open()` (load mirrors, O(terms+docs), no chunk bodies) +- :159 `index(itemID:tokens:now:)` — atomic replace, idempotent; empty tokens = removal; `now` unused (determinism discipline); :199 `remove(itemID:)`; :230 `buildIndex(parameters:)` (cached; invalidated per write); :253 `topK(queryTerms:k:parameters:algorithm:)`; :273 `deleteAll()`; :295 `documentCount` +- Rust twin owns a PRIVATE connection with begin/commit/rollback_batch; Swift shares estate storage — hence Corpus-managed transaction windows in ingestBatch + +### Legacy keyword index — BM25Index.swift +- :34 `public actor BM25Index` — in-memory, Chunk/UUID-typed; NO LONGER used by Corpus (kept public for external callers); :49 init(tokenizer:parameters:); :58 `index(_ chunks:)`; :78 `remove(_:)`; :95 `documentCount()`; :107 `topK(_ k:for tokens:)` — pre-tokenized input; tie-break uuidString ASC + +### Fusion — Engine/Fusion.swift +- :48 `enum Fusion`; :74 `fuse(rankedLists:laneScores:weights:rrfK: 60)` — fusedScore = Σ weight·1/(rrfK+rank), rank 1-based; per-lane dedup (best rank only); precondition rrfK > 0; sort fusedScore DESC, itemID ASC; :164 `fuse(scoredLists:weights:rrfK:)` — position = rank, CALLER must pre-sort +- MMR: `mmrLambda` exists only as a HybridRecallConfiguration field — NOT implemented anywhere; do not document MMR as active + +### Hybrid recall — HybridRecall.swift +- :33 `HybridRecallConfiguration` — vectorWeight 0.6 / keywordWeight 0.4 / rrfK 60 / mmrLambda nil (unread); :52 `enum HybridRecall` +- :84 `recall(probe:query:modelID:limit:vectorStore:invertedIndex:bundleStore:configuration:)` — candidateK = max(limit×4, 32) per lane; vector kNN filtered to modelID; UUID hits CANONICALIZED via UUID(uuidString:).uuidString (P3-secfix: lowercase Rust-written ids must fuse with uppercase Swift ids); vectorScore = Hamming (0 = best, kept), keywordScore 0 → nil (BM25 real matches strictly positive); unhydratable ids silently dropped; telemetry post-hoc (corpuskit.recall.*) + +### Trainable-basis seam — TrainableEmbeddingBasis.swift, BasisStore.swift +- TrainableEmbeddingBasis.swift:50 `protocol TrainableEmbeddingBasis: AnyObject, Sendable` — conformers: RI/PPMI/LSA/NMF only +- :70 `trainOnCorpus(texts:)` — ADDITIVE (never retrain a live provider — reconstruct fresh first); :78 `serializeBasis()`; :98 `reconstructBasis(from:)` — INSTANCE method (type-erased witness routes to concrete init(deserializing:)); round-trip law = identical embeddings; throws decodingFailure, never crashes +- :132 `addToCounts(text:)` / :141 `serializeCounts()` / :149 `restoreCounts(from:)` / :154 `countsVocabularySize` — P3 maintained-counts seam, batch-boundary snapshots; infrastructure only (reindex still trains from text) +- No wall-clock reads anywhere in training — pure function of (texts, seeds) +- Rust divergence: EmbeddingProvider is a supertrait there (no trait cross-cast) +- BasisStore.swift:67 `PersistedBasis`; :98 `public actor BasisStore`; :112 schema v2 (`corpus_provider_basis`, PK (model_id, model_version), trained_at TEXT ISO8601, trained_chunk_count anchor, ext slot); :151 `upsert(_:)` (in-place, one row per key); :177 `load(modelID:modelVersion:)`; :195 `deleteAll()`; decoder accepts BOTH TypedValue timestamp forms (same reopen-bug class as BundleStore) + +### Basis codec — BasisCodec.swift (CorpusKitProviders) +- :43 `basisFormatVersion: UInt8 = 1`; :52 `struct BasisWriter` — LE only; f32 as bitPattern (:94); strings u32-len UTF-8 (:99); maps byte-sorted keys (:122/:136 via :148 lhsLess raw-UTF-8 compare — matches Rust str Ord, NOT Swift String <) +- :168 `struct BasisReader` — :192 `expectMagic` / :206 `expectVersion` reject wrong/future blobs with decodingFailure; all reads bounds-checked. Frame = MAGIC(4) | version(1) | payload. No nested-map primitive (PPMI inlines its own) + +### Deterministic tokenizer — DeterministicTokenizer.swift +- :16 `struct DeterministicTokenizer: Tokenizer`; :23 init(vocabID "deterministic-v1", vocabSize 30522, maxTokens 128); :33 `tokenize(_:)` — FNV-1a 32 fold into [2, vocabSize); 0=pad 1=unk reserved; empty input → [pad]. NOT a model vocab — real-model output from these ids is garbage (real WordPiece/SentencePiece = v1.1 model-bundle mission) + +### Shared training inputs — TermDocumentCounts.swift, ReducedVocab.swift +- TermDocumentCounts.swift:56 `struct TermDocumentCounts` — :61 vocab (ENCOUNTER-ORDER indices, cross-port byte contract), :64 tfCounts, :68 dfCounts; :110 `addDocument(_:)` (full); :156 `addDocumentForCountsAnchor(_:)` (vocab+docCount only, no TF/DF — re-tokenize-at-retrain decision); :91 init(restoredVocab:documentCount:) — TF rows exist but EMPTY after restore; NOT thread-safe +- ReducedVocab.swift:33 `defaultReducedVocabCap = 512` (ADR-022); :38 `ReducedVocabulary` (keptTerms/termToColumn/fullIndexToColumn/size); :63 `selectReducedVocabulary(vocab:dfCounts:documentCount:cap:)` — below cap = exact NO-OP (fixture compatibility); above cap: drop df<2, rank df DESC, tie-break raw-UTF-8-byte order (Rust parity); documentCount reserved/unused + +### Honest signals — RandomIndexingProvider.swift, PpmiProvider.swift, LsaProvider.swift, NmfProvider.swift, FdcProvider.swift +- RandomIndexing: :89 riDimension 2048, :93 riNonzeros 10, :98 riWindow 4, :102 riProjectionSeed 0x5249_5F56_315F_4D58 "RI_V1_MX"; :121 `riIndexVector(term:)` — FNV64(lowercased) → SplitMix64, EXACTLY 20 draws (pos %2048 bias-free, sign &1), collision last-wins (constant draw count = cross-port PRNG alignment); :162 `final class RandomIndexingProvider`; :206 `train(terms:window:)` additive; :249 embed / :268 embedFloat / :308 embedPair; :377 serializeBasis "RIB1" (vocab IS the basis, no finalize); :452 serializeCounts "RICT" (same payload, distinct magic on purpose) +- PPMI: :107/:111/:115 same D/K/window as RI (shared index space); :120 ppmiProjectionSeed 0x5050_4D49_5F56_314D "PPMI_V1M"; :153 `final class PpmiProvider`; :226 `train` (counts); :282 `finalize()` — ppmi(t,c)=max(0, ln P(t,c) − ln P(t) − ln P(c)); contextVec = Σ ppmi·riIndexVector(c); idempotent; :346 embed/:365 embedFloat/:397 embedPair; :464 basis "PPB1" (derived vectors only, unnormalized in store); :521 counts "PPMC" (raw additive state incl. nested coCount); NOT plain RI — do not reduce +- LSA: :109 lsaProjectionSeed 0x4C53415F56315F4D "LSA_V1_M"; :114 lsaDefaultRank 64; :165 svdSweeps PINNED 30 (change invalidates all conformance vectors); :148 `final class LsaProvider`; :227 `train(document:)`; :258 `finalize()` — ReducedVocab → tf=ln(1+c), idf=max(0, ln((N+1)/(df+1))) → JacobiSVD (wide matrix transposed+swapped back); query fold-in (1/σ)·Vt·q, σ<1e-9 skipped; :487 `documentEmbedding(at:)` exact U·Σ; :547 basis "LSB1" (reduced vocab + idf + RAW U/σ/Vt, port-neutral); :632 counts "LSAC" (anchors only) +- NMF: :110 nmfProjectionSeed 0x4E4D465F56315F4D "NMF_V1_M"; :114 rank 32; :119 iterations 100; :124 nmfFactorizationSeed 0xDEADBEEFCAFEBABE (pinned); :159 `final class NmfProvider`; :266 `finalize()` — V (terms×docs), log-TF NO idf, ALS with tolerance=0 → FIXED iteration count (bit-identity device); query fold-in dot(W[:,r],q)/(‖W[:,r]‖²+1e-9); :499 basis "NMB1" (W and H raw); :588 counts "NMFC" (anchors only) +- FDC: :106 fdcDimension 256; :111 fdcProjectionSeed 0x4644_435F_5631_5F50 "FDC_V1_P"; :137 `fdcNodeVector(code:)` — FNV64 → ONE SplitMix64 advance → LCG (Knuth 6364136223846793005 / 1442695040888963407) → 256 draws → l2Normalize (pipeline deliberately ≠ RI's); :242 `final class FDCProvider` — STATELESS, no training, no BasisCodec; path = FDC.ancestors + [code], node weight 1/(L+1); UNRESOLVED/empty → opt-out ([] / .zero) — honest, never guess; :284 embed/:299 embedFloat/:310 embedPair; determinism inherited from LatticeLib singletons +- Common float-lane tri-state (four trainable providers): `[]` = structural opt-out (untrained) → .unavailableProviderOptOut; throw VectorKitError.embedFloatVocabMiss = trained-but-all-OOV → .unavailableNoVocabHit; vector = signal. `embedPair` collapses the vocab-miss throw to (.zero, []) on ALL providers. Training thread-contract everywhere: Sendable class, but train/finalize must complete before concurrent embeds; read-only after + +### Ensemble factory — DefaultEnsemble.swift +- :38 `enum CorpusEnsemble`; :62 `defaultEnsemble()` — [.randomIndexing, .ppmi, .lsa, .nmf, .fdc] in pinned order ([0] = default signal); FUNCTION not constant (trainable providers are reference types — sharing one array would alias trained state across estates); returned trainables are UNTRAINED + +### Named neural providers — MiniLMTextProvider.swift, MPNetTextProvider.swift, EmbeddingGemmaProvider.swift +- MiniLM: :40 `struct MiniLMTextProvider: EmbeddingProvider`; :48 `inference: @Sendable ([Int32]) async throws -> [Float]` (closure-injected CoreML seam, doctrine §5); :50 init — modelID "minilm-v6", seed 0x4D49_4E4C_4D5F_7631 "MINLM_v1", 384-dim; :64 embed / :80 embedFloat / :95 embedPair (ONE inference pass — separate calls each pay inference) +- MPNet: :30 struct; :38 init — "mpnet-base-v2", seed 0x4D50_4E45_545F_7631 "MPNET_v1", 768-dim; :52/:65/:80 embed/embedFloat/embedPair +- EmbeddingGemma: :32 struct; :40 init — "embedding-gemma-300m", seed 0x454D_4247_4D5F_7631 "EMBGM_v1", 768-dim, stand-in tokenizer vocab 256_000 / maxTokens 2048 (SentencePiece-sized); :58/:71/:86 +- Parity framing: embedding VALUES are a property of the host's model bundle; what is bit-identical cross-port is the kit-owned pipeline (tokens→engram, float lane given a pooled vector) and the full no-host pipeline with DeterministicTokenizer + +### Apple NL providers — NLEmbeddingProvider.swift, NLContextualEmbeddingProvider.swift (Swift-only, ADR-019) +- NLEmbedding: :76 `nlEmbeddingProjectionSeed 0x4150_4E4C_454D_4231` "APNLEMB1" (outside doctrine table, same never-collide rule I-4); :113 `struct NLEmbeddingProvider: EmbeddingProvider, Sendable`; :145 init(language: .english default); :164/:180/:189 embed/embedFloat/embedPair; OS sentence model; no-model-for-language → [] opt-out; l2Normalize via FloatVecOps +- NLContextual: :79 `nlContextualEmbeddingProjectionSeed 0x4150_4E4C_4354_5831` "APNLCTX1"; :121 struct; :152 init; :170/:187/:196; checks `hasAvailableAssets` (free, sync) — NEVER triggers a network fetch; host must prefetch via requestAssets BEFORE constructing; mean-pools token vectors with per-token dimension guard; all failures → [] (asset absence = expected operational state) +- Both `#if canImport(NaturalLanguage)`; no Rust twins; vector dimension comes from the OS model (not pinned) + +## INVARIANTS / GOTCHAS + +- DETERMINISM DISCIPLINE: engines never read `Date()`/`UUID()`; `now` is always caller-supplied. Sanctioned exceptions: RemovedSourceStore audit stamp, telemetry timestamps. Training is a pure function of (texts, pinned seeds). +- UNIVERSAL JOIN KEY: chunk.id.uuidString == VectorKit item_id == inverted-index item_id. Swift canonical uuidString is UPPERCASE; HybridRecall re-canonicalizes vector-lane ids (P3-secfix). Universal tie-break everywhere: score DESC, then id ASC ("smaller id wins"). +- PINNED CONSTANTS (change = new conformance vectors + possible fleet re-key): chunker 800/100; BM25 k1 1.5 / b 0.75; QUANT_SCALE 100 with round-HALF-TO-EVEN; BMW block 128; RRF k 60, weights 0.6/0.4; candidate over-fetch max(limit×4, 32); float-lane sim quantization ×10_000; commit windows 512/4096; ingestMaxAttempts 8; idle drain sleep 15 ms; lease TTL 15 s / standby poll 3 s; RI/PPMI D 2048 K 10 window 4; LSA rank 64 sweeps 30; NMF rank 32 iterations 100 tolerance 0 factorization seed 0xDEADBEEFCAFEBABE; reduced-vocab cap 512; Chunk.namespaceBytes; basisFormatVersion 1. +- PROJECTION SEEDS partition vector storage by model and must be unique + frozen: deterministic 0xC05B_D15C_A15D_1B00, MINLM_v1, MPNET_v1, EMBGM_v1, RI_V1_MX, PPMI_V1M, LSA_V1_M, NMF_V1_M, FDC_V1_P, APNLEMB1, APNLCTX1. All projection goes through SubstrateML.FloatSimHash — ad-hoc projections are banned from the kit graph. +- ONLY TWO TRAIN TRIGGERS: first-ingest auto-train (gate = no persisted basis) and explicit `reindex(now:)`. trainOnCorpus is ADDITIVE — always reconstruct fresh from the slot's freshBasisBlob before retraining; growth-threshold auto-retrain is future policy (anchors persisted, path unwired — "HALF A"). +- TOMBSTONE CONSULTATION IS UNENFORCED: every chunk-replay path (reindex, first-ingest train, count, any future rebuild) MUST subtract RemovedSourceStore.removedIDs() or removed sources resurrect on the governor's auto-reindex. +- BundleStore.insert returns ONLY newly-inserted chunks; fold derived state over the returned subset. count(asOf:) ignores asOf. +- DUAL TypedValue DECODE (BundleStore, BasisStore): SQLite round-trips UUID/HLC/timestamp as primitives, InMemory keeps semantic forms; decoders must accept both; InMemory-only tests cannot catch the regression (proven bug class: silent total data loss on reopen). +- STREAM SCOPING: every queue op in CorpusIngestQueue uses stream "encode"; the shared queue.sqlite may carry other streams; an unscoped awaitDrain deadlocks. IngestJob JSON field names are a frozen cross-port wire contract. +- appendOnly means two different things: sync conflict policy (SyncManifest, `.appendOnly`, safe via content addressing) vs BundleStore schema flag (`appendOnly: false`, required so scrubText can UPDATE). Do not conflate. +- Chunks are immutable BY CONVENTION (no update API), not by DB trigger; edit = delete + reinsert with new id; metadata is the only legal per-chunk side-data slot (doctrine §2). +- keywordTokens override breaks the single-tokenizer guarantee shared by BM25 + distributional providers (convention, not compiler-enforced). +- MMR (`mmrLambda`) is declared but NOT implemented on any recall path. BM25Index is legacy — Corpus uses InvertedIndexStore. +- Actors everywhere (Corpus, BundleStore, InvertedIndexStore, BasisStore, CorpusProviderCountsStore, RemovedSourceStore, BM25Index): writes serialized per instance; embedding compute deliberately escapes the actor via Sendable providers for parallelism. +- Retry-in-place (8 attempts) is sound ONLY because ingest is idempotent via content-addressed ids — not a general recipe. +- Rust conformance gates: SharedVectors JSON (BM25 impacts, embedding vectors, basis blobs) read by BOTH legs; rust-providers tests pin basis serialization byte-for-byte for RI/PPMI/LSA/NMF and canonical vectors for RI/PPMI/FDC; NL providers exempt (Swift-only). diff --git a/packages/kits/CorpusKit/docs/DETAILS.md b/packages/kits/CorpusKit/docs/DETAILS.md new file mode 100644 index 0000000..e3b6568 --- /dev/null +++ b/packages/kits/CorpusKit/docs/DETAILS.md @@ -0,0 +1,931 @@ +--- +doc: DETAILS +package: CorpusKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/CorpusKit/BasisStore.swift + blob: 48850906faa7c2fe4aac2859a1c4e892cff32cab + - path: Sources/CorpusKit/BM25Index.swift + blob: 06fb90cd40e81f013e01a8a6c4c6f94e71bf33f3 + - path: Sources/CorpusKit/BundleStore.swift + blob: 419b1c0609597cdd68bf623ed37bd40a0171597b + - path: Sources/CorpusKit/Chunk.swift + blob: d5a1be1bb08858f5f7bd59bb141a8a0ba6f1dfbe + - path: Sources/CorpusKit/Chunker.swift + blob: a2718e06d1715f539ff633e7037c70e10ecb7a2d + - path: Sources/CorpusKit/CorpusIngestQueue.swift + blob: 2c32133701ce728bc017d4ddad51b052cae990db + - path: Sources/CorpusKit/CorpusKit.swift + blob: 4518f15fdb798c3a203c4a9db949f4d6172f540d + - path: Sources/CorpusKit/CorpusKitError.swift + blob: 68ac8d0a248bc9c2dd1885b0bc531ac4ed9cb91d + - path: Sources/CorpusKit/CorpusProviderCountsStore.swift + blob: c92160041765cc8546501c5ac4d8a2b769656e93 + - path: Sources/CorpusKit/Engine/BM25Weighting.swift + blob: 622f45870ab1118d7590cce1b379a063619b4714 + - path: Sources/CorpusKit/Engine/Fusion.swift + blob: d128ed9bc206612fc7c2e849a2e77d03d8a6cafa + - path: Sources/CorpusKit/Engine/InvertedIndex.swift + blob: 1273adcb3794b1997c93488182fc5ed95b21f9ec + - path: Sources/CorpusKit/Engine/InvertedIndexStore.swift + blob: 242baf05e5c846c719c403599a9a407bba646f5b + - path: Sources/CorpusKit/Engine/SparseTypes.swift + blob: 54654e2c49b09d31f60c06503216a2b281939f87 + - path: Sources/CorpusKit/HybridRecall.swift + blob: 21d9fb3415b699c6469a1e80d5e84da1e43981ca + - path: Sources/CorpusKit/RemovedSourceStore.swift + blob: 138a2a094ff369ee69eafa04eeba77206a58e5f4 + - path: Sources/CorpusKit/SyncManifest.swift + blob: 39af591d1fbf1f213c93eb143c213258e41c6c4b + - path: Sources/CorpusKit/Tokenizer.swift + blob: 603028510f91b3c6d75cdda1cb0a1db1c59eee28 + - path: Sources/CorpusKit/TrainableEmbeddingBasis.swift + blob: 4722def84980b3a8987adb090a1a702ac789f8ab + - path: Sources/CorpusKitProviders/BasisCodec.swift + blob: d107e1efd6341648fd8f717c7956a15b98c1b29f + - path: Sources/CorpusKitProviders/DefaultEnsemble.swift + blob: c58168f0991cc4e4c3ec490e2272bdc1a5a17be1 + - path: Sources/CorpusKitProviders/DeterministicTokenizer.swift + blob: 0586b3a4ae93dc0b58ef8f62c0d104a81dcfefe3 + - path: Sources/CorpusKitProviders/EmbeddingGemmaProvider.swift + blob: 593cd5952aad04fe0390e133e68cc07cad983d86 + - path: Sources/CorpusKitProviders/FdcProvider.swift + blob: 96f3ffaf64c7b17e2f617f49ce06cd649e70a027 + - path: Sources/CorpusKitProviders/LsaProvider.swift + blob: 3870f0d24659cb27ba13dc2cba9f94debb6b5c07 + - path: Sources/CorpusKitProviders/MiniLMTextProvider.swift + blob: 35c739a37e9ef1a92098458b48c5f5a06f11050f + - path: Sources/CorpusKitProviders/MPNetTextProvider.swift + blob: 6f13fc6dcd4733460cad366e78273f542c65a844 + - path: Sources/CorpusKitProviders/NLContextualEmbeddingProvider.swift + blob: 17d1acc363bab9c5d0bb807041e0b4c3d66fa0ee + - path: Sources/CorpusKitProviders/NLEmbeddingProvider.swift + blob: 2d714a051a1b6bd4dbde1ecd0e182a81f94b7008 + - path: Sources/CorpusKitProviders/NmfProvider.swift + blob: 43f0e339a42426d68788a03e8a216890c73eb05a + - path: Sources/CorpusKitProviders/PpmiProvider.swift + blob: 282e2185cb7e3d066979ea23b74e096ee337545b + - path: Sources/CorpusKitProviders/RandomIndexingProvider.swift + blob: 552b55b1be93fb57b9e7daf123ecf3a73df7abef + - path: Sources/CorpusKitProviders/ReducedVocab.swift + blob: fb50a8566f9ef3b2a9c650102274b20894d6542d + - path: Sources/CorpusKitProviders/TermDocumentCounts.swift + blob: e72231cf3e50799b6bbeac6a165b80410dc40317 +--- + +# CorpusKit Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear in pipeline order: +the content types, the stores, the keyword engine, fusion and recall, the +trainable-basis machinery, the `Corpus` facade and its ingest queue, and +finally the providers target. + +## Chunk.swift + +This file provides `Chunk`, the fundamental retrievable unit, and +`ScoredChunk`, the result type recall returns. + +A chunk's identity is content-addressed: its UUID is computed, not +assigned. `deriveID(sourceID:startOffset:text:)` builds an RFC 4122 +version 5 UUID from a SHA-1 hash over the source identifier, the start +offset, and the exact text. The three fields are joined with the Unicode +unit separator so that no two different inputs can collide by ambiguous +concatenation. Content addressing is what makes the rest of the kit work: +re-ingesting the same source reproduces the same identities, so a repeat +insert is a harmless no-op, and two federated devices writing the same +chunk converge on one row instead of conflicting. The fixed namespace +bytes are permanent. Changing them would re-key every chunk in every +estate and break the join to existing vector rows, because the chunk id +doubles as the VectorKit item identifier. + +Two initializers exist. The content-addressed one computes the id and is +the path normal ingestion uses. The explicit-id one reconstructs a chunk +whose id is already known, such as a decoded storage row; the caller must +ensure the id truly matches the content. The `hlc` field is a hybrid +logical clock stamp — a timestamp that also encodes causal order — and is +always caller-supplied. `ScoredChunk` pairs a chunk with its fused +`score` plus the raw `vectorScore` and `keywordScore`, kept separate so +callers can diagnose which lane produced a hit. + +## Chunker.swift + +This file provides `Chunker`, which splits raw source text into ordered +chunks with sentence-aware boundaries. + +Chunks should not cut a sentence in half, because a half sentence embeds +poorly and reads worse. The chunker therefore segments the text into +sentences first, delegating to `EideticLib.sentences(_:)` so segmentation +logic lives in one shared place. It then fills a buffer greedily: sentences +accumulate until adding the next one would pass the target size, the buffer +flushes into a chunk, and the tail of the flushed chunk carries over as +overlap into the next buffer. Overlap means a match near a boundary still +brings its surrounding context along. Offsets into the original text are +tracked exactly, because the offset is part of each chunk's +content-addressed identity. + +`ChunkerConfiguration` holds `targetChars` (default 800), `overlapChars` +(default 100), and `respectSentences` (default true). Its initializer +clamps nonsense values — overlap can never reach the target, which would +loop forever. `Chunker.chunk(text:sourceID:configuration:hlcGenerator:)` +is the single entry point. The HLC generator is passed `inout` because the +chunker is the sole authority for chunk order within one call: it stamps +each chunk in emission order. Changing the default sizes changes chunk +boundaries and therefore chunk identities for re-ingested content, so the +defaults are pinned to the substrate reference. + +## Tokenizer.swift + +This file provides the `Tokenizer` protocol and the single canonical +keyword tokenizer. + +A tokenizer serves two different masters. An embedding model needs its own +vocabulary of integer token ids, so the protocol requires +`tokenize(_:) -> [Int32]` plus identity fields (`vocabID`, `maxTokens`, +pad and unknown ids). Keyword search needs plain words, so the protocol +also requires `keywordTokens(_:) -> [String]`. The protocol extension +supplies a default for the second, delegating to the free function +`defaultKeywordTokens(_:)`: lowercase the text, then keep runs of +alphabetic or ASCII-digit characters and split on everything else. + +There is exactly one definition of keyword tokenization in the module for +a reason. BM25 and every distributional embedding signal must agree on +what a "term" is, or the keyword lane and the semantic lane would score +different vocabularies and hybrid recall would quietly degrade. The +function is also parity-critical: the Rust port implements the same rules, +and committed conformance vectors on both legs break if it changes. A +provider that overrides `keywordTokens` breaks this guarantee by +convention, not by compiler error. + +## CorpusKitError.swift + +This file provides `CorpusKitError`, the module's single error enum. + +Seven flat cases cover the failure classes: `encodingFailure`, +`decodingFailure`, `tokenizerUnavailable`, `modelUnavailable`, +`embeddingFailed`, `storeUnavailable`, and `notTrainable`. Each carries a +plain message string, because callers mostly log or surface the message +rather than branch on structured data. `notTrainable` exists for +`EmbeddingModel.reconstruct(from:)`: providers without a trained basis +(the deterministic provider, the named neural models, the stateless FDC +provider) surface this error instead of silently substituting a wrong +provider. The enum is `Equatable` on its message strings, so tests that +compare errors must construct the exact message. + +## BundleStore.swift + +This file provides `BundleStore`, the persistence layer for chunks — the +content half of every content-plus-vector bundle. The vector half lives in +VectorKit, joined by the chunk's UUID string. + +`BundleStore` is an actor wrapping `any Storage` from PersistenceKit, so +the application picks the backend (SQLite or in-memory) and the kit does +not. It owns schema version 3: a `chunks` table of ten columns plus a +`corpus_metadata` table, with indices on `source_id` and `hlc`. Inserts do +not go straight to the row store. They pass through a hashing decorator +that computes a content hash with `MerkleHash.leaf` on every write and +emits dirty-chain events. A Merkle hash chain lets the estate prove its +content has not drifted: each chunk hashes, each source's chunks combine +into a per-corpus root, and all corpus roots combine into one global root. +Because the hashing callback is synchronous, a small lock-guarded +`ParentChainCache` pre-stages each chunk's parent identifiers before the +insert. The corpus and root identifiers derive from fixed namespace +strings via SHA-256, so both language legs compute identical chains. + +`insert(_:)` is idempotent by design. It attempts a plain insert per chunk +and treats a duplicate-key error as the documented no-op path — first +write wins, which is sound only because identities are content-addressed. +It returns only the chunks that were actually new, in input order, so +callers maintaining derived state never double-count a re-ingested chunk. +Read paths (`get`, `getMany`, `chunksForSource`, `allChunks`, +`allSourceIDs`, `count`, `chunkSourcePairs`) are thin query wrappers; +`chunkSourcePairs()` deliberately omits chunk bodies so opening a corpus +stays cheap. `scrubText(sourceID:)` is the hard-delete seam: it zeroes the +`text` column through a direct update, which is why the schema declares +the table `appendOnly: false` even though the API treats chunks as +immutable. Immutability here is a convention enforced by the surface, not +a database trigger; the sync layer's separate `appendOnly` conflict policy +should not be confused with this flag. + +One decoding rule deserves emphasis. The SQLite backend round-trips UUIDs +as text and HLCs as packed integers, while the in-memory backend preserves +the semantic typed values. `decodeChunk` and its helpers accept both +forms. A past semantic-only decoder silently dropped every persisted chunk +on reopen, and in-memory tests never caught it. Any new decode path must +handle both forms. + +## RemovedSourceStore.swift + +This file provides `RemovedSourceStore`, the tombstone table that makes +source removal stick. + +Chunk rows are never deleted, so "removing" a source can only delete its +vectors and keyword postings. Without a durable marker, any rebuild that +replays `allChunks()` — an explicit reindex, or the autonomic governor's +scheduled one — would re-embed the removed source and silently resurrect +it. The store is therefore the single source of truth every rebuild path +must consult. The presence of a row is the entire state: there is no +boolean column, per the fleet-wide schema rule. Reactivation is symmetric: +re-ingesting a source clears its tombstone, so ingestion itself is the +undo. + +`markRemoved(_:now:)` upserts a tombstone with a caller-supplied +timestamp. `clearRemoved(_:)` deletes it. `removedIDs()` returns the full +set that rebuilds must subtract. `deleteAll()` supports index destruction. +Nothing enforces that a new rebuild path remembers to consult this store; +that is a blast-radius obligation on every future change. + +## CorpusProviderCountsStore.swift + +This file provides `CorpusProviderCountsStore`, persistence for the raw +statistics a trainable embedding signal accumulates between retrains. + +The design splits cheap from heavy. The `counts` column is an opaque blob +the provider alone serializes; the store never decodes it and never +imports the providers target. Two small integer columns — document count +and vocabulary size — are lifted out of the blob so a staleness check can +ask "has the corpus grown enough to retrain?" with one tiny query instead +of deserializing a large blob. Rows are keyed by `(model_id, +model_version)`, matching how basis rows and vector rows are keyed, +because counts are only valid for the exact provider version that +accumulated them. `upsert(_:)` replaces the row whole; additive merging is +the provider's job before it calls in. + +The file is candid that this is half a feature. Counts are persisted and +restored, but `Corpus.reindex` still retrains from raw chunk text rather +than from this table; the counts-backed retrain path and vector +re-projection are future work. Documentation should not present this store +as the current retrain mechanism. + +## SyncManifest.swift + +This file provides `CorpusKitSync.manifest(zoneIdentifier:)`, the +declarative sync contract for the `chunks` table. + +The manifest declares one bidirectional synced table with primary key `id` +and conflict policy `appendOnly`. That policy is safe precisely because +chunks are content-addressed and immutable: two devices can never produce +conflicting edits to the same id, only identical re-derivations, so the +sync layer needs no merge strategy. The kit performs no sync itself; the +application hands this manifest and a storage instance to ConvergenceKit. +When VectorKit sync is also enabled, both tables should share one zone so +chunks and their vectors stay join-compatible on every device. + +## Engine/SparseTypes.swift + +This file provides the value types of the sparse retrieval lane: +`ImpactPosting`, `SparseHit`, `FusedHit`, and the `LaneTag` alias. + +The load-bearing decision is that `ImpactPosting.impact` is an integer. A +float BM25 weight is quantized once at index build; from then on the whole +query path is integer arithmetic, which is what makes the Swift and Rust +legs bit-identical. `SparseHit` is the consumer surface: the integer score +divided back by the quantization scale. `FusedHit` carries the fused score +plus a `perLane` map of raw per-lane scores, preserved so later selection +stages can read lane signals without recomputation. `LaneTag` is a type +alias to VectorKit's enum rather than a second enum, because two identical +Swift enums are still distinct types and would make case names ambiguous +for consumers importing both kits. Posting lists are always sorted by item +id ascending — the WAND algorithm's pivoting invariant — and fused results +sort by score descending, then item id ascending. + +## Engine/BM25Weighting.swift + +This file provides BM25 as an impact-weighting scheme that feeds the +inverted index, plus the quantizer and query helpers. + +BM25 scores a document for a term by combining the term's rarity (inverse +document frequency, IDF) with its frequency in the document, damped by +document length. The whole float computation happens exactly once, at +index build. `build(termFreqs:docLengths:parameters:)` evaluates the +classic formula per term and document, then quantizes each contribution +with `quantizeImpact(_:)` — multiply by 100 and round half to even. The +rounding mode is pinned deliberately: Swift's default rounding differs +from banker's rounding at exact halves, and both legs must agree. Term +strings map to dense integer ids in sorted order, so runs are +reproducible. `BM25Parameters` pins the defaults `k1 = 1.5` and +`b = 0.75`, tunable per estate. `queryPairs(queryTerms:termMapping:)` +turns query terms into (term id, weight 100) pairs, dropping unknown terms +and deduplicating repeats so each term contributes once. + +## Engine/InvertedIndex.swift + +This file provides the generic weighted inverted index with two exact +top-k algorithms: WAND and Block-Max WAND. + +An inverted index maps each term to a posting list — the items containing +that term, each with a pre-quantized impact. Scoring an item for a query +is an integer dot product over shared terms. The naive approach scores +every candidate; WAND ("Weak AND") skips most of them. It keeps one cursor +per query term, sorted by current position, and computes a pivot: the +first point where the accumulated best-case impacts could beat the current +k-th best score. Items before the pivot cannot win and are skipped +wholesale. Block-Max WAND refines this with per-block maxima (block size +128): if even the tighter block bound cannot beat the threshold, the whole +block is skipped. Both algorithms are exact — they return precisely the +same top-k as a full scan — and `exhaustiveScan(query:k:)` ships as the +reference oracle for conformance tests. + +The index is immutable after construction; mutation means rebuilding, +and serializing rebuilds is the wrapper's job. The internal bounded heap +implements the universal tie-break: equal scores resolve toward the +smaller item id, so results never depend on hash or insertion order. Item +ids compare as strings. The pinned constants `invertedIndexQuantScale` +(100) and `invertedIndexBlockSize` (128) are part of the cross-port +contract. + +## Engine/InvertedIndexStore.swift + +This file provides `InvertedIndexStore`, the persistent wrapper that lets +keyword state survive restarts without replaying chunk bodies. + +The store persists only raw statistics: a term-frequency table and a +document-length table in two small SQLite tables. The weighted index +itself is derived. On demand, `buildIndex(parameters:)` runs the BM25 +build over the in-memory mirrors and caches the result; every write +invalidates the cache. Persisting statistics instead of weighted postings +means changing `k1` or `b` never requires a data migration — only an +in-memory rebuild. `open()` loads all rows once, a cost proportional to +terms plus documents, never to chunk text. + +`index(itemID:tokens:now:)` replaces a document's terms atomically and is +idempotent; empty tokens remove the item. `remove(itemID:)` and +`deleteAll()` complete the mutation surface, and +`topK(queryTerms:k:parameters:algorithm:)` is the one-call query path. The +actor serializes all mutation. The Rust twin owns a private database +connection with explicit batch methods; the Swift store instead shares the +estate's storage, which is why the facade manages transaction windows +around it during bulk ingest. + +## BM25Index.swift + +This file provides `BM25Index`, the original in-memory keyword index, +preserved as a public primitive. + +The `Corpus` facade no longer uses it — durability required +`InvertedIndexStore` — but external callers that built on it keep a +working, chunk-typed surface. It holds term frequencies keyed by chunk +UUID string, tokenizes chunk text itself through an injected `Tokenizer`, +and delegates scoring to the same engine layer (BM25 weighting plus +Block-Max WAND), caching the built index between writes. +`index(_:)`, `remove(_:)`, `documentCount()`, and `topK(_:for:)` form the +surface; `topK` takes pre-tokenized terms, and the caller must tokenize +with the same vocabulary used at index time. Ties break by UUID string +order, which is not numeric UUID order but is identical on both legs. + +## Engine/Fusion.swift + +This file provides `Fusion`, the generalized weighted Reciprocal Rank +Fusion engine. + +Reciprocal Rank Fusion (RRF) merges ranked lists without comparing their +raw scores, which live on incompatible scales. Each lane contributes +`weight × 1 / (rrfK + rank)` for every item it ranked; the sums decide the +final order. The constant `rrfK` (default 60, from the original RRF paper) +damps the advantage of rank one over rank two. The function deduplicates +within each lane — only an item's best rank counts, because a duplicate +would illegally double its contribution — and demands `rrfK > 0`, since +zero or negative values corrupt the formula. Output sorts by fused score +descending, then item id ascending. + +Two overloads exist. `fuse(rankedLists:laneScores:weights:rrfK:)` takes +explicit ranks and optional raw scores to carry through into `perLane`. +`fuse(scoredLists:weights:rrfK:)` treats array position as rank; the +caller must pre-sort. The engine is a pure function over ranks and +weights — deterministic and reentrant. One caution: the configuration type +in `HybridRecall.swift` reserves an MMR field, but no diversification is +implemented here or anywhere on this path yet. + +## HybridRecall.swift + +This file provides `HybridRecall.recall(...)`, the canonical two-lane +retrieval pipeline. + +The pipeline over-fetches a candidate window of `max(limit × 4, 32)` from +each lane, because fusion needs headroom: an item ranked eleventh in both +lanes can out-fuse an item ranked first in only one. The vector lane runs +VectorKit's nearest-neighbor search concurrently while the query is +tokenized and the keyword lane queries the inverted index. Both hit lists +become ranked lists keyed by canonical UUID strings. Canonicalization is a +deliberate security-review fix: a lowercase UUID written by the Rust leg +and an uppercase Swift keyword hit for the same item would otherwise never +fuse. `Fusion.fuse` merges the lanes with the configured weights, the +result truncates to the limit, and the winners hydrate from the bundle +store in fused order. + +`HybridRecallConfiguration` pins the defaults: vector weight 0.6, keyword +weight 0.4, `rrfK` 60, and an `mmrLambda` slot that is currently declared +but never read. Score mapping is asymmetric on purpose: a vector score of +zero is a perfect Hamming match and is kept, while a keyword score of zero +means "did not match" and maps to nil. Telemetry (latency and per-lane +counts) fires at the operation boundary where it cannot affect results; +with monitoring off it costs one atomic load per metric. + +## TrainableEmbeddingBasis.swift + +This file provides the `TrainableEmbeddingBasis` protocol, the seam that +lets the core drive provider training without importing the providers +target. + +Layering runs one way: providers depend on core, never the reverse. The +`Corpus` holds providers as type-erased values, so it needs a protocol to +ask "can you train, serialize, and reconstruct yourself?" Providers that +cannot — the deterministic provider, the named neural models, FDC, the +Apple NL providers — simply do not conform, and the facade surfaces +`CorpusKitError.notTrainable`. `reconstructBasis(from:)` is an instance +method rather than an initializer for exactly this reason: invoked on a +type-erased witness, it routes to the correct concrete type's +deserializing initializer. + +The protocol has two halves. The basis half — `trainOnCorpus(texts:)`, +`serializeBasis()`, `reconstructBasis(from:)` — covers full training and +the round-trip law: a reconstructed provider embeds byte-identically to +the trained original. The counts half — `addToCounts(text:)`, +`serializeCounts()`, `restoreCounts(from:)`, `countsVocabularySize` — +maintains raw additive statistics incrementally, snapshotted at batch +boundaries because per-chunk serialization would be quadratic over an +import. Training must never read the wall clock; it is a pure function of +the texts and fixed seeds. The Rust port cannot cross-cast trait objects, +so there the embedding trait is a supertrait instead — a documented, +sanctioned divergence. + +## BasisStore.swift + +This file provides `BasisStore`, persistence for trained basis blobs, so a +reopened corpus embeds immediately instead of retraining. + +One row per `(model_id, model_version)` in the `corpus_provider_basis` +table holds the opaque little-endian blob, a trained-at timestamp +(caller-supplied, stored as ISO 8601 text per the schema rules), and a +trained-chunk-count anchor reserved for a future auto-retrain policy. +The composite key matters: a blob trained for one provider must never load +into another, and the key matches how every vector row is keyed. Retrain +upserts in place, so exactly one row exists per provider — no history, no +orphans. Schema version 2 adds a nullable JSON `ext` column as a +forward-compatibility slot that version 1.0 writes as null and never +reads. + +Like `BundleStore`, the decoder tolerates both typed-value forms — the +in-memory backend's semantic timestamps and SQLite's ISO text — because a +semantic-only reader would silently drop every row on reopen and semantic +recall would go dark on any restored estate. `upsert(_:)`, +`load(modelID:modelVersion:)`, and `deleteAll()` form the whole surface. + +## CorpusKit.swift + +This file provides the public entry point: the `Corpus` actor, the +`EmbeddingModel` selection enum, the `FloatLaneOutcome` result type, and +the `EncodeSpeed` quality-of-service knob. It is the largest file in the +package because it is the composition root: everything else exists so this +file can wire it together. + +### The Corpus Actor and Its Provider Slots + +A `Corpus` composes the bundle store, the persistent keyword index, the +vector store, the basis and counts stores, the tombstone store, and one +slot per configured embedding signal. It seals VectorKit behind its own +surface: no VectorKit type appears in a public signature except the +deliberate `sharedVectorStore` escape hatch, which lends the estate's one +vector store to the orchestrator so no second store is built over the same +table. + +Each provider slot holds three things. First, the serving provider, which +embeds queries and chunks. Second, for trainable signals, a +`freshBasisBlob` — the serialized untrained basis captured at +construction. This is the from-scratch factory: training is additive, so +retraining a live provider would count the corpus twice; every retrain +instead reconstructs a fresh provider from this blob, which makes reindex +idempotent and canonical across ports. Third, a separate counts +accumulator, deliberately not the serving provider, because growing a +vocabulary in place would desync a factorized basis from its frozen +factors. Slot zero is the default signal: every single-signal entry point +delegates to it, so a one-model corpus behaves exactly like the old +single-provider design. + +### Opening, Ingesting, Reindexing + +`init(storage:models:)` migrates six schemas, resolves each slot (a +persisted basis reconstructs a trained provider; a corrupt blob throws +rather than serving untrained), opens the keyword index, and warm-loads a +chunk-to-source map from a body-free projection — the whole cold start +avoids reading chunk text. `ingest(_:sourceID:now:)` chunks the text, +clears any tombstone (re-ingest reactivates), inserts idempotently, +indexes keywords, folds counts, and embeds. Embedding is two-phase: any +trainable slot with no persisted basis triggers the one-and-only implicit +first-ingest training over the full corpus snapshot; all other slots fold +in under their frozen basis. Fold-in embeddings compute concurrently off +the actor — providers are `Sendable` values — and land in one batched +vector write. `ingestBatch(_:)` produces output identical to per-item +ingest but commits in windows of 512 items or 4,096 rows, long enough to +amortize disk syncs and short enough not to starve concurrent captures, +and fans embedding out in contiguous slices per core. + +`reindex(now:)` is the explicit retrain trigger: reconstruct fresh, train +on all active chunks (tombstoned sources excluded), install, persist the +basis, and re-embed every active chunk under every slot. Only two train +triggers exist in the whole kit — first ingest and explicit reindex. + +### Recall, Removal, Observation + +`recall(_:limit:now:)` embeds the query on the default signal and +delegates to `HybridRecall`. `bm25TopKBySource(query:limit:)` is the pure +keyword lane aggregated to source granularity. The dense float lane — +`floatNearest`, `floatNearestPerSignal`, `floatFarthestPerSignal` — ranks +by true cosine similarity and never throws; unavailable states are typed +`FloatLaneOutcome` values (`.unavailableProviderOptOut`, +`.unavailableNoVocabHit`, `.unavailableNoFloatRows`, `.emptyQuery`, +`.storeError`), because a dark lane is an expected condition, not an +error. The farthest variant answers "what is unlike this?" and aggregates +by each source's closest chunk, so a source only counts as unlike when +even its best chunk is far. + +`remove(sourceID:)` suppresses recall: it deletes keyword rows and every +model's vectors and writes the tombstone. `expunge(sourceID:)` scrubs the +verbatim text first, then removes, so content is destroyed even if a later +step fails. `destroyRecallIndex()` wipes every derived structure while +chunk rows survive. `count()`, `indexedSourceIDs()`, +`maintainedVocabAnchor()`, and the two Merkle root accessors round out the +observational surface. + +### EmbeddingModel and the Small Types + +`EmbeddingModel` names every signal the corpus can hold: `.deterministic` +(the permanent federation-grade baseline — a hash-based, lexical, fully +reproducible signal with a pinned seed), the three named neural models +(`.miniLM`, `.mpNet`, `.embeddingGemma`) that take a host-supplied +inference closure, the four trainable statistical signals +(`.randomIndexing`, `.ppmi`, `.lsa`, `.nmf`) that carry pre-built +providers, stateless `.fdc`, and the Apple-only `.nlEmbedding` and +`.nlContextualEmbedding`. `isTrainable` reports whether the carried +provider conforms to the training seam, and `reconstruct(from:)` routes a +persisted blob to the right concrete type. `EncodeSpeed` selects the +embed-concurrency cap: `.foreground` uses all cores, `.background` roughly +a quarter. A private `CorpusDefaultTokenizer` duplicates the providers' +deterministic tokenizer to avoid a circular dependency, and a private +`CorpusTextProvider` implements the tokenize-infer-project pipeline for +the named models, computing the pooled vector once per chunk for both the +engram and the float row. + +## CorpusIngestQueue.swift + +This file provides the asynchronous ingest pipeline as an extension on +`Corpus`: a durable queue, a background drain worker, and a single-drainer +lease. It exists so CorpusKit is a complete standalone substrate — any +consumer gets queued, multi-core encoding with no orchestrator. + +`mountIngestQueue()` picks the backend by estate durability. A SQLite +estate gets a sibling `queue.sqlite` file derived deterministically from +the estate configuration — encrypted with the same key as the estate, +replacing an earlier plaintext directory queue that was a real security +hole beside an encrypted estate. An in-memory estate gets a transient +store under a fixed constant UUID, avoiding random-identity +nondeterminism. Because the physical queue can carry other streams, every +operation here is scoped to the `"encode"` stream; an unscoped wait would +deadlock on jobs this drainer never claims. + +The drain loop coordinates through a `DrainLease`: one live drainer per +estate, with crash recovery on first acquisition that resets orphaned +in-flight jobs — safe precisely because the lease guarantees no other live +drainer holds them. A losing process becomes a warm standby that re-checks +every three seconds, bounded by the lease's fifteen-second staleness +window. Each drain pass claims the whole available batch, decodes jobs +(undecodable ones are terminally blocked, empty ones completed), runs +`ingestBatch` once for the batch, and retires the batch in one bulk reply. +While passes keep draining jobs, the loop spins without sleeping and +defers the vector-index publish until the burst ends — one index rebuild +per burst instead of one per pass, turning a quadratic bulk import linear. +Idle, it sleeps fifteen milliseconds, the near-realtime latency floor. A +failing item retries in place up to eight attempts before a terminal +blocked reply; in-place retry is sound only because ingest is idempotent. + +`enqueueIngest(_:sourceID:now:)` and `enqueueIngestBatch(_:)` stamp jobs +with caller-supplied instants — never the wall clock — and the batch +variant wraps all inserts in one transaction, which removed the last +full-core bottleneck of bulk imports on encrypted SQLite. +`awaitIngestDrain(timeout:)` is the barrier importers use to know writes +are searchable. `setOnEncoded(_:)` installs the one callback CorpusKit +ever makes toward an orchestrator. The `IngestJob` wire format's JSON +field names are a pinned cross-port contract with the Rust twin. + +## BasisCodec.swift + +This file provides the shared binary codec every trainable provider uses +to serialize bases and counts. + +The byte layout is the cross-port contract: the same trained state must +serialize to the same bytes on Swift and Rust, which rules out JSON (float +formatting, key order, and whitespace differ across ecosystems). The rules +are fixed: everything little-endian; floats written as raw IEEE-754 bit +patterns so negative zero and NaN round-trip exactly; strings +length-prefixed UTF-8; maps written with keys in ascending raw-byte order. +That byte-order sort exists because Swift's default string comparison is +Unicode-canonical while Rust's is byte order — the writer compares raw +UTF-8 to match. Every blob is framed with a four-byte magic tag and a +format version byte, currently 1. + +`BasisWriter` is an append-only cursor with typed write methods; +`BasisReader` is a bounds-checked sequential reader whose +`expectMagic(_:)` and `expectVersion(_:)` reject wrong-provider or +future-format blobs with `CorpusKitError.decodingFailure` — never a crash, +never a silent misparse. Both are value types with no shared state. + +## DeterministicTokenizer.swift + +This file provides `DeterministicTokenizer`, the model-agnostic stand-in +tokenizer that ships as the version 1.0 default for the named neural +providers. + +It is a hash, not a vocabulary. Words split by the canonical keyword +rules, then each word folds through FNV-1a into an id in the range two +through the vocabulary size; ids zero and one are reserved sentinels for +padding and unknown. Empty input returns a single pad token, never an +empty array. Because both legs fold through the same hash, conformance +harnesses get identical ids for identical input. The defaults (vocabulary +30,522, maximum 128 tokens) match the BERT family. The critical caveat: +feeding these ids into a real embedding model produces garbage, because +they have no relation to the model's true vocabulary. Real WordPiece and +SentencePiece tokenizers arrive with the version 1.1 model-bundle mission. + +## TermDocumentCounts.swift + +This file provides `TermDocumentCounts`, the shared count builder feeding +the LSA and NMF providers. + +It owns three things: the vocabulary, built in encounter order (a term's +column index is fixed by the first document that mentions it, which keeps +matrix columns stable for a fixed document sequence); per-document term +frequencies; and per-term document frequencies. It deliberately does not +own weighting or factorization — those belong to the consuming providers, +which weight the same counts differently. `addDocument(_:)` is the full +training path. `addDocumentForCountsAnchor(_:)` is the lightweight +incremental path: it grows the vocabulary and the document count but keeps +no frequencies, because the heavy inputs are re-derived by re-tokenizing +the corpus at retrain time, bounding maintained state to the vocabulary's +size. The restored-vocabulary initializer likewise seeds a deserialized +provider with truthful metadata and empty frequency rows. The builder is +not thread-safe; all writes must finish before reads. + +## ReducedVocab.swift + +This file provides the shared vocabulary-reduction step for the dense +factorizations, LSA and NMF. + +A dense matrix over tens of thousands of terms is unfactorizable on a +device — the comment estimates ten-to-the-fifteenth operations. The fix is +to keep only the most informative columns. Below the cap (512 by default) +the function is a strict no-op, so small estates and every conformance +fixture behave exactly as before reduction existed. Above the cap it drops +terms seen in only one document (pure noise), ranks the rest by document +frequency descending — terms that co-occur across many documents carry the +latent structure a factorization can find — and breaks ties by raw UTF-8 +byte order, matching Rust's string ordering so both legs select identical +vocabularies. The selection is shared rather than per-provider because +informativeness is a corpus property, identical for both factorizations. +`ReducedVocabulary` freezes the kept terms, the projection map, and the +row-remapping table. + +## RandomIndexingProvider.swift + +This file provides `RandomIndexingProvider`, the first honest +distributional signal in the dense lane. + +Random Indexing gives every term a deterministic sparse "index vector": +2,048 dimensions with exactly ten nonzero entries of plus or minus one. +The generator seeds a counter-based random stream from the FNV hash of +the lowercased term and draws exactly twenty values — ten positions, ten +signs — with collisions resolved last-wins rather than by rejection, so +the draw count is constant and the Swift and Rust streams stay aligned. A +term's meaning is then learned by addition: sliding a window of four over +training text, each term accumulates the index vectors of its neighbors +into a context vector. Terms that keep similar company converge, which is +genuine co-occurrence semantics at almost no computational cost, and the +accumulation is incremental by construction. + +Embedding text sums the context vectors of its in-vocabulary terms and +normalizes to unit length. The float lane is honest about failure: an +untrained provider opts out with an empty vector, and a trained provider +whose query is entirely out of vocabulary throws a typed vocabulary-miss +error that the facade maps to the right dark-lane outcome. The basis blob +(magic `RIB1`) is the whole vocabulary map — Random Indexing has no +separate finalize step — and the counts blob (`RICT`) carries the same +payload under a distinct magic so a counts row can never be misread as a +basis row. The projection seed spells `RI_V1_MX`. + +## PpmiProvider.swift + +This file provides `PpmiProvider`, the co-occurrence signal weighted by +positive pointwise mutual information. + +PPMI asks of each word pair: do these words co-occur more than chance +would predict? The score is the logarithm of the observed co-occurrence +probability over the product of the individual probabilities, floored at +zero. Frequent-but-meaningless neighbors (the "the" problem) score near +zero; genuinely associated pairs keep full weight. Training is two-phase. +Phase one counts: a sliding window of four accumulates pair counts, term +counts, and totals, additively across calls. Phase two, `finalize()`, +converts counts to context vectors: each term's vector is the +PPMI-weighted sum of its neighbors' Random Indexing index vectors, in the +same 2,048-dimension space with the same generator, so the two signals are +directly comparable. The file warns explicitly that this is mathematically +distinct from unweighted Random Indexing and must not be "simplified" +into it. + +Embedding, opt-out, and vocabulary-miss behavior mirror the Random +Indexing provider. The basis blob (`PPB1`) persists only the derived +vectors; the count tables are training scratch. The counts blob (`PPMC`) +persists the raw additive state — including the nested pair-count map, +serialized with byte-sorted keys — so a retrain can resume counting +without re-tokenizing. Stored vectors are kept unnormalized; +normalization happens at embed time so tests can inspect raw sums. The +projection seed spells `PPMI_V1M` and must never equal the Random +Indexing seed, because the seed partitions vector storage by model. + +## LsaProvider.swift + +This file provides `LsaProvider`, the latent semantic analysis signal. + +LSA finds hidden topic structure by factorizing a term-document matrix +with the singular value decomposition (SVD). Words that appear in similar +documents land near each other in the latent space, so synonyms co-locate +even when they never co-occur. Training builds on the shared count +builder: reduce the vocabulary (the ADR-022 step), weight each cell by +log term frequency times smoothed IDF (add-one smoothing on both sides, +so an unseen query term still gets positive weight), and decompose with +`JacobiSVD` from SubstrateML at rank 64. The Jacobi method is chosen +because its sweep count is pinned — thirty sweeps, exactly — making the +factorization a fixed computation rather than a convergence-dependent +one; changing the sweep count invalidates every conformance vector. Wide +matrices are transposed for the decomposition and the factors swapped +back, so downstream code sees one orientation. + +New text embeds by the classical fold-in formula: project the query's +weighted term vector through the factor matrix, scaling by the inverse +singular values and skipping values below a small floor. Training +documents use the exact projection instead. The dark-lane contract +matches the other trainable providers. The basis blob (`LSB1`) persists +the reduced vocabulary, the IDF weights, and the raw factor matrices — +port-neutral, so each leg re-derives its own document vectors — and the +counts blob (`LSAC`) persists only the vocabulary and document-count +anchors, per the re-tokenize-at-retrain decision. The projection seed +spells `LSA_V1_M`. + +## NmfProvider.swift + +This file provides `NmfProvider`, the non-negative matrix factorization +signal. + +NMF factorizes the term-document matrix into two non-negative factors, so +every latent dimension reads as an additive combination of terms — parts, +not contrasts. The matrix is oriented terms-by-documents, weighted by log +term frequency only: NMF requires non-negative input, and its update +steps are most stable with uniformly scaled entries, so IDF is deliberately +omitted. Factorization reuses SubstrateML's alternating least squares at +rank 32 with two pinned determinism devices: the convergence tolerance is +set to zero, which forces exactly one hundred iterations on every platform +regardless of floating-point convergence behavior, and factor +initialization is seeded with a fixed constant. Document embeddings are +the normalized factor columns, precomputed at finalize. + +Queries fold in through a pseudo-inverse projection onto each factor +column, with a small epsilon guarding the denominator, then normalize. +Serialization follows the family pattern: the basis blob (`NMB1`) carries +configuration, the reduced vocabulary, and both raw factor matrices; the +counts blob (`NMFC`) carries anchors only. The projection seed spells +`NMF_V1_M`. The dark-lane contract matches the other trainable providers. + +## FdcProvider.swift + +This file provides `FDCProvider`, the taxonomic co-classification signal — +the one honest signal that needs no training at all. + +The provider classifies text with LatticeLib's FDC engine, which lives in +the moot-semantics repository and is not reimplemented here. `FDC.encode` +returns a lattice code or nothing; `FDC.ancestors` returns the code's +chain up the classification hierarchy. The provider turns that chain into +geometry: each node in the chain gets a deterministic 256-dimension unit +vector generated from the hash of its code string, and the vectors sum +with weight one over depth-plus-one — shared roots give any two texts a +similarity floor, and shared deep ancestors add to it. Texts filed near +each other in the subject hierarchy therefore embed near each other, +regardless of surface wording. The node-vector generator is deliberately +a different pipeline from Random Indexing's, and the input domains cannot +collide. + +Honesty governs the edges. Text the classifier cannot resolve returns +UNRESOLVED, and the provider opts out with an empty float vector and a +zero engram — unclassifiable text must not contribute false similarity. +The provider is stateless and `Sendable`; its determinism rests on +LatticeLib's own agreement property. The projection seed spells +`FDC_V1_P`, and the free function `fdcNodeVector(code:)` is public so +conformance tests can pin individual node vectors. + +## DefaultEnsemble.swift + +This file provides `CorpusEnsemble.defaultEnsemble()`, the single source +of truth for the production recall ensemble. + +The factory returns the five honest signals in fixed order: Random +Indexing, PPMI, LSA, NMF, FDC. Order is load-bearing — the first element +becomes the corpus's default signal. It is a function rather than a shared +constant because the four trainable providers are reference types holding +mutable trained state; a shared array would alias one provider instance +across every estate. Fresh construction gives each estate its own +untrained providers, which the corpus lifecycle then trains and persists +under their own model identifiers. The factory lives in the providers +target because it names concrete types; the core's `EmbeddingModel` enum +never does. + +## MiniLMTextProvider.swift, MPNetTextProvider.swift, EmbeddingGemmaProvider.swift + +These three files provide the named neural embedding providers: MiniLM-L6 +v2 (384 dimensions), mpnet-base-v2 (768), and EmbeddingGemma 300M (768). +They share one structure, so they are described together; each fact below +holds for all three unless noted. + +Each provider runs the same pipeline: tokenize, run the host-injected +inference closure to get a pooled float vector, and project that vector to +a 256-bit engram with `FloatSimHash` from SubstrateML. An engram is a +fixed-size binary fingerprint; similar vectors produce similar engrams, so +the vector lane can compare chunks by fast Hamming distance. The inference +closure is the doctrine's model seam: the provider never holds a CoreML +model, which keeps it testable without a model bundle and leaves model +loading — different on iOS, macOS, and CI — to the host, composed once at +startup. Each provider pins its own projection seed so its fingerprints +are model-tagged: `MINLM_v1`, `MPNET_v1`, and `EMBGM_v1` in ASCII. Seeds +partition vector storage by model and must never change or collide. + +Each exposes three surfaces. `embed` returns the engram; `embedFloat` +returns the raw pooled vector for the true-cosine float lane; `embedPair` +runs inference once and derives both, halving the cost of ingest, which +needs both per chunk. Empty input short-circuits to a zero engram and an +empty vector. All three default to the `DeterministicTokenizer` stand-in — +EmbeddingGemma's is sized for its SentencePiece vocabulary of 256,000 and +context of 2,048 tokens — so until real tokenizers ship, embedding values +are a property of the host's model bundle. What the kit itself owns, the +token-to-engram pipeline given a pooled vector, is bit-identical across +ports. + +## NLEmbeddingProvider.swift + +This file provides `NLEmbeddingProvider`, the Apple sentence-embedding +signal built on the operating system's bundled `NLEmbedding` model. + +It is the cheap, immediate Apple-native lane: no download, no CoreML seam, +no training — the OS framework is the model, so there is nothing for a +host to inject. The provider looks up the OS sentence model for its +configured language, embeds the text, casts the result to floats, and +normalizes to unit length with the substrate's canonical vector +operations, keeping the float lane's cosine on a unit sphere like every +other provider. Absence is graceful: no model for the language, or text +the model cannot embed, returns an empty vector — a typed lane opt-out, +not an error. The projection seed spells `APNLEMB1`, defined here rather +than in the doctrine's CoreML seed table but under the same never-collide +rule. The file is compiled only where NaturalLanguage exists; there is no +Rust counterpart, a sanctioned Swift-only divergence recorded in ADR-019, +and recall fusion simply handles the absent lane elsewhere. + +## NLContextualEmbeddingProvider.swift + +This file provides `NLContextualEmbeddingProvider`, the higher-quality +Apple lane built on the on-device `NLContextualEmbedding` transformer. + +The transformer needs a per-language downloadable asset that may be +absent, and the file's central rule is that an embed call must never +trigger a network fetch as a side effect. The provider checks asset +availability with a free, synchronous call and opts out with an empty +vector when the asset is missing; prefetching assets is the host +application's job, done before constructing the provider. When assets +exist, the provider runs the transformer, mean-pools the per-token vectors +(the conventional strategy for a transformer without a sentence-pooling +head, with a defensive dimension guard that skips malformed token +vectors), and normalizes. Every failure mode collapses to the empty-vector +opt-out, because a missing asset is an expected operational state. The +projection seed spells `APNLCTX1`, distinct from the sentence provider's +so the two lanes key to separate storage partitions. Apple-only, no Rust +counterpart, per ADR-019. + +## Rust Port and Conformance + +The `rust/` directory mirrors the core target and the `rust-providers/` +directory mirrors the providers target, matching the two-target split so +core consumers never depend on provider code. The core crate ships the +same chunk, chunker, tokenizer, store, engine, hybrid-recall, ingest-queue, +and `Corpus` types; concurrency uses locks where Swift uses actors, and +the queue and inverted-index store own private connections where Swift +shares the estate's storage. The providers crate ships the deterministic +tokenizer, the shared count and vocabulary-reduction code, the basis +codec, the five honest signals, and the three named neural providers over +the same host-injected inference seam. Neither crate bundles model +weights. + +Three fixture families gate the ports. The shared canonical vectors in +`Tests/SharedVectors/` — BM25 impacts, per-provider embedding vectors, and +serialized basis blobs — are read by both legs and must reproduce byte for +byte. The Rust test suite additionally pins basis serialization +byte-for-byte for all four trainable providers and round-trips the counts +seam. The two Apple NaturalLanguage providers are the one sanctioned +divergence: they exist only in Swift, and the parity baseline is the +deterministic and classical providers. When you change either leg, run +both test suites; the fixtures are the contract. diff --git a/packages/kits/CorpusKit/docs/OVERVIEW.md b/packages/kits/CorpusKit/docs/OVERVIEW.md new file mode 100644 index 0000000..4cf6972 --- /dev/null +++ b/packages/kits/CorpusKit/docs/OVERVIEW.md @@ -0,0 +1,202 @@ +--- +doc: OVERVIEW +package: CorpusKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/CorpusKit/BasisStore.swift + blob: 48850906faa7c2fe4aac2859a1c4e892cff32cab + - path: Sources/CorpusKit/BM25Index.swift + blob: 06fb90cd40e81f013e01a8a6c4c6f94e71bf33f3 + - path: Sources/CorpusKit/BundleStore.swift + blob: 419b1c0609597cdd68bf623ed37bd40a0171597b + - path: Sources/CorpusKit/Chunk.swift + blob: d5a1be1bb08858f5f7bd59bb141a8a0ba6f1dfbe + - path: Sources/CorpusKit/Chunker.swift + blob: a2718e06d1715f539ff633e7037c70e10ecb7a2d + - path: Sources/CorpusKit/CorpusIngestQueue.swift + blob: 2c32133701ce728bc017d4ddad51b052cae990db + - path: Sources/CorpusKit/CorpusKit.swift + blob: 4518f15fdb798c3a203c4a9db949f4d6172f540d + - path: Sources/CorpusKit/CorpusKitError.swift + blob: 68ac8d0a248bc9c2dd1885b0bc531ac4ed9cb91d + - path: Sources/CorpusKit/CorpusProviderCountsStore.swift + blob: c92160041765cc8546501c5ac4d8a2b769656e93 + - path: Sources/CorpusKit/Engine/BM25Weighting.swift + blob: 622f45870ab1118d7590cce1b379a063619b4714 + - path: Sources/CorpusKit/Engine/Fusion.swift + blob: d128ed9bc206612fc7c2e849a2e77d03d8a6cafa + - path: Sources/CorpusKit/Engine/InvertedIndex.swift + blob: 1273adcb3794b1997c93488182fc5ed95b21f9ec + - path: Sources/CorpusKit/Engine/InvertedIndexStore.swift + blob: 242baf05e5c846c719c403599a9a407bba646f5b + - path: Sources/CorpusKit/Engine/SparseTypes.swift + blob: 54654e2c49b09d31f60c06503216a2b281939f87 + - path: Sources/CorpusKit/HybridRecall.swift + blob: 21d9fb3415b699c6469a1e80d5e84da1e43981ca + - path: Sources/CorpusKit/RemovedSourceStore.swift + blob: 138a2a094ff369ee69eafa04eeba77206a58e5f4 + - path: Sources/CorpusKit/SyncManifest.swift + blob: 39af591d1fbf1f213c93eb143c213258e41c6c4b + - path: Sources/CorpusKit/Tokenizer.swift + blob: 603028510f91b3c6d75cdda1cb0a1db1c59eee28 + - path: Sources/CorpusKit/TrainableEmbeddingBasis.swift + blob: 4722def84980b3a8987adb090a1a702ac789f8ab + - path: Sources/CorpusKitProviders/BasisCodec.swift + blob: d107e1efd6341648fd8f717c7956a15b98c1b29f + - path: Sources/CorpusKitProviders/DefaultEnsemble.swift + blob: c58168f0991cc4e4c3ec490e2272bdc1a5a17be1 + - path: Sources/CorpusKitProviders/DeterministicTokenizer.swift + blob: 0586b3a4ae93dc0b58ef8f62c0d104a81dcfefe3 + - path: Sources/CorpusKitProviders/EmbeddingGemmaProvider.swift + blob: 593cd5952aad04fe0390e133e68cc07cad983d86 + - path: Sources/CorpusKitProviders/FdcProvider.swift + blob: 96f3ffaf64c7b17e2f617f49ce06cd649e70a027 + - path: Sources/CorpusKitProviders/LsaProvider.swift + blob: 3870f0d24659cb27ba13dc2cba9f94debb6b5c07 + - path: Sources/CorpusKitProviders/MiniLMTextProvider.swift + blob: 35c739a37e9ef1a92098458b48c5f5a06f11050f + - path: Sources/CorpusKitProviders/MPNetTextProvider.swift + blob: 6f13fc6dcd4733460cad366e78273f542c65a844 + - path: Sources/CorpusKitProviders/NLContextualEmbeddingProvider.swift + blob: 17d1acc363bab9c5d0bb807041e0b4c3d66fa0ee + - path: Sources/CorpusKitProviders/NLEmbeddingProvider.swift + blob: 2d714a051a1b6bd4dbde1ecd0e182a81f94b7008 + - path: Sources/CorpusKitProviders/NmfProvider.swift + blob: 43f0e339a42426d68788a03e8a216890c73eb05a + - path: Sources/CorpusKitProviders/PpmiProvider.swift + blob: 282e2185cb7e3d066979ea23b74e096ee337545b + - path: Sources/CorpusKitProviders/RandomIndexingProvider.swift + blob: 552b55b1be93fb57b9e7daf123ecf3a73df7abef + - path: Sources/CorpusKitProviders/ReducedVocab.swift + blob: fb50a8566f9ef3b2a9c650102274b20894d6542d + - path: Sources/CorpusKitProviders/TermDocumentCounts.swift + blob: e72231cf3e50799b6bbeac6a165b80410dc40317 +--- + +# CorpusKit Overview + +## What This Kit Does + +CorpusKit stores text and finds it again by meaning as well as by keywords. +It is the retrieval tier of MOOTx01, an on-device AI memory system. The +technique it implements is called retrieval-augmented generation, or RAG. In +RAG, an AI looks up relevant stored material before it answers, so its +answers rest on real text instead of on guesswork. + +A kit is a larger package that composes libraries into a subsystem. CorpusKit +composes storage, indexing, and embedding libraries into one database-like +surface. Callers hand it text; it splits the text into chunks, stores each +chunk, and indexes it two ways. A chunk is a piece of source text with a +stable identity, sized to a few sentences. Later, callers hand it a query and +receive the most relevant chunks back, scored and ranked. + +The kit stands alone. A developer can use it as a private RAG database with +no other MOOT components. Inside MOOTx01, the GeniusLocusKit orchestrator +uses it as the estate's recall engine. An estate is one user's complete +memory store. + +## The Problem It Solves + +Recall must work on the device, deterministically, and without leaking text. +Cloud embedding services see private content, need a network, and change +without notice. If recall depended on them, a user's memory would be neither +private nor reproducible. Federation raises the stakes: MOOTx01 estates can +share memories across devices, and shared recall only works when every +device computes the same result from the same input — the agreement +property. + +CorpusKit answers with two ranked lanes that both run entirely on device. A +lane is one independent way of scoring how well a chunk matches a query. The +keyword lane uses BM25, a standard formula that rewards chunks containing +the query's rarer words. The semantic lane uses embeddings. An embedding is +a list of numbers (a vector) that represents what a text means; texts with +similar meaning get nearby vectors. The two lanes are fused into one ranking +by Reciprocal Rank Fusion, a simple rule that rewards chunks ranked high in +either lane. + +For the semantic lane, the kit ships an ensemble of five "honest" signals: +Random Indexing, PPMI, LSA, NMF, and FDC. Honest means each signal reflects +real word co-occurrence or real classification structure, never a disguised +hash of the surface text. All five are classical statistical methods. They +need no neural network, cost little, run identically on every platform, and +are gated by shared conformance fixtures — recorded input and output pairs +that the Swift leg and the Rust leg (in `rust/` and `rust-providers/`) must +reproduce exactly, byte for byte. Optional higher-quality neural providers +(MiniLM, mpnet, EmbeddingGemma, and two Apple NaturalLanguage providers) +plug into the same seam when a host supplies the model. + +## How It Works + +Ingestion runs in a pipeline. Text enters through an ingest queue backed by +QueueKit, so callers never wait on encoding. A background drain worker takes +batches from the queue and hands them to the `Corpus` actor, the kit's +central type. The `Corpus` splits each text into chunks with sentence-aware +boundaries. Each chunk receives a content-addressed identity: its UUID is +computed from its source, offset, and exact text. The same content always +produces the same identity, so re-ingesting a document is a harmless no-op +and two federated devices converge on identical rows. + +Each stored chunk is then indexed twice. The keyword side tokenizes the +chunk and records term frequencies in a persistent inverted index — a table +mapping each word to the chunks that contain it. The semantic side runs the +chunk through every configured embedding signal and writes the resulting +vectors to VectorKit, the sibling kit that owns vector storage and +nearest-neighbor search. Content and vectors are joined by the chunk's UUID +string, so a chunk and its meaning never drift apart. + +Four of the five honest signals are trainable: they learn a basis from the +corpus itself. A basis is the trained reference data a signal needs to embed +new text — for example, the word co-occurrence vectors Random Indexing +accumulates. Training happens exactly twice: automatically on first ingest, +and again whenever a caller requests an explicit reindex. Trained bases are +serialized to a pinned little-endian byte format and persisted, so a +reopened corpus embeds immediately without retraining. + +Recall runs the pipeline in reverse. The query is embedded once, the vector +lane fetches its nearest neighbors, the keyword lane fetches its best BM25 +matches, and Reciprocal Rank Fusion merges the two rankings with pinned +weights (0.6 vector, 0.4 keyword). The winners are hydrated from the chunk +store and returned as scored chunks. Ties always break toward the smaller +identifier, so results are deterministic down to the last position. + +Deletion is honest about its limits. Chunk rows are immutable, so removing a +source deletes its index rows and vectors and records a tombstone that every +rebuild consults; expunging additionally scrubs the stored text itself. + +## How the Pieces Fit + +Figure 1 shows the kit's topology — its major parts and how data moves +between them. + +![Figure 1. Topology of CorpusKit](topology.svg) + +*Figure 1. Topology of CorpusKit. Ingested text flows through the queue and +the `Corpus` actor into the chunk store, the keyword index, and the vector +store. A query fans out to both index lanes, and Reciprocal Rank Fusion +merges them into scored chunks. Dashed regions mark the external kits and +the persisted tables.* + +The `Corpus` actor is the seam everything passes through. It owns the chunk +store (`BundleStore`), the persistent keyword index (`InvertedIndexStore`), +the basis and counts stores, the tombstone store, and one slot per embedding +signal. It hides VectorKit behind its own surface, so consumers never touch +vector storage directly. The engine layer beneath it — sparse types, BM25 +weighting, the WAND/Block-Max WAND inverted index, and the fusion function — +is pure computation with no storage of its own. + +The package splits into two targets on purpose. The `CorpusKit` core target +holds the storage, the engines, and the protocols. The `CorpusKitProviders` +target holds every concrete embedding provider and tokenizer. Consumers that +only need storage and BM25 never pull in provider code or model seams. + +## What Ships in the Package + +The package ships the two Swift targets, thirty-four Swift source files in +all, and two mirror Rust crates: `rust/` for the core and `rust-providers/` +for the providers. Shared canonical vectors in `Tests/SharedVectors/` gate +both legs: BM25 impacts, per-provider embeddings, and serialized basis blobs +must match byte for byte. The kit bundles no model weights. Hosts that want +neural embeddings inject an inference function; everything the kit itself +computes is deterministic and reproducible from the sources alone. diff --git a/packages/kits/CorpusKit/docs/topology.svg b/packages/kits/CorpusKit/docs/topology.svg new file mode 100644 index 0000000..228e080 --- /dev/null +++ b/packages/kits/CorpusKit/docs/topology.svg @@ -0,0 +1,126 @@ + + + + + + + + + + + + + CorpusKit: text in, scored chunks out + + + + Text in + enqueueIngest + + + Ingest queue + QueueKit · encode stream + + + Corpus actor + composition root + + + Chunker + 800 chars · 100 overlap + + + + + + + + + Embedding ensemble + RI · PPMI · LSA · NMF · FDC + + + + + + VectorKit (external kit) + + VectorStore + engrams + float rows + + + + + + Persisted tables (PersistenceKit; SQLite or in-memory, chosen by the host) + + + BundleStore + chunks · Merkle roots + + + InvertedIndexStore + BM25 stats · WAND/BMW + + + Basis · counts · tombstones + trained bases · removed sources + + + + + + + + + Query + recall(query) + + + HybridRecall + Fusion + RRF k=60 · weights 0.6/0.4 + + + ScoredChunk results + score DESC, id ASC + + + + + + + + + diff --git a/packages/kits/LocusKit/docs/AGENT_MAP.md b/packages/kits/LocusKit/docs/AGENT_MAP.md new file mode 100644 index 0000000..2f1c7f1 --- /dev/null +++ b/packages/kits/LocusKit/docs/AGENT_MAP.md @@ -0,0 +1,481 @@ +--- +doc: AGENT_MAP +package: LocusKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/LocusKit/Adjectives.swift + blob: d95d3ad8b02c61b166be790138b710b595f30c3d + - path: Sources/LocusKit/Association.swift + blob: 4b6f442c58dd6f182b3f5273348aa892eb1a5c55 + - path: Sources/LocusKit/AssociationOperational.swift + blob: 3228571f3f04aa662fa45c3ea098a97e0cd6cc96 + - path: Sources/LocusKit/AuditTypes.swift + blob: 872b7b19dc652f11b695361101d026b31a1a581d + - path: Sources/LocusKit/BitmapEvaluator.swift + blob: f2e25b7a0037131f4ea230bff62d33601a4af1f5 + - path: Sources/LocusKit/BitmapOps.swift + blob: 44dc29d22b712eb8631e07531e0bd3398d367197 + - path: Sources/LocusKit/BundleMaterializer.swift + blob: 59eae4efba9df236c0a7953df2b9649053e8133f + - path: Sources/LocusKit/ContainerFingerprintStore.swift + blob: fa7cb95c176cae53f0f24546112d87701ac1cf55 + - path: Sources/LocusKit/DefaultWings.swift + blob: a3469fce1fab0754cd379aa2e3742562dbc01717 + - path: Sources/LocusKit/DiaryEntry.swift + blob: 7b6d202251630f26b4a88110ecf4c91a5712c0b9 + - path: Sources/LocusKit/DiaryOperational.swift + blob: a5dc4ac124da1217fbaa716a63a194005d70523f + - path: Sources/LocusKit/Drawer.swift + blob: cc75fb3b8bc2450614dae65fb6d976d3f025cf5b + - path: Sources/LocusKit/DrawerFingerprint.swift + blob: 30bf6bf545f1699c1a5f7f31053ce77e65508b06 + - path: Sources/LocusKit/DrawerOperational.swift + blob: 26588caeb8b69f8ebb9973eac35216b957a1461c + - path: Sources/LocusKit/DrawerStateValidator.swift + blob: 0fcc6d5affd39342d2263c841c5fd67894b7f163 + - path: Sources/LocusKit/DrawerStore.swift + blob: 4379bc3c3477c9ac0a29c533d517fc8a67fbf184 + - path: Sources/LocusKit/Estate.swift + blob: 668ad245361b96317e5fa9ad9a35ccc897c2c138 + - path: Sources/LocusKit/EstateAudit.swift + blob: f17f901ea714fee7b7c2a6b8dc3c18e6684586d8 + - path: Sources/LocusKit/EstateIdentityKeyStore.swift + blob: 6a0cfc4dcc0c539c5ad0b5af4400dc0e0ee532e6 + - path: Sources/LocusKit/EstateTypes.swift + blob: 9afca8d07eee38ac8f0ad12fcb26aec4f732c1e8 + - path: Sources/LocusKit/EstateVerbs.swift + blob: ae30cf0be732d3b68bdb94e0193fd37171def37b + - path: Sources/LocusKit/Filter.swift + blob: 4260cd963863bbc6f31f87d7f037e92ae7d95e16 + - path: Sources/LocusKit/Fingerprint256Adapters.swift + blob: a7db452b9cc676b4af53a1c0f78c98d15df9c6c7 + - path: Sources/LocusKit/ForbiddenCombinationValidator.swift + blob: 809c5fdf1e7a6bcc219b9410f6f247eec65b275d + - path: Sources/LocusKit/Frames.swift + blob: fea5a8d8a487cc28ef967941eab8929b1d2b44df + - path: Sources/LocusKit/KGFact.swift + blob: 3d018021557fa23cdce8aa803cd99ce6be109ccd + - path: Sources/LocusKit/KGFactOperational.swift + blob: 635c214307e50e506ee9263fb7847cbcbbc2e165 + - path: Sources/LocusKit/LearnedReference.swift + blob: 9425889adb88a2a6348d134c0acbd6f1103cf8b8 + - path: Sources/LocusKit/LearnedReferenceOperational.swift + blob: c4a4154017577072f258996ca74c9d240cec7fcf + - path: Sources/LocusKit/LocusKit.swift + blob: f201f00a27e2b906a0fc1502bfdef55282844119 + - path: Sources/LocusKit/LocusKitError.swift + blob: cf9ea82e072c56c69fb2e88a62c1835391a3768e + - path: Sources/LocusKit/LocusKitSchema.swift + blob: f672290df86b6c6df6ca5d243a20e62a0ca1c023 + - path: Sources/LocusKit/LocusKitTelemetry.swift + blob: 732f995c52e8af16477f34243b5d898080cec7da + - path: Sources/LocusKit/LocusKitVocabulary.swift + blob: 8d207c675d5487931127b5fb34380a832df2c025 + - path: Sources/LocusKit/Manifest.swift + blob: 206d4ce171a49d96512f68e6398746edcc2ea13a + - path: Sources/LocusKit/MerkleRollup.swift + blob: ad7b32eb7d9ed1938e1c88c8abfedecb40500562 + - path: Sources/LocusKit/Node.swift + blob: 8d59f6721beca08b957c60eddc96c67df9cb39bb + - path: Sources/LocusKit/NodeBundleStore.swift + blob: f24b4bd9fa5b7ef0f4c63b3e9f066c484af17e56 + - path: Sources/LocusKit/NodeStore.swift + blob: 5fe41a84a4594365c16fe8616df24eb828092dbe + - path: Sources/LocusKit/Proposal.swift + blob: e92d684ddb11ec7d9babfc577f9b68d7730c0ec2 + - path: Sources/LocusKit/ProposalOperational.swift + blob: 65477d9fcafe5bdd9e53927833a08b27f6779247 + - path: Sources/LocusKit/Provenance.swift + blob: 95d989f9cf53084c29e2c9d506c2f18f6fd9bb07 + - path: Sources/LocusKit/RecallStream.swift + blob: bb038171818090f8b7b7db0cb39d25d4563ed9e1 + - path: Sources/LocusKit/RecallTraceItem.swift + blob: 1a96a2daf6dc5edb46b8ac6934b6f8670227a3dd + - path: Sources/LocusKit/SourceCatalogEntry.swift + blob: b91b6b61cb59150270488c7451bb8556449a3bc1 + - path: Sources/LocusKit/Summaries.swift + blob: fa624f6b00d8bf40f1ccc02821e3e2b94bf94f6e + - path: Sources/LocusKit/Tunnel.swift + blob: 8da22988147aab0a9451f06ec681d10891da1e95 + - path: Sources/LocusKit/TunnelOperational.swift + blob: a5b9b8d3b5b7990ebcb9d9c6209564a7307aa13e +--- + +# AGENT_MAP — LocusKit + +PURPOSE: storage substrate for one MOOTx01 estate (MemPalace). Nine gated noun tables (drawer/tunnel/diary/kg_fact/proposal/association/learned_reference/source_catalog/node) over PersistenceKit; every state-changing write routes through SubstrateLib's AuditGate (legal-transition + forbidden-combination check) and appends one sealed AuditEvent atomically with the row update. Read side: RecallFrame → [Filter] chain → fingerprint-pruned, four-tier BitmapEvaluator (bitmap/structured/content/order) → paged RecallStream. Estate (actor) is the sole public facade; GeniusLocusKit composes on top. + +DEPS: imports SubstrateLib+SubstrateTypes+SubstrateKernel (AuditGate, BitField, RowStateAutomaton, HLC, SimHash, ORReduce, FNV, MerkleHash — DO NOT REIMPLEMENT), SubstrateML (LexRank-adjacent kernels used indirectly), PersistenceKit (Storage/RowStore/AuditLog/Transaction abstraction — SQLite/Postgres/InMemory backends), IntellectusLib (opt-in telemetry, off-path ~1ns), LatticeLib (QIDClosure — pinned P31/P279 ancestor snapshot, DrawerFingerprint lattice block only). Imported by: GeniusLocusKit (composes on top — never the reverse). Rust port in rust/ (57 files under src/+tests/) mirrors every type + the bitmap/state/fingerprint math; conformance tests in rust/tests/*.rs gate byte-identity per axis. + +ENTRY POINTS (most callers need only these): +- Estate.swift:187 `Estate.open(storage:owner:identityKeyStore:) -> Estate` — open existing estate, validates bitmap_layout_version, mints/loads Ed25519 identity +- Estate.swift:314 `Estate.create(storage:owner:manifest:) -> Estate` — create new estate +- EstateVerbs.swift:69 `Estate.capture(_ frame: CaptureFrame) -> Drawer` — file a drawer +- EstateVerbs.swift:619 `Estate.recall(_ frame: RecallFrame) -> RecallStream` — query drawers +- EstateVerbs.swift:1228 `Estate.mutate(rowID:kind:payload:)` — move state/adjective axis +- EstateVerbs.swift:1030 `Estate.expunge(rowID:reason:confirmation:now:)` — hard delete + audit + +## Symbol Table + +### Module — LocusKit.swift +- :1 doc-only file; no public symbols. Public types live in neighbouring files. + +### Errors — LocusKitError.swift +- :11 `enum LocusKitError: Error, Sendable, Equatable` — every LocusKit failure mode +- :18 `.databaseUnavailable(String)` / :21 `.drawerNotFound(id:)` / :23 `.tunnelNotFound` / :27 `.diaryEntryNotFound` / :32 `.recallTraceItemNotFound` +- :38 `.sqliteError(String)` / :42 `.schemaTooNew(found:expected:)` +- :48 `.invalidContent(String)` — msg IS the contract, tests assert on text +- :59 `.disciplineViolation(from:to:reason:)` — illegal transition / forbidden combo; from/to are raw Int, not State (keeps LocusKitError dependency-free of Adjectives) +- :72 `.corruptStoredValue(table:column:storedText:)` — fail-loud, never fabricate a default +- :81 `.notSupported(String)` +- :90 `description: String` — English render consumed by GLK's gate-rejection text parser (soft contract) + +### Schema — LocusKitSchema.swift +- :80 `kitID = "LocusKit"` / :90 `version = 8` (no migration ladder — no estate data has shipped) +- :97 `static var schema: SchemaDeclaration` — 17 tables + generated columns + indices, ALL declarative (zero raw SQL) +- Every persistent table carries nullable `.json` `ext` column (ADR-012 forward-compat slot) +- Generated columns: `g_state_cluster` (adjectiveBitmap & 0x3F, on drawers/kg_facts/proposals), `g_provenance_source`/`g_provenance_confirmation`/`g_operational_channel` (drawers only) — indexed like ordinary columns +- Header comment: bitmap reservation map (assigned vs FREE bit ranges per column) — consult before claiming a new flag bit + +### Vocabulary — LocusKitVocabulary.swift +- :27 `enum LocusKitVocabulary` — LocusKit's operational+provenance FieldSlot union (adjective basis supplied by SubstrateLib itself) +- :61 `frozen() -> Result` — called once at DrawerStore.init; frozen for estate lifetime + +### Estate types — EstateTypes.swift +- :15 `typealias RowID = String` +- :27 `struct FrameFilteredDrawers` — :30 `admissible: [Drawer]`, :32 `loadedIDs: Set` (loaded-but-filtered vs never-loaded distinction) +- :51 `struct OwnerCredentials` — :55 `ownerIdentifier: String` (non-empty enforced at Estate.open/create) +- :76 `struct LatticeAnchor: Equatable, Codable` — udcCode(required)/udcFacets/wikidataQID/wikidataQidsSecondary; :101 `.udc(_:)` convenience +- :117 `enum EstateError` — :121 substrateUnavailable / :128 manifestMismatch(key:found:expected:) / :133 emptyOwnerIdentifier / :140 keychainError(status:) + +### Estate facade — Estate.swift +- :27 `actor Estate` — owns :45 `store: DrawerStore` (internal, EstateVerbs-only), :50 `containerFP: ContainerFingerprintStore`, :57 `nodeStore: NodeStore` (public) +- :36 `expectedBitmapLayoutVersion = "v1.0"` — Estate.open refuses mismatch +- :187 `open(storage:owner:identityKeyStore:)` — mints Ed25519 keypair on first open (public key → manifest, private key → identityKeyStore); backfills containerFP via rebuildAll +- :314 `create(storage:owner:manifest:)` — no keypair minted (deferred to first open) +- :401 `close()` — no-op; caller's storage owns teardown +- :412/:429/:448 `allDrawers(...)` overloads — full/limit/hydrationLevel variants +- :462 `tombstonedLineageIDs() -> Set` — cluster-C block-set for vault re-import guard +- :474 `roomLevelFingerprints()` — passthrough to containerFP.roomLevelEntries +- :485/:494 `getDrawers(ids:...)` — batch by-id, hydration-level aware +- :527 `getDrawers(ids:matchingFrame:hydrationLevel:) -> FrameFilteredDrawers` — frame-faithful O(candidates) by-id load; forces .full when chain has content predicate +- :566 `hydrateBodies(ids:)` — late body hydration for dense-first pool pattern +- :578 `tunnelsFromWing(_:)`, :587 `recentRecallTraces`, :594/:604 `allTunnels`/`allActiveTunnels`, :612/:621 `retireTunnel`/`unretireTunnel`, :635 `pruneRecallTraces`, :651 `markRecallTracesUsed`, :657/:665 `countRecallTraces`/`countDrawerRows` +- :674/:681/:689/:698/:706/:713 `allProposals`/`allAssociations`/`allLearnedReferences`/`allKGFacts`/`allKGFactsIncludingRetired`/`allDiaryEntries` +- :722 `resolveNodeNames(parentNodeIds:)` +- :737 `manifest: ManifestValues { get async throws }` — re-reads on every access +- :745 `estateUUID: UUID` — stable, parsed once at open +- :759 `retrievePrivateSigningKeyData() -> Data?` +- :780/:789 `meta(key:)`/`setMeta(key:value:)` — public per-estate KV surface (caller must namespace keys) +- :70 `RecallInternalRead` enum + :97 `_testForceInternalReadError` — TEST-ONLY single-use fault-injection seam for recall's internal reads + +### Estate audit — EstateAudit.swift +- :53 `auditTrail(rowID:) -> [AuditEvent]` — full sealed history, HLC order; empty until first mutation (capture = INSERT, not mutation) +- :83 `bitmapState(rowID:asOf:) -> BitmapState` — folds audit log via AuditLogFold.projectStateAt; throws drawerNotFound if no events ≤ asOf + +### Identity keys — EstateIdentityKeyStore.swift +- :23 `protocol EstateIdentityKeyStore: Sendable` — loadPrivateKey/storePrivateKey(forEstateID:) +- :48 `struct KeychainEstateIdentityKeyStore` — kSecClassGenericPassword, service "com.mootx01.estate.identity", account=estateUUID, kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly; production +- :130 `final class InMemoryEstateIdentityKeyStore: @unchecked Sendable` — NSLock-guarded dict; test-only; :147 `_storedPrivateKey(forEstateID:)` test peek + +### Verbs — EstateVerbs.swift (extension Estate) +- :69 `capture(_ frame: CaptureFrame) -> Drawer` — validates 5 required fields; assembles 3 bitmaps via BitField.writeField; resolves wing/room to node ids; routes through :432 `addDrawerCovered` (the ONLY sanctioned add chokepoint — bundles store.addDrawer + containerFP.orIn) +- :239 `captureBatch(_ frames: [CaptureFrame]) -> [Drawer]` — bulk path; ONE transaction for all fresh inserts via `store.insertFreshBatch`; Merkle rollup deliberately deferred (NT_R1) +- :487 `capture(_ frame: TunnelCaptureFrame) -> Tunnel` — standalone tunnel capture (2nd of the two nouns capture accepts) +- :573 `recallCandidateCap = 256` — scan bound floor for director-style callers +- :619 `recall(_ frame: RecallFrame) -> RecallStream` — non-throwing; internal-read failures surfaced via `RecallStream.degradedStages`, never silently emptied +- :743 `enum RecallStage` — stable stage-id strings (locus.liveRows.readFailed, locus.roomFingerprints.readFailed, locus.roomDrawerRead.readFailed, locus.bitmapEval.failed, recall.trace_write_failed) +- :800 `liveRows(for:forcedFault:)` (private) — fingerprint-pruning OR bounded-scan dispatch; no-blob fast path when chain has no content predicate +- :960 `withdraw(rowID:reason:)` — state → .withdrawn via mutateState(.retract) (NOT mutateAdjective — must go through automaton) +- :1030 `expunge(rowID:reason:confirmation:now:) -> AuditEvent?` — requires confirmation:true; walks full lineage chain; rolls up Merkle for every affected room +- :1085 `expungeReturningUnsealedEvent(...)` / :1141 `sealExpungeAudit(_:)` / :1155 `sealExpungeOrphanAudit(...)` — GLK's 3-step §B-2a orchestration (storage tombstone → cross-kit delete → seal); sealAudit:false is the ONLY split path +- :1178 `sealExpungeOrphanAuditSynthetic(rowID:now:)` / :1196 `tombstonedRowsWithoutExpungeAudit()` — crash-window recovery sweep support +- :1228 `mutate(rowID:kind:payload:)` — dispatches MutationKind → confirm(provenance)/reject|contest|resolve|accept|supersede(state)/revive(state, full per-source-state legality table incl. living-successor lineage guard)/correctSensitivity|correctTrust|correctExportability(adjective field) +- :1528 `reanchor(rowID:toRoom:toWing:toLattice:)` — placement move, requires ≥1 non-nil arg; toWing rejects empty/whitespace +- :1578 `reanchorAnchor(rowID:toLattice:changedBy:now:)` — deterministic-timestamp variant for Q-ID-completion acceptance path +- :1615 `propose(_ frame: ProposeFrame, now:) -> Proposal` — derives candidateState+latticeAnchor from target drawer +- :1690 `associate(_ frame: AssociateFrame, now:) -> Association` — derives latticeAnchor from endpoint A +- :1774 `learn(_ frame: LearnFrame, now:) -> LearnedReference` — catalogs source once by handle; NEVER fabricates a sentinel anchor +- :1936 `defaultWing() -> String` (private) — returns "Agentic Memory" (ADR-016, fixed, not owner-derived) +- :1971 `seedWing(_:hint:addedBy:embeddingModelID:now:)` — files ordinary recallable hint drawer per default wing at provision + +### Frames — Frames.swift +- :24 `struct CaptureFrame` — every named capture-time axis; nearly all defaults reproduce pre-slot all-zero bitmap byte-identically +- :178 `struct TunnelCaptureFrame` — no content/anchor/embedding slots (edges carry none) +- :237 `struct RecallFrame` — filterChain/hydrationLevel/limit/ordering/asOf/traceLimit (traceLimit nil = ZERO trace writes, the default) +- :287 `enum MutationKind` — .confirm/.reject/.contest/.resolve/.supersede/.revive/.accept/.correctSensitivity(_)/.correctTrust(_)/.correctExportability(_) +- :338 `struct LearnFrame` — source(SourceCatalogEntry)/handle/mode/refreshPolicy +- :382 `struct ProposeFrame` — target/kind/justification/confirmation/generatedBy/confidence (LocusKit substrate-axis kind, NOT GLK's Brain-layer kind) +- :436 `struct AssociateFrame` — a/b/weight +- :455 `enum HydrationLevel` — .structured/.full/.bitmapOnly +- :476 `enum Ordering` — .byCaptureTimeDesc/.byCaptureTimeAsc/.byRoomAsc (NO .byRelevanceDesc — LocusKit has no scoring signal; that's VectorKit/GLK's job) + +### Storage engine — DrawerStore.swift (actor, largest file, ~3850 lines) +- :57 `actor DrawerStore` — owns every table; :90 `init(storage:hlc:)` opens schema idempotently, freezes vocabulary, classifies estate_uuid (absent/present/CORRUPT-throws), derives HLC maker node id by FNV-1a hashing raw stored uuid text +- :291 `addDrawer(_:now:)` (internal — NOT the public add path, use Estate.addDrawerCovered) — dispatches supersession cascade vs plain gatedCapture +- :346 `addDrawerWithCascade(_:priorID:)` (private) — successor+tunnel in ONE transaction (no orphan on tunnel-insert failure), THEN separate gated mutateState(.superseded) on predecessor +- :405 `findActivePredecessor(lineageID:excludingID:)` — indexed g_state_cluster < 3 query +- :439 `livingSuccessorInLineage(lineageID:excludingID:)` — g_state_cluster < RowState.activeClusterUpperBoundRaw(16); wider predicate than findActivePredecessor, used by revive guard +- :468 `lineageChain(for:) -> [String]` — every row sharing lineageID, any state +- :492/:516/:545 `getDrawer(id:)`/`getDrawers(ids:)`/`getDrawers(ids:hydrationLevel:)` — chunked at 900 ids (SQLite 999-var ceiling); :530 `structuredDrawerColumns` = every column except content +- :591/:627/:657 `drawersIn(wing:)`/`drawersIn(wing:room:)`/`drawersBySource(file:)` +- :678/:718 `allDrawers()`/`allDrawers(hydrationLevel:limit:direction:)` — (filedAt,id) compound sort key, portable across SQLite/Postgres/InMemory (NOT rowid); .descending retains newest under a scan cap +- :774/:795/:814 `mutateProvenance`/`mutateAdjective`/`mutateOperational` — all route through :1708 `gatedColumnWrite` (private) which excludes the state field from field-edit writes +- :858 `gatedCaptureBody(_:now:)` (private) — builds @Sendable closure for capture INSERT + AuditGate.admit(verb:.capture, prior:nil); reused by both gatedCapture and addDrawerWithCascade (no nested-transaction support in PersistenceKit v1.0) +- :912 `gatedCapture(_:now:)` (private) — single-drawer non-cascade path +- :932 `insertFreshBatch(_:now:)` (internal) — ALL inserts + audit events in ONE transaction; caller (captureBatch) pre-verifies no active predecessor +- :979 `mutateState(drawerId:to:via:changedBy:reason:now:)` — RMWs prior bitmaps, AuditGate.admit(verb:), writes merged snapshot + audit event atomically +- :1097 `expungeGated(drawerId:...:sealAudit:commitmentKey:commitmentKeyVersion:) -> AuditEvent?` — lineage-wide: gates target via .tombstone, unconditionally zeroes content on EVERY sibling regardless of gate success/failure (destruction contract > state machine), records erasure ledger entries +- :1343/:1378/:1516 `sealExpungeAudit`/`sealExpungeOrphanAudit`/`sealExpungeOrphanForSweep` — deferred-seal + crash-recovery paths +- :1425 `tombstonedRowsWithoutExpungeAudit()` — crash-window orphan enumeration (no "tombstone"/"expungeOrphan" audit event) +- :1469 `tombstonedLineageIDs() -> Set` +- :1576 `reanchorGated(...)` — verb:.mutate (active→active self-loop); anchor delta via priorLatticeAnchor≠afterLatticeAnchor; writes:[] (placement move, no field write) +- :1689 `requireUuid(_:label:) throws -> UUID` (static) — fail-loud on non-UUID row id +- :1783 `declaredSlots(for:)` (static private) — adjective ⇒ Vocabulary.basis (SubstrateLib); operational/provenance ⇒ LocusKitVocabulary.unionSlots +- :1800/:1805 `auditEventsForRow`/`auditEventCountForRow` +- :1832 `addTunnel(_:)` — enforces "one parent per child" for .parent kind (kit-level constraint, not DB unique index) +- :1954/:1963/:1982/:2001 `allActiveTunnels()`/(T13/ADR-021 Phase 7 retirement)/`retireTunnel`/`unretireTunnel` — bit-13 operationalBitmap flag, filtered in-memory (PersistenceKit predicate DSL has no bitmask ops) +- :2019/:2044/:2076 `outlineChildren(of:)`/`outlineAncestors(of:)` (depth cap 256, cycle-safe)/`reparentDrawer(...)` — ADR-017 §11 outline graph over .parent tunnels + order_key +- :2139/:2166 `addKGFact(_:)`/`withdrawKGFact(id:)` — sourceDrawerID may be "" (unanchored-fact sentinel) +- :2228 `addProposal(_:)` — rejects empty latticeAnchor.udcCode +- :2264 `addAssociation(_:)` — rejects empty latticeAnchor.udcCode +- :2328 `addLearnedReference(_:)` / :2372 `addSourceCatalogEntry(_:)` — both reject empty latticeAnchor.udcCode +- :2408/:2428/:2442 `addDiaryEntry`/`readDiary(agentName:lastN:)`/`readDiary(agentName:in:lastN:)` +- :2592/:2614 `insertRecallTrace`/`insertRecallTraces` (batch, one transaction) — :710 in EstateVerbs is the only production write path (opt-in via frame.traceLimit) +- :2713/:2758 `markRecallTraceUsed(id:now:)`/`markRecallTracesUsed(target:since:now:)` — bulk ARIA reward path +- :2692 `pruneRecallTraces(olderThan:)` / :2807 `countRecallTraces()` / :2827 `countDrawerRows()` (raw COUNT(*), bypasses decode — counts corrupt rows too, used as corruption sentinel) +- :2836/:2881/:2932 `listWings()`/`listRooms(in:)`/`taxonomy()` — computed projections, no dedicated wings/rooms table +- :2939/:2948/:2959 `setMeta`/`getMeta`/`readManifest() -> ManifestValues` +- :3095 `resolveNodeNames(parentNodeIds:) -> [String: (wing:room:)]` +- :3143/:3150 `decodeDrawerRows` (throws on any corrupt value — point-lookup path) / `decodeDrawerRowsResilient`→:3419 `decodeDrawerRowsSkipCorrupt` (logs+skips corrupt rows — corpus-scan path; only .corruptStoredValue is caught, everything else rethrows) +- :3682/:3731 `fingerprintsCaptured(in:)`/`fingerprintBitSeries(bit:bucketSeconds:bucketCount:endingAt:)` — feed FFT rhythm-spectrum / moment-summary lenses; effective capture time = eventTime ?? filedAt (ING-01) + +### Drawer value type — Drawer.swift +- :31 `struct Drawer: Equatable, Hashable, Sendable, Codable` — verbatim content, never truncated/normalized +- :37 `id` / :46 `lineageID: UUID` (fresh per-row default) / :49 `content` / :54 `parentNodeId: String` (FK → nodes.id, room-depth node) +- :75 `filedAt: Date` (ingest clock) vs :83 `eventTime: Date` (world-happened clock; ING-01 two-clock ingest; defaults to filedAt) +- :109/:120/:132 `provenance`/`adjectiveBitmap`/`operationalBitmap: Int64` — all default 0 +- :151/:159/:166/:175 `udcCode`(required non-empty, I-5)/`udcFacets`/`wikidataQID`/`wikidataQidsSecondary` +- Custom Codable (:228) backfills missing eventTime → filedAt on decode + +### Drawer operational — DrawerOperational.swift +- :61 `enum CaptureChannel: Int` — typed=0/voiced=1/ocr=2/importedFile=3/sensor=4/actuator=5 (bits 0–5) +- :78 `enum ContentKind: Int` — prose=0…fingerprintOnly=6 (bits 6–11) +- :100 `struct DrawerFeatureFlags: OptionSet` — bits 12–23; hasAttachments/hasVoice/hasImage/hasLinks/isPinned/isKeystone(§7.2)/isLockedZone +- :141/:149/:159/:170/:179/:188 `Drawer.captureChannel`/`.contentKind`/`.featureFlags`/`.hasFeatureFlag(_:)`/`.stateExtensionActive`(bit24)/`.lineageClusteringActive`(bit25) — all fail-safe to neutral default on unrecognised raw + +### State transition bridge — DrawerStateValidator.swift +- :53 `validate(from:to:via:)` — transition-legality only, bridges State↔SubstrateLib.RowState, delegates to RowStateAutomaton.transition +- :107 `validate(from:to:via:targetingFields:)` — legality + field-invariant (S-1 trust≥canonical, S-2) combined; the overload DrawerStore.mutateState actually calls + +### Drawer fingerprint — DrawerFingerprint.swift +- :75 `struct EstateFingerprintFamilies` — 4 hyperplane families derived once per estate from estate UUID via HyperplaneFamily.blockFamilies; :90 `baseSeed(estateUUID:)` +- :106 `fingerprint(of: Drawer) -> Fingerprint256` — 4 SimHash blocks: bitmap-LSH(adj/op/prov) / lattice(udcPrefixHash+qidDirectHash+qidClosureHash via LatticeLib.QIDClosure) / lineage+temporal(lineageHash+captureWeekBucket, ING-01 keys off eventTime) / channel+source(channel/sourceType/captureChannel/sensitivity/estateUUIDByte) +- :161/:171 `captureWeekBucket(_:)` (mod 256, 2020-01-01 epoch) / `udcPrefixHash(_:)` (FNV-16 of first 4 digits) +- I-17: cross-noun null fields (defer pattern, completion bucket, behavioral recency, stream-source bitset) always zero on a Drawer — those belong to AmbientSample only + +### Adjective axes — Adjectives.swift +- :99 `enum State: Int` — 10 cases, 3 clusters at raw boundaries 16/32: A(active/pending/contested/accepted) B(superseded/decayed/withdrawn/expired) C(rejected/tombstoned); :121 `isClusterA` +- :143 `enum Trust: Int, Comparable` — verbatim=0…ambient=6(NEW v0.6) +- :173 `enum AdjectiveSensitivity: Int` — normal=0/elevated=16/restricted=32/secret=48 (scale-gapped); ADR-007 tier predicates :197 `isBulkExportable`/:217 `requiresOwnerKeyForBulk`/:237 `isExcludedFromBulk` +- :256 `enum AdjectiveExportability: Int` — private_=0/public_=32 +- Drawer accessors :270-:301 `state`/`adjectiveSensitivity`/`exportability`/`trust` (bits 0-5/6-11/12-17/18-23); :311/:323/:338 `isCurrentlyBelieved`/`isKnewPast`/`isTerminal` cluster predicates +- :354 `dreamingRecalcRequired` (bit 26, obligation flag, F17) / :370 `sealed` (bit 27, custody trust hint — cached, seal itself is authoritative if they disagree) +- `@guardian-pair` annotations throughout — SwiftSyntax Guardian CI tool cross-checks these raw sets against SubstrateLib's AuditGate.basis; GuardianPairParityTests backstop + +### Provenance axes — Provenance.swift +- :67 `enum SourceType: Int` — user=0…actuator=9 (bits 0-5) +- :102 `enum Channel: Int` — uiTyped=0…dreamMiningResult=8, deviceSensor=15, actuatorOutcome=16 (bits 6-11, gaps reserved) +- CaptureChannel mirrored at bits 12-17 (provenanceCaptureChannel accessor, :224) +- :137 `enum Confirmation: Int` — unconfirmed=0/userConfirmed=1/automatedConfirmed=2/peerConfirmed=3/actuatorConfirmed=4 (bits 18-23); :261 `isUserConfirmed` +- :155 `enum Confidence: Int, Comparable` — null=0/low=16/medium=32/high=48/verified=56 (bits 24-29) +- :177 `enum Sensitivity: Int` — normal/elevated/restricted/secret mirrors AdjectiveSensitivity raws (bits 30-35) +- :188 `enum EnrichmentStatus: Int` — none/qidPending/qidCompleted/closureCached/qidProposed(terminal, NOT re-picked by maintenance scan) (bits 36-41) + +### Bitmap primitives — BitmapOps.swift +- :55 `andMask(_:mask:expected:)` → BitField.maskedEquals +- :94 `thresholdCompare(_:mask:shift:op:value:)` + :69 `enum ThresholdOp` (.lessThan/.lessThanOrEqual/.greaterThanOrEqual) → BitField.extractField+compare +- :124 `shiftExtract(_:shift:mask:)` → BitField.extractField +- NOT math primitives — no conformance gate; real math primitives live in SubstrateLib (XOR/popcount/Hamming/OR-reduce/SimHash/SHA-256) + +### Forbidden combination — ForbiddenCombinationValidator.swift +- :75 `validate(_:) throws` — I-3: sensitivity=secret(raw48,bits6-11) AND exportability=public(raw32,bits12-17) forbidden; hand-derived raw constants (NOT imported enums) so a rename can't silently break the check +- No live call site today (current writes route the equivalent check through AuditGate/SubstrateLib directly) — ready to wire in wherever a write path bypasses the gate + +### Recall filter algebra — Filter.swift +- :8/:12/:15/:20/:25/:30 typealiases `LineageID`(UUID)/`RoomID`(String)/`WingID`(String)/`WikidataQID`(String)/`ProvenanceChannel`(=Channel)/`FeatureFlag`(=DrawerFeatureFlags) +- :48 `enum StateCluster` — knowNow/knewPast/terminal +- :69 `indirect enum Filter` — NO raw bit position ever exposed; state/trust/sensitivity/exportability/provenance/operational/structural/content cases + :166/:168/:170 `.all`/`.any`/`.not` composition +- RecallFrame.filterChain ([Filter]) ≡ implicit Filter.all(chain) + +### Recall evaluator — BitmapEvaluator.swift +- :60 `struct BitmapEvaluator` (internal — DrawerStore param is internal) — 4-stage pipeline +- :62-125 private layout-constant mirror of Adjectives/Provenance/DrawerOperational bit positions (schema-bump-coupled) +- :147 `evaluate(frame:drawers:store:nodeNames:) -> [Drawer]` — bitmap tier (w/ historical reconstruction via AuditLogFold.projectStateAt when frame.asOf set) → structured tier → content tier → sort +- :221 `insertDefaults(_:)` (private) — prepends .currentlyBelieve/.trustworthy/.sensitivityAtMost(.elevated) when caller's chain doesn't already constrain that axis; NO confirmation default +- :290/:304/:314 `chainHasPrunableFilter`/`chainHasContentPredicate`/`chainHasStructuredNameFilter` (static, called by Estate.liveRows/getDrawers(matchingFrame:) before fetch) +- :349 `containerSurvives(chain:fingerprint:) -> Bool` — sound pruning test; only set-bit filters (.hasFeatureFlag) can prove exclusion from an OR fingerprint; :358 `containerProvablyExcludes` +- :384 `evaluateBitmapTier` — tombstone (state==33) excluded UNCONDITIONALLY, independent of caller's chain +- :641 `reconstructAt(rowID:asOf:store:)` (private) — historical bitmap fold +- :654 `sort(_:ordering:nodeNames:)` (private) + +### Tunnel — Tunnel.swift +- :24 `struct Tunnel: Equatable, Hashable, Codable, Sendable` +- :29 `id` (conventionally SHA-256 of canonicalised endpoint pair — NOT enforced at this layer, LOCI-5) +- source/target: wing+room+optional drawerId (nil = "the room itself") +- :63 `kind: TunnelKind` (default .references) / :69/:75/:80 adjective/operational/provenance bitmaps / :99 `orderKey: Double?` (fractional-index sibling order, .parent kind only) + +### Tunnel operational — TunnelOperational.swift +- :38 `enum TunnelKind: Int` — supersedes=0…parent=9 (outline containment, ADR-017 §11; source=child, target=parent, one-parent-per-child kit-level constraint) +- :60/:70/:80/:94 `TunnelDirection`/`TunnelLifecycle`/`TunnelOriginClass`/`TunnelStrength(Comparable, scale-gapped 0/2/4/6)` — bits 0-2/3-5/6-8/9-11 +- :168 `isRetiredBit = 1<<13` / :175 `isRetired` / :184/:202 `withRetired()`/`withUnretired()` — T13/ADR-021 Phase 7 reversible dreaming suspension (NOT tombstone) +- :241 `isDreamedBit = 1<<0` (provenanceBitmap) / :247 `isDreamed` / :256 `withDreamedProvenance()` — OMEGA retires ONLY isDreamed==true tunnels; declared tunnels never retired + +### Diary — DiaryEntry.swift +- :24 `struct DiaryEntry` — agentName/entry/topic/wing/room (all required, unvalidated strings) +- :85/:92 `reward: Double?`/`rewardProvenance: String?` — explicit quality-signal channel (NEURONKIT_SPEC §3.1 step 1a); nil ⇒ dreaming daemon falls back to implicit RecallTraceItem.used (step 1b) + +### Diary operational — DiaryOperational.swift +- :33 `enum DiaryEventClass: Int` — 12 cases (capture=0…auditTombstone=11), bits 0-3 +- :54 `enum DiarySeverity: Int, Comparable` — trace=0/info=2/warning=4/error=6, bits 4-6 +- :67 `enum DiaryActorClass: Int` — user=0…federationPeer=4, bits 7-9 +- :79 `enum DiaryBatchMembership: Int` — standalone=0…batchEnd=3, bits 10-12 +- :129 `requiresFollowup` — bit 13 flag + +### KG fact — KGFact.swift +- :51 `struct KGFact` — subject/predicate/object (free-form strings, no entity vocabulary yet) + :82 `sourceDrawerID` (may be "" = unanchored-fact sentinel) +- Adjective accessors :149/:159/:171/:180 `trust`/`state`/`adjectiveSensitivity`/`exportability` — IDENTICAL bit layout to Drawer's (shared retrieval predicates) + +### KG fact operational — KGFactOperational.swift +- :48 `enum KGExtractorClass: Int` — manual=0…federated=5, bits 0-3 +- :66 `enum KGAssertionKind: Int` — asserted=0/inferred=1/hypothesized=2/contradicted=3, bits 4-6 +- :83 `enum KGSpecificity: Int, Comparable` — general=0…instance=6 (scale-gapped), bits 7-9 +- :106 `enum KGConfidenceBand: Int, Comparable` — unknown=0…certain=6 (scale-gapped), bits 10-12 +- :171 `isCanonical` — bit 13 flag (estate-wide load-bearing promotion) + +### Proposal — Proposal.swift +- :72 `struct Proposal` — targetRowID(empty legal for brand-new-object proposals)/justification/:99 `candidateState: Int64`(adjective set to apply if accepted)/:106 `latticeAnchor`(required, I-16) +- :165 `state: State` — decodes proposal's OWN lifecycle axis (pending→accepted/rejected/withdrawn) from adjectiveBitmap bits 0-5 + +### Proposal operational — ProposalOperational.swift +- :60 `enum ProposalKind: Int` — newTunnel=0…tierAdvisory=8, bits 0-5 (COOKBOOK-SPECIFIED layout — conformance-gated, unlike KGFact's LocusKit-internal layout) +- :81 `enum ProposalTargetObjectType: Int` — drawer=0…systemState=6, bits 6-11 (.noneBrandNew=4 ⇒ targetRowID empty) +- :100 `enum ProposalConfirmationSource: Int` — human=0/agent=1/automatedThreshold=2/actuator=3, bits 12-17 +- :116 `enum ProposalGeneratedByClass: Int` — dreamingDaemon=0…tierAggregator=4, bits 18-23 +- :136 `enum ProposalConfidenceBucket: Int, Comparable` — null=0/low=8/medium=16/high=32/verified=48, bits 24-29 +- :207 `composeOperational(kind:targetObjectType:generatedBy:confidence:) -> Int64` (static) — assembles 4 of 5 axes via BitField.writeField; confirmationSource left at .human default + +### Association — Association.swift +- :66 `struct Association: Equatable, Codable, Sendable` (NOT Hashable — embedded LatticeAnchor isn't) — mirrors Tunnel minus `kind`, plus REQUIRED :103 `latticeAnchor` (I-16, lattice-midpoint of endpoints) + +### Association operational — AssociationOperational.swift +- :47 `struct AssociationSignalSources: OptionSet` — bits 0-11 bitset (coRecall/coConfirmed/dreamPairing/vectorSimilarity/sharedEntity/explicitHuman/fingerprintSimilarity/crossEstate/crossTier/actionOutcome); :75 `mask = 0xFFF` +- :90 `enum AssociationDecayClass: Int, Comparable` — pinned=0/slow=16/normal=32/fast=48, bits 12-17 +- :106 `enum AssociationArity: Int` — binary=0(v1 only, I-23)/nAry=1(reserved), bits 18-19 + +### Learned reference — LearnedReference.swift +- :85 `struct LearnedReference: Equatable, Codable, Sendable` (NOT Hashable) — mirrors Association; :96 `sourceCatalogID`(FK→SourceCatalogEntry.id)/:101 `handle`(reference's own locator)/:107 `latticeAnchor`(required, I-16, inherited from catalog entry — NEVER fabricated) + +### Learned reference operational — LearnedReferenceOperational.swift +- :47 `enum RefreshPolicy: Int, Comparable` — none=0/monthly=16/weekly=24/daily=32/onDemand=48/realtime=56, bits 0-5 +- :68 `enum DriftSeverity: Int, Comparable` — none=0/minor=16/major=32/critical=48, bits 6-11 +- :85 `enum LearnMode: Int` — byReference=0/byIngestion=1, bit 12 +- :96 `enum LearnedReferenceSource: Int` — user=0…pairedEstate=5, bits 13-18 (same vocabulary as SourceKind — acquisition channel == source class) + +### Source catalog — SourceCatalogEntry.swift +- :50 `struct SourceCatalogEntry: Equatable, Codable, Sendable` (NOT Hashable) — id/kind(SourceKind)/handle/:68 `latticeAnchor`(required, I-16, GENUINE anchor every LearnedReference from this source inherits)/firstSeen/addedBy +- :103 `enum SourceKind: Int, CaseIterable` — user=0…pairedEstate=5; :119 `fromRaw(_:)` fail-closed to .user + +### Containment tree node — Node.swift +- :25 `struct Node: Sendable, Equatable, Codable, Hashable` — id/parentId(nil only at depth 0)/:34 `displayName`(first-writer casing)/:38 `lookupName`(NFC+casefold+whitespace-collapse, ALL resolution keys use this)/:41 `depth`(0=estate,1=wing,2=room, write-once)/:44 `lifecycle`(0 active,1 tombstoned)/createdHlc/tombstonedHlc/merkleRoot +- :116 `normalizeLookupName(_:) -> String` (static) — CONFORMANCE-GATED, Swift+Rust must byte-match + +### Node store — NodeStore.swift +- :39 `actor NodeStore` — create-on-demand resolution race-free via actor serialization (no INSERT-OR-IGNORE needed) +- :89 `createNode(displayName:parentId:now:) -> Node` — enforces I-NT-5(parent must exist)/I-NT-2(depth≤2)/I-NT-4(no dup active lookupName under parent); tombstoned nodes invisible to resolution (§5 no-resurrection) +- :165 `createRoot(displayName:now:) -> Node` — I-NT-1 single root, idempotent +- :207/:220/:237 `getNode(id:)`/`rootNode()`/`childNodes(parentId:)` +- :259 `tombstoneNode(id:now:) -> Node?` +- :296 `updateMerkleRoot(nodeId:merkleRoot:now:)` — 32-byte BLOB write + +### Node bundle store — NodeBundleStore.swift +- :27 `actor NodeBundleStore` — :32 `enum BundleKind: String` .activeA("A")/.departedB("B") +- :51/:68 `encodeCounts(_:)`/`decodeCounts(_:n:)` — 256×UInt32 LE = 1024 bytes; decode throws (not traps) on wrong size +- :87/:103/:119 `put`/`get`/`rooms(forWing:kind:)` + +### Bundle materializer — BundleMaterializer.swift +- :34 `struct BundleMaterializer` — first real caller of SubstrateKernel.countFold256 +- :63 `materializeRoom(wing:room:now:) -> CountVector256` — filters to State.isClusterA before fold (Bundle A cannot be incremental — no subtraction) +- :79 `rollUpWing(wing:now:) -> CountVector256` — merges room bundles; associative, so room materialize order doesn't matter + +### Fingerprint256 adapters — Fingerprint256Adapters.swift +- :55 `Fingerprint256.init(int64Column:)` / :63 `.int64Column` — packs/unpacks one Int64 bitmap column into block 0 (blocks 1-3 zero); pure type-shape convention, NOT the cookbook §3.2 Bitmap-LSH SimHash interpretation of block 0 — do not conflate the two uses + +### Container fingerprint pruning — ContainerFingerprintStore.swift +- :39 `struct ContainerFingerprint: Sendable, Equatable` — adjective/operational/provenance Int64 OR-aggregate; :68 `merging(_:)` routes through SubstrateLib.ORReduce at Fingerprint256 width +- :87 `actor ContainerFingerprintStore` — :91 `wingRollupRoom = ""` sentinel for wing-level row +- :105/:123 `get(wing:room:)`/`roomLevelEntries()` — nil result means "must scan", not "empty" (unsound to prune against absence) +- :146 `orIn(wing:room:adjective:operational:provenance:now:)` — incremental maintenance, called on every capture; NO clear-side (stale set bits are harmless over-approximation) +- :167/:182/:200 `rebuildRoom`/`rollUpWing`/`rebuildAll` — periodic tightening; rebuildAll runs once at Estate.open + +### Merkle rollup — MerkleRollup.swift (extension Estate) +- :54 `rollupRoomsForDrawers(_ drawerIds:)` — coalesced per-room rollup for the deferred/off-write-path queue worker +- :71 `rollupMerkleRoots(roomNodeId:now:)` — room→wing→estate, 3-level bottom-up for ONE changed room +- :125 `computeRoomMerkleRoot(roomNodeId:)` — excludes tombstoned AND withdrawn(state=18) drawers from the snapshot (WS2-F1); reads content_hash column when present, else computes leaf on-demand +- :175/:186 `computeEstateOrWingMerkleRoot(parentNodeId:)`/`computeWingMerkleRoot(wingNodeId:)` — same function serves both levels +- :200 `recomputeAllMerkleRoots(now:)` — full bottom-up; :243 `rollupAllMerkleRoots(now:)` alias for batch-capture reindex (NT_R1) +- :265 `createSnapshot(label:now:additionalAttestations:)` — ALWAYS recomputes full tree first (capture defers rollup, so cached roots may be stale) +- :332 `deterministicUUID(from:)` (static) — SHA-256-derived stable UUID for non-UUID drawer ids (version/variant bits stamped) + +### Manifest — Manifest.swift +- :6 `enum ManifestKey: String, CaseIterable` — 18 required + 7 optional keys; :41 `.ed25519PublicKey` (safe, public) / :56 `.ed25519PrivateKeyWrapped` (NEVER written by current code — reserved read-compat seam only, real private key lives in EstateIdentityKeyStore) +- :80 `struct ManifestValues` — typed snapshot from DrawerStore.readManifest() + +### Default wings — DefaultWings.swift +- :18 `defaultWingName = "Agentic Memory"` (ADR-016, fixed — NOT owner-derived) / :26 `hintRoom = "AI_Charter_Hint"` / :32 `hintUDCCode = "001"` / :36 `hintAddedBy = "estate-provision"` +- :44 `struct WingDefinition` — name+hint / :64 `defaultWings: [WingDefinition]` — 7 seeded wings (suggestion, not enforced schema) + +### Recall stream — RecallStream.swift +- :21 `struct RecallStream: AsyncSequence, Sendable` — :26 `defaultPageSize = 50` +- :54 `degradedStages: [String]` — non-throwing failure channel (empty ⇒ genuine result incl. genuine-empty; non-empty ⇒ named internal-read/eval failure) +- :86 `struct RecallPage` — rows/pageIndex/isLast +- :92 `struct AsyncIterator: AsyncIteratorProtocol` — first page synchronous; :129 `hydrate(_:)` (private) rebuilds with content="" ONLY at .bitmapOnly + +### Recall trace — RecallTraceItem.swift +- :28 `flagUsed: Int64 = 1<<0` / :58 `used: Bool` (computed, no stored Bool — bitmap-backed pattern every noun follows) / :47 `score: Double?` (nil = no score, e.g. capture-time-ordered recall) + +### Audit types — AuditTypes.swift +- :26 `struct BitmapState` — rowID/asOf(HLC)/adjectiveBitmap/operationalBitmap/provenanceBitmap; LocusKit's own wrapper over AuditLogFold.projectStateAt's result shape + +### Summaries — Summaries.swift +- :10 `struct WingSummary` — name/drawerCount/roomCount / :33 `struct RoomSummary` — wing/name/drawerCount; BOTH are live computed projections, no dedicated wings/rooms table + +### Telemetry — LocusKitTelemetry.swift +- All functions `@inline(__always)`, `Intellectus.report(_:)` with @autoclosure (never evaluated when monitoring off — ~1ns off-path cost, no lock, no alloc) +- `now:` ALWAYS caller-supplied — never Date() inside this file (determinism contract) +- :50 `emitDrawerCapture` / :90 `emitDrawerQuery` / :123 `emitKGFactAdd` / :149 `emitKGFactQuery` / :175 `emitTunnelAdd` / :199 `emitGateAdmit` / :224 `emitGateReject`(reason truncated to 64 chars) +- Namespace: `locuskit..` + +## INVARIANTS / GOTCHAS + +- SOURCE OF TRUTH IS THE AUDIT LOG, NOT THE LIVE ROW. Every gated write (mutateState/mutateAdjective/mutateOperational/mutateProvenance/expungeGated/reanchorGated) updates the projection column AND appends the sealed AuditEvent inside ONE `storage.transaction(isolation: .serializable)`. Never split these two writes across transactions except via the documented sealAudit:false + sealExpungeAudit(_:) two-step (GLK §B-2a orchestration only). +- The state field (adjectiveBitmap bits 0-5) is VERB-DRIVEN ONLY. `gatedColumnWrite` (whole-column field-edit path) explicitly excludes it. State only ever moves through `mutateState`, which validates `(from, via, to)` against SubstrateLib.RowStateAutomaton BEFORE any write. +- DrawerStore.addDrawer is `internal`, not `public`. The only sanctioned add path is `Estate.addDrawerCovered` (private in EstateVerbs.swift), which bundles the row insert with `containerFP.orIn` — this is what makes "captured a drawer without updating its container fingerprint" structurally impossible. Tests reaching `store.addDrawer` directly via `@testable import` must document why fingerprint coverage is not needed for that path. +- ContainerFingerprintStore has NO clear side by design. Withdrawing/expunging a drawer never clears bits from the room/wing OR-aggregate — a stale set bit only causes an unnecessary scan, never a missed match. `rebuildRoom`/`rollUpWing`/`rebuildAll` periodically tighten; `Estate.open` calls `rebuildAll` once so the aggregate is sound from the first recall. +- Expunge's destruction contract is STRONGER than the state machine. In `expungeGated`, every lineage sibling has its content column unconditionally zeroed even when the gate refuses that sibling's state transition (e.g. an `accepted` row, forbidden from reaching `.tombstoned` by S-3). Leaving verbatim content behind because a state flip was refused would be a privacy violation. +- Tombstone exclusion in `BitmapEvaluator.evaluateBitmapTier` is UNCONDITIONAL and independent of the caller's filter chain — `state == 33` (raw State.tombstoned) never surfaces through recall no matter what filters are supplied. +- `recall(_:)` NEVER THROWS (spec §7.8.1). Internal read/eval failures are surfaced exclusively via `RecallStream.degradedStages` (named stage strings). An empty `rows` with empty `degradedStages` means genuinely no matches; an empty `rows` with a populated `degradedStages` means a failed internal read. +- PersistenceKit v1.0 has NO nested transactions. This is why `gatedCaptureBody` exists as a separable @Sendable closure builder — `addDrawerWithCascade` needs the drawer INSERT and the supersedes-tunnel INSERT in ONE outer transaction, and cannot call `gatedCapture` (which opens its own transaction) from inside another transaction. +- `findActivePredecessor` predicate is `g_state_cluster < 3`; `livingSuccessorInLineage` predicate is `g_state_cluster < RowState.activeClusterUpperBoundRaw (16)`. These are DIFFERENT thresholds for different purposes (supersession-cascade target vs revive-guard living-successor check) — do not conflate them. +- Corpus scans (`allDrawers`, `drawersIn`, etc.) use `decodeDrawerRowsSkipCorrupt`: a `.corruptStoredValue` on one row is logged and skipped, everything else re-thrown. Point lookups (`getDrawer(id:)`, `getDrawers(ids:)`) use the strict `decodeDrawerRows`, which throws on the first corrupt row. Do not swap these two decode paths between call sites. +- `allDrawers(hydrationLevel:limit:direction:)` sorts by compound key `(filedAt, id)`, NOT SQLite `rowid` — `id` is the declared TEXT primary key, present and portable across SQLite/PostgreSQL/InMemory. DESC is the exact byte-for-byte reverse of ASC for any fixed dataset because of this deterministic tie-break. +- `RecallFrame.traceLimit` defaults to `nil`, which writes ZERO recall-trace rows. Only the GLK RecallDirector's primary locus call sets a non-nil traceLimit. Any new caller of `recall(_:)` that leaves traceLimit nil is correct by default; do not add trace writes to a scan-style caller without deliberate reason (write amplification). +- Bitmap layout constants in `BitmapEvaluator.swift` (adjStateMask, provConfirmMask, opChannelMask, etc.) are a PRIVATE MIRROR of the accessor decoders in Adjectives.swift/Provenance.swift/DrawerOperational.swift. A schema bump (`bitmap_layout_version`) that moves a field must update BOTH the accessor and this mirror in the same change. +- `@guardian-pair` comments (Adjectives.swift, DrawerStore.swift mutateState/expungeGated) mark raw-value sets that MUST stay in sync with SubstrateLib's AuditGate.basis legalValues — a build-time Guardian tool checks these; GuardianPairParityTests is the CI backstop. Touching a State/Trust/AdjectiveSensitivity/AdjectiveExportability case requires checking every paired comment. +- Nine nouns share the three-Int64-bitmap pattern (adjective/operational/provenance) EXCEPT: `Tunnel`/`Association`/`LearnedReference` have no `provenance`-axis adjective sharing beyond the bitmap slot layout itself being noun-specific; `Node` has no bitmap columns at all (lifecycle is a plain Int, not a bitmap). `Association`, `Proposal`, `LearnedReference`, `SourceCatalogEntry` all require a non-empty `latticeAnchor.udcCode` at insert (I-16) — `KGFact` and `Tunnel` predate I-16 and carry no anchor requirement. +- `Association`/`Proposal`/`LearnedReference`/`SourceCatalogEntry` are `Equatable, Codable, Sendable` but deliberately NOT `Hashable` (the embedded `LatticeAnchor` isn't Hashable). Do not add `Hashable` conformance without also widening `LatticeAnchor` — nothing today keys a Set/Dictionary on any of these four types. +- `Node.normalizeLookupName(_:)` and `DrawerFingerprint`'s four-block derivation are CONFORMANCE-GATED: the Rust port must produce byte-identical output for identical input. Changing either requires mirroring the change in rust/src/ and passing the corresponding conformance test. +- Pinned/fixed constants — changing any requires a coordinated Swift+Rust update and, where noted, a schema version bump: `LocusKitSchema.version` (8), `Estate.expectedBitmapLayoutVersion` ("v1.0"), `EstateVerbs.recallCandidateCap` (256), `RecallStream.defaultPageSize` (50), `RecallTraceItem.flagUsed`/`Tunnel.isRetiredBit`(bit 13)/`Tunnel.isDreamedBit`(bit 0), `NodeStore` depth ceiling (2), `outlineAncestors` depth ceiling (256), `NodeBundleStore` count-vector width (256 × UInt32 = 1024 bytes), `defaultWingName`("Agentic Memory")/`hintRoom`/`hintUDCCode`/`hintAddedBy`. +- LocusKit ships NO pinned data artifacts (no Resources/ directory of the LatticeLib kind) — its schema is generated fresh from `LocusKitSchema.schema` at every open; the only external pinned artifact it consumes is LatticeLib's Q-ID closure snapshot, reached read-only via `QIDClosure.ancestors(of:)`. diff --git a/packages/kits/LocusKit/docs/DETAILS.md b/packages/kits/LocusKit/docs/DETAILS.md new file mode 100644 index 0000000..5d1d1c2 --- /dev/null +++ b/packages/kits/LocusKit/docs/DETAILS.md @@ -0,0 +1,1315 @@ +--- +doc: DETAILS +package: LocusKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/LocusKit/Adjectives.swift + blob: d95d3ad8b02c61b166be790138b710b595f30c3d + - path: Sources/LocusKit/Association.swift + blob: 4b6f442c58dd6f182b3f5273348aa892eb1a5c55 + - path: Sources/LocusKit/AssociationOperational.swift + blob: 3228571f3f04aa662fa45c3ea098a97e0cd6cc96 + - path: Sources/LocusKit/AuditTypes.swift + blob: 872b7b19dc652f11b695361101d026b31a1a581d + - path: Sources/LocusKit/BitmapEvaluator.swift + blob: f2e25b7a0037131f4ea230bff62d33601a4af1f5 + - path: Sources/LocusKit/BitmapOps.swift + blob: 44dc29d22b712eb8631e07531e0bd3398d367197 + - path: Sources/LocusKit/BundleMaterializer.swift + blob: 59eae4efba9df236c0a7953df2b9649053e8133f + - path: Sources/LocusKit/ContainerFingerprintStore.swift + blob: fa7cb95c176cae53f0f24546112d87701ac1cf55 + - path: Sources/LocusKit/DefaultWings.swift + blob: a3469fce1fab0754cd379aa2e3742562dbc01717 + - path: Sources/LocusKit/DiaryEntry.swift + blob: 7b6d202251630f26b4a88110ecf4c91a5712c0b9 + - path: Sources/LocusKit/DiaryOperational.swift + blob: a5dc4ac124da1217fbaa716a63a194005d70523f + - path: Sources/LocusKit/Drawer.swift + blob: cc75fb3b8bc2450614dae65fb6d976d3f025cf5b + - path: Sources/LocusKit/DrawerFingerprint.swift + blob: 30bf6bf545f1699c1a5f7f31053ce77e65508b06 + - path: Sources/LocusKit/DrawerOperational.swift + blob: 26588caeb8b69f8ebb9973eac35216b957a1461c + - path: Sources/LocusKit/DrawerStateValidator.swift + blob: 0fcc6d5affd39342d2263c841c5fd67894b7f163 + - path: Sources/LocusKit/DrawerStore.swift + blob: 4379bc3c3477c9ac0a29c533d517fc8a67fbf184 + - path: Sources/LocusKit/Estate.swift + blob: 668ad245361b96317e5fa9ad9a35ccc897c2c138 + - path: Sources/LocusKit/EstateAudit.swift + blob: f17f901ea714fee7b7c2a6b8dc3c18e6684586d8 + - path: Sources/LocusKit/EstateIdentityKeyStore.swift + blob: 6a0cfc4dcc0c539c5ad0b5af4400dc0e0ee532e6 + - path: Sources/LocusKit/EstateTypes.swift + blob: 9afca8d07eee38ac8f0ad12fcb26aec4f732c1e8 + - path: Sources/LocusKit/EstateVerbs.swift + blob: ae30cf0be732d3b68bdb94e0193fd37171def37b + - path: Sources/LocusKit/Filter.swift + blob: 4260cd963863bbc6f31f87d7f037e92ae7d95e16 + - path: Sources/LocusKit/Fingerprint256Adapters.swift + blob: a7db452b9cc676b4af53a1c0f78c98d15df9c6c7 + - path: Sources/LocusKit/ForbiddenCombinationValidator.swift + blob: 809c5fdf1e7a6bcc219b9410f6f247eec65b275d + - path: Sources/LocusKit/Frames.swift + blob: fea5a8d8a487cc28ef967941eab8929b1d2b44df + - path: Sources/LocusKit/KGFact.swift + blob: 3d018021557fa23cdce8aa803cd99ce6be109ccd + - path: Sources/LocusKit/KGFactOperational.swift + blob: 635c214307e50e506ee9263fb7847cbcbbc2e165 + - path: Sources/LocusKit/LearnedReference.swift + blob: 9425889adb88a2a6348d134c0acbd6f1103cf8b8 + - path: Sources/LocusKit/LearnedReferenceOperational.swift + blob: c4a4154017577072f258996ca74c9d240cec7fcf + - path: Sources/LocusKit/LocusKit.swift + blob: f201f00a27e2b906a0fc1502bfdef55282844119 + - path: Sources/LocusKit/LocusKitError.swift + blob: cf9ea82e072c56c69fb2e88a62c1835391a3768e + - path: Sources/LocusKit/LocusKitSchema.swift + blob: f672290df86b6c6df6ca5d243a20e62a0ca1c023 + - path: Sources/LocusKit/LocusKitTelemetry.swift + blob: 732f995c52e8af16477f34243b5d898080cec7da + - path: Sources/LocusKit/LocusKitVocabulary.swift + blob: 8d207c675d5487931127b5fb34380a832df2c025 + - path: Sources/LocusKit/Manifest.swift + blob: 206d4ce171a49d96512f68e6398746edcc2ea13a + - path: Sources/LocusKit/MerkleRollup.swift + blob: ad7b32eb7d9ed1938e1c88c8abfedecb40500562 + - path: Sources/LocusKit/Node.swift + blob: 8d59f6721beca08b957c60eddc96c67df9cb39bb + - path: Sources/LocusKit/NodeBundleStore.swift + blob: f24b4bd9fa5b7ef0f4c63b3e9f066c484af17e56 + - path: Sources/LocusKit/NodeStore.swift + blob: 5fe41a84a4594365c16fe8616df24eb828092dbe + - path: Sources/LocusKit/Proposal.swift + blob: e92d684ddb11ec7d9babfc577f9b68d7730c0ec2 + - path: Sources/LocusKit/ProposalOperational.swift + blob: 65477d9fcafe5bdd9e53927833a08b27f6779247 + - path: Sources/LocusKit/Provenance.swift + blob: 95d989f9cf53084c29e2c9d506c2f18f6fd9bb07 + - path: Sources/LocusKit/RecallStream.swift + blob: bb038171818090f8b7b7db0cb39d25d4563ed9e1 + - path: Sources/LocusKit/RecallTraceItem.swift + blob: 1a96a2daf6dc5edb46b8ac6934b6f8670227a3dd + - path: Sources/LocusKit/SourceCatalogEntry.swift + blob: b91b6b61cb59150270488c7451bb8556449a3bc1 + - path: Sources/LocusKit/Summaries.swift + blob: fa624f6b00d8bf40f1ccc02821e3e2b94bf94f6e + - path: Sources/LocusKit/Tunnel.swift + blob: 8da22988147aab0a9451f06ec681d10891da1e95 + - path: Sources/LocusKit/TunnelOperational.swift + blob: a5b9b8d3b5b7990ebcb9d9c6209564a7307aa13e +--- + +# LocusKit Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in +pipeline order: the module surface, the estate lifecycle, the nine +verbs and their frame types, the storage engine, the drawer value type +and its shared bitmap machinery, the recall filter and evaluator, the +remaining eight noun types and their bitmap decoders, the containment +tree and bundle algebra, fingerprint and integrity machinery, manifest +and provisioning data, the recall-result and trace types, and finally +telemetry. + +## LocusKit.swift + +This file provides the module surface: a doc comment describing the +package and pointing to the files that carry the real public types. +LocusKit has no module-level namespace enum of its own (unlike some +sibling libs) — every public type lives in its own file, and this file +exists only to orient a reader arriving at the package for the first +time. + +## LocusKitError.swift + +This file provides `LocusKitError`, the single error enum every +LocusKit method throws. Centralizing every failure mode in one type +lets a caller recover specifically — for example, treating a missing +drawer as a routine miss while still propagating a genuine SQLite +failure. + +The cases split into three groups. Not-found cases +(`drawerNotFound`, `tunnelNotFound`, `diaryEntryNotFound`, +`recallTraceItemNotFound`) name a missing row by id. Storage cases +(`databaseUnavailable`, `sqliteError`, `schemaTooNew`, +`corruptStoredValue`) describe the storage layer misbehaving; +`corruptStoredValue` in particular names the table, column, and raw +stored text, so a caller can locate and repair the offending row +rather than receive a fabricated default value. Domain cases +(`invalidContent`, `disciplineViolation`, `notSupported`) describe +LocusKit's own validation refusing a write: `invalidContent` carries a +human-readable rule violation (a validation-test contract — tests +assert on the message text), and `disciplineViolation` names an +illegal state transition or forbidden bitmap combination using the raw +integer state values rather than the `State` enum, so the error type +does not need to depend on `Adjectives.swift`. + +`LocusKitError.description` renders each case as plain English without +Swift's `EnumCase(...)` noise. This description is consumed at the +GeniusLocusKit boundary, where a caller parses the English text back +out to decide how to react to a rejected write — so the wording is a +soft contract, not just a log message. + +## LocusKitSchema.swift + +This file provides `LocusKitSchema`, the entire on-disk schema declared +as data rather than as SQL text. `Storage.open(schema:)`, a +PersistenceKit primitive, reads this declaration and creates every +table, generated column, and index from it. No file in LocusKit issues +raw `CREATE TABLE` or `CREATE INDEX` strings; the previous +hand-rolled-SQL implementation was replaced by this declarative +approach so that the schema is portable across SQLite, PostgreSQL, and +an in-memory test backend without three copies of the same DDL. + +`LocusKitSchema.schema` assembles seventeen tables: `drawers`, +`tunnels`, `diary`, `manifest`, `kg_facts`, `proposals`, `associations`, +`learnedReferences`, `source_catalog`, `node_bundles`, +`container_fingerprints`, `recall_trace`, `keys`, `nodes`, plus three +tables owned by sibling schema fragments (`ErasureLedgerSchema`, +`SnapshotSchema`). Every persistent noun table carries a nullable +`.json` column named `ext` — a forward-compatible extension slot that +absorbs an unforeseen future field without a schema migration; version +1.0 writes `NULL` into it and never reads it. `LocusKitSchema.version` +is currently 8; the file's header comment documents each version bump +(a BLOB Merkle root, a nullable `content_hash` column, the `nodes` +table, and so on) and states plainly that there is no incremental +migration ladder, because no estate data has shipped under an earlier +version. + +The file's most distinctive feature is its **generated columns** — for +example `g_state_cluster` on `drawers`, computed as +`adjectiveBitmap & 0x3F`. A generated column is a column whose value +SQLite (or the equivalent backend) derives automatically from an +expression over other columns in the same row, so it can be indexed +like an ordinary column without the application computing and storing +it by hand. `LocusKitSchema.indices` then declares an ordinary index on +each generated column, which turns what used to be a hand-written +`CREATE INDEX ... ON drawers (provenance & 0xF)` functional index into +a declarative, backend-portable statement. The file's header comment +also documents a **bitmap reservation map**: which bit ranges of each +Int64 bitmap column are assigned and which are free headroom, so +adding a future flag is a matter of claiming a documented free bit +rather than a schema migration. + +## LocusKitVocabulary.swift + +This file provides `LocusKitVocabulary`, LocusKit's contribution to the +write-gate's vocabulary — the set of legal values every bitmap field +may take. A vocabulary is the fixed rulebook the write gate checks +every proposed value against; a value outside the vocabulary is +refused before it ever reaches storage. + +`LocusKitVocabulary.unionSlots` declares one `FieldSlot` per +operational and provenance field LocusKit defines (capture channel, +content kind, feature flags, source type, and so on), each naming its +bitmap column, bit position, width, and (for enumerated fields) its +legal raw values. The adjective axis's vocabulary — state, sensitivity, +exportability, trust — is supplied by the write gate itself, from +SubstrateLib, not by this file; `unionSlots` is purely LocusKit's own +consumer-side additions. `LocusKitVocabulary.frozen()` compiles this +set into a `Vocabulary` value once, at estate open, via +`VocabularyValidator.freeze`; the resulting frozen vocabulary is what +every subsequent write in the estate's lifetime is checked against, and +it never changes after that first freeze. + +## EstateTypes.swift + +This file provides five small supporting types used across the estate +surface: `RowID` (a plain `String` type alias naming a row's stable +identifier), `FrameFilteredDrawers` (the result of a frame-aware +by-id load, pairing the frame-admissible drawers with the full set of +ids that physically loaded), `OwnerCredentials` (the estate owner's +identifier, validated non-empty at open/create time), `LatticeAnchor` +(the four-field classification anchor every drawer carries — a UDC +code plus optional Wikidata enrichment), and `EstateError` (the error +enum for `Estate` lifecycle failures, distinct from `LocusKitError`). + +`FrameFilteredDrawers.loadedIDs` is reported independently of +`admissible` for a specific reason: a caller needs to distinguish "this +id loaded but failed the frame's filter" (a legitimate drop) from "this +id never loaded" (a transient or partial read that must be treated as +degraded, not silently dropped). `LatticeAnchor.udc(_:)` is a +convenience constructor for the common case of a bare UDC code with no +Wikidata enrichment — the shape most content has before an enrichment +daemon has run over it. `EstateError` covers three failure shapes: +`substrateUnavailable` (the backing storage failed to open), +`manifestMismatch` (a stored manifest value does not match what this +build of the kit expects — most importantly the bitmap layout +version), and `emptyOwnerIdentifier` (raised before any storage call, +so a caller gets a structurally distinct error rather than a generic +one), plus `keychainError` for a failing Keychain call during identity +key handling. + +## Estate.swift + +This file provides `Estate`, the single public entry point to a +LocusKit-backed memory store. An `Estate` is an actor — a Swift type +that serializes access to its own state, so concurrent callers can +never interleave two operations against the same estate unsafely. It +owns one `DrawerStore` (the storage engine, described below), one +`ContainerFingerprintStore` (the recall-pruning aggregates), and one +`NodeStore` (the containment tree), and it is the only public surface +most callers should ever touch. + +`Estate.open(storage:owner:identityKeyStore:)` opens an existing +estate. It refuses to open a database whose manifest carries a +different `bitmap_layout_version` than this build expects — bitmap bit +positions are part of the durable on-disk contract, so opening a +database written by an incompatible future schema would silently +misread every bitmap field. On first open it also mints a fresh +Curve25519 Ed25519 keypair for the estate's federation identity: the +public half is written to the manifest (safe, because a public key has +no confidentiality requirement), and the private half is handed to an +injected `EstateIdentityKeyStore` — the Keychain in production, an +in-memory store in tests — and cached for the lifetime of the `Estate` +instance. `Estate.create(storage:owner:manifest:)` is the sibling +constructor for a brand-new estate; it does not mint the identity +keypair, because that only happens on the first `open` after creation. + +The bulk of the file is a set of typed pass-through reads — +`allDrawers`, `getDrawers(ids:)`, `getDrawers(ids:matchingFrame:hydrationLevel:)`, +`hydrateBodies`, `tunnelsFromWing`, `recentRecallTraces`, +`allTunnels`/`allActiveTunnels`, `retireTunnel`/`unretireTunnel`, +`pruneRecallTraces`, `markRecallTracesUsed`, `countRecallTraces`, +`countDrawerRows`, `allProposals`/`allAssociations`/ +`allLearnedReferences`/`allKGFacts`/`allDiaryEntries`, and +`resolveNodeNames` — that exist so higher kits (chiefly +GeniusLocusKit) never need to import or construct a `DrawerStore` +directly. `getDrawers(ids:matchingFrame:hydrationLevel:)` is the most +elaborate of these: it runs the exact same frame-filtering pipeline +`recall` uses, but over a caller-supplied id set rather than a full +corpus scan, which is the shape a dense-first candidate-pool search +needs. `meta(key:)` and `setMeta(key:value:)` expose the estate +manifest as a general per-estate key-value store for higher kits that +need durable state of their own, on the condition that they namespace +their keys to avoid colliding with LocusKit's own typed manifest keys. + +## EstateAudit.swift + +This file provides `Estate`'s audit and history API: `auditTrail(rowID:)` +and `bitmapState(rowID:asOf:)`. Both delegate to the row's sealed audit +log rather than any separate history table, because the audit log — +not the live row — is the estate's source of truth for what happened +and when. + +`auditTrail(rowID:)` returns every sealed `AuditEvent` for a row in +HLC order (an HLC, or hybrid logical clock, is a timestamp format that +orders events consistently even across independently-clocked +machines). `bitmapState(rowID:asOf:)` reconstructs what a row's three +bitmaps looked like at an earlier point in time by folding its audit +log forward from the beginning up to (and including) the requested HLC, +via SubstrateLib's `AuditLogFold.projectStateAt`. This is what lets a +caller ask "what did this drawer's state look like a week ago" without +LocusKit maintaining a separate append-only history table alongside the +live row — the live row is redundant with the log by construction, and +the log is the only place that needs to remember the past. + +## EstateIdentityKeyStore.swift + +This file provides the protocol and two implementations for persisting +an estate's Ed25519 private signing key outside the manifest table. +The manifest table is ordinary, unencrypted metadata readable by +anyone with database or backup access, so the private half of the +estate's federation identity keypair must never be written there — +only the public half is safe to store in the manifest. + +`EstateIdentityKeyStore` declares two methods, `loadPrivateKey` and +`storePrivateKey`, both keyed by the estate's UUID. +`KeychainEstateIdentityKeyStore` is the production implementation: it +stores the raw key bytes as a `kSecClassGenericPassword` Keychain item, +accessible after the first device unlock following a restart +(`kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly`) but never synced +to iCloud Keychain — matching the same device-bound posture as the +estate's own SQLite file. `InMemoryEstateIdentityKeyStore` is the test +double: a plain dictionary guarded by a lock, with a `_storedPrivateKey` +accessor so a test can assert what was persisted without touching the +real Keychain (which would require entitlements and would pollute +state across test runs). + +## EstateVerbs.swift + +This file provides the nine verb methods — `capture`, `captureBatch`, +`recall`, `withdraw`, `expunge`, `mutate`, `reanchor`, `propose`, +`learn`, `associate` — as an extension on `Estate`. It is declared as a +separate file specifically so it can reach `Estate.store`, which is +declared `internal` rather than `private` in `Estate.swift` for exactly +this reason. + +`capture(_:CaptureFrame)` is the drawer entry point. It validates that +content, room, lattice-anchor UDC code, actor, and embedding-model id +are all non-empty, assembles the three bitmap columns from the frame's +named fields via `BitField.writeField` (never hand-rolled shift/mask +arithmetic), resolves the target wing and room to containment-tree node +ids (creating them on demand), and writes the drawer through +`addDrawerCovered` — the one sanctioned internal chokepoint that +bundles the row insert with the container-fingerprint update, so a +drawer can never be captured without its fingerprint aggregate being +updated in the same call. `captureBatch(_:)` is the bulk-import +sibling: it resolves all wing/room node ids up front with a per-call +cache, batches every fresh (non-superseding) insert into one storage +transaction via `DrawerStore.insertFreshBatch`, and defers the Merkle +rollup entirely — turning a 40,000-drawer import from roughly 34 +minutes of per-row commits into roughly 30 seconds. + +`capture(_:TunnelCaptureFrame)` is the tunnel entry point — `capture` +is legal on exactly two nouns, drawer and tunnel, and this overload +handles the second. `recall(_:RecallFrame)` is the read path: it reads +the live (non-tombstoned) candidate rows via `liveRows`, runs them +through `BitmapEvaluator.evaluate`, optionally writes a bounded number +of recall-trace rows (only when the caller opts in via +`frame.traceLimit`, so internal and bulk-export scans never accumulate +trace rows), and returns a `RecallStream`. Both `liveRows` and the +evaluation step surface any internal read failure as a **named +degraded stage** on the returned stream rather than silently returning +an empty result — this is the mechanism by which a caller can tell a +genuinely empty estate from a recall that failed partway through. + +`withdraw(rowID:reason:)` moves a drawer's state to `.withdrawn` via +`mutateState`, which is the only correct path because a state +transition must go through the automaton's legality check. +`expunge(rowID:reason:confirmation:now:)` is the destructive delete: +it requires an explicit `confirmation: true`, walks the drawer's full +lineage chain (every version sharing its `lineageID`) so a hard delete +scrubs every historical version, and always seals a sealed audit event +recording the fact of the expunge even though the content itself is +gone. `expungeReturningUnsealedEvent` and the paired +`sealExpungeAudit`/`sealExpungeOrphanAudit` methods exist for +GeniusLocusKit's two-step orchestration, where a cross-kit vector +delete must happen between the storage tombstone and the audit seal; +splitting the seal out prevents an arbitrary caller from suppressing +the audit event by accident. + +`mutate(rowID:kind:payload:)` dispatches on `MutationKind` to move a +drawer along one of three axes: confirmation (via `mutateProvenance`), +state (via `mutateState`, with the caller-facing guards for `.resolve`, +`.accept`, and `.revive` implemented here before the store's automaton +check even runs, so the error message is clearer), or an adjective +field (sensitivity, trust, or exportability, via `mutateAdjective`). +The `.revive` case is the most elaborate: it implements the full +per-source-state legality table from the cookbook's revive rule, +including the "living successor" lineage check that refuses to revive +a superseded row while a later version in the same lineage is still +alive. `reanchor(rowID:toRoom:toWing:toLattice:)` moves a drawer's room, +wing, and/or lattice position without touching its bitmaps. +`propose(_:now:)` and `associate(_:now:)` build a `Proposal` or +`Association` from their respective frames, deriving the lattice anchor +from the target row (for propose) or from the first endpoint (for +associate) rather than ever fabricating one. `learn(_:now:)` catalogs +the frame's source (once, keyed by handle) and writes a +`LearnedReference` anchored to that catalog entry's genuine lattice +position — never a sentinel anchor invented from a bare handle. + +`seedWing(_:hint:addedBy:embeddingModelID:now:)` files one ordinary, +fully recallable drawer per default wing at estate provision time, +giving a fresh agent a plain-language description of what each wing is +for. + +## Frames.swift + +This file provides the eight "frame" structs that carry a verb's +arguments: `CaptureFrame`, `TunnelCaptureFrame`, `RecallFrame`, +`LearnFrame`, `ProposeFrame`, `AssociateFrame`, plus the supporting +enums `MutationKind`, `HydrationLevel`, and `Ordering`. Every field on +every frame is named after a domain concern (a capture channel, a +sensitivity tier, a filter chain) — no raw bitmap value or bit position +crosses the public verb boundary anywhere in this file. + +`CaptureFrame` carries every named axis a captured drawer's three +bitmaps encode: channel, sensitivity, kind, and the matching +provenance-side channel, source type, sensitivity, confirmation, and +confidence, plus lineage, room, lattice anchor, actor, embedding model, +optional event time (for backdated bulk ingestion), feature flags, +exportability, and an optional wing. Nearly every field defaults to the +value that produces the same all-zero bitmap a caller got before that +field existed, so adding a new capture-time axis is source-compatible +with every existing caller. `RecallFrame` carries the filter chain, +hydration level, page-size limit, ordering, an optional historical +`asOf` HLC, and `traceLimit` (the opt-in recall-trace write count). +`MutationKind` enumerates the mutate verb's seven cases — `.confirm`, +`.reject`, `.contest`, `.resolve`, `.supersede`, `.revive`, `.accept`, +and the three `.correctSensitivity`/`.correctTrust`/`.correctExportability` +cases carrying their new value. `HydrationLevel` names the three +read-cost tiers — `.bitmapOnly`, `.structured`, `.full` — and +`Ordering` names the three result orderings recall supports; a +relevance ordering is deliberately absent, because LocusKit has no +scoring signal of its own (that lives in VectorKit, composed on top by +GeniusLocusKit), and the file's comment explains that shipping a +`.byRelevanceDesc` case here without a real relevance signal would be +an honesty violation. + +## DrawerStore.swift + +This file provides `DrawerStore`, the actor that owns every table in +the schema and implements the full CRUD and mutation surface for all +nine nouns. It is the largest file in the package and the place where +almost every write in the estate ultimately lands. + +`DrawerStore.init(storage:hlc:)` opens the schema (idempotently — a +second open on an existing database is a no-op for tables and +indices), freezes the write-gate vocabulary once, populates the v1 +manifest defaults (writing each key only if absent, so a first-open +value is never overwritten by a later open), and classifies the +manifest's `estate_uuid` value into one of three outcomes: absent +(a genuinely fresh estate, resolved to a freshly minted UUID), present +and well-formed (the normal case), or present but unparseable — which +throws `corruptStoredValue` rather than silently fabricating a random +replacement UUID, because conflating corruption with a fresh estate +would mask real data loss. The HLC clock's node id is derived from that +same classified value by hashing the raw stored text, so a standalone +estate gets a stable, estate-specific clock identity that a Rust port +of the same estate would derive identically. + +The **gated write path** is the file's central pattern, used by +`addDrawer`, `mutateProvenance`, `mutateAdjective`, `mutateOperational`, +`mutateState`, `expungeGated`, and `reanchorGated`. Each reads the +row's current bitmaps inside a serializable transaction, decomposes the +proposed new value into per-field `FieldWrite`s (never a whole-column +replacement — this is what lets the gate validate each field +independently and check the forbidden-combination invariant on the +merged result), calls `AuditGate.admit` (a SubstrateLib primitive), +and — on success — writes both the merged projection column and the +sealed `AuditEvent` in the same transaction. `gatedColumnWrite` is the +shared helper behind the three whole-column mutators; it excludes the +adjective bitmap's state field from the field list, because state is +verb-driven and must never move through a field edit. `mutateState` +additionally validates that the verb (`RowVerb`) and the caller's +requested target state agree with what the automaton's transition +table produces, so an illegal `(from, via, to)` triple is rejected +before any row is touched. + +`addDrawer` implements the **supersession cascade**: when the new +drawer's `lineageID` matches an active predecessor +(`findActivePredecessor`, an indexed query on the generated state-cluster +column), `addDrawerWithCascade` inserts the successor and a +`.supersedes` tunnel in one transaction (so a failed tunnel insert +never leaves an orphaned successor row), then separately flips the +predecessor's state to `.superseded` through the normal gated +`mutateState` path. `insertFreshBatch` is the bulk-import sibling used +by `EstateVerbs.captureBatch`: every drawer in the batch (all of which +the caller has pre-verified have no active predecessor) is inserted and +gated inside one single transaction, eliminating per-row commit +overhead. + +`expungeGated` is the lineage-wide hard delete: it tombstones the +target drawer through the gate, zeroes its content, sets the +`dreaming_recalc_required` obligation flag, records it in the erasure +ledger, and then repeats a content scrub (and, where the state +transition is legal, a full gated tombstone) across every other member +of the lineage chain. A sibling whose state cannot legally transition +to tombstoned (an audit-grade `accepted` row, forbidden by invariant +S-3) still has its content unconditionally zeroed — the destruction +contract is stronger than the state machine, because leaving verbatim +content behind when the gate refuses the state flip would be a privacy +violation. `sealExpungeAudit`, `sealExpungeOrphanAudit`, and +`sealExpungeOrphanForSweep` support GeniusLocusKit's two-step orphan +recovery when a crash separates the storage tombstone from its audit +seal; `tombstonedRowsWithoutExpungeAudit` is the query the recovery +sweep uses to find rows stuck in that crash window. + +The remainder of the file is CRUD for the other eight nouns +(`addTunnel`/`getTunnel`/`tunnelsFrom`/`tunnelsTo`/`allTunnels`, the +tunnel-retirement pair `retireTunnel`/`unretireTunnel`, the outline +helpers `outlineChildren`/`outlineAncestors`/`reparentDrawer`, +`addKGFact`/`getKGFact`/`kgFacts(forDrawerID:)`/`withdrawKGFact`, +`addProposal`/`getProposal`/`proposals(forTargetRowID:)`, +`addAssociation`/`getAssociation`/`associationsFrom`/`associationsTo`, +`addLearnedReference`/`getLearnedReference`/`learnedReferences(forSourceCatalogID:)`, +`addSourceCatalogEntry`/`getSourceCatalogEntry`/`sourceCatalogEntry(forHandle:)`, +`addDiaryEntry`/`getDiaryEntry`/`readDiary`), the recall-trace surface +(`insertRecallTrace`/`insertRecallTraces`/`recentRecallTraces`/ +`markRecallTraceUsed`/`markRecallTracesUsed`/`pruneRecallTraces`/ +`countRecallTraces`), the manifest surface (`setMeta`/`getMeta`/ +`readManifest`), the node-tree name-resolution helpers +(`roomNodeIdsInWing`/`roomNodeId`/`resolveNodeNames`), the summary +surface (`listWings`/`listRooms`/`taxonomy`), and the temporal-read +surface (`fingerprintsCaptured(in:)`/`fingerprintBitSeries`, which +feed the FFT-based rhythm-spectrum and moment-summary lenses built on +top of LocusKit). Every row-decode function follows the same +resilience pattern: `drawerFromRow` and its siblings throw +`corruptStoredValue` for an unambiguously malformed stored value (a +non-UUID `lineageID`, an unparseable timestamp), while corpus-scan +callers route through `decodeDrawerRowsSkipCorrupt`, which logs and +skips a corrupt row rather than aborting an entire estate-wide scan — +point lookups fail loud, corpus scans degrade gracefully. + +## Drawer.swift + +This file provides `Drawer`, the value type behind verbatim content — +the thing callers file and the thing recall returns. Its `content` +field is preserved unchanged: no truncation, no normalization, because +MemPalace's verbatim-first principle requires that retrieval surface +exactly what was filed. + +Every drawer references its containing room by `parentNodeId`, a +foreign key into the `nodes` table, rather than storing wing and room +names directly — display names are resolved separately via +`DrawerStore.resolveNodeNames`. `filedAt` and `eventTime` are two +distinct clocks: `filedAt` is when the row entered the local store +(the ingest clock, monotonic, the anchor for audit ordering), while +`eventTime` is when the content actually happened or was authored in +the world — for streaming capture the two coincide, but a bulk +historical importer supplies a real, possibly much earlier, `eventTime`. +The three bitmap fields (`provenance`, `adjectiveBitmap`, +`operationalBitmap`) all default to `0`, and the four lattice fields +(`udcCode`, `udcFacets`, `wikidataQID`, `wikidataQidsSecondary`) locate +the drawer in the classification lattice — `udcCode` is required +non-empty at every capture path per invariant I-5. `Drawer`'s custom +`Codable` conformance backfills a missing `eventTime` to `filedAt` on +decode, so a row encoded before the `eventTime` column existed still +decodes to a sensible value. + +## DrawerFingerprint.swift + +This file provides the derivation of a drawer's 256-bit structural +fingerprint — the coordinate system LocusKit uses for structural +similarity and for recall pruning. A fingerprint is built from four +64-bit SimHash blocks, each a projection of one facet of the row +through its own family of random hyperplanes: bitmap (the three +bitmap columns), lattice (UDC prefix, direct Q-ID, and Q-ID +closure), lineage-and-temporal (lineage hash and capture week), and +channel-and-source (channel, source type, capture channel, sensitivity, +and an estate identity hash). + +`EstateFingerprintFamilies` derives the four hyperplane families for +one estate from its UUID, once, at estate open; two independently +started replicas of the same estate — same UUID — always derive the +identical families and therefore the identical fingerprint for +identical drawer content, which is the determinism property recall +pruning depends on. `EstateFingerprintFamilies.fingerprint(of:)` is the +per-drawer entry point: it resolves the drawer's Wikidata Q-ID (if any) +through `LatticeLib.QIDClosure` to get its full ancestor chain, folds +that chain into a 16-bit hash for the lattice block's `qidClosureHash` +slot, and hands all four assembled inputs to `SubstrateLib.SimHash`. A +drawer carrying no lineage-clustering, defer-pattern, or +stream-source data — fields that only apply to a different noun type, +`AmbientSample` — supplies the deterministic null value zero for those +sub-fields, per invariant I-17, which keeps Hamming distance +well-defined across every noun type sharing the same fingerprint shape. + +## DrawerOperational.swift + +This file provides the three named axes packed into +`Drawer.operationalBitmap`: `CaptureChannel` (how the content entered +the system — typed, voiced, OCR, imported file, sensor, or actuator), +`ContentKind` (the shape of the content — prose, code, transcript, +list, structured JSON, image caption, or fingerprint-only), and +`DrawerFeatureFlags` (a non-exclusive bitset of seven named flags: +attachments, voice, image, links, pinned, keystone, and locked-zone). +Because feature flags are a bitset rather than an exclusive choice, +`DrawerFeatureFlags` is declared as an `OptionSet` rather than an enum, +unlike the other two axes in this file. + +Each axis has a computed accessor on `Drawer` — `captureChannel`, +`contentKind`, `featureFlags`, `hasFeatureFlag(_:)`, +`stateExtensionActive`, and `lineageClusteringActive` — that decodes +the relevant bit range and falls back to a safe neutral default (typed +input, prose content) for any raw value this build does not recognize, +so a row written by a future version with a case this build has never +heard of degrades gracefully instead of crashing. + +## DrawerStateValidator.swift + +This file provides `DrawerStateValidator`, a thin bridge from +LocusKit's `State` enum to SubstrateLib's canonical `RowStateAutomaton` +— the single implementation of the row-state legal-transition table +that every LocusKit noun with a state axis is validated against. +`DrawerStateValidator` used to carry its own parallel transition table; +that table permitted four transitions the specification actually +forbids (including `accepted → tombstoned`), so it was retired in +favor of consuming SubstrateLib directly. + +`validate(from:to:via:)` is the transition-legality-only overload, +used by `StateTransitionTests`: it bridges `State` to `RowState`, +looks up the legal target for `(from, via)` in the automaton, and +throws `LocusKitError.disciplineViolation` naming the rule violated if +either the `(from, via)` pair has no legal transition or the caller's +requested `to` disagrees with what the automaton actually produces. +`validate(from:to:via:targetingFields:)` is the overload +`DrawerStore.mutateState` actually calls: it additionally checks +field-level invariants (S-1: an accepted row requires trust at or +above canonical; S-2 and other bitmap-combination rules) against the +post-write bitmap fields, in the same call. + +## Adjectives.swift + +This file provides the four cross-noun adjective axes — `State`, +`Trust`, `AdjectiveSensitivity`, `AdjectiveExportability` — packed into +the low bits of every noun's `adjectiveBitmap`, plus their computed +accessors on `Drawer`. These four enums are the single source of truth +for these axes across every LocusKit consumer (GeniusLocusKit, +NeuronKit, and so on); layers beneath LocusKit cannot import them (the +dependency graph points the other way), so they carry the raw integer +encoding instead and a build-time "Guardian" tool checks that the two +sides never silently drift apart. + +`State`'s ten cases partition into three clusters at raw-value +boundaries 16 and 32, chosen specifically so the cluster of a state can +be computed as `(state >> 4) & 0x3` in one shift-and-mask: Cluster A +("currently believed" — active, pending, contested, accepted), Cluster +B ("knew past" — superseded, decayed, withdrawn, expired), and Cluster +C (terminal — rejected, tombstoned). `Drawer.isCurrentlyBelieved`, +`isKnewPast`, and `isTerminal` expose these three clusters as +predicates. `Trust` orders seven levels from `.verbatim` (unqualified, +as-filed content) through `.canonical` up to `.ambient`; it conforms to +`Comparable` so a filter like "trust at least canonical" reads as an +ordinary comparison rather than raw-value arithmetic. `AdjectiveSensitivity` +and `AdjectiveExportability` are both scale-gapped (their case raw +values skip numbers, leaving room for future intermediate tiers without +disturbing any existing bitmask). `AdjectiveSensitivity`'s three +`ADR-007`-driven predicates — `isBulkExportable`, `requiresOwnerKeyForBulk`, +`isExcludedFromBulk` — are the enforcement hooks VaultKit's bulk-export +path consults to decide whether a drawer may ride a bulk channel +without additional friction. `Drawer.dreamingRecalcRequired` and +`Drawer.sealed` decode two single-bit obligation/trust-hint flags +living above the four main axes. + +## Provenance.swift + +This file provides the seven named axes packed into `Drawer.provenance` +— `SourceType`, `Channel`, a mirrored `CaptureChannel` reference, +`Confirmation`, `Confidence`, `Sensitivity`, and `EnrichmentStatus` — +plus their computed accessors on `Drawer`. Where the adjective bitmap +records a row's *current standing* and the operational bitmap records +*mechanical facts about the content*, the provenance bitmap records +*how the row came into being and how it has been reviewed since*. + +`SourceType` names ten origins (user, observed, imported, canonical, +derived, federation-aggregate, tier-aggregate, paired-estate, ambient, +actuator). `Channel` names the system surface content arrived through +(UI typed or voiced input, an MCP agent, a file import, federation, or +one of three dreaming-daemon channels). `Confirmation` is the review +axis — unconfirmed through user-, automated-, peer-, or +actuator-confirmed — and `isUserConfirmed` is the convenience predicate +retrieval layers use to surface only user-vetted content. +`Confidence` and `Sensitivity` are both scale-gapped, `Confidence` +ordering from `.null` to `.verified` and `Sensitivity` deliberately +mirroring the raw values of `AdjectiveSensitivity` so the two axes — +sensitivity *at capture* versus the estate's current, possibly +since-mutated, access posture — can be compared directly. +`EnrichmentStatus` tracks a Q-ID enrichment daemon's lifecycle for a +row, ending either at `.qidCompleted` or, when deterministic +re-inference fails, at the terminal in-workflow state `.qidProposed`. + +## BitmapOps.swift + +This file provides three small, `@inlinable` field-extraction helpers +— `andMask`, `thresholdCompare`, `shiftExtract` — that `BitmapEvaluator` +uses to translate a `Filter` case into a check against a packed Int64 +bitmap. The file is explicit that these are not math primitives: they +carry no algorithm to prove and no platform-specific optimization to +gate, unlike the substrate primitives (SimHash, Hamming distance, +OR-reduction) that live in SubstrateLib and participate in the +cross-platform conformance gate. Each function delegates its actual +bit manipulation to a SubstrateLib primitive (`BitField.maskedEquals`, +`BitField.extractField`, `BitField.popcount`) so LocusKit never +hand-rolls a shift-and-mask sequence that could silently diverge from +the Rust port's equivalent. + +`andMask(_:mask:expected:)` tests whether a field equals an expected, +already-shifted value in one masked comparison. +`thresholdCompare(_:mask:shift:op:value:)` extracts a field and +compares it against a threshold using one of three orderings +(`.lessThan`, `.lessThanOrEqual`, `.greaterThanOrEqual`) — the +mechanism behind cluster-boundary filters such as "trust below the +action threshold." `shiftExtract(_:shift:mask:)` returns a field's raw +integer value, used wherever the evaluator needs the value itself +rather than a yes/no comparison. + +## ForbiddenCombinationValidator.swift + +This file provides `ForbiddenCombinationValidator`, documenting (and, +where a live call site exists, enforcing) the one constitutional +forbidden bitmap combination: a row must never carry adjective +sensitivity `.secret` together with adjective exportability +`.public_`. Storage can physically represent that combination — nothing +stops the bits from being set — but the verb layer must refuse to +produce it. The file notes that current write paths route this check +through `AuditGate`/SubstrateLib directly, so this validator itself has +no live call site today; it exists as an explicit, reviewable statement +of the rule, ready to be wired in wherever a future write path bypasses +the gate. + +`validate(_:)` extracts the sensitivity field (bits 6–11) and the +exportability field (bits 12–17) from a full `adjectiveBitmap` value +and throws `LocusKitError.disciplineViolation` if both match their +forbidden raw values (48 and 32, respectively) — hand-derived from the +enum cases' shipped raw values rather than imported, specifically so a +future rename of an enum case cannot silently change what this +validator checks. + +## Filter.swift + +This file provides `Filter`, the named recall-filter algebra, plus its +five small supporting type aliases (`LineageID`, `RoomID`, `WingID`, +`WikidataQID`, `ProvenanceChannel`, `FeatureFlag`) and the +`StateCluster` enum. No `Filter` case takes a raw bit position, mask, +or threshold integer — every case names a domain concern directly +(`.trustworthy`, `.inRoom(_:)`, `.contentMatches(_:)`), and +`BitmapEvaluator` is the sole place that translates a `Filter` into the +bitmap primitives internally. + +The cases group by concern: state queries (`.currentlyBelieve`, +`.usedToBelieve`, `.knewOnceAndErased`, `.state(_:)`, +`.stateInCluster(_:)`), trust queries (`.trustworthy`, +`.requiresConfirmation`, `.trust(_:)`, `.trustAtMost(_:)`), sensitivity +and exportability queries, provenance queries (confirmation, source +type, channel, confidence), operational queries (capture channel, +content kind, feature flags), structural queries (room, wing, lineage, +time bounds, lattice anchor and prefix, Wikidata concept), the single +content query `.contentMatches(_:)`, and three composition cases — +`.all`, `.any`, `.not` — that let filters nest arbitrarily. A +`RecallFrame.filterChain`, a plain array of `Filter`, is interpreted as +an implicit `.all` — every filter in the chain must pass. + +## BitmapEvaluator.swift + +This file provides `BitmapEvaluator`, the compiler and interpreter for +a `RecallFrame.filterChain` — the heart of the recall pipeline. It is +declared `internal` rather than `public` because it takes a +`DrawerStore` argument, itself internal; public callers reach it only +indirectly through `Estate.recall`. + +`evaluate(frame:drawers:store:nodeNames:)` runs four stages against the +candidate row set its caller has already narrowed down (via +`Estate.liveRows`, which applies fingerprint pruning ahead of this +call). **Default insertion** first prepends implicit filters for any +concern the caller left unconstrained: state defaults to +`.currentlyBelieve`, trust to `.trustworthy`, and sensitivity to +`.sensitivityAtMost(.elevated)` — the Normal-tier ceiling per ADR-007 — +so recall with an empty filter chain returns only currently-believed, +trustworthy, normal-or-elevated-sensitivity content by default, without +a caller needing to spell any of that out. **Bitmap-tier evaluation** +then compiles each `Filter` case to a check over the row's three raw +bitmaps using the `BitmapOps.swift` primitives, with tombstone exclusion +enforced unconditionally and independent of the caller's chain — a +tombstoned row can never surface through recall no matter what the +chain says. When `frame.asOf` is set, each row's bitmaps are +reconstructed as they stood at that historical HLC via +`AuditLogFold.projectStateAt` before the bitmap tier runs. **Structured-tier** +evaluation then applies room, wing, lineage, time-bound, and lattice +filters that need the drawer's non-bitmap fields (and, for room/wing, +a resolved node-name lookup). **Content-tier** evaluation applies +`.contentMatches` via a case-insensitive substring search over the +verbatim content. A final **ordering** pass sorts the survivors. + +`chainHasPrunableFilter`, `chainHasContentPredicate`, and +`chainHasStructuredNameFilter` are classifiers `Estate.liveRows` and +`Estate.getDrawers(ids:matchingFrame:hydrationLevel:)` consult before +even fetching rows: the first decides whether fingerprint pruning can +apply at all, the second decides whether the fetch needs to pay for the +content blob, and the third decides whether a node-name lookup is +needed. `containerSurvives(chain:fingerprint:)` is the actual pruning +predicate: it returns false only when the chain provably cannot be +satisfied by any row in a container, which is sound only for +set-bit filters like `.hasFeatureFlag` — a threshold or exact-value +filter can never be decided from an OR-reduced fingerprint alone, so +those filters never prune a container, they only ever get evaluated +per-row. + +## Tunnel.swift + +This file provides `Tunnel`, the typed cross-reference between two +locations in the estate. A tunnel is stored directionally — source +endpoint to target endpoint, each carrying a wing, a room, and an +optional specific drawer id (nil meaning "the room itself") — so a +query asking "what does this side know about" never needs to scan both +directions. `kind`, a `TunnelKind`, is the closed, indexed relationship +vocabulary the retrieval layer dispatches on; `label` is a free-form, +unvalidated human annotation. `orderKey` is a fractional-index sibling +ordering value used only by `.parent` tunnels in the outline graph +(ADR-017) — siblings under the same parent sort by ascending +`orderKey` without every sibling needing to be rewritten when one is +inserted between two existing siblings. + +## TunnelOperational.swift + +This file provides `TunnelKind` (the ten-case closed relationship +vocabulary — `.supersedes`, `.references`, `.blocks`, `.validates`, +`.contradicts`, `.derivesFrom`, `.covers`, `.elaborates`, `.respondsTo`, +and `.parent`, the outline-containment edge) and the four operational +axes packed into `Tunnel.operationalBitmap` — `TunnelDirection`, +`TunnelLifecycle`, `TunnelOriginClass`, `TunnelStrength` — plus their +accessors. + +Two bits above those four axes carry standalone boolean flags rather +than enumerated fields: `hasInverse` (whether a paired reverse tunnel +exists) and `isRetired` (bit 13, added for the T13/ADR-021 dreaming +retirement mechanism). `withRetired()` and `withUnretired()` return a +copy of a tunnel with that bit flipped — the reversible mechanism by +which the dreaming pipeline's OMEGA cycle suspends a tunnel from active +reads without tombstoning it, so a later co-recall can re-propose the +same pairing. A separate provenance-bitmap section adds `isDreamed` +(bit 0 of `provenanceBitmap`), set only for tunnels the dreaming +pipeline itself proposed and later accepted, never for a +user-explicit, imported, or federated tunnel — OMEGA's retirement +predicate requires `isDreamed == true`, so a declared tunnel is never +retired regardless of how much or little it is recalled. + +## DiaryEntry.swift + +This file provides `DiaryEntry`, the first-person record of what an +agent thought, did, or learned at a point in time. A diary entry is +stored alongside drawers but queried separately, keyed by `agentName` +and an unvalidated `topic` tag, so one agent's diary never leaks into +another's wing-filtered recall by accident (a convention, not an +enforced constraint — `wing`/`room` are plain required strings the +caller supplies). `reward` and `rewardProvenance` are the explicit +quality-signal channel: when a caller has a direct quality score for an +entry (a user rating, a model confidence value), it is recorded here +directly rather than left for the dreaming daemon's implicit +recall-based reward inference to guess at later. + +## DiaryOperational.swift + +This file provides the four axes packed into +`DiaryEntry.operationalBitmap` — `DiaryEventClass` (twelve cases: what +kind of substrate event this entry records — capture, mutation, +withdraw, expunge, propose, associate, learn, signal emission, +maintenance, migration, training, or audit-tombstone), +`DiarySeverity` (scale-gapped: trace, info, warning, error), +`DiaryActorClass` (who produced the entry: user, substrate daemon, MCP +agent, migration tool, or federation peer), and +`DiaryBatchMembership` (standalone, batch-start, batch-member, or +batch-end) — plus the single `requiresFollowup` flag bit, and their +computed accessors. + +## KGFact.swift + +This file provides `KGFact`, a subject-predicate-object triple +distilled from a verbatim drawer, retaining `sourceDrawerID` as a +backreference so a fact's provenance is always recoverable. `subject`, +`predicate`, and `object` are all free-form strings at this rung — no +entity vocabulary is enforced by the value type itself; that +enforcement, when it arrives, belongs to a later federated +knowledge-graph layer. `sourceDrawerID` may legitimately be the empty +string, an "unanchored fact" sentinel used when a caller asserts a +freestanding triple not extracted from any specific drawer. + +`KGFact` reuses the same three-bitmap pattern every other bitmap-backed +noun uses, and its adjective accessors (`state`, `adjectiveSensitivity`, +`exportability`, `trust`) decode the identical bit layout `Drawer` +uses — a fact and its source drawer can be filtered by the same +retrieval predicates because they share the same encoding. + +## KGFactOperational.swift + +This file provides the four axes plus one flag packed into +`KGFact.operationalBitmap`: `KGExtractorClass` (six cases ordered +roughly by rigor — manual, foundation-model, specialized-model, +rules-based, imported-KG, federated), `KGAssertionKind` (asserted, +inferred, hypothesized, or contradicted — the last recording that +another fact disputes this one without either being retracted, with +resolution deferred to retrieval time), `KGSpecificity` and +`KGConfidenceBand` (both scale-gapped and `Comparable`, letting a +filter read as `fact.specificity >= .specific`), and the `isCanonical` +flag marking a fact promoted to estate-wide, load-bearing status. + +## Proposal.swift + +This file provides `Proposal`, a suggested change awaiting +confirmation — the durable record produced by the substrate's only +autonomous write surface. `Proposal` structurally mirrors `KGFact` with +one addition `KGFact` predates: a required `latticeAnchor`, because +`KGFact` was written before the cookbook universalised the +every-row-has-an-anchor invariant. `candidateState` is the adjective +set the proposal would apply to its target if accepted — the accept +path reads this value to know what to write. `Proposal.state` decodes +the proposal's own lifecycle position (pending while awaiting +confirmation, then accepted, rejected, or withdrawn) from its +adjective bitmap's state field, the same field `Drawer.state` decodes. + +## ProposalOperational.swift + +This file provides the five typed axes packed into +`Proposal.operationalBitmap`: `ProposalKind` (nine cases — what kind of +write this proposal proposes, from `.newTunnel` and `.mutateDrawer` +through the newer `.actionProposal`, `.recordObservation`, and +`.tierAdvisory`), `ProposalTargetObjectType` (which kind of row this +proposal targets, including the `.noneBrandNew` sentinel for a +not-yet-existing target and `.systemState` for a proposal about the +estate itself), `ProposalConfirmationSource` (human, agent, automated +threshold, or actuator), `ProposalGeneratedByClass` (dreaming daemon, +MCP agent, federation sync, manual, or tier aggregator), and +`ProposalConfidenceBucket` (scale-gapped and `Comparable`). This is the +one operational layout the file notes is directly specified by the +engineering cookbook rather than being LocusKit-internal, so a +conformance test pins every field position and raw value against the +cookbook table directly. +`composeOperational(kind:targetObjectType:generatedBy:confidence:)` +assembles a full operational bitmap from four of the five axes in one +call (confirmation source is left at its default until a confirmation +step actually runs), used by autonomic daemon sinks that need to stamp +genuine provenance on the proposals they emit. + +## Association.swift + +This file provides `Association`, the graph edge recording that two +rows belong together — a statistical or dreaming-derived pairing, +distinct from a `Tunnel`'s typed semantic claim. `Association` +structurally mirrors `Tunnel` (source and target endpoints, three +bitmap columns, the Rev 1.0 soft-delete reservation) with two +deliberate differences: it carries no `kind` column — an association's +semantics live entirely in its operational bitmap — and it carries a +required `latticeAnchor`, anchored to the lattice midpoint of its two +endpoints, which `Tunnel` predates and therefore lacks. + +## AssociationOperational.swift + +This file provides the three axes packed into +`Association.operationalBitmap`: `AssociationSignalSources` (a +twelve-bit bitset — an `OptionSet`, not an enum, because more than one +signal can independently contribute evidence for the same pairing: a +co-recall signal, a co-confirmation signal, a dream-pairing signal, a +vector-similarity signal, a shared-entity signal, an explicit-human +signal, a fingerprint-similarity signal, and three v0.36 additions — +cross-estate, cross-tier, and action-outcome), `AssociationDecayClass` +(scale-gapped and `Comparable`: pinned, slow, normal, fast — how +quickly an association ages out of relevance), and `AssociationArity` +(binary today; n-ary reserved for a future version, per invariant +I-23's current binary-only limit). + +## LearnedReference.swift + +This file provides `LearnedReference`, the durable record of an +external reference the grounding-driven `learn` verb brought into the +estate. It structurally mirrors `Association` — an id, content +columns, a required `latticeAnchor`, three bitmap columns, and the +soft-delete reservation — chosen as the template specifically because +`Association` is the freshest content-bearing noun that already +honours the anchor requirement. `sourceCatalogID` is a foreign-key +reference to the `SourceCatalogEntry` this reference was learned from, +stored as an identifier string the same way `KGFact` stores +`sourceDrawerID` rather than embedding the full catalog entry inline. +`handle` is the reference's own locator, distinct from the source's own +locator on the catalog entry it points into. + +## LearnedReferenceOperational.swift + +This file provides the four axes packed into +`LearnedReference.operationalBitmap`: `RefreshPolicy` (scale-gapped and +`Comparable`, ordering none through monthly, weekly, daily, on-demand, +and realtime), `DriftSeverity` (scale-gapped and `Comparable`: none, +minor, major, critical — how far a reference has drifted from its +source since it was last re-grounded), `LearnMode` (a single bit: +whether the reference is held by pointer, `.byReference`, or its +content was ingested wholesale, `.byIngestion`), and +`LearnedReferenceSource` (where the reference was acquired from — user, +federation, household pairing, fleet pairing, tier inheritance, or +paired estate — distinct from `sourceCatalogID`, which names *which* +catalog entry, not *how* the reference arrived). + +## SourceCatalogEntry.swift + +This file provides `SourceCatalogEntry`, the durable, queryable record +of an external source from which references are learned — the +substrate behind the `source` slot of the `learn` verb. Its reason for +existing is grounding integrity: every `LearnedReference` must carry a +genuine lattice anchor, never a fabricated sentinel, and an anchor is a +property of the source (a web domain, a document corpus, a paired +estate), not of each individual handle learned from it. Cataloging a +source once and having every reference from it inherit that catalog +entry's anchor is what makes a genuine anchor available without +inventing one per reference. `SourceKind` names six source classes +(user, federation, household pairing, fleet pairing, tier inheritance, +paired estate) — deliberately the same vocabulary +`LearnedReferenceSource` uses, because the acquisition channel a +reference was learned through is the same vocabulary as the kind of +source it came from. + +## Node.swift + +This file provides `Node`, a container node in the estate's containment +tree: the estate root at depth 0, a wing at depth 1, or a room at depth +2. Every node carries two name fields — `displayName`, which preserves +whatever casing the first writer used, and `lookupName`, a normalized +form (Unicode NFC, trimmed, internal whitespace collapsed, casefolded) +used for resolution and uniqueness enforcement — so "Personal " and +"personal" resolve to the same node while the node's display name +still shows however it was first written. `lifecycle` (0 active, 1 +tombstoned) plus a pair of HLC-typed timestamps +(`createdHlc`/`tombstonedHlc`) let the containment tree support the +same as-of historical read surface as drawers, without needing +wall-clock timestamps to double as the ordering key. +`Node.normalizeLookupName(_:)` is the single, conformance-relevant +normalization function both the Swift and Rust ports must produce +byte-identical results from. + +## NodeStore.swift + +This file provides `NodeStore`, the actor that owns the containment +tree's storage and its create-on-demand resolution: given a display +name and a parent id, find the active node with that normalized lookup +name under that parent, or create one if none exists. `NodeStore` is +an actor specifically so this find-then-insert sequence is race-free +without needing an `INSERT OR IGNORE` or a conflict-column mechanism — +two concurrent requests to create the same wing under the same estate +serialize through the actor and produce exactly one node, because the +second request's `findActiveNode` call cannot run until the first +request's insert has completed. + +`createNode(displayName:parentId:now:)` enforces two structural +invariants before writing: the parent must already exist (I-NT-5), and +the new node's depth (parent's depth plus one) must not exceed 2 +(I-NT-2) — a room cannot have a child, because there is no fourth +tier. `createRoot(displayName:now:)` is the once-per-estate seed for +the depth-0 root, enforcing that exactly one root exists (I-NT-1) by +returning the existing root unchanged if one is already present. +Tombstoned nodes are invisible to resolution — a tombstoned wing does +not block, and cannot be accidentally reused by, a later request for a +wing with the same name (the no-resurrection guard, §5). + +## NodeBundleStore.swift + +This file provides `NodeBundleStore`, persistence for the bundle-algebra +count-vector aggregates — one row per node (room or wing) per bundle +kind, stored in the `node_bundles` table. A count-vector here is a +256-element array of counts, one per fingerprint bit, encoded as 1024 +bytes of little-endian `UInt32` values; `BundleKind.activeA` is the +"active centroid" (a fold of a node's currently active members) and +`BundleKind.departedB` is the "departed accumulator" (eager-folded at +departure time). The per-row drawer fingerprints that feed these +aggregates are never themselves stored — only the folded aggregate is, +which keeps the table small regardless of estate size. + +`put`/`get`/`rooms(forWing:kind:)` are the read/write surface; +`encodeCounts`/`decodeCounts` handle the wire format, with +`decodeCounts` throwing `LocusKitError.invalidContent` (rather than +trapping) when a stored blob is not exactly 1024 bytes, so a corrupt +row surfaces as a recoverable error instead of crashing the process. + +## BundleMaterializer.swift + +This file provides `BundleMaterializer`, the operation that actually +computes a Bundle A aggregate from live drawers — the first real +consumer of `SubstrateKernel.countFold256`. Bundle A cannot be +maintained incrementally (active membership changes over time, and the +fold operation has no subtraction), so it is fully recomputed on +demand, typically by a periodic dreaming tick. + +`materializeRoom(wing:room:now:)` fetches every non-tombstoned drawer +in a room, filters to `State.isClusterA` (currently believed) rows, +derives each survivor's `Fingerprint256` via `EstateFingerprintFamilies`, +folds the set with `kernel.countFold256`, and stores the result. +`rollUpWing(wing:now:)` merges a wing's already-materialized room +bundles into one wing-level bundle; because the count-vector fold is +associative, this merge produces the identical result as folding every +active drawer in the wing directly, so rooms may be materialized in any +order before the wing roll-up runs. + +## Fingerprint256Adapters.swift + +This file provides a small marshaling convention: packing a single +Int64 bitmap column into block 0 of a `Fingerprint256` (with blocks 1–3 +zeroed) so a per-column bit operation — an OR, an XOR, a popcount — can +be routed through a SubstrateLib primitive that operates at the +substrate's canonical 256-bit width, rather than LocusKit reimplementing +the same math at Int64 width. The file is explicit that this is purely +a type-shape convention with no algorithm of its own, and it warns that +this packing is not the same as the cookbook's Bitmap-LSH SimHash +interpretation of block 0 — a caller must not mix the two uses of the +same 64-bit lane. + +`init(int64Column:)` and the `int64Column` computed property are the +pack and unpack directions, used throughout `ContainerFingerprintStore`. + +## ContainerFingerprintStore.swift + +This file provides `ContainerFingerprintStore`, the per-container +OR-reduction aggregates that make fingerprint-based recall pruning +possible: for every room, and rolled up to every wing, the bitwise OR +of the three bitmap fields across every active drawer in that +container. Because OR is monotone, a container's aggregate can only +ever gain bits as content is added; a bit that is absent from the +aggregate is therefore provably absent from every row in the +container, which is exactly the soundness property recall pruning +needs — a filter requiring that bit can safely skip the whole +container without fetching a single row. + +`orIn(wing:room:adjective:operational:provenance:now:)` is the +incremental maintenance path, called on every drawer capture; it folds +the new drawer's bitmaps into both the room-level and the wing-level +row via `ContainerFingerprint.merging(_:)`, which itself routes through +`SubstrateLib.ORReduce.reduce` at canonical 256-bit width using the +Fingerprint256 packing convention above. The clear side is +deliberately absent — withdrawing or expunging a drawer never clears +bits from the aggregate, because a stale set bit is a harmless +over-approximation (it can only cause an unnecessary scan, never a +missed match), while `rebuildRoom`/`rollUpWing`/`rebuildAll` exist to +periodically tighten the aggregate back down from a fresh full scan. +`rebuildAll` is what `Estate.open` calls once at startup, so an +existing estate's aggregate is guaranteed complete and therefore sound +before any recall runs against it. + +## MerkleRollup.swift + +This file provides the Merkle content-integrity rollup: an extension on +`Estate` that computes a hash tree — room, then wing, then estate — +bottom-up over the same containment tree `NodeStore` maintains. A +Merkle tree is a hash tree where each parent's hash is computed from +its children's hashes, so a single top-level root hash can attest to +the exact contents of an entire subtree; changing any leaf changes +every hash on the path up to the root. The rollup is invoked +explicitly, not automatically after every write, because computing it +inline on every drawer capture would be O(room-size) per write and peg +the CPU during a bulk import. + +`computeRoomMerkleRoot(roomNodeId:)` hashes a room's live drawers, +reading each row's stored `content_hash` column when present (written +by a separate hash-on-write hook) or computing a leaf hash on demand +from the drawer's content when absent; it explicitly excludes both +tombstoned rows (irreversibly deleted) and withdrawn rows (state raw +value 18) from the snapshot, because a withdrawn drawer's content has +been retracted by the user and must not remain retrievable through a +content-attesting snapshot. `computeEstateOrWingMerkleRoot(parentNodeId:)` +hashes a set of children's already-computed roots — the same function +serves both the wing level (hashing room roots) and the estate level +(hashing wing roots). `rollupMerkleRoots(roomNodeId:now:)` runs all +three levels for one changed room; `recomputeAllMerkleRoots(now:)` is +the full bottom-up recompute used after bulk import, migration, or +corruption recovery; `rollupAllMerkleRoots(now:)` is a documented alias +of the full recompute, named for the batch-capture reindex path. +`createSnapshot(label:now:additionalAttestations:)` recomputes the +full tree (so a snapshot is never taken against stale roots), then +writes an attestation row for the estate root and every wing, plus any +attestations a higher composition-layer kit supplies. + +## Manifest.swift + +This file provides `ManifestKey`, the eighteen required and seven +optional typed keys of the v1 manifest key-value table, and +`ManifestValues`, the typed, read-only snapshot `DrawerStore.readManifest()` +produces. The manifest table itself is a plain key-value store; this +file is what turns "some strings in a table" into a fully typed record +a caller can rely on. `ManifestKey.ed25519PublicKey` and +`ed25519PrivateKeyWrapped` are the two identity-related keys: +`ed25519PublicKey` is the estate's federation public key, safe to store +here because a public key has no confidentiality requirement, while +`ed25519PrivateKeyWrapped` is a reserved, never-written seam kept only +for backward read-compatibility with an estate opened before the +Keychain migration — the private key itself lives exclusively in +`EstateIdentityKeyStore`. + +## DefaultWings.swift + +This file provides the seven default wing +definitions seeded into a fresh estate at provision time, plus three +related constants: `defaultWingName` ("Agentic Memory," the wing +`capture` uses when a caller supplies no explicit wing), `hintRoom` +("AI_Charter_Hint," the room each seeded wing's hint memory lives in), +and `hintUDCCode` ("001," the UDC Knowledge-class code stamped on hint +drawers). Each `WingDefinition` pairs a wing name with a plain-language +hint describing the wing's role — Agentic Memory, User Canon, Source +Corpus, Personal, Professional, Projects, and Temp — seeded as an +ordinary, fully recallable, user-deletable drawer so a fresh agent has +a working orientation to its own memory structure from the first +session. The set is explicitly a suggestion, not a fixed schema: an +agent may create any additional wing it needs. + +## RecallStream.swift + +This file provides `RecallStream`, the paged async sequence +`Estate.recall` returns. Iterating a `RecallStream` produces one +`RecallPage` at a time — the first page synchronously on the first +`next()` call, later pages lazily — with `isLast` true only on the +final page, so a caller can drive a uniform `for await` loop without +special-casing an empty result: a genuinely empty corpus still emits +exactly one page with zero rows and `isLast == true`. + +`degradedStages` is the channel by which a failed internal read (the +bounded corpus scan, the room-fingerprint enumeration, a surviving +room's drawer read, or the bitmap evaluator itself) becomes observable +to a caller. `recall` never throws — spec §7.8.1 requires it stay +non-throwing — so an internal failure would otherwise be +indistinguishable from a genuinely empty result; a non-empty +`degradedStages` array is what lets a caller (chiefly GeniusLocusKit's +`RecallDirector`) tell the two apart. `AsyncIterator.hydrate(_:)` +applies the requested `HydrationLevel` per page: `.bitmapOnly` rebuilds +each drawer with `content` blanked to the empty string while preserving +every bitmap and metadata field; `.structured` and `.full` pass rows +through unchanged, because the content-stripping decision for +`.structured` was already made upstream, at the storage-fetch layer, +not here. + +## RecallTraceItem.swift + +This file provides `RecallTraceItem`, one record of a single drawer +returned by a recall operation — the substrate for the "later +two-source reward" mechanism a dreaming daemon uses to distinguish rows +a user actually acted on from rows that were returned but ignored. The +`used` flag is bit 0 of `operationalBitmap`, decoded through a +computed property rather than stored as a separate `Bool` column, so +the type carries no stored boolean fields at all — a convention every +bitmap-backed LocusKit noun follows. `score`, when present, is the +recall's own similarity score for the row; `nil` means the recall that +produced this trace carried no score (for example, an ordinary +ordered-by-capture-time query). + +## AuditTypes.swift + +This file provides `BitmapState`, the snapshot of a row's three bitmap +columns at a specific historical HLC, returned by +`Estate.bitmapState(rowID:asOf:)`. It exists as a small, focused return +type precisely because the reconstruction that produces it — +`AuditLogFold.projectStateAt`, folding a row's sealed audit events +forward in HLC order — is a SubstrateLib primitive that returns its +result in a different shape; `BitmapState` is LocusKit's own +public-facing wrapper naming the three bitmaps explicitly rather than +exposing the substrate's internal projection type directly. + +## Summaries.swift + +This file provides `WingSummary` and `RoomSummary`, the two small +aggregate types `DrawerStore.listWings`/`listRooms` produce. Both are +computed projections over the current drawer set — LocusKit has no +separate `wings` or `rooms` table; wing and room identity lives +entirely in the `nodes` table, and a summary's drawer and room counts +are simply whatever the live query finds at the moment it runs, not a +maintained running total. + +## LocusKitTelemetry.swift + +This file provides LocusKit's opt-in +telemetry emission functions, wired through `IntellectusLib.Intellectus.report(_:)`. +Every emit function here follows the same three rules stated in the +file's header: the reported value is always an `@autoclosure` that is +never evaluated when monitoring is disabled (the off-path cost is one +atomic boolean load and a branch — no lock, no allocation); the `now` +timestamp is always caller-supplied, never read from a clock inside the +function, preserving IntellectusLib's own determinism contract; and the +metric namespace follows the fleet-wide `..` +convention (`locuskit.drawer.capture_latency_ms`, +`locuskit.kgfact.add_count`, `locuskit.gate.reject_count`, and so on). + +`emitDrawerCapture`, `emitDrawerQuery`, `emitKGFactAdd`, +`emitKGFactQuery`, `emitTunnelAdd`, `emitGateAdmit`, and +`emitGateReject` are called from the corresponding `DrawerStore` +methods at the exact moment each operation completes (or, for the gate +functions, at the exact moment `AuditGate.admit` returns). Because +telemetry is purely additive — it is never consulted by any control +flow that decides what a method returns — every store method's +functional behavior is byte-identical whether monitoring is enabled or +not. + +## Rust Port and Conformance + +The `rust/` directory contains the second leg of the kit: fifty-seven +source and test files under `rust/src/` and `rust/tests/` mirroring the +Swift implementation file-for-file (`drawer.rs`, `drawer_store.rs`, +`bitmap_evaluator.rs`, `merkle_rollup.rs`, `estate.rs`, and so on), +plus three concrete `DrawerStore` backends the Swift side does not +carry directly — `drawer_store_inmemory.rs`, `drawer_store_sqlite.rs`, +and `drawer_store_postgres.rs`, all built over a shared, storage-agnostic +`DrawerStoreCore`. Conformance tests under `rust/tests/` (adjective and +operational bitmap conformance, provenance bitmap conformance, corrupt +readback, Merkle rollup, outline proofs, recall pruning, temporal +reads, and the LP0 fixed-vector suite) gate byte-for-byte agreement with +the Swift implementation on every bitmap layout, every fingerprint +derivation, and every state-transition legality decision. When you +change a bitmap layout, a fingerprint derivation, or a transition rule +on either leg, mirror the change on the other and run both test suites +— the fixtures and conformance tests are the contract, not a +convenience. diff --git a/packages/kits/LocusKit/docs/OVERVIEW.md b/packages/kits/LocusKit/docs/OVERVIEW.md new file mode 100644 index 0000000..e7d4fc2 --- /dev/null +++ b/packages/kits/LocusKit/docs/OVERVIEW.md @@ -0,0 +1,304 @@ +--- +doc: OVERVIEW +package: LocusKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/LocusKit/Adjectives.swift + blob: d95d3ad8b02c61b166be790138b710b595f30c3d + - path: Sources/LocusKit/Association.swift + blob: 4b6f442c58dd6f182b3f5273348aa892eb1a5c55 + - path: Sources/LocusKit/AssociationOperational.swift + blob: 3228571f3f04aa662fa45c3ea098a97e0cd6cc96 + - path: Sources/LocusKit/AuditTypes.swift + blob: 872b7b19dc652f11b695361101d026b31a1a581d + - path: Sources/LocusKit/BitmapEvaluator.swift + blob: f2e25b7a0037131f4ea230bff62d33601a4af1f5 + - path: Sources/LocusKit/BitmapOps.swift + blob: 44dc29d22b712eb8631e07531e0bd3398d367197 + - path: Sources/LocusKit/BundleMaterializer.swift + blob: 59eae4efba9df236c0a7953df2b9649053e8133f + - path: Sources/LocusKit/ContainerFingerprintStore.swift + blob: fa7cb95c176cae53f0f24546112d87701ac1cf55 + - path: Sources/LocusKit/DefaultWings.swift + blob: a3469fce1fab0754cd379aa2e3742562dbc01717 + - path: Sources/LocusKit/DiaryEntry.swift + blob: 7b6d202251630f26b4a88110ecf4c91a5712c0b9 + - path: Sources/LocusKit/DiaryOperational.swift + blob: a5dc4ac124da1217fbaa716a63a194005d70523f + - path: Sources/LocusKit/Drawer.swift + blob: cc75fb3b8bc2450614dae65fb6d976d3f025cf5b + - path: Sources/LocusKit/DrawerFingerprint.swift + blob: 30bf6bf545f1699c1a5f7f31053ce77e65508b06 + - path: Sources/LocusKit/DrawerOperational.swift + blob: 26588caeb8b69f8ebb9973eac35216b957a1461c + - path: Sources/LocusKit/DrawerStateValidator.swift + blob: 0fcc6d5affd39342d2263c841c5fd67894b7f163 + - path: Sources/LocusKit/DrawerStore.swift + blob: 4379bc3c3477c9ac0a29c533d517fc8a67fbf184 + - path: Sources/LocusKit/Estate.swift + blob: 668ad245361b96317e5fa9ad9a35ccc897c2c138 + - path: Sources/LocusKit/EstateAudit.swift + blob: f17f901ea714fee7b7c2a6b8dc3c18e6684586d8 + - path: Sources/LocusKit/EstateIdentityKeyStore.swift + blob: 6a0cfc4dcc0c539c5ad0b5af4400dc0e0ee532e6 + - path: Sources/LocusKit/EstateTypes.swift + blob: 9afca8d07eee38ac8f0ad12fcb26aec4f732c1e8 + - path: Sources/LocusKit/EstateVerbs.swift + blob: ae30cf0be732d3b68bdb94e0193fd37171def37b + - path: Sources/LocusKit/Filter.swift + blob: 4260cd963863bbc6f31f87d7f037e92ae7d95e16 + - path: Sources/LocusKit/Fingerprint256Adapters.swift + blob: a7db452b9cc676b4af53a1c0f78c98d15df9c6c7 + - path: Sources/LocusKit/ForbiddenCombinationValidator.swift + blob: 809c5fdf1e7a6bcc219b9410f6f247eec65b275d + - path: Sources/LocusKit/Frames.swift + blob: fea5a8d8a487cc28ef967941eab8929b1d2b44df + - path: Sources/LocusKit/KGFact.swift + blob: 3d018021557fa23cdce8aa803cd99ce6be109ccd + - path: Sources/LocusKit/KGFactOperational.swift + blob: 635c214307e50e506ee9263fb7847cbcbbc2e165 + - path: Sources/LocusKit/LearnedReference.swift + blob: 9425889adb88a2a6348d134c0acbd6f1103cf8b8 + - path: Sources/LocusKit/LearnedReferenceOperational.swift + blob: c4a4154017577072f258996ca74c9d240cec7fcf + - path: Sources/LocusKit/LocusKit.swift + blob: f201f00a27e2b906a0fc1502bfdef55282844119 + - path: Sources/LocusKit/LocusKitError.swift + blob: cf9ea82e072c56c69fb2e88a62c1835391a3768e + - path: Sources/LocusKit/LocusKitSchema.swift + blob: f672290df86b6c6df6ca5d243a20e62a0ca1c023 + - path: Sources/LocusKit/LocusKitTelemetry.swift + blob: 732f995c52e8af16477f34243b5d898080cec7da + - path: Sources/LocusKit/LocusKitVocabulary.swift + blob: 8d207c675d5487931127b5fb34380a832df2c025 + - path: Sources/LocusKit/Manifest.swift + blob: 206d4ce171a49d96512f68e6398746edcc2ea13a + - path: Sources/LocusKit/MerkleRollup.swift + blob: ad7b32eb7d9ed1938e1c88c8abfedecb40500562 + - path: Sources/LocusKit/Node.swift + blob: 8d59f6721beca08b957c60eddc96c67df9cb39bb + - path: Sources/LocusKit/NodeBundleStore.swift + blob: f24b4bd9fa5b7ef0f4c63b3e9f066c484af17e56 + - path: Sources/LocusKit/NodeStore.swift + blob: 5fe41a84a4594365c16fe8616df24eb828092dbe + - path: Sources/LocusKit/Proposal.swift + blob: e92d684ddb11ec7d9babfc577f9b68d7730c0ec2 + - path: Sources/LocusKit/ProposalOperational.swift + blob: 65477d9fcafe5bdd9e53927833a08b27f6779247 + - path: Sources/LocusKit/Provenance.swift + blob: 95d989f9cf53084c29e2c9d506c2f18f6fd9bb07 + - path: Sources/LocusKit/RecallStream.swift + blob: bb038171818090f8b7b7db0cb39d25d4563ed9e1 + - path: Sources/LocusKit/RecallTraceItem.swift + blob: 1a96a2daf6dc5edb46b8ac6934b6f8670227a3dd + - path: Sources/LocusKit/SourceCatalogEntry.swift + blob: b91b6b61cb59150270488c7451bb8556449a3bc1 + - path: Sources/LocusKit/Summaries.swift + blob: fa624f6b00d8bf40f1ccc02821e3e2b94bf94f6e + - path: Sources/LocusKit/Tunnel.swift + blob: 8da22988147aab0a9451f06ec681d10891da1e95 + - path: Sources/LocusKit/TunnelOperational.swift + blob: a5b9b8d3b5b7990ebcb9d9c6209564a7307aa13e +--- + +# LocusKit Overview + +## What This Kit Does + +LocusKit is the storage substrate for a MOOTx01 estate. An estate is one +user's complete memory store in MOOTx01, an on-device AI memory system +that stores what an AI observes over time and helps the AI recall it +later. LocusKit is the layer that actually holds the memories on disk +and lets a caller file them, find them, and change their standing over +time. + +LocusKit is a kit, not a library. A library (lib) is a folder of related +code files that does one job well; a kit is a larger package that +composes libraries into a subsystem. Kits may depend on libs, but libs +never depend back on kits. LocusKit depends on several libs — most +importantly PersistenceKit for storage, SubstrateLib and SubstrateTypes +for the write-gate math, SubstrateML for classification math, and +LatticeLib for a taxonomic lookup — and composes them into the estate's +spatial memory surface. A higher kit, GeniusLocusKit, builds on top of +LocusKit; it is out of scope for this document. + +The word "MemPalace" names the spatial metaphor LocusKit implements. +Content lives in drawers, drawers sit in rooms, rooms sit in wings, and +wings sit in one estate. This document calls that three-level structure +the containment tree. + +## The Problem It Solves + +An AI's memory needs more than a place to put text. It needs to know +whether a memory is still believed, how sensitive it is, whether it can +leave the device, how much to trust it, and where it came from. It needs +to change its mind about a memory — mark it superseded, contested, +confirmed, or gone — without losing the history of that change. It needs +to find memories quickly by many different criteria without scanning +every row every time. And it needs all of this to survive a crash +midway through a write. + +LocusKit answers each of these needs with a specific mechanism: + +- **Standing.** Every memory carries three packed 64-bit numbers — + bitmaps — that record its adjective state (is it active, superseded, + contested, accepted, rejected, or gone), its operational facts (how it + was captured, what kind of content it is), and its provenance (where + it came from and how confident the system is in it). A bitmap is a + fixed-size number where different ranges of bits, called fields, each + carry an independent small value. Packing many fields into one number + keeps a row's standing in a single machine word instead of a dozen + separate boolean columns. +- **Safe change over time.** A memory's state does not move freely + from any value to any other. It follows a finite set of legal moves — + for example, an active memory can become contested, but a rejected + memory cannot become accepted. LocusKit routes every state-changing + write through a write gate (`AuditGate`, owned by SubstrateLib) that + checks the move is legal and then writes one sealed, tamper-evident + audit event recording exactly what changed. The audit event, not the + live row, is the source of truth; the live row is a cached projection + of the latest audit event for fast reads. Row and log update together + inside one transaction, so every value ever read back from a row + matches a genuine sealed event in its audit log — the row is never + observed one step ahead of its own history. +- **Fast, layered search.** A caller expresses what it wants as a + chain of named filters — "currently believed," "in this room," + "captured after this date" — rather than raw bitmask arithmetic. + LocusKit compiles that chain through four progressively more + expensive tiers: a bitmap tier (near-free integer comparisons), a + structured tier (room, wing, lattice checks), a content tier + (substring search in the verbatim text), and an ordering pass. Before + any of that, a per-container fingerprint check can rule out whole + rooms or wings without touching a single row. +- **Crash safety.** Every state-changing write happens inside one + database transaction that both updates the live row and appends the + audit event. If the process dies partway through, the transaction + either committed both changes or neither — never a live row that says + one thing and an audit log that says another. + +## How It Works + +### Nine kinds of rows, one set of patterns + +LocusKit stores nine kinds of rows, or nouns: `Drawer` (verbatim +content), `Tunnel` (a typed link between two locations), `DiaryEntry` +(a first-person agent record), `KGFact` (a subject-predicate-object +triple extracted from a drawer), `Proposal` (a suggested change awaiting +confirmation), `Association` (a graph edge recording that two rows +belong together), `LearnedReference` (an external reference brought in +by the `learn` verb), `SourceCatalogEntry` (the durable record of where +a learned reference came from), and `Node` (an entry in the containment +tree — the estate root, a wing, or a room). Most of these nouns share +the three-bitmap pattern described above; each has its own file +decoding its own bitmap layout (for example, `DrawerOperational.swift` +decodes `Drawer.operationalBitmap`, and `KGFactOperational.swift` +decodes `KGFact.operationalBitmap`). + +### The containment tree + +Estates used to store a drawer's wing and room as two plain text +columns. LocusKit now stores them as a three-level tree of `Node` rows: +the estate root at depth 0, wings at depth 1, and rooms at depth 2. A +drawer references its room by a foreign key, `parentNodeId`, rather than +by name. `NodeStore` resolves a wing/room name pair to a node id, +creating the node on first use and returning the existing one +thereafter — so filing a drawer into "Personal / Health" always lands in +the same room node, no matter how many times the name is used. This +containment tree is what a Merkle rollup (below) walks bottom-up, and +what lets a room be renamed without touching every drawer inside it. + +### The write gate + +`DrawerStore` is the actor that owns every table. Nearly every write +that changes a row's standing — capturing a drawer, moving its state, +editing an adjective field — is a "gated" write: `DrawerStore` reads the +row's current bitmaps, hands them to `AuditGate.admit` along with the +proposed change, and either receives back a sealed `AuditEvent` to +persist or a rejection naming the rule that was violated (an illegal +state transition, or a forbidden bitmap combination such as sensitivity +`secret` paired with exportability `public`). `DrawerStateValidator` and +`ForbiddenCombinationValidator` exist mainly as documentation of these +same rules, since the live enforcement now happens inside the gate. + +### The recall pipeline + +`Estate.recall` is the read side. A caller builds a `RecallFrame` +carrying a chain of `Filter` values — the recall filter algebra. No +`Filter` case exposes a raw bit position; each names a domain concern +("trustworthy," "in this wing," "captured after date X"). `Estate.recall` +first asks `ContainerFingerprintStore` whether any room or wing can be +ruled out from its cached per-container OR-fingerprint — a bitwise-OR +of every active drawer's three bitmaps in that container — before +fetching a single row. Surviving rows then pass through +`BitmapEvaluator`, which evaluates the filter chain in four tiers +(bitmap, structured, content, ordering) and can reconstruct a row's +historical bitmap state at a past point in time by folding its audit +log forward (`AuditLogFold.projectStateAt`, a SubstrateLib primitive). + +### Structural similarity and integrity + +Two more subsystems ride alongside the write/read core. `DrawerFingerprint` +derives a 256-bit structural fingerprint for a drawer from its bitmaps, +lattice anchor, lineage, and timing, using SubstrateLib's SimHash +machinery; these fingerprints feed both `ContainerFingerprintStore`'s +pruning aggregates and a separate bundle-algebra subsystem +(`NodeBundleStore`, `BundleMaterializer`) that folds many fingerprints +into one compact count-vector per room or wing. `MerkleRollup` computes +a content-integrity hash tree bottom-up over the same containment +tree — room, then wing, then estate — so a snapshot can attest that its +content has not silently changed. + +### The nine verbs + +`EstateVerbs.swift` adds nine verb methods to `Estate`: `capture` +(file a new drawer or tunnel), `recall` (the pipeline above), `mutate` +(move a row's standing along a named axis), `withdraw` (retract a +drawer), `expunge` (hard-delete a drawer's content while preserving the +audit trail), `reanchor` (move a drawer to a new room or lattice +position), `learn` (bring in an external reference, grounded to its +source's own lattice anchor), `propose` (record a suggested change +awaiting confirmation), and `associate` (record a graph edge between +two rows). Every verb takes a named "frame" struct (`CaptureFrame`, +`RecallFrame`, and so on) as its argument, so no raw bitmap value ever +crosses the public boundary. + +## How the Pieces Fit + +Figure 1 shows the kit's topology — its major parts and how a write and +a read move through them. + +![Figure 1. Topology of LocusKit](topology.svg) + +*Figure 1. Topology of LocusKit. A capture (left) flows through the +write gate into the drawers table and, off the write path, into the +container-fingerprint and Merkle-rollup subsystems. A recall (right) +flows through fingerprint pruning and the four-tier bitmap evaluator. +Dashed boxes mark external kits and libs LocusKit depends on but does +not own.* + +`Estate` is the single public entry point: it owns one `DrawerStore` +(the actor holding every table), one `ContainerFingerprintStore` (the +pruning aggregates), and one `NodeStore` (the containment tree), and it +exposes the nine verbs plus a set of read-only pass-throughs +(`allDrawers`, `allKGFacts`, `tunnelsFromWing`, and so on) that GLK and +other higher-level consumers use instead of reaching into `DrawerStore` +directly. `DrawerStore` itself is declared with an `internal`, not +`public`, add-path (`addDrawer`) so that the only sanctioned way to add +a drawer from outside the file is through `Estate.addDrawerCovered`, +which bundles the row insert with the container-fingerprint update — a +structural guarantee that a drawer can never be captured without also +updating its container's pruning aggregate. + +## What Ships in the Package + +The package ships the Swift sources under `Sources/LocusKit/` and a +parallel Rust port under `rust/`. The Swift package depends on five +sibling libs — `SubstrateLib`, `SubstrateTypes`, `SubstrateKernel`, and +`SubstrateML` for the write-gate and fingerprint math, `PersistenceKit` +for the storage abstraction, `IntellectusLib` for opt-in telemetry, and +`LatticeLib` for the pinned Q-ID ancestor closure `DrawerFingerprint` +hashes into its lattice block. LocusKit ships no pinned data artifacts +of its own; its schema (`LocusKitSchema`) is declared entirely in +PersistenceKit primitives and created fresh at estate open. diff --git a/packages/kits/LocusKit/docs/topology.svg b/packages/kits/LocusKit/docs/topology.svg new file mode 100644 index 0000000..49d80ed --- /dev/null +++ b/packages/kits/LocusKit/docs/topology.svg @@ -0,0 +1,129 @@ + + + + + + + + + + + + LocusKit: capture and recall through the estate facade + + + + Capture + Estate.capture + + + Recall + Estate.recall + + + + Estate + verb facade + (actor) + + + + NodeStore + containment tree + + + + DrawerStore + tables + write gate + + + + ContainerFingerprint + Store (pruning OR) + + + + BitmapEvaluator + 4-tier filter pipeline + + + + MerkleRollup + room → wing → estate + + + + + + + + + + + + + + + + + + + + External kits and libs (owned outside LocusKit) + + + AuditGate + SubstrateLib — legal-transition + + + PersistenceKit + Storage / RowStore / AuditLog + + + LatticeLib QIDClosure + pinned P31/P279 ancestors + + + + + + + every gated write validated here + every table's backend + read by DrawerFingerprint's lattice block + + Figure reads left to right: an input verb enters through Estate, which resolves + containment-tree nodes, writes through DrawerStore's audit-gated path (top), + or prunes and evaluates candidates for a read (bottom). Dashed edges cross a + kit/lib boundary LocusKit depends on but does not own. + diff --git a/packages/kits/VectorKit/docs/AGENT_MAP.md b/packages/kits/VectorKit/docs/AGENT_MAP.md new file mode 100644 index 0000000..b67a508 --- /dev/null +++ b/packages/kits/VectorKit/docs/AGENT_MAP.md @@ -0,0 +1,205 @@ +--- +doc: AGENT_MAP +package: VectorKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/VectorKit/EmbeddingProvider.swift + blob: ad2bf52732b46960b9357a01fea37254d1681561 + - path: Sources/VectorKit/Engine/BruteForceIndex.swift + blob: da3bdac5a6d2b84ff73e4ec66057bcc2acd2b2cb + - path: Sources/VectorKit/Engine/DenseHit.swift + blob: 22289f57f49923647e4b99092c134dbc1910c15e + - path: Sources/VectorKit/Engine/DenseIndex.swift + blob: 010a51a54b6d62c971943070115e8822d9ffeafb + - path: Sources/VectorKit/Engine/DenseMetric.swift + blob: a28578e73ec36943a73d067e7768782311fa2005 + - path: Sources/VectorKit/Engine/FloatBruteForceIndex.swift + blob: 888d5a4079c84939b2cfde93160ee6bc3851adeb + - path: Sources/VectorKit/Engine/MaxSimScorer.swift + blob: 91875a79b8f6eebf6a2fd0a3a9dde85311a50aae + - path: Sources/VectorKit/Engine/MIHIndex.swift + blob: 61e283122542218eaf1f057cd7b9f1022930956f + - path: Sources/VectorKit/Engine/ResidentArrayStore.swift + blob: 21c67979dfc05d761909edec9700849d7cad74a5 + - path: Sources/VectorKit/Engine/ResidentVectorArray.swift + blob: 6e0f689702e4173388324b22fd828559ce0b1ab2 + - path: Sources/VectorKit/Engine/VectorPayload.swift + blob: 9259b4db9380cf9d854abd84a1d5059a0fcff5ec + - path: Sources/VectorKit/Engine/VectorRecordKey.swift + blob: bb4fff18c74c37ccafc9b1eaa01a4ec86b80be20 + - path: Sources/VectorKit/FloatSimHashEmbeddingProvider.swift + blob: efcb85396ceace1373e3017c0f799371a9a5c3bf + - path: Sources/VectorKit/StoredVector.swift + blob: 44702eaf3a0e28ef7f70031fa05751b48a8ecfbf + - path: Sources/VectorKit/VectorKit.swift + blob: 0a6eba27a0501601ee9ac015875de6d71bd4cf05 + - path: Sources/VectorKit/VectorKitError.swift + blob: 89c486eba6992edd583649e37674a67ea95ee317 + - path: Sources/VectorKit/VectorMatch.swift + blob: 24cc2c1bd25f71a7cef60a043c3a640df2368a23 + - path: Sources/VectorKit/VectorStore.swift + blob: 3c7fe4a19eba1142ac82a993cee0e7660a4ffdce +--- + +# AGENT_MAP — VectorKit + +PURPOSE: on-device embedding generation (`EmbeddingProvider` seam) + model-tagged vector storage (`VectorStore`, PersistenceKit-backed) + dual-lane nearest-neighbour search: binary Hamming (Lane A `BruteForceIndex` oracle / Lane B `MIHIndex` sub-linear exact, promoted at `mihThreshold`) and float cosine/l2/dot (`FloatBruteForceIndex`, one index per modelID) + ColBERT MaxSim late-interaction scorer (`MaxSimScorer`, standalone, not wired into VectorStore). + +DEPS: imports EngramLib (Engram type, Hamming kernel via EngramLib.distances/Session — I-7 absolute), SubstrateML (FloatSimHash.project), SubstrateTypes, PersistenceKit (Storage/RowStore/BlobStore, product "PersistenceKit"), IntellectusLib (Intellectus.report telemetry, no-op when disabled). Test target additionally depends on PersistenceKitInMemory, PersistenceKitSQLite. Imported by: CorpusKit / CorpusKitProviders (concrete text embedding providers built on FloatSimHashEmbeddingProvider), GeniusLocusKit (destroyAllVectors as part of estate teardown). Rust port in rust/ mirrors every file (vector_store.rs, engine/{brute_force,mih,float_brute_force,max_sim,resident,resident_store,key,payload,hit,metric,seam}.rs, embedding_provider.rs, simhash_embedding_provider.rs, error.rs); no shared cross-language fixture file — conformance rests on both ports implementing the documented algorithms identically (colex enumeration, sidecar byte layout, budget arithmetic). Float lane is explicitly NOT four-way bit-identical (documented, not a gap). + +ENTRY POINTS (most callers need only these): +- VectorStore.swift:451 `VectorStore.addVector(itemID:engram:modelID:modelVersion:filedAt:)` — write one binary vector +- VectorStore.swift:1134 `VectorStore.findNearest(probe:modelID:limit:) -> [VectorMatch]` — binary Hamming k-NN +- VectorStore.swift:1223 `VectorStore.findNearestFloat(probe:modelID:limit:) -> [VectorMatch]` — float cosine k-NN +- FloatSimHashEmbeddingProvider.swift:63 `FloatSimHashEmbeddingProvider.embed(_:) -> Engram` — text → fingerprint via injected inference + FloatSimHash + +## Symbol Table + +### Module surface +- VectorKit.swift:1 — namespace/header only; no types. Consumers `import EngramLib` separately for `Engram` (not re-exported). + +### Errors — VectorKitError.swift +- :5 `enum VectorKitError: Error, Sendable, Equatable` — concrete cases, never optional+log +- :9 `.embeddingFailed(String)` / :13 `.modelUnavailable(String)` / :17 `.storeUnavailable(String)` / :21 `.notFound` (reserved, unused by current API) / :27 `.invalidPayload(String)` / :31 `.decodingFailure(String)` +- :41 `.int8QuantizationPolicyUndefined(String)` — thrown on every int8 write; policy unratified (VECTORKIT_SPEC §I-4a); remove guard+case only when ratified +- :56 `.embedFloatVocabMiss(String)` — distributional-provider OOV signal, distinct from embeddingFailed + +### Embedding seam — EmbeddingProvider.swift +- :15 `protocol EmbeddingProvider: Sendable` — modelID/modelVersion + embed/embedFloat/embedPair/embedBatch +- :37 `embed(_:) -> Engram` — MUST return `Engram.zero` for empty string (cross-provider contract, mirrored in Rust trait) +- :64 `embedFloat(_:) -> [Float]` — opt-in; default impl (:105) throws embeddingFailed; empty input → `[]` never zero-vector +- :81 `embedPair(_:) -> (engram, floats)` — default impl (:115): two-pass (embed then embedFloat), float opt-out swallowed to `[]` +- :93 `embedBatch(_:) -> [Engram]` — default impl (:123): sequential; override for batched inference + +### Concrete provider — FloatSimHashEmbeddingProvider.swift +- :35 `struct FloatSimHashEmbeddingProvider: EmbeddingProvider` — Swift mirror of Rust vectorkit::FloatSimHashEmbeddingProvider +- :44 `projectionSeed: UInt64` — distinct seeds ⇒ distinct fingerprints for same float vector (I-4 enforced at projection layer) +- :49 `inference: @Sendable (String) async throws -> [Float]` — host-injected; kit owns no tokenizer/model +- :63 `embed(_:)` — empty-string short-circuit BEFORE inference call, then `FloatSimHash.project(vector:seed:)` +- :87 `embedFloat(_:)` — returns the SAME vector embed() projects; no double inference + +### Engine foundation types (Lane F — additive-only, no local field additions) +- VectorRecordKey.swift:33 `struct VectorRecordKey: Sendable, Equatable, Hashable, Comparable` — (itemID, vectorIndex, modelID, modelVersion); ordering IS the partition/tie-break order +- VectorRecordKey.swift:87 `< (lhs:rhs:)` — lexicographic (itemID, vectorIndex, modelID, modelVersion); DO NOT reorder fields +- VectorPayload.swift:76 `enum VectorKind: UInt8` — .binary=0/.float32=1/.int8=2; ON-DISK raw values, never reorder +- VectorPayload.swift:101 `struct VectorPayload` — kind+dim+bytes+scale(int8 only, unused in prod) +- VectorPayload.swift:146 `init(engram:)` — binary, zero-copy wire bytes +- VectorPayload.swift:163 `init(floats:)` — float32, explicit little-endian serialization (byte-order portability, not native-order) +- VectorPayload.swift:187/:200 `asEngram()` / `asFloats()` — throw invalidPayload on kind/size mismatch +- VectorPayload.swift:38 `struct VectorPayloadInput` — bulk-write row bundle (itemID, vectorIndex, payload, modelID, modelVersion, filedAt) +- DenseHit.swift:45 `struct DenseHit: Sendable, Equatable` — key + rawDistance(Int32, dual-purpose: Hamming int OR Float bit pattern) + metric +- DenseHit.swift:101/:121 `hammingDistance` / `floatDistance` — typed accessors reinterpreting rawDistance +- DenseHit.swift:132 `enum LaneTag` — .binaryDense/.floatDense/.sparse/.lateInteraction (fusion/cross-package use) +- DenseMetric.swift:41 `enum FloatMetric` — .cosine/.l2/.dot; VectorKit-owned (ADR-008) +- DenseMetric.swift:58 `enum BinaryMetric` — .hamming/.jaccard (jaccard reserved, BruteForceIndex rejects it) +- DenseMetric.swift:81 `enum DenseMetric` — .binary(BinaryMetric)/.float(FloatMetric) umbrella; :91-:103 shorthand statics +- DenseIndex.swift:67 `enum SearchDirection` — .nearest/.farthest; farthest is bottom-K scan, NOT negated top-K +- DenseIndex.swift:85 `struct MetadataFilter` — modelID/modelVersion wildcard-if-nil; :109 `accepts(_:)` +- DenseIndex.swift:35 `enum IndexKind` — .bruteForce/.mih tag (nominal dispatch, not type-casting) +- DenseIndex.swift:131 `protocol DenseIndex: Sendable` — build/search/add/remove seam; BruteForceIndex is the binary oracle + +### Binary engines +- BruteForceIndex.swift:47 `actor BruteForceIndex: DenseIndex` — Lane A; conformance oracle; ZERO Hamming math in file (I-7) +- BruteForceIndex.swift:101 `search(probe:metric:k:filter:)` — only .binary(.hamming); model-partition slice via O(log m) lookup; sorts (distance ASC, FULL key ASC) — NOT EngramLib.findNearest (different tie-break) +- BruteForceIndex.swift:227 `add(key:vector:)` — tombstone-then-append upsert; :276 `remove(key:)` — tombstone only, no reclaim +- BruteForceIndex.swift:305 `currentSnapshot()` — value-copy for cross-actor read (VectorStore tombstone scans) +- BruteForceIndex.swift:326/:342 `setTombstoneBit` / `buildPartitions` — shared bit-layout + partition-rebuild helpers, mirrored in ResidentArrayStore +- MIHIndex.swift:251 `actor MIHIndex: DenseIndex` — Lane B; sub-linear EXACT Hamming k-NN via Multi-Index Hashing; output MUST equal BruteForceIndex bit-for-bit (BLOCKER conformance gate, MIHIndexTests.swift) +- MIHIndex.swift:72 `enum MIHBandCount: UInt32` — {.m4,.m8,.m16,.m32} ONLY (§1.7: keeps sub_bits∈{64,32,16,8}, no word-straddle) +- MIHIndex.swift:303 `init(bandCount:maskBudget:)` — maskBudget nil ⇒ dynamic max(n, 2^20) per query +- MIHIndex.swift:351 `search(...)` → :450 `knn(...)` — progressive-radius pigeonhole expansion; stop when heap full AND worstDist ≤ r +- MIHIndex.swift:504 enumeration-budget guard — projected flip-mask count vs budget; over-budget + heap not exact ⇒ :575 `bruteScan` fallback (still exact, just O(n)); fires `.notice` log + `vectorkit.mih.enumeration_fallback` metric +- MIHIndex.swift:711 `cumulativeChoose(subBits:rho:)` — Σ C(subBits,d); saturates to Int.max on overflow; MUST match Rust saturating_add bit-for-bit +- MIHIndex.swift:764 `colexFlipMasks(subBits:maxHamming:body:)` — Gosper's-hack colex enumeration, ascending subset size then ascending mask value; canonical order, internal (test-visible) +- MIHIndex.swift:654 `extractBand(from:bandIndex:)` — canonical bit numbering (bit i → word i/64, LSB=0); word-straddle branch unreachable for allowed m + +### Float engine +- FloatBruteForceIndex.swift:60 `actor FloatBruteForceIndex: DenseIndex` — Lane C/D (file header says "Lane C", VectorStore.swift comments say "Lane D" — same type, inconsistent lane label in source, not a functional issue); float32 ONLY; NOT four-way bit-identical (documented, do not "fix") +- FloatBruteForceIndex.swift:84 `build(from:)` — O(1) reference store; array IS the index +- FloatBruteForceIndex.swift:102 `search(...)` — validates probe.kind/.dim vs array stride; cosine treats zero-vector as distance 1.0 (no div-by-zero) +- FloatBruteForceIndex.swift:156 `searchFarthest(...)` — identical scan+distance as search(); only sort direction flips (:233 `rank(...direction:)`) +- FloatBruteForceIndex.swift:273 `add(key:vector:)` — FIRST add establishes stride; later mismatched byte count throws invalidPayload (prevents storage corruption) +- FloatBruteForceIndex.swift:323 `remove(key:)` — tombstone; compaction only on next build() + +### Late-interaction scorer (standalone, not a DenseIndex) +- MaxSimScorer.swift:97 `struct MaxSimScorer: Sendable` — Lane E1, Exact-A exhaustive ColBERT MaxSim; conformance reference for future pruned variants +- MaxSimScorer.swift:148 `score(queryTokens:documents:k:) -> [MaxSimHit]` — Σ(256−min hamming) per query token; documents iterated in SORTED itemID order (dict order is undefined); sort (score DESC, itemID ASC); truncate to k AFTER full sort +- MaxSimScorer.swift:55 `struct MaxSimHit` — itemID + integer score [0, 256×|Q|] +- All distances via `EngramLib.Session.distances` (I-7); session built once per scorer, reused across the whole score() call + +### Resident array (shared data contract, Lane F) +- ResidentVectorArray.swift:65 `struct ResidentVectorArray: Sendable` — packed fixed-stride array; kind/stride/count/storage/keys/modelPartitions/tombstones; measured 87% of pre-resident latency was fetch+decode, 0.4% kernel — this type removes the fetch+decode cost +- ResidentVectorArray.swift:135 `liveCount` — O(count/64) tombstone-bitmap walk; used for sidecar staleness (live-vs-live compare) +- ResidentVectorArray.swift:190 `partitionRange(for:)` — binary search, O(log m) +- ResidentVectorArray.swift:214/:228 `isTombstoned(_:)` / `vectorBytes(at:)` — per-slot accessors every engine scan loop uses +- ResidentVectorArray.swift:44 `struct ModelPartitionEntry` — modelID + half-open Range + +### Resident array persistence — ResidentArrayStore.swift +- :116 `actor ResidentArrayStore` — owns optional `.vec` sidecar; vectors TABLE remains sole durable source; sidecar is regenerable cache only +- :97 `kVecVersion = 0x0002` — format version; adds live_count field after count (discarded on load, recomputed from tombstone bitmap — stale header value cannot corrupt results) +- :102 `kDefaultTombstoneCompactionThreshold = 0.25` +- :183 `load()` — missing/invalid sidecar ⇒ start empty, no crash +- :209 `rebuild(from:)` — full rewrite from sorted [(key,bytes)]; used on stale-sidecar detection +- :247 `append(key:bytes:)` — EAGER write (immediate sidecar rewrite) +- :277 `appendDeferred(key:bytes:)` — WRITE-BEHIND single-add path (production default via VectorStore.addPayload); sets isDirty, no disk write; caller must flush() +- :299 `appendBatch(records:)` — bulk path, ONE sidecar write per batch (not per record) — TASK #24 amortization +- :346 `flush()` — persists pending write-behind mutation; no-op if !isDirty +- :387/:420 `tombstone(key:)` (eager, writes) / `tombstoneDeferred(keys:)` (batch, no write, sets isDirty) +- :448 `compact()` — drops tombstoned slots, sorted-by-key rewrite, deterministic output +- :563/:610/:623 `writeSidecar` / `readSidecar` (mmap via .mappedIfSafe) / `parseSidecar` — every length field bounds-checked before trust; magic "VEC1" (:85 `kVecMagic`) +- On-disk layout: magic(4)|version(2)|kind(1)|stride(4)|count(4)|live_count(4)|tombstone_words(4)|tombstones|vectors|keys(variable)|partition_index(variable); ALL integers little-endian (cross-host byte-identity, arch spec §4.3) + +### Storage-facing types +- StoredVector.swift:20 `struct StoredVector: Sendable, Equatable` — decoded `vectors` row; `engram` non-nil ONLY for binary kind (float/int8 rows: use getPayload) +- VectorMatch.swift:19 `struct VectorMatch: Sendable, Comparable, Equatable` — itemID/distance/modelID; :43 `<` — (distance ASC, itemID ASC) universal tie-break + +### Storage actor — VectorStore.swift +- :128 `actor VectorStore` — the kit's single consumer-facing surface +- :327 `static let schemaDeclaration` — "vectors" table v3; UNIQUE(item_id, vector_index, model_id) == VectorRecordKey minus modelVersion +- :382 `static defaultSidecarURL(for:)` — `.sqlite` → `.vectors.vec`; nil for non-file backends +- :165 `mihThreshold: UInt32 = 50_000` (default) — promotion boundary, overridable at init +- :172 `mihBandCount: MIHBandCount` (default .m16) — pinned per §1.6 for 50k default threshold +- :185/:189/:197 `bruteForceIndex` / `mihIndex` / `hotIndex` — both allocated at init; hotIndex swapped by :1661 `_selectIndex()` (no rebuild on swap) +- :283 `floatIndices: [String: FloatBruteForceIndex]` — ONE PER modelID (uniform stride requirement); map-entry presence == "built" flag +- :451 `addVector(itemID:engram:modelID:modelVersion:filedAt:)` — convenience wrapper over addPayload +- :496 `addPayload(itemID:vectorIndex:payload:modelID:modelVersion:filedAt:)` — rejects .int8 (throws int8QuantizationPolicyUndefined); table upsert THEN resident mirror; matches stale slots by (itemID,vectorIndex,modelID) — NOT full key — so modelVersion changes are treated as replacement (secfix/ws2-coredelete hard-delete contract); emits vectorkit.index.insert_latency_ms +- :669 `addPayloads(_:)` — bulk path; rejects batch containing ANY int8 (no partial writes); immediate mode rebuilds both indexes ONCE from final snapshot; emits vectorkit.index.batch_insert_latency_ms +- :874 `beginDeferredIndex()` / :898 `publishResidentIndex()` — bulk-burst mode: appends skip index rebuild until publish (O(N) not O(N²)); corpus ingest drain wraps a burst in this pair +- :253/:943 `deferredPendingRecords` / `_flushDeferredPending()` — memory-only-path back-pressure valve, capped at `deferredPendingLimit` (default 50_000, ctor param) — secfix/punt-vector unbounded-buffer fix +- :993 `flush()` — persists pending sidecar write-behind mutation +- :1052/:1066/:1088 `getVector` / `getPayload` / `vectors(forItemID:)` — read paths, all decode via :1803 `decodePayload` / :1840 `storedVector` +- :1134 `findNearest(probe:modelID:limit:)` — lazy `_ensureIndexBuilt()`, delegates to hotIndex.search, NO re-sort (engine already ordered); emits vectorkit.search.latency_ms + .result_count +- :1223 `findNearestFloat(probe:modelID:limit:)` — lazy per-model FloatBruteForceIndex build via :1558 `_ensureFloatIndexBuilt`; quantizes cosine distance ×10_000 rounded for cross-language integer comparison +- :1294 `findFarthestFloat(probe:modelID:limit:)` — same as findNearestFloat but calls searchFarthest; anti-similarity +- :1337 `findByKeyword(_:limit:)` — substring LIKE on item_id; NOT full BM25 (CorpusKit's job); emits vectorkit.search.keyword_result_count +- :1377 `deleteVector(itemID:modelID:)` / :1386 `deleteAllVectors(itemID:modelID:)` — both flush pending deferred burst FIRST if dirty, then delete+tombstone; deleteAllVectors invalidates that model's float index (lazy rebuild) +- :1442 `destroyAllVectors()` — full wipe: table+both binary indexes+sidecar+ALL float indices; used by GeniusLocusKit estate teardown +- :1487 `_ensureIndexBuilt()` — idempotent; sidecar trusted iff `snap.liveCount == tableCount` (live-vs-live, NOT snap.count vs table — avoids spurious rebuild after deletes, "C5 fix") +- :1558 `_ensureFloatIndexBuilt(modelID:)` — nil return (no cache) when model has zero float rows, so a later first-ingest can still build a real index +- :1736 `_deleteAndTombstone(itemID:vectorIndex:modelID:)` — scans ALL slots matching (itemID,vectorIndex,modelID) across modelVersions, tombstones every match (no break-after-first) — hard-delete contract +- :1803 `decodePayload(from row:)` — int8 rows ALWAYS decode to nil (symmetric fail-closed read guard, even for hand-crafted rows); guards every Int64→UInt8/UInt32 narrowing conversion against trap + +## INVARIANTS / GOTCHAS + +- I-7 ABSOLUTE: zero Hamming/XOR/popcount arithmetic anywhere in this package outside EngramLib calls. BruteForceIndex, MIHIndex, MaxSimScorer all delegate every distance to EngramLib.distances / EngramLib.Session.distances / EngramLib.distance. A raw popcount anywhere is a conformance violation. +- I-4 ABSOLUTE: cross-model vector comparison is forbidden. Every write carries modelID+modelVersion; every search is scoped to one modelID; float indices are one-per-model because different models emit different dimensions; the resident binary partition index scopes searches by modelID. +- Determinism boundary: binary lane (.hamming) is four-way bit-identical (Swift/Rust × platforms), gated by EngramLib's kernel. Float lane (.cosine/.l2/.dot) is reproducible-within-config ONLY — NOT four-way. Do not add a conformance test asserting float bit-identity; it will be correctly flagged as testing an undocumented, unintended guarantee. +- MIHIndex MUST equal BruteForceIndex output bit-for-bit on every input — this is a BLOCKER gate (MIHIndexTests.swift), not a best-effort property. BruteForceIndex is the oracle; never "fix" MIH by relaxing the gate. +- int8 is REJECTED fail-closed at both write (addPayload/addPayloads throw int8QuantizationPolicyUndefined) and read (decodePayload returns nil for kind==.int8). The case, field, and guards stay until a quantization policy is ratified — do not remove the guard as "dead code." +- Stale-slot matching in addPayload/addPayloads uses (itemID, vectorIndex, modelID) — NOT the full VectorRecordKey — specifically so a modelVersion change is recognized as replacement, not a new sibling slot. Matching by full key here is the recurring bug shape (secfix/ws2-coredelete); do not "simplify" to full-key equality. +- VectorRecordKey ordering (itemID, vectorIndex, modelID, modelVersion) is load-bearing: it is the resident-array partition order, the universal search tie-break, and the sidecar's on-disk key ordering. Do not reorder the Comparable fields. +- MIHBandCount is restricted to {4,8,16,32} by the enum itself (§1.7 conformance restriction) — sub_bits ∈ {64,32,16,8} guarantees no word-straddle in extractBand. Do not add m=2 or other values without a new word-straddle code path and a separate conformance harness. +- MIHIndex's enumeration-budget guard falls back to a full O(n) bruteScan (still EXACT, same heap, same distances) rather than hang on sparse/adversarial data. cumulativeChoose's saturating-overflow arithmetic MUST stay bit-identical to the Rust port's saturating_add so both fall back at the same radius. +- ResidentArrayStore sidecar is a REGENERABLE CACHE, never a second source of truth. The `vectors` table is authoritative. Staleness check compares live-vs-live counts (format 0x0002); comparing total slot counts instead (pre-C5-fix behavior) spuriously rebuilds after every delete. +- Sidecar single-add writes are WRITE-BEHIND (appendDeferred + isDirty): VectorStore.addPayload does not force a disk write per call. Callers relying on immediate on-disk durability of the sidecar must call `flush()`; crash safety does not depend on this because the table write already happened synchronously before the mirror. +- deferredIndexActive (beginDeferredIndex/publishResidentIndex) turns a bulk import from O(N²) to O(N) index rebuilds. deleteVector/deleteAllVectors both force-publish a dirty deferred window before deleting, so a delete never races an unpublished burst. +- deferredPendingRecords (memory-only deferred path) is capped at deferredPendingLimit (default 50,000); exceeding it triggers an intermediate flush that keeps the burst open. This is a back-pressure valve, not an error path — do not treat a flush mid-burst as anomalous. +- Float payload byte order is EXPLICIT little-endian (VectorPayload.init(floats:) / asFloats()), independent of host native endianness — required for the `.vec` sidecar (and any future float sidecar) to be byte-identical across Apple and Linux hosts. +- FloatBruteForceIndex establishes its stride from the FIRST vector added via add(key:vector:); a later vector of different byte count throws rather than corrupting the flat storage buffer. One index = one dimension, always. +- FloatBruteForceIndex is labeled "Lane C" in its own file header but "Lane D" in VectorStore.swift's comments — same type, inconsistent label, not a functional divergence. Do not "fix" one file to match the other without checking whether a Lane-lettering convention doc elsewhere disambiguates it first. +- MaxSimScorer (Lane E1) is NOT wired into VectorStore's public search API in this package version — it is a standalone scorer callers invoke directly with pre-fetched token-Engram arrays. Do not assume `findNearest` performs late interaction. +- Telemetry (Intellectus.report) is off by default; the emitted metrics (vectorkit.index.insert_latency_ms, .batch_insert_latency_ms, vectorkit.search.latency_ms, .result_count, .keyword_result_count, vectorkit.mih.enumeration_fallback) are named constants used by dashboards — renaming any of them is a breaking change to monitoring, not just to code. +- Pinned/default constants — changing any requires updating dependent conformance tests: mihThreshold 50,000, mihBandCount .m16 (sub_bits=16), deferredPendingLimit 50,000, compactionThreshold 0.25, poolSubmitThreshold-equivalent N/A (not used here), sidecar format version 0x0002, MIHBandCount ∈ {4,8,16,32}, cosine-distance quantization scale ×10,000. +- Actor boundaries: VectorStore, BruteForceIndex, MIHIndex, FloatBruteForceIndex, ResidentArrayStore are all actors — all mutation and reads are serialized per actor. ResidentVectorArray, VectorPayload, VectorRecordKey, DenseHit, VectorMatch, StoredVector are plain Sendable value types safe to pass across actor boundaries once constructed. diff --git a/packages/kits/VectorKit/docs/DETAILS.md b/packages/kits/VectorKit/docs/DETAILS.md new file mode 100644 index 0000000..bf45b65 --- /dev/null +++ b/packages/kits/VectorKit/docs/DETAILS.md @@ -0,0 +1,658 @@ +--- +doc: DETAILS +package: VectorKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/VectorKit/EmbeddingProvider.swift + blob: ad2bf52732b46960b9357a01fea37254d1681561 + - path: Sources/VectorKit/Engine/BruteForceIndex.swift + blob: da3bdac5a6d2b84ff73e4ec66057bcc2acd2b2cb + - path: Sources/VectorKit/Engine/DenseHit.swift + blob: 22289f57f49923647e4b99092c134dbc1910c15e + - path: Sources/VectorKit/Engine/DenseIndex.swift + blob: 010a51a54b6d62c971943070115e8822d9ffeafb + - path: Sources/VectorKit/Engine/DenseMetric.swift + blob: a28578e73ec36943a73d067e7768782311fa2005 + - path: Sources/VectorKit/Engine/FloatBruteForceIndex.swift + blob: 888d5a4079c84939b2cfde93160ee6bc3851adeb + - path: Sources/VectorKit/Engine/MaxSimScorer.swift + blob: 91875a79b8f6eebf6a2fd0a3a9dde85311a50aae + - path: Sources/VectorKit/Engine/MIHIndex.swift + blob: 61e283122542218eaf1f057cd7b9f1022930956f + - path: Sources/VectorKit/Engine/ResidentArrayStore.swift + blob: 21c67979dfc05d761909edec9700849d7cad74a5 + - path: Sources/VectorKit/Engine/ResidentVectorArray.swift + blob: 6e0f689702e4173388324b22fd828559ce0b1ab2 + - path: Sources/VectorKit/Engine/VectorPayload.swift + blob: 9259b4db9380cf9d854abd84a1d5059a0fcff5ec + - path: Sources/VectorKit/Engine/VectorRecordKey.swift + blob: bb4fff18c74c37ccafc9b1eaa01a4ec86b80be20 + - path: Sources/VectorKit/FloatSimHashEmbeddingProvider.swift + blob: efcb85396ceace1373e3017c0f799371a9a5c3bf + - path: Sources/VectorKit/StoredVector.swift + blob: 44702eaf3a0e28ef7f70031fa05751b48a8ecfbf + - path: Sources/VectorKit/VectorKit.swift + blob: 0a6eba27a0501601ee9ac015875de6d71bd4cf05 + - path: Sources/VectorKit/VectorKitError.swift + blob: 89c486eba6992edd583649e37674a67ea95ee317 + - path: Sources/VectorKit/VectorMatch.swift + blob: 24cc2c1bd25f71a7cef60a043c3a640df2368a23 + - path: Sources/VectorKit/VectorStore.swift + blob: 3c7fe4a19eba1142ac82a993cee0e7660a4ffdce +--- + +# VectorKit Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in pipeline +order: the module surface and errors, the embedding seam, the shared +engine foundation types, the three search engines, the resident-array +storage layer, and finally the storage-facing types and the `VectorStore` +actor that ties everything together. + +## VectorKit.swift + +This file provides the module surface: a short header comment naming the +kit's public pieces. It defines no types of its own. + +The file explains a boundary worth restating here: VectorKit imports +EngramLib internally to use the `Engram` fingerprint type, but it does not +re-export that import. A caller that wants to construct an `Engram` +directly must `import EngramLib` itself. This keeps every package's import +list explicit, matching the convention used across the rest of the +substrate-dependent kits. + +## VectorKitError.swift + +This file provides `VectorKitError`, the single error type for every +VectorKit operation. Per MOOTx01 convention, errors are concrete named +cases rather than a generic failure plus a logged message, so a caller can +branch on exactly what went wrong. + +`embeddingFailed(String)` reports an inference failure inside an +`EmbeddingProvider`. `modelUnavailable(String)` reports a model that is not +loaded on the current device. `storeUnavailable(String)` reports a failure +opening the backing storage. `notFound` is reserved for a future throwing +read path; today's read functions return `nil` instead. `invalidPayload` +covers a structurally broken `VectorPayload` — wrong kind, wrong byte +count, or a dimension mismatch — and is thrown by the payload's own +decode functions and by every search engine's input validation. +`decodingFailure` covers a malformed key or row that cannot be decoded. + +Two cases record product decisions rather than plain bugs. +`int8QuantizationPolicyUndefined` is thrown whenever a caller tries to +write an `.int8` (quantized) vector: the rules for how to quantize and +later reverse the quantization have not been agreed on yet, so writing one +now would lock in behavior nobody has approved. The case documents that +this is deliberate and reversible — once a policy is ratified, the guards +that throw this error are meant to be removed. `embedFloatVocabMiss` is +thrown by embedding providers whose vocabulary is fixed in advance (for +example, statistical models trained on a specific word list) when none of +a query's words are in that vocabulary; it is distinct from +`embeddingFailed` because the provider is working correctly — the input +simply has nothing in common with what the provider knows. + +## EmbeddingProvider.swift + +This file provides the `EmbeddingProvider` protocol, the single seam +between "text" and "vector" that every embedding source in VectorKit's kit +graph implements. + +The protocol is deliberately narrow: a `modelID` and `modelVersion` (the +tags every stored vector must carry, per spec I-4, so that vectors from +incompatible models are never compared), and three ways to turn text into +numbers. This narrowness is what lets VectorKit remain agnostic about +where inference happens — CoreML, ONNX, a hand-written statistical model — +and let a concrete provider such as `FloatSimHashEmbeddingProvider` or a +sibling package's MiniLM adapter fill in the details. + +`embed(_:)` is the primary method: it returns the 256-bit binary Engram +for a piece of text, throwing `embeddingFailed` or `modelUnavailable` on +failure. Every conformer must return the substrate's canonical zero Engram +for an empty string, because this is the one input every provider is +guaranteed to agree on, and treating it specially lets empty records from +different providers land on the same "identical" partition instead of +colliding by coincidence. + +`embedFloat(_:)` returns the dense float vector a provider computed on its +way to the Engram, before that vector was compressed into 256 bits. The +protocol's default implementation simply throws `embeddingFailed`, meaning +the float lane is opt-in: a provider that has nothing more precise to +offer than the fingerprint declines rather than fabricate numbers. +Providers that do real inference (MiniLM, mpnet, EmbeddingGemma, defined +in a sibling package) override this to return the vector they already +computed, at no extra inference cost. Empty input returns an empty array, +never a vector of zeros, because a vector of zeros would look like a +legitimate — and misleading — nearest neighbor to every genuinely empty +text. + +`embedPair(_:)` is a convenience for callers that need both outputs from +one text. Its default implementation calls `embed` and then `embedFloat` +— two inference passes — swallowing a float-lane opt-out into an empty +array so the default matches historical two-call behavior. A provider that +computes both outputs from a single inference pass should override this to +avoid running its model twice. + +`embedBatch(_:)` embeds several texts, defaulting to a sequential loop; +providers with genuinely batched inference should override it for +throughput. Output order always matches input order. + +## FloatSimHashEmbeddingProvider.swift + +This file provides `FloatSimHashEmbeddingProvider`, the one concrete +`EmbeddingProvider` VectorKit ships. It is a mirror of the Rust +`vectorkit::FloatSimHashEmbeddingProvider`. + +The type is a thin wrapper: it holds a `modelID`, a `modelVersion`, a +`projectionSeed`, and an injected inference closure — `(String) async +throws -> [Float]` — supplied by whatever host owns the actual model. +VectorKit itself never loads a model or owns a tokenizer; concrete text +providers that do (MiniLM, mpnet, EmbeddingGemma) live in a different +package, CorpusKitProviders, and conform to this same protocol using this +type as their low-level building block. + +`embed(_:)` short-circuits to the canonical zero Engram for empty input +before ever calling the inference closure, guaranteeing the empty-string +contract holds even if the closure itself would have produced something +non-zero. Otherwise it calls the inference closure and passes the result +through `SubstrateML.FloatSimHash.project(vector:seed:)` — a shared, +conformance-gated substrate primitive that projects an arbitrary-length +float vector down to a 256-bit fingerprint using a technique called +SimHash. The `projectionSeed` matters here: different seeds turn the same +float vector into different fingerprints, which is what keeps two +different models' outputs from accidentally landing in the same fingerprint +space and being wrongly compared (spec I-4's rule, enforced at the +projection layer as well as at storage). + +`embedFloat(_:)` returns exactly the float vector the inference closure +produced — the same vector `embed(_:)` feeds into the projection — so a +caller using both lanes never pays for two inference passes. + +## Engine/VectorRecordKey.swift + +This file provides `VectorRecordKey`, the identifier every stored vector +carries inside the search engines. Note that this file lives under +`Engine/`, VectorKit's internal folder for the search-engine machinery +shared by every index implementation. + +A key used to be a two-part (item, model) pair, which assumed one vector +per item per model. That assumption breaks for models like ColBERT that +produce one small vector per word instead of one vector per document, so +the key grew a third field. `itemID` names the owning record (a drawer or +a text chunk, as a UUID string). `vectorIndex` is the position of this +vector within its item's sequence — `0` for the ordinary single-vector +case, `0` through `N-1` for a multi-vector item. `modelID` and +`modelVersion` complete the tuple, because vectors from different model +versions are never comparable (spec I-4). + +`VectorRecordKey` is `Comparable`, ordered lexicographically by +`(itemID, vectorIndex, modelID, modelVersion)`. This order is not +incidental: it is the tie-break every search result uses when two matches +land at the same distance, and it is the order the on-disk resident array +is built in, so both the search output and the storage layout agree on +what "sorted" means. + +## Engine/VectorPayload.swift + +This file provides `VectorPayload`, the one envelope every vector's raw +bytes travel in, plus `VectorKind` (the tag for which numeric family the +bytes represent) and `VectorPayloadInput` (a payload bundled with its +storage metadata, used for bulk writes). + +`VectorKind` has three cases. `.binary` (raw value 0) is exactly the +32-byte Engram wire form. `.float32` (raw value 1) is `dim × 4` bytes of +IEEE-754 numbers. `.int8` (raw value 2) is reserved for a future quantized +representation; the case exists so that a future ratified quantization +policy does not require a new wire format, but `VectorStore` currently +rejects every `.int8` write (see `VectorKitError.int8QuantizationPolicyUndefined` +above). The raw values are stored on disk and must never be reordered. + +`VectorPayload.init(engram:)` builds a binary payload directly from an +Engram's wire bytes — no conversion, no loss, so every existing binary +test still holds. `init(floats:)` serializes a `[Float]` to little-endian +IEEE-754 bytes explicitly, byte by byte, rather than relying on the +platform's native byte order. This is a deliberate choice, not an +oversight: it is what lets the on-disk `.vec` sidecar be read back +identically on an Apple device and on a Linux server, which may not share +the same native byte order convention. `asEngram()` and `asFloats()` +reverse these conversions, throwing `invalidPayload` when the payload's +kind or byte count does not match what was asked for. + +## Engine/DenseHit.swift + +This file provides `DenseHit`, the one result shape every search engine +returns, and `LaneTag`, an enum naming which retrieval technique produced +a given score (used by fusion and multi-technique retrieval code outside +this package). + +A `DenseHit` carries the matched `key`, a `rawDistance`, and the `metric` +that produced it. The tricky design point is that `rawDistance` is a +single `Int32` field shared by very different kinds of number: an integer +Hamming distance for the binary lane, or the bit pattern of a `Float` +distance for the float lane. Two computed properties translate it back: +`hammingDistance` simply casts it to `Int` (safe because Hamming distances +are always 0–256), and `floatDistance` reconstructs the `Float` from its +stored bit pattern. Packing both families into one field, rather than +giving each lane its own result type, is what lets code elsewhere in the +kit graph handle a `[DenseHit]` without caring which lane produced it. + +The file's header calls out an "additive-only" rule: any future field +added here must have a default value, so existing callers who build a +`DenseHit` with the current initializer keep compiling. This matters +because both a Swift and a Rust version of this type exist, and the two +must stay in lockstep. + +## Engine/DenseMetric.swift + +This file provides the metric vocabulary for the whole engine seam: +`BinaryMetric` (`.hamming`, `.jaccard`), `FloatMetric` (`.cosine`, `.l2`, +`.dot`), and `DenseMetric`, the umbrella enum wrapping either family so +that `DenseIndex.search` needs only one metric parameter regardless of +which lane it routes to. + +The file's real content is its documentation of a determinism boundary +that recurs throughout this package: `.binary(.hamming)` is "four-way" +bit-identical, because it is pure integer arithmetic computed by a shared, +conformance-gated kernel; `.binary(.jaccard)` is bit-identical up through +its two integer counts, with one final IEEE-754 division that is itself +guaranteed identical because IEEE-754 mandates exact rounding for basic +operations; and every `.float(_)` metric is reproducible only within one +build and platform, never guaranteed identical between Swift and Rust. +This is stated as a documented property of floating-point math, not a +defect to be "fixed" — a warning aimed squarely at a future reviewer who +might otherwise try to force float parity that the underlying arithmetic +cannot honestly provide. + +## Engine/DenseIndex.swift + +This file provides the `DenseIndex` protocol — the single seam that lets +`VectorStore` treat three very different search engines +(`BruteForceIndex`, `MIHIndex`, `FloatBruteForceIndex`) as +interchangeable — along with three supporting types: `IndexKind` (a tag +naming which implementation is behind a given index, for tests that need +to pick the brute-force oracle deliberately), `SearchDirection`, and +`MetadataFilter`. + +`SearchDirection` distinguishes `.nearest` (most similar first, the +default) from `.farthest` (most dissimilar first). Farthest search +supports an "find things unlike this" query. It is not a trick of negating +a nearest-neighbor list — the farthest items are not among the nearest +top-k at all — so the index has to scan and sort toward the opposite end +using the exact same distance calculation, just the opposite sort +direction. + +`MetadataFilter` restricts a search to one `modelID` and, optionally, one +`modelVersion`. Its `accepts(_:)` method is the single predicate every +engine calls per-candidate; a `nil` field is a wildcard. + +The protocol itself declares four operations — `build(from:)`, +`search(probe:metric:k:filter:)`, `add(key:vector:)`, and +`remove(key:)` — and documents the contract every conformer must honor: +results come back sorted by distance ascending, ties broken by the +matched key ascending, and `BruteForceIndex` is the correctness oracle +every other binary engine is measured against. + +## Engine/BruteForceIndex.swift + +This file provides `BruteForceIndex`, the exact linear-scan search engine +for binary (Hamming) vectors, and the conformance oracle every other +binary engine — currently just `MIHIndex` — is checked against. + +The file is built around one hard rule, restated three times in its +comments: it performs zero Hamming arithmetic itself. Every distance is +computed by `EngramLib.distances`, which routes to a shared kernel +selected once per process (NEON on Apple silicon hardware, a scalar +fallback elsewhere) and checked for identical output across four build +configurations. Reimplementing a bitwise XOR-and-count here, even a +correct one, would bypass that check and risk silent divergence between +platforms — this is the file's version of spec I-7. + +`search(probe:metric:k:filter:)` validates the probe (must be exactly 32 +bytes of `.binary` kind) and the metric (only `.binary(.hamming)` is +supported; other requests throw `invalidPayload`), narrows the scan to one +model's slot range when a filter is present (an `O(log m)` lookup into a +sorted partition index, avoiding a full-array walk), collects the live, +un-tombstoned candidates in that range, and hands their Engrams to +`EngramLib.distances` in one batch call. It deliberately avoids +`EngramLib.findNearest`, which applies a different, insertion-order +tie-break; this file sorts the returned distances itself by +`(distance ascending, key ascending)`, because the engine's own contract +requires the full `VectorRecordKey` as the tie-break, not just the array's +insertion position — otherwise two records under the same item but +different model or vector index could be returned inconsistently. + +`add(key:vector:)` implements upsert by tombstoning any existing slot with +the same key before appending the new bytes, then rebuilding the sorted +model-partition index from the updated key list. `remove(key:)` tombstones +every matching slot without touching the underlying storage bytes — actual +space reclamation is `ResidentArrayStore`'s job, not this type's. +`currentSnapshot()` returns a value-type copy of the live array so callers +outside the actor (chiefly `VectorStore`, when it needs to scan for +tombstoning) can read it safely. + +## Engine/FloatBruteForceIndex.swift + +This file provides `FloatBruteForceIndex`, the linear-scan search engine +for the float32 lane — cosine, Euclidean (`l2`), and dot-product distance +— and, unlike the binary lane, this is both the correctness reference and +the production search path: there is no separate accelerated float engine +in this package. + +The file opens with an emphatic warning, repeated in this document because +it protects against a plausible but wrong "fix": float arithmetic here is +reproducible on one build and platform, but it is not, and cannot be +made to be, bit-identical between Swift and Rust or across different +hardware. This is a documented property of IEEE-754 arithmetic, not an +oversight; a reviewer must not try to force it to match the binary lane's +four-way guarantee. + +`build(from:)` simply stores a reference to the supplied array — there is +no secondary structure to construct, so building is `O(1)`; the real cost +is whatever the caller paid to assemble the array. `search(probe:metric: +k:filter:)` validates that the probe is `.float32`, that the requested +metric is a float metric, and that the probe's byte count matches both its +own declared dimension and the array's fixed stride (a mismatch would +otherwise read past the end of a slot and throws instead). It then scans +every live, filter-passing slot, computing one of three distances per +candidate — cosine distance treats a zero vector as maximally distant +rather than crashing on a divide-by-zero; `l2` is the plain Euclidean +formula; `dot` is negated so that "smaller is nearer" holds for every +metric uniformly — and sorts ascending by distance, then by key. +`searchFarthest(probe:metric:k:filter:)` reuses the identical scan and +identical distance math, changing only the sort direction to descending, +which is what makes "find dissimilar items" a real bottom-of-the-list scan +rather than a negated top-of-the-list one. + +`add(key:vector:)` establishes the array's dimension from the first vector +added and rejects any later vector of a different byte count, because a +mismatched stride would silently corrupt the flat storage buffer. +`remove(key:)` tombstones the matching slot; actual compaction happens the +next time `build(from:)` runs with a freshly assembled array. + +## Engine/MaxSimScorer.swift + +This file provides `MaxSimScorer` and `MaxSimHit`, the exhaustive +("Exact-A") implementation of ColBERT-style late-interaction scoring over +binary token fingerprints. + +Some embedding techniques represent one document as many small vectors — +one per word or token — rather than one vector for the whole document. +Comparing two such documents means asking, for every word in the query, +"which word in this document is most like it?" and adding up those best +matches. That sum, `Σ (256 − minimum Hamming distance)` over every query +token, is the MaxSim score this file computes. Because it examines every +query token against every document token for every candidate document, it +never skips a candidate; this exhaustiveness is precisely what makes it +the correctness reference for any faster, pruned variant built later — the +file's header explicitly reserves the accelerated two-stage variant as out +of scope here. + +`score(queryTokens:documents:k:)` iterates the supplied documents in +ascending itemID order — sorting the dictionary's keys explicitly, because +a Swift dictionary's own iteration order is not guaranteed and would make +results non-reproducible — computes each document's MaxSim score, sorts +the results `(score descending, itemID ascending)`, and truncates to `k` +only after the full sort, never before, so a document that would have +scored well is never cut for appearing late in an unsorted pass. Every +Hamming distance again goes through `EngramLib.Session.distances`, +constructed once per `MaxSimScorer` and reused for the whole call, so the +one-time cost of picking the fastest available kernel is paid once rather +than per comparison. + +## Engine/ResidentVectorArray.swift + +This file provides `ResidentVectorArray`, the packed in-memory data shape +every search engine reads from, and `ModelPartitionEntry`, one entry in +its per-model index. This is the shared contract underneath +`BruteForceIndex`, `MIHIndex`, and `FloatBruteForceIndex` — all three read +the identical layout, which is what lets `VectorStore` build the array +once and hand it to whichever engine is currently active. + +The design reason is stated directly in the file's header comment: +measurement on the pre-existing code path showed that fetching and +decoding rows from the database consumed 87% of a search's latency, while +the actual distance kernel took 0.4%. A fixed-stride, contiguous byte +array removes the fetch-and-decode cost from every query after the first, +because the whole array is loaded once and then scanned as a flat block of +memory with no per-row allocation. + +The type stores `kind` and `stride` (bytes per vector slot — 32 for +binary, `dim × 4` for float32), `count` (including tombstoned slots), +`storage` (the packed bytes themselves — on Apple platforms potentially +memory-mapped read-only from a sidecar file), a `keys` array parallel to +`storage`, a sorted `modelPartitions` index, and a `tombstones` bitmap. +`liveCount` walks the tombstone bitmap to compute how many slots are still +valid; `partitionRange(for:)` binary-searches the sorted partitions to +find one model's slot range in `O(log m)`; `isTombstoned(_:)` and +`vectorBytes(at:)` are the two per-slot accessors every engine's scan loop +calls. + +## Engine/ResidentArrayStore.swift + +This file provides `ResidentArrayStore`, the actor that owns the optional +on-disk `.vec` sidecar file: a packed binary cache of a +`ResidentVectorArray` that lets a reopened store skip rebuilding the array +from every database row. + +The file documents its own on-disk format in full — a fixed header +(magic bytes, format version, vector kind, stride, count, a live-slot +count, and the tombstone bitmap), followed by the packed vector bytes, +then variable-length key records, then the model-partition index, with +every multi-byte integer explicitly little-endian so the same file reads +identically on an Apple device and on a Linux server. `writeSidecar` +writes to a temporary file and atomically renames it into place, so a +crash mid-write never leaves a half-written sidecar behind; `readSidecar` +memory-maps the file where the platform supports it (a load-time +optimization, not a difference in the bytes returned) and `parseSidecar` +does the actual decoding, checking every length field against the +remaining buffer size before trusting it, so a corrupted or hand-edited +file is rejected with a `decodingFailure` rather than crashing the +process. + +The file is explicit about one policy: the `vectors` database table is +always the durable source of truth, and this sidecar is a regenerable +cache, never a second copy of record. `load()` reads the sidecar if +present; if it is missing or fails to parse, the store simply starts +empty and waits for `VectorStore` to rebuild it from the table. + +Three write paths exist because a single write policy could not serve +both a low-latency single insert and a large bulk import well. +`append(key:bytes:)` is the eager path: it writes the sidecar +immediately after every single addition. `appendDeferred(key:bytes:)` +is the "write-behind" path a single insert uses in production: it updates +the in-memory array and marks the store dirty without touching disk, +trusting the caller to call `flush()` at a natural pause. This is safe +because the database row was already written durably before this call — +losing an unflushed sidecar only costs a rebuild on the next open, never +data. `appendBatch(records:)` is the bulk-import path: it extends storage, +keys, and the tombstone bitmap for the whole batch in one pass and writes +the sidecar exactly once, so importing a thousand vectors costs one disk +write instead of a thousand. + +`compact()` rewrites the sidecar keeping only live (non-tombstoned) +records, sorted by key for a deterministic, reproducible layout, and is +triggered automatically whenever the tombstone ratio exceeds +`compactionThreshold` (25% by default) after any eager write. + +## StoredVector.swift + +This file provides `StoredVector`, the public, fully-decoded row shape +`VectorStore.vectors(forItemID:)` returns to callers who want the +convenient binary form rather than the raw typed payload. + +Its fields mirror the `vectors` table's columns directly: a stable `id` +assigned on insert, the owning `itemID`, the `vectorIndex` position within +a multi-vector item, `modelID` and `modelVersion` (spec I-4's tags), +the decoded `engram`, and `filedAt`, the time the row was written, +round-tripped through the database's text-based ISO 8601 timestamp column +(which loses sub-millisecond precision — the file notes this explicitly so +a caller comparing timestamps at fine granularity is not surprised). +`StoredVector.engram` is non-nil only for binary rows; a float or int8 row +must be read through `VectorStore.getPayload` instead, since this +convenience type only round-trips the binary case. + +## VectorMatch.swift + +This file provides `VectorMatch`, the public search-result shape +`VectorStore.findNearest` and its float-lane counterparts return. + +It carries `itemID` (the matched record), `distance` (Hamming distance for +the binary lane, an integer in 0…256; a scaled, quantized cosine distance +for the float lane, explained in `VectorStore`'s float-search functions), +and `modelID`, so a caller can confirm which model actually produced the +match. `VectorMatch` conforms to `Comparable`, ordered by `distance` +ascending with ties broken by `itemID` ascending — the same universal +tie-break rule used throughout the engine layer — so a sorted array of +matches reads nearest-to-farthest from front to back without a caller +having to know the sort convention. + +## VectorStore.swift + +This file provides `VectorStore`, the actor every consumer of VectorKit +actually talks to. It is the largest file in the package because it is +where every other piece — the durable table, the resident arrays, the +three search engines, and telemetry — is wired together into one +consistent API. + +### Storage and schema + +`VectorStore` wraps a PersistenceKit `Storage` backend (SQLite, an +in-memory backend for tests, or, in the future, PostgreSQL); the kit never +sees which backend is chosen — that decision belongs to the application. +`schemaDeclaration` is the static schema description passed to +`storage.open(schema:)` before the store is used: one `vectors` table, +version 3, whose `UNIQUE(item_id, vector_index, model_id)` constraint is +exactly `VectorRecordKey` minus `modelVersion` — the constraint that makes +an upsert on a changed model version a true replacement of the old row +rather than a duplicate. + +### Two hot-path structures kept in sync + +Every write updates three things together: the durable `vectors` table +row, the in-memory resident array (through `bruteForceIndex` and +`mihIndex`, which are both always kept current — only one is ever the +active `hotIndex`), and, when a sidecar was configured at construction, +the on-disk cache. `_selectIndex()` compares `liveBinaryCount` against +`mihThreshold` (50,000 by default) after every write that changes the +count, swapping `hotIndex` between `bruteForceIndex` and `mihIndex` with a +plain reference assignment — no rebuild is needed on promotion or +demotion, because both indexes were already current. `init(storage: +sidecarURL:mihThreshold:mihBandCount:deferredPendingLimit:)` allocates both +index actors up front so this swap never needs to construct anything at +query time. + +### Write path + +`addVector(itemID:engram:modelID:modelVersion:filedAt:)` is a convenience +wrapper for the common single binary vector case; it builds a +`VectorPayload` and delegates to `addPayload`. `addPayload(itemID: +vectorIndex:payload:modelID:modelVersion:filedAt:)` is the general write: +it rejects `.int8` payloads immediately (see `VectorKitError` above), +writes the row via an upsert keyed on the table's unique constraint, and +then — only for `.binary` and `.float32` kinds — mirrors the write into +the matching resident array. For a binary write, it first finds any +existing slot at the same logical position (`itemID`, `vectorIndex`, +`modelID`, ignoring `modelVersion`) so that a version change is treated as +a true replacement rather than leaving a stale duplicate slot behind; this +matching is deliberately looser than full key equality specifically to +catch that case. It then tombstones the stale slot (if any) and appends +the new one in both `bruteForceIndex` and `mihIndex`, updates +`liveBinaryCount` only when the write was genuinely new, and calls +`_selectIndex()`. Every write emits a `vectorkit.index.insert_latency_ms` +telemetry metric through IntellectusLib — a short-circuited no-op unless +monitoring has been explicitly turned on, so the cost on the default path +is one boolean check. + +`addPayloads(_:)` is the bulk-import counterpart, and its whole reason for +existing is complexity: importing N vectors one at a time through +`addPayload` costs N sidecar rewrites and, without care, N index rebuilds. +This function upserts every row to the table (unavoidable — the table is +the durable source), then rebuilds both binary indexes exactly once from +the final merged array rather than once per row, cutting the amortized +cost from `O(N²)` bytes written to `O(N)`. `beginDeferredIndex()` and +`publishResidentIndex()` extend this further for very large or +multi-call bulk imports: while a deferred window is open, `addPayloads` +appends to storage but skips the index rebuild entirely, seeding an +in-memory tracked set of live keys so replacement detection stays cheap +across the whole window; `publishResidentIndex()` performs the single +rebuild the whole burst needed, once, when the caller signals the burst is +finished. `deferredPendingRecords`, capped at `deferredPendingLimit` +(50,000 by default), guards the memory-only variant of this path against +unbounded growth if a caller holds the window open indefinitely; crossing +the cap triggers `_flushDeferredPending()`, an internal intermediate merge +that keeps the deferred window open for the caller while bounding peak +memory. + +### Search path + +`findNearest(probe:modelID:limit:)` is the binary-lane search: it lazily +builds the resident array on first use (from the sidecar if one is +current, or from the table if not), then delegates entirely to +`hotIndex.search`, converting the returned `[DenseHit]` into +`[VectorMatch]` without re-sorting — the engine has already applied the +required `(distance ascending, itemID ascending)` order. +`findNearestFloat(probe:modelID:limit:)` is the float-lane equivalent: it +lazily builds a `FloatBruteForceIndex` per model (the map's presence is +the "already built" flag — there is no separate boolean), then searches +by cosine distance and quantizes the resulting `Float` distance to an +integer by multiplying by 10,000 and rounding, so results from different +languages' fixtures can be compared exactly rather than approximately. +`findFarthestFloat(probe:modelID:limit:)` is identical except it calls the +engine's farthest-ranking search, for an anti-similarity "find things +unlike this" query. `findByKeyword(_:limit:)` is a plain substring match +on `item_id`, explicitly documented as a quick pre-filter for hybrid +retrieval, not a real keyword search — full keyword scoring is a different +package's responsibility. + +### Delete path + +`deleteVector(itemID:modelID:)` deletes the row at `vectorIndex` 0 and +tombstones every matching resident slot. `deleteAllVectors(itemID: +modelID:)` deletes every vector index for an item and model, used when a +multi-vector item (all its ColBERT token vectors, for instance) needs +complete removal. Both first flush any in-flight deferred-index burst, +so a delete never races an unpublished bulk import. `destroyAllVectors()` +wipes the entire store — every row, both resident indexes, the sidecar, +and the per-model float indexes — as part of a coordinated estate +teardown; an estate is one user's complete memory store in MOOTx01. + +### Coherence helpers + +`_ensureIndexBuilt()` is the one-time (per process) function that +populates both binary resident indexes, either by trusting a sidecar whose +recorded live-slot count matches the table's live binary-row count, or, if +they disagree, by rebuilding from the table and rewriting the sidecar. +Comparing live-count to live-count, rather than the older approach of +comparing total slot counts, avoids a spurious full rebuild after ordinary +deletions leave tombstoned slots behind. `decodePayload(from:)` and +`storedVector(from:)` are the row-decoding functions every read path +shares; both explicitly guard every narrowing integer conversion (for +example, a negative `dim` or an out-of-range `kind` byte) so that a +hand-crafted or corrupted row is rejected with `nil` rather than crashing +the process on a Swift trap. + +## Rust Port and Conformance + +The `rust/` directory mirrors the Swift implementation file for file: +`vector_store.rs` alongside `engine/brute_force.rs`, `engine/mih.rs`, +`engine/float_brute_force.rs`, `engine/max_sim.rs`, `engine/resident.rs`, +`engine/resident_store.rs`, `engine/key.rs`, `engine/payload.rs`, +`engine/hit.rs`, `engine/metric.rs`, `engine/seam.rs`, plus +`embedding_provider.rs`, `simhash_embedding_provider.rs`, and `error.rs`. +The `.vec` sidecar format, the MIH band-hashing algorithm, and the MaxSim +scoring algorithm are all specified precisely enough in the Swift source +comments (colex enumeration order, the enumeration-budget guard's integer +arithmetic, the little-endian sidecar layout) that both ports are expected +to agree exactly on the binary lane. `rust/tests/` holds integration +suites for bulk ingest, the float lane, int8 rejection, the SimHash +provider, the vector store, and telemetry, exercising the same behaviors +described above. The package's own `MIHIndexTests.swift` gates `MIHIndex` +against `BruteForceIndex` directly within Swift; cross-language +conformance for the binary lane rests on both ports implementing the same +documented algorithm rather than on a single shared fixture file, unlike +LatticeLib's shared JSON fixtures. The float lane is, by design, exempt +from cross-language bit-identity (see `DenseMetric.swift` and +`FloatBruteForceIndex.swift` above); only within-platform reproducibility +and rank correctness are asserted for it, in both languages. diff --git a/packages/kits/VectorKit/docs/INTERFACE_DOCTRINE.md b/packages/kits/VectorKit/docs/INTERFACE_DOCTRINE.md deleted file mode 100644 index 8d4a0af..0000000 --- a/packages/kits/VectorKit/docs/INTERFACE_DOCTRINE.md +++ /dev/null @@ -1,79 +0,0 @@ -# VectorKit Interface Doctrine - -For coding agents using VectorKit in product code or downstream kits. - -## 1. The kit is PersistenceKit-backed - -VectorKit's storage is `any Storage` from PersistenceKit. The application picks the backend (InMemory for tests, SQLite for Apple-platform deployment, PostgreSQL for server-side). VectorKit does not open SQLite directly any more. - -```swift -let storage = SQLiteStorage(configuration: EstateConfiguration( - estateID: estate.id, - backend: .sqlite(url: dbURL) -)) -try await storage.open(schema: VectorStore.schemaDeclaration) -let store = VectorStore(storage: storage) -``` - -A kit consuming VectorStore receives `any Storage` from its caller. It does not pick a backend. - -## 2. Engrams are tagged - -Every vector stored carries `(modelID, modelVersion)` per spec I-4. Cross-model comparisons are forbidden. `findNearest(probe:modelID:limit:)` filters to a single model; the caller is responsible for ensuring the probe was produced by that same model. - -```swift -let probe = try await provider.embed("query text") -let matches = try await store.findNearest( - probe: probe, - modelID: provider.modelID, - limit: 10 -) -``` - -## 3. Use FloatSimHash for new providers - -External embedding providers (MiniLM, BERT, EmbeddingGemma) project dense float vectors through `SubstrateLib.FloatSimHash` to produce 256-bit Engrams. Each provider picks a stable, deterministic seed so its fingerprints are model-tagged independent of vector content. - -```swift -private static let providerSeed: UInt64 = 0x... // unique per model -let engram = FloatSimHash.project(vector: pooledFloats, seed: providerSeed) -``` - -Do not invent ad-hoc projections. Sign-bit folding, hash-into-bits, etc. are gone from the kit graph; they did not preserve cosine similarity faithfully. FloatSimHash does (cookbook section 3.6). - - -VectorKit currently embeds a deterministic stand-in tokenizer inside `MiniLMProvider` so the provider can ship before CorpusKit lands. After mission 7, tokenization moves to CorpusKit and providers consume it through CorpusKit's tokenizer protocol. Do not extend the MiniLMProvider tokenizer; it is internal and migrating. - -If you need text tokenization in a non-MiniLM context today, wait for CorpusKit. If you need it before CorpusKit lands, file a decision record. - -## 5. Hybrid retrieval composes elsewhere - -VectorKit's `findByKeyword` is a coarse substring filter on drawer IDs, kept for backward compatibility with hybrid-retrieval callers that need a fast pre-filter. Full BM25 keyword search with proper tokenization lives in CorpusKit. Composition of vector + keyword scoring (RRF, MMR) lives in NeuronKit's hybrid recall (paper section 10.2). VectorKit stays focused on vector primitives. - -## 6. The schema is owned by VectorStore - -`VectorStore.schemaDeclaration` is a public static. Callers pass it to `Storage.open(schema:)` before constructing the store. The schema declares one table (`vectors`) with the columns and constraints documented inline. Do not reach into the table directly from outside the kit; go through VectorStore's API. - -If you need a column VectorStore does not expose, file a decision record proposing the schema change. Schema version bumps follow the migration rules in PersistenceKit's INTERFACE_DOCTRINE. - -## 7. findNearest is currently a linear scan - -The v1 implementation scans every row in the `vectors` table for the given `modelID` and computes Hamming distance in memory. This matches the pre-refactor SQLite behavior. The migration to PersistenceKit's `VectorIndex` protocol with sqlite-vec or pgvector ANN is a follow-on; it does not change the public API. - -If your call sites need sub-linear retrieval today, gate them on corpus size or fall through to a higher-tier path. P99 on a 10k-row corpus is roughly 50ms on Apple Silicon. - -## 8. Sendable everywhere - -VectorStore is an actor. Its public methods are async. EmbeddingProvider is `Sendable`. StoredVector and VectorMatch are `Sendable` value types. If you wrap VectorKit in another kit, that kit's public surface stays `Sendable` too. - -## 9. When in doubt, file a decision record - -If you find yourself wanting to: - -- Bypass the model tag (modelID, modelVersion) on stored vectors -- Add a new projection besides FloatSimHash -- Expose VectorStore internals (the underlying Storage, the schema, the table name) -- Add a kit dependency from VectorKit on something other than SubstrateLib, EngramLib, or PersistenceKit -- Cross the kit boundary into CorpusKit-shaped tokenization concerns - -Stop. Write a decision record in `docs/decisions/` proposing the change. The kit boundaries are deliberate. diff --git a/packages/kits/VectorKit/docs/OVERVIEW.md b/packages/kits/VectorKit/docs/OVERVIEW.md new file mode 100644 index 0000000..726ee1c --- /dev/null +++ b/packages/kits/VectorKit/docs/OVERVIEW.md @@ -0,0 +1,202 @@ +--- +doc: OVERVIEW +package: VectorKit +repo: moot-memory +authored_commit: ecbe2bc361c83a1e8bc636767d33d0c678f88bd7 +authored_date: 2026-07-04 +sources: + - path: Sources/VectorKit/EmbeddingProvider.swift + blob: ad2bf52732b46960b9357a01fea37254d1681561 + - path: Sources/VectorKit/Engine/BruteForceIndex.swift + blob: da3bdac5a6d2b84ff73e4ec66057bcc2acd2b2cb + - path: Sources/VectorKit/Engine/DenseHit.swift + blob: 22289f57f49923647e4b99092c134dbc1910c15e + - path: Sources/VectorKit/Engine/DenseIndex.swift + blob: 010a51a54b6d62c971943070115e8822d9ffeafb + - path: Sources/VectorKit/Engine/DenseMetric.swift + blob: a28578e73ec36943a73d067e7768782311fa2005 + - path: Sources/VectorKit/Engine/FloatBruteForceIndex.swift + blob: 888d5a4079c84939b2cfde93160ee6bc3851adeb + - path: Sources/VectorKit/Engine/MaxSimScorer.swift + blob: 91875a79b8f6eebf6a2fd0a3a9dde85311a50aae + - path: Sources/VectorKit/Engine/MIHIndex.swift + blob: 61e283122542218eaf1f057cd7b9f1022930956f + - path: Sources/VectorKit/Engine/ResidentArrayStore.swift + blob: 21c67979dfc05d761909edec9700849d7cad74a5 + - path: Sources/VectorKit/Engine/ResidentVectorArray.swift + blob: 6e0f689702e4173388324b22fd828559ce0b1ab2 + - path: Sources/VectorKit/Engine/VectorPayload.swift + blob: 9259b4db9380cf9d854abd84a1d5059a0fcff5ec + - path: Sources/VectorKit/Engine/VectorRecordKey.swift + blob: bb4fff18c74c37ccafc9b1eaa01a4ec86b80be20 + - path: Sources/VectorKit/FloatSimHashEmbeddingProvider.swift + blob: efcb85396ceace1373e3017c0f799371a9a5c3bf + - path: Sources/VectorKit/StoredVector.swift + blob: 44702eaf3a0e28ef7f70031fa05751b48a8ecfbf + - path: Sources/VectorKit/VectorKit.swift + blob: 0a6eba27a0501601ee9ac015875de6d71bd4cf05 + - path: Sources/VectorKit/VectorKitError.swift + blob: 89c486eba6992edd583649e37674a67ea95ee317 + - path: Sources/VectorKit/VectorMatch.swift + blob: 24cc2c1bd25f71a7cef60a043c3a640df2368a23 + - path: Sources/VectorKit/VectorStore.swift + blob: 3c7fe4a19eba1142ac82a993cee0e7660a4ffdce +--- + +# VectorKit Overview + +## What This Library Does + +VectorKit turns a piece of text into a vector — a fixed-size code that +stands in for the text's meaning — and stores that vector so a later query +can find the most similar ones. MOOTx01 is an on-device AI memory system. +It stores what an AI observes over time and helps the AI recall it later. +VectorKit is the part of MOOTx01 that answers "what stored memories are +most like this one?" + +VectorKit stores two kinds of vector for the same piece of text. The first +is a 256-bit binary fingerprint called an `Engram`, defined by a sibling +library, EngramLib. A fingerprint is a short fixed-size code computed from +a piece of content; similar content produces similar fingerprints, so the +system compares things quickly without reading them in full. Two Engrams +are compared by Hamming distance — the number of bit positions where they +differ; smaller means more similar. The second kind is a dense float +vector: a list of several hundred decimal numbers produced directly by an +embedding model such as MiniLM. VectorKit keeps both because they serve +different needs, explained below. + +## The Problem It Solves + +An AI memory system needs to answer "find memories like this one" without +sending private text to a server. VectorKit runs entirely on the device +that captured the memory. + +A 256-bit fingerprint is compact and fast to compare — comparing two +fingerprints is counting differing bits, pure integer arithmetic. Because +the comparison never uses floating-point math, it produces the exact same +answer on every device and every operating system. VectorKit calls this +property "four-way" determinism, and it never computes a fingerprint +comparison itself: every comparison is delegated to EngramLib, which in +turn delegates to a shared, conformance-gated kernel (a conformance +fixture is a recorded input/output pair both an original and a ported +implementation must reproduce exactly; EngramLib's kernel is checked this +way across four build configurations). This is spec I-7 in VectorKit's own +design documents: the kit performs no Hamming math of its own. + +A fingerprint is compact, but compacting hundreds of numbers into 256 bits +throws information away. Some queries need the finer detail the original +float numbers carry — for example, telling a passage from its own echoed +question apart, which the fingerprint's collapsed representation cannot +always do. For those queries VectorKit also stores the float vector and +compares it with cosine distance, a measure of the angle between two +vectors. Float math is reproducible on one platform and one build, but it +is not guaranteed to produce byte-identical results on a different +platform. VectorKit documents this openly as a boundary, not a defect: the +binary fingerprint lane is the four-way-identical lane, and the float lane +is the "reproducible within one configuration" lane. + +Every stored vector is tagged with the model identifier and model version +that produced it. Two different embedding models turn the same word into +different numbers for reasons that have nothing to do with meaning, so +comparing vectors from different models produces a meaningless answer. +VectorKit enforces this rule, called spec I-4, at multiple levels: storage +partitions vectors by model, and every search is scoped to one model. + +Finally, an on-device memory store must stay fast as it grows. Scanning +every stored vector for every query works fine at a few thousand records +but becomes slow at scale. VectorKit solves this by keeping an in-memory +copy of all fingerprints (so no per-query database read is needed) and by +switching, once the count of live fingerprints crosses a threshold, from a +full scan to a sub-linear search structure that returns the identical +answer faster. + +## How It Works + +Writing a vector has two parts. `VectorStore.addPayload` first writes the +vector as one row to a `vectors` table, which is the durable source of +truth — nothing is ever considered stored until this write succeeds. It +then mirrors a binary vector into an in-memory packed array (or, for a +float vector, into a per-model in-memory array) so that later searches +never have to re-read the database. The in-memory array can optionally be +backed by an on-disk cache file, called a sidecar, so that reopening the +store does not require rebuilding the array from every database row. + +Searching has two independent paths, one per vector kind. A binary search +(`findNearest`) compares the query fingerprint against the in-memory array +using Hamming distance. Below a configurable threshold of live fingerprints +(50,000 by default), the search does a full linear scan — always fast +enough at that scale. At or above the threshold, VectorKit switches to a +technique called Multi-Index Hashing, which slices each 256-bit fingerprint +into several shorter bands and uses per-band lookup tables to rule out most +of the collection without touching it. Multi-Index Hashing is provably +exact: it returns precisely the same neighbors the full scan would have +returned, just faster. A test suite checks this by running both searches +on the same random and adversarial inputs and requiring identical output. + +A float search (`findNearestFloat`) compares the query's float vector +against a separate in-memory array holding only that model's float +vectors, using cosine, Euclidean, or dot-product distance. Because +different models produce vectors of different length, VectorKit keeps one +float array per model rather than mixing them. + +A third comparison method, `MaxSimScorer`, serves models that produce many +small vectors per item instead of one — for example, one fingerprint per +word, an approach known as ColBERT-style late interaction. Rather than +compare one vector to one vector, it compares every query-word fingerprint +against every document-word fingerprint and keeps, for each query word, +the best match found in the document; the document's overall score is the +sum of those best matches. This scorer is exhaustive: it never skips a +candidate document, which makes it the correctness reference for any +faster method built later. + +## How the Pieces Fit + +Figure 1 shows the library's topology — its major parts and how data moves +between them. + +![Figure 1. Topology of VectorKit](topology.svg) + +*Figure 1. Topology of VectorKit. Text enters through an `EmbeddingProvider` +and becomes an Engram and, optionally, a float vector. `VectorStore` writes +both to the durable `vectors` table and mirrors them into in-memory +resident arrays. Reads dispatch through the `DenseIndex` seam to one of +three interchangeable search engines. The dashed regions mark the durable +storage boundary and the optional on-disk cache.* + +`EmbeddingProvider` is the seam a host application implements: it supplies +whatever inference technique turns text into numbers (a CoreML model, for +instance). VectorKit's own concrete implementation, +`FloatSimHashEmbeddingProvider`, takes those numbers and projects them into +an Engram fingerprint using a shared substrate primitive, `FloatSimHash`, +so that every provider in the MOOTx01 kit graph produces fingerprints the +same deterministic way. + +`VectorStore` is the actor every caller talks to. It owns the durable +`vectors` table (through a PersistenceKit `Storage` backend such as +SQLite) and the resident in-memory arrays. Three foundation types flow +through every layer beneath it: `VectorRecordKey` (which record this is), +`VectorPayload` (the raw typed bytes of one vector), and `DenseHit` (one +scored search result). All three are shared, additive-only types — no +search engine defines its own private version of them. + +Underneath `VectorStore` sits the `DenseIndex` protocol, a pluggable engine +seam. Three concrete engines implement it: `BruteForceIndex` (the always- +correct binary linear scan and the conformance reference), `MIHIndex` (the +sub-linear binary search gated against `BruteForceIndex`), and +`FloatBruteForceIndex` (the float-lane linear scan, built once per model). +`VectorStore` decides which binary engine is active by comparing the live +fingerprint count against `mihThreshold`, and swaps between them without +rebuilding either — both are always kept in sync with every write. + +`ResidentArrayStore` manages the optional `.vec` sidecar file: a packed, +fixed-format binary cache of the in-memory array. It is a regenerable +cache, never a second source of truth — if it is missing, stale, or +corrupted, VectorStore rebuilds it from the `vectors` table, the only +durable source. + +## What Ships in the Package + +The package ships the Swift sources listed above, a mirrored Rust +implementation in `rust/`, and no bundled data artifacts — VectorKit has +no fixed reference tables of its own; everything it stores comes from +caller-supplied text and caller-supplied embedding models. diff --git a/packages/kits/VectorKit/docs/topology.svg b/packages/kits/VectorKit/docs/topology.svg new file mode 100644 index 0000000..6541f48 --- /dev/null +++ b/packages/kits/VectorKit/docs/topology.svg @@ -0,0 +1,124 @@ + + + + + + + + + + + + + VectorKit: text in, ranked matches out + + + + EmbeddingProvider + text → Engram + floats + + + VectorStore + write + search API + + + VectorMatch + ranked results + + + + + + + Durable storage boundary (PersistenceKit) + + vectors table + SQLite / InMemory / … + + + + Optional on-disk cache (regenerable) + + ResidentArrayStore + .vec sidecar, mmap load + + + + + + + ResidentVectorArray + packed in-memory contract, all engines read this + + + + + + + BruteForceIndex + Lane A · binary oracle + Hamming, always active below threshold + + + MIHIndex + Lane B · sub-linear exact + active at/above mihThreshold + + + FloatBruteForceIndex + Lane C/D · one per modelID + cosine / l2 / dot + + + + + + + + + + DenseIndex.search dispatch (hotIndex or per-model float index) + + + + Standalone (not wired into VectorStore) + + MaxSimScorer + Lane E1 · ColBERT MaxSim + From 331449602b479f414547d2862664a86bd8ded148 Mon Sep 17 00:00:00 2001 From: Fable Date: Sat, 4 Jul 2026 22:15:04 -0500 Subject: [PATCH 2/2] =?UTF-8?q?docs(sdk):=20rewrite=20prose=20to=20the=20f?= =?UTF-8?q?our=20immutable=20laws=20=E2=80=94=20FK=20<=3D=208.0,=20no=20em?= =?UTF-8?q?-dashes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full rewrite of OVERVIEW.md and DETAILS.md per package: Turabian/Chicago, reading level 8.0 or less, no sentence over 2 commas outside simple lists, no 'not X, but Y' or 'it is not X, it is Y' forms. AGENT_MAP and topology.svg scrubbed of em-dashes only. Front matter unchanged. Gated by scripts/docs-style-lint.py (mootx01-ee): 65/65 pass; docs-scrub clean. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Pzkqg3bpbbhfuWRsYomJ3o --- packages/kits/CorpusKit/docs/AGENT_MAP.md | 230 +-- packages/kits/CorpusKit/docs/DETAILS.md | 1365 +++++++------- packages/kits/CorpusKit/docs/OVERVIEW.md | 233 +-- packages/kits/LocusKit/docs/AGENT_MAP.md | 600 +++--- packages/kits/LocusKit/docs/DETAILS.md | 2062 +++++++++++---------- packages/kits/LocusKit/docs/OVERVIEW.md | 360 ++-- packages/kits/LocusKit/docs/topology.svg | 2 +- packages/kits/VectorKit/docs/AGENT_MAP.md | 248 +-- packages/kits/VectorKit/docs/DETAILS.md | 1025 +++++----- packages/kits/VectorKit/docs/OVERVIEW.md | 282 +-- 10 files changed, 3343 insertions(+), 3064 deletions(-) diff --git a/packages/kits/CorpusKit/docs/AGENT_MAP.md b/packages/kits/CorpusKit/docs/AGENT_MAP.md index 075f5ab..cba0ea4 100644 --- a/packages/kits/CorpusKit/docs/AGENT_MAP.md +++ b/packages/kits/CorpusKit/docs/AGENT_MAP.md @@ -75,152 +75,152 @@ sources: blob: e72231cf3e50799b6bbeac6a165b80410dc40317 --- -# AGENT_MAP — CorpusKit +# AGENT_MAP : CorpusKit PURPOSE: standalone on-device RAG kit. Text → chunks (content-addressed UUID) → BundleStore (PersistenceKit) + persistent BM25 inverted index + per-provider vectors (VectorKit) → hybrid recall (Hamming kNN + BM25, weighted RRF) → [ScoredChunk]. Ships two targets: CorpusKit (core: stores, engines, protocols) and CorpusKitProviders (concrete embedding providers/tokenizers). Default production ensemble = five honest deterministic signals (RI/PPMI/LSA/NMF/FDC). DEPS: CorpusKit imports SubstrateTypes, SubstrateLib (MerkleHash), SubstrateML, EngramLib, EideticLib (sentence segmentation), IntellectusLib (telemetry, off-by-default), PersistenceKit (+InMemory, +SQLite), ConvergenceKit (manifest only), VectorKit, QueueKit, Crypto. CorpusKitProviders additionally imports SubstrateKernel (FloatVecOps), LatticeLib (FDC runtime; FDC math NOT reimplemented). Imported by: GeniusLocusKit (orchestrator tier). Rust ports: `rust/` (core, crate corpus-kit) + `rust-providers/` (crate corpus-kit-providers); shared fixtures `Tests/SharedVectors/*.json` read by BOTH legs gate bit-identity. NL providers are Swift-only (ADR-019, no Rust twin). ENTRY POINTS (most callers need only these): -- CorpusKit.swift:702 `Corpus.init(storage:models:)` — open estate corpus; `models[0]` = default signal (:674 single-model convenience) -- CorpusKit.swift:1001 `Corpus.ingest(_ text:sourceID:now:)` — synchronous chunk+index+embed -- CorpusIngestQueue.swift:157 `Corpus.enqueueIngest(_:sourceID:now:)` — async queued ingest (production path) -- CorpusKit.swift:1629 `Corpus.recall(_ query:limit:now:) -> [ScoredChunk]` — hybrid RRF recall on default signal -- DefaultEnsemble.swift:62 `CorpusEnsemble.defaultEnsemble() -> [EmbeddingModel]` — the five production signals, fresh per estate +- CorpusKit.swift:702 `Corpus.init(storage:models:)` : open estate corpus; `models[0]` = default signal (:674 single-model convenience) +- CorpusKit.swift:1001 `Corpus.ingest(_ text:sourceID:now:)` : synchronous chunk+index+embed +- CorpusIngestQueue.swift:157 `Corpus.enqueueIngest(_:sourceID:now:)` : async queued ingest (production path) +- CorpusKit.swift:1629 `Corpus.recall(_ query:limit:now:) -> [ScoredChunk]` : hybrid RRF recall on default signal +- DefaultEnsemble.swift:62 `CorpusEnsemble.defaultEnsemble() -> [EmbeddingModel]` : the five production signals, fresh per estate ## Symbol Table -### Facade — CorpusKit.swift -- :48 `enum FloatLaneOutcome` — `.hits`/`.unavailableProviderOptOut`/`.unavailableNoVocabHit`/`.unavailableNoFloatRows`/`.emptyQuery`/`.storeError`; dark lanes are typed outcomes, NEVER errors -- :134 `enum EmbeddingModel` — `.deterministic` (:147, seed 0xC05B_D15C_A15D_1B00, federation baseline), `.miniLM/.mpNet/.embeddingGemma(inference:)` (:157/:167/:177, host closure), `.randomIndexing/.ppmi/.lsa/.nmf(provider:)` (:194–:233, trainable), `.fdc(provider:)` (:254, stateless), `.nlEmbedding/.nlContextualEmbedding` (:275/:293, Apple-only); `.default = .deterministic` (:297) -- :338 `EmbeddingModel.isTrainable` — true iff carried provider conforms to TrainableEmbeddingBasis -- :362 `EmbeddingModel.reconstruct(from: Data)` — routes blob to concrete type; throws `.notTrainable` -- :417 `enum EncodeSpeed` — `.foreground` (all cores) / `.background` (cores/4, floor 1) -- :426 `public actor Corpus` — composition root; sealed-vector principle (no VectorKit type in public API except :649 `sharedVectorStore`) +### Facade : CorpusKit.swift +- :48 `enum FloatLaneOutcome` : `.hits`/`.unavailableProviderOptOut`/`.unavailableNoVocabHit`/`.unavailableNoFloatRows`/`.emptyQuery`/`.storeError`; dark lanes are typed outcomes, NEVER errors +- :134 `enum EmbeddingModel` : `.deterministic` (:147, seed 0xC05B_D15C_A15D_1B00, federation baseline), `.miniLM/.mpNet/.embeddingGemma(inference:)` (:157/:167/:177, host closure), `.randomIndexing/.ppmi/.lsa/.nmf(provider:)` (:194–:233, trainable), `.fdc(provider:)` (:254, stateless), `.nlEmbedding/.nlContextualEmbedding` (:275/:293, Apple-only); `.default = .deterministic` (:297) +- :338 `EmbeddingModel.isTrainable` : true iff carried provider conforms to TrainableEmbeddingBasis +- :362 `EmbeddingModel.reconstruct(from: Data)` : routes blob to concrete type; throws `.notTrainable` +- :417 `enum EncodeSpeed` : `.foreground` (all cores) / `.background` (cores/4, floor 1) +- :426 `public actor Corpus` : composition root; sealed-vector principle (no VectorKit type in public API except :649 `sharedVectorStore`) - :491 `setEncodeSpeed(_:)`; :641 `onEncoded` callback (set via CorpusIngestQueue.swift:134 `setOnEncoded`) -- :1001 `ingest(_:sourceID:now:)` — idempotent; re-ingest clears tombstone; first-ingest auto-train (gate = no persisted basis, NOT factory-blob presence) -- :1181 `ingestBatch(_:)` — identical output to per-item; commit windows 512 items / 4096 rows (:436–:437); slice-parallel embed; batch-aware first-basis bootstrap (train once on full batch, never per item) -- :1416 `maintainedVocabAnchor() -> Int` — governor's vocab-growth retrain trigger read -- :1488 `reindex(now:)` — THE explicit retrain: reconstruct-fresh → trainOnCorpus(active chunks) → upsert basis → re-embed all under every slot; excludes removed sources +- :1001 `ingest(_:sourceID:now:)` : idempotent; re-ingest clears tombstone; first-ingest auto-train (gate = no persisted basis, NOT factory-blob presence) +- :1181 `ingestBatch(_:)` : identical output to per-item; commit windows 512 items / 4096 rows (:436–:437); slice-parallel embed; batch-aware first-basis bootstrap (train once on full batch, never per item) +- :1416 `maintainedVocabAnchor() -> Int` : governor's vocab-growth retrain trigger read +- :1488 `reindex(now:)` : THE explicit retrain: reconstruct-fresh → trainOnCorpus(active chunks) → upsert basis → re-embed all under every slot; excludes removed sources - :1629 `recall(_:limit:now:)`; :1657 `remove(sourceID:)` (BM25 rows + ALL models' vectors + tombstone; chunks kept); :1711 `expunge(sourceID:)` (scrubText FIRST, then remove); :1738 `destroyRecallIndex()` (all derived state; chunks survive) -- :1779 `bm25TopKBySource(query:limit:)` — pure keyword lane, max-chunk-score per source, frontierK ≤ 256 +- :1779 `bm25TopKBySource(query:limit:)` : pure keyword lane, max-chunk-score per source, frontierK ≤ 256 - :1833 `embed(_:) -> Engram`; :1846 `modelID`; :1863 `embedFloat(_:)` (throws on provider opt-out); :2238 `supportsFloat` -- :1895 `floatNearest(query:limit:) -> FloatLaneOutcome` — never throws; sim = 1 − distance/10_000; source aggregation = MAX chunk cosine -- :2141 `floatNearestPerSignal(query:limit:)` — per-slot dense lanes in slot order, NO fusion (caller fuses); :2214 `floatFarthestPerSignal` — anti-similarity, MIN chunk cosine per source, ascending +- :1895 `floatNearest(query:limit:) -> FloatLaneOutcome` : never throws; sim = 1 − distance/10_000; source aggregation = MAX chunk cosine +- :2141 `floatNearestPerSignal(query:limit:)` : per-slot dense lanes in slot order, NO fusion (caller fuses); :2214 `floatFarthestPerSignal` : anti-similarity, MIN chunk cosine per source, ascending - :2249 `count()` (excludes removed sources); :2266 `indexedSourceIDs()`; :2274/:2280 `corpusMerkleRoot(for:)`/`globalCorpusMerkleRoot()` -- :2300 `EmbeddingModel.makeProvider()` (private) — pinned seeds: miniLM 0x4D49_4E4C_4D5F_7631 "MINLM_v1", mpNet 0x4D50_4E45_545F_7631 "MPNET_v1", embeddingGemma 0x454D_4247_4D5F_7631 "EMBGM_v1"; model IDs corpus-deterministic-v1 / minilm-v6 / mpnet-base-v2 / embedding-gemma-300m -- :2413 `CorpusDefaultTokenizer` (internal) — FNV-1a fold, duplicated from providers to avoid circular dep; :2445 `CorpusTextProvider` (private) — tokenize→inference→FloatSimHash; :2498 `embedPair` computes pooled vector ONCE for both lanes +- :2300 `EmbeddingModel.makeProvider()` (private) : pinned seeds: miniLM 0x4D49_4E4C_4D5F_7631 "MINLM_v1", mpNet 0x4D50_4E45_545F_7631 "MPNET_v1", embeddingGemma 0x454D_4247_4D5F_7631 "EMBGM_v1"; model IDs corpus-deterministic-v1 / minilm-v6 / mpnet-base-v2 / embedding-gemma-300m +- :2413 `CorpusDefaultTokenizer` (internal) : FNV-1a fold, duplicated from providers to avoid circular dep; :2445 `CorpusTextProvider` (private) : tokenize→inference→FloatSimHash; :2498 `embedPair` computes pooled vector ONCE for both lanes - Test seams (never production): :914 init(storage:provider:), :980 `_testForceFloatStoreError`, :656 `_ingestFailureHook` -### Ingest queue — CorpusIngestQueue.swift (extension Corpus) -- :63 `mountIngestQueue()` — idempotent; SQLite estate → encrypted sibling `queue.sqlite` via `EstateConfiguration.queueSibling` (ADR-021 D7/T4; replaced plaintext maildir hole); InMemory estate → fixed :486 `ingestQueueStoreID` (no UUID() nondeterminism) -- :120 `dropIngestQueue()`; :134 `setOnEncoded(_:)` — the ONLY CorpusKit→orchestrator callback -- :157 `enqueueIngest(...)`; :179 `enqueueIngestBatch(...)` — one transaction for all jobs (bulk-import bottleneck fix; caller bounds batch size) -- :205 `awaitIngestDrain(timeout: 30s)` — barrier: drained AND vector index republished; throws drainTimeout -- :229 `ingestQueueDepth() -> (pending, inFlight)` — pending IS stream-scoped, inFlight is NOT (all streams) -- :248 `drainIngestQueueOnce()` — claims whole batch; undecodable → `.blocked` (terminal), empty text → `.done`; batch `ingestBatch` + bulk session reply; falls back serial on batch throw -- :341 `runIngestDrainLoop` (private) — DrainLease single drainer; first-acquire crash recovery `reclaimInFlight`; standby poll 3 s; lease TTL 15 s (failover ≈ 15–18 s, not instant); spin-while-draining, 15 ms idle sleep; vector index published once per burst (O(N) not O(N²)) -- :430 `ingestOneAndReply` (private) — retry in place ≤ :476 `ingestMaxAttempts = 8`, then `.blocked`; sound ONLY because ingest is idempotent -- :480 `encodeStreamID = "encode"` — EVERY queue op must be scoped to it (shared queue.sqlite may carry other streams; unscoped awaitDrain deadlocks) -- :515 `IngestJob` — wire fields `sourceID`/`text`/`capturedAtISO8601` = pinned cross-port JSON contract; :551 `toJob`, :563 `from(job:)` - -### Chunks — Chunk.swift / Chunker.swift -- Chunk.swift:35 `struct Chunk: Sendable, Equatable, Codable` — immutable, content-addressed +### Ingest queue : CorpusIngestQueue.swift (extension Corpus) +- :63 `mountIngestQueue()` : idempotent; SQLite estate → encrypted sibling `queue.sqlite` via `EstateConfiguration.queueSibling` (ADR-021 D7/T4; replaced plaintext maildir hole); InMemory estate → fixed :486 `ingestQueueStoreID` (no UUID() nondeterminism) +- :120 `dropIngestQueue()`; :134 `setOnEncoded(_:)` : the ONLY CorpusKit→orchestrator callback +- :157 `enqueueIngest(...)`; :179 `enqueueIngestBatch(...)` : one transaction for all jobs (bulk-import bottleneck fix; caller bounds batch size) +- :205 `awaitIngestDrain(timeout: 30s)` : barrier: drained AND vector index republished; throws drainTimeout +- :229 `ingestQueueDepth() -> (pending, inFlight)` : pending IS stream-scoped, inFlight is NOT (all streams) +- :248 `drainIngestQueueOnce()` : claims whole batch; undecodable → `.blocked` (terminal), empty text → `.done`; batch `ingestBatch` + bulk session reply; falls back serial on batch throw +- :341 `runIngestDrainLoop` (private) : DrainLease single drainer; first-acquire crash recovery `reclaimInFlight`; standby poll 3 s; lease TTL 15 s (failover ≈ 15–18 s, not instant); spin-while-draining, 15 ms idle sleep; vector index published once per burst (O(N) not O(N²)) +- :430 `ingestOneAndReply` (private) : retry in place ≤ :476 `ingestMaxAttempts = 8`, then `.blocked`; sound ONLY because ingest is idempotent +- :480 `encodeStreamID = "encode"` : EVERY queue op must be scoped to it (shared queue.sqlite may carry other streams; unscoped awaitDrain deadlocks) +- :515 `IngestJob` : wire fields `sourceID`/`text`/`capturedAtISO8601` = pinned cross-port JSON contract; :551 `toJob`, :563 `from(job:)` + +### Chunks : Chunk.swift / Chunker.swift +- Chunk.swift:35 `struct Chunk: Sendable, Equatable, Codable` : immutable, content-addressed - Chunk.swift:68 content-addressed init (normal path); :90 explicit-id init (reconstruction; caller must guarantee id matches content) -- Chunk.swift:129 `Chunk.deriveID(sourceID:startOffset:text:)` — RFC 4122 v5 (SHA-1) over fields joined by \u{1F}; :114 `namespaceBytes` PERMANENT (change re-keys fleet + breaks vector join) -- Chunk.swift:149 `struct ScoredChunk` — chunk + score/vectorScore/keywordScore (per-lane preserved) -- Chunker.swift:28 `ChunkerConfiguration` — targetChars 800 / overlapChars 100 / respectSentences true; init clamps (overlap < target) -- Chunker.swift:49 `Chunker.chunk(text:sourceID:configuration:hlcGenerator:)` — EideticLib.sentences segmentation; greedy fill + tail overlap; hlcGenerator `inout`, stamps emission order; offsets are Character counts - -### Tokenization — Tokenizer.swift -- :10 `protocol Tokenizer: Sendable` — vocabID/maxTokens/padTokenID/unknownTokenID; :27 `tokenize(_:) -> [Int32]` (truncation is implementer's job); :33 `keywordTokens(_:)` (default :62) -- :44 `defaultKeywordTokens(_:)` — lowercase + alphabetic/ASCII-digit runs; THE single keyword tokenizer for BM25 + all distributional providers; parity-critical with Rust; overriding keywordTokens breaks the guarantee (convention, not compiler) - -### Errors — CorpusKitError.swift -- :5 `enum CorpusKitError: Error, Sendable, Equatable` — encodingFailure/decodingFailure/tokenizerUnavailable/modelUnavailable/embeddingFailed/storeUnavailable/:18 notTrainable; Equatable on exact message strings - -### Chunk store — BundleStore.swift -- :94 `public actor BundleStore`; :139 `schemaDeclaration` v3 — `chunks` (10 cols incl. content_hash BLOB, ext JSON) + `corpus_metadata`; indices source_id, hlc -- :199 `init(storage:dirtyChainSink:)` — wraps HashingRowStore (MerkleHash.leaf per insert); :70 `ParentChainCache` bridges sync hash callback -- :268 `insert(_:)` — idempotent (duplicateKey = no-op, first write wins); RETURNS ONLY NEWLY-INSERTED chunks — derived-state callers must fold the returned subset, never the input -- :348 `get(id:asOf:)`; :361 `getMany(ids:asOf:)`; :375 `chunksForSource(_:asOf:)` (start_offset ASC); :399 `allSourceIDs` (full scan, maintenance only); :427 `chunkSourcePairs()` (body-free warm-load projection); :457 `count(asOf:)` — asOf accepted but IGNORED; :461 `allChunks(asOf:)` (hlc ASC) -- :500 `scrubText(sourceID:)` — hard-delete text zeroing via direct UPDATE (why schema appendOnly: false); leaves content_hash stale intentionally -- :564 `corpusMerkleRoot(for:)`; :583 `globalCorpusMerkleRoot()` — corpus/root UUIDs derived from fixed SHA-256 namespace strings (cross-port) -- :632 `decodeChunk(_:)` — MUST accept both TypedValue forms (SQLite primitive .text/.int AND InMemory semantic .uuid/.hlc); historical bug: semantic-only decoder dropped all chunks on reopen; InMemory-only tests cannot catch regressions here - -### Tombstones — RemovedSourceStore.swift +- Chunk.swift:129 `Chunk.deriveID(sourceID:startOffset:text:)` : RFC 4122 v5 (SHA-1) over fields joined by \u{1F}; :114 `namespaceBytes` PERMANENT (change re-keys fleet + breaks vector join) +- Chunk.swift:149 `struct ScoredChunk` : chunk + score/vectorScore/keywordScore (per-lane preserved) +- Chunker.swift:28 `ChunkerConfiguration` : targetChars 800 / overlapChars 100 / respectSentences true; init clamps (overlap < target) +- Chunker.swift:49 `Chunker.chunk(text:sourceID:configuration:hlcGenerator:)` : EideticLib.sentences segmentation; greedy fill + tail overlap; hlcGenerator `inout`, stamps emission order; offsets are Character counts + +### Tokenization : Tokenizer.swift +- :10 `protocol Tokenizer: Sendable` : vocabID/maxTokens/padTokenID/unknownTokenID; :27 `tokenize(_:) -> [Int32]` (truncation is implementer's job); :33 `keywordTokens(_:)` (default :62) +- :44 `defaultKeywordTokens(_:)` : lowercase + alphabetic/ASCII-digit runs; THE single keyword tokenizer for BM25 + all distributional providers; parity-critical with Rust; overriding keywordTokens breaks the guarantee (convention, not compiler) + +### Errors : CorpusKitError.swift +- :5 `enum CorpusKitError: Error, Sendable, Equatable` : encodingFailure/decodingFailure/tokenizerUnavailable/modelUnavailable/embeddingFailed/storeUnavailable/:18 notTrainable; Equatable on exact message strings + +### Chunk store : BundleStore.swift +- :94 `public actor BundleStore`; :139 `schemaDeclaration` v3 : `chunks` (10 cols incl. content_hash BLOB, ext JSON) + `corpus_metadata`; indices source_id, hlc +- :199 `init(storage:dirtyChainSink:)` : wraps HashingRowStore (MerkleHash.leaf per insert); :70 `ParentChainCache` bridges sync hash callback +- :268 `insert(_:)` : idempotent (duplicateKey = no-op, first write wins); RETURNS ONLY NEWLY-INSERTED chunks : derived-state callers must fold the returned subset, never the input +- :348 `get(id:asOf:)`; :361 `getMany(ids:asOf:)`; :375 `chunksForSource(_:asOf:)` (start_offset ASC); :399 `allSourceIDs` (full scan, maintenance only); :427 `chunkSourcePairs()` (body-free warm-load projection); :457 `count(asOf:)` : asOf accepted but IGNORED; :461 `allChunks(asOf:)` (hlc ASC) +- :500 `scrubText(sourceID:)` : hard-delete text zeroing via direct UPDATE (why schema appendOnly: false); leaves content_hash stale intentionally +- :564 `corpusMerkleRoot(for:)`; :583 `globalCorpusMerkleRoot()` : corpus/root UUIDs derived from fixed SHA-256 namespace strings (cross-port) +- :632 `decodeChunk(_:)` : MUST accept both TypedValue forms (SQLite primitive .text/.int AND InMemory semantic .uuid/.hlc); historical bug: semantic-only decoder dropped all chunks on reopen; InMemory-only tests cannot catch regressions here + +### Tombstones : RemovedSourceStore.swift - :44 `public actor RemovedSourceStore`; :52 schemaDeclaration (own kitID "CorpusKitRemovedSources") -- :76 `markRemoved(_:now:)` (idempotent upsert; row presence IS the state — no Bool column); :90 `clearRemoved(_:)` (re-ingest = the undo); :99 `removedIDs() -> Set` — EVERY rebuild path (reindex, first-ingest train, count) must subtract this set (unenforced convention; resurrection bug class); :118 `deleteAll()` +- :76 `markRemoved(_:now:)` (idempotent upsert; row presence IS the state : no Bool column); :90 `clearRemoved(_:)` (re-ingest = the undo); :99 `removedIDs() -> Set` : EVERY rebuild path (reindex, first-ingest train, count) must subtract this set (unenforced convention; resurrection bug class); :118 `deleteAll()` -### Provider counts — CorpusProviderCountsStore.swift +### Provider counts : CorpusProviderCountsStore.swift - :71 `PersistedCounts` (modelID/modelVersion/counts blob/documentCount/vocabSize/updatedAt); :101 `CountsGrowthAnchor` (cheap pair, no blob) - :112 `public actor CorpusProviderCountsStore`; :121 schema (kitID "CorpusKitCounts", PK (model_id, model_version), ext slot) -- :156 `upsert(_:)` — full-row replace (caller folds blob first, no atomic increment); :173 `load(...)`; :191 `growthAnchor(...)` — the staleness-check read; :210 `deleteAll()` -- STATUS: "HALF A" — counts persisted/restored, but `Corpus.reindex` still retrains from raw chunk text; counts-backed retrain + vector re-projection (HALF B) not wired +- :156 `upsert(_:)` : full-row replace (caller folds blob first, no atomic increment); :173 `load(...)`; :191 `growthAnchor(...)` : the staleness-check read; :210 `deleteAll()` +- STATUS: "HALF A" : counts persisted/restored, but `Corpus.reindex` still retrains from raw chunk text; counts-backed retrain + vector re-projection (HALF B) not wired -### Sync — SyncManifest.swift -- :11 `enum CorpusKitSync`; :17 `manifest(zoneIdentifier:)` — chunks table, bidirectional, PK id, conflictPolicy `.appendOnly` (safe because content-addressed); kitID "CorpusKit", schemaVersion 1. Declarative only. Sync-layer appendOnly ≠ BundleStore schema `appendOnly: false` — different systems, same word +### Sync : SyncManifest.swift +- :11 `enum CorpusKitSync`; :17 `manifest(zoneIdentifier:)` : chunks table, bidirectional, PK id, conflictPolicy `.appendOnly` (safe because content-addressed); kitID "CorpusKit", schemaVersion 1. Declarative only. Sync-layer appendOnly ≠ BundleStore schema `appendOnly: false` : different systems, same word -### Sparse engine — Engine/SparseTypes.swift, Engine/BM25Weighting.swift, Engine/InvertedIndex.swift +### Sparse engine : Engine/SparseTypes.swift, Engine/BM25Weighting.swift, Engine/InvertedIndex.swift - SparseTypes.swift:39 `typealias LaneTag = VectorKit.LaneTag` (single owner, avoids ambiguity); :56 `ImpactPosting` (impact Int32, quantized ONCE at build); :94 `SparseHit` (impact Float = int/100); :136 `FusedHit` (fusedScore + perLane raw scores; absent key = no hit in lane) -- BM25Weighting.swift:29 `BM25Parameters` — k1 1.5 / b 0.75 pinned defaults; :50 `quantizeImpact(_:)` — round HALF-TO-EVEN × 100 (Swift default rounding differs at .5 — do not "simplify"); :82 `buildTermIDMap` (sorted term-id assignment); :110 `build(termFreqs:docLengths:parameters:)` — float BM25 math exactly once; IDF = ln((N−df+0.5)/(df+0.5)+1); :164 `queryPairs` — OOV dropped, duplicates deduped, weight = 100 -- InvertedIndex.swift:37 `invertedIndexQuantScale = 100`; :42 `invertedIndexBlockSize = 128` (pinned for conformance traces); :116 `struct InvertedIndex: Sendable` — immutable after init (init sorts postings itemID ASC); :195 `enum Algorithm` .wand/.blockMaxWand (result-identical); :211 `topK(query:k:algorithm:)` — EXACT top-k, integer-only path, tie-break smaller itemID wins; :246 `exhaustiveScan(query:k:)` — DAAT conformance oracle, not production. Item IDs compare as STRINGS (uuidString lexicographic ≠ numeric UUID order, but consistent cross-port) - -### Persistent keyword index — Engine/InvertedIndexStore.swift -- :47 `public actor InvertedIndexStore`; :55 schemaDeclaration (kitID "InvertedIndexStore", tables iix_termfreqs/iix_doclens — RAW statistics only, weighted index derived+cached, so k1/b changes need no migration); :105 init (storage pre-opened/migrated); :114 `open()` (load mirrors, O(terms+docs), no chunk bodies) -- :159 `index(itemID:tokens:now:)` — atomic replace, idempotent; empty tokens = removal; `now` unused (determinism discipline); :199 `remove(itemID:)`; :230 `buildIndex(parameters:)` (cached; invalidated per write); :253 `topK(queryTerms:k:parameters:algorithm:)`; :273 `deleteAll()`; :295 `documentCount` -- Rust twin owns a PRIVATE connection with begin/commit/rollback_batch; Swift shares estate storage — hence Corpus-managed transaction windows in ingestBatch - -### Legacy keyword index — BM25Index.swift -- :34 `public actor BM25Index` — in-memory, Chunk/UUID-typed; NO LONGER used by Corpus (kept public for external callers); :49 init(tokenizer:parameters:); :58 `index(_ chunks:)`; :78 `remove(_:)`; :95 `documentCount()`; :107 `topK(_ k:for tokens:)` — pre-tokenized input; tie-break uuidString ASC - -### Fusion — Engine/Fusion.swift -- :48 `enum Fusion`; :74 `fuse(rankedLists:laneScores:weights:rrfK: 60)` — fusedScore = Σ weight·1/(rrfK+rank), rank 1-based; per-lane dedup (best rank only); precondition rrfK > 0; sort fusedScore DESC, itemID ASC; :164 `fuse(scoredLists:weights:rrfK:)` — position = rank, CALLER must pre-sort -- MMR: `mmrLambda` exists only as a HybridRecallConfiguration field — NOT implemented anywhere; do not document MMR as active - -### Hybrid recall — HybridRecall.swift -- :33 `HybridRecallConfiguration` — vectorWeight 0.6 / keywordWeight 0.4 / rrfK 60 / mmrLambda nil (unread); :52 `enum HybridRecall` -- :84 `recall(probe:query:modelID:limit:vectorStore:invertedIndex:bundleStore:configuration:)` — candidateK = max(limit×4, 32) per lane; vector kNN filtered to modelID; UUID hits CANONICALIZED via UUID(uuidString:).uuidString (P3-secfix: lowercase Rust-written ids must fuse with uppercase Swift ids); vectorScore = Hamming (0 = best, kept), keywordScore 0 → nil (BM25 real matches strictly positive); unhydratable ids silently dropped; telemetry post-hoc (corpuskit.recall.*) - -### Trainable-basis seam — TrainableEmbeddingBasis.swift, BasisStore.swift -- TrainableEmbeddingBasis.swift:50 `protocol TrainableEmbeddingBasis: AnyObject, Sendable` — conformers: RI/PPMI/LSA/NMF only -- :70 `trainOnCorpus(texts:)` — ADDITIVE (never retrain a live provider — reconstruct fresh first); :78 `serializeBasis()`; :98 `reconstructBasis(from:)` — INSTANCE method (type-erased witness routes to concrete init(deserializing:)); round-trip law = identical embeddings; throws decodingFailure, never crashes -- :132 `addToCounts(text:)` / :141 `serializeCounts()` / :149 `restoreCounts(from:)` / :154 `countsVocabularySize` — P3 maintained-counts seam, batch-boundary snapshots; infrastructure only (reindex still trains from text) -- No wall-clock reads anywhere in training — pure function of (texts, seeds) +- BM25Weighting.swift:29 `BM25Parameters` : k1 1.5 / b 0.75 pinned defaults; :50 `quantizeImpact(_:)` : round HALF-TO-EVEN × 100 (Swift default rounding differs at .5 : do not "simplify"); :82 `buildTermIDMap` (sorted term-id assignment); :110 `build(termFreqs:docLengths:parameters:)` : float BM25 math exactly once; IDF = ln((N−df+0.5)/(df+0.5)+1); :164 `queryPairs` : OOV dropped, duplicates deduped, weight = 100 +- InvertedIndex.swift:37 `invertedIndexQuantScale = 100`; :42 `invertedIndexBlockSize = 128` (pinned for conformance traces); :116 `struct InvertedIndex: Sendable` : immutable after init (init sorts postings itemID ASC); :195 `enum Algorithm` .wand/.blockMaxWand (result-identical); :211 `topK(query:k:algorithm:)` : EXACT top-k, integer-only path, tie-break smaller itemID wins; :246 `exhaustiveScan(query:k:)` : DAAT conformance oracle, not production. Item IDs compare as STRINGS (uuidString lexicographic ≠ numeric UUID order, but consistent cross-port) + +### Persistent keyword index : Engine/InvertedIndexStore.swift +- :47 `public actor InvertedIndexStore`; :55 schemaDeclaration (kitID "InvertedIndexStore", tables iix_termfreqs/iix_doclens : RAW statistics only, weighted index derived+cached, so k1/b changes need no migration); :105 init (storage pre-opened/migrated); :114 `open()` (load mirrors, O(terms+docs), no chunk bodies) +- :159 `index(itemID:tokens:now:)` : atomic replace, idempotent; empty tokens = removal; `now` unused (determinism discipline); :199 `remove(itemID:)`; :230 `buildIndex(parameters:)` (cached; invalidated per write); :253 `topK(queryTerms:k:parameters:algorithm:)`; :273 `deleteAll()`; :295 `documentCount` +- Rust twin owns a PRIVATE connection with begin/commit/rollback_batch; Swift shares estate storage : hence Corpus-managed transaction windows in ingestBatch + +### Legacy keyword index : BM25Index.swift +- :34 `public actor BM25Index` : in-memory, Chunk/UUID-typed; NO LONGER used by Corpus (kept public for external callers); :49 init(tokenizer:parameters:); :58 `index(_ chunks:)`; :78 `remove(_:)`; :95 `documentCount()`; :107 `topK(_ k:for tokens:)` : pre-tokenized input; tie-break uuidString ASC + +### Fusion : Engine/Fusion.swift +- :48 `enum Fusion`; :74 `fuse(rankedLists:laneScores:weights:rrfK: 60)` : fusedScore = Σ weight·1/(rrfK+rank), rank 1-based; per-lane dedup (best rank only); precondition rrfK > 0; sort fusedScore DESC, itemID ASC; :164 `fuse(scoredLists:weights:rrfK:)` : position = rank, CALLER must pre-sort +- MMR: `mmrLambda` exists only as a HybridRecallConfiguration field : NOT implemented anywhere; do not document MMR as active + +### Hybrid recall : HybridRecall.swift +- :33 `HybridRecallConfiguration` : vectorWeight 0.6 / keywordWeight 0.4 / rrfK 60 / mmrLambda nil (unread); :52 `enum HybridRecall` +- :84 `recall(probe:query:modelID:limit:vectorStore:invertedIndex:bundleStore:configuration:)` : candidateK = max(limit×4, 32) per lane; vector kNN filtered to modelID; UUID hits CANONICALIZED via UUID(uuidString:).uuidString (P3-secfix: lowercase Rust-written ids must fuse with uppercase Swift ids); vectorScore = Hamming (0 = best, kept), keywordScore 0 → nil (BM25 real matches strictly positive); unhydratable ids silently dropped; telemetry post-hoc (corpuskit.recall.*) + +### Trainable-basis seam : TrainableEmbeddingBasis.swift, BasisStore.swift +- TrainableEmbeddingBasis.swift:50 `protocol TrainableEmbeddingBasis: AnyObject, Sendable` : conformers: RI/PPMI/LSA/NMF only +- :70 `trainOnCorpus(texts:)` : ADDITIVE (never retrain a live provider : reconstruct fresh first); :78 `serializeBasis()`; :98 `reconstructBasis(from:)` : INSTANCE method (type-erased witness routes to concrete init(deserializing:)); round-trip law = identical embeddings; throws decodingFailure, never crashes +- :132 `addToCounts(text:)` / :141 `serializeCounts()` / :149 `restoreCounts(from:)` / :154 `countsVocabularySize` : P3 maintained-counts seam, batch-boundary snapshots; infrastructure only (reindex still trains from text) +- No wall-clock reads anywhere in training : pure function of (texts, seeds) - Rust divergence: EmbeddingProvider is a supertrait there (no trait cross-cast) - BasisStore.swift:67 `PersistedBasis`; :98 `public actor BasisStore`; :112 schema v2 (`corpus_provider_basis`, PK (model_id, model_version), trained_at TEXT ISO8601, trained_chunk_count anchor, ext slot); :151 `upsert(_:)` (in-place, one row per key); :177 `load(modelID:modelVersion:)`; :195 `deleteAll()`; decoder accepts BOTH TypedValue timestamp forms (same reopen-bug class as BundleStore) -### Basis codec — BasisCodec.swift (CorpusKitProviders) -- :43 `basisFormatVersion: UInt8 = 1`; :52 `struct BasisWriter` — LE only; f32 as bitPattern (:94); strings u32-len UTF-8 (:99); maps byte-sorted keys (:122/:136 via :148 lhsLess raw-UTF-8 compare — matches Rust str Ord, NOT Swift String <) -- :168 `struct BasisReader` — :192 `expectMagic` / :206 `expectVersion` reject wrong/future blobs with decodingFailure; all reads bounds-checked. Frame = MAGIC(4) | version(1) | payload. No nested-map primitive (PPMI inlines its own) +### Basis codec : BasisCodec.swift (CorpusKitProviders) +- :43 `basisFormatVersion: UInt8 = 1`; :52 `struct BasisWriter` : LE only; f32 as bitPattern (:94); strings u32-len UTF-8 (:99); maps byte-sorted keys (:122/:136 via :148 lhsLess raw-UTF-8 compare : matches Rust str Ord, NOT Swift String <) +- :168 `struct BasisReader` : :192 `expectMagic` / :206 `expectVersion` reject wrong/future blobs with decodingFailure; all reads bounds-checked. Frame = MAGIC(4) | version(1) | payload. No nested-map primitive (PPMI inlines its own) -### Deterministic tokenizer — DeterministicTokenizer.swift -- :16 `struct DeterministicTokenizer: Tokenizer`; :23 init(vocabID "deterministic-v1", vocabSize 30522, maxTokens 128); :33 `tokenize(_:)` — FNV-1a 32 fold into [2, vocabSize); 0=pad 1=unk reserved; empty input → [pad]. NOT a model vocab — real-model output from these ids is garbage (real WordPiece/SentencePiece = v1.1 model-bundle mission) +### Deterministic tokenizer : DeterministicTokenizer.swift +- :16 `struct DeterministicTokenizer: Tokenizer`; :23 init(vocabID "deterministic-v1", vocabSize 30522, maxTokens 128); :33 `tokenize(_:)` : FNV-1a 32 fold into [2, vocabSize); 0=pad 1=unk reserved; empty input → [pad]. NOT a model vocab : real-model output from these ids is garbage (real WordPiece/SentencePiece = v1.1 model-bundle mission) -### Shared training inputs — TermDocumentCounts.swift, ReducedVocab.swift -- TermDocumentCounts.swift:56 `struct TermDocumentCounts` — :61 vocab (ENCOUNTER-ORDER indices, cross-port byte contract), :64 tfCounts, :68 dfCounts; :110 `addDocument(_:)` (full); :156 `addDocumentForCountsAnchor(_:)` (vocab+docCount only, no TF/DF — re-tokenize-at-retrain decision); :91 init(restoredVocab:documentCount:) — TF rows exist but EMPTY after restore; NOT thread-safe -- ReducedVocab.swift:33 `defaultReducedVocabCap = 512` (ADR-022); :38 `ReducedVocabulary` (keptTerms/termToColumn/fullIndexToColumn/size); :63 `selectReducedVocabulary(vocab:dfCounts:documentCount:cap:)` — below cap = exact NO-OP (fixture compatibility); above cap: drop df<2, rank df DESC, tie-break raw-UTF-8-byte order (Rust parity); documentCount reserved/unused +### Shared training inputs : TermDocumentCounts.swift, ReducedVocab.swift +- TermDocumentCounts.swift:56 `struct TermDocumentCounts` : :61 vocab (ENCOUNTER-ORDER indices, cross-port byte contract), :64 tfCounts, :68 dfCounts; :110 `addDocument(_:)` (full); :156 `addDocumentForCountsAnchor(_:)` (vocab+docCount only, no TF/DF : re-tokenize-at-retrain decision); :91 init(restoredVocab:documentCount:) : TF rows exist but EMPTY after restore; NOT thread-safe +- ReducedVocab.swift:33 `defaultReducedVocabCap = 512` (ADR-022); :38 `ReducedVocabulary` (keptTerms/termToColumn/fullIndexToColumn/size); :63 `selectReducedVocabulary(vocab:dfCounts:documentCount:cap:)` : below cap = exact NO-OP (fixture compatibility); above cap: drop df<2, rank df DESC, tie-break raw-UTF-8-byte order (Rust parity); documentCount reserved/unused -### Honest signals — RandomIndexingProvider.swift, PpmiProvider.swift, LsaProvider.swift, NmfProvider.swift, FdcProvider.swift -- RandomIndexing: :89 riDimension 2048, :93 riNonzeros 10, :98 riWindow 4, :102 riProjectionSeed 0x5249_5F56_315F_4D58 "RI_V1_MX"; :121 `riIndexVector(term:)` — FNV64(lowercased) → SplitMix64, EXACTLY 20 draws (pos %2048 bias-free, sign &1), collision last-wins (constant draw count = cross-port PRNG alignment); :162 `final class RandomIndexingProvider`; :206 `train(terms:window:)` additive; :249 embed / :268 embedFloat / :308 embedPair; :377 serializeBasis "RIB1" (vocab IS the basis, no finalize); :452 serializeCounts "RICT" (same payload, distinct magic on purpose) -- PPMI: :107/:111/:115 same D/K/window as RI (shared index space); :120 ppmiProjectionSeed 0x5050_4D49_5F56_314D "PPMI_V1M"; :153 `final class PpmiProvider`; :226 `train` (counts); :282 `finalize()` — ppmi(t,c)=max(0, ln P(t,c) − ln P(t) − ln P(c)); contextVec = Σ ppmi·riIndexVector(c); idempotent; :346 embed/:365 embedFloat/:397 embedPair; :464 basis "PPB1" (derived vectors only, unnormalized in store); :521 counts "PPMC" (raw additive state incl. nested coCount); NOT plain RI — do not reduce -- LSA: :109 lsaProjectionSeed 0x4C53415F56315F4D "LSA_V1_M"; :114 lsaDefaultRank 64; :165 svdSweeps PINNED 30 (change invalidates all conformance vectors); :148 `final class LsaProvider`; :227 `train(document:)`; :258 `finalize()` — ReducedVocab → tf=ln(1+c), idf=max(0, ln((N+1)/(df+1))) → JacobiSVD (wide matrix transposed+swapped back); query fold-in (1/σ)·Vt·q, σ<1e-9 skipped; :487 `documentEmbedding(at:)` exact U·Σ; :547 basis "LSB1" (reduced vocab + idf + RAW U/σ/Vt, port-neutral); :632 counts "LSAC" (anchors only) -- NMF: :110 nmfProjectionSeed 0x4E4D465F56315F4D "NMF_V1_M"; :114 rank 32; :119 iterations 100; :124 nmfFactorizationSeed 0xDEADBEEFCAFEBABE (pinned); :159 `final class NmfProvider`; :266 `finalize()` — V (terms×docs), log-TF NO idf, ALS with tolerance=0 → FIXED iteration count (bit-identity device); query fold-in dot(W[:,r],q)/(‖W[:,r]‖²+1e-9); :499 basis "NMB1" (W and H raw); :588 counts "NMFC" (anchors only) -- FDC: :106 fdcDimension 256; :111 fdcProjectionSeed 0x4644_435F_5631_5F50 "FDC_V1_P"; :137 `fdcNodeVector(code:)` — FNV64 → ONE SplitMix64 advance → LCG (Knuth 6364136223846793005 / 1442695040888963407) → 256 draws → l2Normalize (pipeline deliberately ≠ RI's); :242 `final class FDCProvider` — STATELESS, no training, no BasisCodec; path = FDC.ancestors + [code], node weight 1/(L+1); UNRESOLVED/empty → opt-out ([] / .zero) — honest, never guess; :284 embed/:299 embedFloat/:310 embedPair; determinism inherited from LatticeLib singletons +### Honest signals : RandomIndexingProvider.swift, PpmiProvider.swift, LsaProvider.swift, NmfProvider.swift, FdcProvider.swift +- RandomIndexing: :89 riDimension 2048, :93 riNonzeros 10, :98 riWindow 4, :102 riProjectionSeed 0x5249_5F56_315F_4D58 "RI_V1_MX"; :121 `riIndexVector(term:)` : FNV64(lowercased) → SplitMix64, EXACTLY 20 draws (pos %2048 bias-free, sign &1), collision last-wins (constant draw count = cross-port PRNG alignment); :162 `final class RandomIndexingProvider`; :206 `train(terms:window:)` additive; :249 embed / :268 embedFloat / :308 embedPair; :377 serializeBasis "RIB1" (vocab IS the basis, no finalize); :452 serializeCounts "RICT" (same payload, distinct magic on purpose) +- PPMI: :107/:111/:115 same D/K/window as RI (shared index space); :120 ppmiProjectionSeed 0x5050_4D49_5F56_314D "PPMI_V1M"; :153 `final class PpmiProvider`; :226 `train` (counts); :282 `finalize()` : ppmi(t,c)=max(0, ln P(t,c) − ln P(t) − ln P(c)); contextVec = Σ ppmi·riIndexVector(c); idempotent; :346 embed/:365 embedFloat/:397 embedPair; :464 basis "PPB1" (derived vectors only, unnormalized in store); :521 counts "PPMC" (raw additive state incl. nested coCount); NOT plain RI : do not reduce +- LSA: :109 lsaProjectionSeed 0x4C53415F56315F4D "LSA_V1_M"; :114 lsaDefaultRank 64; :165 svdSweeps PINNED 30 (change invalidates all conformance vectors); :148 `final class LsaProvider`; :227 `train(document:)`; :258 `finalize()` : ReducedVocab → tf=ln(1+c), idf=max(0, ln((N+1)/(df+1))) → JacobiSVD (wide matrix transposed+swapped back); query fold-in (1/σ)·Vt·q, σ<1e-9 skipped; :487 `documentEmbedding(at:)` exact U·Σ; :547 basis "LSB1" (reduced vocab + idf + RAW U/σ/Vt, port-neutral); :632 counts "LSAC" (anchors only) +- NMF: :110 nmfProjectionSeed 0x4E4D465F56315F4D "NMF_V1_M"; :114 rank 32; :119 iterations 100; :124 nmfFactorizationSeed 0xDEADBEEFCAFEBABE (pinned); :159 `final class NmfProvider`; :266 `finalize()` : V (terms×docs), log-TF NO idf, ALS with tolerance=0 → FIXED iteration count (bit-identity device); query fold-in dot(W[:,r],q)/(‖W[:,r]‖²+1e-9); :499 basis "NMB1" (W and H raw); :588 counts "NMFC" (anchors only) +- FDC: :106 fdcDimension 256; :111 fdcProjectionSeed 0x4644_435F_5631_5F50 "FDC_V1_P"; :137 `fdcNodeVector(code:)` : FNV64 → ONE SplitMix64 advance → LCG (Knuth 6364136223846793005 / 1442695040888963407) → 256 draws → l2Normalize (pipeline deliberately ≠ RI's); :242 `final class FDCProvider` : STATELESS, no training, no BasisCodec; path = FDC.ancestors + [code], node weight 1/(L+1); UNRESOLVED/empty → opt-out ([] / .zero) : honest, never guess; :284 embed/:299 embedFloat/:310 embedPair; determinism inherited from LatticeLib singletons - Common float-lane tri-state (four trainable providers): `[]` = structural opt-out (untrained) → .unavailableProviderOptOut; throw VectorKitError.embedFloatVocabMiss = trained-but-all-OOV → .unavailableNoVocabHit; vector = signal. `embedPair` collapses the vocab-miss throw to (.zero, []) on ALL providers. Training thread-contract everywhere: Sendable class, but train/finalize must complete before concurrent embeds; read-only after -### Ensemble factory — DefaultEnsemble.swift -- :38 `enum CorpusEnsemble`; :62 `defaultEnsemble()` — [.randomIndexing, .ppmi, .lsa, .nmf, .fdc] in pinned order ([0] = default signal); FUNCTION not constant (trainable providers are reference types — sharing one array would alias trained state across estates); returned trainables are UNTRAINED +### Ensemble factory : DefaultEnsemble.swift +- :38 `enum CorpusEnsemble`; :62 `defaultEnsemble()` : [.randomIndexing, .ppmi, .lsa, .nmf, .fdc] in pinned order ([0] = default signal); FUNCTION not constant (trainable providers are reference types : sharing one array would alias trained state across estates); returned trainables are UNTRAINED -### Named neural providers — MiniLMTextProvider.swift, MPNetTextProvider.swift, EmbeddingGemmaProvider.swift -- MiniLM: :40 `struct MiniLMTextProvider: EmbeddingProvider`; :48 `inference: @Sendable ([Int32]) async throws -> [Float]` (closure-injected CoreML seam, doctrine §5); :50 init — modelID "minilm-v6", seed 0x4D49_4E4C_4D5F_7631 "MINLM_v1", 384-dim; :64 embed / :80 embedFloat / :95 embedPair (ONE inference pass — separate calls each pay inference) -- MPNet: :30 struct; :38 init — "mpnet-base-v2", seed 0x4D50_4E45_545F_7631 "MPNET_v1", 768-dim; :52/:65/:80 embed/embedFloat/embedPair -- EmbeddingGemma: :32 struct; :40 init — "embedding-gemma-300m", seed 0x454D_4247_4D5F_7631 "EMBGM_v1", 768-dim, stand-in tokenizer vocab 256_000 / maxTokens 2048 (SentencePiece-sized); :58/:71/:86 +### Named neural providers : MiniLMTextProvider.swift, MPNetTextProvider.swift, EmbeddingGemmaProvider.swift +- MiniLM: :40 `struct MiniLMTextProvider: EmbeddingProvider`; :48 `inference: @Sendable ([Int32]) async throws -> [Float]` (closure-injected CoreML seam, doctrine §5); :50 init : modelID "minilm-v6", seed 0x4D49_4E4C_4D5F_7631 "MINLM_v1", 384-dim; :64 embed / :80 embedFloat / :95 embedPair (ONE inference pass : separate calls each pay inference) +- MPNet: :30 struct; :38 init : "mpnet-base-v2", seed 0x4D50_4E45_545F_7631 "MPNET_v1", 768-dim; :52/:65/:80 embed/embedFloat/embedPair +- EmbeddingGemma: :32 struct; :40 init : "embedding-gemma-300m", seed 0x454D_4247_4D5F_7631 "EMBGM_v1", 768-dim, stand-in tokenizer vocab 256_000 / maxTokens 2048 (SentencePiece-sized); :58/:71/:86 - Parity framing: embedding VALUES are a property of the host's model bundle; what is bit-identical cross-port is the kit-owned pipeline (tokens→engram, float lane given a pooled vector) and the full no-host pipeline with DeterministicTokenizer -### Apple NL providers — NLEmbeddingProvider.swift, NLContextualEmbeddingProvider.swift (Swift-only, ADR-019) +### Apple NL providers : NLEmbeddingProvider.swift, NLContextualEmbeddingProvider.swift (Swift-only, ADR-019) - NLEmbedding: :76 `nlEmbeddingProjectionSeed 0x4150_4E4C_454D_4231` "APNLEMB1" (outside doctrine table, same never-collide rule I-4); :113 `struct NLEmbeddingProvider: EmbeddingProvider, Sendable`; :145 init(language: .english default); :164/:180/:189 embed/embedFloat/embedPair; OS sentence model; no-model-for-language → [] opt-out; l2Normalize via FloatVecOps -- NLContextual: :79 `nlContextualEmbeddingProjectionSeed 0x4150_4E4C_4354_5831` "APNLCTX1"; :121 struct; :152 init; :170/:187/:196; checks `hasAvailableAssets` (free, sync) — NEVER triggers a network fetch; host must prefetch via requestAssets BEFORE constructing; mean-pools token vectors with per-token dimension guard; all failures → [] (asset absence = expected operational state) +- NLContextual: :79 `nlContextualEmbeddingProjectionSeed 0x4150_4E4C_4354_5831` "APNLCTX1"; :121 struct; :152 init; :170/:187/:196; checks `hasAvailableAssets` (free, sync) : NEVER triggers a network fetch; host must prefetch via requestAssets BEFORE constructing; mean-pools token vectors with per-token dimension guard; all failures → [] (asset absence = expected operational state) - Both `#if canImport(NaturalLanguage)`; no Rust twins; vector dimension comes from the OS model (not pinned) ## INVARIANTS / GOTCHAS @@ -228,8 +228,8 @@ ENTRY POINTS (most callers need only these): - DETERMINISM DISCIPLINE: engines never read `Date()`/`UUID()`; `now` is always caller-supplied. Sanctioned exceptions: RemovedSourceStore audit stamp, telemetry timestamps. Training is a pure function of (texts, pinned seeds). - UNIVERSAL JOIN KEY: chunk.id.uuidString == VectorKit item_id == inverted-index item_id. Swift canonical uuidString is UPPERCASE; HybridRecall re-canonicalizes vector-lane ids (P3-secfix). Universal tie-break everywhere: score DESC, then id ASC ("smaller id wins"). - PINNED CONSTANTS (change = new conformance vectors + possible fleet re-key): chunker 800/100; BM25 k1 1.5 / b 0.75; QUANT_SCALE 100 with round-HALF-TO-EVEN; BMW block 128; RRF k 60, weights 0.6/0.4; candidate over-fetch max(limit×4, 32); float-lane sim quantization ×10_000; commit windows 512/4096; ingestMaxAttempts 8; idle drain sleep 15 ms; lease TTL 15 s / standby poll 3 s; RI/PPMI D 2048 K 10 window 4; LSA rank 64 sweeps 30; NMF rank 32 iterations 100 tolerance 0 factorization seed 0xDEADBEEFCAFEBABE; reduced-vocab cap 512; Chunk.namespaceBytes; basisFormatVersion 1. -- PROJECTION SEEDS partition vector storage by model and must be unique + frozen: deterministic 0xC05B_D15C_A15D_1B00, MINLM_v1, MPNET_v1, EMBGM_v1, RI_V1_MX, PPMI_V1M, LSA_V1_M, NMF_V1_M, FDC_V1_P, APNLEMB1, APNLCTX1. All projection goes through SubstrateML.FloatSimHash — ad-hoc projections are banned from the kit graph. -- ONLY TWO TRAIN TRIGGERS: first-ingest auto-train (gate = no persisted basis) and explicit `reindex(now:)`. trainOnCorpus is ADDITIVE — always reconstruct fresh from the slot's freshBasisBlob before retraining; growth-threshold auto-retrain is future policy (anchors persisted, path unwired — "HALF A"). +- PROJECTION SEEDS partition vector storage by model and must be unique + frozen: deterministic 0xC05B_D15C_A15D_1B00, MINLM_v1, MPNET_v1, EMBGM_v1, RI_V1_MX, PPMI_V1M, LSA_V1_M, NMF_V1_M, FDC_V1_P, APNLEMB1, APNLCTX1. All projection goes through SubstrateML.FloatSimHash : ad-hoc projections are banned from the kit graph. +- ONLY TWO TRAIN TRIGGERS: first-ingest auto-train (gate = no persisted basis) and explicit `reindex(now:)`. trainOnCorpus is ADDITIVE : always reconstruct fresh from the slot's freshBasisBlob before retraining; growth-threshold auto-retrain is future policy (anchors persisted, path unwired : "HALF A"). - TOMBSTONE CONSULTATION IS UNENFORCED: every chunk-replay path (reindex, first-ingest train, count, any future rebuild) MUST subtract RemovedSourceStore.removedIDs() or removed sources resurrect on the governor's auto-reindex. - BundleStore.insert returns ONLY newly-inserted chunks; fold derived state over the returned subset. count(asOf:) ignores asOf. - DUAL TypedValue DECODE (BundleStore, BasisStore): SQLite round-trips UUID/HLC/timestamp as primitives, InMemory keeps semantic forms; decoders must accept both; InMemory-only tests cannot catch the regression (proven bug class: silent total data loss on reopen). @@ -237,7 +237,7 @@ ENTRY POINTS (most callers need only these): - appendOnly means two different things: sync conflict policy (SyncManifest, `.appendOnly`, safe via content addressing) vs BundleStore schema flag (`appendOnly: false`, required so scrubText can UPDATE). Do not conflate. - Chunks are immutable BY CONVENTION (no update API), not by DB trigger; edit = delete + reinsert with new id; metadata is the only legal per-chunk side-data slot (doctrine §2). - keywordTokens override breaks the single-tokenizer guarantee shared by BM25 + distributional providers (convention, not compiler-enforced). -- MMR (`mmrLambda`) is declared but NOT implemented on any recall path. BM25Index is legacy — Corpus uses InvertedIndexStore. +- MMR (`mmrLambda`) is declared but NOT implemented on any recall path. BM25Index is legacy : Corpus uses InvertedIndexStore. - Actors everywhere (Corpus, BundleStore, InvertedIndexStore, BasisStore, CorpusProviderCountsStore, RemovedSourceStore, BM25Index): writes serialized per instance; embedding compute deliberately escapes the actor via Sendable providers for parallelism. -- Retry-in-place (8 attempts) is sound ONLY because ingest is idempotent via content-addressed ids — not a general recipe. +- Retry-in-place (8 attempts) is sound ONLY because ingest is idempotent via content-addressed ids : not a general recipe. - Rust conformance gates: SharedVectors JSON (BM25 impacts, embedding vectors, basis blobs) read by BOTH legs; rust-providers tests pin basis serialization byte-for-byte for RI/PPMI/LSA/NMF and canonical vectors for RI/PPMI/FDC; NL providers exempt (Swift-only). diff --git a/packages/kits/CorpusKit/docs/DETAILS.md b/packages/kits/CorpusKit/docs/DETAILS.md index e3b6568..696b0df 100644 --- a/packages/kits/CorpusKit/docs/DETAILS.md +++ b/packages/kits/CorpusKit/docs/DETAILS.md @@ -77,855 +77,912 @@ sources: # CorpusKit Details -This document walks through every source file in the package. Read -`OVERVIEW.md` first for the big picture. Files appear in pipeline order: -the content types, the stores, the keyword engine, fusion and recall, the -trainable-basis machinery, the `Corpus` facade and its ingest queue, and -finally the providers target. +This document walks through each source file in the package. Read +`OVERVIEW.md` first for the big picture. This document follows pipeline +order. It starts with the content types and the stores. Next comes the +keyword engine, plus fusion and recall. After that comes the +trainable-basis machinery. It ends with the `Corpus` facade, the ingest +queue, and the providers target. ## Chunk.swift -This file provides `Chunk`, the fundamental retrievable unit, and -`ScoredChunk`, the result type recall returns. - -A chunk's identity is content-addressed: its UUID is computed, not -assigned. `deriveID(sourceID:startOffset:text:)` builds an RFC 4122 -version 5 UUID from a SHA-1 hash over the source identifier, the start -offset, and the exact text. The three fields are joined with the Unicode -unit separator so that no two different inputs can collide by ambiguous -concatenation. Content addressing is what makes the rest of the kit work: -re-ingesting the same source reproduces the same identities, so a repeat -insert is a harmless no-op, and two federated devices writing the same +This file holds `Chunk`, the basic retrievable unit. It also +holds `ScoredChunk`, the result type recall returns. + +A chunk's id is content-addressed. Its UUID is computed, never +assigned. The function `deriveID(sourceID:startOffset:text:)` builds an +RFC 4122 version 5 UUID. It hashes the source identifier, the start +offset, and the exact text with SHA-1. The three fields join through the +Unicode unit separator. That separator stops different inputs from +colliding through ambiguous concatenation. + +Content addressing is what makes the rest of the kit work. Re-ingesting +the same source reproduces the same identities. A repeat insert becomes +a harmless no-op. Two federated devices that write the same chunk converge on one row instead of conflicting. The fixed namespace -bytes are permanent. Changing them would re-key every chunk in every -estate and break the join to existing vector rows, because the chunk id -doubles as the VectorKit item identifier. - -Two initializers exist. The content-addressed one computes the id and is -the path normal ingestion uses. The explicit-id one reconstructs a chunk -whose id is already known, such as a decoded storage row; the caller must -ensure the id truly matches the content. The `hlc` field is a hybrid -logical clock stamp — a timestamp that also encodes causal order — and is -always caller-supplied. `ScoredChunk` pairs a chunk with its fused -`score` plus the raw `vectorScore` and `keywordScore`, kept separate so -callers can diagnose which lane produced a hit. +bytes are permanent. Changing them would re-key each chunk in each +estate. It would also break the join to existing vector rows, since +the chunk id doubles as the VectorKit item identifier. + +Two initializers exist. The content-addressed one computes the id. +Normal ingestion always takes this path. The explicit-id one +reconstructs a chunk whose id is already known, such as a decoded +storage row. The caller must ensure that id truly matches the content. +The `hlc` field is a hybrid logical clock stamp, a timestamp that also +encodes causal order. It is always caller-supplied. `ScoredChunk` pairs +a chunk with its fused `score`. It also carries the raw `vectorScore` +and `keywordScore`. The kit keeps these two scores separate. A caller +can then diagnose which lane produced a hit. ## Chunker.swift -This file provides `Chunker`, which splits raw source text into ordered +This file holds `Chunker`. It splits raw source text into ordered chunks with sentence-aware boundaries. -Chunks should not cut a sentence in half, because a half sentence embeds -poorly and reads worse. The chunker therefore segments the text into -sentences first, delegating to `EideticLib.sentences(_:)` so segmentation -logic lives in one shared place. It then fills a buffer greedily: sentences -accumulate until adding the next one would pass the target size, the buffer -flushes into a chunk, and the tail of the flushed chunk carries over as -overlap into the next buffer. Overlap means a match near a boundary still -brings its surrounding context along. Offsets into the original text are -tracked exactly, because the offset is part of each chunk's -content-addressed identity. - -`ChunkerConfiguration` holds `targetChars` (default 800), `overlapChars` -(default 100), and `respectSentences` (default true). Its initializer -clamps nonsense values — overlap can never reach the target, which would -loop forever. `Chunker.chunk(text:sourceID:configuration:hlcGenerator:)` -is the single entry point. The HLC generator is passed `inout` because the -chunker is the sole authority for chunk order within one call: it stamps -each chunk in emission order. Changing the default sizes changes chunk -boundaries and therefore chunk identities for re-ingested content, so the -defaults are pinned to the substrate reference. +A chunk should never cut a sentence in half. A half sentence embeds +poorly and reads worse. The chunker segments the text into sentences +first, for that reason. It delegates to `EideticLib.sentences(_:)`, so +segmentation logic lives in one shared place. It then fills a buffer +greedily. Sentences accumulate until adding the next one would pass the +target size. At that point the buffer flushes into a chunk. The tail of +the flushed chunk carries over as overlap into the next buffer. Overlap +means a match near a boundary still brings its surrounding context +along. Offsets into the original text are tracked exactly, since the +offset is part of each chunk's content-addressed id. + +`ChunkerConfiguration` holds three settings: `targetChars` (default 800), +`overlapChars` (default 100), and `respectSentences` (default true). Its +initializer clamps nonsense values. Overlap can never reach the target, +since that would loop forever. `Chunker.chunk(text:sourceID:configuration:hlcGenerator:)` +is the single entry point. The HLC generator passes in as `inout`. The +chunker is the sole authority for chunk order within one call, and it +stamps each chunk in emission order. Changing the default sizes changes +chunk boundaries. That in turn changes chunk identities for re-ingested +content, so the defaults are pinned to the substrate reference. ## Tokenizer.swift -This file provides the `Tokenizer` protocol and the single canonical -keyword tokenizer. - -A tokenizer serves two different masters. An embedding model needs its own -vocabulary of integer token ids, so the protocol requires -`tokenize(_:) -> [Int32]` plus identity fields (`vocabID`, `maxTokens`, -pad and unknown ids). Keyword search needs plain words, so the protocol -also requires `keywordTokens(_:) -> [String]`. The protocol extension -supplies a default for the second, delegating to the free function -`defaultKeywordTokens(_:)`: lowercase the text, then keep runs of -alphabetic or ASCII-digit characters and split on everything else. - -There is exactly one definition of keyword tokenization in the module for -a reason. BM25 and every distributional embedding signal must agree on -what a "term" is, or the keyword lane and the semantic lane would score -different vocabularies and hybrid recall would quietly degrade. The -function is also parity-critical: the Rust port implements the same rules, -and committed conformance vectors on both legs break if it changes. A -provider that overrides `keywordTokens` breaks this guarantee by -convention, not by compiler error. +This file holds the `Tokenizer` protocol. It also holds the single +canonical keyword tokenizer. + +A tokenizer serves two different masters. An embedding model needs its +own vocabulary of integer token ids. The protocol requires +`tokenize(_:) -> [Int32]` for that reason, plus id fields: `vocabID`, `maxTokens`, +and pad and unknown ids. Keyword search needs plain words instead. The +protocol also requires `keywordTokens(_:) -> [String]`. The protocol +extension supplies a default for that second method. It delegates to the +free function `defaultKeywordTokens(_:)`, which lowercases the text, then +keeps runs of alphabetic or ASCII-digit characters and splits on +everything else. + +There is just one definition of keyword tokenization in the module, +for a reason. BM25 and each distributional embedding signal must agree +on what a "term" is. Otherwise the keyword lane and the semantic lane +would score different vocabularies. Hybrid recall would then quietly +degrade. The function is also parity-critical. The Rust port implements +the same rules, and committed conformance vectors on both legs break if +the rules change. A provider that overrides `keywordTokens` breaks this +guarantee by convention, not by compiler error. ## CorpusKitError.swift -This file provides `CorpusKitError`, the module's single error enum. +This file holds `CorpusKitError`, the module's single error enum. Seven flat cases cover the failure classes: `encodingFailure`, `decodingFailure`, `tokenizerUnavailable`, `modelUnavailable`, -`embeddingFailed`, `storeUnavailable`, and `notTrainable`. Each carries a -plain message string, because callers mostly log or surface the message -rather than branch on structured data. `notTrainable` exists for -`EmbeddingModel.reconstruct(from:)`: providers without a trained basis -(the deterministic provider, the named neural models, the stateless FDC -provider) surface this error instead of silently substituting a wrong -provider. The enum is `Equatable` on its message strings, so tests that -compare errors must construct the exact message. +`embeddingFailed`, `storeUnavailable`, and `notTrainable`. Each case +carries a plain message string. Callers mostly log the message, or +surface it, rather than branch on structured data. The `notTrainable` +case exists for `EmbeddingModel.reconstruct(from:)`. Providers without a +trained basis surface this error instead of silently substituting a +wrong provider. Three provider families fall into this group: the +deterministic provider, the named neural models, and the stateless FDC +provider. The enum is `Equatable` on its message strings. Tests that +compare errors must so construct the exact message. ## BundleStore.swift -This file provides `BundleStore`, the persistence layer for chunks — the -content half of every content-plus-vector bundle. The vector half lives in -VectorKit, joined by the chunk's UUID string. - -`BundleStore` is an actor wrapping `any Storage` from PersistenceKit, so -the application picks the backend (SQLite or in-memory) and the kit does -not. It owns schema version 3: a `chunks` table of ten columns plus a -`corpus_metadata` table, with indices on `source_id` and `hlc`. Inserts do -not go straight to the row store. They pass through a hashing decorator -that computes a content hash with `MerkleHash.leaf` on every write and -emits dirty-chain events. A Merkle hash chain lets the estate prove its -content has not drifted: each chunk hashes, each source's chunks combine -into a per-corpus root, and all corpus roots combine into one global root. -Because the hashing callback is synchronous, a small lock-guarded -`ParentChainCache` pre-stages each chunk's parent identifiers before the -insert. The corpus and root identifiers derive from fixed namespace -strings via SHA-256, so both language legs compute identical chains. - -`insert(_:)` is idempotent by design. It attempts a plain insert per chunk -and treats a duplicate-key error as the documented no-op path — first -write wins, which is sound only because identities are content-addressed. -It returns only the chunks that were actually new, in input order, so -callers maintaining derived state never double-count a re-ingested chunk. -Read paths (`get`, `getMany`, `chunksForSource`, `allChunks`, -`allSourceIDs`, `count`, `chunkSourcePairs`) are thin query wrappers; -`chunkSourcePairs()` deliberately omits chunk bodies so opening a corpus -stays cheap. `scrubText(sourceID:)` is the hard-delete seam: it zeroes the -`text` column through a direct update, which is why the schema declares -the table `appendOnly: false` even though the API treats chunks as -immutable. Immutability here is a convention enforced by the surface, not -a database trigger; the sync layer's separate `appendOnly` conflict policy -should not be confused with this flag. - -One decoding rule deserves emphasis. The SQLite backend round-trips UUIDs -as text and HLCs as packed integers, while the in-memory backend preserves -the semantic typed values. `decodeChunk` and its helpers accept both -forms. A past semantic-only decoder silently dropped every persisted chunk -on reopen, and in-memory tests never caught it. Any new decode path must -handle both forms. +This file holds `BundleStore`, the persistence layer for chunks. It +is the content half of each content-plus-vector bundle. The vector half +lives in VectorKit, joined by the chunk's UUID string. + +`BundleStore` is an actor. It wraps `any Storage` from PersistenceKit. +The application picks the backend, SQLite or in-memory. The kit itself +does not. It owns schema version 3: a `chunks` table of ten columns, plus a +`corpus_metadata` table, with indices on `source_id` and `hlc`. Inserts +do not go straight to the row store. They pass through a hashing +decorator first. That decorator computes a content hash with +`MerkleHash.leaf` on each write and emits dirty-chain events. A Merkle +hash chain lets the estate prove its content has not drifted. Each chunk +hashes. Each source's chunks combine into a per-corpus root. All corpus +roots combine into one global root. Since the hashing callback is +synchronous, a small lock-guarded `ParentChainCache` pre-stages each +chunk's parent identifiers before the insert. The corpus and root +identifiers derive from fixed namespace strings, through SHA-256, so +both language legs compute the same chains. + +`insert(_:)` is idempotent by design. It attempts a plain insert per +chunk. It treats a duplicate-key error as the documented no-op path. +First write wins, and this is sound only since identities are +content-addressed. The method returns only the chunks that were actually +new, in input order. A caller that maintains derived state never +double-counts a re-ingested chunk this way. Seven methods form the read +surface: `get`, `getMany`, `chunksForSource`, `allChunks`, +`allSourceIDs`, `count`, and `chunkSourcePairs`. All are thin query +wrappers. `chunkSourcePairs()` omits chunk bodies on purpose, so opening +a corpus stays cheap. +`scrubText(sourceID:)` is the hard-delete seam. It zeroes the `text` +column through a direct update. This is why the schema declares the +table `appendOnly: false`, even though the API treats chunks as +immutable. Immutability here is a convention the surface enforces, not a +database trigger. The sync layer has its own separate `appendOnly` +conflict policy, and the two should not be confused. + +One decoding rule deserves emphasis. The SQLite backend round-trips +UUIDs as text and HLCs as packed integers. The in-memory backend +preserves the semantic typed values instead. `decodeChunk` and its +helpers accept both forms. A past semantic-only decoder silently +dropped each persisted chunk on reopen, and in-memory tests never +caught it. Any new decode path must handle both forms. ## RemovedSourceStore.swift -This file provides `RemovedSourceStore`, the tombstone table that makes +This file holds `RemovedSourceStore`, the tombstone table that makes source removal stick. -Chunk rows are never deleted, so "removing" a source can only delete its -vectors and keyword postings. Without a durable marker, any rebuild that -replays `allChunks()` — an explicit reindex, or the autonomic governor's -scheduled one — would re-embed the removed source and silently resurrect -it. The store is therefore the single source of truth every rebuild path -must consult. The presence of a row is the entire state: there is no -boolean column, per the fleet-wide schema rule. Reactivation is symmetric: -re-ingesting a source clears its tombstone, so ingestion itself is the -undo. +Chunk rows are never deleted. So "removing" a source can only delete its +vectors and its keyword postings. Without a durable marker, any rebuild +that replays `allChunks()` would re-embed the removed source. That +includes an explicit reindex, or the autonomic governor's scheduled one. +The source would then silently resurrect. The store is, for that +reason, the single source of truth each rebuild path must consult. The presence of +a row is the entire state. There is no boolean column, per the +fleet-wide schema rule. Reactivation is symmetric: re-ingesting a source +clears its tombstone, so ingestion itself is the undo. `markRemoved(_:now:)` upserts a tombstone with a caller-supplied timestamp. `clearRemoved(_:)` deletes it. `removedIDs()` returns the full -set that rebuilds must subtract. `deleteAll()` supports index destruction. -Nothing enforces that a new rebuild path remembers to consult this store; -that is a blast-radius obligation on every future change. +set that rebuilds must subtract. `deleteAll()` supports index +destruction. Nothing enforces that a new rebuild path remembers to +consult this store. That remains a blast-radius obligation on each +future change. ## CorpusProviderCountsStore.swift -This file provides `CorpusProviderCountsStore`, persistence for the raw -statistics a trainable embedding signal accumulates between retrains. +This file holds `CorpusProviderCountsStore`. It persists the raw +statistics a trainable embedding signal builds up between retrains. -The design splits cheap from heavy. The `counts` column is an opaque blob -the provider alone serializes; the store never decodes it and never -imports the providers target. Two small integer columns — document count -and vocabulary size — are lifted out of the blob so a staleness check can -ask "has the corpus grown enough to retrain?" with one tiny query instead -of deserializing a large blob. Rows are keyed by `(model_id, -model_version)`, matching how basis rows and vector rows are keyed, -because counts are only valid for the exact provider version that -accumulated them. `upsert(_:)` replaces the row whole; additive merging is -the provider's job before it calls in. +The design splits cheap state from heavy state. The `counts` column is +an opaque blob the provider alone serializes. The store never decodes +it, and never imports the providers target. Two small integer columns, +document count and vocabulary size, are lifted out of the blob. A +staleness check can then ask "has the corpus grown enough to retrain?" +with one tiny query, instead of deserializing a large blob. Rows key by +`(model_id, model_version)`. This matches how basis rows and vector rows +are keyed, since counts are only valid for the exact provider version +that accumulated them. `upsert(_:)` replaces the row whole. Additive +merging is the provider's job before it calls in. The file is candid that this is half a feature. Counts are persisted and -restored, but `Corpus.reindex` still retrains from raw chunk text rather -than from this table; the counts-backed retrain path and vector -re-projection are future work. Documentation should not present this store -as the current retrain mechanism. +restored. But `Corpus.reindex` still retrains from raw chunk text rather +than from this table. The counts-backed retrain path, and the matching +vector re-projection, are future work. Documentation should not present +this store as the current retrain mechanism. ## SyncManifest.swift -This file provides `CorpusKitSync.manifest(zoneIdentifier:)`, the +This file holds `CorpusKitSync.manifest(zoneIdentifier:)`, the declarative sync contract for the `chunks` table. -The manifest declares one bidirectional synced table with primary key `id` -and conflict policy `appendOnly`. That policy is safe precisely because -chunks are content-addressed and immutable: two devices can never produce -conflicting edits to the same id, only identical re-derivations, so the -sync layer needs no merge strategy. The kit performs no sync itself; the -application hands this manifest and a storage instance to ConvergenceKit. -When VectorKit sync is also enabled, both tables should share one zone so -chunks and their vectors stay join-compatible on every device. +The manifest declares one bidirectional synced table. Its primary key is +`id`, and its conflict policy is `appendOnly`. That policy is safe, +since chunks are content-addressed and immutable. Two devices can +never produce conflicting edits to the same id. They can only produce +the same re-derivations, so the sync layer needs no merge strategy. The kit performs no sync itself. The application hands this +manifest, and a storage instance, to ConvergenceKit. When VectorKit sync +is also enabled, both tables should share one zone. That keeps chunks +and their vectors join-compatible on each device. ## Engine/SparseTypes.swift -This file provides the value types of the sparse retrieval lane: +This file holds the value types of the sparse retrieval lane: `ImpactPosting`, `SparseHit`, `FusedHit`, and the `LaneTag` alias. -The load-bearing decision is that `ImpactPosting.impact` is an integer. A -float BM25 weight is quantized once at index build; from then on the whole -query path is integer arithmetic, which is what makes the Swift and Rust -legs bit-identical. `SparseHit` is the consumer surface: the integer score -divided back by the quantization scale. `FusedHit` carries the fused score -plus a `perLane` map of raw per-lane scores, preserved so later selection -stages can read lane signals without recomputation. `LaneTag` is a type -alias to VectorKit's enum rather than a second enum, because two identical -Swift enums are still distinct types and would make case names ambiguous -for consumers importing both kits. Posting lists are always sorted by item -id ascending — the WAND algorithm's pivoting invariant — and fused results -sort by score descending, then item id ascending. +The load-bearing decision is that `ImpactPosting.impact` is an integer. +A float BM25 weight quantizes once, at index build time. From then on +the whole query path runs on integer arithmetic. That is what makes the +Swift and Rust legs bit-identical. `SparseHit` is the consumer surface. +It divides the integer score back by the quantization scale. +`FusedHit` carries the fused score, plus a `perLane` map of raw +per-lane scores. Those raw scores stay available so later selection +stages can read lane signals without recomputing them. `LaneTag` is a +type alias to VectorKit's enum, not a second enum. Two matching Swift +enums would still be distinct types, and that would make case names +ambiguous for a consumer that imports both kits. Posting lists always +sort by item id ascending, the WAND algorithm's pivoting invariant. +Fused results sort by score descending, then by item id ascending. ## Engine/BM25Weighting.swift -This file provides BM25 as an impact-weighting scheme that feeds the +This file holds BM25 as an impact-weighting scheme that feeds the inverted index, plus the quantizer and query helpers. -BM25 scores a document for a term by combining the term's rarity (inverse -document frequency, IDF) with its frequency in the document, damped by -document length. The whole float computation happens exactly once, at -index build. `build(termFreqs:docLengths:parameters:)` evaluates the -classic formula per term and document, then quantizes each contribution -with `quantizeImpact(_:)` — multiply by 100 and round half to even. The -rounding mode is pinned deliberately: Swift's default rounding differs -from banker's rounding at exact halves, and both legs must agree. Term -strings map to dense integer ids in sorted order, so runs are -reproducible. `BM25Parameters` pins the defaults `k1 = 1.5` and -`b = 0.75`, tunable per estate. `queryPairs(queryTerms:termMapping:)` -turns query terms into (term id, weight 100) pairs, dropping unknown terms -and deduplicating repeats so each term contributes once. +BM25 scores a document for a term by combining two things: the term's +rarity, called inverse document frequency (IDF), and its frequency in +the document. Document length damps the score. The whole float +computation happens just once, at index build. `build(termFreqs:docLengths:parameters:)` +evaluates the classic formula per term and document. It then quantizes +each contribution with `quantizeImpact(_:)`, which multiplies by 100 and +rounds half to even. The rounding mode is pinned on purpose. Swift's +default rounding differs from banker's rounding at exact halves, and +both legs must agree. Term strings map to dense integer ids in sorted +order, so runs stay reproducible. `BM25Parameters` pins the defaults: +`k1` equals 1.5, and `b` equals 0.75. Both values are tunable per +estate. `queryPairs(queryTerms:termMapping:)` turns query terms into +term-id and weight-100 pairs. It drops unknown terms. It deduplicates +repeats, so each term contributes just once. ## Engine/InvertedIndex.swift -This file provides the generic weighted inverted index with two exact -top-k algorithms: WAND and Block-Max WAND. - -An inverted index maps each term to a posting list — the items containing -that term, each with a pre-quantized impact. Scoring an item for a query -is an integer dot product over shared terms. The naive approach scores -every candidate; WAND ("Weak AND") skips most of them. It keeps one cursor -per query term, sorted by current position, and computes a pivot: the -first point where the accumulated best-case impacts could beat the current -k-th best score. Items before the pivot cannot win and are skipped -wholesale. Block-Max WAND refines this with per-block maxima (block size -128): if even the tighter block bound cannot beat the threshold, the whole -block is skipped. Both algorithms are exact — they return precisely the -same top-k as a full scan — and `exhaustiveScan(query:k:)` ships as the -reference oracle for conformance tests. - -The index is immutable after construction; mutation means rebuilding, +This file holds the generic weighted inverted index. It ships two +exact top-k algorithms: WAND and Block-Max WAND. + +An inverted index maps each term to a posting list, the items containing +that term, each carrying a pre-quantized impact. Scoring an item for a +query is an integer dot product over shared terms. The naive approach +scores each candidate. WAND, short for "Weak AND," skips most of them +instead. It keeps one cursor per query term, sorted by current position. +It computes a pivot: the first point where the accumulated best-case +impacts could beat the current k-th best score. Items before the pivot +cannot win, so the algorithm skips them wholesale. Block-Max WAND +refines this idea with per-block maxima, using a block size of 128. If +even the tighter block bound cannot beat the threshold, the whole block +gets skipped. Both algorithms are exact. They return precisely the same +top-k as a full scan. `exhaustiveScan(query:k:)` ships as the reference +oracle for conformance tests. + +The index is immutable after construction. Mutation means rebuilding it, and serializing rebuilds is the wrapper's job. The internal bounded heap implements the universal tie-break: equal scores resolve toward the -smaller item id, so results never depend on hash or insertion order. Item -ids compare as strings. The pinned constants `invertedIndexQuantScale` -(100) and `invertedIndexBlockSize` (128) are part of the cross-port -contract. +smaller item id, so results never depend on hash order or insertion +order. Item ids compare as strings. Two pinned constants belong to the +cross-port contract: `invertedIndexQuantScale` (100) and +`invertedIndexBlockSize` (128). ## Engine/InvertedIndexStore.swift -This file provides `InvertedIndexStore`, the persistent wrapper that lets -keyword state survive restarts without replaying chunk bodies. +This file holds `InvertedIndexStore`, the persistent wrapper that +lets keyword state survive restarts without replaying chunk bodies. The store persists only raw statistics: a term-frequency table and a -document-length table in two small SQLite tables. The weighted index -itself is derived. On demand, `buildIndex(parameters:)` runs the BM25 -build over the in-memory mirrors and caches the result; every write -invalidates the cache. Persisting statistics instead of weighted postings -means changing `k1` or `b` never requires a data migration — only an -in-memory rebuild. `open()` loads all rows once, a cost proportional to -terms plus documents, never to chunk text. - -`index(itemID:tokens:now:)` replaces a document's terms atomically and is -idempotent; empty tokens remove the item. `remove(itemID:)` and -`deleteAll()` complete the mutation surface, and -`topK(queryTerms:k:parameters:algorithm:)` is the one-call query path. The -actor serializes all mutation. The Rust twin owns a private database -connection with explicit batch methods; the Swift store instead shares the -estate's storage, which is why the facade manages transaction windows -around it during bulk ingest. +document-length table, in two small SQLite tables. The weighted index +itself is derived, not stored. On demand, `buildIndex(parameters:)` runs +the BM25 build over the in-memory mirrors and caches the result. Each +write invalidates the cache. Persisting statistics instead of weighted +postings means changing `k1` or `b` never requires a data migration. It +only requires an in-memory rebuild. `open()` loads all rows once, at a +cost proportional to terms plus documents, never to chunk text. + +`index(itemID:tokens:now:)` replaces a document's terms atomically, and +it is idempotent. Empty tokens remove the item. `remove(itemID:)` and +`deleteAll()` complete the mutation surface. `topK(queryTerms:k:parameters:algorithm:)` +is the one-call query path. The actor serializes all mutation. The Rust +twin owns a private database connection with explicit batch methods. +The Swift store instead shares the estate's storage, which is why the +facade manages transaction windows around it during bulk ingest. ## BM25Index.swift -This file provides `BM25Index`, the original in-memory keyword index, -preserved as a public primitive. +This file holds `BM25Index`, the original in-memory keyword index. It +is preserved as a public primitive. -The `Corpus` facade no longer uses it — durability required -`InvertedIndexStore` — but external callers that built on it keep a -working, chunk-typed surface. It holds term frequencies keyed by chunk -UUID string, tokenizes chunk text itself through an injected `Tokenizer`, -and delegates scoring to the same engine layer (BM25 weighting plus -Block-Max WAND), caching the built index between writes. +The `Corpus` facade no longer uses it. Durability required +`InvertedIndexStore` instead. External callers that built on the older +type keep a working, chunk-typed surface. It holds term frequencies +keyed by chunk UUID string. It tokenizes chunk text itself, through an +injected `Tokenizer`. It delegates scoring to the same engine layer: BM25 +weighting, plus Block-Max WAND. The built index caches between writes. `index(_:)`, `remove(_:)`, `documentCount()`, and `topK(_:for:)` form the -surface; `topK` takes pre-tokenized terms, and the caller must tokenize +surface. `topK` takes pre-tokenized terms, and the caller must tokenize with the same vocabulary used at index time. Ties break by UUID string -order, which is not numeric UUID order but is identical on both legs. +order. That order differs from numeric UUID order. It stays the same +on both legs, still. ## Engine/Fusion.swift -This file provides `Fusion`, the generalized weighted Reciprocal Rank +This file holds `Fusion`, the generalized weighted Reciprocal Rank Fusion engine. -Reciprocal Rank Fusion (RRF) merges ranked lists without comparing their -raw scores, which live on incompatible scales. Each lane contributes -`weight × 1 / (rrfK + rank)` for every item it ranked; the sums decide the -final order. The constant `rrfK` (default 60, from the original RRF paper) -damps the advantage of rank one over rank two. The function deduplicates -within each lane — only an item's best rank counts, because a duplicate -would illegally double its contribution — and demands `rrfK > 0`, since -zero or negative values corrupt the formula. Output sorts by fused score -descending, then item id ascending. +Reciprocal Rank Fusion, called RRF, merges ranked lists. It never +compares raw scores, since those scores live on incompatible scales. Each +lane contributes `weight × 1 / (rrfK + rank)` for each item it ranked. +The sums decide the final order. The constant `rrfK` defaults to 60, +from the original RRF paper, and it damps the advantage of rank one over +rank two. The function deduplicates within each lane. Only an item's +best rank counts there, since a duplicate would illegally double its +contribution. The function also demands `rrfK > 0`, since zero or +negative values would corrupt the formula. Output sorts by fused score +descending, then by item id ascending. Two overloads exist. `fuse(rankedLists:laneScores:weights:rrfK:)` takes -explicit ranks and optional raw scores to carry through into `perLane`. -`fuse(scoredLists:weights:rrfK:)` treats array position as rank; the -caller must pre-sort. The engine is a pure function over ranks and -weights — deterministic and reentrant. One caution: the configuration type -in `HybridRecall.swift` reserves an MMR field, but no diversification is -implemented here or anywhere on this path yet. +explicit ranks. It takes optional raw scores too, to carry through into +`perLane`. `fuse(scoredLists:weights:rrfK:)` treats array position as +rank instead, so the caller must pre-sort its input. The engine is a +pure function over ranks and weights. It is deterministic and reentrant. +One caution applies here. The configuration type in +`HybridRecall.swift` reserves an MMR field. No diversification logic +exists yet, on this path or anywhere else. ## HybridRecall.swift -This file provides `HybridRecall.recall(...)`, the canonical two-lane +This file holds `HybridRecall.recall(...)`, the canonical two-lane retrieval pipeline. -The pipeline over-fetches a candidate window of `max(limit × 4, 32)` from -each lane, because fusion needs headroom: an item ranked eleventh in both -lanes can out-fuse an item ranked first in only one. The vector lane runs -VectorKit's nearest-neighbor search concurrently while the query is -tokenized and the keyword lane queries the inverted index. Both hit lists -become ranked lists keyed by canonical UUID strings. Canonicalization is a -deliberate security-review fix: a lowercase UUID written by the Rust leg -and an uppercase Swift keyword hit for the same item would otherwise never -fuse. `Fusion.fuse` merges the lanes with the configured weights, the -result truncates to the limit, and the winners hydrate from the bundle -store in fused order. - -`HybridRecallConfiguration` pins the defaults: vector weight 0.6, keyword -weight 0.4, `rrfK` 60, and an `mmrLambda` slot that is currently declared -but never read. Score mapping is asymmetric on purpose: a vector score of -zero is a perfect Hamming match and is kept, while a keyword score of zero -means "did not match" and maps to nil. Telemetry (latency and per-lane -counts) fires at the operation boundary where it cannot affect results; -with monitoring off it costs one atomic load per metric. +The pipeline over-fetches a candidate window from each lane. That +window's size is `max(limit × 4, 32)`, since fusion needs headroom. An +item ranked eleventh in both lanes can out-fuse an item ranked first in +only one. The vector lane runs VectorKit's nearest-neighbor search +concurrently, while the query gets tokenized and the keyword lane +queries the inverted index. Both hit lists become ranked lists, keyed by +canonical UUID strings. Canonicalization is a deliberate security-review +fix. A lowercase UUID written by the Rust leg, and an uppercase Swift +keyword hit for the same item, would otherwise never fuse. `Fusion.fuse` +merges the lanes with the configured weights. The result truncates to +the limit. The winners then hydrate from the bundle store, in fused +order. + +`HybridRecallConfiguration` pins four defaults. The vector weight is +0.6. The keyword weight is 0.4. The `rrfK` value is 60. The +`mmrLambda` slot exists, but the pipeline never reads it. Score mapping +is asymmetric on purpose. A vector score of zero is a perfect Hamming +match, and the pipeline keeps it. A keyword score of zero instead means +"did not match," and it maps to nil. Telemetry fires at the operation +boundary, where it cannot affect results. It covers latency and +per-lane counts. With monitoring off it costs one atomic load per +metric. ## TrainableEmbeddingBasis.swift -This file provides the `TrainableEmbeddingBasis` protocol, the seam that -lets the core drive provider training without importing the providers -target. +This file holds the `TrainableEmbeddingBasis` protocol. It is the +seam that lets the core drive provider training without importing the +providers target. Layering runs one way: providers depend on core, never the reverse. The -`Corpus` holds providers as type-erased values, so it needs a protocol to -ask "can you train, serialize, and reconstruct yourself?" Providers that -cannot — the deterministic provider, the named neural models, FDC, the -Apple NL providers — simply do not conform, and the facade surfaces -`CorpusKitError.notTrainable`. `reconstructBasis(from:)` is an instance -method rather than an initializer for exactly this reason: invoked on a -type-erased witness, it routes to the correct concrete type's -deserializing initializer. - -The protocol has two halves. The basis half — `trainOnCorpus(texts:)`, -`serializeBasis()`, `reconstructBasis(from:)` — covers full training and -the round-trip law: a reconstructed provider embeds byte-identically to -the trained original. The counts half — `addToCounts(text:)`, -`serializeCounts()`, `restoreCounts(from:)`, `countsVocabularySize` — -maintains raw additive statistics incrementally, snapshotted at batch -boundaries because per-chunk serialization would be quadratic over an -import. Training must never read the wall clock; it is a pure function of -the texts and fixed seeds. The Rust port cannot cross-cast trait objects, -so there the embedding trait is a supertrait instead — a documented, -sanctioned divergence. +`Corpus` holds providers as type-erased values. It needs a protocol to +ask one question: can the provider train, serialize, and reconstruct +itself? Some providers +cannot: the deterministic provider, the named neural models, FDC, and +the Apple NL providers. They simply do not conform, and the facade +surfaces `CorpusKitError.notTrainable`. `reconstructBasis(from:)` is an +instance method rather than an initializer, for exactly this reason. +Invoked on a type-erased witness, it routes to the correct concrete +type's deserializing initializer. + +The protocol has two halves. The basis half covers full training and the +round-trip law: a reconstructed provider embeds byte-identically to the +trained original. It includes `trainOnCorpus(texts:)`, `serializeBasis()`, +and `reconstructBasis(from:)`. The counts half maintains raw additive +statistics incrementally instead. It includes `addToCounts(text:)`, +`serializeCounts()`, `restoreCounts(from:)`, and `countsVocabularySize`. +Snapshots happen at batch boundaries, since per-chunk serialization +would be quadratic over an import. Training must never read the wall +clock. It is a pure function of the texts and fixed seeds. The Rust port +cannot cross-cast trait objects, so there the embedding trait is a +supertrait instead, a documented and sanctioned divergence. ## BasisStore.swift -This file provides `BasisStore`, persistence for trained basis blobs, so a -reopened corpus embeds immediately instead of retraining. - -One row per `(model_id, model_version)` in the `corpus_provider_basis` -table holds the opaque little-endian blob, a trained-at timestamp -(caller-supplied, stored as ISO 8601 text per the schema rules), and a -trained-chunk-count anchor reserved for a future auto-retrain policy. -The composite key matters: a blob trained for one provider must never load -into another, and the key matches how every vector row is keyed. Retrain -upserts in place, so exactly one row exists per provider — no history, no -orphans. Schema version 2 adds a nullable JSON `ext` column as a -forward-compatibility slot that version 1.0 writes as null and never -reads. - -Like `BundleStore`, the decoder tolerates both typed-value forms — the -in-memory backend's semantic timestamps and SQLite's ISO text — because a -semantic-only reader would silently drop every row on reopen and semantic -recall would go dark on any restored estate. `upsert(_:)`, -`load(modelID:modelVersion:)`, and `deleteAll()` form the whole surface. +This file holds `BasisStore`, persistence for trained basis blobs, so +a reopened corpus embeds right away instead of retraining. + +One row per `(model_id, model_version)` lives in the +`corpus_provider_basis` table. Each row holds the opaque little-endian +blob, a trained-at timestamp, and a trained-chunk-count anchor reserved +for a future auto-retrain policy. The timestamp is caller-supplied, and +it stores as ISO 8601 text, per the schema rules. The composite key +matters: a blob trained for one provider must never load into another, +and the key matches how each vector row is keyed. Retrain upserts in +place, so just one row exists per provider. There is no history, and +there are no orphans. Schema version 2 adds a nullable JSON `ext` column +as a forward-compatibility slot. Version 1.0 writes that column as null +and never reads it. + +Like `BundleStore`, the decoder tolerates both typed-value forms: the +in-memory backend's semantic timestamps, and SQLite's ISO text. A +semantic-only reader would otherwise silently drop each row on reopen, +and semantic recall would go dark on any restored estate. +`upsert(_:)`, `load(modelID:modelVersion:)`, and `deleteAll()` form the +whole surface. ## CorpusKit.swift -This file provides the public entry point: the `Corpus` actor, the +This file holds the public entry point: the `Corpus` actor, the `EmbeddingModel` selection enum, the `FloatLaneOutcome` result type, and the `EncodeSpeed` quality-of-service knob. It is the largest file in the -package because it is the composition root: everything else exists so this -file can wire it together. +package, since it is the composition root. Everything else exists so +this file can wire it together. ### The Corpus Actor and Its Provider Slots -A `Corpus` composes the bundle store, the persistent keyword index, the -vector store, the basis and counts stores, the tombstone store, and one -slot per configured embedding signal. It seals VectorKit behind its own -surface: no VectorKit type appears in a public signature except the -deliberate `sharedVectorStore` escape hatch, which lends the estate's one -vector store to the orchestrator so no second store is built over the same -table. - -Each provider slot holds three things. First, the serving provider, which -embeds queries and chunks. Second, for trainable signals, a -`freshBasisBlob` — the serialized untrained basis captured at -construction. This is the from-scratch factory: training is additive, so -retraining a live provider would count the corpus twice; every retrain -instead reconstructs a fresh provider from this blob, which makes reindex -idempotent and canonical across ports. Third, a separate counts -accumulator, deliberately not the serving provider, because growing a -vocabulary in place would desync a factorized basis from its frozen -factors. Slot zero is the default signal: every single-signal entry point -delegates to it, so a one-model corpus behaves exactly like the old +A `Corpus` composes the bundle store, the persistent keyword index, and +the vector store. It also composes the basis store, the counts store, +and the tombstone store. It holds one slot per configured embedding +signal, too. It seals VectorKit behind its own surface. No VectorKit type appears in a +public signature, except the deliberate `sharedVectorStore` escape +hatch. That hatch lends the estate's one vector store to the +orchestrator, so no second store gets built over the same table. + +Each provider slot holds three things. First, the serving provider, +which embeds queries and chunks. Second, trainable signals get a +`freshBasisBlob`. This is the serialized untrained basis, captured at +construction. It works as the from-scratch factory. Training is additive, +so retraining a live provider would count the corpus twice. Each +retrain instead reconstructs a fresh provider from this blob. That makes +reindex idempotent and canonical across ports. Third, a separate counts +accumulator, kept apart from the serving provider on purpose. Growing +a vocabulary in place would desync a factorized basis from its frozen +factors. Slot zero is the default signal. Each single-signal entry +point delegates to it, so a one-model corpus behaves just like the old single-provider design. ### Opening, Ingesting, Reindexing -`init(storage:models:)` migrates six schemas, resolves each slot (a -persisted basis reconstructs a trained provider; a corrupt blob throws -rather than serving untrained), opens the keyword index, and warm-loads a -chunk-to-source map from a body-free projection — the whole cold start -avoids reading chunk text. `ingest(_:sourceID:now:)` chunks the text, -clears any tombstone (re-ingest reactivates), inserts idempotently, -indexes keywords, folds counts, and embeds. Embedding is two-phase: any -trainable slot with no persisted basis triggers the one-and-only implicit -first-ingest training over the full corpus snapshot; all other slots fold -in under their frozen basis. Fold-in embeddings compute concurrently off -the actor — providers are `Sendable` values — and land in one batched -vector write. `ingestBatch(_:)` produces output identical to per-item -ingest but commits in windows of 512 items or 4,096 rows, long enough to -amortize disk syncs and short enough not to starve concurrent captures, -and fans embedding out in contiguous slices per core. - -`reindex(now:)` is the explicit retrain trigger: reconstruct fresh, train -on all active chunks (tombstoned sources excluded), install, persist the -basis, and re-embed every active chunk under every slot. Only two train -triggers exist in the whole kit — first ingest and explicit reindex. +`init(storage:models:)` migrates six schemas. It resolves each slot: a +persisted basis reconstructs a trained provider, and a corrupt blob +throws rather than serving untrained data. It opens the keyword index. +It warm-loads a chunk-to-source map from a body-free projection, so the +whole cold start avoids reading chunk text. `ingest(_:sourceID:now:)` +performs six steps in order. It chunks the text. It clears any +tombstone, since re-ingest reactivates a source. It inserts the chunks +idempotently. It indexes keywords. It folds counts. It embeds the +chunk. Embedding runs in two phases. Any trainable slot with no persisted basis triggers the +one-and-only implicit first-ingest training, over the full corpus +snapshot. Each other slot folds in under its frozen basis instead. +Fold-in embeddings compute concurrently off the actor, since providers +are `Sendable` values, and they land in one batched vector write. +`ingestBatch(_:)` produces output identical to per-item ingest. It +commits in windows of 512 items or 4,096 rows, long enough to amortize +disk syncs and short enough not to starve concurrent captures. It also +fans embedding work out in contiguous slices per core. + +`reindex(now:)` is the explicit retrain trigger. It performs five steps +in order. It reconstructs fresh providers. It trains on all active +chunks, excluding tombstoned sources. It installs the result. It +persists the basis. It re-embeds each active chunk under each slot. +Only two train triggers exist in the whole kit: first ingest, and +explicit reindex. ### Recall, Removal, Observation `recall(_:limit:now:)` embeds the query on the default signal and -delegates to `HybridRecall`. `bm25TopKBySource(query:limit:)` is the pure -keyword lane aggregated to source granularity. The dense float lane — -`floatNearest`, `floatNearestPerSignal`, `floatFarthestPerSignal` — ranks -by true cosine similarity and never throws; unavailable states are typed -`FloatLaneOutcome` values (`.unavailableProviderOptOut`, -`.unavailableNoVocabHit`, `.unavailableNoFloatRows`, `.emptyQuery`, -`.storeError`), because a dark lane is an expected condition, not an -error. The farthest variant answers "what is unlike this?" and aggregates -by each source's closest chunk, so a source only counts as unlike when -even its best chunk is far. - -`remove(sourceID:)` suppresses recall: it deletes keyword rows and every -model's vectors and writes the tombstone. `expunge(sourceID:)` scrubs the -verbatim text first, then removes, so content is destroyed even if a later -step fails. `destroyRecallIndex()` wipes every derived structure while -chunk rows survive. `count()`, `indexedSourceIDs()`, -`maintainedVocabAnchor()`, and the two Merkle root accessors round out the -observational surface. +delegates to `HybridRecall`. `bm25TopKBySource(query:limit:)` is the +pure keyword lane, aggregated to source granularity. The dense float +lane covers three methods: `floatNearest`, `floatNearestPerSignal`, and +`floatFarthestPerSignal`. All three rank by true cosine similarity. None +of them ever throws. Unavailable states are typed `FloatLaneOutcome` values instead: +`.unavailableProviderOptOut`, `.unavailableNoVocabHit`, +`.unavailableNoFloatRows`, `.emptyQuery`, and `.storeError`. A dark lane +is an expected condition here, not an error. The farthest variant +answers "what is unlike this?" It aggregates by each source's closest +chunk, so a source only counts as unlike when even its best chunk is +far. + +`remove(sourceID:)` suppresses recall. It deletes keyword rows and each +model's vectors, and it writes the tombstone. `expunge(sourceID:)` goes +further: it scrubs the verbatim text first, then removes the source, so +content is destroyed even if a later step fails. `destroyRecallIndex()` +wipes each derived structure while chunk rows survive. Four more +methods round out the observational surface: `count()`, +`indexedSourceIDs()`, `maintainedVocabAnchor()`, and two Merkle-root +readers. ### EmbeddingModel and the Small Types -`EmbeddingModel` names every signal the corpus can hold: `.deterministic` -(the permanent federation-grade baseline — a hash-based, lexical, fully -reproducible signal with a pinned seed), the three named neural models -(`.miniLM`, `.mpNet`, `.embeddingGemma`) that take a host-supplied -inference closure, the four trainable statistical signals -(`.randomIndexing`, `.ppmi`, `.lsa`, `.nmf`) that carry pre-built -providers, stateless `.fdc`, and the Apple-only `.nlEmbedding` and -`.nlContextualEmbedding`. `isTrainable` reports whether the carried -provider conforms to the training seam, and `reconstruct(from:)` routes a -persisted blob to the right concrete type. `EncodeSpeed` selects the -embed-concurrency cap: `.foreground` uses all cores, `.background` roughly -a quarter. A private `CorpusDefaultTokenizer` duplicates the providers' -deterministic tokenizer to avoid a circular dependency, and a private +`EmbeddingModel` names each signal the corpus can hold. `.deterministic` +is the permanent federation-grade baseline. It is hash-based and +lexical. It is fully reproducible, with a pinned seed. Three named +neural models follow it: `.miniLM`, `.mpNet`, and `.embeddingGemma`. +Each one takes a host-supplied inference closure. Four trainable statistical +signals carry pre-built providers: `.randomIndexing`, `.ppmi`, `.lsa`, +and `.nmf`. The stateless `.fdc` signal comes next. Two Apple-only +signals close the list: `.nlEmbedding` and `.nlContextualEmbedding`. +`isTrainable` reports whether the carried provider conforms to the +training seam. `reconstruct(from:)` routes a persisted blob to the right +concrete type. `EncodeSpeed` selects the embed-concurrency cap. +`.foreground` uses all cores, and `.background` uses roughly a quarter. +A private `CorpusDefaultTokenizer` duplicates the providers' +deterministic tokenizer, to avoid a circular dependency. A private `CorpusTextProvider` implements the tokenize-infer-project pipeline for -the named models, computing the pooled vector once per chunk for both the -engram and the float row. +the named models. It computes the pooled vector once per chunk, for +both the engram and the float row. ## CorpusIngestQueue.swift -This file provides the asynchronous ingest pipeline as an extension on -`Corpus`: a durable queue, a background drain worker, and a single-drainer -lease. It exists so CorpusKit is a complete standalone substrate — any -consumer gets queued, multi-core encoding with no orchestrator. +This file holds the asynchronous ingest pipeline. It arrives as an +extension on `Corpus`. Three pieces make up the pipeline: a durable +queue, a background drain worker, and a single-drainer lease. It exists +so CorpusKit is a complete standalone substrate. Any consumer gets +queued, multi-core encoding with no orchestrator. `mountIngestQueue()` picks the backend by estate durability. A SQLite -estate gets a sibling `queue.sqlite` file derived deterministically from -the estate configuration — encrypted with the same key as the estate, -replacing an earlier plaintext directory queue that was a real security -hole beside an encrypted estate. An in-memory estate gets a transient -store under a fixed constant UUID, avoiding random-identity -nondeterminism. Because the physical queue can carry other streams, every -operation here is scoped to the `"encode"` stream; an unscoped wait would -deadlock on jobs this drainer never claims. - -The drain loop coordinates through a `DrainLease`: one live drainer per -estate, with crash recovery on first acquisition that resets orphaned -in-flight jobs — safe precisely because the lease guarantees no other live -drainer holds them. A losing process becomes a warm standby that re-checks -every three seconds, bounded by the lease's fifteen-second staleness -window. Each drain pass claims the whole available batch, decodes jobs -(undecodable ones are terminally blocked, empty ones completed), runs -`ingestBatch` once for the batch, and retires the batch in one bulk reply. -While passes keep draining jobs, the loop spins without sleeping and -defers the vector-index publish until the burst ends — one index rebuild -per burst instead of one per pass, turning a quadratic bulk import linear. -Idle, it sleeps fifteen milliseconds, the near-realtime latency floor. A -failing item retries in place up to eight attempts before a terminal -blocked reply; in-place retry is sound only because ingest is idempotent. +estate gets a sibling `queue.sqlite` file, derived deterministically +from the estate configuration. That file is encrypted with the same key +as the estate. It replaces an earlier plaintext directory queue, which +was a real security hole beside an encrypted estate. An in-memory estate +gets a transient store instead, under a fixed constant UUID, which +avoids random-id nondeterminism. Since the physical queue can +carry other streams, each operation here is scoped to the `"encode"` +stream. An unscoped wait would deadlock on jobs this drainer never +claims. + +The drain loop coordinates through a `DrainLease`. Only one live drainer +exists per estate, with crash recovery on first acquisition that resets +orphaned in-flight jobs. That recovery is safe precisely since the +lease guarantees no other live drainer holds them. A losing process +becomes a warm standby instead. It re-checks each three seconds, +bounded by the lease's fifteen-second staleness window. Each drain pass +claims the whole available batch. It decodes jobs: undecodable ones are +terminally blocked, and empty ones complete right away. It runs +`ingestBatch` once for the whole batch, and it retires the batch in one +bulk reply. While passes keep draining jobs, the loop spins without +sleeping. It defers the vector-index publish until the burst ends, so +the kit runs one index rebuild per burst instead of one per pass. That +turns a quadratic bulk import linear. When idle, the loop sleeps fifteen +milliseconds, the near-realtime latency floor. A failing item retries in +place, up to eight attempts, before a terminal blocked reply. In-place +retry is sound only since ingest is idempotent. `enqueueIngest(_:sourceID:now:)` and `enqueueIngestBatch(_:)` stamp jobs -with caller-supplied instants — never the wall clock — and the batch -variant wraps all inserts in one transaction, which removed the last -full-core bottleneck of bulk imports on encrypted SQLite. +with caller-supplied instants, never the wall clock. The batch variant +wraps all inserts in one transaction, which removed the last full-core +bottleneck of bulk imports on encrypted SQLite. `awaitIngestDrain(timeout:)` is the barrier importers use to know writes are searchable. `setOnEncoded(_:)` installs the one callback CorpusKit ever makes toward an orchestrator. The `IngestJob` wire format's JSON -field names are a pinned cross-port contract with the Rust twin. +field names form a pinned cross-port contract with the Rust twin. ## BasisCodec.swift -This file provides the shared binary codec every trainable provider uses +This file holds the shared binary codec each trainable provider uses to serialize bases and counts. -The byte layout is the cross-port contract: the same trained state must -serialize to the same bytes on Swift and Rust, which rules out JSON (float -formatting, key order, and whitespace differ across ecosystems). The rules -are fixed: everything little-endian; floats written as raw IEEE-754 bit -patterns so negative zero and NaN round-trip exactly; strings -length-prefixed UTF-8; maps written with keys in ascending raw-byte order. -That byte-order sort exists because Swift's default string comparison is -Unicode-canonical while Rust's is byte order — the writer compares raw -UTF-8 to match. Every blob is framed with a four-byte magic tag and a -format version byte, currently 1. - -`BasisWriter` is an append-only cursor with typed write methods; -`BasisReader` is a bounds-checked sequential reader whose -`expectMagic(_:)` and `expectVersion(_:)` reject wrong-provider or -future-format blobs with `CorpusKitError.decodingFailure` — never a crash, -never a silent misparse. Both are value types with no shared state. +The byte layout is the cross-port contract. The same trained state must +serialize to the same bytes on Swift and on Rust, which rules out JSON. +Float formatting, key order, and whitespace all differ across +ecosystems. The rules stay fixed: everything is little-endian. Floats +write as raw IEEE-754 bit patterns, so negative zero and NaN round-trip +exactly. Strings are length-prefixed UTF-8. Maps write with keys in +ascending raw-byte order. That byte-order sort exists since Swift's +default string comparison is Unicode-canonical, while Rust's is byte +order. The writer compares raw UTF-8 to match Rust. Each blob is framed +with a four-byte magic tag and a format version byte, currently 1. + +`BasisWriter` is an append-only cursor with typed write methods. +`BasisReader` is a bounds-checked sequential reader. Its +`expectMagic(_:)` and `expectVersion(_:)` methods reject wrong-provider +or future-format blobs with `CorpusKitError.decodingFailure`, never a +crash and never a silent misparse. Both types are value types with no +shared state. ## DeterministicTokenizer.swift -This file provides `DeterministicTokenizer`, the model-agnostic stand-in +This file holds `DeterministicTokenizer`, the model-agnostic stand-in tokenizer that ships as the version 1.0 default for the named neural providers. It is a hash, not a vocabulary. Words split by the canonical keyword -rules, then each word folds through FNV-1a into an id in the range two -through the vocabulary size; ids zero and one are reserved sentinels for -padding and unknown. Empty input returns a single pad token, never an -empty array. Because both legs fold through the same hash, conformance -harnesses get identical ids for identical input. The defaults (vocabulary -30,522, maximum 128 tokens) match the BERT family. The critical caveat: -feeding these ids into a real embedding model produces garbage, because -they have no relation to the model's true vocabulary. Real WordPiece and -SentencePiece tokenizers arrive with the version 1.1 model-bundle mission. +rules first. Each word then folds through FNV-1a into an id in the range +two through the vocabulary size. Ids zero and one are reserved +sentinels, for padding and unknown tokens. Empty input returns a single +pad token, never an empty array. Since both legs fold through the same +hash, conformance harnesses get the same ids for the same input. The +defaults match the BERT family. The vocabulary size is 30522. The token +maximum is 128. One caveat matters most: feeding these ids into a real +embedding model produces garbage, since they carry no relation to the +model's true vocabulary. Real WordPiece and SentencePiece tokenizers +arrive with the version 1.1 model-bundle mission. ## TermDocumentCounts.swift -This file provides `TermDocumentCounts`, the shared count builder feeding -the LSA and NMF providers. - -It owns three things: the vocabulary, built in encounter order (a term's -column index is fixed by the first document that mentions it, which keeps -matrix columns stable for a fixed document sequence); per-document term -frequencies; and per-term document frequencies. It deliberately does not -own weighting or factorization — those belong to the consuming providers, -which weight the same counts differently. `addDocument(_:)` is the full -training path. `addDocumentForCountsAnchor(_:)` is the lightweight -incremental path: it grows the vocabulary and the document count but keeps -no frequencies, because the heavy inputs are re-derived by re-tokenizing -the corpus at retrain time, bounding maintained state to the vocabulary's +This file holds `TermDocumentCounts`, the shared count builder that +feeds the LSA and NMF providers. + +It owns three things: the vocabulary, per-document term frequencies, and +per-term document frequencies. The vocabulary builds in encounter order. +A term's column index is fixed by the first document that mentions it, +which keeps matrix columns stable for a fixed document sequence. The +builder does not own weighting or factoring, on purpose. Those +belong to the consuming providers, which weight the same counts +differently. `addDocument(_:)` is the full training path. +`addDocumentForCountsAnchor(_:)` is the lightweight incremental path +instead. It grows the vocabulary and the document count, but it keeps no +frequencies, since the heavy inputs get re-derived by re-tokenizing the +corpus at retrain time. That bounds maintained state to the vocabulary's size. The restored-vocabulary initializer likewise seeds a deserialized provider with truthful metadata and empty frequency rows. The builder is -not thread-safe; all writes must finish before reads. +not thread-safe. All writes must finish before any reads. ## ReducedVocab.swift -This file provides the shared vocabulary-reduction step for the dense +This file holds the shared vocabulary-reduction step for the dense factorizations, LSA and NMF. A dense matrix over tens of thousands of terms is unfactorizable on a -device — the comment estimates ten-to-the-fifteenth operations. The fix is -to keep only the most informative columns. Below the cap (512 by default) -the function is a strict no-op, so small estates and every conformance -fixture behave exactly as before reduction existed. Above the cap it drops -terms seen in only one document (pure noise), ranks the rest by document -frequency descending — terms that co-occur across many documents carry the -latent structure a factorization can find — and breaks ties by raw UTF-8 -byte order, matching Rust's string ordering so both legs select identical -vocabularies. The selection is shared rather than per-provider because -informativeness is a corpus property, identical for both factorizations. -`ReducedVocabulary` freezes the kept terms, the projection map, and the -row-remapping table. +device. The code comment estimates ten-to-the-fifteenth operations for +that case. The fix is to keep only the most informative columns. +Below the cap, 512 by default, the function is a strict no-op. Small +estates and each conformance fixture behave just as before reduction +existed. Above the cap, the function drops terms seen in only one +document, since those are pure noise. It ranks the rest by document +frequency descending, since terms that co-occur across many documents +carry the latent structure a factorization can find. It breaks ties by +raw UTF-8 byte order, to match Rust's string ordering, so both legs +select the same vocabularies. The selection is shared rather than +per-provider, since informativeness is a corpus property. It stays the +same for both factorizations. `ReducedVocabulary` freezes the kept +terms, the projection map, and the row-remapping table. ## RandomIndexingProvider.swift -This file provides `RandomIndexingProvider`, the first honest +This file holds `RandomIndexingProvider`, the first honest distributional signal in the dense lane. -Random Indexing gives every term a deterministic sparse "index vector": -2,048 dimensions with exactly ten nonzero entries of plus or minus one. +Random Indexing gives each term a deterministic sparse "index vector": +2,048 dimensions, with exactly ten nonzero entries of plus or minus one. The generator seeds a counter-based random stream from the FNV hash of -the lowercased term and draws exactly twenty values — ten positions, ten -signs — with collisions resolved last-wins rather than by rejection, so -the draw count is constant and the Swift and Rust streams stay aligned. A -term's meaning is then learned by addition: sliding a window of four over -training text, each term accumulates the index vectors of its neighbors -into a context vector. Terms that keep similar company converge, which is -genuine co-occurrence semantics at almost no computational cost, and the -accumulation is incremental by construction. +the lowercased term. It draws exactly twenty values: ten positions, then +ten signs. Collisions resolve last-wins rather than by rejection, so the +draw count stays constant and the Swift and Rust streams stay aligned. A +term's meaning is then learned by addition. A window of four slides over +the training text, and each term builds up the index vectors of its +neighbors into a context vector. Terms that keep similar company +converge. This is genuine co-occurrence semantics at almost no +computational cost, and the accumulation is incremental by construction. Embedding text sums the context vectors of its in-vocabulary terms and -normalizes to unit length. The float lane is honest about failure: an -untrained provider opts out with an empty vector, and a trained provider +normalizes to unit length. The float lane is honest about failure. An +untrained provider opts out with an empty vector. A trained provider whose query is entirely out of vocabulary throws a typed vocabulary-miss -error that the facade maps to the right dark-lane outcome. The basis blob -(magic `RIB1`) is the whole vocabulary map — Random Indexing has no -separate finalize step — and the counts blob (`RICT`) carries the same -payload under a distinct magic so a counts row can never be misread as a -basis row. The projection seed spells `RI_V1_MX`. +error instead, and the facade maps that error to the right dark-lane +outcome. The basis blob carries the magic tag `RIB1`. It is the whole +vocabulary map, since Random Indexing has no separate finalize step. +The counts blob carries the same payload, under the distinct magic tag +`RICT`. A counts row can so never be misread as a basis row. The +projection seed spells +`RI_V1_MX`. ## PpmiProvider.swift -This file provides `PpmiProvider`, the co-occurrence signal weighted by +This file holds `PpmiProvider`, the co-occurrence signal weighted by positive pointwise mutual information. -PPMI asks of each word pair: do these words co-occur more than chance -would predict? The score is the logarithm of the observed co-occurrence +PPMI asks of each word pair whether the words co-occur more than chance +would predict. The score is the logarithm of the observed co-occurrence probability over the product of the individual probabilities, floored at -zero. Frequent-but-meaningless neighbors (the "the" problem) score near -zero; genuinely associated pairs keep full weight. Training is two-phase. -Phase one counts: a sliding window of four accumulates pair counts, term -counts, and totals, additively across calls. Phase two, `finalize()`, -converts counts to context vectors: each term's vector is the -PPMI-weighted sum of its neighbors' Random Indexing index vectors, in the -same 2,048-dimension space with the same generator, so the two signals are -directly comparable. The file warns explicitly that this is mathematically -distinct from unweighted Random Indexing and must not be "simplified" -into it. - -Embedding, opt-out, and vocabulary-miss behavior mirror the Random -Indexing provider. The basis blob (`PPB1`) persists only the derived -vectors; the count tables are training scratch. The counts blob (`PPMC`) -persists the raw additive state — including the nested pair-count map, -serialized with byte-sorted keys — so a retrain can resume counting -without re-tokenizing. Stored vectors are kept unnormalized; -normalization happens at embed time so tests can inspect raw sums. The -projection seed spells `PPMI_V1M` and must never equal the Random -Indexing seed, because the seed partitions vector storage by model. +zero. Frequent-but-meaningless neighbors, the "the" problem, score near +zero. Genuinely associated pairs keep full weight instead. Training runs +in two phases. Phase one counts: a sliding window of four builds up +pair counts, term counts, and totals, additively across calls. Phase +two, `finalize()`, converts counts into context vectors. Each term's +vector becomes the PPMI-weighted sum of its neighbors' Random Indexing +index vectors. That sum lives in the same 2048-dimension space, with +the same generator. The two signals stay directly comparable, as a +result. The file warns outright that this differs mathematically from +unweighted Random +Indexing, and it must never be "simplified" into it. + +Embedding, opt-out, and vocabulary-miss behavior all mirror the Random +Indexing provider. The basis blob, `PPB1`, persists only the derived +vectors. The count tables are training scratch instead. The counts +blob, `PPMC`, persists the raw additive state. That state includes the +nested pair-count map. The map serializes with byte-sorted keys. A +retrain can then resume counting, without re-tokenizing. Stored vectors stay unnormalized. +Normalization happens at embed time instead, so tests can inspect raw +sums. The projection seed spells `PPMI_V1M`. It must never equal the +Random Indexing seed, since the seed partitions vector storage by +model. ## LsaProvider.swift -This file provides `LsaProvider`, the latent semantic analysis signal. - -LSA finds hidden topic structure by factorizing a term-document matrix -with the singular value decomposition (SVD). Words that appear in similar -documents land near each other in the latent space, so synonyms co-locate -even when they never co-occur. Training builds on the shared count -builder: reduce the vocabulary (the ADR-022 step), weight each cell by -log term frequency times smoothed IDF (add-one smoothing on both sides, -so an unseen query term still gets positive weight), and decompose with -`JacobiSVD` from SubstrateML at rank 64. The Jacobi method is chosen -because its sweep count is pinned — thirty sweeps, exactly — making the -factorization a fixed computation rather than a convergence-dependent -one; changing the sweep count invalidates every conformance vector. Wide -matrices are transposed for the decomposition and the factors swapped -back, so downstream code sees one orientation. - -New text embeds by the classical fold-in formula: project the query's -weighted term vector through the factor matrix, scaling by the inverse -singular values and skipping values below a small floor. Training -documents use the exact projection instead. The dark-lane contract -matches the other trainable providers. The basis blob (`LSB1`) persists -the reduced vocabulary, the IDF weights, and the raw factor matrices — -port-neutral, so each leg re-derives its own document vectors — and the -counts blob (`LSAC`) persists only the vocabulary and document-count -anchors, per the re-tokenize-at-retrain decision. The projection seed -spells `LSA_V1_M`. +This file holds `LsaProvider`, the latent semantic analysis signal. + +LSA finds hidden topic structure by factorizing a term-document matrix, +with the singular value decomposition, or SVD. Words that appear in +similar documents land near each other in the latent space, so synonyms +co-locate even when they never co-occur. Training builds on the shared +count builder. It reduces the vocabulary first, the ADR-022 step. It +weights each cell by log term frequency times smoothed IDF, with +add-one smoothing on both sides, so an unseen query term still gets a +positive weight. It decomposes with `JacobiSVD` from SubstrateML, at +rank 64. The Jacobi method is chosen since its sweep count is pinned +at exactly thirty sweeps. That makes the factorization a fixed +computation, rather than a convergence-dependent one. Changing the sweep +count would invalidate each conformance vector. Wide matrices transpose +for the decomposition, and the factors swap back afterward, so +downstream code always sees one orientation. + +New text embeds through the classical fold-in formula. The query's +weighted term vector projects through the factor matrix. It scales by +the inverse singular values, and it skips values below a small floor. +Training documents use the exact projection instead. The dark-lane +contract matches the other trainable providers. The basis blob carries +the magic tag `LSB1`. It persists three things: the reduced vocabulary, +the IDF weights, and the raw factor matrices. This form is port-neutral, +so each leg re-derives its own document vectors. The counts blob +carries the magic tag `LSAC`. It persists only the vocabulary and +document-count anchors, per the re-tokenize-at-retrain decision. The +projection seed spells `LSA_V1_M`. ## NmfProvider.swift -This file provides `NmfProvider`, the non-negative matrix factorization +This file holds `NmfProvider`, the non-negative matrix factorization signal. -NMF factorizes the term-document matrix into two non-negative factors, so -every latent dimension reads as an additive combination of terms — parts, -not contrasts. The matrix is oriented terms-by-documents, weighted by log -term frequency only: NMF requires non-negative input, and its update -steps are most stable with uniformly scaled entries, so IDF is deliberately -omitted. Factorization reuses SubstrateML's alternating least squares at -rank 32 with two pinned determinism devices: the convergence tolerance is -set to zero, which forces exactly one hundred iterations on every platform -regardless of floating-point convergence behavior, and factor -initialization is seeded with a fixed constant. Document embeddings are -the normalized factor columns, precomputed at finalize. +NMF factorizes the term-document matrix into two non-negative factors, +so each latent dimension reads as an additive combination of terms: +parts, not contrasts. The matrix orients terms-by-documents, weighted by +log term frequency only. NMF requires non-negative input, and its +update steps are most stable with uniformly scaled entries, so IDF is +left out on purpose. Factorization reuses SubstrateML's alternating +least squares, at rank 32, with two pinned determinism devices. The +convergence tolerance sets to zero, which forces exactly one hundred +iterations on each platform, regardless of floating-point convergence +behavior. Factor initialization seeds with a fixed constant. Document +embeddings are the normalized factor columns, precomputed at finalize. Queries fold in through a pseudo-inverse projection onto each factor column, with a small epsilon guarding the denominator, then normalize. -Serialization follows the family pattern: the basis blob (`NMB1`) carries -configuration, the reduced vocabulary, and both raw factor matrices; the -counts blob (`NMFC`) carries anchors only. The projection seed spells -`NMF_V1_M`. The dark-lane contract matches the other trainable providers. +Serialization follows the family pattern. The basis blob, `NMB1`, +carries configuration, the reduced vocabulary, and both raw factor +matrices. The counts blob, `NMFC`, carries anchors only. The projection +seed spells `NMF_V1_M`. The dark-lane contract matches the other +trainable providers. ## FdcProvider.swift -This file provides `FDCProvider`, the taxonomic co-classification signal — -the one honest signal that needs no training at all. - -The provider classifies text with LatticeLib's FDC engine, which lives in -the moot-semantics repository and is not reimplemented here. `FDC.encode` -returns a lattice code or nothing; `FDC.ancestors` returns the code's -chain up the classification hierarchy. The provider turns that chain into -geometry: each node in the chain gets a deterministic 256-dimension unit -vector generated from the hash of its code string, and the vectors sum -with weight one over depth-plus-one — shared roots give any two texts a -similarity floor, and shared deep ancestors add to it. Texts filed near -each other in the subject hierarchy therefore embed near each other, -regardless of surface wording. The node-vector generator is deliberately -a different pipeline from Random Indexing's, and the input domains cannot -collide. +This file holds `FDCProvider`, the taxonomic co-classification +signal. It is the one honest signal that needs no training at all. + +The provider classifies text with LatticeLib's FDC engine. That engine +lives in the moot-semantics repository, and it is not reimplemented +here. `FDC.encode` returns a lattice code, or nothing. `FDC.ancestors` +returns the code's chain up the classification hierarchy. The provider +turns that chain into geometry. Each node in the chain gets a +deterministic 256-dimension unit vector, generated from the hash of its +code string. The vectors sum with weight one over depth-plus-one. Shared +roots give any two texts a similarity floor, and shared deep ancestors +add to it. Texts filed near each other in the subject hierarchy then +embed near each other, regardless of surface wording. The node-vector +generator runs a different pipeline from Random Indexing's, on purpose. +The two input domains cannot collide. Honesty governs the edges. Text the classifier cannot resolve returns -UNRESOLVED, and the provider opts out with an empty float vector and a -zero engram — unclassifiable text must not contribute false similarity. -The provider is stateless and `Sendable`; its determinism rests on -LatticeLib's own agreement property. The projection seed spells -`FDC_V1_P`, and the free function `fdcNodeVector(code:)` is public so +UNRESOLVED. The provider then opts out, with an empty float vector and a +zero engram, since unclassifiable text must not contribute false +similarity. The provider is stateless and `Sendable`. Its determinism +rests on LatticeLib's own agreement property. The projection seed spells +`FDC_V1_P`. The free function `fdcNodeVector(code:)` is public, so conformance tests can pin individual node vectors. ## DefaultEnsemble.swift -This file provides `CorpusEnsemble.defaultEnsemble()`, the single source +This file holds `CorpusEnsemble.defaultEnsemble()`, the single source of truth for the production recall ensemble. The factory returns the five honest signals in fixed order: Random -Indexing, PPMI, LSA, NMF, FDC. Order is load-bearing — the first element -becomes the corpus's default signal. It is a function rather than a shared -constant because the four trainable providers are reference types holding -mutable trained state; a shared array would alias one provider instance -across every estate. Fresh construction gives each estate its own -untrained providers, which the corpus lifecycle then trains and persists -under their own model identifiers. The factory lives in the providers -target because it names concrete types; the core's `EmbeddingModel` enum -never does. +Indexing, PPMI, LSA, NMF, and FDC. Order is load-bearing here, since +the first element becomes the corpus's default signal. It is a function +rather than a shared constant, since the four trainable providers are +reference types holding mutable trained state. A shared array would +alias one provider instance across each estate. Fresh construction +gives each estate its own untrained providers instead. The corpus +lifecycle then trains and persists those providers under their own +model identifiers. The factory lives in the providers target, since +it names concrete types, and the core's `EmbeddingModel` enum never +does. ## MiniLMTextProvider.swift, MPNetTextProvider.swift, EmbeddingGemmaProvider.swift -These three files provide the named neural embedding providers: MiniLM-L6 -v2 (384 dimensions), mpnet-base-v2 (768), and EmbeddingGemma 300M (768). -They share one structure, so they are described together; each fact below -holds for all three unless noted. - -Each provider runs the same pipeline: tokenize, run the host-injected -inference closure to get a pooled float vector, and project that vector to -a 256-bit engram with `FloatSimHash` from SubstrateML. An engram is a -fixed-size binary fingerprint; similar vectors produce similar engrams, so -the vector lane can compare chunks by fast Hamming distance. The inference -closure is the doctrine's model seam: the provider never holds a CoreML -model, which keeps it testable without a model bundle and leaves model -loading — different on iOS, macOS, and CI — to the host, composed once at -startup. Each provider pins its own projection seed so its fingerprints -are model-tagged: `MINLM_v1`, `MPNET_v1`, and `EMBGM_v1` in ASCII. Seeds -partition vector storage by model and must never change or collide. - -Each exposes three surfaces. `embed` returns the engram; `embedFloat` -returns the raw pooled vector for the true-cosine float lane; `embedPair` -runs inference once and derives both, halving the cost of ingest, which -needs both per chunk. Empty input short-circuits to a zero engram and an -empty vector. All three default to the `DeterministicTokenizer` stand-in — -EmbeddingGemma's is sized for its SentencePiece vocabulary of 256,000 and -context of 2,048 tokens — so until real tokenizers ship, embedding values -are a property of the host's model bundle. What the kit itself owns, the -token-to-engram pipeline given a pooled vector, is bit-identical across -ports. +These three files hold the named neural embedding providers. +MiniLM-L6 v2 uses 384 dimensions. mpnet-base-v2 uses 768. EmbeddingGemma +300M also uses 768. They share one structure, so this section describes +them together. Each fact below holds for all three, unless the text +says otherwise. + +Each provider runs the same pipeline. It tokenizes the input. It runs +the host-injected inference closure to get a pooled float vector. It +projects that vector to a 256-bit engram with `FloatSimHash` from +SubstrateML. An engram is a fixed-size binary fingerprint. Similar +vectors produce similar engrams, so the vector lane can compare chunks +by fast Hamming distance. The inference closure is the doctrine's model +seam. The provider never holds a CoreML model, which keeps it testable +without a model bundle. Model loading differs on iOS, macOS, and CI. +The host owns that step. It composes the model once, at startup. Each +provider pins its own projection seed. The seed keeps fingerprints +model-tagged, in ASCII: `MINLM_v1`, `MPNET_v1`, and `EMBGM_v1`. Seeds +partition vector storage by model, and they must never change or +collide. + +Each provider exposes three surfaces. `embed` returns the engram. +`embedFloat` returns the raw pooled vector, for the true-cosine float +lane. `embedPair` runs inference once and derives both results, which +halves the cost of ingest, since ingest needs both per chunk. Empty +input short-circuits to a zero engram and an empty vector. All three +default to the `DeterministicTokenizer` stand-in. EmbeddingGemma's copy +is sized for its own SentencePiece vocabulary. That vocabulary holds +256000 terms, with a context of 2048 tokens. Until real tokenizers ship, embedding values stay a +property of the host's model bundle. What the kit itself owns, the +token-to-engram pipeline given a pooled vector, stays bit-identical +across ports. ## NLEmbeddingProvider.swift -This file provides `NLEmbeddingProvider`, the Apple sentence-embedding +This file holds `NLEmbeddingProvider`, the Apple sentence-embedding signal built on the operating system's bundled `NLEmbedding` model. -It is the cheap, immediate Apple-native lane: no download, no CoreML seam, -no training — the OS framework is the model, so there is nothing for a -host to inject. The provider looks up the OS sentence model for its -configured language, embeds the text, casts the result to floats, and -normalizes to unit length with the substrate's canonical vector -operations, keeping the float lane's cosine on a unit sphere like every -other provider. Absence is graceful: no model for the language, or text -the model cannot embed, returns an empty vector — a typed lane opt-out, -not an error. The projection seed spells `APNLEMB1`, defined here rather -than in the doctrine's CoreML seed table but under the same never-collide -rule. The file is compiled only where NaturalLanguage exists; there is no -Rust counterpart, a sanctioned Swift-only divergence recorded in ADR-019, -and recall fusion simply handles the absent lane elsewhere. +It is the cheap, immediate Apple-native lane. It needs no download. It +needs no CoreML seam. It needs no training either, since the OS +framework is the model itself. There is nothing for a host to inject. + +The provider looks up the OS sentence model for its configured +language. It embeds the text. It casts the result to floats. It then +normalizes the vector to unit length, using the substrate's canonical +vector operations. This keeps the float lane's cosine on a unit sphere, +like each other provider. + +Absence stays graceful. Two cases return an empty vector: no model +exists for the language, or the model cannot embed the given text. +Either case counts as a typed lane opt-out, never an error. + +The projection seed spells `APNLEMB1`. It is defined here, rather than +in the doctrine's CoreML seed table, but it still follows the same +never-collide rule. The file compiles only where NaturalLanguage +exists. There is no Rust counterpart here. ADR-019 records this as a +sanctioned Swift-only divergence. Recall fusion simply handles the +absent lane elsewhere. ## NLContextualEmbeddingProvider.swift -This file provides `NLContextualEmbeddingProvider`, the higher-quality +This file holds `NLContextualEmbeddingProvider`, the higher-quality Apple lane built on the on-device `NLContextualEmbedding` transformer. The transformer needs a per-language downloadable asset that may be -absent, and the file's central rule is that an embed call must never +absent. The file's central rule holds that an embed call must never trigger a network fetch as a side effect. The provider checks asset -availability with a free, synchronous call and opts out with an empty -vector when the asset is missing; prefetching assets is the host -application's job, done before constructing the provider. When assets -exist, the provider runs the transformer, mean-pools the per-token vectors -(the conventional strategy for a transformer without a sentence-pooling -head, with a defensive dimension guard that skips malformed token -vectors), and normalizes. Every failure mode collapses to the empty-vector -opt-out, because a missing asset is an expected operational state. The -projection seed spells `APNLCTX1`, distinct from the sentence provider's -so the two lanes key to separate storage partitions. Apple-only, no Rust +availability with a free, synchronous call. It opts out with an empty +vector when the asset is missing. Prefetching assets is the host +application's job, done before the provider gets constructed. When +assets exist, the provider runs the transformer. It mean-pools the +per-token vectors, the conventional strategy for a transformer without a +sentence-pooling head, with a defensive dimension guard that skips +malformed token vectors. Then it normalizes the result. Each failure +mode collapses to the empty-vector opt-out, since a missing asset is +an expected operational state. The projection seed spells `APNLCTX1`, +distinct from the sentence provider's, so the two lanes key to separate +storage partitions. This provider is Apple-only, with no Rust counterpart, per ADR-019. ## Rust Port and Conformance -The `rust/` directory mirrors the core target and the `rust-providers/` -directory mirrors the providers target, matching the two-target split so -core consumers never depend on provider code. The core crate ships the -same chunk, chunker, tokenizer, store, engine, hybrid-recall, ingest-queue, -and `Corpus` types; concurrency uses locks where Swift uses actors, and -the queue and inverted-index store own private connections where Swift -shares the estate's storage. The providers crate ships the deterministic -tokenizer, the shared count and vocabulary-reduction code, the basis -codec, the five honest signals, and the three named neural providers over -the same host-injected inference seam. Neither crate bundles model -weights. +The `rust/` directory mirrors the core target. The `rust-providers/` +directory mirrors the providers target, matching the two-target split, +so core consumers never depend on provider code. The core crate ships +the same chunk, chunker, tokenizer, store, engine, hybrid-recall, +ingest-queue, and `Corpus` types. Concurrency uses locks where Swift uses +actors. The queue and the inverted-index store own private connections +there, where Swift instead shares the estate's storage. The providers +crate ships the deterministic tokenizer and the shared +count-and-vocabulary-reduction code. It also ships the basis codec, the +five honest signals, and the three named neural providers. All of them +share the same host-injected inference seam. Neither crate bundles +model weights. Three fixture families gate the ports. The shared canonical vectors in -`Tests/SharedVectors/` — BM25 impacts, per-provider embedding vectors, and -serialized basis blobs — are read by both legs and must reproduce byte for -byte. The Rust test suite additionally pins basis serialization -byte-for-byte for all four trainable providers and round-trips the counts -seam. The two Apple NaturalLanguage providers are the one sanctioned -divergence: they exist only in Swift, and the parity baseline is the -deterministic and classical providers. When you change either leg, run -both test suites; the fixtures are the contract. +`Tests/SharedVectors/` cover BM25 impacts, per-provider embedding +vectors, and serialized basis blobs. Both legs read them, and both must +reproduce them byte for byte. The Rust test suite also pins +basis serialization byte-for-byte for all four trainable providers, and +it round-trips the counts seam. The two Apple NaturalLanguage providers +are the one sanctioned divergence. They exist only in Swift, and the +parity baseline is the deterministic and classical providers. When you +change either leg, run both test suites. The fixtures are the contract. diff --git a/packages/kits/CorpusKit/docs/OVERVIEW.md b/packages/kits/CorpusKit/docs/OVERVIEW.md index 4cf6972..b84e656 100644 --- a/packages/kits/CorpusKit/docs/OVERVIEW.md +++ b/packages/kits/CorpusKit/docs/OVERVIEW.md @@ -79,124 +79,145 @@ sources: ## What This Kit Does -CorpusKit stores text and finds it again by meaning as well as by keywords. -It is the retrieval tier of MOOTx01, an on-device AI memory system. The -technique it implements is called retrieval-augmented generation, or RAG. In -RAG, an AI looks up relevant stored material before it answers, so its -answers rest on real text instead of on guesswork. - -A kit is a larger package that composes libraries into a subsystem. CorpusKit -composes storage, indexing, and embedding libraries into one database-like -surface. Callers hand it text; it splits the text into chunks, stores each -chunk, and indexes it two ways. A chunk is a piece of source text with a -stable identity, sized to a few sentences. Later, callers hand it a query and -receive the most relevant chunks back, scored and ranked. - -The kit stands alone. A developer can use it as a private RAG database with -no other MOOT components. Inside MOOTx01, the GeniusLocusKit orchestrator -uses it as the estate's recall engine. An estate is one user's complete -memory store. +CorpusKit stores text. It finds that text again by meaning, and also by +keyword. It is the retrieval tier of MOOTx01, an on-device AI memory +system. The technique it uses has a name: retrieval-augmented generation, +or RAG. In RAG, an AI looks up stored material first. Then it answers. +The answer rests on real text, not on guesswork. + +A kit is a larger package. It composes libraries into one subsystem. +CorpusKit composes storage, indexing, and embedding libraries. Together +they form one database-like surface. Callers hand it text. The kit +splits the text into chunks. It stores each chunk, then indexes it two +ways. A chunk is a piece of source text with a stable identity. Each +chunk is sized to a few sentences. Later, a caller hands the kit a +query. The kit returns the most relevant chunks, scored and ranked. + +The kit stands alone. A developer can use it as a private RAG database. +No other MOOT component is required. Inside MOOTx01, the GeniusLocusKit +orchestrator uses CorpusKit differently. It treats the kit as the +estate's recall engine. An estate is one user's complete memory store. ## The Problem It Solves -Recall must work on the device, deterministically, and without leaking text. -Cloud embedding services see private content, need a network, and change -without notice. If recall depended on them, a user's memory would be neither -private nor reproducible. Federation raises the stakes: MOOTx01 estates can -share memories across devices, and shared recall only works when every -device computes the same result from the same input — the agreement -property. - -CorpusKit answers with two ranked lanes that both run entirely on device. A -lane is one independent way of scoring how well a chunk matches a query. The -keyword lane uses BM25, a standard formula that rewards chunks containing -the query's rarer words. The semantic lane uses embeddings. An embedding is -a list of numbers (a vector) that represents what a text means; texts with -similar meaning get nearby vectors. The two lanes are fused into one ranking -by Reciprocal Rank Fusion, a simple rule that rewards chunks ranked high in -either lane. - -For the semantic lane, the kit ships an ensemble of five "honest" signals: -Random Indexing, PPMI, LSA, NMF, and FDC. Honest means each signal reflects -real word co-occurrence or real classification structure, never a disguised -hash of the surface text. All five are classical statistical methods. They -need no neural network, cost little, run identically on every platform, and -are gated by shared conformance fixtures — recorded input and output pairs -that the Swift leg and the Rust leg (in `rust/` and `rust-providers/`) must -reproduce exactly, byte for byte. Optional higher-quality neural providers -(MiniLM, mpnet, EmbeddingGemma, and two Apple NaturalLanguage providers) -plug into the same seam when a host supplies the model. +Recall must work on the device. It must be deterministic. It must never +leak text. Cloud embedding services fail all three tests. They see +private content. They need a network connection. They change without +notice. If recall depended on such a service, a user's memory would be +neither private nor reproducible. + +Federation raises the stakes further. MOOTx01 estates can share memories +across devices. Shared recall only works when every device computes the +same result from the same input. Call this the agreement property. + +CorpusKit answers with two ranked lanes. Both run entirely on the +device. A lane is one independent way to score how well a chunk matches +a query. The keyword lane uses BM25. BM25 is a standard formula. It +rewards chunks that contain the query's rarer words. The semantic lane +uses embeddings instead. An embedding is a list of numbers, called a +vector, that represents what a text means. Texts with similar meaning +get nearby vectors. A rule called Reciprocal Rank Fusion then merges the +two lanes into one ranking. The rule is simple: it rewards any chunk +ranked high in either lane. + +For the semantic lane, the kit ships an ensemble of five signals. Each +one is "honest." Honest means the signal reflects real word +co-occurrence, or real classification structure. It is never a disguised +hash of the surface text. The five signals are Random Indexing, PPMI, +LSA, NMF, and FDC. All five are classical statistical methods. None +needs a neural network. Each costs little to run. Each runs identically +on every platform. Shared conformance fixtures gate all five: recorded +input and output pairs that the Swift leg and the Rust leg must +reproduce exactly, byte for byte. The Rust leg lives in `rust/` and +`rust-providers/`. Optional neural providers plug into the same seam +when a host supplies a model. Four ship today: MiniLM, mpnet, +EmbeddingGemma, and two Apple NaturalLanguage providers. ## How It Works -Ingestion runs in a pipeline. Text enters through an ingest queue backed by -QueueKit, so callers never wait on encoding. A background drain worker takes -batches from the queue and hands them to the `Corpus` actor, the kit's -central type. The `Corpus` splits each text into chunks with sentence-aware -boundaries. Each chunk receives a content-addressed identity: its UUID is -computed from its source, offset, and exact text. The same content always -produces the same identity, so re-ingesting a document is a harmless no-op -and two federated devices converge on identical rows. - -Each stored chunk is then indexed twice. The keyword side tokenizes the -chunk and records term frequencies in a persistent inverted index — a table -mapping each word to the chunks that contain it. The semantic side runs the -chunk through every configured embedding signal and writes the resulting -vectors to VectorKit, the sibling kit that owns vector storage and -nearest-neighbor search. Content and vectors are joined by the chunk's UUID -string, so a chunk and its meaning never drift apart. - -Four of the five honest signals are trainable: they learn a basis from the -corpus itself. A basis is the trained reference data a signal needs to embed -new text — for example, the word co-occurrence vectors Random Indexing -accumulates. Training happens exactly twice: automatically on first ingest, -and again whenever a caller requests an explicit reindex. Trained bases are -serialized to a pinned little-endian byte format and persisted, so a -reopened corpus embeds immediately without retraining. - -Recall runs the pipeline in reverse. The query is embedded once, the vector -lane fetches its nearest neighbors, the keyword lane fetches its best BM25 -matches, and Reciprocal Rank Fusion merges the two rankings with pinned -weights (0.6 vector, 0.4 keyword). The winners are hydrated from the chunk -store and returned as scored chunks. Ties always break toward the smaller -identifier, so results are deterministic down to the last position. - -Deletion is honest about its limits. Chunk rows are immutable, so removing a -source deletes its index rows and vectors and records a tombstone that every -rebuild consults; expunging additionally scrubs the stored text itself. +Ingestion runs as a pipeline. Text enters through an ingest queue backed +by QueueKit. Callers never wait on encoding to finish. A background +drain worker takes batches from the queue. It hands each batch to the +`Corpus` actor, the kit's central type. The `Corpus` splits each text +into chunks with sentence-aware boundaries. Each chunk then receives a +content-addressed identity. Its UUID is computed from its source, its +offset, and its exact text. The same content always produces the same +identity. A repeat ingest of one document is therefore a harmless +no-op. Two federated devices that ingest the same content converge on +identical rows. + +Each stored chunk is indexed twice. The keyword side tokenizes the +chunk. It records term frequencies in a persistent inverted index. An +inverted index is a table that maps each word to the chunks that contain +it. The semantic side runs the chunk through every configured embedding +signal. It writes the resulting vectors to VectorKit, the sibling kit +that owns vector storage and nearest-neighbor search. Content and +vectors join on the chunk's UUID string. A chunk and its meaning never +drift apart. + +Four of the five honest signals are trainable. They learn a basis from +the corpus itself. A basis is the trained reference data a signal needs +to embed new text. One example is the word co-occurrence vectors that +Random Indexing accumulates. Training happens exactly twice: once +automatically on first ingest, and again whenever a caller requests an +explicit reindex. Trained bases serialize to a pinned little-endian byte +format and persist to storage. A reopened corpus embeds text right away. +It never has to retrain first. + +Recall runs the same pipeline in reverse. The query is embedded once. +The vector lane fetches its nearest neighbors. The keyword lane fetches +its best BM25 matches. Reciprocal Rank Fusion merges the two rankings. +It uses pinned weights: 0.6 for the vector lane and 0.4 for the keyword +lane. The winning chunks hydrate from the chunk store, then return to +the caller as scored chunks. Ties always break toward the smaller +identifier. Results stay deterministic down to the last position. + +Deletion is honest about its own limits. Chunk rows are immutable, so +removing a source cannot erase its chunk rows. Instead it deletes the +source's index rows and vectors. It also records a tombstone, which +every rebuild consults afterward. Expunging goes one step further: it +scrubs the stored text itself. ## How the Pieces Fit -Figure 1 shows the kit's topology — its major parts and how data moves -between them. +Figure 1 shows the kit's topology. It shows the major parts, and how +data moves between them. ![Figure 1. Topology of CorpusKit](topology.svg) -*Figure 1. Topology of CorpusKit. Ingested text flows through the queue and -the `Corpus` actor into the chunk store, the keyword index, and the vector -store. A query fans out to both index lanes, and Reciprocal Rank Fusion -merges them into scored chunks. Dashed regions mark the external kits and -the persisted tables.* - -The `Corpus` actor is the seam everything passes through. It owns the chunk -store (`BundleStore`), the persistent keyword index (`InvertedIndexStore`), -the basis and counts stores, the tombstone store, and one slot per embedding -signal. It hides VectorKit behind its own surface, so consumers never touch -vector storage directly. The engine layer beneath it — sparse types, BM25 -weighting, the WAND/Block-Max WAND inverted index, and the fusion function — -is pure computation with no storage of its own. - -The package splits into two targets on purpose. The `CorpusKit` core target -holds the storage, the engines, and the protocols. The `CorpusKitProviders` -target holds every concrete embedding provider and tokenizer. Consumers that -only need storage and BM25 never pull in provider code or model seams. +*Figure 1. Topology of CorpusKit. Ingested text flows through the queue +and the `Corpus` actor into the chunk store, the keyword index, and the +vector store. A query fans out to both index lanes. Reciprocal Rank +Fusion merges them into scored chunks. Dashed regions mark the external +kits and the persisted tables.* + +The `Corpus` actor is the seam that everything passes through. It owns +the chunk store, called `BundleStore`. It owns the persistent keyword +index, called `InvertedIndexStore`. It also owns the basis store, the +counts store, and the tombstone store. It holds one slot per embedding +signal, too. The actor hides VectorKit behind its own surface. Consumers +never touch vector storage directly. + +Beneath the actor sits an engine layer. The layer holds the sparse +types, the BM25 weighting code, the inverted index, and the fusion +function. The inverted index runs two exact algorithms, WAND and +Block-Max WAND. The whole engine layer is pure computation. It holds no +storage of its own. + +The package splits into two targets on purpose. The `CorpusKit` core +target holds the storage layer, the engines, and the protocols. The +`CorpusKitProviders` target holds every concrete embedding provider and +tokenizer. A consumer that needs only storage and BM25 search never has +to pull in provider code or model seams. ## What Ships in the Package -The package ships the two Swift targets, thirty-four Swift source files in -all, and two mirror Rust crates: `rust/` for the core and `rust-providers/` -for the providers. Shared canonical vectors in `Tests/SharedVectors/` gate -both legs: BM25 impacts, per-provider embeddings, and serialized basis blobs -must match byte for byte. The kit bundles no model weights. Hosts that want -neural embeddings inject an inference function; everything the kit itself -computes is deterministic and reproducible from the sources alone. +The package ships two Swift targets. Together they hold thirty-four +Swift source files. It also ships two mirror Rust crates: `rust/` for +the core and `rust-providers/` for the providers. Shared canonical +vectors in `Tests/SharedVectors/` gate both legs. Three things must +match byte for byte across the legs: BM25 impacts, per-provider +embeddings, and serialized basis blobs. The kit bundles no model weights +of its own. A host that wants neural embeddings must inject an inference +function. Everything the kit itself computes stays deterministic. It +stays reproducible from the sources alone. diff --git a/packages/kits/LocusKit/docs/AGENT_MAP.md b/packages/kits/LocusKit/docs/AGENT_MAP.md index 2f1c7f1..e1cac03 100644 --- a/packages/kits/LocusKit/docs/AGENT_MAP.md +++ b/packages/kits/LocusKit/docs/AGENT_MAP.md @@ -103,358 +103,358 @@ sources: blob: a5b9b8d3b5b7990ebcb9d9c6209564a7307aa13e --- -# AGENT_MAP — LocusKit +# AGENT_MAP : LocusKit PURPOSE: storage substrate for one MOOTx01 estate (MemPalace). Nine gated noun tables (drawer/tunnel/diary/kg_fact/proposal/association/learned_reference/source_catalog/node) over PersistenceKit; every state-changing write routes through SubstrateLib's AuditGate (legal-transition + forbidden-combination check) and appends one sealed AuditEvent atomically with the row update. Read side: RecallFrame → [Filter] chain → fingerprint-pruned, four-tier BitmapEvaluator (bitmap/structured/content/order) → paged RecallStream. Estate (actor) is the sole public facade; GeniusLocusKit composes on top. -DEPS: imports SubstrateLib+SubstrateTypes+SubstrateKernel (AuditGate, BitField, RowStateAutomaton, HLC, SimHash, ORReduce, FNV, MerkleHash — DO NOT REIMPLEMENT), SubstrateML (LexRank-adjacent kernels used indirectly), PersistenceKit (Storage/RowStore/AuditLog/Transaction abstraction — SQLite/Postgres/InMemory backends), IntellectusLib (opt-in telemetry, off-path ~1ns), LatticeLib (QIDClosure — pinned P31/P279 ancestor snapshot, DrawerFingerprint lattice block only). Imported by: GeniusLocusKit (composes on top — never the reverse). Rust port in rust/ (57 files under src/+tests/) mirrors every type + the bitmap/state/fingerprint math; conformance tests in rust/tests/*.rs gate byte-identity per axis. +DEPS: imports SubstrateLib+SubstrateTypes+SubstrateKernel (AuditGate, BitField, RowStateAutomaton, HLC, SimHash, ORReduce, FNV, MerkleHash : DO NOT REIMPLEMENT), SubstrateML (LexRank-adjacent kernels used indirectly), PersistenceKit (Storage/RowStore/AuditLog/Transaction abstraction : SQLite/Postgres/InMemory backends), IntellectusLib (opt-in telemetry, off-path ~1ns), LatticeLib (QIDClosure : pinned P31/P279 ancestor snapshot, DrawerFingerprint lattice block only). Imported by: GeniusLocusKit (composes on top : never the reverse). Rust port in rust/ (57 files under src/+tests/) mirrors every type + the bitmap/state/fingerprint math; conformance tests in rust/tests/*.rs gate byte-identity per axis. ENTRY POINTS (most callers need only these): -- Estate.swift:187 `Estate.open(storage:owner:identityKeyStore:) -> Estate` — open existing estate, validates bitmap_layout_version, mints/loads Ed25519 identity -- Estate.swift:314 `Estate.create(storage:owner:manifest:) -> Estate` — create new estate -- EstateVerbs.swift:69 `Estate.capture(_ frame: CaptureFrame) -> Drawer` — file a drawer -- EstateVerbs.swift:619 `Estate.recall(_ frame: RecallFrame) -> RecallStream` — query drawers -- EstateVerbs.swift:1228 `Estate.mutate(rowID:kind:payload:)` — move state/adjective axis -- EstateVerbs.swift:1030 `Estate.expunge(rowID:reason:confirmation:now:)` — hard delete + audit +- Estate.swift:187 `Estate.open(storage:owner:identityKeyStore:) -> Estate` : open existing estate, validates bitmap_layout_version, mints/loads Ed25519 identity +- Estate.swift:314 `Estate.create(storage:owner:manifest:) -> Estate` : create new estate +- EstateVerbs.swift:69 `Estate.capture(_ frame: CaptureFrame) -> Drawer` : file a drawer +- EstateVerbs.swift:619 `Estate.recall(_ frame: RecallFrame) -> RecallStream` : query drawers +- EstateVerbs.swift:1228 `Estate.mutate(rowID:kind:payload:)` : move state/adjective axis +- EstateVerbs.swift:1030 `Estate.expunge(rowID:reason:confirmation:now:)` : hard delete + audit ## Symbol Table -### Module — LocusKit.swift +### Module : LocusKit.swift - :1 doc-only file; no public symbols. Public types live in neighbouring files. -### Errors — LocusKitError.swift -- :11 `enum LocusKitError: Error, Sendable, Equatable` — every LocusKit failure mode +### Errors : LocusKitError.swift +- :11 `enum LocusKitError: Error, Sendable, Equatable` : every LocusKit failure mode - :18 `.databaseUnavailable(String)` / :21 `.drawerNotFound(id:)` / :23 `.tunnelNotFound` / :27 `.diaryEntryNotFound` / :32 `.recallTraceItemNotFound` - :38 `.sqliteError(String)` / :42 `.schemaTooNew(found:expected:)` -- :48 `.invalidContent(String)` — msg IS the contract, tests assert on text -- :59 `.disciplineViolation(from:to:reason:)` — illegal transition / forbidden combo; from/to are raw Int, not State (keeps LocusKitError dependency-free of Adjectives) -- :72 `.corruptStoredValue(table:column:storedText:)` — fail-loud, never fabricate a default +- :48 `.invalidContent(String)` : msg IS the contract, tests assert on text +- :59 `.disciplineViolation(from:to:reason:)` : illegal transition / forbidden combo; from/to are raw Int, not State (keeps LocusKitError dependency-free of Adjectives) +- :72 `.corruptStoredValue(table:column:storedText:)` : fail-loud, never fabricate a default - :81 `.notSupported(String)` -- :90 `description: String` — English render consumed by GLK's gate-rejection text parser (soft contract) +- :90 `description: String` : English render consumed by GLK's gate-rejection text parser (soft contract) -### Schema — LocusKitSchema.swift -- :80 `kitID = "LocusKit"` / :90 `version = 8` (no migration ladder — no estate data has shipped) -- :97 `static var schema: SchemaDeclaration` — 17 tables + generated columns + indices, ALL declarative (zero raw SQL) +### Schema : LocusKitSchema.swift +- :80 `kitID = "LocusKit"` / :90 `version = 8` (no migration ladder : no estate data has shipped) +- :97 `static var schema: SchemaDeclaration` : 17 tables + generated columns + indices, ALL declarative (zero raw SQL) - Every persistent table carries nullable `.json` `ext` column (ADR-012 forward-compat slot) -- Generated columns: `g_state_cluster` (adjectiveBitmap & 0x3F, on drawers/kg_facts/proposals), `g_provenance_source`/`g_provenance_confirmation`/`g_operational_channel` (drawers only) — indexed like ordinary columns -- Header comment: bitmap reservation map (assigned vs FREE bit ranges per column) — consult before claiming a new flag bit +- Generated columns: `g_state_cluster` (adjectiveBitmap & 0x3F, on drawers/kg_facts/proposals), `g_provenance_source`/`g_provenance_confirmation`/`g_operational_channel` (drawers only) : indexed like ordinary columns +- Header comment: bitmap reservation map (assigned vs FREE bit ranges per column) : consult before claiming a new flag bit -### Vocabulary — LocusKitVocabulary.swift -- :27 `enum LocusKitVocabulary` — LocusKit's operational+provenance FieldSlot union (adjective basis supplied by SubstrateLib itself) -- :61 `frozen() -> Result` — called once at DrawerStore.init; frozen for estate lifetime +### Vocabulary : LocusKitVocabulary.swift +- :27 `enum LocusKitVocabulary` : LocusKit's operational+provenance FieldSlot union (adjective basis supplied by SubstrateLib itself) +- :61 `frozen() -> Result` : called once at DrawerStore.init; frozen for estate lifetime -### Estate types — EstateTypes.swift +### Estate types : EstateTypes.swift - :15 `typealias RowID = String` -- :27 `struct FrameFilteredDrawers` — :30 `admissible: [Drawer]`, :32 `loadedIDs: Set` (loaded-but-filtered vs never-loaded distinction) -- :51 `struct OwnerCredentials` — :55 `ownerIdentifier: String` (non-empty enforced at Estate.open/create) -- :76 `struct LatticeAnchor: Equatable, Codable` — udcCode(required)/udcFacets/wikidataQID/wikidataQidsSecondary; :101 `.udc(_:)` convenience -- :117 `enum EstateError` — :121 substrateUnavailable / :128 manifestMismatch(key:found:expected:) / :133 emptyOwnerIdentifier / :140 keychainError(status:) - -### Estate facade — Estate.swift -- :27 `actor Estate` — owns :45 `store: DrawerStore` (internal, EstateVerbs-only), :50 `containerFP: ContainerFingerprintStore`, :57 `nodeStore: NodeStore` (public) -- :36 `expectedBitmapLayoutVersion = "v1.0"` — Estate.open refuses mismatch -- :187 `open(storage:owner:identityKeyStore:)` — mints Ed25519 keypair on first open (public key → manifest, private key → identityKeyStore); backfills containerFP via rebuildAll -- :314 `create(storage:owner:manifest:)` — no keypair minted (deferred to first open) -- :401 `close()` — no-op; caller's storage owns teardown -- :412/:429/:448 `allDrawers(...)` overloads — full/limit/hydrationLevel variants -- :462 `tombstonedLineageIDs() -> Set` — cluster-C block-set for vault re-import guard -- :474 `roomLevelFingerprints()` — passthrough to containerFP.roomLevelEntries -- :485/:494 `getDrawers(ids:...)` — batch by-id, hydration-level aware -- :527 `getDrawers(ids:matchingFrame:hydrationLevel:) -> FrameFilteredDrawers` — frame-faithful O(candidates) by-id load; forces .full when chain has content predicate -- :566 `hydrateBodies(ids:)` — late body hydration for dense-first pool pattern +- :27 `struct FrameFilteredDrawers` : :30 `admissible: [Drawer]`, :32 `loadedIDs: Set` (loaded-but-filtered vs never-loaded distinction) +- :51 `struct OwnerCredentials` : :55 `ownerIdentifier: String` (non-empty enforced at Estate.open/create) +- :76 `struct LatticeAnchor: Equatable, Codable` : udcCode(required)/udcFacets/wikidataQID/wikidataQidsSecondary; :101 `.udc(_:)` convenience +- :117 `enum EstateError` : :121 substrateUnavailable / :128 manifestMismatch(key:found:expected:) / :133 emptyOwnerIdentifier / :140 keychainError(status:) + +### Estate facade : Estate.swift +- :27 `actor Estate` : owns :45 `store: DrawerStore` (internal, EstateVerbs-only), :50 `containerFP: ContainerFingerprintStore`, :57 `nodeStore: NodeStore` (public) +- :36 `expectedBitmapLayoutVersion = "v1.0"` : Estate.open refuses mismatch +- :187 `open(storage:owner:identityKeyStore:)` : mints Ed25519 keypair on first open (public key → manifest, private key → identityKeyStore); backfills containerFP via rebuildAll +- :314 `create(storage:owner:manifest:)` : no keypair minted (deferred to first open) +- :401 `close()` : no-op; caller's storage owns teardown +- :412/:429/:448 `allDrawers(...)` overloads : full/limit/hydrationLevel variants +- :462 `tombstonedLineageIDs() -> Set` : cluster-C block-set for vault re-import guard +- :474 `roomLevelFingerprints()` : passthrough to containerFP.roomLevelEntries +- :485/:494 `getDrawers(ids:...)` : batch by-id, hydration-level aware +- :527 `getDrawers(ids:matchingFrame:hydrationLevel:) -> FrameFilteredDrawers` : frame-faithful O(candidates) by-id load; forces .full when chain has content predicate +- :566 `hydrateBodies(ids:)` : late body hydration for dense-first pool pattern - :578 `tunnelsFromWing(_:)`, :587 `recentRecallTraces`, :594/:604 `allTunnels`/`allActiveTunnels`, :612/:621 `retireTunnel`/`unretireTunnel`, :635 `pruneRecallTraces`, :651 `markRecallTracesUsed`, :657/:665 `countRecallTraces`/`countDrawerRows` - :674/:681/:689/:698/:706/:713 `allProposals`/`allAssociations`/`allLearnedReferences`/`allKGFacts`/`allKGFactsIncludingRetired`/`allDiaryEntries` - :722 `resolveNodeNames(parentNodeIds:)` -- :737 `manifest: ManifestValues { get async throws }` — re-reads on every access -- :745 `estateUUID: UUID` — stable, parsed once at open +- :737 `manifest: ManifestValues { get async throws }` : re-reads on every access +- :745 `estateUUID: UUID` : stable, parsed once at open - :759 `retrievePrivateSigningKeyData() -> Data?` -- :780/:789 `meta(key:)`/`setMeta(key:value:)` — public per-estate KV surface (caller must namespace keys) -- :70 `RecallInternalRead` enum + :97 `_testForceInternalReadError` — TEST-ONLY single-use fault-injection seam for recall's internal reads - -### Estate audit — EstateAudit.swift -- :53 `auditTrail(rowID:) -> [AuditEvent]` — full sealed history, HLC order; empty until first mutation (capture = INSERT, not mutation) -- :83 `bitmapState(rowID:asOf:) -> BitmapState` — folds audit log via AuditLogFold.projectStateAt; throws drawerNotFound if no events ≤ asOf - -### Identity keys — EstateIdentityKeyStore.swift -- :23 `protocol EstateIdentityKeyStore: Sendable` — loadPrivateKey/storePrivateKey(forEstateID:) -- :48 `struct KeychainEstateIdentityKeyStore` — kSecClassGenericPassword, service "com.mootx01.estate.identity", account=estateUUID, kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly; production -- :130 `final class InMemoryEstateIdentityKeyStore: @unchecked Sendable` — NSLock-guarded dict; test-only; :147 `_storedPrivateKey(forEstateID:)` test peek - -### Verbs — EstateVerbs.swift (extension Estate) -- :69 `capture(_ frame: CaptureFrame) -> Drawer` — validates 5 required fields; assembles 3 bitmaps via BitField.writeField; resolves wing/room to node ids; routes through :432 `addDrawerCovered` (the ONLY sanctioned add chokepoint — bundles store.addDrawer + containerFP.orIn) -- :239 `captureBatch(_ frames: [CaptureFrame]) -> [Drawer]` — bulk path; ONE transaction for all fresh inserts via `store.insertFreshBatch`; Merkle rollup deliberately deferred (NT_R1) -- :487 `capture(_ frame: TunnelCaptureFrame) -> Tunnel` — standalone tunnel capture (2nd of the two nouns capture accepts) -- :573 `recallCandidateCap = 256` — scan bound floor for director-style callers -- :619 `recall(_ frame: RecallFrame) -> RecallStream` — non-throwing; internal-read failures surfaced via `RecallStream.degradedStages`, never silently emptied -- :743 `enum RecallStage` — stable stage-id strings (locus.liveRows.readFailed, locus.roomFingerprints.readFailed, locus.roomDrawerRead.readFailed, locus.bitmapEval.failed, recall.trace_write_failed) -- :800 `liveRows(for:forcedFault:)` (private) — fingerprint-pruning OR bounded-scan dispatch; no-blob fast path when chain has no content predicate -- :960 `withdraw(rowID:reason:)` — state → .withdrawn via mutateState(.retract) (NOT mutateAdjective — must go through automaton) -- :1030 `expunge(rowID:reason:confirmation:now:) -> AuditEvent?` — requires confirmation:true; walks full lineage chain; rolls up Merkle for every affected room -- :1085 `expungeReturningUnsealedEvent(...)` / :1141 `sealExpungeAudit(_:)` / :1155 `sealExpungeOrphanAudit(...)` — GLK's 3-step §B-2a orchestration (storage tombstone → cross-kit delete → seal); sealAudit:false is the ONLY split path -- :1178 `sealExpungeOrphanAuditSynthetic(rowID:now:)` / :1196 `tombstonedRowsWithoutExpungeAudit()` — crash-window recovery sweep support -- :1228 `mutate(rowID:kind:payload:)` — dispatches MutationKind → confirm(provenance)/reject|contest|resolve|accept|supersede(state)/revive(state, full per-source-state legality table incl. living-successor lineage guard)/correctSensitivity|correctTrust|correctExportability(adjective field) -- :1528 `reanchor(rowID:toRoom:toWing:toLattice:)` — placement move, requires ≥1 non-nil arg; toWing rejects empty/whitespace -- :1578 `reanchorAnchor(rowID:toLattice:changedBy:now:)` — deterministic-timestamp variant for Q-ID-completion acceptance path -- :1615 `propose(_ frame: ProposeFrame, now:) -> Proposal` — derives candidateState+latticeAnchor from target drawer -- :1690 `associate(_ frame: AssociateFrame, now:) -> Association` — derives latticeAnchor from endpoint A -- :1774 `learn(_ frame: LearnFrame, now:) -> LearnedReference` — catalogs source once by handle; NEVER fabricates a sentinel anchor -- :1936 `defaultWing() -> String` (private) — returns "Agentic Memory" (ADR-016, fixed, not owner-derived) -- :1971 `seedWing(_:hint:addedBy:embeddingModelID:now:)` — files ordinary recallable hint drawer per default wing at provision - -### Frames — Frames.swift -- :24 `struct CaptureFrame` — every named capture-time axis; nearly all defaults reproduce pre-slot all-zero bitmap byte-identically -- :178 `struct TunnelCaptureFrame` — no content/anchor/embedding slots (edges carry none) -- :237 `struct RecallFrame` — filterChain/hydrationLevel/limit/ordering/asOf/traceLimit (traceLimit nil = ZERO trace writes, the default) -- :287 `enum MutationKind` — .confirm/.reject/.contest/.resolve/.supersede/.revive/.accept/.correctSensitivity(_)/.correctTrust(_)/.correctExportability(_) -- :338 `struct LearnFrame` — source(SourceCatalogEntry)/handle/mode/refreshPolicy -- :382 `struct ProposeFrame` — target/kind/justification/confirmation/generatedBy/confidence (LocusKit substrate-axis kind, NOT GLK's Brain-layer kind) -- :436 `struct AssociateFrame` — a/b/weight -- :455 `enum HydrationLevel` — .structured/.full/.bitmapOnly -- :476 `enum Ordering` — .byCaptureTimeDesc/.byCaptureTimeAsc/.byRoomAsc (NO .byRelevanceDesc — LocusKit has no scoring signal; that's VectorKit/GLK's job) - -### Storage engine — DrawerStore.swift (actor, largest file, ~3850 lines) -- :57 `actor DrawerStore` — owns every table; :90 `init(storage:hlc:)` opens schema idempotently, freezes vocabulary, classifies estate_uuid (absent/present/CORRUPT-throws), derives HLC maker node id by FNV-1a hashing raw stored uuid text -- :291 `addDrawer(_:now:)` (internal — NOT the public add path, use Estate.addDrawerCovered) — dispatches supersession cascade vs plain gatedCapture -- :346 `addDrawerWithCascade(_:priorID:)` (private) — successor+tunnel in ONE transaction (no orphan on tunnel-insert failure), THEN separate gated mutateState(.superseded) on predecessor -- :405 `findActivePredecessor(lineageID:excludingID:)` — indexed g_state_cluster < 3 query -- :439 `livingSuccessorInLineage(lineageID:excludingID:)` — g_state_cluster < RowState.activeClusterUpperBoundRaw(16); wider predicate than findActivePredecessor, used by revive guard -- :468 `lineageChain(for:) -> [String]` — every row sharing lineageID, any state -- :492/:516/:545 `getDrawer(id:)`/`getDrawers(ids:)`/`getDrawers(ids:hydrationLevel:)` — chunked at 900 ids (SQLite 999-var ceiling); :530 `structuredDrawerColumns` = every column except content +- :780/:789 `meta(key:)`/`setMeta(key:value:)` : public per-estate KV surface (caller must namespace keys) +- :70 `RecallInternalRead` enum + :97 `_testForceInternalReadError` : TEST-ONLY single-use fault-injection seam for recall's internal reads + +### Estate audit : EstateAudit.swift +- :53 `auditTrail(rowID:) -> [AuditEvent]` : full sealed history, HLC order; empty until first mutation (capture = INSERT, not mutation) +- :83 `bitmapState(rowID:asOf:) -> BitmapState` : folds audit log via AuditLogFold.projectStateAt; throws drawerNotFound if no events ≤ asOf + +### Identity keys : EstateIdentityKeyStore.swift +- :23 `protocol EstateIdentityKeyStore: Sendable` : loadPrivateKey/storePrivateKey(forEstateID:) +- :48 `struct KeychainEstateIdentityKeyStore` : kSecClassGenericPassword, service "com.mootx01.estate.identity", account=estateUUID, kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly; production +- :130 `final class InMemoryEstateIdentityKeyStore: @unchecked Sendable` : NSLock-guarded dict; test-only; :147 `_storedPrivateKey(forEstateID:)` test peek + +### Verbs : EstateVerbs.swift (extension Estate) +- :69 `capture(_ frame: CaptureFrame) -> Drawer` : validates 5 required fields; assembles 3 bitmaps via BitField.writeField; resolves wing/room to node ids; routes through :432 `addDrawerCovered` (the ONLY sanctioned add chokepoint : bundles store.addDrawer + containerFP.orIn) +- :239 `captureBatch(_ frames: [CaptureFrame]) -> [Drawer]` : bulk path; ONE transaction for all fresh inserts via `store.insertFreshBatch`; Merkle rollup deliberately deferred (NT_R1) +- :487 `capture(_ frame: TunnelCaptureFrame) -> Tunnel` : standalone tunnel capture (2nd of the two nouns capture accepts) +- :573 `recallCandidateCap = 256` : scan bound floor for director-style callers +- :619 `recall(_ frame: RecallFrame) -> RecallStream` : non-throwing; internal-read failures surfaced via `RecallStream.degradedStages`, never silently emptied +- :743 `enum RecallStage` : stable stage-id strings (locus.liveRows.readFailed, locus.roomFingerprints.readFailed, locus.roomDrawerRead.readFailed, locus.bitmapEval.failed, recall.trace_write_failed) +- :800 `liveRows(for:forcedFault:)` (private) : fingerprint-pruning OR bounded-scan dispatch; no-blob fast path when chain has no content predicate +- :960 `withdraw(rowID:reason:)` : state → .withdrawn via mutateState(.retract) (NOT mutateAdjective : must go through automaton) +- :1030 `expunge(rowID:reason:confirmation:now:) -> AuditEvent?` : requires confirmation:true; walks full lineage chain; rolls up Merkle for every affected room +- :1085 `expungeReturningUnsealedEvent(...)` / :1141 `sealExpungeAudit(_:)` / :1155 `sealExpungeOrphanAudit(...)` : GLK's 3-step §B-2a orchestration (storage tombstone → cross-kit delete → seal); sealAudit:false is the ONLY split path +- :1178 `sealExpungeOrphanAuditSynthetic(rowID:now:)` / :1196 `tombstonedRowsWithoutExpungeAudit()` : crash-window recovery sweep support +- :1228 `mutate(rowID:kind:payload:)` : dispatches MutationKind → confirm(provenance)/reject|contest|resolve|accept|supersede(state)/revive(state, full per-source-state legality table incl. living-successor lineage guard)/correctSensitivity|correctTrust|correctExportability(adjective field) +- :1528 `reanchor(rowID:toRoom:toWing:toLattice:)` : placement move, requires ≥1 non-nil arg; toWing rejects empty/whitespace +- :1578 `reanchorAnchor(rowID:toLattice:changedBy:now:)` : deterministic-timestamp variant for Q-ID-completion acceptance path +- :1615 `propose(_ frame: ProposeFrame, now:) -> Proposal` : derives candidateState+latticeAnchor from target drawer +- :1690 `associate(_ frame: AssociateFrame, now:) -> Association` : derives latticeAnchor from endpoint A +- :1774 `learn(_ frame: LearnFrame, now:) -> LearnedReference` : catalogs source once by handle; NEVER fabricates a sentinel anchor +- :1936 `defaultWing() -> String` (private) : returns "Agentic Memory" (ADR-016, fixed, not owner-derived) +- :1971 `seedWing(_:hint:addedBy:embeddingModelID:now:)` : files ordinary recallable hint drawer per default wing at provision + +### Frames : Frames.swift +- :24 `struct CaptureFrame` : every named capture-time axis; nearly all defaults reproduce pre-slot all-zero bitmap byte-identically +- :178 `struct TunnelCaptureFrame` : no content/anchor/embedding slots (edges carry none) +- :237 `struct RecallFrame` : filterChain/hydrationLevel/limit/ordering/asOf/traceLimit (traceLimit nil = ZERO trace writes, the default) +- :287 `enum MutationKind` : .confirm/.reject/.contest/.resolve/.supersede/.revive/.accept/.correctSensitivity(_)/.correctTrust(_)/.correctExportability(_) +- :338 `struct LearnFrame` : source(SourceCatalogEntry)/handle/mode/refreshPolicy +- :382 `struct ProposeFrame` : target/kind/justification/confirmation/generatedBy/confidence (LocusKit substrate-axis kind, NOT GLK's Brain-layer kind) +- :436 `struct AssociateFrame` : a/b/weight +- :455 `enum HydrationLevel` : .structured/.full/.bitmapOnly +- :476 `enum Ordering` : .byCaptureTimeDesc/.byCaptureTimeAsc/.byRoomAsc (NO .byRelevanceDesc : LocusKit has no scoring signal; that's VectorKit/GLK's job) + +### Storage engine : DrawerStore.swift (actor, largest file, ~3850 lines) +- :57 `actor DrawerStore` : owns every table; :90 `init(storage:hlc:)` opens schema idempotently, freezes vocabulary, classifies estate_uuid (absent/present/CORRUPT-throws), derives HLC maker node id by FNV-1a hashing raw stored uuid text +- :291 `addDrawer(_:now:)` (internal : NOT the public add path, use Estate.addDrawerCovered) : dispatches supersession cascade vs plain gatedCapture +- :346 `addDrawerWithCascade(_:priorID:)` (private) : successor+tunnel in ONE transaction (no orphan on tunnel-insert failure), THEN separate gated mutateState(.superseded) on predecessor +- :405 `findActivePredecessor(lineageID:excludingID:)` : indexed g_state_cluster < 3 query +- :439 `livingSuccessorInLineage(lineageID:excludingID:)` : g_state_cluster < RowState.activeClusterUpperBoundRaw(16); wider predicate than findActivePredecessor, used by revive guard +- :468 `lineageChain(for:) -> [String]` : every row sharing lineageID, any state +- :492/:516/:545 `getDrawer(id:)`/`getDrawers(ids:)`/`getDrawers(ids:hydrationLevel:)` : chunked at 900 ids (SQLite 999-var ceiling); :530 `structuredDrawerColumns` = every column except content - :591/:627/:657 `drawersIn(wing:)`/`drawersIn(wing:room:)`/`drawersBySource(file:)` -- :678/:718 `allDrawers()`/`allDrawers(hydrationLevel:limit:direction:)` — (filedAt,id) compound sort key, portable across SQLite/Postgres/InMemory (NOT rowid); .descending retains newest under a scan cap -- :774/:795/:814 `mutateProvenance`/`mutateAdjective`/`mutateOperational` — all route through :1708 `gatedColumnWrite` (private) which excludes the state field from field-edit writes -- :858 `gatedCaptureBody(_:now:)` (private) — builds @Sendable closure for capture INSERT + AuditGate.admit(verb:.capture, prior:nil); reused by both gatedCapture and addDrawerWithCascade (no nested-transaction support in PersistenceKit v1.0) -- :912 `gatedCapture(_:now:)` (private) — single-drawer non-cascade path -- :932 `insertFreshBatch(_:now:)` (internal) — ALL inserts + audit events in ONE transaction; caller (captureBatch) pre-verifies no active predecessor -- :979 `mutateState(drawerId:to:via:changedBy:reason:now:)` — RMWs prior bitmaps, AuditGate.admit(verb:), writes merged snapshot + audit event atomically -- :1097 `expungeGated(drawerId:...:sealAudit:commitmentKey:commitmentKeyVersion:) -> AuditEvent?` — lineage-wide: gates target via .tombstone, unconditionally zeroes content on EVERY sibling regardless of gate success/failure (destruction contract > state machine), records erasure ledger entries -- :1343/:1378/:1516 `sealExpungeAudit`/`sealExpungeOrphanAudit`/`sealExpungeOrphanForSweep` — deferred-seal + crash-recovery paths -- :1425 `tombstonedRowsWithoutExpungeAudit()` — crash-window orphan enumeration (no "tombstone"/"expungeOrphan" audit event) +- :678/:718 `allDrawers()`/`allDrawers(hydrationLevel:limit:direction:)` : (filedAt,id) compound sort key, portable across SQLite/Postgres/InMemory (NOT rowid); .descending retains newest under a scan cap +- :774/:795/:814 `mutateProvenance`/`mutateAdjective`/`mutateOperational` : all route through :1708 `gatedColumnWrite` (private) which excludes the state field from field-edit writes +- :858 `gatedCaptureBody(_:now:)` (private) : builds @Sendable closure for capture INSERT + AuditGate.admit(verb:.capture, prior:nil); reused by both gatedCapture and addDrawerWithCascade (no nested-transaction support in PersistenceKit v1.0) +- :912 `gatedCapture(_:now:)` (private) : single-drawer non-cascade path +- :932 `insertFreshBatch(_:now:)` (internal) : ALL inserts + audit events in ONE transaction; caller (captureBatch) pre-verifies no active predecessor +- :979 `mutateState(drawerId:to:via:changedBy:reason:now:)` : RMWs prior bitmaps, AuditGate.admit(verb:), writes merged snapshot + audit event atomically +- :1097 `expungeGated(drawerId:...:sealAudit:commitmentKey:commitmentKeyVersion:) -> AuditEvent?` : lineage-wide: gates target via .tombstone, unconditionally zeroes content on EVERY sibling regardless of gate success/failure (destruction contract > state machine), records erasure ledger entries +- :1343/:1378/:1516 `sealExpungeAudit`/`sealExpungeOrphanAudit`/`sealExpungeOrphanForSweep` : deferred-seal + crash-recovery paths +- :1425 `tombstonedRowsWithoutExpungeAudit()` : crash-window orphan enumeration (no "tombstone"/"expungeOrphan" audit event) - :1469 `tombstonedLineageIDs() -> Set` -- :1576 `reanchorGated(...)` — verb:.mutate (active→active self-loop); anchor delta via priorLatticeAnchor≠afterLatticeAnchor; writes:[] (placement move, no field write) -- :1689 `requireUuid(_:label:) throws -> UUID` (static) — fail-loud on non-UUID row id -- :1783 `declaredSlots(for:)` (static private) — adjective ⇒ Vocabulary.basis (SubstrateLib); operational/provenance ⇒ LocusKitVocabulary.unionSlots +- :1576 `reanchorGated(...)` : verb:.mutate (active→active self-loop); anchor delta via priorLatticeAnchor≠afterLatticeAnchor; writes:[] (placement move, no field write) +- :1689 `requireUuid(_:label:) throws -> UUID` (static) : fail-loud on non-UUID row id +- :1783 `declaredSlots(for:)` (static private) : adjective ⇒ Vocabulary.basis (SubstrateLib); operational/provenance ⇒ LocusKitVocabulary.unionSlots - :1800/:1805 `auditEventsForRow`/`auditEventCountForRow` -- :1832 `addTunnel(_:)` — enforces "one parent per child" for .parent kind (kit-level constraint, not DB unique index) -- :1954/:1963/:1982/:2001 `allActiveTunnels()`/(T13/ADR-021 Phase 7 retirement)/`retireTunnel`/`unretireTunnel` — bit-13 operationalBitmap flag, filtered in-memory (PersistenceKit predicate DSL has no bitmask ops) -- :2019/:2044/:2076 `outlineChildren(of:)`/`outlineAncestors(of:)` (depth cap 256, cycle-safe)/`reparentDrawer(...)` — ADR-017 §11 outline graph over .parent tunnels + order_key -- :2139/:2166 `addKGFact(_:)`/`withdrawKGFact(id:)` — sourceDrawerID may be "" (unanchored-fact sentinel) -- :2228 `addProposal(_:)` — rejects empty latticeAnchor.udcCode -- :2264 `addAssociation(_:)` — rejects empty latticeAnchor.udcCode -- :2328 `addLearnedReference(_:)` / :2372 `addSourceCatalogEntry(_:)` — both reject empty latticeAnchor.udcCode +- :1832 `addTunnel(_:)` : enforces "one parent per child" for .parent kind (kit-level constraint, not DB unique index) +- :1954/:1963/:1982/:2001 `allActiveTunnels()`/(T13/ADR-021 Phase 7 retirement)/`retireTunnel`/`unretireTunnel` : bit-13 operationalBitmap flag, filtered in-memory (PersistenceKit predicate DSL has no bitmask ops) +- :2019/:2044/:2076 `outlineChildren(of:)`/`outlineAncestors(of:)` (depth cap 256, cycle-safe)/`reparentDrawer(...)` : ADR-017 §11 outline graph over .parent tunnels + order_key +- :2139/:2166 `addKGFact(_:)`/`withdrawKGFact(id:)` : sourceDrawerID may be "" (unanchored-fact sentinel) +- :2228 `addProposal(_:)` : rejects empty latticeAnchor.udcCode +- :2264 `addAssociation(_:)` : rejects empty latticeAnchor.udcCode +- :2328 `addLearnedReference(_:)` / :2372 `addSourceCatalogEntry(_:)` : both reject empty latticeAnchor.udcCode - :2408/:2428/:2442 `addDiaryEntry`/`readDiary(agentName:lastN:)`/`readDiary(agentName:in:lastN:)` -- :2592/:2614 `insertRecallTrace`/`insertRecallTraces` (batch, one transaction) — :710 in EstateVerbs is the only production write path (opt-in via frame.traceLimit) -- :2713/:2758 `markRecallTraceUsed(id:now:)`/`markRecallTracesUsed(target:since:now:)` — bulk ARIA reward path -- :2692 `pruneRecallTraces(olderThan:)` / :2807 `countRecallTraces()` / :2827 `countDrawerRows()` (raw COUNT(*), bypasses decode — counts corrupt rows too, used as corruption sentinel) -- :2836/:2881/:2932 `listWings()`/`listRooms(in:)`/`taxonomy()` — computed projections, no dedicated wings/rooms table +- :2592/:2614 `insertRecallTrace`/`insertRecallTraces` (batch, one transaction) : :710 in EstateVerbs is the only production write path (opt-in via frame.traceLimit) +- :2713/:2758 `markRecallTraceUsed(id:now:)`/`markRecallTracesUsed(target:since:now:)` : bulk ARIA reward path +- :2692 `pruneRecallTraces(olderThan:)` / :2807 `countRecallTraces()` / :2827 `countDrawerRows()` (raw COUNT(*), bypasses decode : counts corrupt rows too, used as corruption sentinel) +- :2836/:2881/:2932 `listWings()`/`listRooms(in:)`/`taxonomy()` : computed projections, no dedicated wings/rooms table - :2939/:2948/:2959 `setMeta`/`getMeta`/`readManifest() -> ManifestValues` - :3095 `resolveNodeNames(parentNodeIds:) -> [String: (wing:room:)]` -- :3143/:3150 `decodeDrawerRows` (throws on any corrupt value — point-lookup path) / `decodeDrawerRowsResilient`→:3419 `decodeDrawerRowsSkipCorrupt` (logs+skips corrupt rows — corpus-scan path; only .corruptStoredValue is caught, everything else rethrows) -- :3682/:3731 `fingerprintsCaptured(in:)`/`fingerprintBitSeries(bit:bucketSeconds:bucketCount:endingAt:)` — feed FFT rhythm-spectrum / moment-summary lenses; effective capture time = eventTime ?? filedAt (ING-01) +- :3143/:3150 `decodeDrawerRows` (throws on any corrupt value : point-lookup path) / `decodeDrawerRowsResilient`→:3419 `decodeDrawerRowsSkipCorrupt` (logs+skips corrupt rows : corpus-scan path; only .corruptStoredValue is caught, everything else rethrows) +- :3682/:3731 `fingerprintsCaptured(in:)`/`fingerprintBitSeries(bit:bucketSeconds:bucketCount:endingAt:)` : feed FFT rhythm-spectrum / moment-summary lenses; effective capture time = eventTime ?? filedAt (ING-01) -### Drawer value type — Drawer.swift -- :31 `struct Drawer: Equatable, Hashable, Sendable, Codable` — verbatim content, never truncated/normalized +### Drawer value type : Drawer.swift +- :31 `struct Drawer: Equatable, Hashable, Sendable, Codable` : verbatim content, never truncated/normalized - :37 `id` / :46 `lineageID: UUID` (fresh per-row default) / :49 `content` / :54 `parentNodeId: String` (FK → nodes.id, room-depth node) - :75 `filedAt: Date` (ingest clock) vs :83 `eventTime: Date` (world-happened clock; ING-01 two-clock ingest; defaults to filedAt) -- :109/:120/:132 `provenance`/`adjectiveBitmap`/`operationalBitmap: Int64` — all default 0 +- :109/:120/:132 `provenance`/`adjectiveBitmap`/`operationalBitmap: Int64` : all default 0 - :151/:159/:166/:175 `udcCode`(required non-empty, I-5)/`udcFacets`/`wikidataQID`/`wikidataQidsSecondary` - Custom Codable (:228) backfills missing eventTime → filedAt on decode -### Drawer operational — DrawerOperational.swift -- :61 `enum CaptureChannel: Int` — typed=0/voiced=1/ocr=2/importedFile=3/sensor=4/actuator=5 (bits 0–5) -- :78 `enum ContentKind: Int` — prose=0…fingerprintOnly=6 (bits 6–11) -- :100 `struct DrawerFeatureFlags: OptionSet` — bits 12–23; hasAttachments/hasVoice/hasImage/hasLinks/isPinned/isKeystone(§7.2)/isLockedZone -- :141/:149/:159/:170/:179/:188 `Drawer.captureChannel`/`.contentKind`/`.featureFlags`/`.hasFeatureFlag(_:)`/`.stateExtensionActive`(bit24)/`.lineageClusteringActive`(bit25) — all fail-safe to neutral default on unrecognised raw +### Drawer operational : DrawerOperational.swift +- :61 `enum CaptureChannel: Int` : typed=0/voiced=1/ocr=2/importedFile=3/sensor=4/actuator=5 (bits 0–5) +- :78 `enum ContentKind: Int` : prose=0…fingerprintOnly=6 (bits 6–11) +- :100 `struct DrawerFeatureFlags: OptionSet` : bits 12–23; hasAttachments/hasVoice/hasImage/hasLinks/isPinned/isKeystone(§7.2)/isLockedZone +- :141/:149/:159/:170/:179/:188 `Drawer.captureChannel`/`.contentKind`/`.featureFlags`/`.hasFeatureFlag(_:)`/`.stateExtensionActive`(bit24)/`.lineageClusteringActive`(bit25) : all fail-safe to neutral default on unrecognised raw -### State transition bridge — DrawerStateValidator.swift -- :53 `validate(from:to:via:)` — transition-legality only, bridges State↔SubstrateLib.RowState, delegates to RowStateAutomaton.transition -- :107 `validate(from:to:via:targetingFields:)` — legality + field-invariant (S-1 trust≥canonical, S-2) combined; the overload DrawerStore.mutateState actually calls +### State transition bridge : DrawerStateValidator.swift +- :53 `validate(from:to:via:)` : transition-legality only, bridges State↔SubstrateLib.RowState, delegates to RowStateAutomaton.transition +- :107 `validate(from:to:via:targetingFields:)` : legality + field-invariant (S-1 trust≥canonical, S-2) combined; the overload DrawerStore.mutateState actually calls -### Drawer fingerprint — DrawerFingerprint.swift -- :75 `struct EstateFingerprintFamilies` — 4 hyperplane families derived once per estate from estate UUID via HyperplaneFamily.blockFamilies; :90 `baseSeed(estateUUID:)` -- :106 `fingerprint(of: Drawer) -> Fingerprint256` — 4 SimHash blocks: bitmap-LSH(adj/op/prov) / lattice(udcPrefixHash+qidDirectHash+qidClosureHash via LatticeLib.QIDClosure) / lineage+temporal(lineageHash+captureWeekBucket, ING-01 keys off eventTime) / channel+source(channel/sourceType/captureChannel/sensitivity/estateUUIDByte) +### Drawer fingerprint : DrawerFingerprint.swift +- :75 `struct EstateFingerprintFamilies` : 4 hyperplane families derived once per estate from estate UUID via HyperplaneFamily.blockFamilies; :90 `baseSeed(estateUUID:)` +- :106 `fingerprint(of: Drawer) -> Fingerprint256` : 4 SimHash blocks: bitmap-LSH(adj/op/prov) / lattice(udcPrefixHash+qidDirectHash+qidClosureHash via LatticeLib.QIDClosure) / lineage+temporal(lineageHash+captureWeekBucket, ING-01 keys off eventTime) / channel+source(channel/sourceType/captureChannel/sensitivity/estateUUIDByte) - :161/:171 `captureWeekBucket(_:)` (mod 256, 2020-01-01 epoch) / `udcPrefixHash(_:)` (FNV-16 of first 4 digits) -- I-17: cross-noun null fields (defer pattern, completion bucket, behavioral recency, stream-source bitset) always zero on a Drawer — those belong to AmbientSample only +- I-17: cross-noun null fields (defer pattern, completion bucket, behavioral recency, stream-source bitset) always zero on a Drawer : those belong to AmbientSample only -### Adjective axes — Adjectives.swift -- :99 `enum State: Int` — 10 cases, 3 clusters at raw boundaries 16/32: A(active/pending/contested/accepted) B(superseded/decayed/withdrawn/expired) C(rejected/tombstoned); :121 `isClusterA` -- :143 `enum Trust: Int, Comparable` — verbatim=0…ambient=6(NEW v0.6) -- :173 `enum AdjectiveSensitivity: Int` — normal=0/elevated=16/restricted=32/secret=48 (scale-gapped); ADR-007 tier predicates :197 `isBulkExportable`/:217 `requiresOwnerKeyForBulk`/:237 `isExcludedFromBulk` -- :256 `enum AdjectiveExportability: Int` — private_=0/public_=32 +### Adjective axes : Adjectives.swift +- :99 `enum State: Int` : 10 cases, 3 clusters at raw boundaries 16/32: A(active/pending/contested/accepted) B(superseded/decayed/withdrawn/expired) C(rejected/tombstoned); :121 `isClusterA` +- :143 `enum Trust: Int, Comparable` : verbatim=0…ambient=6(NEW v0.6) +- :173 `enum AdjectiveSensitivity: Int` : normal=0/elevated=16/restricted=32/secret=48 (scale-gapped); ADR-007 tier predicates :197 `isBulkExportable`/:217 `requiresOwnerKeyForBulk`/:237 `isExcludedFromBulk` +- :256 `enum AdjectiveExportability: Int` : private_=0/public_=32 - Drawer accessors :270-:301 `state`/`adjectiveSensitivity`/`exportability`/`trust` (bits 0-5/6-11/12-17/18-23); :311/:323/:338 `isCurrentlyBelieved`/`isKnewPast`/`isTerminal` cluster predicates -- :354 `dreamingRecalcRequired` (bit 26, obligation flag, F17) / :370 `sealed` (bit 27, custody trust hint — cached, seal itself is authoritative if they disagree) -- `@guardian-pair` annotations throughout — SwiftSyntax Guardian CI tool cross-checks these raw sets against SubstrateLib's AuditGate.basis; GuardianPairParityTests backstop +- :354 `dreamingRecalcRequired` (bit 26, obligation flag, F17) / :370 `sealed` (bit 27, custody trust hint : cached, seal itself is authoritative if they disagree) +- `@guardian-pair` annotations throughout : SwiftSyntax Guardian CI tool cross-checks these raw sets against SubstrateLib's AuditGate.basis; GuardianPairParityTests backstop -### Provenance axes — Provenance.swift -- :67 `enum SourceType: Int` — user=0…actuator=9 (bits 0-5) -- :102 `enum Channel: Int` — uiTyped=0…dreamMiningResult=8, deviceSensor=15, actuatorOutcome=16 (bits 6-11, gaps reserved) +### Provenance axes : Provenance.swift +- :67 `enum SourceType: Int` : user=0…actuator=9 (bits 0-5) +- :102 `enum Channel: Int` : uiTyped=0…dreamMiningResult=8, deviceSensor=15, actuatorOutcome=16 (bits 6-11, gaps reserved) - CaptureChannel mirrored at bits 12-17 (provenanceCaptureChannel accessor, :224) -- :137 `enum Confirmation: Int` — unconfirmed=0/userConfirmed=1/automatedConfirmed=2/peerConfirmed=3/actuatorConfirmed=4 (bits 18-23); :261 `isUserConfirmed` -- :155 `enum Confidence: Int, Comparable` — null=0/low=16/medium=32/high=48/verified=56 (bits 24-29) -- :177 `enum Sensitivity: Int` — normal/elevated/restricted/secret mirrors AdjectiveSensitivity raws (bits 30-35) -- :188 `enum EnrichmentStatus: Int` — none/qidPending/qidCompleted/closureCached/qidProposed(terminal, NOT re-picked by maintenance scan) (bits 36-41) +- :137 `enum Confirmation: Int` : unconfirmed=0/userConfirmed=1/automatedConfirmed=2/peerConfirmed=3/actuatorConfirmed=4 (bits 18-23); :261 `isUserConfirmed` +- :155 `enum Confidence: Int, Comparable` : null=0/low=16/medium=32/high=48/verified=56 (bits 24-29) +- :177 `enum Sensitivity: Int` : normal/elevated/restricted/secret mirrors AdjectiveSensitivity raws (bits 30-35) +- :188 `enum EnrichmentStatus: Int` : none/qidPending/qidCompleted/closureCached/qidProposed(terminal, NOT re-picked by maintenance scan) (bits 36-41) -### Bitmap primitives — BitmapOps.swift +### Bitmap primitives : BitmapOps.swift - :55 `andMask(_:mask:expected:)` → BitField.maskedEquals - :94 `thresholdCompare(_:mask:shift:op:value:)` + :69 `enum ThresholdOp` (.lessThan/.lessThanOrEqual/.greaterThanOrEqual) → BitField.extractField+compare - :124 `shiftExtract(_:shift:mask:)` → BitField.extractField -- NOT math primitives — no conformance gate; real math primitives live in SubstrateLib (XOR/popcount/Hamming/OR-reduce/SimHash/SHA-256) +- NOT math primitives : no conformance gate; real math primitives live in SubstrateLib (XOR/popcount/Hamming/OR-reduce/SimHash/SHA-256) -### Forbidden combination — ForbiddenCombinationValidator.swift -- :75 `validate(_:) throws` — I-3: sensitivity=secret(raw48,bits6-11) AND exportability=public(raw32,bits12-17) forbidden; hand-derived raw constants (NOT imported enums) so a rename can't silently break the check -- No live call site today (current writes route the equivalent check through AuditGate/SubstrateLib directly) — ready to wire in wherever a write path bypasses the gate +### Forbidden combination : ForbiddenCombinationValidator.swift +- :75 `validate(_:) throws` : I-3: sensitivity=secret(raw48,bits6-11) AND exportability=public(raw32,bits12-17) forbidden; hand-derived raw constants (NOT imported enums) so a rename can't silently break the check +- No live call site today (current writes route the equivalent check through AuditGate/SubstrateLib directly) : ready to wire in wherever a write path bypasses the gate -### Recall filter algebra — Filter.swift +### Recall filter algebra : Filter.swift - :8/:12/:15/:20/:25/:30 typealiases `LineageID`(UUID)/`RoomID`(String)/`WingID`(String)/`WikidataQID`(String)/`ProvenanceChannel`(=Channel)/`FeatureFlag`(=DrawerFeatureFlags) -- :48 `enum StateCluster` — knowNow/knewPast/terminal -- :69 `indirect enum Filter` — NO raw bit position ever exposed; state/trust/sensitivity/exportability/provenance/operational/structural/content cases + :166/:168/:170 `.all`/`.any`/`.not` composition +- :48 `enum StateCluster` : knowNow/knewPast/terminal +- :69 `indirect enum Filter` : NO raw bit position ever exposed; state/trust/sensitivity/exportability/provenance/operational/structural/content cases + :166/:168/:170 `.all`/`.any`/`.not` composition - RecallFrame.filterChain ([Filter]) ≡ implicit Filter.all(chain) -### Recall evaluator — BitmapEvaluator.swift -- :60 `struct BitmapEvaluator` (internal — DrawerStore param is internal) — 4-stage pipeline +### Recall evaluator : BitmapEvaluator.swift +- :60 `struct BitmapEvaluator` (internal : DrawerStore param is internal) : 4-stage pipeline - :62-125 private layout-constant mirror of Adjectives/Provenance/DrawerOperational bit positions (schema-bump-coupled) -- :147 `evaluate(frame:drawers:store:nodeNames:) -> [Drawer]` — bitmap tier (w/ historical reconstruction via AuditLogFold.projectStateAt when frame.asOf set) → structured tier → content tier → sort -- :221 `insertDefaults(_:)` (private) — prepends .currentlyBelieve/.trustworthy/.sensitivityAtMost(.elevated) when caller's chain doesn't already constrain that axis; NO confirmation default +- :147 `evaluate(frame:drawers:store:nodeNames:) -> [Drawer]` : bitmap tier (w/ historical reconstruction via AuditLogFold.projectStateAt when frame.asOf set) → structured tier → content tier → sort +- :221 `insertDefaults(_:)` (private) : prepends .currentlyBelieve/.trustworthy/.sensitivityAtMost(.elevated) when caller's chain doesn't already constrain that axis; NO confirmation default - :290/:304/:314 `chainHasPrunableFilter`/`chainHasContentPredicate`/`chainHasStructuredNameFilter` (static, called by Estate.liveRows/getDrawers(matchingFrame:) before fetch) -- :349 `containerSurvives(chain:fingerprint:) -> Bool` — sound pruning test; only set-bit filters (.hasFeatureFlag) can prove exclusion from an OR fingerprint; :358 `containerProvablyExcludes` -- :384 `evaluateBitmapTier` — tombstone (state==33) excluded UNCONDITIONALLY, independent of caller's chain -- :641 `reconstructAt(rowID:asOf:store:)` (private) — historical bitmap fold +- :349 `containerSurvives(chain:fingerprint:) -> Bool` : sound pruning test; only set-bit filters (.hasFeatureFlag) can prove exclusion from an OR fingerprint; :358 `containerProvablyExcludes` +- :384 `evaluateBitmapTier` : tombstone (state==33) excluded UNCONDITIONALLY, independent of caller's chain +- :641 `reconstructAt(rowID:asOf:store:)` (private) : historical bitmap fold - :654 `sort(_:ordering:nodeNames:)` (private) -### Tunnel — Tunnel.swift +### Tunnel : Tunnel.swift - :24 `struct Tunnel: Equatable, Hashable, Codable, Sendable` -- :29 `id` (conventionally SHA-256 of canonicalised endpoint pair — NOT enforced at this layer, LOCI-5) +- :29 `id` (conventionally SHA-256 of canonicalised endpoint pair : NOT enforced at this layer, LOCI-5) - source/target: wing+room+optional drawerId (nil = "the room itself") - :63 `kind: TunnelKind` (default .references) / :69/:75/:80 adjective/operational/provenance bitmaps / :99 `orderKey: Double?` (fractional-index sibling order, .parent kind only) -### Tunnel operational — TunnelOperational.swift -- :38 `enum TunnelKind: Int` — supersedes=0…parent=9 (outline containment, ADR-017 §11; source=child, target=parent, one-parent-per-child kit-level constraint) -- :60/:70/:80/:94 `TunnelDirection`/`TunnelLifecycle`/`TunnelOriginClass`/`TunnelStrength(Comparable, scale-gapped 0/2/4/6)` — bits 0-2/3-5/6-8/9-11 -- :168 `isRetiredBit = 1<<13` / :175 `isRetired` / :184/:202 `withRetired()`/`withUnretired()` — T13/ADR-021 Phase 7 reversible dreaming suspension (NOT tombstone) -- :241 `isDreamedBit = 1<<0` (provenanceBitmap) / :247 `isDreamed` / :256 `withDreamedProvenance()` — OMEGA retires ONLY isDreamed==true tunnels; declared tunnels never retired - -### Diary — DiaryEntry.swift -- :24 `struct DiaryEntry` — agentName/entry/topic/wing/room (all required, unvalidated strings) -- :85/:92 `reward: Double?`/`rewardProvenance: String?` — explicit quality-signal channel (NEURONKIT_SPEC §3.1 step 1a); nil ⇒ dreaming daemon falls back to implicit RecallTraceItem.used (step 1b) - -### Diary operational — DiaryOperational.swift -- :33 `enum DiaryEventClass: Int` — 12 cases (capture=0…auditTombstone=11), bits 0-3 -- :54 `enum DiarySeverity: Int, Comparable` — trace=0/info=2/warning=4/error=6, bits 4-6 -- :67 `enum DiaryActorClass: Int` — user=0…federationPeer=4, bits 7-9 -- :79 `enum DiaryBatchMembership: Int` — standalone=0…batchEnd=3, bits 10-12 -- :129 `requiresFollowup` — bit 13 flag - -### KG fact — KGFact.swift -- :51 `struct KGFact` — subject/predicate/object (free-form strings, no entity vocabulary yet) + :82 `sourceDrawerID` (may be "" = unanchored-fact sentinel) -- Adjective accessors :149/:159/:171/:180 `trust`/`state`/`adjectiveSensitivity`/`exportability` — IDENTICAL bit layout to Drawer's (shared retrieval predicates) - -### KG fact operational — KGFactOperational.swift -- :48 `enum KGExtractorClass: Int` — manual=0…federated=5, bits 0-3 -- :66 `enum KGAssertionKind: Int` — asserted=0/inferred=1/hypothesized=2/contradicted=3, bits 4-6 -- :83 `enum KGSpecificity: Int, Comparable` — general=0…instance=6 (scale-gapped), bits 7-9 -- :106 `enum KGConfidenceBand: Int, Comparable` — unknown=0…certain=6 (scale-gapped), bits 10-12 -- :171 `isCanonical` — bit 13 flag (estate-wide load-bearing promotion) - -### Proposal — Proposal.swift -- :72 `struct Proposal` — targetRowID(empty legal for brand-new-object proposals)/justification/:99 `candidateState: Int64`(adjective set to apply if accepted)/:106 `latticeAnchor`(required, I-16) -- :165 `state: State` — decodes proposal's OWN lifecycle axis (pending→accepted/rejected/withdrawn) from adjectiveBitmap bits 0-5 - -### Proposal operational — ProposalOperational.swift -- :60 `enum ProposalKind: Int` — newTunnel=0…tierAdvisory=8, bits 0-5 (COOKBOOK-SPECIFIED layout — conformance-gated, unlike KGFact's LocusKit-internal layout) -- :81 `enum ProposalTargetObjectType: Int` — drawer=0…systemState=6, bits 6-11 (.noneBrandNew=4 ⇒ targetRowID empty) -- :100 `enum ProposalConfirmationSource: Int` — human=0/agent=1/automatedThreshold=2/actuator=3, bits 12-17 -- :116 `enum ProposalGeneratedByClass: Int` — dreamingDaemon=0…tierAggregator=4, bits 18-23 -- :136 `enum ProposalConfidenceBucket: Int, Comparable` — null=0/low=8/medium=16/high=32/verified=48, bits 24-29 -- :207 `composeOperational(kind:targetObjectType:generatedBy:confidence:) -> Int64` (static) — assembles 4 of 5 axes via BitField.writeField; confirmationSource left at .human default - -### Association — Association.swift -- :66 `struct Association: Equatable, Codable, Sendable` (NOT Hashable — embedded LatticeAnchor isn't) — mirrors Tunnel minus `kind`, plus REQUIRED :103 `latticeAnchor` (I-16, lattice-midpoint of endpoints) - -### Association operational — AssociationOperational.swift -- :47 `struct AssociationSignalSources: OptionSet` — bits 0-11 bitset (coRecall/coConfirmed/dreamPairing/vectorSimilarity/sharedEntity/explicitHuman/fingerprintSimilarity/crossEstate/crossTier/actionOutcome); :75 `mask = 0xFFF` -- :90 `enum AssociationDecayClass: Int, Comparable` — pinned=0/slow=16/normal=32/fast=48, bits 12-17 -- :106 `enum AssociationArity: Int` — binary=0(v1 only, I-23)/nAry=1(reserved), bits 18-19 - -### Learned reference — LearnedReference.swift -- :85 `struct LearnedReference: Equatable, Codable, Sendable` (NOT Hashable) — mirrors Association; :96 `sourceCatalogID`(FK→SourceCatalogEntry.id)/:101 `handle`(reference's own locator)/:107 `latticeAnchor`(required, I-16, inherited from catalog entry — NEVER fabricated) - -### Learned reference operational — LearnedReferenceOperational.swift -- :47 `enum RefreshPolicy: Int, Comparable` — none=0/monthly=16/weekly=24/daily=32/onDemand=48/realtime=56, bits 0-5 -- :68 `enum DriftSeverity: Int, Comparable` — none=0/minor=16/major=32/critical=48, bits 6-11 -- :85 `enum LearnMode: Int` — byReference=0/byIngestion=1, bit 12 -- :96 `enum LearnedReferenceSource: Int` — user=0…pairedEstate=5, bits 13-18 (same vocabulary as SourceKind — acquisition channel == source class) - -### Source catalog — SourceCatalogEntry.swift -- :50 `struct SourceCatalogEntry: Equatable, Codable, Sendable` (NOT Hashable) — id/kind(SourceKind)/handle/:68 `latticeAnchor`(required, I-16, GENUINE anchor every LearnedReference from this source inherits)/firstSeen/addedBy -- :103 `enum SourceKind: Int, CaseIterable` — user=0…pairedEstate=5; :119 `fromRaw(_:)` fail-closed to .user - -### Containment tree node — Node.swift -- :25 `struct Node: Sendable, Equatable, Codable, Hashable` — id/parentId(nil only at depth 0)/:34 `displayName`(first-writer casing)/:38 `lookupName`(NFC+casefold+whitespace-collapse, ALL resolution keys use this)/:41 `depth`(0=estate,1=wing,2=room, write-once)/:44 `lifecycle`(0 active,1 tombstoned)/createdHlc/tombstonedHlc/merkleRoot -- :116 `normalizeLookupName(_:) -> String` (static) — CONFORMANCE-GATED, Swift+Rust must byte-match - -### Node store — NodeStore.swift -- :39 `actor NodeStore` — create-on-demand resolution race-free via actor serialization (no INSERT-OR-IGNORE needed) -- :89 `createNode(displayName:parentId:now:) -> Node` — enforces I-NT-5(parent must exist)/I-NT-2(depth≤2)/I-NT-4(no dup active lookupName under parent); tombstoned nodes invisible to resolution (§5 no-resurrection) -- :165 `createRoot(displayName:now:) -> Node` — I-NT-1 single root, idempotent +### Tunnel operational : TunnelOperational.swift +- :38 `enum TunnelKind: Int` : supersedes=0…parent=9 (outline containment, ADR-017 §11; source=child, target=parent, one-parent-per-child kit-level constraint) +- :60/:70/:80/:94 `TunnelDirection`/`TunnelLifecycle`/`TunnelOriginClass`/`TunnelStrength(Comparable, scale-gapped 0/2/4/6)` : bits 0-2/3-5/6-8/9-11 +- :168 `isRetiredBit = 1<<13` / :175 `isRetired` / :184/:202 `withRetired()`/`withUnretired()` : T13/ADR-021 Phase 7 reversible dreaming suspension (NOT tombstone) +- :241 `isDreamedBit = 1<<0` (provenanceBitmap) / :247 `isDreamed` / :256 `withDreamedProvenance()` : OMEGA retires ONLY isDreamed==true tunnels; declared tunnels never retired + +### Diary : DiaryEntry.swift +- :24 `struct DiaryEntry` : agentName/entry/topic/wing/room (all required, unvalidated strings) +- :85/:92 `reward: Double?`/`rewardProvenance: String?` : explicit quality-signal channel (NEURONKIT_SPEC §3.1 step 1a); nil ⇒ dreaming daemon falls back to implicit RecallTraceItem.used (step 1b) + +### Diary operational : DiaryOperational.swift +- :33 `enum DiaryEventClass: Int` : 12 cases (capture=0…auditTombstone=11), bits 0-3 +- :54 `enum DiarySeverity: Int, Comparable` : trace=0/info=2/warning=4/error=6, bits 4-6 +- :67 `enum DiaryActorClass: Int` : user=0…federationPeer=4, bits 7-9 +- :79 `enum DiaryBatchMembership: Int` : standalone=0…batchEnd=3, bits 10-12 +- :129 `requiresFollowup` : bit 13 flag + +### KG fact : KGFact.swift +- :51 `struct KGFact` : subject/predicate/object (free-form strings, no entity vocabulary yet) + :82 `sourceDrawerID` (may be "" = unanchored-fact sentinel) +- Adjective accessors :149/:159/:171/:180 `trust`/`state`/`adjectiveSensitivity`/`exportability` : IDENTICAL bit layout to Drawer's (shared retrieval predicates) + +### KG fact operational : KGFactOperational.swift +- :48 `enum KGExtractorClass: Int` : manual=0…federated=5, bits 0-3 +- :66 `enum KGAssertionKind: Int` : asserted=0/inferred=1/hypothesized=2/contradicted=3, bits 4-6 +- :83 `enum KGSpecificity: Int, Comparable` : general=0…instance=6 (scale-gapped), bits 7-9 +- :106 `enum KGConfidenceBand: Int, Comparable` : unknown=0…certain=6 (scale-gapped), bits 10-12 +- :171 `isCanonical` : bit 13 flag (estate-wide load-bearing promotion) + +### Proposal : Proposal.swift +- :72 `struct Proposal` : targetRowID(empty legal for brand-new-object proposals)/justification/:99 `candidateState: Int64`(adjective set to apply if accepted)/:106 `latticeAnchor`(required, I-16) +- :165 `state: State` : decodes proposal's OWN lifecycle axis (pending→accepted/rejected/withdrawn) from adjectiveBitmap bits 0-5 + +### Proposal operational : ProposalOperational.swift +- :60 `enum ProposalKind: Int` : newTunnel=0…tierAdvisory=8, bits 0-5 (COOKBOOK-SPECIFIED layout : conformance-gated, unlike KGFact's LocusKit-internal layout) +- :81 `enum ProposalTargetObjectType: Int` : drawer=0…systemState=6, bits 6-11 (.noneBrandNew=4 ⇒ targetRowID empty) +- :100 `enum ProposalConfirmationSource: Int` : human=0/agent=1/automatedThreshold=2/actuator=3, bits 12-17 +- :116 `enum ProposalGeneratedByClass: Int` : dreamingDaemon=0…tierAggregator=4, bits 18-23 +- :136 `enum ProposalConfidenceBucket: Int, Comparable` : null=0/low=8/medium=16/high=32/verified=48, bits 24-29 +- :207 `composeOperational(kind:targetObjectType:generatedBy:confidence:) -> Int64` (static) : assembles 4 of 5 axes via BitField.writeField; confirmationSource left at .human default + +### Association : Association.swift +- :66 `struct Association: Equatable, Codable, Sendable` (NOT Hashable : embedded LatticeAnchor isn't) : mirrors Tunnel minus `kind`, plus REQUIRED :103 `latticeAnchor` (I-16, lattice-midpoint of endpoints) + +### Association operational : AssociationOperational.swift +- :47 `struct AssociationSignalSources: OptionSet` : bits 0-11 bitset (coRecall/coConfirmed/dreamPairing/vectorSimilarity/sharedEntity/explicitHuman/fingerprintSimilarity/crossEstate/crossTier/actionOutcome); :75 `mask = 0xFFF` +- :90 `enum AssociationDecayClass: Int, Comparable` : pinned=0/slow=16/normal=32/fast=48, bits 12-17 +- :106 `enum AssociationArity: Int` : binary=0(v1 only, I-23)/nAry=1(reserved), bits 18-19 + +### Learned reference : LearnedReference.swift +- :85 `struct LearnedReference: Equatable, Codable, Sendable` (NOT Hashable) : mirrors Association; :96 `sourceCatalogID`(FK→SourceCatalogEntry.id)/:101 `handle`(reference's own locator)/:107 `latticeAnchor`(required, I-16, inherited from catalog entry : NEVER fabricated) + +### Learned reference operational : LearnedReferenceOperational.swift +- :47 `enum RefreshPolicy: Int, Comparable` : none=0/monthly=16/weekly=24/daily=32/onDemand=48/realtime=56, bits 0-5 +- :68 `enum DriftSeverity: Int, Comparable` : none=0/minor=16/major=32/critical=48, bits 6-11 +- :85 `enum LearnMode: Int` : byReference=0/byIngestion=1, bit 12 +- :96 `enum LearnedReferenceSource: Int` : user=0…pairedEstate=5, bits 13-18 (same vocabulary as SourceKind : acquisition channel == source class) + +### Source catalog : SourceCatalogEntry.swift +- :50 `struct SourceCatalogEntry: Equatable, Codable, Sendable` (NOT Hashable) : id/kind(SourceKind)/handle/:68 `latticeAnchor`(required, I-16, GENUINE anchor every LearnedReference from this source inherits)/firstSeen/addedBy +- :103 `enum SourceKind: Int, CaseIterable` : user=0…pairedEstate=5; :119 `fromRaw(_:)` fail-closed to .user + +### Containment tree node : Node.swift +- :25 `struct Node: Sendable, Equatable, Codable, Hashable` : id/parentId(nil only at depth 0)/:34 `displayName`(first-writer casing)/:38 `lookupName`(NFC+casefold+whitespace-collapse, ALL resolution keys use this)/:41 `depth`(0=estate,1=wing,2=room, write-once)/:44 `lifecycle`(0 active,1 tombstoned)/createdHlc/tombstonedHlc/merkleRoot +- :116 `normalizeLookupName(_:) -> String` (static) : CONFORMANCE-GATED, Swift+Rust must byte-match + +### Node store : NodeStore.swift +- :39 `actor NodeStore` : create-on-demand resolution race-free via actor serialization (no INSERT-OR-IGNORE needed) +- :89 `createNode(displayName:parentId:now:) -> Node` : enforces I-NT-5(parent must exist)/I-NT-2(depth≤2)/I-NT-4(no dup active lookupName under parent); tombstoned nodes invisible to resolution (§5 no-resurrection) +- :165 `createRoot(displayName:now:) -> Node` : I-NT-1 single root, idempotent - :207/:220/:237 `getNode(id:)`/`rootNode()`/`childNodes(parentId:)` - :259 `tombstoneNode(id:now:) -> Node?` -- :296 `updateMerkleRoot(nodeId:merkleRoot:now:)` — 32-byte BLOB write +- :296 `updateMerkleRoot(nodeId:merkleRoot:now:)` : 32-byte BLOB write -### Node bundle store — NodeBundleStore.swift -- :27 `actor NodeBundleStore` — :32 `enum BundleKind: String` .activeA("A")/.departedB("B") -- :51/:68 `encodeCounts(_:)`/`decodeCounts(_:n:)` — 256×UInt32 LE = 1024 bytes; decode throws (not traps) on wrong size +### Node bundle store : NodeBundleStore.swift +- :27 `actor NodeBundleStore` : :32 `enum BundleKind: String` .activeA("A")/.departedB("B") +- :51/:68 `encodeCounts(_:)`/`decodeCounts(_:n:)` : 256×UInt32 LE = 1024 bytes; decode throws (not traps) on wrong size - :87/:103/:119 `put`/`get`/`rooms(forWing:kind:)` -### Bundle materializer — BundleMaterializer.swift -- :34 `struct BundleMaterializer` — first real caller of SubstrateKernel.countFold256 -- :63 `materializeRoom(wing:room:now:) -> CountVector256` — filters to State.isClusterA before fold (Bundle A cannot be incremental — no subtraction) -- :79 `rollUpWing(wing:now:) -> CountVector256` — merges room bundles; associative, so room materialize order doesn't matter - -### Fingerprint256 adapters — Fingerprint256Adapters.swift -- :55 `Fingerprint256.init(int64Column:)` / :63 `.int64Column` — packs/unpacks one Int64 bitmap column into block 0 (blocks 1-3 zero); pure type-shape convention, NOT the cookbook §3.2 Bitmap-LSH SimHash interpretation of block 0 — do not conflate the two uses - -### Container fingerprint pruning — ContainerFingerprintStore.swift -- :39 `struct ContainerFingerprint: Sendable, Equatable` — adjective/operational/provenance Int64 OR-aggregate; :68 `merging(_:)` routes through SubstrateLib.ORReduce at Fingerprint256 width -- :87 `actor ContainerFingerprintStore` — :91 `wingRollupRoom = ""` sentinel for wing-level row -- :105/:123 `get(wing:room:)`/`roomLevelEntries()` — nil result means "must scan", not "empty" (unsound to prune against absence) -- :146 `orIn(wing:room:adjective:operational:provenance:now:)` — incremental maintenance, called on every capture; NO clear-side (stale set bits are harmless over-approximation) -- :167/:182/:200 `rebuildRoom`/`rollUpWing`/`rebuildAll` — periodic tightening; rebuildAll runs once at Estate.open - -### Merkle rollup — MerkleRollup.swift (extension Estate) -- :54 `rollupRoomsForDrawers(_ drawerIds:)` — coalesced per-room rollup for the deferred/off-write-path queue worker -- :71 `rollupMerkleRoots(roomNodeId:now:)` — room→wing→estate, 3-level bottom-up for ONE changed room -- :125 `computeRoomMerkleRoot(roomNodeId:)` — excludes tombstoned AND withdrawn(state=18) drawers from the snapshot (WS2-F1); reads content_hash column when present, else computes leaf on-demand -- :175/:186 `computeEstateOrWingMerkleRoot(parentNodeId:)`/`computeWingMerkleRoot(wingNodeId:)` — same function serves both levels -- :200 `recomputeAllMerkleRoots(now:)` — full bottom-up; :243 `rollupAllMerkleRoots(now:)` alias for batch-capture reindex (NT_R1) -- :265 `createSnapshot(label:now:additionalAttestations:)` — ALWAYS recomputes full tree first (capture defers rollup, so cached roots may be stale) -- :332 `deterministicUUID(from:)` (static) — SHA-256-derived stable UUID for non-UUID drawer ids (version/variant bits stamped) - -### Manifest — Manifest.swift -- :6 `enum ManifestKey: String, CaseIterable` — 18 required + 7 optional keys; :41 `.ed25519PublicKey` (safe, public) / :56 `.ed25519PrivateKeyWrapped` (NEVER written by current code — reserved read-compat seam only, real private key lives in EstateIdentityKeyStore) -- :80 `struct ManifestValues` — typed snapshot from DrawerStore.readManifest() - -### Default wings — DefaultWings.swift -- :18 `defaultWingName = "Agentic Memory"` (ADR-016, fixed — NOT owner-derived) / :26 `hintRoom = "AI_Charter_Hint"` / :32 `hintUDCCode = "001"` / :36 `hintAddedBy = "estate-provision"` -- :44 `struct WingDefinition` — name+hint / :64 `defaultWings: [WingDefinition]` — 7 seeded wings (suggestion, not enforced schema) - -### Recall stream — RecallStream.swift -- :21 `struct RecallStream: AsyncSequence, Sendable` — :26 `defaultPageSize = 50` -- :54 `degradedStages: [String]` — non-throwing failure channel (empty ⇒ genuine result incl. genuine-empty; non-empty ⇒ named internal-read/eval failure) -- :86 `struct RecallPage` — rows/pageIndex/isLast -- :92 `struct AsyncIterator: AsyncIteratorProtocol` — first page synchronous; :129 `hydrate(_:)` (private) rebuilds with content="" ONLY at .bitmapOnly - -### Recall trace — RecallTraceItem.swift -- :28 `flagUsed: Int64 = 1<<0` / :58 `used: Bool` (computed, no stored Bool — bitmap-backed pattern every noun follows) / :47 `score: Double?` (nil = no score, e.g. capture-time-ordered recall) - -### Audit types — AuditTypes.swift -- :26 `struct BitmapState` — rowID/asOf(HLC)/adjectiveBitmap/operationalBitmap/provenanceBitmap; LocusKit's own wrapper over AuditLogFold.projectStateAt's result shape - -### Summaries — Summaries.swift -- :10 `struct WingSummary` — name/drawerCount/roomCount / :33 `struct RoomSummary` — wing/name/drawerCount; BOTH are live computed projections, no dedicated wings/rooms table - -### Telemetry — LocusKitTelemetry.swift -- All functions `@inline(__always)`, `Intellectus.report(_:)` with @autoclosure (never evaluated when monitoring off — ~1ns off-path cost, no lock, no alloc) -- `now:` ALWAYS caller-supplied — never Date() inside this file (determinism contract) +### Bundle materializer : BundleMaterializer.swift +- :34 `struct BundleMaterializer` : first real caller of SubstrateKernel.countFold256 +- :63 `materializeRoom(wing:room:now:) -> CountVector256` : filters to State.isClusterA before fold (Bundle A cannot be incremental : no subtraction) +- :79 `rollUpWing(wing:now:) -> CountVector256` : merges room bundles; associative, so room materialize order doesn't matter + +### Fingerprint256 adapters : Fingerprint256Adapters.swift +- :55 `Fingerprint256.init(int64Column:)` / :63 `.int64Column` : packs/unpacks one Int64 bitmap column into block 0 (blocks 1-3 zero); pure type-shape convention, NOT the cookbook §3.2 Bitmap-LSH SimHash interpretation of block 0 : do not conflate the two uses + +### Container fingerprint pruning : ContainerFingerprintStore.swift +- :39 `struct ContainerFingerprint: Sendable, Equatable` : adjective/operational/provenance Int64 OR-aggregate; :68 `merging(_:)` routes through SubstrateLib.ORReduce at Fingerprint256 width +- :87 `actor ContainerFingerprintStore` : :91 `wingRollupRoom = ""` sentinel for wing-level row +- :105/:123 `get(wing:room:)`/`roomLevelEntries()` : nil result means "must scan", not "empty" (unsound to prune against absence) +- :146 `orIn(wing:room:adjective:operational:provenance:now:)` : incremental maintenance, called on every capture; NO clear-side (stale set bits are harmless over-approximation) +- :167/:182/:200 `rebuildRoom`/`rollUpWing`/`rebuildAll` : periodic tightening; rebuildAll runs once at Estate.open + +### Merkle rollup : MerkleRollup.swift (extension Estate) +- :54 `rollupRoomsForDrawers(_ drawerIds:)` : coalesced per-room rollup for the deferred/off-write-path queue worker +- :71 `rollupMerkleRoots(roomNodeId:now:)` : room→wing→estate, 3-level bottom-up for ONE changed room +- :125 `computeRoomMerkleRoot(roomNodeId:)` : excludes tombstoned AND withdrawn(state=18) drawers from the snapshot (WS2-F1); reads content_hash column when present, else computes leaf on-demand +- :175/:186 `computeEstateOrWingMerkleRoot(parentNodeId:)`/`computeWingMerkleRoot(wingNodeId:)` : same function serves both levels +- :200 `recomputeAllMerkleRoots(now:)` : full bottom-up; :243 `rollupAllMerkleRoots(now:)` alias for batch-capture reindex (NT_R1) +- :265 `createSnapshot(label:now:additionalAttestations:)` : ALWAYS recomputes full tree first (capture defers rollup, so cached roots may be stale) +- :332 `deterministicUUID(from:)` (static) : SHA-256-derived stable UUID for non-UUID drawer ids (version/variant bits stamped) + +### Manifest : Manifest.swift +- :6 `enum ManifestKey: String, CaseIterable` : 18 required + 7 optional keys; :41 `.ed25519PublicKey` (safe, public) / :56 `.ed25519PrivateKeyWrapped` (NEVER written by current code : reserved read-compat seam only, real private key lives in EstateIdentityKeyStore) +- :80 `struct ManifestValues` : typed snapshot from DrawerStore.readManifest() + +### Default wings : DefaultWings.swift +- :18 `defaultWingName = "Agentic Memory"` (ADR-016, fixed : NOT owner-derived) / :26 `hintRoom = "AI_Charter_Hint"` / :32 `hintUDCCode = "001"` / :36 `hintAddedBy = "estate-provision"` +- :44 `struct WingDefinition` : name+hint / :64 `defaultWings: [WingDefinition]` : 7 seeded wings (suggestion, not enforced schema) + +### Recall stream : RecallStream.swift +- :21 `struct RecallStream: AsyncSequence, Sendable` : :26 `defaultPageSize = 50` +- :54 `degradedStages: [String]` : non-throwing failure channel (empty ⇒ genuine result incl. genuine-empty; non-empty ⇒ named internal-read/eval failure) +- :86 `struct RecallPage` : rows/pageIndex/isLast +- :92 `struct AsyncIterator: AsyncIteratorProtocol` : first page synchronous; :129 `hydrate(_:)` (private) rebuilds with content="" ONLY at .bitmapOnly + +### Recall trace : RecallTraceItem.swift +- :28 `flagUsed: Int64 = 1<<0` / :58 `used: Bool` (computed, no stored Bool : bitmap-backed pattern every noun follows) / :47 `score: Double?` (nil = no score, e.g. capture-time-ordered recall) + +### Audit types : AuditTypes.swift +- :26 `struct BitmapState` : rowID/asOf(HLC)/adjectiveBitmap/operationalBitmap/provenanceBitmap; LocusKit's own wrapper over AuditLogFold.projectStateAt's result shape + +### Summaries : Summaries.swift +- :10 `struct WingSummary` : name/drawerCount/roomCount / :33 `struct RoomSummary` : wing/name/drawerCount; BOTH are live computed projections, no dedicated wings/rooms table + +### Telemetry : LocusKitTelemetry.swift +- All functions `@inline(__always)`, `Intellectus.report(_:)` with @autoclosure (never evaluated when monitoring off : ~1ns off-path cost, no lock, no alloc) +- `now:` ALWAYS caller-supplied : never Date() inside this file (determinism contract) - :50 `emitDrawerCapture` / :90 `emitDrawerQuery` / :123 `emitKGFactAdd` / :149 `emitKGFactQuery` / :175 `emitTunnelAdd` / :199 `emitGateAdmit` / :224 `emitGateReject`(reason truncated to 64 chars) - Namespace: `locuskit..` @@ -462,20 +462,20 @@ ENTRY POINTS (most callers need only these): - SOURCE OF TRUTH IS THE AUDIT LOG, NOT THE LIVE ROW. Every gated write (mutateState/mutateAdjective/mutateOperational/mutateProvenance/expungeGated/reanchorGated) updates the projection column AND appends the sealed AuditEvent inside ONE `storage.transaction(isolation: .serializable)`. Never split these two writes across transactions except via the documented sealAudit:false + sealExpungeAudit(_:) two-step (GLK §B-2a orchestration only). - The state field (adjectiveBitmap bits 0-5) is VERB-DRIVEN ONLY. `gatedColumnWrite` (whole-column field-edit path) explicitly excludes it. State only ever moves through `mutateState`, which validates `(from, via, to)` against SubstrateLib.RowStateAutomaton BEFORE any write. -- DrawerStore.addDrawer is `internal`, not `public`. The only sanctioned add path is `Estate.addDrawerCovered` (private in EstateVerbs.swift), which bundles the row insert with `containerFP.orIn` — this is what makes "captured a drawer without updating its container fingerprint" structurally impossible. Tests reaching `store.addDrawer` directly via `@testable import` must document why fingerprint coverage is not needed for that path. -- ContainerFingerprintStore has NO clear side by design. Withdrawing/expunging a drawer never clears bits from the room/wing OR-aggregate — a stale set bit only causes an unnecessary scan, never a missed match. `rebuildRoom`/`rollUpWing`/`rebuildAll` periodically tighten; `Estate.open` calls `rebuildAll` once so the aggregate is sound from the first recall. +- DrawerStore.addDrawer is `internal`, not `public`. The only sanctioned add path is `Estate.addDrawerCovered` (private in EstateVerbs.swift), which bundles the row insert with `containerFP.orIn` : this is what makes "captured a drawer without updating its container fingerprint" structurally impossible. Tests reaching `store.addDrawer` directly via `@testable import` must document why fingerprint coverage is not needed for that path. +- ContainerFingerprintStore has NO clear side by design. Withdrawing/expunging a drawer never clears bits from the room/wing OR-aggregate : a stale set bit only causes an unnecessary scan, never a missed match. `rebuildRoom`/`rollUpWing`/`rebuildAll` periodically tighten; `Estate.open` calls `rebuildAll` once so the aggregate is sound from the first recall. - Expunge's destruction contract is STRONGER than the state machine. In `expungeGated`, every lineage sibling has its content column unconditionally zeroed even when the gate refuses that sibling's state transition (e.g. an `accepted` row, forbidden from reaching `.tombstoned` by S-3). Leaving verbatim content behind because a state flip was refused would be a privacy violation. -- Tombstone exclusion in `BitmapEvaluator.evaluateBitmapTier` is UNCONDITIONAL and independent of the caller's filter chain — `state == 33` (raw State.tombstoned) never surfaces through recall no matter what filters are supplied. +- Tombstone exclusion in `BitmapEvaluator.evaluateBitmapTier` is UNCONDITIONAL and independent of the caller's filter chain : `state == 33` (raw State.tombstoned) never surfaces through recall no matter what filters are supplied. - `recall(_:)` NEVER THROWS (spec §7.8.1). Internal read/eval failures are surfaced exclusively via `RecallStream.degradedStages` (named stage strings). An empty `rows` with empty `degradedStages` means genuinely no matches; an empty `rows` with a populated `degradedStages` means a failed internal read. -- PersistenceKit v1.0 has NO nested transactions. This is why `gatedCaptureBody` exists as a separable @Sendable closure builder — `addDrawerWithCascade` needs the drawer INSERT and the supersedes-tunnel INSERT in ONE outer transaction, and cannot call `gatedCapture` (which opens its own transaction) from inside another transaction. -- `findActivePredecessor` predicate is `g_state_cluster < 3`; `livingSuccessorInLineage` predicate is `g_state_cluster < RowState.activeClusterUpperBoundRaw (16)`. These are DIFFERENT thresholds for different purposes (supersession-cascade target vs revive-guard living-successor check) — do not conflate them. +- PersistenceKit v1.0 has NO nested transactions. This is why `gatedCaptureBody` exists as a separable @Sendable closure builder : `addDrawerWithCascade` needs the drawer INSERT and the supersedes-tunnel INSERT in ONE outer transaction, and cannot call `gatedCapture` (which opens its own transaction) from inside another transaction. +- `findActivePredecessor` predicate is `g_state_cluster < 3`; `livingSuccessorInLineage` predicate is `g_state_cluster < RowState.activeClusterUpperBoundRaw (16)`. These are DIFFERENT thresholds for different purposes (supersession-cascade target vs revive-guard living-successor check) : do not conflate them. - Corpus scans (`allDrawers`, `drawersIn`, etc.) use `decodeDrawerRowsSkipCorrupt`: a `.corruptStoredValue` on one row is logged and skipped, everything else re-thrown. Point lookups (`getDrawer(id:)`, `getDrawers(ids:)`) use the strict `decodeDrawerRows`, which throws on the first corrupt row. Do not swap these two decode paths between call sites. -- `allDrawers(hydrationLevel:limit:direction:)` sorts by compound key `(filedAt, id)`, NOT SQLite `rowid` — `id` is the declared TEXT primary key, present and portable across SQLite/PostgreSQL/InMemory. DESC is the exact byte-for-byte reverse of ASC for any fixed dataset because of this deterministic tie-break. +- `allDrawers(hydrationLevel:limit:direction:)` sorts by compound key `(filedAt, id)`, NOT SQLite `rowid` : `id` is the declared TEXT primary key, present and portable across SQLite/PostgreSQL/InMemory. DESC is the exact byte-for-byte reverse of ASC for any fixed dataset because of this deterministic tie-break. - `RecallFrame.traceLimit` defaults to `nil`, which writes ZERO recall-trace rows. Only the GLK RecallDirector's primary locus call sets a non-nil traceLimit. Any new caller of `recall(_:)` that leaves traceLimit nil is correct by default; do not add trace writes to a scan-style caller without deliberate reason (write amplification). - Bitmap layout constants in `BitmapEvaluator.swift` (adjStateMask, provConfirmMask, opChannelMask, etc.) are a PRIVATE MIRROR of the accessor decoders in Adjectives.swift/Provenance.swift/DrawerOperational.swift. A schema bump (`bitmap_layout_version`) that moves a field must update BOTH the accessor and this mirror in the same change. -- `@guardian-pair` comments (Adjectives.swift, DrawerStore.swift mutateState/expungeGated) mark raw-value sets that MUST stay in sync with SubstrateLib's AuditGate.basis legalValues — a build-time Guardian tool checks these; GuardianPairParityTests is the CI backstop. Touching a State/Trust/AdjectiveSensitivity/AdjectiveExportability case requires checking every paired comment. -- Nine nouns share the three-Int64-bitmap pattern (adjective/operational/provenance) EXCEPT: `Tunnel`/`Association`/`LearnedReference` have no `provenance`-axis adjective sharing beyond the bitmap slot layout itself being noun-specific; `Node` has no bitmap columns at all (lifecycle is a plain Int, not a bitmap). `Association`, `Proposal`, `LearnedReference`, `SourceCatalogEntry` all require a non-empty `latticeAnchor.udcCode` at insert (I-16) — `KGFact` and `Tunnel` predate I-16 and carry no anchor requirement. -- `Association`/`Proposal`/`LearnedReference`/`SourceCatalogEntry` are `Equatable, Codable, Sendable` but deliberately NOT `Hashable` (the embedded `LatticeAnchor` isn't Hashable). Do not add `Hashable` conformance without also widening `LatticeAnchor` — nothing today keys a Set/Dictionary on any of these four types. +- `@guardian-pair` comments (Adjectives.swift, DrawerStore.swift mutateState/expungeGated) mark raw-value sets that MUST stay in sync with SubstrateLib's AuditGate.basis legalValues : a build-time Guardian tool checks these; GuardianPairParityTests is the CI backstop. Touching a State/Trust/AdjectiveSensitivity/AdjectiveExportability case requires checking every paired comment. +- Nine nouns share the three-Int64-bitmap pattern (adjective/operational/provenance) EXCEPT: `Tunnel`/`Association`/`LearnedReference` have no `provenance`-axis adjective sharing beyond the bitmap slot layout itself being noun-specific; `Node` has no bitmap columns at all (lifecycle is a plain Int, not a bitmap). `Association`, `Proposal`, `LearnedReference`, `SourceCatalogEntry` all require a non-empty `latticeAnchor.udcCode` at insert (I-16) : `KGFact` and `Tunnel` predate I-16 and carry no anchor requirement. +- `Association`/`Proposal`/`LearnedReference`/`SourceCatalogEntry` are `Equatable, Codable, Sendable` but deliberately NOT `Hashable` (the embedded `LatticeAnchor` isn't Hashable). Do not add `Hashable` conformance without also widening `LatticeAnchor` : nothing today keys a Set/Dictionary on any of these four types. - `Node.normalizeLookupName(_:)` and `DrawerFingerprint`'s four-block derivation are CONFORMANCE-GATED: the Rust port must produce byte-identical output for identical input. Changing either requires mirroring the change in rust/src/ and passing the corresponding conformance test. -- Pinned/fixed constants — changing any requires a coordinated Swift+Rust update and, where noted, a schema version bump: `LocusKitSchema.version` (8), `Estate.expectedBitmapLayoutVersion` ("v1.0"), `EstateVerbs.recallCandidateCap` (256), `RecallStream.defaultPageSize` (50), `RecallTraceItem.flagUsed`/`Tunnel.isRetiredBit`(bit 13)/`Tunnel.isDreamedBit`(bit 0), `NodeStore` depth ceiling (2), `outlineAncestors` depth ceiling (256), `NodeBundleStore` count-vector width (256 × UInt32 = 1024 bytes), `defaultWingName`("Agentic Memory")/`hintRoom`/`hintUDCCode`/`hintAddedBy`. -- LocusKit ships NO pinned data artifacts (no Resources/ directory of the LatticeLib kind) — its schema is generated fresh from `LocusKitSchema.schema` at every open; the only external pinned artifact it consumes is LatticeLib's Q-ID closure snapshot, reached read-only via `QIDClosure.ancestors(of:)`. +- Pinned/fixed constants : changing any requires a coordinated Swift+Rust update and, where noted, a schema version bump: `LocusKitSchema.version` (8), `Estate.expectedBitmapLayoutVersion` ("v1.0"), `EstateVerbs.recallCandidateCap` (256), `RecallStream.defaultPageSize` (50), `RecallTraceItem.flagUsed`/`Tunnel.isRetiredBit`(bit 13)/`Tunnel.isDreamedBit`(bit 0), `NodeStore` depth ceiling (2), `outlineAncestors` depth ceiling (256), `NodeBundleStore` count-vector width (256 × UInt32 = 1024 bytes), `defaultWingName`("Agentic Memory")/`hintRoom`/`hintUDCCode`/`hintAddedBy`. +- LocusKit ships NO pinned data artifacts (no Resources/ directory of the LatticeLib kind) : its schema is generated fresh from `LocusKitSchema.schema` at every open; the only external pinned artifact it consumes is LatticeLib's Q-ID closure snapshot, reached read-only via `QIDClosure.ancestors(of:)`. diff --git a/packages/kits/LocusKit/docs/DETAILS.md b/packages/kits/LocusKit/docs/DETAILS.md index 5d1d1c2..d60564c 100644 --- a/packages/kits/LocusKit/docs/DETAILS.md +++ b/packages/kits/LocusKit/docs/DETAILS.md @@ -107,1209 +107,1309 @@ sources: This document walks through every source file in the package. Read `OVERVIEW.md` first for the big picture. Files appear here in -pipeline order: the module surface, the estate lifecycle, the nine -verbs and their frame types, the storage engine, the drawer value type -and its shared bitmap machinery, the recall filter and evaluator, the -remaining eight noun types and their bitmap decoders, the containment -tree and bundle algebra, fingerprint and integrity machinery, manifest -and provisioning data, the recall-result and trace types, and finally -telemetry. +pipeline order. First come the module surface and the estate +lifecycle. Next come the nine verbs and their frame types. Then comes +the storage engine. After that comes the drawer value type and its +shared bitmap machinery. Next comes the recall filter and its +evaluator. Then come the remaining eight noun types and their bitmap +decoders. After that comes the containment tree and the bundle +algebra. Next comes the fingerprint and integrity machinery. Then +comes the manifest and setup data. Next come the recall-result +and trace types. Telemetry comes last. ## LocusKit.swift -This file provides the module surface: a doc comment describing the -package and pointing to the files that carry the real public types. -LocusKit has no module-level namespace enum of its own (unlike some -sibling libs) — every public type lives in its own file, and this file -exists only to orient a reader arriving at the package for the first -time. +This file provides the module surface. It carries a doc comment +describing the package. That comment points to the files holding the +real public types. LocusKit has no module-level namespace enum of its +own, unlike some sibling libs. Every public type lives in its own +file. This file exists only to orient a reader arriving at the +package for the first time. ## LocusKitError.swift This file provides `LocusKitError`, the single error enum every LocusKit method throws. Centralizing every failure mode in one type -lets a caller recover specifically — for example, treating a missing -drawer as a routine miss while still propagating a genuine SQLite -failure. - -The cases split into three groups. Not-found cases -(`drawerNotFound`, `tunnelNotFound`, `diaryEntryNotFound`, -`recallTraceItemNotFound`) name a missing row by id. Storage cases -(`databaseUnavailable`, `sqliteError`, `schemaTooNew`, -`corruptStoredValue`) describe the storage layer misbehaving; -`corruptStoredValue` in particular names the table, column, and raw -stored text, so a caller can locate and repair the offending row -rather than receive a fabricated default value. Domain cases -(`invalidContent`, `disciplineViolation`, `notSupported`) describe -LocusKit's own validation refusing a write: `invalidContent` carries a -human-readable rule violation (a validation-test contract — tests -assert on the message text), and `disciplineViolation` names an -illegal state transition or forbidden bitmap combination using the raw -integer state values rather than the `State` enum, so the error type -does not need to depend on `Adjectives.swift`. - -`LocusKitError.description` renders each case as plain English without -Swift's `EnumCase(...)` noise. This description is consumed at the -GeniusLocusKit boundary, where a caller parses the English text back -out to decide how to react to a rejected write — so the wording is a -soft contract, not just a log message. +lets a caller recover in exact ways. For example, a caller can treat a +missing drawer as a routine miss. It can still propagate a genuine +SQLite failure. + +The cases split into three groups. The not-found group names a +missing row by id: `drawerNotFound`, `tunnelNotFound`, +`diaryEntryNotFound`, and `recallTraceItemNotFound`. The storage group +describes the storage layer misbehaving: `databaseUnavailable`, +`sqliteError`, `schemaTooNew`, and `corruptStoredValue`. +`corruptStoredValue` names the table, column, and raw stored text. So +a caller can locate and repair the offending row, rather than receive +a fabricated default value. The domain group describes LocusKit's own +validation refusing a write: `invalidContent`, `disciplineViolation`, +and `notSupported`. `invalidContent` carries a human-readable rule +violation. Tests assert on this message text, as a validation-test +contract. `disciplineViolation` names an illegal state transition or +forbidden bitmap combination. It uses the raw integer state values, +rather than the `State` enum. So the error type does not need to +depend on `Adjectives.swift`. + +`LocusKitError.description` renders each case as plain English, +without Swift's `EnumCase(...)` noise. GeniusLocusKit consumes this +description at its boundary. A caller there parses the English text +back out, to decide how to react to a rejected write. So the wording +is a soft contract, not just a log message. ## LocusKitSchema.swift -This file provides `LocusKitSchema`, the entire on-disk schema declared -as data rather than as SQL text. `Storage.open(schema:)`, a -PersistenceKit primitive, reads this declaration and creates every -table, generated column, and index from it. No file in LocusKit issues -raw `CREATE TABLE` or `CREATE INDEX` strings; the previous -hand-rolled-SQL implementation was replaced by this declarative -approach so that the schema is portable across SQLite, PostgreSQL, and -an in-memory test backend without three copies of the same DDL. - -`LocusKitSchema.schema` assembles seventeen tables: `drawers`, -`tunnels`, `diary`, `manifest`, `kg_facts`, `proposals`, `associations`, -`learnedReferences`, `source_catalog`, `node_bundles`, -`container_fingerprints`, `recall_trace`, `keys`, `nodes`, plus three -tables owned by sibling schema fragments (`ErasureLedgerSchema`, -`SnapshotSchema`). Every persistent noun table carries a nullable -`.json` column named `ext` — a forward-compatible extension slot that -absorbs an unforeseen future field without a schema migration; version -1.0 writes `NULL` into it and never reads it. `LocusKitSchema.version` -is currently 8; the file's header comment documents each version bump -(a BLOB Merkle root, a nullable `content_hash` column, the `nodes` -table, and so on) and states plainly that there is no incremental -migration ladder, because no estate data has shipped under an earlier -version. - -The file's most distinctive feature is its **generated columns** — for -example `g_state_cluster` on `drawers`, computed as +This file provides `LocusKitSchema`, the entire on-disk schema. The +schema is declared as data, not as SQL text. `Storage.open(schema:)`, +a PersistenceKit primitive, reads this declaration. It creates every +table, generated column, and index from it. No file in LocusKit +issues a raw `CREATE TABLE` or `CREATE INDEX` string. A hand-rolled +SQL implementation used to do this. This declarative approach +replaced it. So the schema is portable across SQLite, PostgreSQL, and +an in-memory test backend. It needs no three copies of the same DDL. + +`LocusKitSchema.schema` assembles seventeen tables. Fourteen of them +are: `drawers`, `tunnels`, `diary`, `manifest`, `kg_facts`, +`proposals`, `associations`, `learnedReferences`, `source_catalog`, +`node_bundles`, `container_fingerprints`, `recall_trace`, `keys`, and +`nodes`. Three more tables come from two sibling schema fragments: +`ErasureLedgerSchema` and `SnapshotSchema`. Every persistent noun +table carries a nullable `.json` column named `ext`. This column is a +forward-compatible extension slot. It absorbs an unforeseen future +field, without a schema migration. Version 1.0 writes `NULL` into it +and never reads it. `LocusKitSchema.version` is currently 8. The +file's header comment documents each version bump. Bumps include a +BLOB Merkle root, a nullable `content_hash` column, and the `nodes` +table, among others. The comment states plainly that there is no +incremental migration ladder. No estate data has shipped under an +earlier version. + +The file's most distinctive feature is its generated columns. +`g_state_cluster` on `drawers` is one example. It is computed as `adjectiveBitmap & 0x3F`. A generated column is a column whose value -SQLite (or the equivalent backend) derives automatically from an -expression over other columns in the same row, so it can be indexed -like an ordinary column without the application computing and storing -it by hand. `LocusKitSchema.indices` then declares an ordinary index on -each generated column, which turns what used to be a hand-written -`CREATE INDEX ... ON drawers (provenance & 0xF)` functional index into -a declarative, backend-portable statement. The file's header comment -also documents a **bitmap reservation map**: which bit ranges of each -Int64 bitmap column are assigned and which are free headroom, so -adding a future flag is a matter of claiming a documented free bit -rather than a schema migration. +the backend derives on its own. The backend computes it from an +expression over other columns in the same row. So a generated column +can be indexed like an ordinary column, without the application +computing and storing it by hand. `LocusKitSchema.indices` then +declares an ordinary index on each generated column. This turns a +hand-written functional index into a declarative, backend-portable +statement. The old form looked like `CREATE INDEX ... ON drawers +(provenance & 0xF)`. The file's header comment also documents a +bitmap reservation map. This map shows which bit ranges of each Int64 +bitmap column are assigned, and which are free headroom. So adding a +future flag means claiming a documented free bit, not running a +schema migration. ## LocusKitVocabulary.swift -This file provides `LocusKitVocabulary`, LocusKit's contribution to the -write-gate's vocabulary — the set of legal values every bitmap field -may take. A vocabulary is the fixed rulebook the write gate checks -every proposed value against; a value outside the vocabulary is -refused before it ever reaches storage. +This file provides `LocusKitVocabulary`, LocusKit's contribution to +the write gate's vocabulary. A vocabulary here is the set of legal +values a bitmap field may take. It is the fixed rulebook the write +gate checks every proposed value against. A value outside the +vocabulary is refused before it ever reaches storage. `LocusKitVocabulary.unionSlots` declares one `FieldSlot` per -operational and provenance field LocusKit defines (capture channel, -content kind, feature flags, source type, and so on), each naming its -bitmap column, bit position, width, and (for enumerated fields) its -legal raw values. The adjective axis's vocabulary — state, sensitivity, -exportability, trust — is supplied by the write gate itself, from -SubstrateLib, not by this file; `unionSlots` is purely LocusKit's own -consumer-side additions. `LocusKitVocabulary.frozen()` compiles this -set into a `Vocabulary` value once, at estate open, via -`VocabularyValidator.freeze`; the resulting frozen vocabulary is what -every subsequent write in the estate's lifetime is checked against, and -it never changes after that first freeze. +operational and provenance field LocusKit defines. These fields +include capture channel, content kind, feature flags, and source +type. Each slot names its bitmap column, bit position, and width. An +enumerated field's slot also lists its legal raw values. The +adjective axis's vocabulary covers state, sensitivity, exportability, +and trust. The write gate itself supplies that vocabulary, from +SubstrateLib, not this file. `unionSlots` is purely LocusKit's own +consumer-side addition. `LocusKitVocabulary.frozen()` compiles this +set into a `Vocabulary` value once, at estate open, through +`VocabularyValidator.freeze`. Every later write in the estate's +lifetime is checked against this frozen vocabulary. It never changes +after that first freeze. ## EstateTypes.swift -This file provides five small supporting types used across the estate -surface: `RowID` (a plain `String` type alias naming a row's stable -identifier), `FrameFilteredDrawers` (the result of a frame-aware -by-id load, pairing the frame-admissible drawers with the full set of -ids that physically loaded), `OwnerCredentials` (the estate owner's -identifier, validated non-empty at open/create time), `LatticeAnchor` -(the four-field classification anchor every drawer carries — a UDC -code plus optional Wikidata enrichment), and `EstateError` (the error -enum for `Estate` lifecycle failures, distinct from `LocusKitError`). - -`FrameFilteredDrawers.loadedIDs` is reported independently of -`admissible` for a specific reason: a caller needs to distinguish "this -id loaded but failed the frame's filter" (a legitimate drop) from "this -id never loaded" (a transient or partial read that must be treated as -degraded, not silently dropped). `LatticeAnchor.udc(_:)` is a -convenience constructor for the common case of a bare UDC code with no -Wikidata enrichment — the shape most content has before an enrichment -daemon has run over it. `EstateError` covers three failure shapes: -`substrateUnavailable` (the backing storage failed to open), -`manifestMismatch` (a stored manifest value does not match what this -build of the kit expects — most importantly the bitmap layout -version), and `emptyOwnerIdentifier` (raised before any storage call, -so a caller gets a structurally distinct error rather than a generic -one), plus `keychainError` for a failing Keychain call during identity -key handling. +This file provides five small supporting types used across the +estate surface. `RowID` is a plain `String` type alias. It names a +row's stable identifier. `FrameFilteredDrawers` is the result of a +frame-aware by-id load. It pairs the frame-admissible drawers with +the full set of ids that physically loaded. `OwnerCredentials` is the +estate owner's identifier. It is validated non-empty at open or +create time. `LatticeAnchor` is the four-field classification anchor +every drawer carries. It holds a UDC code, plus optional Wikidata +enrichment. `EstateError` is the error enum for `Estate` lifecycle +failures, distinct from `LocusKitError`. + +`FrameFilteredDrawers.loadedIDs` is reported apart from +`admissible`, for a specific reason. A caller needs to tell two cases +apart. The first case is an id that loaded but failed the frame's +filter, a legitimate drop. The second case is an id that never +loaded, a transient or partial read that must be treated as degraded, +not silently dropped. `LatticeAnchor.udc(_:)` is a convenience +constructor. It handles the common case of a bare UDC code with no +Wikidata enrichment, the shape most content has before an enrichment +daemon has run over it. `EstateError` covers three failure shapes. +`substrateUnavailable` means the backing storage failed to open. +`manifestMismatch` means a stored manifest value does not match what +this build expects. Most importantly, the bitmap layout version must +match. `emptyOwnerIdentifier` is raised before any storage call. So a +caller gets a structurally distinct error, rather than a generic one. +`keychainError` covers a failing Keychain call during identity key +handling. ## Estate.swift This file provides `Estate`, the single public entry point to a -LocusKit-backed memory store. An `Estate` is an actor — a Swift type -that serializes access to its own state, so concurrent callers can -never interleave two operations against the same estate unsafely. It -owns one `DrawerStore` (the storage engine, described below), one -`ContainerFingerprintStore` (the recall-pruning aggregates), and one -`NodeStore` (the containment tree), and it is the only public surface -most callers should ever touch. +LocusKit-backed memory store. An `Estate` is an actor. An actor is a +Swift type that serializes access to its own state. So concurrent +callers can never interleave two operations against the same estate +unsafely. `Estate` owns one `DrawerStore`, the storage engine +described below. It owns one `ContainerFingerprintStore`, the +recall-pruning totals. It owns one `NodeStore`, the containment +tree. It is the only public surface most callers should ever touch. `Estate.open(storage:owner:identityKeyStore:)` opens an existing estate. It refuses to open a database whose manifest carries a -different `bitmap_layout_version` than this build expects — bitmap bit -positions are part of the durable on-disk contract, so opening a +different `bitmap_layout_version` than this build expects. Bitmap bit +positions are part of the durable on-disk contract. So opening a database written by an incompatible future schema would silently -misread every bitmap field. On first open it also mints a fresh -Curve25519 Ed25519 keypair for the estate's federation identity: the -public half is written to the manifest (safe, because a public key has -no confidentiality requirement), and the private half is handed to an -injected `EstateIdentityKeyStore` — the Keychain in production, an -in-memory store in tests — and cached for the lifetime of the `Estate` -instance. `Estate.create(storage:owner:manifest:)` is the sibling -constructor for a brand-new estate; it does not mint the identity -keypair, because that only happens on the first `open` after creation. - -The bulk of the file is a set of typed pass-through reads — -`allDrawers`, `getDrawers(ids:)`, `getDrawers(ids:matchingFrame:hydrationLevel:)`, -`hydrateBodies`, `tunnelsFromWing`, `recentRecallTraces`, -`allTunnels`/`allActiveTunnels`, `retireTunnel`/`unretireTunnel`, -`pruneRecallTraces`, `markRecallTracesUsed`, `countRecallTraces`, -`countDrawerRows`, `allProposals`/`allAssociations`/ -`allLearnedReferences`/`allKGFacts`/`allDiaryEntries`, and -`resolveNodeNames` — that exist so higher kits (chiefly -GeniusLocusKit) never need to import or construct a `DrawerStore` -directly. `getDrawers(ids:matchingFrame:hydrationLevel:)` is the most -elaborate of these: it runs the exact same frame-filtering pipeline -`recall` uses, but over a caller-supplied id set rather than a full -corpus scan, which is the shape a dense-first candidate-pool search -needs. `meta(key:)` and `setMeta(key:value:)` expose the estate -manifest as a general per-estate key-value store for higher kits that -need durable state of their own, on the condition that they namespace -their keys to avoid colliding with LocusKit's own typed manifest keys. +misread every bitmap field. On first open, it also mints a fresh +Curve25519 Ed25519 keypair for the estate's federation identity. The +public half is written to the manifest, safe because a public key has +no confidentiality requirement. The private half is handed to an +injected `EstateIdentityKeyStore`, the Keychain in production, an +in-memory store in tests. It is cached for the lifetime of the +`Estate` instance. `Estate.create(storage:owner:manifest:)` is the +sibling constructor for a brand-new estate. It does not mint the +identity keypair, because that only happens on the first `open` after +creation. + +The bulk of the file is a set of typed pass-through reads. These +exist so higher kits, chiefly GeniusLocusKit, never need to import or +construct a `DrawerStore`. They include `allDrawers`, +`getDrawers(ids:)`, `hydrateBodies`, `tunnelsFromWing`, +`recentRecallTraces`, `allTunnels`, `retireTunnel`, +`pruneRecallTraces`, `countDrawerRows`, `allProposals`, +`allAssociations`, `allLearnedReferences`, `allKGFacts`, +`allDiaryEntries`, and `resolveNodeNames`. +`getDrawers(ids:matchingFrame:hydrationLevel:)` is the most elaborate +of these. It runs the exact same frame-filtering pipeline `recall` +uses, but over a caller-supplied id set rather than a full corpus +scan. This is the shape a dense-first candidate-pool search needs. +`meta(key:)` and `setMeta(key:value:)` expose the estate manifest as +a general per-estate key-value store. Higher kits can use it for +durable state of their own, on the condition that they namespace +their keys to avoid colliding with LocusKit's own typed manifest +keys. ## EstateAudit.swift -This file provides `Estate`'s audit and history API: `auditTrail(rowID:)` -and `bitmapState(rowID:asOf:)`. Both delegate to the row's sealed audit -log rather than any separate history table, because the audit log — -not the live row — is the estate's source of truth for what happened -and when. +This file provides `Estate`'s audit and history API: +`auditTrail(rowID:)` and `bitmapState(rowID:asOf:)`. Both delegate to +the row's sealed audit log, rather than to any separate history +table. The audit log, not the live row, is the estate's source of +truth for what happened and when. `auditTrail(rowID:)` returns every sealed `AuditEvent` for a row in -HLC order (an HLC, or hybrid logical clock, is a timestamp format that -orders events consistently even across independently-clocked -machines). `bitmapState(rowID:asOf:)` reconstructs what a row's three -bitmaps looked like at an earlier point in time by folding its audit -log forward from the beginning up to (and including) the requested HLC, -via SubstrateLib's `AuditLogFold.projectStateAt`. This is what lets a -caller ask "what did this drawer's state look like a week ago" without -LocusKit maintaining a separate append-only history table alongside the -live row — the live row is redundant with the log by construction, and -the log is the only place that needs to remember the past. +HLC order. An HLC, or hybrid logical clock, is a timestamp format. It +puts events in one order, even across machines with their own +clocks. `bitmapState(rowID:asOf:)` rebuilds what a row's three +bitmaps looked like at an earlier point in time. It does this by +folding the row's audit log forward, from the beginning up to and +including the requested HLC. This folding runs through SubstrateLib's +`AuditLogFold.projectStateAt`. This is what lets a caller ask what a +drawer's state looked like a week ago. LocusKit needs no separate +append-only history table alongside the live row for this. The live +row is redundant with the log by construction. The log is the only +place that needs to remember the past. ## EstateIdentityKeyStore.swift -This file provides the protocol and two implementations for persisting -an estate's Ed25519 private signing key outside the manifest table. -The manifest table is ordinary, unencrypted metadata readable by -anyone with database or backup access, so the private half of the -estate's federation identity keypair must never be written there — -only the public half is safe to store in the manifest. - -`EstateIdentityKeyStore` declares two methods, `loadPrivateKey` and -`storePrivateKey`, both keyed by the estate's UUID. -`KeychainEstateIdentityKeyStore` is the production implementation: it -stores the raw key bytes as a `kSecClassGenericPassword` Keychain item, -accessible after the first device unlock following a restart -(`kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly`) but never synced -to iCloud Keychain — matching the same device-bound posture as the -estate's own SQLite file. `InMemoryEstateIdentityKeyStore` is the test -double: a plain dictionary guarded by a lock, with a `_storedPrivateKey` -accessor so a test can assert what was persisted without touching the -real Keychain (which would require entitlements and would pollute -state across test runs). +This file provides the protocol and two implementations for +persisting an estate's Ed25519 private signing key outside the +manifest table. The manifest table is ordinary, unencrypted metadata. +Anyone with database or backup access can read it. So the private +half of the estate's federation identity keypair must never be +written there. Only the public half is safe to store in the +manifest. + +`EstateIdentityKeyStore` declares two methods: `loadPrivateKey` and +`storePrivateKey`. Both are keyed by the estate's UUID. +`KeychainEstateIdentityKeyStore` is the production implementation. It +stores the raw key bytes as a `kSecClassGenericPassword` Keychain +item. The item stays accessible after the first device unlock +following a restart. It is never synced to iCloud Keychain. This +matches the same device-bound posture as the estate's own SQLite +file. `InMemoryEstateIdentityKeyStore` is the test double. It is a +plain dictionary guarded by a lock. It carries a `_storedPrivateKey` +accessor. A test can use this accessor to check what was persisted, +without touching the real Keychain. The real Keychain would require +entitlements. It would also pollute state across test runs. ## EstateVerbs.swift -This file provides the nine verb methods — `capture`, `captureBatch`, -`recall`, `withdraw`, `expunge`, `mutate`, `reanchor`, `propose`, -`learn`, `associate` — as an extension on `Estate`. It is declared as a -separate file specifically so it can reach `Estate.store`, which is -declared `internal` rather than `private` in `Estate.swift` for exactly -this reason. - -`capture(_:CaptureFrame)` is the drawer entry point. It validates that -content, room, lattice-anchor UDC code, actor, and embedding-model id -are all non-empty, assembles the three bitmap columns from the frame's -named fields via `BitField.writeField` (never hand-rolled shift/mask -arithmetic), resolves the target wing and room to containment-tree node -ids (creating them on demand), and writes the drawer through -`addDrawerCovered` — the one sanctioned internal chokepoint that -bundles the row insert with the container-fingerprint update, so a -drawer can never be captured without its fingerprint aggregate being -updated in the same call. `captureBatch(_:)` is the bulk-import -sibling: it resolves all wing/room node ids up front with a per-call -cache, batches every fresh (non-superseding) insert into one storage -transaction via `DrawerStore.insertFreshBatch`, and defers the Merkle -rollup entirely — turning a 40,000-drawer import from roughly 34 -minutes of per-row commits into roughly 30 seconds. - -`capture(_:TunnelCaptureFrame)` is the tunnel entry point — `capture` +This file provides the nine verb methods as an extension on `Estate`: +`capture`, `captureBatch`, `recall`, `withdraw`, `expunge`, `mutate`, +`reanchor`, `propose`, `learn`, and `associate`. It is declared as a +separate file for a specific reason. It needs to reach `Estate.store`, +which is declared `internal` rather than `private` in `Estate.swift` +for exactly this reason. + +`capture(_:CaptureFrame)` is the drawer entry point. It validates +that content and room are both non-empty. It also validates the +lattice-anchor UDC code, the actor, and the embedding-model id. It +assembles the three bitmap +columns from the frame's named fields, through `BitField.writeField`, +never hand-rolled shift-and-mask arithmetic. It resolves the target +wing and room to containment-tree node ids, creating them on demand. +It writes the drawer through `addDrawerCovered`, the one sanctioned +internal chokepoint. That chokepoint bundles the row insert with the +container-fingerprint update. So a drawer can never be captured +without its fingerprint total being updated in the same call. +`captureBatch(_:)` is the bulk-import sibling. It resolves all wing +and room node ids up front, with a per-call cache. It batches every +fresh, non-superseding insert into one storage transaction, through +`DrawerStore.insertFreshBatch`. It defers the Merkle rollup entirely. +This turns a 40,000-drawer import from roughly 34 minutes of per-row +commits into roughly 30 seconds. + +`capture(_:TunnelCaptureFrame)` is the tunnel entry point. `capture` is legal on exactly two nouns, drawer and tunnel, and this overload -handles the second. `recall(_:RecallFrame)` is the read path: it reads -the live (non-tombstoned) candidate rows via `liveRows`, runs them -through `BitmapEvaluator.evaluate`, optionally writes a bounded number -of recall-trace rows (only when the caller opts in via -`frame.traceLimit`, so internal and bulk-export scans never accumulate -trace rows), and returns a `RecallStream`. Both `liveRows` and the -evaluation step surface any internal read failure as a **named -degraded stage** on the returned stream rather than silently returning -an empty result — this is the mechanism by which a caller can tell a -genuinely empty estate from a recall that failed partway through. - -`withdraw(rowID:reason:)` moves a drawer's state to `.withdrawn` via -`mutateState`, which is the only correct path because a state +handles the second. `recall(_:RecallFrame)` is the read path. It +reads the live, non-tombstoned, candidate rows through `liveRows`. It +runs them through `BitmapEvaluator.evaluate`. It optionally writes a +bounded number of recall-trace rows, only when the caller opts in +through `frame.traceLimit`. So internal and bulk-export scans never +accumulate trace rows. `recall` returns a `RecallStream`. Both +`liveRows` and the evaluation step surface any internal read failure +as a named degraded stage on the returned stream, rather than +silently returning an empty result. This is the mechanism by which a +caller can tell a genuinely empty estate from a recall that failed +partway through. + +`withdraw(rowID:reason:)` moves a drawer's state to `.withdrawn`, +through `mutateState`. This is the only correct path, because a state transition must go through the automaton's legality check. -`expunge(rowID:reason:confirmation:now:)` is the destructive delete: -it requires an explicit `confirmation: true`, walks the drawer's full -lineage chain (every version sharing its `lineageID`) so a hard delete -scrubs every historical version, and always seals a sealed audit event -recording the fact of the expunge even though the content itself is -gone. `expungeReturningUnsealedEvent` and the paired -`sealExpungeAudit`/`sealExpungeOrphanAudit` methods exist for -GeniusLocusKit's two-step orchestration, where a cross-kit vector -delete must happen between the storage tombstone and the audit seal; -splitting the seal out prevents an arbitrary caller from suppressing -the audit event by accident. +`expunge(rowID:reason:confirmation:now:)` is the destructive delete. +It requires an explicit `confirmation: true`. It walks the drawer's +full lineage chain, every version sharing its `lineageID`, so a hard +delete scrubs every historical version. It always seals a sealed +audit event recording the fact of the expunge, even though the +content itself is gone. `expungeReturningUnsealedEvent` and the +paired `sealExpungeAudit` and `sealExpungeOrphanAudit` methods exist +for GeniusLocusKit's two-step orchestration. There, a cross-kit +vector delete must happen between the storage tombstone and the +audit seal. Splitting the seal out prevents any caller from +suppressing the audit event by accident. `mutate(rowID:kind:payload:)` dispatches on `MutationKind` to move a -drawer along one of three axes: confirmation (via `mutateProvenance`), -state (via `mutateState`, with the caller-facing guards for `.resolve`, -`.accept`, and `.revive` implemented here before the store's automaton -check even runs, so the error message is clearer), or an adjective -field (sensitivity, trust, or exportability, via `mutateAdjective`). -The `.revive` case is the most elaborate: it implements the full -per-source-state legality table from the cookbook's revive rule, -including the "living successor" lineage check that refuses to revive -a superseded row while a later version in the same lineage is still -alive. `reanchor(rowID:toRoom:toWing:toLattice:)` moves a drawer's room, -wing, and/or lattice position without touching its bitmaps. +drawer along one of three axes. The first axis is confirmation, +through `mutateProvenance`. The second is state, through +`mutateState`. The caller-facing guards for `.resolve`, `.accept`, +and `.revive` are implemented here. This happens before the store's +automaton check even runs. So the error message is clearer. The third axis is +an adjective field, sensitivity, trust, or exportability, through +`mutateAdjective`. The `.revive` case is the most elaborate. It +implements the full per-source-state legality table from the +cookbook's revive rule. This includes the living-successor lineage +check. That check refuses to revive a superseded row while a later +version in the same lineage is still alive. +`reanchor(rowID:toRoom:toWing:toLattice:)` moves a drawer's room, +wing, and lattice position, without touching its bitmaps. `propose(_:now:)` and `associate(_:now:)` build a `Proposal` or -`Association` from their respective frames, deriving the lattice anchor -from the target row (for propose) or from the first endpoint (for -associate) rather than ever fabricating one. `learn(_:now:)` catalogs -the frame's source (once, keyed by handle) and writes a +`Association` from their respective frames. Each derives its lattice +anchor from a specific source. For propose, the anchor comes from +the target row. For associate, it comes from the first endpoint. +Neither ever fabricates an anchor. `learn(_:now:)` +catalogs the frame's source, once, keyed by handle. It writes a `LearnedReference` anchored to that catalog entry's genuine lattice -position — never a sentinel anchor invented from a bare handle. +position, never a sentinel anchor invented from a bare handle. `seedWing(_:hint:addedBy:embeddingModelID:now:)` files one ordinary, -fully recallable drawer per default wing at estate provision time, -giving a fresh agent a plain-language description of what each wing is -for. +fully recallable drawer per default wing, at estate provision time. +This gives a fresh agent a plain-language description of what each +wing is for. ## Frames.swift -This file provides the eight "frame" structs that carry a verb's -arguments: `CaptureFrame`, `TunnelCaptureFrame`, `RecallFrame`, -`LearnFrame`, `ProposeFrame`, `AssociateFrame`, plus the supporting -enums `MutationKind`, `HydrationLevel`, and `Ordering`. Every field on -every frame is named after a domain concern (a capture channel, a -sensitivity tier, a filter chain) — no raw bitmap value or bit position +This file provides the frame structs that carry a verb's arguments: +`CaptureFrame`, `TunnelCaptureFrame`, `RecallFrame`, `LearnFrame`, +`ProposeFrame`, and `AssociateFrame`. It also provides three +supporting enums: `MutationKind`, `HydrationLevel`, and `Ordering`. +Every field on +every frame is named after a domain concern, a capture channel, a +sensitivity tier, a filter chain. No raw bitmap value or bit position crosses the public verb boundary anywhere in this file. `CaptureFrame` carries every named axis a captured drawer's three bitmaps encode: channel, sensitivity, kind, and the matching provenance-side channel, source type, sensitivity, confirmation, and -confidence, plus lineage, room, lattice anchor, actor, embedding model, -optional event time (for backdated bulk ingestion), feature flags, -exportability, and an optional wing. Nearly every field defaults to the -value that produces the same all-zero bitmap a caller got before that -field existed, so adding a new capture-time axis is source-compatible -with every existing caller. `RecallFrame` carries the filter chain, -hydration level, page-size limit, ordering, an optional historical -`asOf` HLC, and `traceLimit` (the opt-in recall-trace write count). -`MutationKind` enumerates the mutate verb's seven cases — `.confirm`, -`.reject`, `.contest`, `.resolve`, `.supersede`, `.revive`, `.accept`, -and the three `.correctSensitivity`/`.correctTrust`/`.correctExportability` -cases carrying their new value. `HydrationLevel` names the three -read-cost tiers — `.bitmapOnly`, `.structured`, `.full` — and -`Ordering` names the three result orderings recall supports; a -relevance ordering is deliberately absent, because LocusKit has no -scoring signal of its own (that lives in VectorKit, composed on top by -GeniusLocusKit), and the file's comment explains that shipping a -`.byRelevanceDesc` case here without a real relevance signal would be -an honesty violation. +confidence. It also carries lineage, room, lattice anchor, actor, and +embedding model. It carries an optional event time too, for backdated +bulk ingestion. It also carries feature flags, exportability, and an +optional wing. Nearly every field defaults to the value that produced +the same all-zero bitmap a caller got before that field existed. So +adding a new capture-time axis stays source-compatible with every +existing caller. `RecallFrame` carries the filter chain, hydration +level, page-size limit, and ordering. It also carries an optional +past `asOf` HLC and `traceLimit`. +`traceLimit` is the opt-in recall-trace write count. + +`MutationKind` enumerates the mutate verb's seven cases: `.confirm`, +`.reject`, `.contest`, `.resolve`, `.supersede`, `.revive`, and +`.accept`. Three more cases carry a new value: `.correctSensitivity`, +`.correctTrust`, and `.correctExportability`. `HydrationLevel` names +the three read-cost tiers: `.bitmapOnly`, `.structured`, and `.full`. +`Ordering` names the three result orderings recall supports. A +relevance ordering is absent on purpose. LocusKit has no scoring +signal of its own. That signal lives in VectorKit, composed on top by +GeniusLocusKit. The file's comment explains that shipping a +`.byRelevanceDesc` case here, without a real relevance signal, would +be an honesty violation. ## DrawerStore.swift This file provides `DrawerStore`, the actor that owns every table in -the schema and implements the full CRUD and mutation surface for all -nine nouns. It is the largest file in the package and the place where -almost every write in the estate ultimately lands. - -`DrawerStore.init(storage:hlc:)` opens the schema (idempotently — a -second open on an existing database is a no-op for tables and -indices), freezes the write-gate vocabulary once, populates the v1 -manifest defaults (writing each key only if absent, so a first-open -value is never overwritten by a later open), and classifies the -manifest's `estate_uuid` value into one of three outcomes: absent -(a genuinely fresh estate, resolved to a freshly minted UUID), present -and well-formed (the normal case), or present but unparseable — which -throws `corruptStoredValue` rather than silently fabricating a random -replacement UUID, because conflating corruption with a fresh estate -would mask real data loss. The HLC clock's node id is derived from that -same classified value by hashing the raw stored text, so a standalone -estate gets a stable, estate-specific clock identity that a Rust port -of the same estate would derive identically. - -The **gated write path** is the file's central pattern, used by -`addDrawer`, `mutateProvenance`, `mutateAdjective`, `mutateOperational`, -`mutateState`, `expungeGated`, and `reanchorGated`. Each reads the -row's current bitmaps inside a serializable transaction, decomposes the -proposed new value into per-field `FieldWrite`s (never a whole-column -replacement — this is what lets the gate validate each field -independently and check the forbidden-combination invariant on the -merged result), calls `AuditGate.admit` (a SubstrateLib primitive), -and — on success — writes both the merged projection column and the -sealed `AuditEvent` in the same transaction. `gatedColumnWrite` is the -shared helper behind the three whole-column mutators; it excludes the -adjective bitmap's state field from the field list, because state is -verb-driven and must never move through a field edit. `mutateState` -additionally validates that the verb (`RowVerb`) and the caller's -requested target state agree with what the automaton's transition -table produces, so an illegal `(from, via, to)` triple is rejected -before any row is touched. - -`addDrawer` implements the **supersession cascade**: when the new -drawer's `lineageID` matches an active predecessor -(`findActivePredecessor`, an indexed query on the generated state-cluster -column), `addDrawerWithCascade` inserts the successor and a -`.supersedes` tunnel in one transaction (so a failed tunnel insert -never leaves an orphaned successor row), then separately flips the -predecessor's state to `.superseded` through the normal gated +the schema. It implements the full CRUD and mutation surface for all +nine nouns. It is the largest file in the package. Almost every write +in the estate lands here in the end. + +`DrawerStore.init(storage:hlc:)` opens the schema. This is +idempotent: a second open on an existing database is a no-op for +tables and indices. It freezes the write-gate vocabulary once. It +populates the v1 manifest defaults, writing each key only if absent, +so a first-open value is never overwritten by a later open. It +classifies the manifest's `estate_uuid` value into one of three +outcomes. The first outcome is absent: a genuinely fresh estate, +resolved to a freshly minted UUID. The second outcome is present and +well-formed, the normal case. The third outcome is present but +unparseable. This third outcome throws `corruptStoredValue`, rather +than silently fabricating a random replacement UUID, because +conflating corruption with a fresh estate would mask real data loss. +The HLC clock's node id is derived from that same classified value, +by hashing the raw stored text. So a standalone estate gets a stable, +estate-specific clock identity. A Rust port of the same estate +derives that identity identically. + +The gated write path is the file's central pattern. It is used by +`addDrawer`, `mutateProvenance`, `mutateAdjective`, +`mutateOperational`, `mutateState`, `expungeGated`, and +`reanchorGated`. Each of these reads the row's current bitmaps inside +a serializable transaction. Each decomposes the proposed new value +into per-field `FieldWrite`s, never a whole-column replacement. This +is what lets the gate validate each field on its own. It also lets +the gate check the forbidden-combination invariant on the merged +result. Each calls `AuditGate.admit`, a SubstrateLib primitive. On +success, each writes both the merged projection column and the +sealed `AuditEvent`, in the same transaction. `gatedColumnWrite` is +the shared helper behind the three whole-column mutators. It excludes +the adjective bitmap's state field from the field list, because +state is verb-driven and must never move through a field edit. +`mutateState` also checks the verb against the caller's requested +target state. Both must agree with what the automaton's transition +table produces. The verb itself is a `RowVerb` value. So an illegal +from-via-to triple is rejected before any row is touched. + +`addDrawer` implements the supersession cascade. Sometimes the new +drawer's `lineageID` matches an active predecessor. +`findActivePredecessor` finds this match, through an indexed query on +the generated state-cluster column. When a match exists, +`addDrawerWithCascade` inserts the successor and a `.supersedes` +tunnel in one transaction. So a failed tunnel insert never leaves an +orphaned successor row. `addDrawerWithCascade` then separately flips +the predecessor's state to `.superseded`, through the normal gated `mutateState` path. `insertFreshBatch` is the bulk-import sibling used -by `EstateVerbs.captureBatch`: every drawer in the batch (all of which -the caller has pre-verified have no active predecessor) is inserted and -gated inside one single transaction, eliminating per-row commit -overhead. - -`expungeGated` is the lineage-wide hard delete: it tombstones the -target drawer through the gate, zeroes its content, sets the -`dreaming_recalc_required` obligation flag, records it in the erasure -ledger, and then repeats a content scrub (and, where the state -transition is legal, a full gated tombstone) across every other member -of the lineage chain. A sibling whose state cannot legally transition -to tombstoned (an audit-grade `accepted` row, forbidden by invariant -S-3) still has its content unconditionally zeroed — the destruction -contract is stronger than the state machine, because leaving verbatim -content behind when the gate refuses the state flip would be a privacy -violation. `sealExpungeAudit`, `sealExpungeOrphanAudit`, and -`sealExpungeOrphanForSweep` support GeniusLocusKit's two-step orphan -recovery when a crash separates the storage tombstone from its audit -seal; `tombstonedRowsWithoutExpungeAudit` is the query the recovery -sweep uses to find rows stuck in that crash window. - -The remainder of the file is CRUD for the other eight nouns -(`addTunnel`/`getTunnel`/`tunnelsFrom`/`tunnelsTo`/`allTunnels`, the -tunnel-retirement pair `retireTunnel`/`unretireTunnel`, the outline -helpers `outlineChildren`/`outlineAncestors`/`reparentDrawer`, -`addKGFact`/`getKGFact`/`kgFacts(forDrawerID:)`/`withdrawKGFact`, -`addProposal`/`getProposal`/`proposals(forTargetRowID:)`, -`addAssociation`/`getAssociation`/`associationsFrom`/`associationsTo`, -`addLearnedReference`/`getLearnedReference`/`learnedReferences(forSourceCatalogID:)`, -`addSourceCatalogEntry`/`getSourceCatalogEntry`/`sourceCatalogEntry(forHandle:)`, -`addDiaryEntry`/`getDiaryEntry`/`readDiary`), the recall-trace surface -(`insertRecallTrace`/`insertRecallTraces`/`recentRecallTraces`/ -`markRecallTraceUsed`/`markRecallTracesUsed`/`pruneRecallTraces`/ -`countRecallTraces`), the manifest surface (`setMeta`/`getMeta`/ -`readManifest`), the node-tree name-resolution helpers -(`roomNodeIdsInWing`/`roomNodeId`/`resolveNodeNames`), the summary -surface (`listWings`/`listRooms`/`taxonomy`), and the temporal-read -surface (`fingerprintsCaptured(in:)`/`fingerprintBitSeries`, which -feed the FFT-based rhythm-spectrum and moment-summary lenses built on -top of LocusKit). Every row-decode function follows the same -resilience pattern: `drawerFromRow` and its siblings throw -`corruptStoredValue` for an unambiguously malformed stored value (a -non-UUID `lineageID`, an unparseable timestamp), while corpus-scan -callers route through `decodeDrawerRowsSkipCorrupt`, which logs and -skips a corrupt row rather than aborting an entire estate-wide scan — -point lookups fail loud, corpus scans degrade gracefully. +by `EstateVerbs.captureBatch`. Every drawer in the batch already has +no active predecessor. The caller pre-verifies this before the batch +runs. Each drawer is inserted and gated inside one single +transaction. This eliminates per-row commit overhead. + +`expungeGated` is the lineage-wide hard delete. It tombstones the +target drawer through the gate. It zeroes the drawer's content. It +sets the `dreaming_recalc_required` obligation flag. It records the +event in the erasure ledger. It then repeats a content scrub across +every other member of the lineage chain. Where the state transition +is legal, it also repeats a full gated tombstone. A sibling whose +state cannot legally transition to tombstoned, an audit-grade +`accepted` row forbidden by invariant S-3, still has its content +zeroed no matter what. The destruction contract is stronger than the +state machine. Leaving verbatim content behind, when the gate refuses +the state flip, would be a privacy violation. `sealExpungeAudit`, +`sealExpungeOrphanAudit`, and `sealExpungeOrphanForSweep` support +GeniusLocusKit's two-step orphan recovery. This recovery matters when +a crash separates the storage tombstone from its audit seal. +`tombstonedRowsWithoutExpungeAudit` is the query the recovery sweep +uses to find rows stuck in that crash window. + +The remainder of the file provides CRUD for the other eight nouns. +Tunnel CRUD includes `addTunnel`, `getTunnel`, `tunnelsFrom`, +`tunnelsTo`, and `allTunnels`. It also includes the tunnel-retirement +pair `retireTunnel` and `unretireTunnel`. It includes the outline +helpers `outlineChildren`, `outlineAncestors`, and `reparentDrawer`. +Fact CRUD +includes `addKGFact`, `getKGFact`, `kgFacts(forDrawerID:)`, and +`withdrawKGFact`. Proposal CRUD includes `addProposal`, `getProposal`, +and `proposals(forTargetRowID:)`. Association CRUD includes +`addAssociation`, `getAssociation`, `associationsFrom`, and +`associationsTo`. Learned-reference CRUD includes +`addLearnedReference`, `getLearnedReference`, and +`learnedReferences(forSourceCatalogID:)`. Source-catalog CRUD includes +`addSourceCatalogEntry`, `getSourceCatalogEntry`, and +`sourceCatalogEntry(forHandle:)`. Diary CRUD includes `addDiaryEntry`, +`getDiaryEntry`, and `readDiary`. The file also provides the +recall-trace surface: `insertRecallTrace`, `insertRecallTraces`, +`recentRecallTraces`, `markRecallTraceUsed`, `markRecallTracesUsed`, +`pruneRecallTraces`, and `countRecallTraces`. It provides the +manifest surface: `setMeta`, `getMeta`, and `readManifest`. It +provides the node-tree name-resolution helpers: `roomNodeIdsInWing`, +`roomNodeId`, and `resolveNodeNames`. It provides the summary +surface: `listWings`, `listRooms`, and `taxonomy`. It provides the +temporal-read surface: `fingerprintsCaptured(in:)` and +`fingerprintBitSeries`. These feed the FFT-based rhythm-spectrum and +moment-summary lenses built on top of LocusKit. + +Every row-decode function follows the same resilience pattern. +`drawerFromRow` and its siblings throw `corruptStoredValue` for a +clearly malformed stored value, such as a non-UUID `lineageID` +or an unparseable timestamp. Corpus-scan callers instead route +through `decodeDrawerRowsSkipCorrupt`. This function logs and skips a +corrupt row, rather than aborting an entire estate-wide scan. Point +lookups fail loud. Corpus scans degrade gracefully. ## Drawer.swift -This file provides `Drawer`, the value type behind verbatim content — -the thing callers file and the thing recall returns. Its `content` -field is preserved unchanged: no truncation, no normalization, because -MemPalace's verbatim-first principle requires that retrieval surface -exactly what was filed. +This file provides `Drawer`, the value type behind verbatim content. +A `Drawer` is the thing callers file, and the thing recall returns. +Its `content` field is preserved unchanged, with no truncation and no +normalization. MemPalace's verbatim-first principle requires that +retrieval surface exactly what was filed. Every drawer references its containing room by `parentNodeId`, a foreign key into the `nodes` table, rather than storing wing and room -names directly — display names are resolved separately via +names. Display names are resolved separately, through `DrawerStore.resolveNodeNames`. `filedAt` and `eventTime` are two -distinct clocks: `filedAt` is when the row entered the local store -(the ingest clock, monotonic, the anchor for audit ordering), while -`eventTime` is when the content actually happened or was authored in -the world — for streaming capture the two coincide, but a bulk -historical importer supplies a real, possibly much earlier, `eventTime`. -The three bitmap fields (`provenance`, `adjectiveBitmap`, -`operationalBitmap`) all default to `0`, and the four lattice fields -(`udcCode`, `udcFacets`, `wikidataQID`, `wikidataQidsSecondary`) locate -the drawer in the classification lattice — `udcCode` is required -non-empty at every capture path per invariant I-5. `Drawer`'s custom -`Codable` conformance backfills a missing `eventTime` to `filedAt` on -decode, so a row encoded before the `eventTime` column existed still -decodes to a sensible value. +distinct clocks. `filedAt` is the ingest clock. It marks when the row +entered the local store. It is monotonic, and it anchors audit +ordering. `eventTime` is when the content happened, or was +authored, in the world. For streaming capture the two clocks +coincide. A bulk backfill importer instead supplies a real, +possibly much earlier, `eventTime`. The three bitmap fields, +`provenance`, `adjectiveBitmap`, and `operationalBitmap`, all default +to zero. The four lattice fields are `udcCode`, `udcFacets`, +`wikidataQID`, and `wikidataQidsSecondary`. They locate the drawer in +the classification lattice. `udcCode` is required non-empty at every +capture path, per invariant I-5. `Drawer`'s custom `Codable` +conformance backfills a missing `eventTime` to `filedAt` on decode. So +a row encoded before the `eventTime` column existed still decodes to +a sensible value. ## DrawerFingerprint.swift This file provides the derivation of a drawer's 256-bit structural -fingerprint — the coordinate system LocusKit uses for structural -similarity and for recall pruning. A fingerprint is built from four -64-bit SimHash blocks, each a projection of one facet of the row -through its own family of random hyperplanes: bitmap (the three -bitmap columns), lattice (UDC prefix, direct Q-ID, and Q-ID -closure), lineage-and-temporal (lineage hash and capture week), and -channel-and-source (channel, source type, capture channel, sensitivity, -and an estate identity hash). +fingerprint. This fingerprint is the coordinate system LocusKit uses +for structural similarity and for recall pruning. A fingerprint is +built from four 64-bit SimHash blocks. Each block is a projection of +one facet of the row, through its own family of random hyperplanes. +The first block is bitmap: the three bitmap columns. The second is +lattice: UDC prefix, direct Q-ID, and Q-ID closure. The third is +lineage-and-temporal: lineage hash and capture week. The fourth is +channel-and-source: channel, source type, capture channel, +sensitivity, and an estate identity hash. `EstateFingerprintFamilies` derives the four hyperplane families for -one estate from its UUID, once, at estate open; two independently -started replicas of the same estate — same UUID — always derive the -identical families and therefore the identical fingerprint for -identical drawer content, which is the determinism property recall -pruning depends on. `EstateFingerprintFamilies.fingerprint(of:)` is the -per-drawer entry point: it resolves the drawer's Wikidata Q-ID (if any) -through `LatticeLib.QIDClosure` to get its full ancestor chain, folds -that chain into a 16-bit hash for the lattice block's `qidClosureHash` -slot, and hands all four assembled inputs to `SubstrateLib.SimHash`. A -drawer carrying no lineage-clustering, defer-pattern, or -stream-source data — fields that only apply to a different noun type, -`AmbientSample` — supplies the deterministic null value zero for those -sub-fields, per invariant I-17, which keeps Hamming distance -well-defined across every noun type sharing the same fingerprint shape. +one estate from its UUID, once, at estate open. Two replicas of the +same estate, started on their own, still share the same UUID. So +they always derive the same families. So they derive the same +fingerprint for the same drawer content. This determinism property +is what recall pruning depends on. `EstateFingerprintFamilies.fingerprint(of:)` +is the per-drawer entry point. It resolves the drawer's Wikidata +Q-ID, if the drawer has one, through `LatticeLib.QIDClosure`. This +gives the drawer's full ancestor chain. It folds that chain into a +16-bit hash for the lattice block's `qidClosureHash` slot. It hands +all four assembled inputs to `SubstrateLib.SimHash`. Some fields do +not apply to a drawer: lineage-clustering, defer-pattern, and +stream-source data. These fields only apply to a different noun type, +`AmbientSample`. For a drawer, they default to a null value of zero, +per invariant I-17. This default keeps Hamming distance well-defined +across every noun type that shares the same fingerprint shape. ## DrawerOperational.swift This file provides the three named axes packed into -`Drawer.operationalBitmap`: `CaptureChannel` (how the content entered -the system — typed, voiced, OCR, imported file, sensor, or actuator), -`ContentKind` (the shape of the content — prose, code, transcript, -list, structured JSON, image caption, or fingerprint-only), and -`DrawerFeatureFlags` (a non-exclusive bitset of seven named flags: -attachments, voice, image, links, pinned, keystone, and locked-zone). -Because feature flags are a bitset rather than an exclusive choice, -`DrawerFeatureFlags` is declared as an `OptionSet` rather than an enum, -unlike the other two axes in this file. - -Each axis has a computed accessor on `Drawer` — `captureChannel`, +`Drawer.operationalBitmap`. The first is `CaptureChannel`: how the +content entered the system. Its values are typed, voiced, OCR, +imported file, sensor, and actuator. The second is `ContentKind`: the +shape of the content. Its values are prose, code, transcript, list, +structured JSON, image caption, and fingerprint-only. The third is +`DrawerFeatureFlags`: a non-exclusive bitset of seven named flags. The +flags are attachments, voice, image, links, pinned, keystone, and +locked-zone. Feature flags form a bitset, not an exclusive choice. So +`DrawerFeatureFlags` is declared as an `OptionSet`. The other two axes +in this file are declared as enums instead. + +Each axis has a computed accessor on `Drawer`: `captureChannel`, `contentKind`, `featureFlags`, `hasFeatureFlag(_:)`, -`stateExtensionActive`, and `lineageClusteringActive` — that decodes -the relevant bit range and falls back to a safe neutral default (typed -input, prose content) for any raw value this build does not recognize, -so a row written by a future version with a case this build has never -heard of degrades gracefully instead of crashing. +`stateExtensionActive`, and `lineageClusteringActive`. Each accessor +decodes the relevant bit range. Each falls back to a safe neutral +default, typed input or prose content, for any raw value this build +does not recognize. So a row written by a future version, with a +case this build has never heard of, degrades gracefully instead of +crashing. ## DrawerStateValidator.swift This file provides `DrawerStateValidator`, a thin bridge from -LocusKit's `State` enum to SubstrateLib's canonical `RowStateAutomaton` -— the single implementation of the row-state legal-transition table -that every LocusKit noun with a state axis is validated against. -`DrawerStateValidator` used to carry its own parallel transition table; -that table permitted four transitions the specification actually -forbids (including `accepted → tombstoned`), so it was retired in -favor of consuming SubstrateLib directly. +LocusKit's `State` enum to SubstrateLib's canonical +`RowStateAutomaton`. This automaton is the single implementation of +the row-state legal-transition table. Every LocusKit noun with a +state axis is validated against it. `DrawerStateValidator` used to +carry its own parallel transition table. That table permitted four +transitions the specification actually forbids, including `accepted` +to `tombstoned`. So it was retired, in favor of using +SubstrateLib itself. `validate(from:to:via:)` is the transition-legality-only overload, -used by `StateTransitionTests`: it bridges `State` to `RowState`, -looks up the legal target for `(from, via)` in the automaton, and -throws `LocusKitError.disciplineViolation` naming the rule violated if -either the `(from, via)` pair has no legal transition or the caller's -requested `to` disagrees with what the automaton actually produces. +used by `StateTransitionTests`. It bridges `State` to `RowState`. It +looks up the legal target for the from-and-via pair in the automaton. +It throws `LocusKitError.disciplineViolation` when a check fails, +naming the rule violated. A check fails in either of two ways. The +from-via pair may have no legal transition. Or the caller's requested +target may disagree with what the automaton actually produces. `validate(from:to:via:targetingFields:)` is the overload -`DrawerStore.mutateState` actually calls: it additionally checks -field-level invariants (S-1: an accepted row requires trust at or -above canonical; S-2 and other bitmap-combination rules) against the -post-write bitmap fields, in the same call. +`DrawerStore.mutateState` actually calls. It also checks +field-level invariants against the post-write bitmap fields, in the +same call. One such invariant, S-1, requires that an accepted row +have trust at or above canonical. Other invariants cover further +bitmap-combination rules, such as S-2. ## Adjectives.swift -This file provides the four cross-noun adjective axes — `State`, -`Trust`, `AdjectiveSensitivity`, `AdjectiveExportability` — packed into -the low bits of every noun's `adjectiveBitmap`, plus their computed -accessors on `Drawer`. These four enums are the single source of truth -for these axes across every LocusKit consumer (GeniusLocusKit, -NeuronKit, and so on); layers beneath LocusKit cannot import them (the -dependency graph points the other way), so they carry the raw integer -encoding instead and a build-time "Guardian" tool checks that the two -sides never silently drift apart. - -`State`'s ten cases partition into three clusters at raw-value -boundaries 16 and 32, chosen specifically so the cluster of a state can -be computed as `(state >> 4) & 0x3` in one shift-and-mask: Cluster A -("currently believed" — active, pending, contested, accepted), Cluster -B ("knew past" — superseded, decayed, withdrawn, expired), and Cluster -C (terminal — rejected, tombstoned). `Drawer.isCurrentlyBelieved`, -`isKnewPast`, and `isTerminal` expose these three clusters as -predicates. `Trust` orders seven levels from `.verbatim` (unqualified, -as-filed content) through `.canonical` up to `.ambient`; it conforms to -`Comparable` so a filter like "trust at least canonical" reads as an -ordinary comparison rather than raw-value arithmetic. `AdjectiveSensitivity` -and `AdjectiveExportability` are both scale-gapped (their case raw -values skip numbers, leaving room for future intermediate tiers without -disturbing any existing bitmask). `AdjectiveSensitivity`'s three -`ADR-007`-driven predicates — `isBulkExportable`, `requiresOwnerKeyForBulk`, -`isExcludedFromBulk` — are the enforcement hooks VaultKit's bulk-export -path consults to decide whether a drawer may ride a bulk channel -without additional friction. `Drawer.dreamingRecalcRequired` and -`Drawer.sealed` decode two single-bit obligation/trust-hint flags -living above the four main axes. +This file provides the four cross-noun adjective axes: `State`, +`Trust`, `AdjectiveSensitivity`, and `AdjectiveExportability`. It also +provides their computed accessors on `Drawer`. These four enums are +the single source of truth for these axes, across every LocusKit +consumer, including GeniusLocusKit and NeuronKit. Layers beneath +LocusKit cannot import them, because the dependency graph points the +other way. So those layers carry the raw integer encoding instead. A +build-time Guardian tool checks that the two sides never silently +drift apart. + +`State` has ten cases. They partition into three clusters, at +raw-value boundaries sixteen and thirty-two. These boundaries let a +state's cluster be computed as one shift-and-mask operation. Cluster +A is "currently believed": active, pending, contested, and accepted. +Cluster B is "knew past": superseded, decayed, withdrawn, and +expired. Cluster C is terminal: rejected and tombstoned. +`Drawer.isCurrentlyBelieved`, `isKnewPast`, and `isTerminal` expose +these three clusters as predicates. + +`Trust` orders seven levels. The lowest is `.verbatim`, unqualified +as-filed content. Levels rise through `.canonical` up to the highest, +`.ambient`. It conforms to +`Comparable`. So a filter like "trust at least canonical" reads as an +ordinary comparison, rather than raw-value arithmetic. +`AdjectiveSensitivity` and `AdjectiveExportability` are both +scale-gapped. Their case raw values skip numbers, leaving room for +future intermediate tiers without disturbing any existing bitmask. +`AdjectiveSensitivity` carries three `ADR-007`-driven predicates: +`isBulkExportable`, `requiresOwnerKeyForBulk`, and +`isExcludedFromBulk`. These are the enforcement hooks VaultKit's +bulk-export path consults, to decide whether a drawer may ride a bulk +channel without additional friction. `Drawer.dreamingRecalcRequired` +and `Drawer.sealed` decode two single-bit obligation and trust-hint +flags, living above the four main axes. ## Provenance.swift -This file provides the seven named axes packed into `Drawer.provenance` -— `SourceType`, `Channel`, a mirrored `CaptureChannel` reference, -`Confirmation`, `Confidence`, `Sensitivity`, and `EnrichmentStatus` — -plus their computed accessors on `Drawer`. Where the adjective bitmap -records a row's *current standing* and the operational bitmap records -*mechanical facts about the content*, the provenance bitmap records -*how the row came into being and how it has been reviewed since*. - -`SourceType` names ten origins (user, observed, imported, canonical, -derived, federation-aggregate, tier-aggregate, paired-estate, ambient, -actuator). `Channel` names the system surface content arrived through -(UI typed or voiced input, an MCP agent, a file import, federation, or -one of three dreaming-daemon channels). `Confirmation` is the review -axis — unconfirmed through user-, automated-, peer-, or -actuator-confirmed — and `isUserConfirmed` is the convenience predicate -retrieval layers use to surface only user-vetted content. -`Confidence` and `Sensitivity` are both scale-gapped, `Confidence` -ordering from `.null` to `.verified` and `Sensitivity` deliberately -mirroring the raw values of `AdjectiveSensitivity` so the two axes — -sensitivity *at capture* versus the estate's current, possibly -since-mutated, access posture — can be compared directly. -`EnrichmentStatus` tracks a Q-ID enrichment daemon's lifecycle for a -row, ending either at `.qidCompleted` or, when deterministic -re-inference fails, at the terminal in-workflow state `.qidProposed`. +This file provides the seven named axes packed into +`Drawer.provenance`: `SourceType`, `Channel`, a mirrored +`CaptureChannel` reference, `Confirmation`, `Confidence`, +`Sensitivity`, and `EnrichmentStatus`. It also provides their +computed accessors on `Drawer`. The adjective bitmap records a row's +current standing. The operational bitmap records mechanical facts +about the content. The provenance bitmap records something +different: how the row came into being, and how it has been reviewed +since. + +`SourceType` names ten origins: user, observed, imported, canonical, +derived, federation-aggregate, tier-aggregate, paired-estate, +ambient, and actuator. `Channel` names the system surface content +arrived through: UI typed input, UI voiced input, an MCP agent, a +file import, federation, or a dreaming-daemon channel. `Confirmation` +is the review axis. Its levels run from unconfirmed through +user-confirmed, automated-confirmed, peer-confirmed, and +actuator-confirmed. `isUserConfirmed` is the +convenience predicate retrieval layers use, to surface only +user-vetted content. `Confidence` and `Sensitivity` are both +scale-gapped. `Confidence` orders from `.null` to `.verified`. +`Sensitivity` mirrors the raw values of +`AdjectiveSensitivity` on purpose. So the two axes can be compared directly: +sensitivity at capture, versus the estate's current access posture, +which may since have been mutated. `EnrichmentStatus` tracks a Q-ID +enrichment daemon's lifecycle for a row. It usually ends at +`.qidCompleted`. When deterministic re-inference fails instead, it +ends at the terminal in-workflow state `.qidProposed`. ## BitmapOps.swift -This file provides three small, `@inlinable` field-extraction helpers -— `andMask`, `thresholdCompare`, `shiftExtract` — that `BitmapEvaluator` -uses to translate a `Filter` case into a check against a packed Int64 -bitmap. The file is explicit that these are not math primitives: they -carry no algorithm to prove and no platform-specific optimization to -gate, unlike the substrate primitives (SimHash, Hamming distance, -OR-reduction) that live in SubstrateLib and participate in the -cross-platform conformance gate. Each function delegates its actual -bit manipulation to a SubstrateLib primitive (`BitField.maskedEquals`, -`BitField.extractField`, `BitField.popcount`) so LocusKit never -hand-rolls a shift-and-mask sequence that could silently diverge from -the Rust port's equivalent. +This file provides three small, `@inlinable` field-extraction +helpers: `andMask`, `thresholdCompare`, and `shiftExtract`. +`BitmapEvaluator` uses them to translate a `Filter` case into a check +against a packed Int64 bitmap. These helpers are not math primitives. +They carry no algorithm to prove. They need no platform-specific +optimization to gate. The substrate primitives are different: SimHash, +Hamming distance, and OR-reduction live in SubstrateLib. Those +primitives participate in the cross-platform conformance gate. Each +function here delegates its actual bit manipulation to a SubstrateLib +primitive: `BitField.maskedEquals`, `BitField.extractField`, or +`BitField.popcount`. So LocusKit never hand-rolls a shift-and-mask +sequence that could silently diverge from the Rust port's +equivalent. `andMask(_:mask:expected:)` tests whether a field equals an expected, -already-shifted value in one masked comparison. -`thresholdCompare(_:mask:shift:op:value:)` extracts a field and -compares it against a threshold using one of three orderings -(`.lessThan`, `.lessThanOrEqual`, `.greaterThanOrEqual`) — the -mechanism behind cluster-boundary filters such as "trust below the -action threshold." `shiftExtract(_:shift:mask:)` returns a field's raw -integer value, used wherever the evaluator needs the value itself -rather than a yes/no comparison. +already-shifted value, in one masked comparison. +`thresholdCompare(_:mask:shift:op:value:)` extracts a field. It +compares the field against a threshold. The comparison uses one of +three orderings: `.lessThan`, `.lessThanOrEqual`, and +`.greaterThanOrEqual`. This is +the mechanism behind cluster-boundary filters, such as "trust below +the action threshold." `shiftExtract(_:shift:mask:)` returns a +field's raw integer value. It is used wherever the evaluator needs +the value itself, rather than a yes-or-no comparison. ## ForbiddenCombinationValidator.swift -This file provides `ForbiddenCombinationValidator`, documenting (and, -where a live call site exists, enforcing) the one constitutional -forbidden bitmap combination: a row must never carry adjective -sensitivity `.secret` together with adjective exportability -`.public_`. Storage can physically represent that combination — nothing -stops the bits from being set — but the verb layer must refuse to -produce it. The file notes that current write paths route this check -through `AuditGate`/SubstrateLib directly, so this validator itself has -no live call site today; it exists as an explicit, reviewable statement -of the rule, ready to be wired in wherever a future write path bypasses -the gate. - -`validate(_:)` extracts the sensitivity field (bits 6–11) and the -exportability field (bits 12–17) from a full `adjectiveBitmap` value -and throws `LocusKitError.disciplineViolation` if both match their -forbidden raw values (48 and 32, respectively) — hand-derived from the -enum cases' shipped raw values rather than imported, specifically so a +This file provides `ForbiddenCombinationValidator`. It documents one +core rule: a forbidden bitmap combination. Where a live +call site exists, it also enforces that rule. The rule: a row must +never carry adjective sensitivity `.secret` together with adjective +exportability `.public_`. Storage can physically represent that +combination. Nothing stops the bits from being set. The verb layer +still must refuse to produce it. Current write paths route this +check through `AuditGate` and SubstrateLib. So this +validator itself has no live call site today. It exists as an +explicit, reviewable statement of the rule. It stays ready to be +wired in, wherever a future write path bypasses the gate. + +`validate(_:)` extracts two fields from a full `adjectiveBitmap` +value. The sensitivity field spans bits six through eleven. The +exportability field spans bits twelve through seventeen. It throws +`LocusKitError.disciplineViolation` if both match their forbidden raw +values, 48 and 32. These raw values are hand-derived from the enum +cases' shipped values, rather than imported. This is deliberate, so a future rename of an enum case cannot silently change what this validator checks. ## Filter.swift -This file provides `Filter`, the named recall-filter algebra, plus its -five small supporting type aliases (`LineageID`, `RoomID`, `WingID`, -`WikidataQID`, `ProvenanceChannel`, `FeatureFlag`) and the -`StateCluster` enum. No `Filter` case takes a raw bit position, mask, -or threshold integer — every case names a domain concern directly -(`.trustworthy`, `.inRoom(_:)`, `.contentMatches(_:)`), and -`BitmapEvaluator` is the sole place that translates a `Filter` into the -bitmap primitives internally. - -The cases group by concern: state queries (`.currentlyBelieve`, -`.usedToBelieve`, `.knewOnceAndErased`, `.state(_:)`, -`.stateInCluster(_:)`), trust queries (`.trustworthy`, -`.requiresConfirmation`, `.trust(_:)`, `.trustAtMost(_:)`), sensitivity -and exportability queries, provenance queries (confirmation, source -type, channel, confidence), operational queries (capture channel, -content kind, feature flags), structural queries (room, wing, lineage, -time bounds, lattice anchor and prefix, Wikidata concept), the single -content query `.contentMatches(_:)`, and three composition cases — -`.all`, `.any`, `.not` — that let filters nest arbitrarily. A -`RecallFrame.filterChain`, a plain array of `Filter`, is interpreted as -an implicit `.all` — every filter in the chain must pass. +This file provides `Filter`, the named recall-filter algebra. It also +provides five small supporting type aliases: `LineageID`, `RoomID`, +`WingID`, `WikidataQID`, and `ProvenanceChannel`, `FeatureFlag`, plus +the `StateCluster` enum. No `Filter` case takes a raw bit position, +mask, or threshold integer. Every case names a domain concern, +such as `.trustworthy`, `.inRoom(_:)`, or +`.contentMatches(_:)`. `BitmapEvaluator` is the sole place that +translates a `Filter` into the bitmap primitives internally. + +The cases group by concern. State queries include +`.currentlyBelieve`, `.usedToBelieve`, `.knewOnceAndErased`, +`.state(_:)`, and `.stateInCluster(_:)`. Trust queries include +`.trustworthy`, `.requiresConfirmation`, `.trust(_:)`, and +`.trustAtMost(_:)`. Other groups exist too. Sensitivity and +exportability queries form one group. Provenance queries form +another: confirmation, source type, channel, and confidence. +Operational queries form a third: capture channel, content kind, and +feature flags. Structural queries cover room, wing, lineage, time +bounds, lattice anchor and prefix, and Wikidata concept. One content +query, `.contentMatches(_:)`, searches text. Three +composition cases let filters nest arbitrarily: `.all`, `.any`, and +`.not`. A `RecallFrame.filterChain` is a plain array of `Filter` +values. It is interpreted as an implicit `.all`. Every filter in the +chain must pass. ## BitmapEvaluator.swift -This file provides `BitmapEvaluator`, the compiler and interpreter for -a `RecallFrame.filterChain` — the heart of the recall pipeline. It is -declared `internal` rather than `public` because it takes a -`DrawerStore` argument, itself internal; public callers reach it only -indirectly through `Estate.recall`. - -`evaluate(frame:drawers:store:nodeNames:)` runs four stages against the -candidate row set its caller has already narrowed down (via -`Estate.liveRows`, which applies fingerprint pruning ahead of this -call). **Default insertion** first prepends implicit filters for any -concern the caller left unconstrained: state defaults to -`.currentlyBelieve`, trust to `.trustworthy`, and sensitivity to -`.sensitivityAtMost(.elevated)` — the Normal-tier ceiling per ADR-007 — -so recall with an empty filter chain returns only currently-believed, -trustworthy, normal-or-elevated-sensitivity content by default, without -a caller needing to spell any of that out. **Bitmap-tier evaluation** -then compiles each `Filter` case to a check over the row's three raw -bitmaps using the `BitmapOps.swift` primitives, with tombstone exclusion -enforced unconditionally and independent of the caller's chain — a -tombstoned row can never surface through recall no matter what the -chain says. When `frame.asOf` is set, each row's bitmaps are -reconstructed as they stood at that historical HLC via -`AuditLogFold.projectStateAt` before the bitmap tier runs. **Structured-tier** -evaluation then applies room, wing, lineage, time-bound, and lattice -filters that need the drawer's non-bitmap fields (and, for room/wing, -a resolved node-name lookup). **Content-tier** evaluation applies -`.contentMatches` via a case-insensitive substring search over the -verbatim content. A final **ordering** pass sorts the survivors. +This file provides `BitmapEvaluator`, the compiler and interpreter +for a `RecallFrame.filterChain`. This is the heart of the recall +pipeline. It is declared `internal` rather than `public`, because it +takes a `DrawerStore` argument, itself internal. Public callers reach +it only indirectly, through `Estate.recall`. + +`evaluate(frame:drawers:store:nodeNames:)` runs four stages. These +stages run against the candidate row set its caller has already +narrowed down, through `Estate.liveRows`, which applies fingerprint +pruning ahead of this call. The first stage is default insertion. It +prepends implicit filters for any concern the caller left +unconstrained. State defaults to `.currentlyBelieve`. Trust defaults +to `.trustworthy`. Sensitivity defaults to +`.sensitivityAtMost(.elevated)`, the Normal-tier ceiling per ADR-007. +So recall with an empty filter chain returns only content by default: +currently believed, trustworthy, and normal-or-elevated in +sensitivity. A caller need not spell any of this out. The second +stage is bitmap-tier evaluation. It compiles each `Filter` case to a +check over the row's three raw bitmaps, using the `BitmapOps.swift` +primitives. Tombstone exclusion is always enforced here, apart from +the caller's chain. A tombstoned row can never surface +through recall, no matter what the chain says. When `frame.asOf` is +set, each row's bitmaps are first rebuilt as they stood at that +past HLC. This rebuild runs through +`AuditLogFold.projectStateAt`, before the bitmap tier runs. The third +stage is structured-tier evaluation. It applies filters that need the +drawer's non-bitmap fields: room, wing, lineage, time bounds, and +lattice. For room and wing filters, it also needs a resolved +node-name lookup. The fourth stage is content-tier evaluation. It +applies `.contentMatches`, through a case-insensitive substring +search over the verbatim content. A final ordering pass then sorts +the survivors. `chainHasPrunableFilter`, `chainHasContentPredicate`, and -`chainHasStructuredNameFilter` are classifiers `Estate.liveRows` and -`Estate.getDrawers(ids:matchingFrame:hydrationLevel:)` consult before -even fetching rows: the first decides whether fingerprint pruning can -apply at all, the second decides whether the fetch needs to pay for the -content blob, and the third decides whether a node-name lookup is -needed. `containerSurvives(chain:fingerprint:)` is the actual pruning -predicate: it returns false only when the chain provably cannot be -satisfied by any row in a container, which is sound only for -set-bit filters like `.hasFeatureFlag` — a threshold or exact-value -filter can never be decided from an OR-reduced fingerprint alone, so -those filters never prune a container, they only ever get evaluated -per-row. +`chainHasStructuredNameFilter` are classifiers. `Estate.liveRows` and +`Estate.getDrawers(ids:matchingFrame:hydrationLevel:)` consult them +before even fetching rows. The first classifier decides whether +fingerprint pruning can apply at all. The second decides whether the +fetch needs to pay for the content blob. The third decides whether a +node-name lookup is needed. `containerSurvives(chain:fingerprint:)` +is the actual pruning predicate. It returns false only when the chain +provably cannot be satisfied by any row in a container. This is sound +only for set-bit filters, such as `.hasFeatureFlag`. A threshold or +exact-value filter can never be decided from an OR-reduced fingerprint +alone. So those filters never prune a container. They only ever get +evaluated per row. ## Tunnel.swift This file provides `Tunnel`, the typed cross-reference between two -locations in the estate. A tunnel is stored directionally — source -endpoint to target endpoint, each carrying a wing, a room, and an -optional specific drawer id (nil meaning "the room itself") — so a -query asking "what does this side know about" never needs to scan both -directions. `kind`, a `TunnelKind`, is the closed, indexed relationship -vocabulary the retrieval layer dispatches on; `label` is a free-form, -unvalidated human annotation. `orderKey` is a fractional-index sibling -ordering value used only by `.parent` tunnels in the outline graph -(ADR-017) — siblings under the same parent sort by ascending -`orderKey` without every sibling needing to be rewritten when one is -inserted between two existing siblings. +locations in the estate. A tunnel is stored directionally: from a +source endpoint to a target endpoint. Each endpoint carries a wing and a +room. Each endpoint also carries an optional specific drawer id. A +`nil` id means the room itself is the endpoint. So a query asking +what this side knows about never needs to scan both directions. +`kind` is a `TunnelKind`. This is the closed, indexed relationship +vocabulary the retrieval layer dispatches on. `label` is +a free-form, unvalidated human annotation. `orderKey` is a +fractional-index sibling ordering value. It is used only by +`.parent` tunnels in the outline graph, per ADR-017. Siblings under +the same parent sort by ascending `orderKey`. So no sibling needs to +be rewritten when a new one is inserted between two existing +siblings. ## TunnelOperational.swift -This file provides `TunnelKind` (the ten-case closed relationship -vocabulary — `.supersedes`, `.references`, `.blocks`, `.validates`, -`.contradicts`, `.derivesFrom`, `.covers`, `.elaborates`, `.respondsTo`, -and `.parent`, the outline-containment edge) and the four operational -axes packed into `Tunnel.operationalBitmap` — `TunnelDirection`, -`TunnelLifecycle`, `TunnelOriginClass`, `TunnelStrength` — plus their -accessors. - -Two bits above those four axes carry standalone boolean flags rather -than enumerated fields: `hasInverse` (whether a paired reverse tunnel -exists) and `isRetired` (bit 13, added for the T13/ADR-021 dreaming -retirement mechanism). `withRetired()` and `withUnretired()` return a -copy of a tunnel with that bit flipped — the reversible mechanism by -which the dreaming pipeline's OMEGA cycle suspends a tunnel from active -reads without tombstoning it, so a later co-recall can re-propose the -same pairing. A separate provenance-bitmap section adds `isDreamed` -(bit 0 of `provenanceBitmap`), set only for tunnels the dreaming -pipeline itself proposed and later accepted, never for a -user-explicit, imported, or federated tunnel — OMEGA's retirement -predicate requires `isDreamed == true`, so a declared tunnel is never -retired regardless of how much or little it is recalled. +This file provides `TunnelKind`, the ten-case closed relationship +vocabulary. Its cases are: `.supersedes`, `.references`, `.blocks`, +`.validates`, `.contradicts`, `.derivesFrom`, `.covers`, +`.elaborates`, `.respondsTo`, and `.parent`, the outline-containment +edge. This file also provides the four operational axes packed into +`Tunnel.operationalBitmap`: `TunnelDirection`, `TunnelLifecycle`, +`TunnelOriginClass`, and `TunnelStrength`, plus their accessors. + +Two bits above those four axes carry standalone boolean flags, +rather than enumerated fields. `hasInverse` records whether a paired +reverse tunnel exists. `isRetired` is bit 13. The T13 mechanism, per +ADR-021, added this bit for dreaming retirement. `withRetired()` and +`withUnretired()` return a copy of a tunnel with that bit flipped. +This is the reversible mechanism by which the dreaming pipeline's +OMEGA cycle suspends a tunnel from active reads, without tombstoning +it. So a later co-recall can re-propose the same pairing. A separate +provenance-bitmap section adds `isDreamed`, bit 0 of +`provenanceBitmap`. This flag is set only for tunnels the dreaming +pipeline itself proposed and later accepted. It is never set for a +user-explicit, imported, or federated tunnel. OMEGA's retirement +predicate requires `isDreamed == true`. So a declared tunnel is never +retired, no matter how much or little it is recalled. ## DiaryEntry.swift -This file provides `DiaryEntry`, the first-person record of what an -agent thought, did, or learned at a point in time. A diary entry is -stored alongside drawers but queried separately, keyed by `agentName` -and an unvalidated `topic` tag, so one agent's diary never leaks into -another's wing-filtered recall by accident (a convention, not an -enforced constraint — `wing`/`room` are plain required strings the -caller supplies). `reward` and `rewardProvenance` are the explicit -quality-signal channel: when a caller has a direct quality score for an -entry (a user rating, a model confidence value), it is recorded here -directly rather than left for the dreaming daemon's implicit -recall-based reward inference to guess at later. +This file provides `DiaryEntry`. This is the first-person record of +what an agent thought, did, or learned at a point in time. A diary +entry is stored alongside drawers. It is queried separately, keyed by +`agentName` and an unvalidated `topic` tag. So one agent's diary +never leaks into another agent's wing-filtered recall by accident. +This is a convention, not an enforced constraint. `wing` and `room` +are plain required strings the caller supplies. `reward` and +`rewardProvenance` are the explicit quality-signal channel. When a +caller has a direct quality score for an entry, such as a user rating +or a model confidence value, it is recorded here. Otherwise +it is left for the dreaming daemon's implicit recall-based reward +inference to guess at later. ## DiaryOperational.swift This file provides the four axes packed into -`DiaryEntry.operationalBitmap` — `DiaryEventClass` (twelve cases: what -kind of substrate event this entry records — capture, mutation, +`DiaryEntry.operationalBitmap`, plus a single flag bit and their +computed accessors. `DiaryEventClass` has twelve cases. It records +what kind of substrate event this entry describes: capture, mutation, withdraw, expunge, propose, associate, learn, signal emission, -maintenance, migration, training, or audit-tombstone), -`DiarySeverity` (scale-gapped: trace, info, warning, error), -`DiaryActorClass` (who produced the entry: user, substrate daemon, MCP -agent, migration tool, or federation peer), and -`DiaryBatchMembership` (standalone, batch-start, batch-member, or -batch-end) — plus the single `requiresFollowup` flag bit, and their -computed accessors. +maintenance, migration, training, or audit-tombstone. `DiarySeverity` +is scale-gapped, with four levels: trace, info, warning, and error. +`DiaryActorClass` records who produced the entry: user, substrate +daemon, MCP agent, migration tool, or federation peer. +`DiaryBatchMembership` has four values: standalone, batch-start, +batch-member, and batch-end. The single flag bit is +`requiresFollowup`. ## KGFact.swift This file provides `KGFact`, a subject-predicate-object triple -distilled from a verbatim drawer, retaining `sourceDrawerID` as a -backreference so a fact's provenance is always recoverable. `subject`, -`predicate`, and `object` are all free-form strings at this rung — no -entity vocabulary is enforced by the value type itself; that +distilled from a verbatim drawer. It retains `sourceDrawerID` as a +backreference, so a fact's provenance is always recoverable. +`subject`, `predicate`, and `object` are all free-form strings at +this rung. The value type itself enforces no entity vocabulary. That enforcement, when it arrives, belongs to a later federated -knowledge-graph layer. `sourceDrawerID` may legitimately be the empty -string, an "unanchored fact" sentinel used when a caller asserts a -freestanding triple not extracted from any specific drawer. - -`KGFact` reuses the same three-bitmap pattern every other bitmap-backed -noun uses, and its adjective accessors (`state`, `adjectiveSensitivity`, -`exportability`, `trust`) decode the identical bit layout `Drawer` -uses — a fact and its source drawer can be filtered by the same -retrieval predicates because they share the same encoding. +knowledge-graph layer. `sourceDrawerID` may legitimately be an empty +string. An empty string is an unanchored-fact sentinel. A caller uses +it when asserting a freestanding triple, one not extracted from any +specific drawer. + +`KGFact` reuses the same three-bitmap pattern every other +bitmap-backed noun uses. Its adjective accessors are `state`, +`adjectiveSensitivity`, `exportability`, and `trust`. These decode +the same bit layout `Drawer` uses. So a fact and its source drawer +can be filtered by the same retrieval predicates. They share the same +encoding. ## KGFactOperational.swift -This file provides the four axes plus one flag packed into -`KGFact.operationalBitmap`: `KGExtractorClass` (six cases ordered -roughly by rigor — manual, foundation-model, specialized-model, -rules-based, imported-KG, federated), `KGAssertionKind` (asserted, -inferred, hypothesized, or contradicted — the last recording that -another fact disputes this one without either being retracted, with -resolution deferred to retrieval time), `KGSpecificity` and -`KGConfidenceBand` (both scale-gapped and `Comparable`, letting a -filter read as `fact.specificity >= .specific`), and the `isCanonical` -flag marking a fact promoted to estate-wide, load-bearing status. +This file provides the four axes, plus one flag, packed into +`KGFact.operationalBitmap`. `KGExtractorClass` has six cases. They are ordered +roughly by rigor: manual, foundation-model, specialized-model, +rules-based, imported-KG, and federated. `KGAssertionKind` has four +cases: asserted, inferred, hypothesized, and contradicted. The last +case records that another fact disputes this one, without either +fact being retracted. Resolution is deferred to retrieval time. +`KGSpecificity` and `KGConfidenceBand` are both scale-gapped and +`Comparable`. So a filter can read as `fact.specificity >= +.specific`. The `isCanonical` flag marks a fact promoted to +estate-wide, load-bearing status. ## Proposal.swift This file provides `Proposal`, a suggested change awaiting -confirmation — the durable record produced by the substrate's only -autonomous write surface. `Proposal` structurally mirrors `KGFact` with -one addition `KGFact` predates: a required `latticeAnchor`, because -`KGFact` was written before the cookbook universalised the -every-row-has-an-anchor invariant. `candidateState` is the adjective -set the proposal would apply to its target if accepted — the accept -path reads this value to know what to write. `Proposal.state` decodes -the proposal's own lifecycle position (pending while awaiting -confirmation, then accepted, rejected, or withdrawn) from its -adjective bitmap's state field, the same field `Drawer.state` decodes. +confirmation. It is the durable record produced by the substrate's +only autonomous write surface. `Proposal` structurally mirrors +`KGFact`, with one addition `KGFact` predates: a required +`latticeAnchor`. `KGFact` was written before the cookbook +universalized the every-row-has-an-anchor invariant. `candidateState` +is the adjective set the proposal would apply to its target, if +accepted. The accept path reads this value, to know what to write. +`Proposal.state` decodes the proposal's own lifecycle position, from +its adjective bitmap's state field, the same field `Drawer.state` +decodes. This position starts at pending, while the proposal awaits +confirmation. It then moves to accepted, rejected, or withdrawn. ## ProposalOperational.swift This file provides the five typed axes packed into -`Proposal.operationalBitmap`: `ProposalKind` (nine cases — what kind of -write this proposal proposes, from `.newTunnel` and `.mutateDrawer` -through the newer `.actionProposal`, `.recordObservation`, and -`.tierAdvisory`), `ProposalTargetObjectType` (which kind of row this -proposal targets, including the `.noneBrandNew` sentinel for a -not-yet-existing target and `.systemState` for a proposal about the -estate itself), `ProposalConfirmationSource` (human, agent, automated -threshold, or actuator), `ProposalGeneratedByClass` (dreaming daemon, -MCP agent, federation sync, manual, or tier aggregator), and -`ProposalConfidenceBucket` (scale-gapped and `Comparable`). This is the -one operational layout the file notes is directly specified by the -engineering cookbook rather than being LocusKit-internal, so a -conformance test pins every field position and raw value against the -cookbook table directly. +`Proposal.operationalBitmap`. `ProposalKind` has nine cases. It +records what kind of write this proposal proposes: `.newTunnel`, +`.mutateDrawer`, and seven more, including the newer +`.actionProposal`, `.recordObservation`, and `.tierAdvisory`. +`ProposalTargetObjectType` records which kind of row this proposal +targets. This includes the `.noneBrandNew` sentinel, for a +not-yet-existing target. It also includes `.systemState`, for a +proposal about the estate itself. `ProposalConfirmationSource` has +four values: human, +agent, automated threshold, and actuator. `ProposalGeneratedByClass` +has five values: dreaming daemon, MCP agent, federation sync, manual, +and tier aggregator. `ProposalConfidenceBucket` is scale-gapped and +`Comparable`. + +This file notes that the operational layout for `ProposalKind` and +its siblings comes straight from the engineering cookbook, rather +than being LocusKit-internal. So a conformance test pins every field +position and raw value against the cookbook table. `composeOperational(kind:targetObjectType:generatedBy:confidence:)` -assembles a full operational bitmap from four of the five axes in one -call (confirmation source is left at its default until a confirmation -step actually runs), used by autonomic daemon sinks that need to stamp -genuine provenance on the proposals they emit. +assembles a full operational bitmap from four of the five axes, in +one call. Confirmation source is left at its default, until a +confirmation step actually runs. Autonomic daemon sinks use this +function. They need to stamp genuine provenance on the proposals they +emit. ## Association.swift This file provides `Association`, the graph edge recording that two -rows belong together — a statistical or dreaming-derived pairing, -distinct from a `Tunnel`'s typed semantic claim. `Association` -structurally mirrors `Tunnel` (source and target endpoints, three -bitmap columns, the Rev 1.0 soft-delete reservation) with two -deliberate differences: it carries no `kind` column — an association's -semantics live entirely in its operational bitmap — and it carries a -required `latticeAnchor`, anchored to the lattice midpoint of its two -endpoints, which `Tunnel` predates and therefore lacks. +rows belong together. This is a statistical or dreaming-derived +pairing, distinct from a `Tunnel`'s typed semantic claim. +`Association` structurally mirrors `Tunnel`: a source and target +endpoint, three bitmap columns, and the Rev 1.0 soft-delete +reservation. Two differences set it apart. It carries no `kind` +column. An association's semantics live entirely in its operational +bitmap. It also carries a required `latticeAnchor`, anchored to the +lattice midpoint of its two endpoints. `Tunnel` predates this +requirement, so it lacks the field. ## AssociationOperational.swift This file provides the three axes packed into -`Association.operationalBitmap`: `AssociationSignalSources` (a -twelve-bit bitset — an `OptionSet`, not an enum, because more than one -signal can independently contribute evidence for the same pairing: a -co-recall signal, a co-confirmation signal, a dream-pairing signal, a -vector-similarity signal, a shared-entity signal, an explicit-human -signal, a fingerprint-similarity signal, and three v0.36 additions — -cross-estate, cross-tier, and action-outcome), `AssociationDecayClass` -(scale-gapped and `Comparable`: pinned, slow, normal, fast — how -quickly an association ages out of relevance), and `AssociationArity` -(binary today; n-ary reserved for a future version, per invariant -I-23's current binary-only limit). +`Association.operationalBitmap`. `AssociationSignalSources` is a +twelve-bit bitset, an `OptionSet` rather than an enum, because more +than one signal can support the same pairing on its own. Its +signals include a co-recall signal, a co-confirmation signal, a +dream-pairing signal, a vector-similarity signal, a shared-entity +signal, an explicit-human signal, a fingerprint-similarity signal, +and three v0.36 additions: cross-estate, cross-tier, and +action-outcome. `AssociationDecayClass` is scale-gapped and +`Comparable`, with four levels: pinned, slow, normal, and fast. This +axis records how quickly an association ages out of relevance. +`AssociationArity` is binary today. An n-ary form is reserved for a +future version, per invariant I-23's current binary-only limit. ## LearnedReference.swift This file provides `LearnedReference`, the durable record of an -external reference the grounding-driven `learn` verb brought into the -estate. It structurally mirrors `Association` — an id, content +external reference the grounding-driven `learn` verb brought into +the estate. It structurally mirrors `Association`: an id, content columns, a required `latticeAnchor`, three bitmap columns, and the -soft-delete reservation — chosen as the template specifically because -`Association` is the freshest content-bearing noun that already -honours the anchor requirement. `sourceCatalogID` is a foreign-key -reference to the `SourceCatalogEntry` this reference was learned from, -stored as an identifier string the same way `KGFact` stores -`sourceDrawerID` rather than embedding the full catalog entry inline. -`handle` is the reference's own locator, distinct from the source's own -locator on the catalog entry it points into. +soft-delete reservation. `Association` was chosen as the template +because it is the freshest content-bearing noun that already honors +the anchor requirement. `sourceCatalogID` is a +foreign-key reference to the `SourceCatalogEntry` this reference was +learned from. It is stored as an identifier string, the same way +`KGFact` stores `sourceDrawerID`, rather than embedding the full +catalog entry inline. `handle` is the reference's own locator. It is +distinct from the source's own locator, on the catalog entry it +points into. ## LearnedReferenceOperational.swift This file provides the four axes packed into -`LearnedReference.operationalBitmap`: `RefreshPolicy` (scale-gapped and -`Comparable`, ordering none through monthly, weekly, daily, on-demand, -and realtime), `DriftSeverity` (scale-gapped and `Comparable`: none, -minor, major, critical — how far a reference has drifted from its -source since it was last re-grounded), `LearnMode` (a single bit: -whether the reference is held by pointer, `.byReference`, or its -content was ingested wholesale, `.byIngestion`), and -`LearnedReferenceSource` (where the reference was acquired from — user, -federation, household pairing, fleet pairing, tier inheritance, or -paired estate — distinct from `sourceCatalogID`, which names *which* -catalog entry, not *how* the reference arrived). +`LearnedReference.operationalBitmap`. `RefreshPolicy` is scale-gapped +and `Comparable`. Its order runs from none through monthly, weekly, +daily, on-demand, and realtime. `DriftSeverity` is scale-gapped and +`Comparable`, with four levels: none, minor, major, and critical. +This axis records how far a reference has drifted from its source, +since it was last re-grounded. `LearnMode` is a single bit. It +records how the reference was acquired. `.byReference` means the +reference is held by pointer. `.byIngestion` means its content was +ingested wholesale. `LearnedReferenceSource` records where the reference was acquired +from: user, federation, household pairing, fleet pairing, tier +inheritance, or paired estate. This is distinct from +`sourceCatalogID`, which names which catalog entry the reference came +from, not how the reference arrived. ## SourceCatalogEntry.swift -This file provides `SourceCatalogEntry`, the durable, queryable record -of an external source from which references are learned — the -substrate behind the `source` slot of the `learn` verb. Its reason for -existing is grounding integrity: every `LearnedReference` must carry a -genuine lattice anchor, never a fabricated sentinel, and an anchor is a -property of the source (a web domain, a document corpus, a paired -estate), not of each individual handle learned from it. Cataloging a -source once and having every reference from it inherit that catalog -entry's anchor is what makes a genuine anchor available without -inventing one per reference. `SourceKind` names six source classes -(user, federation, household pairing, fleet pairing, tier inheritance, -paired estate) — deliberately the same vocabulary -`LearnedReferenceSource` uses, because the acquisition channel a -reference was learned through is the same vocabulary as the kind of -source it came from. +This file provides `SourceCatalogEntry`, the durable, queryable +record of an external source from which references are learned. It +is the substrate behind the `source` slot of the `learn` verb. Its +reason for existing is grounding integrity. Every `LearnedReference` +must carry a genuine lattice anchor, never a fabricated sentinel. An +anchor is a property of the source: a web domain, a document corpus, +or a paired estate. It is not a property of each individual handle +learned from it. Cataloging a source once lets every reference from +it inherit that catalog entry's anchor. This is what makes a genuine +anchor available, without inventing one per reference. `SourceKind` +names six source classes: user, federation, household pairing, fleet +pairing, tier inheritance, and paired estate. This is the same +vocabulary `LearnedReferenceSource` uses, on purpose. The acquisition +channel a reference was learned through is the same vocabulary as the +kind of source it came from. ## Node.swift -This file provides `Node`, a container node in the estate's containment -tree: the estate root at depth 0, a wing at depth 1, or a room at depth -2. Every node carries two name fields — `displayName`, which preserves -whatever casing the first writer used, and `lookupName`, a normalized -form (Unicode NFC, trimmed, internal whitespace collapsed, casefolded) -used for resolution and uniqueness enforcement — so "Personal " and -"personal" resolve to the same node while the node's display name -still shows however it was first written. `lifecycle` (0 active, 1 -tombstoned) plus a pair of HLC-typed timestamps -(`createdHlc`/`tombstonedHlc`) let the containment tree support the -same as-of historical read surface as drawers, without needing -wall-clock timestamps to double as the ordering key. -`Node.normalizeLookupName(_:)` is the single, conformance-relevant -normalization function both the Swift and Rust ports must produce -byte-identical results from. +This file provides `Node`, a container node in the estate's +containment tree. The estate root sits at depth zero. A wing sits at +depth one. A room sits at depth two. Every node carries two name +fields. `displayName` preserves whatever casing the first writer +used. `lookupName` is a normalized form: Unicode NFC, trimmed, with +internal whitespace collapsed and casefolded. This normalized form is +used for lookup and to keep names unique. So "Personal " and +"personal" resolve to the same node. The node's display name still +shows the casing it was first written in. `lifecycle` has two +values, active or tombstoned. A pair of HLC-typed timestamps, +`createdHlc` and `tombstonedHlc`, support this. They let the +containment tree read its own past state, the same way drawers do. +Neither needs a wall-clock timestamp to double as the +ordering key. `Node.normalizeLookupName(_:)` is the single, +conformance-relevant normalization function. Both the Swift and Rust +ports must produce byte-identical results from it. ## NodeStore.swift This file provides `NodeStore`, the actor that owns the containment -tree's storage and its create-on-demand resolution: given a display -name and a parent id, find the active node with that normalized lookup -name under that parent, or create one if none exists. `NodeStore` is -an actor specifically so this find-then-insert sequence is race-free -without needing an `INSERT OR IGNORE` or a conflict-column mechanism — -two concurrent requests to create the same wing under the same estate -serialize through the actor and produce exactly one node, because the -second request's `findActiveNode` call cannot run until the first -request's insert has completed. +tree's storage. It also owns the tree's create-on-demand lookup. +Given a display name and a parent id, it finds the active node with +that normalized lookup name under that parent, or creates one if +none exists. `NodeStore` is an actor for a specific reason. This +find-then-insert sequence stays race-free without needing an `INSERT +OR IGNORE` clause or a conflict-column mechanism. Two concurrent +requests to create the same wing under the same estate serialize +through the actor. So they produce exactly one node. The second +request's `findActiveNode` call cannot run until the first request's +insert has completed. `createNode(displayName:parentId:now:)` enforces two structural -invariants before writing: the parent must already exist (I-NT-5), and -the new node's depth (parent's depth plus one) must not exceed 2 -(I-NT-2) — a room cannot have a child, because there is no fourth -tier. `createRoot(displayName:now:)` is the once-per-estate seed for -the depth-0 root, enforcing that exactly one root exists (I-NT-1) by -returning the existing root unchanged if one is already present. -Tombstoned nodes are invisible to resolution — a tombstoned wing does -not block, and cannot be accidentally reused by, a later request for a -wing with the same name (the no-resurrection guard, §5). +invariants before writing. The parent must already exist, per +invariant I-NT-5. The new node's depth is the parent's depth plus +one. This depth must not exceed two, per invariant I-NT-2. A room cannot have a +child, because there is no fourth tier. `createRoot(displayName:now:)` +is the once-per-estate seed for the depth-zero root. It enforces that +exactly one root exists, per invariant I-NT-1, by returning the +existing root unchanged if one is already present. Tombstoned nodes +are invisible to lookup. A tombstoned wing does not block a later +request for a wing with the same name. It also cannot be reused by +mistake, for that later request. This is the no-resurrection guard, per +section five. ## NodeBundleStore.swift -This file provides `NodeBundleStore`, persistence for the bundle-algebra -count-vector aggregates — one row per node (room or wing) per bundle -kind, stored in the `node_bundles` table. A count-vector here is a -256-element array of counts, one per fingerprint bit, encoded as 1024 -bytes of little-endian `UInt32` values; `BundleKind.activeA` is the -"active centroid" (a fold of a node's currently active members) and -`BundleKind.departedB` is the "departed accumulator" (eager-folded at -departure time). The per-row drawer fingerprints that feed these -aggregates are never themselves stored — only the folded aggregate is, -which keeps the table small regardless of estate size. - -`put`/`get`/`rooms(forWing:kind:)` are the read/write surface; -`encodeCounts`/`decodeCounts` handle the wire format, with -`decodeCounts` throwing `LocusKitError.invalidContent` (rather than -trapping) when a stored blob is not exactly 1024 bytes, so a corrupt -row surfaces as a recoverable error instead of crashing the process. +This file provides `NodeBundleStore`, persistence for the +bundle-algebra count-vector totals. One row exists per node, room +or wing, per bundle kind. Each row is stored in the `node_bundles` +table. A count-vector here is a 256-element array of counts, one per +fingerprint bit, encoded as 1024 bytes of little-endian `UInt32` +values. `BundleKind.activeA` is the active centroid, a fold of a +node's currently active members. `BundleKind.departedB` is the +departed accumulator, eager-folded at departure time. The per-row +drawer fingerprints that feed these totals are never themselves +stored. Only the folded total is stored. This keeps the table +small, no matter the estate size. + +`put`, `get`, and `rooms(forWing:kind:)` are the read-and-write +surface. `encodeCounts` and `decodeCounts` handle the wire format. +`decodeCounts` throws `LocusKitError.invalidContent`, rather than +trapping, when a stored blob is not exactly 1024 bytes. So a corrupt +row surfaces as a recoverable error, instead of crashing the process. ## BundleMaterializer.swift -This file provides `BundleMaterializer`, the operation that actually -computes a Bundle A aggregate from live drawers — the first real -consumer of `SubstrateKernel.countFold256`. Bundle A cannot be -maintained incrementally (active membership changes over time, and the -fold operation has no subtraction), so it is fully recomputed on -demand, typically by a periodic dreaming tick. +This file provides `BundleMaterializer`, the operation that computes +a Bundle A total from live drawers. It is the first +real consumer of `SubstrateKernel.countFold256`. Bundle A cannot be +maintained bit by bit. Active membership changes over time, and +the fold operation has no subtraction. So Bundle A is fully +recomputed on demand, typically by a periodic dreaming tick. `materializeRoom(wing:room:now:)` fetches every non-tombstoned drawer -in a room, filters to `State.isClusterA` (currently believed) rows, -derives each survivor's `Fingerprint256` via `EstateFingerprintFamilies`, -folds the set with `kernel.countFold256`, and stores the result. -`rollUpWing(wing:now:)` merges a wing's already-materialized room -bundles into one wing-level bundle; because the count-vector fold is -associative, this merge produces the identical result as folding every -active drawer in the wing directly, so rooms may be materialized in any -order before the wing roll-up runs. +in a room. It filters to `State.isClusterA` rows, the currently +believed ones. It derives each survivor's `Fingerprint256`, through +`EstateFingerprintFamilies`. It folds the set with +`kernel.countFold256`. It stores the result. `rollUpWing(wing:now:)` +merges a wing's already-materialized room bundles into one +wing-level bundle. The count-vector fold is associative. So this +merge produces the same result as folding every active drawer +in the wing. Rooms may be materialized in any order before +the wing roll-up runs. ## Fingerprint256Adapters.swift -This file provides a small marshaling convention: packing a single -Int64 bitmap column into block 0 of a `Fingerprint256` (with blocks 1–3 -zeroed) so a per-column bit operation — an OR, an XOR, a popcount — can -be routed through a SubstrateLib primitive that operates at the -substrate's canonical 256-bit width, rather than LocusKit reimplementing -the same math at Int64 width. The file is explicit that this is purely -a type-shape convention with no algorithm of its own, and it warns that -this packing is not the same as the cookbook's Bitmap-LSH SimHash -interpretation of block 0 — a caller must not mix the two uses of the -same 64-bit lane. +This file provides a small marshaling convention. It packs a single +Int64 bitmap column into block zero of a `Fingerprint256`, with +blocks one through three zeroed. This lets a per-column bit +operation route through a SubstrateLib primitive instead: an OR, an +XOR, or a popcount. That primitive operates at the substrate's +canonical 256-bit width. LocusKit avoids reimplementing the same math at Int64 width this way. +This file is explicit that the convention is purely a type-shape +choice, with no algorithm of its own. The file also warns that this +packing differs from the cookbook's Bitmap-LSH SimHash interpretation +of block zero. A caller must not mix the two uses of the same 64-bit +lane. `init(int64Column:)` and the `int64Column` computed property are the -pack and unpack directions, used throughout `ContainerFingerprintStore`. +pack and unpack directions. They are used throughout +`ContainerFingerprintStore`. ## ContainerFingerprintStore.swift This file provides `ContainerFingerprintStore`, the per-container -OR-reduction aggregates that make fingerprint-based recall pruning -possible: for every room, and rolled up to every wing, the bitwise OR -of the three bitmap fields across every active drawer in that -container. Because OR is monotone, a container's aggregate can only -ever gain bits as content is added; a bit that is absent from the -aggregate is therefore provably absent from every row in the -container, which is exactly the soundness property recall pruning -needs — a filter requiring that bit can safely skip the whole -container without fetching a single row. +OR-reduction totals. These totals make fingerprint-based +recall pruning possible. For every room, the store keeps the bitwise +OR of the three bitmap fields, across every active drawer in that +container. It also rolls this total up to every wing. OR is +monotone. So a container's total can only ever gain bits as content +is added. So a bit absent from the total is provably absent from +every row in the container. This is exactly the soundness property +recall pruning needs. A filter requiring that bit can safely skip the +whole container, without fetching a single row. `orIn(wing:room:adjective:operational:provenance:now:)` is the -incremental maintenance path, called on every drawer capture; it folds -the new drawer's bitmaps into both the room-level and the wing-level -row via `ContainerFingerprint.merging(_:)`, which itself routes through -`SubstrateLib.ORReduce.reduce` at canonical 256-bit width using the -Fingerprint256 packing convention above. The clear side is -deliberately absent — withdrawing or expunging a drawer never clears -bits from the aggregate, because a stale set bit is a harmless -over-approximation (it can only cause an unnecessary scan, never a -missed match), while `rebuildRoom`/`rollUpWing`/`rebuildAll` exist to -periodically tighten the aggregate back down from a fresh full scan. -`rebuildAll` is what `Estate.open` calls once at startup, so an -existing estate's aggregate is guaranteed complete and therefore sound -before any recall runs against it. +incremental maintenance path. It runs on every drawer capture. It +folds the new drawer's bitmaps into both the room-level and the +wing-level row, through `ContainerFingerprint.merging(_:)`. That +function itself routes through `SubstrateLib.ORReduce.reduce`, at +canonical 256-bit width, using the `Fingerprint256` packing +convention above. The clear side is absent on purpose. Withdrawing +or expunging a drawer never clears bits from the total. A stale +set bit is a harmless over-approximation. It can only cause an +unnecessary scan, never a missed match. Meanwhile, three other +functions exist for a different purpose. They are `rebuildRoom`, +`rollUpWing`, and `rebuildAll`. They tighten the total back down, +now and then, from a fresh full scan. `rebuildAll` is what +`Estate.open` calls once at startup. So an existing estate's +total is guaranteed complete, and thus sound, before any +recall runs against it. ## MerkleRollup.swift -This file provides the Merkle content-integrity rollup: an extension on -`Estate` that computes a hash tree — room, then wing, then estate — -bottom-up over the same containment tree `NodeStore` maintains. A -Merkle tree is a hash tree where each parent's hash is computed from -its children's hashes, so a single top-level root hash can attest to -the exact contents of an entire subtree; changing any leaf changes -every hash on the path up to the root. The rollup is invoked -explicitly, not automatically after every write, because computing it -inline on every drawer capture would be O(room-size) per write and peg -the CPU during a bulk import. - -`computeRoomMerkleRoot(roomNodeId:)` hashes a room's live drawers, -reading each row's stored `content_hash` column when present (written -by a separate hash-on-write hook) or computing a leaf hash on demand -from the drawer's content when absent; it explicitly excludes both -tombstoned rows (irreversibly deleted) and withdrawn rows (state raw -value 18) from the snapshot, because a withdrawn drawer's content has -been retracted by the user and must not remain retrievable through a -content-attesting snapshot. `computeEstateOrWingMerkleRoot(parentNodeId:)` -hashes a set of children's already-computed roots — the same function -serves both the wing level (hashing room roots) and the estate level -(hashing wing roots). `rollupMerkleRoots(roomNodeId:now:)` runs all -three levels for one changed room; `recomputeAllMerkleRoots(now:)` is -the full bottom-up recompute used after bulk import, migration, or -corruption recovery; `rollupAllMerkleRoots(now:)` is a documented alias -of the full recompute, named for the batch-capture reindex path. -`createSnapshot(label:now:additionalAttestations:)` recomputes the -full tree (so a snapshot is never taken against stale roots), then -writes an attestation row for the estate root and every wing, plus any -attestations a higher composition-layer kit supplies. +This file provides the Merkle content-integrity rollup, an extension +on `Estate`. It computes a hash tree from the bottom up: room, then +wing, then estate. This hash tree covers the same containment tree +`NodeStore` maintains. A Merkle tree is a hash tree where each +parent's hash is computed from its children's hashes. So a single +top-level root hash can attest to the exact contents of an entire +subtree. Changing any leaf changes every hash on the path up to the +root. A caller must invoke the rollup by hand. It does not run on +its own after every write. Computing it inline on every drawer +capture would be +costly. The cost would be proportional to room size, on every write. +This would peg the CPU during a bulk import. + +`computeRoomMerkleRoot(roomNodeId:)` hashes a room's live drawers. It +reads each row's stored `content_hash` column when present, written +by a separate hash-on-write hook. It computes a leaf hash on demand +from the drawer's content when the column is absent. It always +excludes two kinds of rows from the snapshot. The first kind is +tombstoned rows. These rows are irreversibly deleted. The second kind +is withdrawn rows, at state raw value 18. A withdrawn drawer's +content has been retracted by the user. So it must not remain +retrievable through a content-attesting snapshot. +`computeEstateOrWingMerkleRoot(parentNodeId:)` hashes a set of +children's already-computed roots. The same function serves two +levels. At the wing level, it hashes room roots. At the estate level, +it hashes wing roots. `rollupMerkleRoots(roomNodeId:now:)` runs all +three levels for one changed room. `recomputeAllMerkleRoots(now:)` is +the full bottom-up recompute, used after bulk import, migration, or +corruption recovery. `rollupAllMerkleRoots(now:)` is a documented +alias of the full recompute, named for the batch-capture reindex +path. `createSnapshot(label:now:additionalAttestations:)` recomputes +the full tree, so a snapshot is never taken against stale roots. It +then writes an attestation row for the estate root and every wing, +plus any attestations a higher composition-layer kit supplies. ## Manifest.swift This file provides `ManifestKey`, the eighteen required and seven -optional typed keys of the v1 manifest key-value table, and -`ManifestValues`, the typed, read-only snapshot `DrawerStore.readManifest()` -produces. The manifest table itself is a plain key-value store; this -file is what turns "some strings in a table" into a fully typed record -a caller can rely on. `ManifestKey.ed25519PublicKey` and -`ed25519PrivateKeyWrapped` are the two identity-related keys: -`ed25519PublicKey` is the estate's federation public key, safe to store -here because a public key has no confidentiality requirement, while -`ed25519PrivateKeyWrapped` is a reserved, never-written seam kept only -for backward read-compatibility with an estate opened before the -Keychain migration — the private key itself lives exclusively in +optional typed keys of the v1 manifest key-value table. It also +provides `ManifestValues`, the typed, read-only snapshot +`DrawerStore.readManifest()` produces. The manifest table itself is +a plain key-value store. This file turns a table of plain strings +into a fully typed record a caller can rely on. +`ManifestKey.ed25519PublicKey` and `ed25519PrivateKeyWrapped` are the +two identity-related keys. `ed25519PublicKey` is the estate's +federation public key. It is safe to store here, because a public +key has no confidentiality requirement. `ed25519PrivateKeyWrapped` is +different: a reserved, never-written seam. It exists only for +backward read-compatibility, with an estate opened before the +Keychain migration. The private key itself lives exclusively in `EstateIdentityKeyStore`. ## DefaultWings.swift -This file provides the seven default wing -definitions seeded into a fresh estate at provision time, plus three -related constants: `defaultWingName` ("Agentic Memory," the wing -`capture` uses when a caller supplies no explicit wing), `hintRoom` -("AI_Charter_Hint," the room each seeded wing's hint memory lives in), -and `hintUDCCode` ("001," the UDC Knowledge-class code stamped on hint -drawers). Each `WingDefinition` pairs a wing name with a plain-language -hint describing the wing's role — Agentic Memory, User Canon, Source -Corpus, Personal, Professional, Projects, and Temp — seeded as an -ordinary, fully recallable, user-deletable drawer so a fresh agent has -a working orientation to its own memory structure from the first -session. The set is explicitly a suggestion, not a fixed schema: an -agent may create any additional wing it needs. +This file provides the seven default wing definitions. These are +seeded into a fresh estate at provision time. It also provides three +related constants. `defaultWingName` is "Agentic Memory," the wing +`capture` uses when a caller supplies no explicit wing. `hintRoom` is +"AI_Charter_Hint," the room each seeded wing's hint memory lives in. +`hintUDCCode` is "001," the UDC Knowledge-class code stamped on hint +drawers. Each `WingDefinition` pairs a wing name with a +plain-language hint describing the wing's role. The seven wings are: +Agentic Memory, User Canon, Source Corpus, Personal, Professional, +Projects, and Temp. Each hint is seeded as an ordinary, fully +recallable, user-deletable drawer. So a fresh agent has a working +orientation to its own memory structure, from its first session. The +set is only a suggestion, not a fixed schema. An agent may +create any extra wing it needs. ## RecallStream.swift This file provides `RecallStream`, the paged async sequence `Estate.recall` returns. Iterating a `RecallStream` produces one -`RecallPage` at a time — the first page synchronously on the first -`next()` call, later pages lazily — with `isLast` true only on the -final page, so a caller can drive a uniform `for await` loop without -special-casing an empty result: a genuinely empty corpus still emits -exactly one page with zero rows and `isLast == true`. - -`degradedStages` is the channel by which a failed internal read (the -bounded corpus scan, the room-fingerprint enumeration, a surviving -room's drawer read, or the bitmap evaluator itself) becomes observable -to a caller. `recall` never throws — spec §7.8.1 requires it stay -non-throwing — so an internal failure would otherwise be -indistinguishable from a genuinely empty result; a non-empty -`degradedStages` array is what lets a caller (chiefly GeniusLocusKit's -`RecallDirector`) tell the two apart. `AsyncIterator.hydrate(_:)` -applies the requested `HydrationLevel` per page: `.bitmapOnly` rebuilds -each drawer with `content` blanked to the empty string while preserving -every bitmap and metadata field; `.structured` and `.full` pass rows -through unchanged, because the content-stripping decision for -`.structured` was already made upstream, at the storage-fetch layer, -not here. +`RecallPage` at a time. The first page arrives synchronously, on the +first `next()` call. Later pages arrive lazily. `isLast` is true only +on the final page. So a caller can drive a uniform `for await` loop, +without special-casing an empty result. A genuinely empty corpus +still emits exactly one page, with zero rows and `isLast == true`. + +`degradedStages` is the channel by which a failed internal read +becomes observable to a caller. A failed read might come from several places: the bounded corpus +scan, the room-fingerprint enumeration, a room's drawer read, or the +bitmap evaluator itself. `recall` never throws. +Section 7.8.1 of the specification requires it stay non-throwing. So +an internal failure would otherwise be indistinguishable from a +genuinely empty result. A non-empty `degradedStages` array is what +lets a caller, chiefly GeniusLocusKit's `RecallDirector`, tell the +two apart. `AsyncIterator.hydrate(_:)` applies the requested +`HydrationLevel` per page. At `.bitmapOnly`, it rebuilds each drawer +with `content` blanked to the empty string, while preserving every +bitmap and metadata field. At `.structured` and `.full`, rows pass +through unchanged. The content-stripping decision for `.structured` +was already made upstream, at the storage-fetch layer, not here. ## RecallTraceItem.swift This file provides `RecallTraceItem`, one record of a single drawer -returned by a recall operation — the substrate for the "later -two-source reward" mechanism a dreaming daemon uses to distinguish rows -a user actually acted on from rows that were returned but ignored. The -`used` flag is bit 0 of `operationalBitmap`, decoded through a -computed property rather than stored as a separate `Bool` column, so -the type carries no stored boolean fields at all — a convention every -bitmap-backed LocusKit noun follows. `score`, when present, is the -recall's own similarity score for the row; `nil` means the recall that -produced this trace carried no score (for example, an ordinary -ordered-by-capture-time query). +returned by a recall operation. This is the substrate for the "later +two-source reward" mechanism. A dreaming daemon uses this mechanism +to distinguish rows a user acted on from rows that were +returned but ignored. The `used` flag is bit 0 of `operationalBitmap`, +decoded through a computed property rather than stored as a separate +`Bool` column. So the type carries no stored boolean fields at all. +Every bitmap-backed LocusKit noun follows this convention. `score`, +when present, is the recall's own similarity score for the row. A +`nil` score means the recall that produced this trace carried no +score, for example an ordinary ordered-by-capture-time query. ## AuditTypes.swift -This file provides `BitmapState`, the snapshot of a row's three bitmap -columns at a specific historical HLC, returned by -`Estate.bitmapState(rowID:asOf:)`. It exists as a small, focused return -type precisely because the reconstruction that produces it — -`AuditLogFold.projectStateAt`, folding a row's sealed audit events -forward in HLC order — is a SubstrateLib primitive that returns its -result in a different shape; `BitmapState` is LocusKit's own -public-facing wrapper naming the three bitmaps explicitly rather than -exposing the substrate's internal projection type directly. +This file provides `BitmapState`, the snapshot of a row's three +bitmap columns at a specific past HLC. `Estate.bitmapState(rowID:asOf:)` +returns this snapshot. `BitmapState` exists as a small, focused +return type, for a precise reason. The reconstruction that produces +it, `AuditLogFold.projectStateAt`, is a SubstrateLib primitive. It +folds a row's sealed audit events forward in HLC order. Yet it +returns its result in a different shape. `BitmapState` is LocusKit's +own public-facing wrapper. It names the three bitmaps by hand, +rather than exposing the substrate's internal projection type +directly. ## Summaries.swift This file provides `WingSummary` and `RoomSummary`, the two small -aggregate types `DrawerStore.listWings`/`listRooms` produce. Both are -computed projections over the current drawer set — LocusKit has no -separate `wings` or `rooms` table; wing and room identity lives -entirely in the `nodes` table, and a summary's drawer and room counts -are simply whatever the live query finds at the moment it runs, not a -maintained running total. +summary types `DrawerStore.listWings` and `listRooms` produce. Both +are computed projections over the current drawer set. LocusKit has +no separate `wings` or `rooms` table. Wing and room identity lives +entirely in the `nodes` table. A summary's drawer and room counts are +simply whatever the live query finds, at the moment it runs. They are +not a maintained running total. ## LocusKitTelemetry.swift -This file provides LocusKit's opt-in -telemetry emission functions, wired through `IntellectusLib.Intellectus.report(_:)`. -Every emit function here follows the same three rules stated in the -file's header: the reported value is always an `@autoclosure` that is -never evaluated when monitoring is disabled (the off-path cost is one -atomic boolean load and a branch — no lock, no allocation); the `now` -timestamp is always caller-supplied, never read from a clock inside the -function, preserving IntellectusLib's own determinism contract; and the -metric namespace follows the fleet-wide `..` -convention (`locuskit.drawer.capture_latency_ms`, -`locuskit.kgfact.add_count`, `locuskit.gate.reject_count`, and so on). +This file provides LocusKit's opt-in telemetry emission functions. +They are wired through `IntellectusLib.Intellectus.report(_:)`. Every +emit function here follows the same three rules, stated in the +file's header. First, the reported value is always an +`@autoclosure`. It is never evaluated when monitoring is disabled. +The off-path cost is one atomic boolean load and a branch: no lock, +no allocation. Second, the `now` timestamp is always caller-supplied, +never read from a clock inside the function. This preserves +IntellectusLib's own determinism contract. Third, the metric +namespace follows the fleet-wide `..` convention. +Examples include `locuskit.drawer.capture_latency_ms`, +`locuskit.kgfact.add_count`, and `locuskit.gate.reject_count`. `emitDrawerCapture`, `emitDrawerQuery`, `emitKGFactAdd`, `emitKGFactQuery`, `emitTunnelAdd`, `emitGateAdmit`, and -`emitGateReject` are called from the corresponding `DrawerStore` -methods at the exact moment each operation completes (or, for the gate -functions, at the exact moment `AuditGate.admit` returns). Because -telemetry is purely additive — it is never consulted by any control -flow that decides what a method returns — every store method's -functional behavior is byte-identical whether monitoring is enabled or -not. +`emitGateReject` share one pattern. Each is called from its +corresponding `DrawerStore` method. Each call happens at the exact +moment its operation completes. For the gate functions, the call +happens when `AuditGate.admit` returns instead. Telemetry is purely additive. It is never +consulted by any control flow that decides what a method returns. So +every store method's functional behavior is byte-identical, whether +monitoring is enabled or not. ## Rust Port and Conformance -The `rust/` directory contains the second leg of the kit: fifty-seven -source and test files under `rust/src/` and `rust/tests/` mirroring the -Swift implementation file-for-file (`drawer.rs`, `drawer_store.rs`, -`bitmap_evaluator.rs`, `merkle_rollup.rs`, `estate.rs`, and so on), -plus three concrete `DrawerStore` backends the Swift side does not -carry directly — `drawer_store_inmemory.rs`, `drawer_store_sqlite.rs`, -and `drawer_store_postgres.rs`, all built over a shared, storage-agnostic -`DrawerStoreCore`. Conformance tests under `rust/tests/` (adjective and -operational bitmap conformance, provenance bitmap conformance, corrupt -readback, Merkle rollup, outline proofs, recall pruning, temporal -reads, and the LP0 fixed-vector suite) gate byte-for-byte agreement with -the Swift implementation on every bitmap layout, every fingerprint -derivation, and every state-transition legality decision. When you -change a bitmap layout, a fingerprint derivation, or a transition rule -on either leg, mirror the change on the other and run both test suites -— the fixtures and conformance tests are the contract, not a -convenience. +The `rust/` directory contains the second leg of the kit. It holds +fifty-seven source and test files, under `rust/src/` and +`rust/tests/`. These files mirror the Swift implementation +file-for-file: `drawer.rs`, `drawer_store.rs`, `bitmap_evaluator.rs`, +`merkle_rollup.rs`, `estate.rs`, and more. The Rust side also carries +three concrete `DrawerStore` backends the Swift side does not +carry: `drawer_store_inmemory.rs`, `drawer_store_sqlite.rs`, and +`drawer_store_postgres.rs`. All three build over a shared, +storage-agnostic `DrawerStoreCore`. + +Conformance tests under `rust/tests/` gate byte-for-byte agreement +with the Swift implementation. These tests cover adjective and +operational bitmap conformance, provenance bitmap conformance, +corrupt readback, Merkle rollup, outline proofs, recall pruning, +temporal reads, and the LP0 fixed-vector suite. They gate agreement +on every bitmap layout, every fingerprint derivation, and every +state-transition legality decision. Suppose you change a bitmap layout, a fingerprint derivation, or a +transition rule on either leg. Mirror that change on the other leg. +Then run both test suites. The fixtures +and conformance tests are the contract, not a convenience. diff --git a/packages/kits/LocusKit/docs/OVERVIEW.md b/packages/kits/LocusKit/docs/OVERVIEW.md index e7d4fc2..063b643 100644 --- a/packages/kits/LocusKit/docs/OVERVIEW.md +++ b/packages/kits/LocusKit/docs/OVERVIEW.md @@ -107,198 +107,238 @@ sources: ## What This Kit Does -LocusKit is the storage substrate for a MOOTx01 estate. An estate is one -user's complete memory store in MOOTx01, an on-device AI memory system -that stores what an AI observes over time and helps the AI recall it -later. LocusKit is the layer that actually holds the memories on disk -and lets a caller file them, find them, and change their standing over -time. - -LocusKit is a kit, not a library. A library (lib) is a folder of related -code files that does one job well; a kit is a larger package that -composes libraries into a subsystem. Kits may depend on libs, but libs -never depend back on kits. LocusKit depends on several libs — most -importantly PersistenceKit for storage, SubstrateLib and SubstrateTypes -for the write-gate math, SubstrateML for classification math, and -LatticeLib for a taxonomic lookup — and composes them into the estate's -spatial memory surface. A higher kit, GeniusLocusKit, builds on top of -LocusKit; it is out of scope for this document. - -The word "MemPalace" names the spatial metaphor LocusKit implements. -Content lives in drawers, drawers sit in rooms, rooms sit in wings, and -wings sit in one estate. This document calls that three-level structure -the containment tree. +LocusKit is the storage substrate for a MOOTx01 estate. An estate is +one user's complete memory store. MOOTx01 is an on-device AI memory +system. It stores what an AI observes over time. It helps the AI +recall that memory later. LocusKit is the layer that holds memories on +disk. It lets a caller file a memory, find it, and change its standing +over time. + +LocusKit is a kit, not a library. A library, or lib, is a folder of +related files that does one job well. A kit is a larger package. A kit +composes several libraries into one subsystem. Kits may depend on +libs. Libs never depend back on kits. + +LocusKit depends on several libs. `PersistenceKit` handles storage. +`SubstrateLib` and `SubstrateTypes` supply the write-gate math. +`SubstrateML` supplies classification math. `LatticeLib` supplies a +taxonomic lookup. LocusKit composes these libs into the estate's +spatial memory surface. A higher kit, `GeniusLocusKit`, builds on top +of LocusKit. This document does not cover `GeniusLocusKit`. + +The word `MemPalace` names the spatial metaphor LocusKit implements. +Content lives in drawers. Drawers sit in rooms. Rooms sit in wings. +Wings sit in one estate. This document calls that three-level +structure the containment tree. ## The Problem It Solves -An AI's memory needs more than a place to put text. It needs to know -whether a memory is still believed, how sensitive it is, whether it can -leave the device, how much to trust it, and where it came from. It needs -to change its mind about a memory — mark it superseded, contested, -confirmed, or gone — without losing the history of that change. It needs -to find memories quickly by many different criteria without scanning -every row every time. And it needs all of this to survive a crash -midway through a write. - -LocusKit answers each of these needs with a specific mechanism: - -- **Standing.** Every memory carries three packed 64-bit numbers — - bitmaps — that record its adjective state (is it active, superseded, - contested, accepted, rejected, or gone), its operational facts (how it - was captured, what kind of content it is), and its provenance (where - it came from and how confident the system is in it). A bitmap is a - fixed-size number where different ranges of bits, called fields, each - carry an independent small value. Packing many fields into one number - keeps a row's standing in a single machine word instead of a dozen - separate boolean columns. -- **Safe change over time.** A memory's state does not move freely - from any value to any other. It follows a finite set of legal moves — - for example, an active memory can become contested, but a rejected - memory cannot become accepted. LocusKit routes every state-changing - write through a write gate (`AuditGate`, owned by SubstrateLib) that - checks the move is legal and then writes one sealed, tamper-evident - audit event recording exactly what changed. The audit event, not the - live row, is the source of truth; the live row is a cached projection - of the latest audit event for fast reads. Row and log update together - inside one transaction, so every value ever read back from a row - matches a genuine sealed event in its audit log — the row is never - observed one step ahead of its own history. -- **Fast, layered search.** A caller expresses what it wants as a - chain of named filters — "currently believed," "in this room," - "captured after this date" — rather than raw bitmask arithmetic. - LocusKit compiles that chain through four progressively more - expensive tiers: a bitmap tier (near-free integer comparisons), a - structured tier (room, wing, lattice checks), a content tier - (substring search in the verbatim text), and an ordering pass. Before - any of that, a per-container fingerprint check can rule out whole - rooms or wings without touching a single row. +An AI's memory needs more than a place to store text. It must know if +a memory is still believed. It must know how sensitive a memory is. It +must know if a memory can leave the device. It must know how much to +trust a memory and where the memory came from. It needs to change its +mind about a memory. It must mark a memory as superseded, contested, +confirmed, or gone. It must do this without losing the history of that +change. It needs to find memories fast, by many criteria. It must do +this without scanning every row each time. It needs all of this to +survive a crash partway through a write. + +LocusKit answers each of these needs with one mechanism per need. + +- **Standing.** Every memory carries three packed 64-bit numbers. Each + number is called a bitmap. One bitmap records the memory's adjective + state: active, superseded, contested, accepted, rejected, or gone. A + second bitmap records operational facts. It records the capture + method and the content kind. A third bitmap records provenance: + where the memory came from and how much the system trusts it. A + bitmap is a fixed-size number. Different ranges of bits inside it + are called fields. Each field holds one small, independent value. + Packing many fields into one number keeps a memory's standing in a + single machine word. Otherwise the record would need a dozen + separate columns. +- **Safe change over time.** A memory's state does not move freely. It + can move only along a fixed set of legal paths. For example, an + active memory can become contested. A rejected memory cannot become + accepted. LocusKit sends every state-changing write through a write + gate, called `AuditGate`. `AuditGate` is owned by SubstrateLib. The + gate checks that the move is legal. It then writes one sealed, + tamper-evident audit event. That event records exactly what changed. + The audit event, not the live row, is the source of truth. The live + row is just a cached copy of the latest audit event, kept for fast + reads. Row and log update together inside one transaction. So every + value read back from a row matches a real sealed event in its audit + log. The row is never observed one step ahead of its own history. +- **Fast, layered search.** A caller expresses a query as a chain of + named filters. Filters name a concern in plain language, such as + currently believed, in this room, or captured after this date. + LocusKit compiles that chain through four tiers, from cheap to + costly. The first tier is the bitmap tier. It runs near-free integer + checks. The second tier is the structured tier. It checks room, + wing, and lattice fields. The third tier is the content tier. It + searches the verbatim text for a substring. The fourth tier is the + ordering pass. It sorts the surviving rows. Before any of that, a + per-container fingerprint check can rule out whole rooms or wings. + It does this without loading a single row. - **Crash safety.** Every state-changing write happens inside one - database transaction that both updates the live row and appends the - audit event. If the process dies partway through, the transaction - either committed both changes or neither — never a live row that says - one thing and an audit log that says another. + database transaction. That transaction updates the live row and + appends the audit event in one step. If the process dies partway + through, the transaction commits both changes or neither. The live + row and the audit log can never disagree about what happened. ## How It Works ### Nine kinds of rows, one set of patterns -LocusKit stores nine kinds of rows, or nouns: `Drawer` (verbatim -content), `Tunnel` (a typed link between two locations), `DiaryEntry` -(a first-person agent record), `KGFact` (a subject-predicate-object -triple extracted from a drawer), `Proposal` (a suggested change awaiting -confirmation), `Association` (a graph edge recording that two rows -belong together), `LearnedReference` (an external reference brought in -by the `learn` verb), `SourceCatalogEntry` (the durable record of where -a learned reference came from), and `Node` (an entry in the containment -tree — the estate root, a wing, or a room). Most of these nouns share -the three-bitmap pattern described above; each has its own file -decoding its own bitmap layout (for example, `DrawerOperational.swift` -decodes `Drawer.operationalBitmap`, and `KGFactOperational.swift` -decodes `KGFact.operationalBitmap`). +LocusKit stores nine kinds of rows. This document calls each kind a +noun. The nine nouns are: `Drawer`, `Tunnel`, `DiaryEntry`, `KGFact`, +`Proposal`, `Association`, `LearnedReference`, `SourceCatalogEntry`, +and `Node`. `Drawer` holds verbatim content. `Tunnel` is a typed link +between two locations. `DiaryEntry` is a first-person agent record. +`KGFact` is a subject-predicate-object triple pulled from a drawer. +`Proposal` is a suggested change waiting for confirmation. +`Association` is a graph edge that records two rows belonging +together. `LearnedReference` is an external reference brought in by +the `learn` verb. `SourceCatalogEntry` is the durable record of where +a learned reference came from. `Node` is an entry in the containment +tree: the estate root, a wing, or a room. + +Most of these nouns share the same three-bitmap pattern described +above. Each noun has its own file that decodes its own bitmap layout. +For example, `DrawerOperational.swift` decodes +`Drawer.operationalBitmap`. `KGFactOperational.swift` decodes +`KGFact.operationalBitmap`. ### The containment tree Estates used to store a drawer's wing and room as two plain text -columns. LocusKit now stores them as a three-level tree of `Node` rows: -the estate root at depth 0, wings at depth 1, and rooms at depth 2. A -drawer references its room by a foreign key, `parentNodeId`, rather than -by name. `NodeStore` resolves a wing/room name pair to a node id, -creating the node on first use and returning the existing one -thereafter — so filing a drawer into "Personal / Health" always lands in -the same room node, no matter how many times the name is used. This -containment tree is what a Merkle rollup (below) walks bottom-up, and -what lets a room be renamed without touching every drawer inside it. +columns. LocusKit now stores them as a three-level tree of `Node` +rows. The estate root sits at depth zero. Wings sit at depth one. +Rooms sit at depth two. A drawer references its room through a foreign +key, `parentNodeId`, rather than by name. + +`NodeStore` resolves a wing-and-room name pair to a node id. It +creates the node on first use. It returns the existing node on every +later use. So filing a drawer into "Personal / Health" always lands in +the same room node, no matter how many times the name is used. A +Merkle rollup, described below, walks this tree from bottom to top. +Renaming a room never requires touching every drawer inside it. ### The write gate `DrawerStore` is the actor that owns every table. Nearly every write -that changes a row's standing — capturing a drawer, moving its state, -editing an adjective field — is a "gated" write: `DrawerStore` reads the -row's current bitmaps, hands them to `AuditGate.admit` along with the -proposed change, and either receives back a sealed `AuditEvent` to -persist or a rejection naming the rule that was violated (an illegal -state transition, or a forbidden bitmap combination such as sensitivity -`secret` paired with exportability `public`). `DrawerStateValidator` and -`ForbiddenCombinationValidator` exist mainly as documentation of these -same rules, since the live enforcement now happens inside the gate. +that changes a row's standing is a gated write. A gated write happens +when a caller captures a drawer, moves its state, or edits an +adjective field. + +For a gated write, `DrawerStore` reads the row's current bitmaps. It +hands those bitmaps to `AuditGate.admit`, along with the proposed +change. `AuditGate.admit` returns one of two things: a sealed +`AuditEvent` to store, or a rejection naming the broken rule. A broken +rule might be an illegal state transition. It might also be a +forbidden bitmap combination, such as sensitivity `secret` paired with +exportability `public`. + +`DrawerStateValidator` and `ForbiddenCombinationValidator` mainly +document these same rules. The live enforcement now happens inside the +gate itself. ### The recall pipeline -`Estate.recall` is the read side. A caller builds a `RecallFrame` -carrying a chain of `Filter` values — the recall filter algebra. No -`Filter` case exposes a raw bit position; each names a domain concern -("trustworthy," "in this wing," "captured after date X"). `Estate.recall` -first asks `ContainerFingerprintStore` whether any room or wing can be -ruled out from its cached per-container OR-fingerprint — a bitwise-OR -of every active drawer's three bitmaps in that container — before -fetching a single row. Surviving rows then pass through -`BitmapEvaluator`, which evaluates the filter chain in four tiers -(bitmap, structured, content, ordering) and can reconstruct a row's -historical bitmap state at a past point in time by folding its audit -log forward (`AuditLogFold.projectStateAt`, a SubstrateLib primitive). +`Estate.recall` is the read side of LocusKit. A caller builds a +`RecallFrame`. The frame carries a chain of `Filter` values. This +chain is the recall filter algebra. No `Filter` case exposes a raw bit +position. Each `Filter` case names a domain concern instead, such as +trustworthy, in this wing, or captured after a date. + +`Estate.recall` first asks `ContainerFingerprintStore` whether a room +or wing can be ruled out. It checks a cached per-container +fingerprint, a bitwise OR of every active drawer's three bitmaps in +that container. This check happens before LocusKit fetches a single +row. + +Rows that survive this check then pass through `BitmapEvaluator`. +`BitmapEvaluator` runs the filter chain in four tiers: bitmap, +structured, content, and ordering. `BitmapEvaluator` can also rebuild +a row's bitmap state at a past point in time. It does this by folding +the row's audit log forward. The folding function is +`AuditLogFold.projectStateAt`, a SubstrateLib primitive. ### Structural similarity and integrity -Two more subsystems ride alongside the write/read core. `DrawerFingerprint` -derives a 256-bit structural fingerprint for a drawer from its bitmaps, -lattice anchor, lineage, and timing, using SubstrateLib's SimHash -machinery; these fingerprints feed both `ContainerFingerprintStore`'s -pruning aggregates and a separate bundle-algebra subsystem -(`NodeBundleStore`, `BundleMaterializer`) that folds many fingerprints -into one compact count-vector per room or wing. `MerkleRollup` computes -a content-integrity hash tree bottom-up over the same containment -tree — room, then wing, then estate — so a snapshot can attest that its -content has not silently changed. +Two more subsystems sit alongside the write-and-read core. +`DrawerFingerprint` derives a 256-bit structural fingerprint for each +drawer. It builds this fingerprint from the drawer's bitmaps, lattice +anchor, lineage, and timing. `DrawerFingerprint` uses SubstrateLib's +SimHash machinery to do this. These fingerprints feed two systems. The +first is `ContainerFingerprintStore`'s pruning aggregates. The second +is a bundle-algebra subsystem, made of `NodeBundleStore` and +`BundleMaterializer`. This subsystem folds many fingerprints into one +compact count vector per room or wing. + +`MerkleRollup` computes a content-integrity hash tree from the bottom +up. It walks the same containment tree: room, then wing, then estate. +This lets a snapshot attest that its content has not silently changed. ### The nine verbs -`EstateVerbs.swift` adds nine verb methods to `Estate`: `capture` -(file a new drawer or tunnel), `recall` (the pipeline above), `mutate` -(move a row's standing along a named axis), `withdraw` (retract a -drawer), `expunge` (hard-delete a drawer's content while preserving the -audit trail), `reanchor` (move a drawer to a new room or lattice -position), `learn` (bring in an external reference, grounded to its -source's own lattice anchor), `propose` (record a suggested change -awaiting confirmation), and `associate` (record a graph edge between -two rows). Every verb takes a named "frame" struct (`CaptureFrame`, -`RecallFrame`, and so on) as its argument, so no raw bitmap value ever -crosses the public boundary. +`EstateVerbs.swift` adds nine verb methods to `Estate`. The nine verbs +are: `capture`, `recall`, `mutate`, `withdraw`, `expunge`, `reanchor`, +`learn`, `propose`, and `associate`. + +`capture` files a new drawer or tunnel. `recall` runs the pipeline +described above. `mutate` moves a row's standing along a named axis. +`withdraw` retracts a drawer. `expunge` hard-deletes a drawer's +content but keeps its audit trail. `reanchor` moves a drawer to a new +room or lattice position. `learn` brings in an external reference, +grounded to its source's own lattice anchor. `propose` records a +suggested change awaiting confirmation. `associate` records a graph +edge between two rows. + +Every verb takes a named frame struct as its argument, such as +`CaptureFrame` or `RecallFrame`. So no raw bitmap value ever crosses +the public boundary. ## How the Pieces Fit -Figure 1 shows the kit's topology — its major parts and how a write and -a read move through them. +Figure 1 shows the kit's topology. It shows the major parts and how a +write and a read move through each one. ![Figure 1. Topology of LocusKit](topology.svg) -*Figure 1. Topology of LocusKit. A capture (left) flows through the -write gate into the drawers table and, off the write path, into the -container-fingerprint and Merkle-rollup subsystems. A recall (right) -flows through fingerprint pruning and the four-tier bitmap evaluator. -Dashed boxes mark external kits and libs LocusKit depends on but does -not own.* - -`Estate` is the single public entry point: it owns one `DrawerStore` -(the actor holding every table), one `ContainerFingerprintStore` (the -pruning aggregates), and one `NodeStore` (the containment tree), and it -exposes the nine verbs plus a set of read-only pass-throughs -(`allDrawers`, `allKGFacts`, `tunnelsFromWing`, and so on) that GLK and -other higher-level consumers use instead of reaching into `DrawerStore` -directly. `DrawerStore` itself is declared with an `internal`, not -`public`, add-path (`addDrawer`) so that the only sanctioned way to add -a drawer from outside the file is through `Estate.addDrawerCovered`, -which bundles the row insert with the container-fingerprint update — a -structural guarantee that a drawer can never be captured without also -updating its container's pruning aggregate. +*Figure 1. Topology of LocusKit. On the left, a capture flows through +the write gate into the drawers table. Off the write path, it also +updates the container-fingerprint and Merkle-rollup subsystems. On the +right, a recall flows through fingerprint pruning and the four-tier +bitmap evaluator. Dashed boxes mark external kits and libs that +LocusKit depends on but does not own.* + +`Estate` is the single public entry point to LocusKit. It owns one +`DrawerStore`, the actor holding every table. It owns one +`ContainerFingerprintStore`, the pruning aggregates. It owns one +`NodeStore`, the containment tree. `Estate` exposes the nine verbs. It +also exposes read-only pass-throughs, such as `allDrawers`, +`allKGFacts`, and `tunnelsFromWing`. GLK and other higher-level +consumers use these pass-throughs instead of reaching into +`DrawerStore` directly. + +`DrawerStore` itself keeps its add path internal, not public. Its add +method, `addDrawer`, stays internal. So the only sanctioned way to add +a drawer from outside the file is through `Estate.addDrawerCovered`. +That method bundles the row insert with the container-fingerprint +update. This is a structural guarantee: a drawer can never be captured +without also updating its container's pruning aggregate. ## What Ships in the Package -The package ships the Swift sources under `Sources/LocusKit/` and a -parallel Rust port under `rust/`. The Swift package depends on five -sibling libs — `SubstrateLib`, `SubstrateTypes`, `SubstrateKernel`, and -`SubstrateML` for the write-gate and fingerprint math, `PersistenceKit` -for the storage abstraction, `IntellectusLib` for opt-in telemetry, and -`LatticeLib` for the pinned Q-ID ancestor closure `DrawerFingerprint` -hashes into its lattice block. LocusKit ships no pinned data artifacts -of its own; its schema (`LocusKitSchema`) is declared entirely in -PersistenceKit primitives and created fresh at estate open. +The package ships the Swift sources under `Sources/LocusKit/`. It also +ships a parallel Rust port under `rust/`. The Swift package depends on +five sibling libs. Four of them supply write-gate and fingerprint +math: `SubstrateLib`, `SubstrateTypes`, `SubstrateKernel`, and +`SubstrateML`. `PersistenceKit` supplies the storage abstraction. +`IntellectusLib` supplies opt-in telemetry. `LatticeLib` supplies the +pinned Q-ID ancestor closure that `DrawerFingerprint` hashes into its +lattice block. + +LocusKit ships no pinned data artifacts of its own. Its schema, +`LocusKitSchema`, is declared entirely in `PersistenceKit` primitives. +The schema is created fresh each time an estate opens. diff --git a/packages/kits/LocusKit/docs/topology.svg b/packages/kits/LocusKit/docs/topology.svg index 49d80ed..b2048d7 100644 --- a/packages/kits/LocusKit/docs/topology.svg +++ b/packages/kits/LocusKit/docs/topology.svg @@ -103,7 +103,7 @@ sources: AuditGate - SubstrateLib — legal-transition + SubstrateLib : legal-transition PersistenceKit diff --git a/packages/kits/VectorKit/docs/AGENT_MAP.md b/packages/kits/VectorKit/docs/AGENT_MAP.md index b67a508..8ff7873 100644 --- a/packages/kits/VectorKit/docs/AGENT_MAP.md +++ b/packages/kits/VectorKit/docs/AGENT_MAP.md @@ -43,163 +43,163 @@ sources: blob: 3c7fe4a19eba1142ac82a993cee0e7660a4ffdce --- -# AGENT_MAP — VectorKit +# AGENT_MAP | VectorKit PURPOSE: on-device embedding generation (`EmbeddingProvider` seam) + model-tagged vector storage (`VectorStore`, PersistenceKit-backed) + dual-lane nearest-neighbour search: binary Hamming (Lane A `BruteForceIndex` oracle / Lane B `MIHIndex` sub-linear exact, promoted at `mihThreshold`) and float cosine/l2/dot (`FloatBruteForceIndex`, one index per modelID) + ColBERT MaxSim late-interaction scorer (`MaxSimScorer`, standalone, not wired into VectorStore). -DEPS: imports EngramLib (Engram type, Hamming kernel via EngramLib.distances/Session — I-7 absolute), SubstrateML (FloatSimHash.project), SubstrateTypes, PersistenceKit (Storage/RowStore/BlobStore, product "PersistenceKit"), IntellectusLib (Intellectus.report telemetry, no-op when disabled). Test target additionally depends on PersistenceKitInMemory, PersistenceKitSQLite. Imported by: CorpusKit / CorpusKitProviders (concrete text embedding providers built on FloatSimHashEmbeddingProvider), GeniusLocusKit (destroyAllVectors as part of estate teardown). Rust port in rust/ mirrors every file (vector_store.rs, engine/{brute_force,mih,float_brute_force,max_sim,resident,resident_store,key,payload,hit,metric,seam}.rs, embedding_provider.rs, simhash_embedding_provider.rs, error.rs); no shared cross-language fixture file — conformance rests on both ports implementing the documented algorithms identically (colex enumeration, sidecar byte layout, budget arithmetic). Float lane is explicitly NOT four-way bit-identical (documented, not a gap). +DEPS: imports EngramLib (Engram type, Hamming kernel via EngramLib.distances/Session | I-7 absolute), SubstrateML (FloatSimHash.project), SubstrateTypes, PersistenceKit (Storage/RowStore/BlobStore, product "PersistenceKit"), IntellectusLib (Intellectus.report telemetry, no-op when disabled). Test target additionally depends on PersistenceKitInMemory, PersistenceKitSQLite. Imported by: CorpusKit / CorpusKitProviders (concrete text embedding providers built on FloatSimHashEmbeddingProvider), GeniusLocusKit (destroyAllVectors as part of estate teardown). Rust port in rust/ mirrors every file (vector_store.rs, engine/{brute_force,mih,float_brute_force,max_sim,resident,resident_store,key,payload,hit,metric,seam}.rs, embedding_provider.rs, simhash_embedding_provider.rs, error.rs); no shared cross-language fixture file | conformance rests on both ports implementing the documented algorithms identically (colex enumeration, sidecar byte layout, budget arithmetic). Float lane is explicitly NOT four-way bit-identical (documented, not a gap). ENTRY POINTS (most callers need only these): -- VectorStore.swift:451 `VectorStore.addVector(itemID:engram:modelID:modelVersion:filedAt:)` — write one binary vector -- VectorStore.swift:1134 `VectorStore.findNearest(probe:modelID:limit:) -> [VectorMatch]` — binary Hamming k-NN -- VectorStore.swift:1223 `VectorStore.findNearestFloat(probe:modelID:limit:) -> [VectorMatch]` — float cosine k-NN -- FloatSimHashEmbeddingProvider.swift:63 `FloatSimHashEmbeddingProvider.embed(_:) -> Engram` — text → fingerprint via injected inference + FloatSimHash +- VectorStore.swift:451 `VectorStore.addVector(itemID:engram:modelID:modelVersion:filedAt:)` | write one binary vector +- VectorStore.swift:1134 `VectorStore.findNearest(probe:modelID:limit:) -> [VectorMatch]` | binary Hamming k-NN +- VectorStore.swift:1223 `VectorStore.findNearestFloat(probe:modelID:limit:) -> [VectorMatch]` | float cosine k-NN +- FloatSimHashEmbeddingProvider.swift:63 `FloatSimHashEmbeddingProvider.embed(_:) -> Engram` | text → fingerprint via injected inference + FloatSimHash ## Symbol Table ### Module surface -- VectorKit.swift:1 — namespace/header only; no types. Consumers `import EngramLib` separately for `Engram` (not re-exported). +- VectorKit.swift:1 | namespace/header only; no types. Consumers `import EngramLib` separately for `Engram` (not re-exported). -### Errors — VectorKitError.swift -- :5 `enum VectorKitError: Error, Sendable, Equatable` — concrete cases, never optional+log +### Errors | VectorKitError.swift +- :5 `enum VectorKitError: Error, Sendable, Equatable` | concrete cases, never optional+log - :9 `.embeddingFailed(String)` / :13 `.modelUnavailable(String)` / :17 `.storeUnavailable(String)` / :21 `.notFound` (reserved, unused by current API) / :27 `.invalidPayload(String)` / :31 `.decodingFailure(String)` -- :41 `.int8QuantizationPolicyUndefined(String)` — thrown on every int8 write; policy unratified (VECTORKIT_SPEC §I-4a); remove guard+case only when ratified -- :56 `.embedFloatVocabMiss(String)` — distributional-provider OOV signal, distinct from embeddingFailed - -### Embedding seam — EmbeddingProvider.swift -- :15 `protocol EmbeddingProvider: Sendable` — modelID/modelVersion + embed/embedFloat/embedPair/embedBatch -- :37 `embed(_:) -> Engram` — MUST return `Engram.zero` for empty string (cross-provider contract, mirrored in Rust trait) -- :64 `embedFloat(_:) -> [Float]` — opt-in; default impl (:105) throws embeddingFailed; empty input → `[]` never zero-vector -- :81 `embedPair(_:) -> (engram, floats)` — default impl (:115): two-pass (embed then embedFloat), float opt-out swallowed to `[]` -- :93 `embedBatch(_:) -> [Engram]` — default impl (:123): sequential; override for batched inference - -### Concrete provider — FloatSimHashEmbeddingProvider.swift -- :35 `struct FloatSimHashEmbeddingProvider: EmbeddingProvider` — Swift mirror of Rust vectorkit::FloatSimHashEmbeddingProvider -- :44 `projectionSeed: UInt64` — distinct seeds ⇒ distinct fingerprints for same float vector (I-4 enforced at projection layer) -- :49 `inference: @Sendable (String) async throws -> [Float]` — host-injected; kit owns no tokenizer/model -- :63 `embed(_:)` — empty-string short-circuit BEFORE inference call, then `FloatSimHash.project(vector:seed:)` -- :87 `embedFloat(_:)` — returns the SAME vector embed() projects; no double inference - -### Engine foundation types (Lane F — additive-only, no local field additions) -- VectorRecordKey.swift:33 `struct VectorRecordKey: Sendable, Equatable, Hashable, Comparable` — (itemID, vectorIndex, modelID, modelVersion); ordering IS the partition/tie-break order -- VectorRecordKey.swift:87 `< (lhs:rhs:)` — lexicographic (itemID, vectorIndex, modelID, modelVersion); DO NOT reorder fields -- VectorPayload.swift:76 `enum VectorKind: UInt8` — .binary=0/.float32=1/.int8=2; ON-DISK raw values, never reorder -- VectorPayload.swift:101 `struct VectorPayload` — kind+dim+bytes+scale(int8 only, unused in prod) -- VectorPayload.swift:146 `init(engram:)` — binary, zero-copy wire bytes -- VectorPayload.swift:163 `init(floats:)` — float32, explicit little-endian serialization (byte-order portability, not native-order) -- VectorPayload.swift:187/:200 `asEngram()` / `asFloats()` — throw invalidPayload on kind/size mismatch -- VectorPayload.swift:38 `struct VectorPayloadInput` — bulk-write row bundle (itemID, vectorIndex, payload, modelID, modelVersion, filedAt) -- DenseHit.swift:45 `struct DenseHit: Sendable, Equatable` — key + rawDistance(Int32, dual-purpose: Hamming int OR Float bit pattern) + metric -- DenseHit.swift:101/:121 `hammingDistance` / `floatDistance` — typed accessors reinterpreting rawDistance -- DenseHit.swift:132 `enum LaneTag` — .binaryDense/.floatDense/.sparse/.lateInteraction (fusion/cross-package use) -- DenseMetric.swift:41 `enum FloatMetric` — .cosine/.l2/.dot; VectorKit-owned (ADR-008) -- DenseMetric.swift:58 `enum BinaryMetric` — .hamming/.jaccard (jaccard reserved, BruteForceIndex rejects it) -- DenseMetric.swift:81 `enum DenseMetric` — .binary(BinaryMetric)/.float(FloatMetric) umbrella; :91-:103 shorthand statics -- DenseIndex.swift:67 `enum SearchDirection` — .nearest/.farthest; farthest is bottom-K scan, NOT negated top-K -- DenseIndex.swift:85 `struct MetadataFilter` — modelID/modelVersion wildcard-if-nil; :109 `accepts(_:)` -- DenseIndex.swift:35 `enum IndexKind` — .bruteForce/.mih tag (nominal dispatch, not type-casting) -- DenseIndex.swift:131 `protocol DenseIndex: Sendable` — build/search/add/remove seam; BruteForceIndex is the binary oracle +- :41 `.int8QuantizationPolicyUndefined(String)` | thrown on every int8 write; policy unratified (VECTORKIT_SPEC §I-4a); remove guard+case only when ratified +- :56 `.embedFloatVocabMiss(String)` | distributional-provider OOV signal, distinct from embeddingFailed + +### Embedding seam | EmbeddingProvider.swift +- :15 `protocol EmbeddingProvider: Sendable` | modelID/modelVersion + embed/embedFloat/embedPair/embedBatch +- :37 `embed(_:) -> Engram` | MUST return `Engram.zero` for empty string (cross-provider contract, mirrored in Rust trait) +- :64 `embedFloat(_:) -> [Float]` | opt-in; default impl (:105) throws embeddingFailed; empty input → `[]` never zero-vector +- :81 `embedPair(_:) -> (engram, floats)` | default impl (:115): two-pass (embed then embedFloat), float opt-out swallowed to `[]` +- :93 `embedBatch(_:) -> [Engram]` | default impl (:123): sequential; override for batched inference + +### Concrete provider | FloatSimHashEmbeddingProvider.swift +- :35 `struct FloatSimHashEmbeddingProvider: EmbeddingProvider` | Swift mirror of Rust vectorkit::FloatSimHashEmbeddingProvider +- :44 `projectionSeed: UInt64` | distinct seeds ⇒ distinct fingerprints for same float vector (I-4 enforced at projection layer) +- :49 `inference: @Sendable (String) async throws -> [Float]` | host-injected; kit owns no tokenizer/model +- :63 `embed(_:)` | empty-string short-circuit BEFORE inference call, then `FloatSimHash.project(vector:seed:)` +- :87 `embedFloat(_:)` | returns the SAME vector embed() projects; no double inference + +### Engine foundation types (Lane F | additive-only, no local field additions) +- VectorRecordKey.swift:33 `struct VectorRecordKey: Sendable, Equatable, Hashable, Comparable` | (itemID, vectorIndex, modelID, modelVersion); ordering IS the partition/tie-break order +- VectorRecordKey.swift:87 `< (lhs:rhs:)` | lexicographic (itemID, vectorIndex, modelID, modelVersion); DO NOT reorder fields +- VectorPayload.swift:76 `enum VectorKind: UInt8` | .binary=0/.float32=1/.int8=2; ON-DISK raw values, never reorder +- VectorPayload.swift:101 `struct VectorPayload` | kind+dim+bytes+scale(int8 only, unused in prod) +- VectorPayload.swift:146 `init(engram:)` | binary, zero-copy wire bytes +- VectorPayload.swift:163 `init(floats:)` | float32, explicit little-endian serialization (byte-order portability, not native-order) +- VectorPayload.swift:187/:200 `asEngram()` / `asFloats()` | throw invalidPayload on kind/size mismatch +- VectorPayload.swift:38 `struct VectorPayloadInput` | bulk-write row bundle (itemID, vectorIndex, payload, modelID, modelVersion, filedAt) +- DenseHit.swift:45 `struct DenseHit: Sendable, Equatable` | key + rawDistance(Int32, dual-purpose: Hamming int OR Float bit pattern) + metric +- DenseHit.swift:101/:121 `hammingDistance` / `floatDistance` | typed accessors reinterpreting rawDistance +- DenseHit.swift:132 `enum LaneTag` | .binaryDense/.floatDense/.sparse/.lateInteraction (fusion/cross-package use) +- DenseMetric.swift:41 `enum FloatMetric` | .cosine/.l2/.dot; VectorKit-owned (ADR-008) +- DenseMetric.swift:58 `enum BinaryMetric` | .hamming/.jaccard (jaccard reserved, BruteForceIndex rejects it) +- DenseMetric.swift:81 `enum DenseMetric` | .binary(BinaryMetric)/.float(FloatMetric) umbrella; :91-:103 shorthand statics +- DenseIndex.swift:67 `enum SearchDirection` | .nearest/.farthest; farthest is bottom-K scan, NOT negated top-K +- DenseIndex.swift:85 `struct MetadataFilter` | modelID/modelVersion wildcard-if-nil; :109 `accepts(_:)` +- DenseIndex.swift:35 `enum IndexKind` | .bruteForce/.mih tag (nominal dispatch, not type-casting) +- DenseIndex.swift:131 `protocol DenseIndex: Sendable` | build/search/add/remove seam; BruteForceIndex is the binary oracle ### Binary engines -- BruteForceIndex.swift:47 `actor BruteForceIndex: DenseIndex` — Lane A; conformance oracle; ZERO Hamming math in file (I-7) -- BruteForceIndex.swift:101 `search(probe:metric:k:filter:)` — only .binary(.hamming); model-partition slice via O(log m) lookup; sorts (distance ASC, FULL key ASC) — NOT EngramLib.findNearest (different tie-break) -- BruteForceIndex.swift:227 `add(key:vector:)` — tombstone-then-append upsert; :276 `remove(key:)` — tombstone only, no reclaim -- BruteForceIndex.swift:305 `currentSnapshot()` — value-copy for cross-actor read (VectorStore tombstone scans) -- BruteForceIndex.swift:326/:342 `setTombstoneBit` / `buildPartitions` — shared bit-layout + partition-rebuild helpers, mirrored in ResidentArrayStore -- MIHIndex.swift:251 `actor MIHIndex: DenseIndex` — Lane B; sub-linear EXACT Hamming k-NN via Multi-Index Hashing; output MUST equal BruteForceIndex bit-for-bit (BLOCKER conformance gate, MIHIndexTests.swift) -- MIHIndex.swift:72 `enum MIHBandCount: UInt32` — {.m4,.m8,.m16,.m32} ONLY (§1.7: keeps sub_bits∈{64,32,16,8}, no word-straddle) -- MIHIndex.swift:303 `init(bandCount:maskBudget:)` — maskBudget nil ⇒ dynamic max(n, 2^20) per query -- MIHIndex.swift:351 `search(...)` → :450 `knn(...)` — progressive-radius pigeonhole expansion; stop when heap full AND worstDist ≤ r -- MIHIndex.swift:504 enumeration-budget guard — projected flip-mask count vs budget; over-budget + heap not exact ⇒ :575 `bruteScan` fallback (still exact, just O(n)); fires `.notice` log + `vectorkit.mih.enumeration_fallback` metric -- MIHIndex.swift:711 `cumulativeChoose(subBits:rho:)` — Σ C(subBits,d); saturates to Int.max on overflow; MUST match Rust saturating_add bit-for-bit -- MIHIndex.swift:764 `colexFlipMasks(subBits:maxHamming:body:)` — Gosper's-hack colex enumeration, ascending subset size then ascending mask value; canonical order, internal (test-visible) -- MIHIndex.swift:654 `extractBand(from:bandIndex:)` — canonical bit numbering (bit i → word i/64, LSB=0); word-straddle branch unreachable for allowed m +- BruteForceIndex.swift:47 `actor BruteForceIndex: DenseIndex` | Lane A; conformance oracle; ZERO Hamming math in file (I-7) +- BruteForceIndex.swift:101 `search(probe:metric:k:filter:)` | only .binary(.hamming); model-partition slice via O(log m) lookup; sorts (distance ASC, FULL key ASC) | NOT EngramLib.findNearest (different tie-break) +- BruteForceIndex.swift:227 `add(key:vector:)` | tombstone-then-append upsert; :276 `remove(key:)` | tombstone only, no reclaim +- BruteForceIndex.swift:305 `currentSnapshot()` | value-copy for cross-actor read (VectorStore tombstone scans) +- BruteForceIndex.swift:326/:342 `setTombstoneBit` / `buildPartitions` | shared bit-layout + partition-rebuild helpers, mirrored in ResidentArrayStore +- MIHIndex.swift:251 `actor MIHIndex: DenseIndex` | Lane B; sub-linear EXACT Hamming k-NN via Multi-Index Hashing; output MUST equal BruteForceIndex bit-for-bit (BLOCKER conformance gate, MIHIndexTests.swift) +- MIHIndex.swift:72 `enum MIHBandCount: UInt32` | {.m4,.m8,.m16,.m32} ONLY (§1.7: keeps sub_bits∈{64,32,16,8}, no word-straddle) +- MIHIndex.swift:303 `init(bandCount:maskBudget:)` | maskBudget nil ⇒ dynamic max(n, 2^20) per query +- MIHIndex.swift:351 `search(...)` → :450 `knn(...)` | progressive-radius pigeonhole expansion; stop when heap full AND worstDist ≤ r +- MIHIndex.swift:504 enumeration-budget guard | projected flip-mask count vs budget; over-budget + heap not exact ⇒ :575 `bruteScan` fallback (still exact, just O(n)); fires `.notice` log + `vectorkit.mih.enumeration_fallback` metric +- MIHIndex.swift:711 `cumulativeChoose(subBits:rho:)` | Σ C(subBits,d); saturates to Int.max on overflow; MUST match Rust saturating_add bit-for-bit +- MIHIndex.swift:764 `colexFlipMasks(subBits:maxHamming:body:)` | Gosper's-hack colex enumeration, ascending subset size then ascending mask value; canonical order, internal (test-visible) +- MIHIndex.swift:654 `extractBand(from:bandIndex:)` | canonical bit numbering (bit i → word i/64, LSB=0); word-straddle branch unreachable for allowed m ### Float engine -- FloatBruteForceIndex.swift:60 `actor FloatBruteForceIndex: DenseIndex` — Lane C/D (file header says "Lane C", VectorStore.swift comments say "Lane D" — same type, inconsistent lane label in source, not a functional issue); float32 ONLY; NOT four-way bit-identical (documented, do not "fix") -- FloatBruteForceIndex.swift:84 `build(from:)` — O(1) reference store; array IS the index -- FloatBruteForceIndex.swift:102 `search(...)` — validates probe.kind/.dim vs array stride; cosine treats zero-vector as distance 1.0 (no div-by-zero) -- FloatBruteForceIndex.swift:156 `searchFarthest(...)` — identical scan+distance as search(); only sort direction flips (:233 `rank(...direction:)`) -- FloatBruteForceIndex.swift:273 `add(key:vector:)` — FIRST add establishes stride; later mismatched byte count throws invalidPayload (prevents storage corruption) -- FloatBruteForceIndex.swift:323 `remove(key:)` — tombstone; compaction only on next build() +- FloatBruteForceIndex.swift:60 `actor FloatBruteForceIndex: DenseIndex` | Lane C/D (file header says "Lane C", VectorStore.swift comments say "Lane D" | same type, inconsistent lane label in source, not a functional issue); float32 ONLY; NOT four-way bit-identical (documented, do not "fix") +- FloatBruteForceIndex.swift:84 `build(from:)` | O(1) reference store; array IS the index +- FloatBruteForceIndex.swift:102 `search(...)` | validates probe.kind/.dim vs array stride; cosine treats zero-vector as distance 1.0 (no div-by-zero) +- FloatBruteForceIndex.swift:156 `searchFarthest(...)` | identical scan+distance as search(); only sort direction flips (:233 `rank(...direction:)`) +- FloatBruteForceIndex.swift:273 `add(key:vector:)` | FIRST add establishes stride; later mismatched byte count throws invalidPayload (prevents storage corruption) +- FloatBruteForceIndex.swift:323 `remove(key:)` | tombstone; compaction only on next build() ### Late-interaction scorer (standalone, not a DenseIndex) -- MaxSimScorer.swift:97 `struct MaxSimScorer: Sendable` — Lane E1, Exact-A exhaustive ColBERT MaxSim; conformance reference for future pruned variants -- MaxSimScorer.swift:148 `score(queryTokens:documents:k:) -> [MaxSimHit]` — Σ(256−min hamming) per query token; documents iterated in SORTED itemID order (dict order is undefined); sort (score DESC, itemID ASC); truncate to k AFTER full sort -- MaxSimScorer.swift:55 `struct MaxSimHit` — itemID + integer score [0, 256×|Q|] +- MaxSimScorer.swift:97 `struct MaxSimScorer: Sendable` | Lane E1, Exact-A exhaustive ColBERT MaxSim; conformance reference for future pruned variants +- MaxSimScorer.swift:148 `score(queryTokens:documents:k:) -> [MaxSimHit]` | Σ(256−min hamming) per query token; documents iterated in SORTED itemID order (dict order is undefined); sort (score DESC, itemID ASC); truncate to k AFTER full sort +- MaxSimScorer.swift:55 `struct MaxSimHit` | itemID + integer score [0, 256×|Q|] - All distances via `EngramLib.Session.distances` (I-7); session built once per scorer, reused across the whole score() call ### Resident array (shared data contract, Lane F) -- ResidentVectorArray.swift:65 `struct ResidentVectorArray: Sendable` — packed fixed-stride array; kind/stride/count/storage/keys/modelPartitions/tombstones; measured 87% of pre-resident latency was fetch+decode, 0.4% kernel — this type removes the fetch+decode cost -- ResidentVectorArray.swift:135 `liveCount` — O(count/64) tombstone-bitmap walk; used for sidecar staleness (live-vs-live compare) -- ResidentVectorArray.swift:190 `partitionRange(for:)` — binary search, O(log m) -- ResidentVectorArray.swift:214/:228 `isTombstoned(_:)` / `vectorBytes(at:)` — per-slot accessors every engine scan loop uses -- ResidentVectorArray.swift:44 `struct ModelPartitionEntry` — modelID + half-open Range - -### Resident array persistence — ResidentArrayStore.swift -- :116 `actor ResidentArrayStore` — owns optional `.vec` sidecar; vectors TABLE remains sole durable source; sidecar is regenerable cache only -- :97 `kVecVersion = 0x0002` — format version; adds live_count field after count (discarded on load, recomputed from tombstone bitmap — stale header value cannot corrupt results) +- ResidentVectorArray.swift:65 `struct ResidentVectorArray: Sendable` | packed fixed-stride array; kind/stride/count/storage/keys/modelPartitions/tombstones; measured 87% of pre-resident latency was fetch+decode, 0.4% kernel | this type removes the fetch+decode cost +- ResidentVectorArray.swift:135 `liveCount` | O(count/64) tombstone-bitmap walk; used for sidecar staleness (live-vs-live compare) +- ResidentVectorArray.swift:190 `partitionRange(for:)` | binary search, O(log m) +- ResidentVectorArray.swift:214/:228 `isTombstoned(_:)` / `vectorBytes(at:)` | per-slot accessors every engine scan loop uses +- ResidentVectorArray.swift:44 `struct ModelPartitionEntry` | modelID + half-open Range + +### Resident array persistence | ResidentArrayStore.swift +- :116 `actor ResidentArrayStore` | owns optional `.vec` sidecar; vectors TABLE remains sole durable source; sidecar is regenerable cache only +- :97 `kVecVersion = 0x0002` | format version; adds live_count field after count (discarded on load, recomputed from tombstone bitmap | stale header value cannot corrupt results) - :102 `kDefaultTombstoneCompactionThreshold = 0.25` -- :183 `load()` — missing/invalid sidecar ⇒ start empty, no crash -- :209 `rebuild(from:)` — full rewrite from sorted [(key,bytes)]; used on stale-sidecar detection -- :247 `append(key:bytes:)` — EAGER write (immediate sidecar rewrite) -- :277 `appendDeferred(key:bytes:)` — WRITE-BEHIND single-add path (production default via VectorStore.addPayload); sets isDirty, no disk write; caller must flush() -- :299 `appendBatch(records:)` — bulk path, ONE sidecar write per batch (not per record) — TASK #24 amortization -- :346 `flush()` — persists pending write-behind mutation; no-op if !isDirty +- :183 `load()` | missing/invalid sidecar ⇒ start empty, no crash +- :209 `rebuild(from:)` | full rewrite from sorted [(key,bytes)]; used on stale-sidecar detection +- :247 `append(key:bytes:)` | EAGER write (immediate sidecar rewrite) +- :277 `appendDeferred(key:bytes:)` | WRITE-BEHIND single-add path (production default via VectorStore.addPayload); sets isDirty, no disk write; caller must flush() +- :299 `appendBatch(records:)` | bulk path, ONE sidecar write per batch (not per record) | TASK #24 amortization +- :346 `flush()` | persists pending write-behind mutation; no-op if !isDirty - :387/:420 `tombstone(key:)` (eager, writes) / `tombstoneDeferred(keys:)` (batch, no write, sets isDirty) -- :448 `compact()` — drops tombstoned slots, sorted-by-key rewrite, deterministic output -- :563/:610/:623 `writeSidecar` / `readSidecar` (mmap via .mappedIfSafe) / `parseSidecar` — every length field bounds-checked before trust; magic "VEC1" (:85 `kVecMagic`) +- :448 `compact()` | drops tombstoned slots, sorted-by-key rewrite, deterministic output +- :563/:610/:623 `writeSidecar` / `readSidecar` (mmap via .mappedIfSafe) / `parseSidecar` | every length field bounds-checked before trust; magic "VEC1" (:85 `kVecMagic`) - On-disk layout: magic(4)|version(2)|kind(1)|stride(4)|count(4)|live_count(4)|tombstone_words(4)|tombstones|vectors|keys(variable)|partition_index(variable); ALL integers little-endian (cross-host byte-identity, arch spec §4.3) ### Storage-facing types -- StoredVector.swift:20 `struct StoredVector: Sendable, Equatable` — decoded `vectors` row; `engram` non-nil ONLY for binary kind (float/int8 rows: use getPayload) -- VectorMatch.swift:19 `struct VectorMatch: Sendable, Comparable, Equatable` — itemID/distance/modelID; :43 `<` — (distance ASC, itemID ASC) universal tie-break - -### Storage actor — VectorStore.swift -- :128 `actor VectorStore` — the kit's single consumer-facing surface -- :327 `static let schemaDeclaration` — "vectors" table v3; UNIQUE(item_id, vector_index, model_id) == VectorRecordKey minus modelVersion -- :382 `static defaultSidecarURL(for:)` — `.sqlite` → `.vectors.vec`; nil for non-file backends -- :165 `mihThreshold: UInt32 = 50_000` (default) — promotion boundary, overridable at init -- :172 `mihBandCount: MIHBandCount` (default .m16) — pinned per §1.6 for 50k default threshold -- :185/:189/:197 `bruteForceIndex` / `mihIndex` / `hotIndex` — both allocated at init; hotIndex swapped by :1661 `_selectIndex()` (no rebuild on swap) -- :283 `floatIndices: [String: FloatBruteForceIndex]` — ONE PER modelID (uniform stride requirement); map-entry presence == "built" flag -- :451 `addVector(itemID:engram:modelID:modelVersion:filedAt:)` — convenience wrapper over addPayload -- :496 `addPayload(itemID:vectorIndex:payload:modelID:modelVersion:filedAt:)` — rejects .int8 (throws int8QuantizationPolicyUndefined); table upsert THEN resident mirror; matches stale slots by (itemID,vectorIndex,modelID) — NOT full key — so modelVersion changes are treated as replacement (secfix/ws2-coredelete hard-delete contract); emits vectorkit.index.insert_latency_ms -- :669 `addPayloads(_:)` — bulk path; rejects batch containing ANY int8 (no partial writes); immediate mode rebuilds both indexes ONCE from final snapshot; emits vectorkit.index.batch_insert_latency_ms -- :874 `beginDeferredIndex()` / :898 `publishResidentIndex()` — bulk-burst mode: appends skip index rebuild until publish (O(N) not O(N²)); corpus ingest drain wraps a burst in this pair -- :253/:943 `deferredPendingRecords` / `_flushDeferredPending()` — memory-only-path back-pressure valve, capped at `deferredPendingLimit` (default 50_000, ctor param) — secfix/punt-vector unbounded-buffer fix -- :993 `flush()` — persists pending sidecar write-behind mutation -- :1052/:1066/:1088 `getVector` / `getPayload` / `vectors(forItemID:)` — read paths, all decode via :1803 `decodePayload` / :1840 `storedVector` -- :1134 `findNearest(probe:modelID:limit:)` — lazy `_ensureIndexBuilt()`, delegates to hotIndex.search, NO re-sort (engine already ordered); emits vectorkit.search.latency_ms + .result_count -- :1223 `findNearestFloat(probe:modelID:limit:)` — lazy per-model FloatBruteForceIndex build via :1558 `_ensureFloatIndexBuilt`; quantizes cosine distance ×10_000 rounded for cross-language integer comparison -- :1294 `findFarthestFloat(probe:modelID:limit:)` — same as findNearestFloat but calls searchFarthest; anti-similarity -- :1337 `findByKeyword(_:limit:)` — substring LIKE on item_id; NOT full BM25 (CorpusKit's job); emits vectorkit.search.keyword_result_count -- :1377 `deleteVector(itemID:modelID:)` / :1386 `deleteAllVectors(itemID:modelID:)` — both flush pending deferred burst FIRST if dirty, then delete+tombstone; deleteAllVectors invalidates that model's float index (lazy rebuild) -- :1442 `destroyAllVectors()` — full wipe: table+both binary indexes+sidecar+ALL float indices; used by GeniusLocusKit estate teardown -- :1487 `_ensureIndexBuilt()` — idempotent; sidecar trusted iff `snap.liveCount == tableCount` (live-vs-live, NOT snap.count vs table — avoids spurious rebuild after deletes, "C5 fix") -- :1558 `_ensureFloatIndexBuilt(modelID:)` — nil return (no cache) when model has zero float rows, so a later first-ingest can still build a real index -- :1736 `_deleteAndTombstone(itemID:vectorIndex:modelID:)` — scans ALL slots matching (itemID,vectorIndex,modelID) across modelVersions, tombstones every match (no break-after-first) — hard-delete contract -- :1803 `decodePayload(from row:)` — int8 rows ALWAYS decode to nil (symmetric fail-closed read guard, even for hand-crafted rows); guards every Int64→UInt8/UInt32 narrowing conversion against trap +- StoredVector.swift:20 `struct StoredVector: Sendable, Equatable` | decoded `vectors` row; `engram` non-nil ONLY for binary kind (float/int8 rows: use getPayload) +- VectorMatch.swift:19 `struct VectorMatch: Sendable, Comparable, Equatable` | itemID/distance/modelID; :43 `<` | (distance ASC, itemID ASC) universal tie-break + +### Storage actor | VectorStore.swift +- :128 `actor VectorStore` | the kit's single consumer-facing surface +- :327 `static let schemaDeclaration` | "vectors" table v3; UNIQUE(item_id, vector_index, model_id) == VectorRecordKey minus modelVersion +- :382 `static defaultSidecarURL(for:)` | `.sqlite` → `.vectors.vec`; nil for non-file backends +- :165 `mihThreshold: UInt32 = 50_000` (default) | promotion boundary, overridable at init +- :172 `mihBandCount: MIHBandCount` (default .m16) | pinned per §1.6 for 50k default threshold +- :185/:189/:197 `bruteForceIndex` / `mihIndex` / `hotIndex` | both allocated at init; hotIndex swapped by :1661 `_selectIndex()` (no rebuild on swap) +- :283 `floatIndices: [String: FloatBruteForceIndex]` | ONE PER modelID (uniform stride requirement); map-entry presence == "built" flag +- :451 `addVector(itemID:engram:modelID:modelVersion:filedAt:)` | convenience wrapper over addPayload +- :496 `addPayload(itemID:vectorIndex:payload:modelID:modelVersion:filedAt:)` | rejects .int8 (throws int8QuantizationPolicyUndefined); table upsert THEN resident mirror; matches stale slots by (itemID,vectorIndex,modelID) | NOT full key | so modelVersion changes are treated as replacement (secfix/ws2-coredelete hard-delete contract); emits vectorkit.index.insert_latency_ms +- :669 `addPayloads(_:)` | bulk path; rejects batch containing ANY int8 (no partial writes); immediate mode rebuilds both indexes ONCE from final snapshot; emits vectorkit.index.batch_insert_latency_ms +- :874 `beginDeferredIndex()` / :898 `publishResidentIndex()` | bulk-burst mode: appends skip index rebuild until publish (O(N) not O(N²)); corpus ingest drain wraps a burst in this pair +- :253/:943 `deferredPendingRecords` / `_flushDeferredPending()` | memory-only-path back-pressure valve, capped at `deferredPendingLimit` (default 50_000, ctor param) | secfix/punt-vector unbounded-buffer fix +- :993 `flush()` | persists pending sidecar write-behind mutation +- :1052/:1066/:1088 `getVector` / `getPayload` / `vectors(forItemID:)` | read paths, all decode via :1803 `decodePayload` / :1840 `storedVector` +- :1134 `findNearest(probe:modelID:limit:)` | lazy `_ensureIndexBuilt()`, delegates to hotIndex.search, NO re-sort (engine already ordered); emits vectorkit.search.latency_ms + .result_count +- :1223 `findNearestFloat(probe:modelID:limit:)` | lazy per-model FloatBruteForceIndex build via :1558 `_ensureFloatIndexBuilt`; quantizes cosine distance ×10_000 rounded for cross-language integer comparison +- :1294 `findFarthestFloat(probe:modelID:limit:)` | same as findNearestFloat but calls searchFarthest; anti-similarity +- :1337 `findByKeyword(_:limit:)` | substring LIKE on item_id; NOT full BM25 (CorpusKit's job); emits vectorkit.search.keyword_result_count +- :1377 `deleteVector(itemID:modelID:)` / :1386 `deleteAllVectors(itemID:modelID:)` | both flush pending deferred burst FIRST if dirty, then delete+tombstone; deleteAllVectors invalidates that model's float index (lazy rebuild) +- :1442 `destroyAllVectors()` | full wipe: table+both binary indexes+sidecar+ALL float indices; used by GeniusLocusKit estate teardown +- :1487 `_ensureIndexBuilt()` | idempotent; sidecar trusted iff `snap.liveCount == tableCount` (live-vs-live, NOT snap.count vs table | avoids spurious rebuild after deletes, "C5 fix") +- :1558 `_ensureFloatIndexBuilt(modelID:)` | nil return (no cache) when model has zero float rows, so a later first-ingest can still build a real index +- :1736 `_deleteAndTombstone(itemID:vectorIndex:modelID:)` | scans ALL slots matching (itemID,vectorIndex,modelID) across modelVersions, tombstones every match (no break-after-first) | hard-delete contract +- :1803 `decodePayload(from row:)` | int8 rows ALWAYS decode to nil (symmetric fail-closed read guard, even for hand-crafted rows); guards every Int64→UInt8/UInt32 narrowing conversion against trap ## INVARIANTS / GOTCHAS - I-7 ABSOLUTE: zero Hamming/XOR/popcount arithmetic anywhere in this package outside EngramLib calls. BruteForceIndex, MIHIndex, MaxSimScorer all delegate every distance to EngramLib.distances / EngramLib.Session.distances / EngramLib.distance. A raw popcount anywhere is a conformance violation. - I-4 ABSOLUTE: cross-model vector comparison is forbidden. Every write carries modelID+modelVersion; every search is scoped to one modelID; float indices are one-per-model because different models emit different dimensions; the resident binary partition index scopes searches by modelID. -- Determinism boundary: binary lane (.hamming) is four-way bit-identical (Swift/Rust × platforms), gated by EngramLib's kernel. Float lane (.cosine/.l2/.dot) is reproducible-within-config ONLY — NOT four-way. Do not add a conformance test asserting float bit-identity; it will be correctly flagged as testing an undocumented, unintended guarantee. -- MIHIndex MUST equal BruteForceIndex output bit-for-bit on every input — this is a BLOCKER gate (MIHIndexTests.swift), not a best-effort property. BruteForceIndex is the oracle; never "fix" MIH by relaxing the gate. -- int8 is REJECTED fail-closed at both write (addPayload/addPayloads throw int8QuantizationPolicyUndefined) and read (decodePayload returns nil for kind==.int8). The case, field, and guards stay until a quantization policy is ratified — do not remove the guard as "dead code." -- Stale-slot matching in addPayload/addPayloads uses (itemID, vectorIndex, modelID) — NOT the full VectorRecordKey — specifically so a modelVersion change is recognized as replacement, not a new sibling slot. Matching by full key here is the recurring bug shape (secfix/ws2-coredelete); do not "simplify" to full-key equality. +- Determinism boundary: binary lane (.hamming) is four-way bit-identical (Swift/Rust × platforms), gated by EngramLib's kernel. Float lane (.cosine/.l2/.dot) is reproducible-within-config ONLY | NOT four-way. Do not add a conformance test asserting float bit-identity; it will be correctly flagged as testing an undocumented, unintended guarantee. +- MIHIndex MUST equal BruteForceIndex output bit-for-bit on every input | this is a BLOCKER gate (MIHIndexTests.swift), not a best-effort property. BruteForceIndex is the oracle; never "fix" MIH by relaxing the gate. +- int8 is REJECTED fail-closed at both write (addPayload/addPayloads throw int8QuantizationPolicyUndefined) and read (decodePayload returns nil for kind==.int8). The case, field, and guards stay until a quantization policy is ratified | do not remove the guard as "dead code." +- Stale-slot matching in addPayload/addPayloads uses (itemID, vectorIndex, modelID) | NOT the full VectorRecordKey | specifically so a modelVersion change is recognized as replacement, not a new sibling slot. Matching by full key here is the recurring bug shape (secfix/ws2-coredelete); do not "simplify" to full-key equality. - VectorRecordKey ordering (itemID, vectorIndex, modelID, modelVersion) is load-bearing: it is the resident-array partition order, the universal search tie-break, and the sidecar's on-disk key ordering. Do not reorder the Comparable fields. -- MIHBandCount is restricted to {4,8,16,32} by the enum itself (§1.7 conformance restriction) — sub_bits ∈ {64,32,16,8} guarantees no word-straddle in extractBand. Do not add m=2 or other values without a new word-straddle code path and a separate conformance harness. +- MIHBandCount is restricted to {4,8,16,32} by the enum itself (§1.7 conformance restriction) | sub_bits ∈ {64,32,16,8} guarantees no word-straddle in extractBand. Do not add m=2 or other values without a new word-straddle code path and a separate conformance harness. - MIHIndex's enumeration-budget guard falls back to a full O(n) bruteScan (still EXACT, same heap, same distances) rather than hang on sparse/adversarial data. cumulativeChoose's saturating-overflow arithmetic MUST stay bit-identical to the Rust port's saturating_add so both fall back at the same radius. - ResidentArrayStore sidecar is a REGENERABLE CACHE, never a second source of truth. The `vectors` table is authoritative. Staleness check compares live-vs-live counts (format 0x0002); comparing total slot counts instead (pre-C5-fix behavior) spuriously rebuilds after every delete. - Sidecar single-add writes are WRITE-BEHIND (appendDeferred + isDirty): VectorStore.addPayload does not force a disk write per call. Callers relying on immediate on-disk durability of the sidecar must call `flush()`; crash safety does not depend on this because the table write already happened synchronously before the mirror. - deferredIndexActive (beginDeferredIndex/publishResidentIndex) turns a bulk import from O(N²) to O(N) index rebuilds. deleteVector/deleteAllVectors both force-publish a dirty deferred window before deleting, so a delete never races an unpublished burst. -- deferredPendingRecords (memory-only deferred path) is capped at deferredPendingLimit (default 50,000); exceeding it triggers an intermediate flush that keeps the burst open. This is a back-pressure valve, not an error path — do not treat a flush mid-burst as anomalous. -- Float payload byte order is EXPLICIT little-endian (VectorPayload.init(floats:) / asFloats()), independent of host native endianness — required for the `.vec` sidecar (and any future float sidecar) to be byte-identical across Apple and Linux hosts. +- deferredPendingRecords (memory-only deferred path) is capped at deferredPendingLimit (default 50,000); exceeding it triggers an intermediate flush that keeps the burst open. This is a back-pressure valve, not an error path | do not treat a flush mid-burst as anomalous. +- Float payload byte order is EXPLICIT little-endian (VectorPayload.init(floats:) / asFloats()), independent of host native endianness | required for the `.vec` sidecar (and any future float sidecar) to be byte-identical across Apple and Linux hosts. - FloatBruteForceIndex establishes its stride from the FIRST vector added via add(key:vector:); a later vector of different byte count throws rather than corrupting the flat storage buffer. One index = one dimension, always. -- FloatBruteForceIndex is labeled "Lane C" in its own file header but "Lane D" in VectorStore.swift's comments — same type, inconsistent label, not a functional divergence. Do not "fix" one file to match the other without checking whether a Lane-lettering convention doc elsewhere disambiguates it first. -- MaxSimScorer (Lane E1) is NOT wired into VectorStore's public search API in this package version — it is a standalone scorer callers invoke directly with pre-fetched token-Engram arrays. Do not assume `findNearest` performs late interaction. -- Telemetry (Intellectus.report) is off by default; the emitted metrics (vectorkit.index.insert_latency_ms, .batch_insert_latency_ms, vectorkit.search.latency_ms, .result_count, .keyword_result_count, vectorkit.mih.enumeration_fallback) are named constants used by dashboards — renaming any of them is a breaking change to monitoring, not just to code. -- Pinned/default constants — changing any requires updating dependent conformance tests: mihThreshold 50,000, mihBandCount .m16 (sub_bits=16), deferredPendingLimit 50,000, compactionThreshold 0.25, poolSubmitThreshold-equivalent N/A (not used here), sidecar format version 0x0002, MIHBandCount ∈ {4,8,16,32}, cosine-distance quantization scale ×10,000. -- Actor boundaries: VectorStore, BruteForceIndex, MIHIndex, FloatBruteForceIndex, ResidentArrayStore are all actors — all mutation and reads are serialized per actor. ResidentVectorArray, VectorPayload, VectorRecordKey, DenseHit, VectorMatch, StoredVector are plain Sendable value types safe to pass across actor boundaries once constructed. +- FloatBruteForceIndex is labeled "Lane C" in its own file header but "Lane D" in VectorStore.swift's comments | same type, inconsistent label, not a functional divergence. Do not "fix" one file to match the other without checking whether a Lane-lettering convention doc elsewhere disambiguates it first. +- MaxSimScorer (Lane E1) is NOT wired into VectorStore's public search API in this package version | it is a standalone scorer callers invoke directly with pre-fetched token-Engram arrays. Do not assume `findNearest` performs late interaction. +- Telemetry (Intellectus.report) is off by default; the emitted metrics (vectorkit.index.insert_latency_ms, .batch_insert_latency_ms, vectorkit.search.latency_ms, .result_count, .keyword_result_count, vectorkit.mih.enumeration_fallback) are named constants used by dashboards | renaming any of them is a breaking change to monitoring, not just to code. +- Pinned/default constants | changing any requires updating dependent conformance tests: mihThreshold 50,000, mihBandCount .m16 (sub_bits=16), deferredPendingLimit 50,000, compactionThreshold 0.25, poolSubmitThreshold-equivalent N/A (not used here), sidecar format version 0x0002, MIHBandCount ∈ {4,8,16,32}, cosine-distance quantization scale ×10,000. +- Actor boundaries: VectorStore, BruteForceIndex, MIHIndex, FloatBruteForceIndex, ResidentArrayStore are all actors | all mutation and reads are serialized per actor. ResidentVectorArray, VectorPayload, VectorRecordKey, DenseHit, VectorMatch, StoredVector are plain Sendable value types safe to pass across actor boundaries once constructed. diff --git a/packages/kits/VectorKit/docs/DETAILS.md b/packages/kits/VectorKit/docs/DETAILS.md index bf45b65..95ce027 100644 --- a/packages/kits/VectorKit/docs/DETAILS.md +++ b/packages/kits/VectorKit/docs/DETAILS.md @@ -47,96 +47,102 @@ sources: This document walks through every source file in the package. Read `OVERVIEW.md` first for the big picture. Files appear here in pipeline -order: the module surface and errors, the embedding seam, the shared -engine foundation types, the three search engines, the resident-array -storage layer, and finally the storage-facing types and the `VectorStore` -actor that ties everything together. +order. First comes the module surface and errors. Then the embedding +seam. Then the shared engine foundation types. Then the three search +engines. Then the resident-array storage layer. Last comes the +storage-facing types and the `VectorStore` actor that ties everything +together. ## VectorKit.swift -This file provides the module surface: a short header comment naming the -kit's public pieces. It defines no types of its own. +This file provides the module surface. It is a short header comment +naming the kit's public pieces. It defines no types of its own. -The file explains a boundary worth restating here: VectorKit imports -EngramLib internally to use the `Engram` fingerprint type, but it does not +The file explains a boundary worth restating here. VectorKit imports +EngramLib internally to use the `Engram` fingerprint type. It does not re-export that import. A caller that wants to construct an `Engram` -directly must `import EngramLib` itself. This keeps every package's import -list explicit, matching the convention used across the rest of the -substrate-dependent kits. +directly must `import EngramLib` itself. This keeps every package's +import list explicit. It matches the convention used across the rest of +the substrate-dependent kits. ## VectorKitError.swift This file provides `VectorKitError`, the single error type for every VectorKit operation. Per MOOTx01 convention, errors are concrete named -cases rather than a generic failure plus a logged message, so a caller can -branch on exactly what went wrong. +cases. They are not a generic failure plus a logged message. A caller +can branch on exactly what went wrong. `embeddingFailed(String)` reports an inference failure inside an -`EmbeddingProvider`. `modelUnavailable(String)` reports a model that is not -loaded on the current device. `storeUnavailable(String)` reports a failure -opening the backing storage. `notFound` is reserved for a future throwing -read path; today's read functions return `nil` instead. `invalidPayload` -covers a structurally broken `VectorPayload` — wrong kind, wrong byte -count, or a dimension mismatch — and is thrown by the payload's own -decode functions and by every search engine's input validation. -`decodingFailure` covers a malformed key or row that cannot be decoded. +`EmbeddingProvider`. `modelUnavailable(String)` reports a model that is +not loaded on the current device. `storeUnavailable(String)` reports a +failure opening the backing storage. `notFound` is reserved for a future +throwing read path. Today's read functions return `nil` instead. +`invalidPayload` covers a structurally broken `VectorPayload`. This +means a wrong kind, a wrong byte count, or a dimension mismatch. The +payload's own decode functions throw it. So does every search engine's +input validation. `decodingFailure` covers a malformed key or row that +cannot be decoded. Two cases record product decisions rather than plain bugs. `int8QuantizationPolicyUndefined` is thrown whenever a caller tries to -write an `.int8` (quantized) vector: the rules for how to quantize and -later reverse the quantization have not been agreed on yet, so writing one -now would lock in behavior nobody has approved. The case documents that -this is deliberate and reversible — once a policy is ratified, the guards -that throw this error are meant to be removed. `embedFloatVocabMiss` is -thrown by embedding providers whose vocabulary is fixed in advance (for -example, statistical models trained on a specific word list) when none of -a query's words are in that vocabulary; it is distinct from -`embeddingFailed` because the provider is working correctly — the input -simply has nothing in common with what the provider knows. +write an `.int8`, or quantized, vector. The rules for how to quantize, +and later reverse the quantization, have not been agreed on yet. Writing +one now would lock in behavior nobody has approved. The case documents +that this is deliberate and reversible. Once a policy is ratified, the +guards that throw this error are meant to be removed. +`embedFloatVocabMiss` is thrown by embedding providers whose vocabulary +is fixed in advance. An example is a statistical model trained on a +specific word list. The error fires when none of a query's words are in +that vocabulary. It is distinct from `embeddingFailed`, because the +provider is working correctly here. The input simply has nothing in +common with what the provider knows. ## EmbeddingProvider.swift -This file provides the `EmbeddingProvider` protocol, the single seam -between "text" and "vector" that every embedding source in VectorKit's kit -graph implements. - -The protocol is deliberately narrow: a `modelID` and `modelVersion` (the -tags every stored vector must carry, per spec I-4, so that vectors from -incompatible models are never compared), and three ways to turn text into -numbers. This narrowness is what lets VectorKit remain agnostic about -where inference happens — CoreML, ONNX, a hand-written statistical model — -and let a concrete provider such as `FloatSimHashEmbeddingProvider` or a -sibling package's MiniLM adapter fill in the details. - -`embed(_:)` is the primary method: it returns the 256-bit binary Engram -for a piece of text, throwing `embeddingFailed` or `modelUnavailable` on -failure. Every conformer must return the substrate's canonical zero Engram -for an empty string, because this is the one input every provider is -guaranteed to agree on, and treating it specially lets empty records from -different providers land on the same "identical" partition instead of +This file provides the `EmbeddingProvider` protocol. It is the single +seam between text and vector that every embedding source in VectorKit's +kit graph implements. + +The protocol is deliberately narrow. It defines a `modelID` and a +`modelVersion`. These are the tags every stored vector must carry, per +spec I-4. This tagging ensures that vectors from incompatible models +are never compared. The protocol also defines three ways to turn text +into numbers. This narrowness is what +lets VectorKit remain agnostic about where inference happens: CoreML, +ONNX, or a hand-written statistical model. A concrete provider, such as +`FloatSimHashEmbeddingProvider` or a sibling package's MiniLM adapter, +fills in the details. + +`embed(_:)` is the primary method. It returns the 256-bit binary Engram +for a piece of text. It throws `embeddingFailed` or `modelUnavailable` +on failure. Every conformer must return the substrate's canonical zero +Engram for an empty string. This is the one input every provider is +guaranteed to agree on. Treating it specially lets empty records from +different providers land on the same identical partition, instead of colliding by coincidence. -`embedFloat(_:)` returns the dense float vector a provider computed on its -way to the Engram, before that vector was compressed into 256 bits. The -protocol's default implementation simply throws `embeddingFailed`, meaning -the float lane is opt-in: a provider that has nothing more precise to -offer than the fingerprint declines rather than fabricate numbers. -Providers that do real inference (MiniLM, mpnet, EmbeddingGemma, defined -in a sibling package) override this to return the vector they already -computed, at no extra inference cost. Empty input returns an empty array, -never a vector of zeros, because a vector of zeros would look like a -legitimate — and misleading — nearest neighbor to every genuinely empty -text. +`embedFloat(_:)` returns the dense float vector a provider computed on +its way to the Engram. This is the vector before it was compressed into +256 bits. The protocol's default implementation simply throws +`embeddingFailed`. This means the float lane is opt-in. A provider with +nothing more precise to offer than the fingerprint declines, rather than +fabricate numbers. Providers that do real inference are defined in a +sibling package. Examples are MiniLM, mpnet, and EmbeddingGemma. They +override this method to return the vector they already computed, at no +extra inference cost. Empty input returns an empty array, never a +vector of zeros. A vector of zeros would look like a legitimate, and +misleading, nearest neighbor to every genuinely empty text. `embedPair(_:)` is a convenience for callers that need both outputs from -one text. Its default implementation calls `embed` and then `embedFloat` -— two inference passes — swallowing a float-lane opt-out into an empty -array so the default matches historical two-call behavior. A provider that -computes both outputs from a single inference pass should override this to -avoid running its model twice. - -`embedBatch(_:)` embeds several texts, defaulting to a sequential loop; -providers with genuinely batched inference should override it for +one text. Its default implementation calls `embed` and then +`embedFloat`, two inference passes. It swallows a float-lane opt-out +into an empty array, so the default matches historical two-call +behavior. A provider might compute both outputs from a single inference +pass. That provider should override this method, to avoid running its +model twice. + +`embedBatch(_:)` embeds several texts. It defaults to a sequential loop. +Providers with genuinely batched inference should override it for throughput. Output order always matches input order. ## FloatSimHashEmbeddingProvider.swift @@ -145,352 +151,376 @@ This file provides `FloatSimHashEmbeddingProvider`, the one concrete `EmbeddingProvider` VectorKit ships. It is a mirror of the Rust `vectorkit::FloatSimHashEmbeddingProvider`. -The type is a thin wrapper: it holds a `modelID`, a `modelVersion`, a -`projectionSeed`, and an injected inference closure — `(String) async -throws -> [Float]` — supplied by whatever host owns the actual model. -VectorKit itself never loads a model or owns a tokenizer; concrete text -providers that do (MiniLM, mpnet, EmbeddingGemma) live in a different -package, CorpusKitProviders, and conform to this same protocol using this -type as their low-level building block. - -`embed(_:)` short-circuits to the canonical zero Engram for empty input -before ever calling the inference closure, guaranteeing the empty-string -contract holds even if the closure itself would have produced something -non-zero. Otherwise it calls the inference closure and passes the result -through `SubstrateML.FloatSimHash.project(vector:seed:)` — a shared, -conformance-gated substrate primitive that projects an arbitrary-length +The type is a thin wrapper. It holds a `modelID`, a `modelVersion`, a +`projectionSeed`, and an injected inference closure. That closure has +the shape `(String) async throws -> [Float]`, and the host that owns the +actual model supplies it. VectorKit itself never loads a model or owns a +tokenizer. Concrete text providers that do live in a different package, +CorpusKitProviders. Examples are MiniLM, mpnet, and EmbeddingGemma. They +conform to this same protocol, using this type as their low-level +building block. + +`embed(_:)` short-circuits to the canonical zero Engram for empty input. +It does this before ever calling the inference closure. This guarantees +the empty-string contract holds, even if the closure itself would have +produced something non-zero. Otherwise it calls the inference closure. +It then passes the result through +`SubstrateML.FloatSimHash.project(vector:seed:)`. This is a shared, +conformance-gated substrate primitive. It projects an arbitrary-length float vector down to a 256-bit fingerprint using a technique called -SimHash. The `projectionSeed` matters here: different seeds turn the same -float vector into different fingerprints, which is what keeps two -different models' outputs from accidentally landing in the same fingerprint -space and being wrongly compared (spec I-4's rule, enforced at the -projection layer as well as at storage). +SimHash. The `projectionSeed` matters here. Different seeds turn the +same float vector into different fingerprints. This is what keeps two +different models' outputs from accidentally landing in the same +fingerprint space and being wrongly compared. This is spec I-4's rule, +enforced at the projection layer as well as at storage. `embedFloat(_:)` returns exactly the float vector the inference closure -produced — the same vector `embed(_:)` feeds into the projection — so a -caller using both lanes never pays for two inference passes. +produced. This is the same vector `embed(_:)` feeds into the projection. +So a caller using both lanes never pays for two inference passes. ## Engine/VectorRecordKey.swift -This file provides `VectorRecordKey`, the identifier every stored vector -carries inside the search engines. Note that this file lives under -`Engine/`, VectorKit's internal folder for the search-engine machinery -shared by every index implementation. - -A key used to be a two-part (item, model) pair, which assumed one vector -per item per model. That assumption breaks for models like ColBERT that -produce one small vector per word instead of one vector per document, so -the key grew a third field. `itemID` names the owning record (a drawer or -a text chunk, as a UUID string). `vectorIndex` is the position of this -vector within its item's sequence — `0` for the ordinary single-vector -case, `0` through `N-1` for a multi-vector item. `modelID` and -`modelVersion` complete the tuple, because vectors from different model -versions are never comparable (spec I-4). - -`VectorRecordKey` is `Comparable`, ordered lexicographically by +This file provides `VectorRecordKey`, the identifier every stored +vector carries inside the search engines. Note that this file lives +under `Engine/`, VectorKit's internal folder for the search-engine +machinery shared by every index implementation. + +A key used to be a two-part pair of item and model. That design assumed +one vector per item per model. The assumption breaks for models like +ColBERT that produce one small vector per word instead of one vector +per document. So the key grew a third field. `itemID` names the owning +record, a drawer or a text chunk, as a UUID string. `vectorIndex` is the +position of this vector within its item's sequence. It is `0` for the +ordinary single-vector case, and `0` through `N-1` for a multi-vector +item. `modelID` and `modelVersion` complete the tuple. Vectors from +different model versions are never comparable, per spec I-4. + +`VectorRecordKey` is `Comparable`. It is ordered lexicographically by `(itemID, vectorIndex, modelID, modelVersion)`. This order is not -incidental: it is the tie-break every search result uses when two matches -land at the same distance, and it is the order the on-disk resident array -is built in, so both the search output and the storage layout agree on -what "sorted" means. +incidental. It is the tie-break every search result uses when two +matches land at the same distance. It is also the order the on-disk +resident array is built in. So both the search output and the storage +layout agree on what sorted means. ## Engine/VectorPayload.swift This file provides `VectorPayload`, the one envelope every vector's raw -bytes travel in, plus `VectorKind` (the tag for which numeric family the -bytes represent) and `VectorPayloadInput` (a payload bundled with its -storage metadata, used for bulk writes). - -`VectorKind` has three cases. `.binary` (raw value 0) is exactly the -32-byte Engram wire form. `.float32` (raw value 1) is `dim × 4` bytes of -IEEE-754 numbers. `.int8` (raw value 2) is reserved for a future quantized -representation; the case exists so that a future ratified quantization -policy does not require a new wire format, but `VectorStore` currently -rejects every `.int8` write (see `VectorKitError.int8QuantizationPolicyUndefined` -above). The raw values are stored on disk and must never be reordered. +bytes travel in. It also provides `VectorKind` and `VectorPayloadInput`. +`VectorKind` tags which numeric family the bytes represent. +`VectorPayloadInput` bundles a payload with its storage metadata, for +bulk writes. + +`VectorKind` has three cases. `.binary`, raw value `0`, is exactly the +32-byte Engram wire form. `.float32`, raw value `1`, is `dim × 4` bytes +of IEEE-754 numbers. `.int8`, raw value `2`, is reserved for a future +quantized representation. The case exists so that a future ratified +quantization policy does not require a new wire format. `VectorStore` +currently rejects every `.int8` write. See +`VectorKitError.int8QuantizationPolicyUndefined` above. The raw values +are stored on disk and must never be reordered. `VectorPayload.init(engram:)` builds a binary payload directly from an -Engram's wire bytes — no conversion, no loss, so every existing binary -test still holds. `init(floats:)` serializes a `[Float]` to little-endian -IEEE-754 bytes explicitly, byte by byte, rather than relying on the -platform's native byte order. This is a deliberate choice, not an -oversight: it is what lets the on-disk `.vec` sidecar be read back -identically on an Apple device and on a Linux server, which may not share -the same native byte order convention. `asEngram()` and `asFloats()` -reverse these conversions, throwing `invalidPayload` when the payload's -kind or byte count does not match what was asked for. +Engram's wire bytes. There is no conversion and no loss, so every +existing binary test still holds. `init(floats:)` serializes a `[Float]` +to little-endian IEEE-754 bytes explicitly, byte by byte, rather than +relying on the platform's native byte order. This is a deliberate +choice, and not an oversight. It is what lets the on-disk `.vec` sidecar +be read back identically on an Apple device and on a Linux server. The +two platforms may not share the same native byte order convention. +`asEngram()` and +`asFloats()` reverse these conversions. They throw `invalidPayload` when +the payload's kind or byte count does not match what was asked for. ## Engine/DenseHit.swift This file provides `DenseHit`, the one result shape every search engine -returns, and `LaneTag`, an enum naming which retrieval technique produced -a given score (used by fusion and multi-technique retrieval code outside -this package). - -A `DenseHit` carries the matched `key`, a `rawDistance`, and the `metric` -that produced it. The tricky design point is that `rawDistance` is a -single `Int32` field shared by very different kinds of number: an integer -Hamming distance for the binary lane, or the bit pattern of a `Float` -distance for the float lane. Two computed properties translate it back: -`hammingDistance` simply casts it to `Int` (safe because Hamming distances -are always 0–256), and `floatDistance` reconstructs the `Float` from its -stored bit pattern. Packing both families into one field, rather than -giving each lane its own result type, is what lets code elsewhere in the -kit graph handle a `[DenseHit]` without caring which lane produced it. - -The file's header calls out an "additive-only" rule: any future field -added here must have a default value, so existing callers who build a -`DenseHit` with the current initializer keep compiling. This matters -because both a Swift and a Rust version of this type exist, and the two -must stay in lockstep. +returns. It also provides `LaneTag`. This is an enum naming which +retrieval technique produced a given score. Fusion and multi-technique +retrieval code outside this package uses it. + +A `DenseHit` carries the matched `key`, a `rawDistance`, and the +`metric` that produced it. The tricky design point is that `rawDistance` +is a single `Int32` field shared by two very different kinds of number. +It can hold an integer Hamming distance for the binary lane. It can also +hold the bit pattern of a `Float` distance for the float lane. Two +computed properties translate it back. `hammingDistance` simply casts +it to `Int`. This is safe because Hamming distances are always in the +range 0 to 256. `floatDistance` reconstructs the `Float` from its stored +bit pattern. Packing both families into one field has one payoff. Code +elsewhere in the kit graph can handle a `[DenseHit]`. It need not care +which lane produced it. This design avoids giving each lane its own +result type. + +The file's header calls out an additive-only rule. Any future field +added here must have a default value. This lets existing callers who +build a `DenseHit` with the current initializer keep compiling. This +matters because both a Swift and a Rust version of this type exist. The +two must stay in lockstep. ## Engine/DenseMetric.swift -This file provides the metric vocabulary for the whole engine seam: -`BinaryMetric` (`.hamming`, `.jaccard`), `FloatMetric` (`.cosine`, `.l2`, -`.dot`), and `DenseMetric`, the umbrella enum wrapping either family so -that `DenseIndex.search` needs only one metric parameter regardless of -which lane it routes to. - -The file's real content is its documentation of a determinism boundary -that recurs throughout this package: `.binary(.hamming)` is "four-way" -bit-identical, because it is pure integer arithmetic computed by a shared, -conformance-gated kernel; `.binary(.jaccard)` is bit-identical up through -its two integer counts, with one final IEEE-754 division that is itself -guaranteed identical because IEEE-754 mandates exact rounding for basic -operations; and every `.float(_)` metric is reproducible only within one -build and platform, never guaranteed identical between Swift and Rust. -This is stated as a documented property of floating-point math, not a -defect to be "fixed" — a warning aimed squarely at a future reviewer who -might otherwise try to force float parity that the underlying arithmetic +This file provides the metric vocabulary for the whole engine seam. It +defines `BinaryMetric`, with cases `.hamming` and `.jaccard`. It defines +`FloatMetric`, with cases `.cosine`, `.l2`, and `.dot`. It defines +`DenseMetric`, the umbrella enum wrapping either family. This means +`DenseIndex.search` needs only one metric parameter, regardless of which +lane it routes to. + +The file's real content is its documentation of a determinism boundary. +This boundary recurs throughout this package. `.binary(.hamming)` is +four-way bit-identical, because it is pure integer arithmetic computed +by a shared, conformance-gated kernel. `.binary(.jaccard)` is +bit-identical through its two integer counts. It ends with one final +IEEE-754 division, which is itself guaranteed identical, because +IEEE-754 mandates exact rounding for basic operations. Every `.float(_)` +metric is reproducible only within one build and platform. It is never +guaranteed identical between Swift and Rust. This is stated as a +documented property of floating-point math. It is not a defect to be +fixed. It is a warning aimed at a future reviewer. That reviewer might +otherwise try to force float parity that the underlying arithmetic cannot honestly provide. ## Engine/DenseIndex.swift -This file provides the `DenseIndex` protocol — the single seam that lets -`VectorStore` treat three very different search engines -(`BruteForceIndex`, `MIHIndex`, `FloatBruteForceIndex`) as -interchangeable — along with three supporting types: `IndexKind` (a tag -naming which implementation is behind a given index, for tests that need -to pick the brute-force oracle deliberately), `SearchDirection`, and -`MetadataFilter`. - -`SearchDirection` distinguishes `.nearest` (most similar first, the -default) from `.farthest` (most dissimilar first). Farthest search -supports an "find things unlike this" query. It is not a trick of negating -a nearest-neighbor list — the farthest items are not among the nearest -top-k at all — so the index has to scan and sort toward the opposite end -using the exact same distance calculation, just the opposite sort -direction. - -`MetadataFilter` restricts a search to one `modelID` and, optionally, one -`modelVersion`. Its `accepts(_:)` method is the single predicate every -engine calls per-candidate; a `nil` field is a wildcard. - -The protocol itself declares four operations — `build(from:)`, +This file provides the `DenseIndex` protocol. This is the single seam +that lets `VectorStore` treat three very different search engines, +`BruteForceIndex`, `MIHIndex`, and `FloatBruteForceIndex`, as +interchangeable. The file also provides three supporting types: +`IndexKind`, `SearchDirection`, and `MetadataFilter`. `IndexKind` is a +tag naming which implementation is behind a given index. Tests use it +to pick the brute-force oracle deliberately. + +`SearchDirection` has two cases. `.nearest` is most similar first, and +is the default. `.farthest` is most dissimilar first. Farthest search +supports a query for things unlike this one. It is not a trick of +negating a nearest-neighbor list. The farthest items are not among the +nearest top-k at all. So the index has to scan and sort toward the +opposite end. It uses the exact same distance calculation, only the +opposite sort direction. + +`MetadataFilter` restricts a search to one `modelID` and, optionally, +one `modelVersion`. Its `accepts(_:)` method is the single predicate +every engine calls per candidate. A `nil` field acts as a wildcard. + +The protocol itself declares four operations: `build(from:)`, `search(probe:metric:k:filter:)`, `add(key:vector:)`, and -`remove(key:)` — and documents the contract every conformer must honor: -results come back sorted by distance ascending, ties broken by the -matched key ascending, and `BruteForceIndex` is the correctness oracle -every other binary engine is measured against. +`remove(key:)`. It documents the contract every conformer must honor. +Results come back sorted by distance ascending. Ties break by the +matched key ascending. `BruteForceIndex` is the correctness oracle every +other binary engine is measured against. ## Engine/BruteForceIndex.swift -This file provides `BruteForceIndex`, the exact linear-scan search engine -for binary (Hamming) vectors, and the conformance oracle every other -binary engine — currently just `MIHIndex` — is checked against. +This file provides `BruteForceIndex`. This is the exact linear-scan +search engine for binary vectors, using Hamming distance. It is also +the conformance oracle every other binary engine is checked against. +Today that is just `MIHIndex`. The file is built around one hard rule, restated three times in its -comments: it performs zero Hamming arithmetic itself. Every distance is -computed by `EngramLib.distances`, which routes to a shared kernel -selected once per process (NEON on Apple silicon hardware, a scalar -fallback elsewhere) and checked for identical output across four build -configurations. Reimplementing a bitwise XOR-and-count here, even a -correct one, would bypass that check and risk silent divergence between -platforms — this is the file's version of spec I-7. - -`search(probe:metric:k:filter:)` validates the probe (must be exactly 32 -bytes of `.binary` kind) and the metric (only `.binary(.hamming)` is -supported; other requests throw `invalidPayload`), narrows the scan to one -model's slot range when a filter is present (an `O(log m)` lookup into a -sorted partition index, avoiding a full-array walk), collects the live, -un-tombstoned candidates in that range, and hands their Engrams to +comments. It performs zero Hamming arithmetic itself. Every distance is +computed by `EngramLib.distances`. This routes to a shared kernel, +selected once per process. The kernel uses NEON on Apple silicon +hardware, and a scalar fallback elsewhere. It is checked for identical +output across four build configurations. Reimplementing a bitwise +XOR-and-count here, even a correct one, would bypass that check. It +would risk silent divergence between platforms. This is the file's +version of spec I-7. + +`search(probe:metric:k:filter:)` validates the probe, which must be +exactly 32 bytes of `.binary` kind, and the metric. Only +`.binary(.hamming)` is supported, and other requests throw +`invalidPayload`. It narrows the scan to one model's slot range when a +filter is present. This is an `O(log m)` lookup into a sorted partition +index. The lookup avoids a full-array walk. It collects the live, +un-tombstoned +candidates in that range and hands their Engrams to `EngramLib.distances` in one batch call. It deliberately avoids `EngramLib.findNearest`, which applies a different, insertion-order -tie-break; this file sorts the returned distances itself by -`(distance ascending, key ascending)`, because the engine's own contract -requires the full `VectorRecordKey` as the tie-break, not just the array's -insertion position — otherwise two records under the same item but -different model or vector index could be returned inconsistently. - -`add(key:vector:)` implements upsert by tombstoning any existing slot with -the same key before appending the new bytes, then rebuilding the sorted -model-partition index from the updated key list. `remove(key:)` tombstones -every matching slot without touching the underlying storage bytes — actual -space reclamation is `ResidentArrayStore`'s job, not this type's. -`currentSnapshot()` returns a value-type copy of the live array so callers -outside the actor (chiefly `VectorStore`, when it needs to scan for -tombstoning) can read it safely. +tie-break. This file sorts the returned distances itself, by distance +ascending and then key ascending. The engine's own contract requires the +full `VectorRecordKey` as the tie-break, not just the array's insertion +position. Otherwise two records under the same item but different model +or vector index could be returned inconsistently. + +`add(key:vector:)` implements upsert by tombstoning any existing slot +with the same key before appending the new bytes. It then rebuilds the +sorted model-partition index from the updated key list. `remove(key:)` +tombstones every matching slot without touching the underlying storage +bytes. Actual space reclamation is `ResidentArrayStore`'s job, not this +type's. `currentSnapshot()` returns a value-type copy of the live array. +This lets callers outside the actor, chiefly `VectorStore` when it needs +to scan for tombstoning, read it safely. ## Engine/FloatBruteForceIndex.swift -This file provides `FloatBruteForceIndex`, the linear-scan search engine -for the float32 lane — cosine, Euclidean (`l2`), and dot-product distance -— and, unlike the binary lane, this is both the correctness reference and -the production search path: there is no separate accelerated float engine -in this package. +This file provides `FloatBruteForceIndex`. This is the linear-scan +search engine for the float32 lane. It supports three float metrics: +cosine, Euclidean, and dot-product distance. The code calls Euclidean +distance `l2`. Unlike the binary lane, this is both the correctness +reference and the production search path. There is no separate +accelerated float engine in this package. -The file opens with an emphatic warning, repeated in this document because -it protects against a plausible but wrong "fix": float arithmetic here is -reproducible on one build and platform, but it is not, and cannot be +The file opens with an emphatic warning, repeated in this document +because it protects against a plausible but wrong fix. Float arithmetic +here is reproducible on one build and platform. It is not, and cannot be made to be, bit-identical between Swift and Rust or across different hardware. This is a documented property of IEEE-754 arithmetic, not an -oversight; a reviewer must not try to force it to match the binary lane's -four-way guarantee. - -`build(from:)` simply stores a reference to the supplied array — there is -no secondary structure to construct, so building is `O(1)`; the real cost -is whatever the caller paid to assemble the array. `search(probe:metric: -k:filter:)` validates that the probe is `.float32`, that the requested -metric is a float metric, and that the probe's byte count matches both its -own declared dimension and the array's fixed stride (a mismatch would -otherwise read past the end of a slot and throws instead). It then scans -every live, filter-passing slot, computing one of three distances per -candidate — cosine distance treats a zero vector as maximally distant -rather than crashing on a divide-by-zero; `l2` is the plain Euclidean -formula; `dot` is negated so that "smaller is nearer" holds for every -metric uniformly — and sorts ascending by distance, then by key. -`searchFarthest(probe:metric:k:filter:)` reuses the identical scan and -identical distance math, changing only the sort direction to descending, -which is what makes "find dissimilar items" a real bottom-of-the-list scan -rather than a negated top-of-the-list one. - -`add(key:vector:)` establishes the array's dimension from the first vector -added and rejects any later vector of a different byte count, because a +oversight. A reviewer must not try to force it to match the binary +lane's four-way guarantee. + +`build(from:)` simply stores a reference to the supplied array. There is +no secondary structure to construct, so building is `O(1)`. The real +cost is whatever the caller paid to assemble the array. +`search(probe:metric:k:filter:)` validates three things. The probe must +be `.float32`. The requested metric must be a float metric. The probe's +byte count must match both its own declared dimension and the array's +fixed stride. A mismatch would otherwise read past the end of a slot, so +it throws instead. It then scans every live, filter-passing slot, computing +one of three distances per candidate. Cosine distance treats a zero +vector as maximally distant rather than crashing on a divide-by-zero. +`l2` is the plain Euclidean formula. `dot` is negated so that smaller is +nearer holds for every metric uniformly. The scan sorts ascending by +distance, then by key. `searchFarthest(probe:metric:k:filter:)` reuses +the identical scan and identical distance math. It changes only the sort +direction to descending. This is what makes finding dissimilar items a +real bottom-of-the-list scan, rather than a negated top-of-the-list one. + +`add(key:vector:)` establishes the array's dimension from the first +vector added. It rejects any later vector of a different byte count. A mismatched stride would silently corrupt the flat storage buffer. -`remove(key:)` tombstones the matching slot; actual compaction happens the -next time `build(from:)` runs with a freshly assembled array. +`remove(key:)` tombstones the matching slot. Actual compaction +happens the next time `build(from:)` runs with a freshly assembled +array. ## Engine/MaxSimScorer.swift -This file provides `MaxSimScorer` and `MaxSimHit`, the exhaustive -("Exact-A") implementation of ColBERT-style late-interaction scoring over -binary token fingerprints. - -Some embedding techniques represent one document as many small vectors — -one per word or token — rather than one vector for the whole document. -Comparing two such documents means asking, for every word in the query, -"which word in this document is most like it?" and adding up those best -matches. That sum, `Σ (256 − minimum Hamming distance)` over every query -token, is the MaxSim score this file computes. Because it examines every -query token against every document token for every candidate document, it -never skips a candidate; this exhaustiveness is precisely what makes it -the correctness reference for any faster, pruned variant built later — the -file's header explicitly reserves the accelerated two-stage variant as out -of scope here. +This file provides `MaxSimScorer` and `MaxSimHit`. Together these form +the exhaustive, or Exact-A, implementation of ColBERT-style +late-interaction scoring over binary token fingerprints. + +Some embedding techniques represent one document as many small vectors. +One vector exists per word or token, not one vector for the whole +document. Comparing two such documents means asking one question, for +every word in the query. Which word in this document is most like it. +The scorer then adds up +those best matches. That sum, `Σ (256 − minimum Hamming distance)` over +every query token, is the MaxSim score this file computes. It examines +every query token against every document token for every candidate +document. So it never skips a candidate. This exhaustiveness is +precisely what makes it the correctness reference for any faster, pruned +variant built later. The file's header explicitly reserves the +accelerated two-stage variant as out of scope here. `score(queryTokens:documents:k:)` iterates the supplied documents in -ascending itemID order — sorting the dictionary's keys explicitly, because -a Swift dictionary's own iteration order is not guaranteed and would make -results non-reproducible — computes each document's MaxSim score, sorts -the results `(score descending, itemID ascending)`, and truncates to `k` -only after the full sort, never before, so a document that would have -scored well is never cut for appearing late in an unsorted pass. Every -Hamming distance again goes through `EngramLib.Session.distances`, -constructed once per `MaxSimScorer` and reused for the whole call, so the -one-time cost of picking the fastest available kernel is paid once rather -than per comparison. +ascending itemID order. It sorts the dictionary's keys explicitly. A +Swift dictionary's own iteration order is not guaranteed. Relying on it +would make results non-reproducible. It computes each document's MaxSim +score. It sorts the results by score descending, then itemID ascending. +It truncates to `k` only after the full sort, never before. So a +document that would have scored well is never cut for appearing late in +an unsorted pass. Every Hamming distance again goes through +`EngramLib.Session.distances`. This session is constructed once per +`MaxSimScorer` and reused for the whole call. The one-time cost of +picking the fastest available kernel is paid once, rather than per +comparison. ## Engine/ResidentVectorArray.swift -This file provides `ResidentVectorArray`, the packed in-memory data shape -every search engine reads from, and `ModelPartitionEntry`, one entry in -its per-model index. This is the shared contract underneath -`BruteForceIndex`, `MIHIndex`, and `FloatBruteForceIndex` — all three read -the identical layout, which is what lets `VectorStore` build the array -once and hand it to whichever engine is currently active. - -The design reason is stated directly in the file's header comment: -measurement on the pre-existing code path showed that fetching and -decoding rows from the database consumed 87% of a search's latency, while -the actual distance kernel took 0.4%. A fixed-stride, contiguous byte -array removes the fetch-and-decode cost from every query after the first, -because the whole array is loaded once and then scanned as a flat block of -memory with no per-row allocation. - -The type stores `kind` and `stride` (bytes per vector slot — 32 for -binary, `dim × 4` for float32), `count` (including tombstoned slots), -`storage` (the packed bytes themselves — on Apple platforms potentially -memory-mapped read-only from a sidecar file), a `keys` array parallel to -`storage`, a sorted `modelPartitions` index, and a `tombstones` bitmap. -`liveCount` walks the tombstone bitmap to compute how many slots are still -valid; `partitionRange(for:)` binary-searches the sorted partitions to -find one model's slot range in `O(log m)`; `isTombstoned(_:)` and -`vectorBytes(at:)` are the two per-slot accessors every engine's scan loop -calls. +This file provides `ResidentVectorArray`, the packed in-memory data +shape every search engine reads from. It also provides +`ModelPartitionEntry`, one entry in its per-model index. This is the +shared contract underneath `BruteForceIndex`, `MIHIndex`, and +`FloatBruteForceIndex`. All three read the identical layout. This is +what lets `VectorStore` build the array once and hand it to whichever +engine is currently active. + +The design reason is stated directly in the file's header comment. +Measurement on the pre-existing code path found a bottleneck. Fetching +and decoding rows from the database consumed eighty-seven percent of a +search's latency. The actual distance kernel took under one percent. A +fixed-stride, contiguous byte array removes the fetch-and-decode cost +from every query after the first. The whole array is loaded once. It is +then scanned as a flat block of memory with no per-row allocation. + +The type stores `kind` and `stride`, the bytes per vector slot: thirty-two +for binary, `dim × 4` for float32. It stores `count`, including +tombstoned slots, and `storage`, the packed bytes themselves. On Apple +platforms, `storage` may be memory-mapped read-only from a sidecar file. +It also stores a `keys` array parallel to `storage`, a sorted +`modelPartitions` index, and a `tombstones` bitmap. `liveCount` walks the +tombstone bitmap to compute how many slots are still valid. +`partitionRange(for:)` binary-searches the sorted partitions to find one +model's slot range in `O(log m)`. `isTombstoned(_:)` and +`vectorBytes(at:)` are the two per-slot accessors every engine's scan +loop calls. ## Engine/ResidentArrayStore.swift -This file provides `ResidentArrayStore`, the actor that owns the optional -on-disk `.vec` sidecar file: a packed binary cache of a -`ResidentVectorArray` that lets a reopened store skip rebuilding the array -from every database row. - -The file documents its own on-disk format in full — a fixed header -(magic bytes, format version, vector kind, stride, count, a live-slot -count, and the tombstone bitmap), followed by the packed vector bytes, -then variable-length key records, then the model-partition index, with -every multi-byte integer explicitly little-endian so the same file reads -identically on an Apple device and on a Linux server. `writeSidecar` -writes to a temporary file and atomically renames it into place, so a -crash mid-write never leaves a half-written sidecar behind; `readSidecar` -memory-maps the file where the platform supports it (a load-time -optimization, not a difference in the bytes returned) and `parseSidecar` -does the actual decoding, checking every length field against the -remaining buffer size before trusting it, so a corrupted or hand-edited -file is rejected with a `decodingFailure` rather than crashing the -process. - -The file is explicit about one policy: the `vectors` database table is -always the durable source of truth, and this sidecar is a regenerable -cache, never a second copy of record. `load()` reads the sidecar if -present; if it is missing or fails to parse, the store simply starts -empty and waits for `VectorStore` to rebuild it from the table. - -Three write paths exist because a single write policy could not serve -both a low-latency single insert and a large bulk import well. -`append(key:bytes:)` is the eager path: it writes the sidecar -immediately after every single addition. `appendDeferred(key:bytes:)` -is the "write-behind" path a single insert uses in production: it updates -the in-memory array and marks the store dirty without touching disk, -trusting the caller to call `flush()` at a natural pause. This is safe -because the database row was already written durably before this call — -losing an unflushed sidecar only costs a rebuild on the next open, never -data. `appendBatch(records:)` is the bulk-import path: it extends storage, -keys, and the tombstone bitmap for the whole batch in one pass and writes -the sidecar exactly once, so importing a thousand vectors costs one disk -write instead of a thousand. - -`compact()` rewrites the sidecar keeping only live (non-tombstoned) -records, sorted by key for a deterministic, reproducible layout, and is -triggered automatically whenever the tombstone ratio exceeds -`compactionThreshold` (25% by default) after any eager write. +This file provides `ResidentArrayStore`, the actor that owns the +optional on-disk `.vec` sidecar file. This file is a packed binary cache +of a `ResidentVectorArray`. It lets a reopened store skip rebuilding the +array from every database row. + +The file documents its own on-disk format in full. A fixed header holds +magic bytes, format version, vector kind, stride, count, a live-slot +count, and the tombstone bitmap. This header is followed by the packed +vector bytes, then variable-length key records, then the +model-partition index. Every multi-byte integer is explicitly +little-endian. So the same file reads identically on an Apple device +and on a Linux server. `writeSidecar` writes to a temporary file and +atomically renames it into place. So a crash mid-write never leaves a +half-written sidecar behind. `readSidecar` memory-maps the file where +the platform supports it. This is a load-time optimization, not a +difference in the bytes returned. `parseSidecar` does the actual +decoding. It +checks every length field against the remaining buffer size before +trusting it. So a corrupted or hand-edited file is rejected with a +`decodingFailure`, rather than crashing the process. + +The file is explicit about one policy. The `vectors` database table is +always the durable source of truth. This sidecar is a regenerable cache, +and never a second copy of record. `load()` reads the sidecar if +present. If it is missing or fails to parse, the store simply starts +empty. It then waits for `VectorStore` to rebuild it from the table. + +Three write paths exist. A single write policy could not serve both a +low-latency single insert and a large bulk import well. +`append(key:bytes:)` is the eager path. It writes the sidecar +immediately after every single addition. `appendDeferred(key:bytes:)` is +the write-behind path a single insert uses in production. It updates the +in-memory array and marks the store dirty, without touching disk. It +trusts the caller to call `flush()` at a natural pause. This is safe +because the database row was already written durably before this call. +Losing an unflushed sidecar only costs a rebuild on the next open, never +data. `appendBatch(records:)` is the bulk-import path. It extends +storage, keys, and the tombstone bitmap for the whole batch in one pass. +It writes the sidecar exactly once, so importing a thousand vectors +costs one disk write instead of a thousand. + +`compact()` rewrites the sidecar, keeping only live, non-tombstoned, +records. It sorts them by key for a deterministic, reproducible layout. +This is triggered automatically whenever the tombstone ratio exceeds +`compactionThreshold`, twenty-five percent by default, after any eager +write. ## StoredVector.swift -This file provides `StoredVector`, the public, fully-decoded row shape -`VectorStore.vectors(forItemID:)` returns to callers who want the -convenient binary form rather than the raw typed payload. - -Its fields mirror the `vectors` table's columns directly: a stable `id` -assigned on insert, the owning `itemID`, the `vectorIndex` position within -a multi-vector item, `modelID` and `modelVersion` (spec I-4's tags), -the decoded `engram`, and `filedAt`, the time the row was written, -round-tripped through the database's text-based ISO 8601 timestamp column -(which loses sub-millisecond precision — the file notes this explicitly so -a caller comparing timestamps at fine granularity is not surprised). -`StoredVector.engram` is non-nil only for binary rows; a float or int8 row -must be read through `VectorStore.getPayload` instead, since this +This file provides `StoredVector`. This is the public, fully decoded row +shape that `VectorStore.vectors(forItemID:)` returns. Callers want this +convenient binary form, rather than the raw typed payload. + +Its fields mirror the `vectors` table's columns directly. The fields +are `id`, `itemID`, `vectorIndex`, `modelID`, `modelVersion`, `engram`, +and `filedAt`. The `id` is a stable value assigned on insert. The +`itemID` is the owning record. The `vectorIndex` is the position within +a multi-vector item. The `modelID` and `modelVersion` are spec I-4's +tags. The `engram` is the decoded fingerprint. The `filedAt` field is +the time the row was written. This last field is round-tripped through +the database's text-based ISO 8601 timestamp column. That column loses +sub-millisecond precision. The file notes this explicitly, so a caller +comparing timestamps at fine granularity is not surprised. +`StoredVector.engram` is non-nil only for binary rows. A float or int8 +row must be read through `VectorStore.getPayload` instead. This convenience type only round-trips the binary case. ## VectorMatch.swift @@ -498,161 +528,176 @@ convenience type only round-trips the binary case. This file provides `VectorMatch`, the public search-result shape `VectorStore.findNearest` and its float-lane counterparts return. -It carries `itemID` (the matched record), `distance` (Hamming distance for -the binary lane, an integer in 0…256; a scaled, quantized cosine distance -for the float lane, explained in `VectorStore`'s float-search functions), -and `modelID`, so a caller can confirm which model actually produced the -match. `VectorMatch` conforms to `Comparable`, ordered by `distance` -ascending with ties broken by `itemID` ascending — the same universal -tie-break rule used throughout the engine layer — so a sorted array of -matches reads nearest-to-farthest from front to back without a caller -having to know the sort convention. +It carries `itemID`, `distance`, and `modelID`. The `itemID` is the +matched record. The `modelID` lets a caller confirm which model +actually produced the match. `distance` is a Hamming distance for the +binary lane, an integer from zero to two hundred fifty-six. For the +float lane it is a scaled, quantized cosine distance, explained in +`VectorStore`'s float-search functions. +`VectorMatch` conforms to `Comparable`. It is ordered by `distance` +ascending, with ties broken by `itemID` ascending. This is the same +universal tie-break rule used throughout the engine layer. So a sorted +array of matches reads nearest to farthest, from front to back. A +caller need not know the sort convention. ## VectorStore.swift This file provides `VectorStore`, the actor every consumer of VectorKit -actually talks to. It is the largest file in the package because it is -where every other piece — the durable table, the resident arrays, the -three search engines, and telemetry — is wired together into one -consistent API. +actually talks to. It is the largest file in the package. Every other +piece is wired together here into one consistent API. This includes the +durable `vectors` table, the resident arrays, the three search engines, +and telemetry. ### Storage and schema -`VectorStore` wraps a PersistenceKit `Storage` backend (SQLite, an -in-memory backend for tests, or, in the future, PostgreSQL); the kit never -sees which backend is chosen — that decision belongs to the application. -`schemaDeclaration` is the static schema description passed to -`storage.open(schema:)` before the store is used: one `vectors` table, -version 3, whose `UNIQUE(item_id, vector_index, model_id)` constraint is -exactly `VectorRecordKey` minus `modelVersion` — the constraint that makes -an upsert on a changed model version a true replacement of the old row -rather than a duplicate. +`VectorStore` wraps a PersistenceKit `Storage` backend. Supported +backends include SQLite and an in-memory test backend. A PostgreSQL +backend may be added in the future. The kit never sees which backend is +chosen. That decision belongs to the application. `schemaDeclaration` is +the static schema description passed +to `storage.open(schema:)` before the store is used. It describes one +`vectors` table, version three. Its `UNIQUE(item_id, vector_index, +model_id)` constraint is exactly `VectorRecordKey` minus `modelVersion`. +This constraint makes an upsert on a changed model version a true +replacement of the old row. It is not treated as a duplicate. ### Two hot-path structures kept in sync Every write updates three things together: the durable `vectors` table -row, the in-memory resident array (through `bruteForceIndex` and -`mihIndex`, which are both always kept current — only one is ever the -active `hotIndex`), and, when a sidecar was configured at construction, -the on-disk cache. `_selectIndex()` compares `liveBinaryCount` against -`mihThreshold` (50,000 by default) after every write that changes the -count, swapping `hotIndex` between `bruteForceIndex` and `mihIndex` with a -plain reference assignment — no rebuild is needed on promotion or -demotion, because both indexes were already current. `init(storage: -sidecarURL:mihThreshold:mihBandCount:deferredPendingLimit:)` allocates both -index actors up front so this swap never needs to construct anything at -query time. +row, the in-memory resident array, and the on-disk cache. The resident +array updates through `bruteForceIndex` and `mihIndex`. The on-disk +cache updates only when a sidecar was configured at construction. Both +`bruteForceIndex` and `mihIndex` are always kept current. Only one is +ever the active `hotIndex`. +`_selectIndex()` compares `liveBinaryCount` against `mihThreshold`, +fifty thousand by default, after every write that changes the count. It +swaps `hotIndex` between `bruteForceIndex` and `mihIndex` with a plain +reference assignment. No rebuild is needed on promotion or demotion, +because both indexes were already current. +`init(storage:sidecarURL:mihThreshold:mihBandCount:deferredPendingLimit:)` +allocates both index actors up front. So this swap never needs to +construct anything at query time. ### Write path -`addVector(itemID:engram:modelID:modelVersion:filedAt:)` is a convenience -wrapper for the common single binary vector case; it builds a -`VectorPayload` and delegates to `addPayload`. `addPayload(itemID: -vectorIndex:payload:modelID:modelVersion:filedAt:)` is the general write: -it rejects `.int8` payloads immediately (see `VectorKitError` above), -writes the row via an upsert keyed on the table's unique constraint, and -then — only for `.binary` and `.float32` kinds — mirrors the write into -the matching resident array. For a binary write, it first finds any -existing slot at the same logical position (`itemID`, `vectorIndex`, -`modelID`, ignoring `modelVersion`) so that a version change is treated as -a true replacement rather than leaving a stale duplicate slot behind; this -matching is deliberately looser than full key equality specifically to -catch that case. It then tombstones the stale slot (if any) and appends -the new one in both `bruteForceIndex` and `mihIndex`, updates -`liveBinaryCount` only when the write was genuinely new, and calls -`_selectIndex()`. Every write emits a `vectorkit.index.insert_latency_ms` -telemetry metric through IntellectusLib — a short-circuited no-op unless -monitoring has been explicitly turned on, so the cost on the default path -is one boolean check. - -`addPayloads(_:)` is the bulk-import counterpart, and its whole reason for -existing is complexity: importing N vectors one at a time through -`addPayload` costs N sidecar rewrites and, without care, N index rebuilds. -This function upserts every row to the table (unavoidable — the table is -the durable source), then rebuilds both binary indexes exactly once from -the final merged array rather than once per row, cutting the amortized -cost from `O(N²)` bytes written to `O(N)`. `beginDeferredIndex()` and -`publishResidentIndex()` extend this further for very large or -multi-call bulk imports: while a deferred window is open, `addPayloads` -appends to storage but skips the index rebuild entirely, seeding an -in-memory tracked set of live keys so replacement detection stays cheap -across the whole window; `publishResidentIndex()` performs the single -rebuild the whole burst needed, once, when the caller signals the burst is -finished. `deferredPendingRecords`, capped at `deferredPendingLimit` -(50,000 by default), guards the memory-only variant of this path against -unbounded growth if a caller holds the window open indefinitely; crossing -the cap triggers `_flushDeferredPending()`, an internal intermediate merge -that keeps the deferred window open for the caller while bounding peak +`addVector(itemID:engram:modelID:modelVersion:filedAt:)` is a +convenience wrapper for the common single binary vector case. It builds +a `VectorPayload` and delegates to `addPayload`. +`addPayload(itemID:vectorIndex:payload:modelID:modelVersion:filedAt:)` +is the general write. It rejects `.int8` payloads immediately. See +`VectorKitError` above for that case. It writes the row through an +upsert keyed on the table's unique constraint. It then mirrors the +write into the matching resident array, but only for `.binary` and +`.float32` kinds. For a binary write, `addPayload` first finds any +existing slot at the same logical position. This position is defined by +`itemID`, `vectorIndex`, and `modelID`. The match ignores `modelVersion`. +This lets a version change be treated as a true replacement, rather +than leaving a stale duplicate slot behind. This matching is +deliberately looser than full key equality, specifically to catch that +case. It then tombstones the stale slot, if any, and appends the new +one in both +`bruteForceIndex` and `mihIndex`. It updates `liveBinaryCount` only when +the write was genuinely new, and calls `_selectIndex()`. Every write +emits a `vectorkit.index.insert_latency_ms` telemetry metric through +IntellectusLib. This metric is a short-circuited no-op unless monitoring +has been explicitly turned on. So the cost on the default path is one +boolean check. + +`addPayloads(_:)` is the bulk-import counterpart. Its whole reason for +existing is complexity. Importing N vectors one at a time through +`addPayload` costs N sidecar rewrites and, without care, N index +rebuilds. This function upserts every row to the table, unavoidable +because the table is the durable source. It then rebuilds both binary +indexes exactly once, from the final merged array rather than once per +row. This cuts the amortized cost from `O(N²)` bytes written to `O(N)`. +`beginDeferredIndex()` and `publishResidentIndex()` extend this further +for very large or multi-call bulk imports. While a deferred window is +open, `addPayloads` appends to storage but skips the index rebuild +entirely. It seeds an in-memory tracked set of live keys, so replacement +detection stays cheap across the whole window. `publishResidentIndex()` +performs the single rebuild the whole burst needed, once, when the +caller signals the burst is finished. `deferredPendingRecords` is +capped at `deferredPendingLimit`, fifty thousand by default. This cap +guards the memory-only variant of this path against unbounded growth. +This applies if a caller holds the window open indefinitely. Crossing +the cap +triggers `_flushDeferredPending()`, an internal intermediate merge that +keeps the deferred window open for the caller, while bounding peak memory. ### Search path -`findNearest(probe:modelID:limit:)` is the binary-lane search: it lazily -builds the resident array on first use (from the sidecar if one is -current, or from the table if not), then delegates entirely to -`hotIndex.search`, converting the returned `[DenseHit]` into -`[VectorMatch]` without re-sorting — the engine has already applied the -required `(distance ascending, itemID ascending)` order. -`findNearestFloat(probe:modelID:limit:)` is the float-lane equivalent: it -lazily builds a `FloatBruteForceIndex` per model (the map's presence is -the "already built" flag — there is no separate boolean), then searches -by cosine distance and quantizes the resulting `Float` distance to an -integer by multiplying by 10,000 and rounding, so results from different -languages' fixtures can be compared exactly rather than approximately. -`findFarthestFloat(probe:modelID:limit:)` is identical except it calls the -engine's farthest-ranking search, for an anti-similarity "find things -unlike this" query. `findByKeyword(_:limit:)` is a plain substring match -on `item_id`, explicitly documented as a quick pre-filter for hybrid -retrieval, not a real keyword search — full keyword scoring is a different +`findNearest(probe:modelID:limit:)` is the binary-lane search. It +lazily builds the resident array on first use. It builds this from the +sidecar if one is current, or from the table if not. It then delegates +entirely to `hotIndex.search`, converting the returned `[DenseHit]` +into `[VectorMatch]` without re-sorting. The engine has already applied +the required order of distance ascending, then itemID ascending. +`findNearestFloat(probe:modelID:limit:)` is the float-lane equivalent. +It lazily builds a `FloatBruteForceIndex` per model. The map's presence +is the already-built flag. There is no separate boolean. It then +searches by cosine distance. It quantizes the resulting `Float` distance +to an integer, by multiplying by ten thousand and rounding. So +results from different languages' fixtures can be compared exactly, +rather than approximately. `findFarthestFloat(probe:modelID:limit:)` is +identical, except it calls the engine's farthest-ranking search, for an +anti-similarity query for things unlike this one. +`findByKeyword(_:limit:)` is a plain substring match on `item_id`. It is +explicitly documented as a quick pre-filter for hybrid retrieval, and +not a real keyword search. Full keyword scoring is a different package's responsibility. ### Delete path -`deleteVector(itemID:modelID:)` deletes the row at `vectorIndex` 0 and -tombstones every matching resident slot. `deleteAllVectors(itemID: -modelID:)` deletes every vector index for an item and model, used when a -multi-vector item (all its ColBERT token vectors, for instance) needs -complete removal. Both first flush any in-flight deferred-index burst, -so a delete never races an unpublished bulk import. `destroyAllVectors()` -wipes the entire store — every row, both resident indexes, the sidecar, -and the per-model float indexes — as part of a coordinated estate -teardown; an estate is one user's complete memory store in MOOTx01. +`deleteVector(itemID:modelID:)` deletes the row at `vectorIndex` zero +and tombstones every matching resident slot. +`deleteAllVectors(itemID:modelID:)` deletes every vector index for an +item and model. This is used when a multi-vector item, such as all of a +ColBERT item's token vectors, needs complete removal. Both functions +first flush any in-flight deferred-index burst, so a delete never races +an unpublished bulk import. `destroyAllVectors()` wipes the entire +store: every row, both resident indexes, the sidecar, and the per-model +float indexes. This runs as part of a coordinated estate teardown. An +estate is one user's complete memory store in MOOTx01. ### Coherence helpers -`_ensureIndexBuilt()` is the one-time (per process) function that -populates both binary resident indexes, either by trusting a sidecar whose -recorded live-slot count matches the table's live binary-row count, or, if -they disagree, by rebuilding from the table and rewriting the sidecar. -Comparing live-count to live-count, rather than the older approach of -comparing total slot counts, avoids a spurious full rebuild after ordinary -deletions leave tombstoned slots behind. `decodePayload(from:)` and -`storedVector(from:)` are the row-decoding functions every read path -shares; both explicitly guard every narrowing integer conversion (for -example, a negative `dim` or an out-of-range `kind` byte) so that a -hand-crafted or corrupted row is rejected with `nil` rather than crashing -the process on a Swift trap. +`_ensureIndexBuilt()` is the one-time, per process, function that +populates both binary resident indexes. It trusts a sidecar when its +recorded live-slot count matches the table's live binary-row count. If +the counts disagree, it rebuilds from the table and rewrites the +sidecar. Comparing live count to live count avoids a spurious full +rebuild after ordinary deletions leave tombstoned slots behind. The +older approach compared total slot counts instead. +`decodePayload(from:)` and `storedVector(from:)` are the row-decoding +functions every read path shares. Both explicitly guard every narrowing +integer conversion, for example a negative `dim` or an out-of-range +`kind` byte. So a hand-crafted or corrupted row is rejected with `nil`, +rather than crashing the process on a Swift trap. ## Rust Port and Conformance -The `rust/` directory mirrors the Swift implementation file for file: -`vector_store.rs` alongside `engine/brute_force.rs`, `engine/mih.rs`, -`engine/float_brute_force.rs`, `engine/max_sim.rs`, `engine/resident.rs`, -`engine/resident_store.rs`, `engine/key.rs`, `engine/payload.rs`, -`engine/hit.rs`, `engine/metric.rs`, `engine/seam.rs`, plus -`embedding_provider.rs`, `simhash_embedding_provider.rs`, and `error.rs`. -The `.vec` sidecar format, the MIH band-hashing algorithm, and the MaxSim -scoring algorithm are all specified precisely enough in the Swift source -comments (colex enumeration order, the enumeration-budget guard's integer -arithmetic, the little-endian sidecar layout) that both ports are expected -to agree exactly on the binary lane. `rust/tests/` holds integration -suites for bulk ingest, the float lane, int8 rejection, the SimHash -provider, the vector store, and telemetry, exercising the same behaviors -described above. The package's own `MIHIndexTests.swift` gates `MIHIndex` -against `BruteForceIndex` directly within Swift; cross-language -conformance for the binary lane rests on both ports implementing the same -documented algorithm rather than on a single shared fixture file, unlike +The `rust/` directory mirrors the Swift implementation file for file. +It provides `vector_store.rs`, alongside `engine/brute_force.rs`, +`engine/mih.rs`, `engine/float_brute_force.rs`, `engine/max_sim.rs`, +`engine/resident.rs`, `engine/resident_store.rs`, `engine/key.rs`, +`engine/payload.rs`, `engine/hit.rs`, `engine/metric.rs`, +`engine/seam.rs`, plus `embedding_provider.rs`, +`simhash_embedding_provider.rs`, and `error.rs`. Three things are +specified precisely enough in the Swift source comments: the `.vec` +sidecar format, the MIH band-hashing algorithm, and the MaxSim scoring +algorithm. This includes colex enumeration order, the enumeration-budget +guard's integer arithmetic, and the little-endian sidecar layout. Both +ports are expected to agree exactly on the binary lane. `rust/tests/` +holds integration suites for bulk ingest, the float lane, int8 +rejection, the SimHash provider, the vector store, and telemetry. These +suites exercise the same behaviors described above. The package's own +`MIHIndexTests.swift` gates `MIHIndex` against `BruteForceIndex` +directly within Swift. Cross-language conformance for the binary lane +rests on both ports implementing the same documented algorithm. It does +not rest on a single shared fixture file. This differs from LatticeLib's shared JSON fixtures. The float lane is, by design, exempt -from cross-language bit-identity (see `DenseMetric.swift` and -`FloatBruteForceIndex.swift` above); only within-platform reproducibility -and rank correctness are asserted for it, in both languages. +from cross-language +bit-identity. See `DenseMetric.swift` and `FloatBruteForceIndex.swift` +above. Only within-platform reproducibility and rank correctness are +asserted for it, in both languages. diff --git a/packages/kits/VectorKit/docs/OVERVIEW.md b/packages/kits/VectorKit/docs/OVERVIEW.md index 726ee1c..4aaccfc 100644 --- a/packages/kits/VectorKit/docs/OVERVIEW.md +++ b/packages/kits/VectorKit/docs/OVERVIEW.md @@ -47,156 +47,172 @@ sources: ## What This Library Does -VectorKit turns a piece of text into a vector — a fixed-size code that -stands in for the text's meaning — and stores that vector so a later query -can find the most similar ones. MOOTx01 is an on-device AI memory system. -It stores what an AI observes over time and helps the AI recall it later. -VectorKit is the part of MOOTx01 that answers "what stored memories are -most like this one?" - -VectorKit stores two kinds of vector for the same piece of text. The first -is a 256-bit binary fingerprint called an `Engram`, defined by a sibling -library, EngramLib. A fingerprint is a short fixed-size code computed from -a piece of content; similar content produces similar fingerprints, so the -system compares things quickly without reading them in full. Two Engrams -are compared by Hamming distance — the number of bit positions where they -differ; smaller means more similar. The second kind is a dense float -vector: a list of several hundred decimal numbers produced directly by an -embedding model such as MiniLM. VectorKit keeps both because they serve -different needs, explained below. +VectorKit turns a piece of text into a vector. A vector is a fixed-size +code that stands in for the text's meaning. VectorKit stores that vector. +A later query can then find the most similar ones. MOOTx01 is an +on-device AI memory system. It stores what an AI observes over time. It +helps the AI recall this later. VectorKit is the part of MOOTx01 that +answers one question: which stored memories are most like this one. + +VectorKit stores two kinds of vector for the same piece of text. The +first is a 256-bit binary fingerprint called an `Engram`. A sibling +library, EngramLib, defines this type. A fingerprint is a short +fixed-size code computed from a piece of content. Similar content +produces similar fingerprints. The system can then compare things +quickly without reading them in full. Two Engrams are compared by +Hamming distance. This is the count of bit positions where they differ. +A smaller count means the two are more similar. The second kind is a +dense float vector. This is a list of several hundred decimal numbers. +An embedding model, such as MiniLM, produces this list directly. +VectorKit keeps both kinds because they serve different needs. The next +section explains why. ## The Problem It Solves -An AI memory system needs to answer "find memories like this one" without -sending private text to a server. VectorKit runs entirely on the device -that captured the memory. - -A 256-bit fingerprint is compact and fast to compare — comparing two -fingerprints is counting differing bits, pure integer arithmetic. Because -the comparison never uses floating-point math, it produces the exact same -answer on every device and every operating system. VectorKit calls this -property "four-way" determinism, and it never computes a fingerprint -comparison itself: every comparison is delegated to EngramLib, which in -turn delegates to a shared, conformance-gated kernel (a conformance -fixture is a recorded input/output pair both an original and a ported -implementation must reproduce exactly; EngramLib's kernel is checked this -way across four build configurations). This is spec I-7 in VectorKit's own -design documents: the kit performs no Hamming math of its own. - -A fingerprint is compact, but compacting hundreds of numbers into 256 bits -throws information away. Some queries need the finer detail the original -float numbers carry — for example, telling a passage from its own echoed -question apart, which the fingerprint's collapsed representation cannot -always do. For those queries VectorKit also stores the float vector and -compares it with cosine distance, a measure of the angle between two -vectors. Float math is reproducible on one platform and one build, but it -is not guaranteed to produce byte-identical results on a different -platform. VectorKit documents this openly as a boundary, not a defect: the -binary fingerprint lane is the four-way-identical lane, and the float lane -is the "reproducible within one configuration" lane. - -Every stored vector is tagged with the model identifier and model version -that produced it. Two different embedding models turn the same word into -different numbers for reasons that have nothing to do with meaning, so -comparing vectors from different models produces a meaningless answer. -VectorKit enforces this rule, called spec I-4, at multiple levels: storage -partitions vectors by model, and every search is scoped to one model. +An AI memory system must answer one question: find memories like this +one. It must do so without sending private text to a server. VectorKit +runs entirely on the device that captured the memory. + +A 256-bit fingerprint is compact and fast to compare. Comparing two +fingerprints means counting differing bits. This is pure integer +arithmetic. The comparison never uses floating-point math. So it +produces the exact same answer on every device and every operating +system. VectorKit calls this property four-way determinism. VectorKit +never computes a fingerprint comparison itself. Every comparison is +delegated to EngramLib. EngramLib in turn delegates to a shared, +conformance-gated kernel. A conformance fixture is a recorded +input-output pair. Both an original and a ported implementation must +reproduce it exactly. EngramLib's kernel is checked this way across four +build configurations. This rule is spec I-7 in VectorKit's own design +documents. The kit performs no Hamming math of its own. + +A fingerprint is compact, but this compacting has a cost. Packing +hundreds of numbers into 256 bits throws information away. Some queries +need the finer detail that the original float numbers carry. One +example is telling a passage apart from its own echoed question. The +fingerprint's collapsed form cannot always make that distinction. For +these queries, VectorKit also stores the float vector. It compares this +vector with cosine distance, a measure of the angle between two vectors. +Float math is reproducible on one platform and one build. It is not +guaranteed to produce byte-identical results on a different platform. +VectorKit documents this openly as a boundary, and not as a defect. The +binary fingerprint lane is the four-way-identical lane. The float lane +is the lane that stays reproducible within one configuration. + +Every stored vector carries the model identifier and model version that +produced it. Two different embedding models turn the same word into +different numbers. These differences have nothing to do with meaning. +So comparing vectors from different models gives a meaningless answer. +VectorKit enforces this rule at multiple levels. The rule is called spec +I-4. Storage partitions vectors by model. Every search is scoped to one +model. Finally, an on-device memory store must stay fast as it grows. Scanning -every stored vector for every query works fine at a few thousand records -but becomes slow at scale. VectorKit solves this by keeping an in-memory -copy of all fingerprints (so no per-query database read is needed) and by -switching, once the count of live fingerprints crosses a threshold, from a -full scan to a sub-linear search structure that returns the identical -answer faster. +every stored vector for every query works fine at a few thousand +records. It becomes slow at a larger scale. VectorKit solves this two +ways. It keeps an in-memory copy of all fingerprints, so no per-query +database read is needed. It also switches search strategy once the +count of live fingerprints crosses a threshold. Below the threshold it +does a full scan. At or above it, VectorKit switches to a sub-linear +search structure. This structure returns the identical answer faster. ## How It Works -Writing a vector has two parts. `VectorStore.addPayload` first writes the -vector as one row to a `vectors` table, which is the durable source of -truth — nothing is ever considered stored until this write succeeds. It -then mirrors a binary vector into an in-memory packed array (or, for a -float vector, into a per-model in-memory array) so that later searches -never have to re-read the database. The in-memory array can optionally be -backed by an on-disk cache file, called a sidecar, so that reopening the -store does not require rebuilding the array from every database row. - -Searching has two independent paths, one per vector kind. A binary search -(`findNearest`) compares the query fingerprint against the in-memory array -using Hamming distance. Below a configurable threshold of live fingerprints -(50,000 by default), the search does a full linear scan — always fast -enough at that scale. At or above the threshold, VectorKit switches to a -technique called Multi-Index Hashing, which slices each 256-bit fingerprint -into several shorter bands and uses per-band lookup tables to rule out most -of the collection without touching it. Multi-Index Hashing is provably -exact: it returns precisely the same neighbors the full scan would have -returned, just faster. A test suite checks this by running both searches -on the same random and adversarial inputs and requiring identical output. - -A float search (`findNearestFloat`) compares the query's float vector -against a separate in-memory array holding only that model's float -vectors, using cosine, Euclidean, or dot-product distance. Because -different models produce vectors of different length, VectorKit keeps one -float array per model rather than mixing them. - -A third comparison method, `MaxSimScorer`, serves models that produce many -small vectors per item instead of one — for example, one fingerprint per -word, an approach known as ColBERT-style late interaction. Rather than -compare one vector to one vector, it compares every query-word fingerprint -against every document-word fingerprint and keeps, for each query word, -the best match found in the document; the document's overall score is the -sum of those best matches. This scorer is exhaustive: it never skips a -candidate document, which makes it the correctness reference for any +Writing a vector has two parts. `VectorStore.addPayload` first writes +the vector as one row to a `vectors` table. This table is the durable +source of truth. Nothing counts as stored until this write succeeds. +`addPayload` then mirrors the vector into an in-memory packed array. A +binary vector goes into one shared array. A float vector goes into a +per-model array instead. This mirroring means a later search never has +to re-read the database. The in-memory array can optionally be backed +by an on-disk cache file, called a sidecar. This lets the store reopen +without rebuilding the array from every database row. + +Searching has two independent paths, one per vector kind. A binary +search, `findNearest`, compares the query fingerprint against the +in-memory array using Hamming distance. Below a configurable threshold +of live fingerprints, the search does a full linear scan. The default +threshold is fifty thousand fingerprints. A full scan stays fast enough +below that scale. At or above the threshold, VectorKit switches to a +technique called Multi-Index Hashing. This technique slices each 256-bit +fingerprint into several shorter bands. It uses per-band lookup tables +to rule out most of the collection without touching it. Multi-Index +Hashing is provably exact. It returns precisely the same neighbors the +full scan would have returned, only faster. A test suite checks this by +running both searches on the same random and adversarial inputs. Both +searches must produce identical output. + +A float search, `findNearestFloat`, compares the query's float vector +against a separate in-memory array. That array holds only one model's +float vectors. The comparison uses cosine, Euclidean, or dot-product +distance. Different models produce vectors of different length. +Because of this, VectorKit keeps one float array per model rather than +mixing them. + +A third comparison method, `MaxSimScorer`, serves a different kind of +model. Some models produce many small vectors per item instead of one. +One example is a fingerprint per word, an approach known as ColBERT-style +late interaction. Rather than compare one vector to one vector, +`MaxSimScorer` compares every query-word fingerprint against every +document-word fingerprint. For each query word, it keeps the best match +found in the document. The document's overall score is the sum of those +best matches. This scorer is exhaustive. It never skips a candidate +document. That property makes it the correctness reference for any faster method built later. ## How the Pieces Fit -Figure 1 shows the library's topology — its major parts and how data moves -between them. +Figure 1 shows the library's topology. It shows the major parts and how +data moves between them. ![Figure 1. Topology of VectorKit](topology.svg) -*Figure 1. Topology of VectorKit. Text enters through an `EmbeddingProvider` -and becomes an Engram and, optionally, a float vector. `VectorStore` writes -both to the durable `vectors` table and mirrors them into in-memory -resident arrays. Reads dispatch through the `DenseIndex` seam to one of -three interchangeable search engines. The dashed regions mark the durable -storage boundary and the optional on-disk cache.* - -`EmbeddingProvider` is the seam a host application implements: it supplies -whatever inference technique turns text into numbers (a CoreML model, for -instance). VectorKit's own concrete implementation, -`FloatSimHashEmbeddingProvider`, takes those numbers and projects them into -an Engram fingerprint using a shared substrate primitive, `FloatSimHash`, -so that every provider in the MOOTx01 kit graph produces fingerprints the -same deterministic way. +*Figure 1. Topology of VectorKit. Text enters through an +`EmbeddingProvider`. It becomes an Engram and, optionally, a float +vector. `VectorStore` writes both to the durable `vectors` table. It +also mirrors them into in-memory resident arrays. Reads dispatch through +the `DenseIndex` seam to one of three interchangeable search engines. +The dashed regions mark the durable storage boundary and the optional +on-disk cache.* + +`EmbeddingProvider` is the seam a host application implements. It +supplies whatever inference technique turns text into numbers, for +example a CoreML model. VectorKit ships one concrete implementation of +its own, `FloatSimHashEmbeddingProvider`. This type takes those numbers +and projects them into an Engram fingerprint. It uses a shared substrate +primitive, `FloatSimHash`, to do this. Every provider in the MOOTx01 kit +graph then produces fingerprints the same deterministic way. `VectorStore` is the actor every caller talks to. It owns the durable -`vectors` table (through a PersistenceKit `Storage` backend such as -SQLite) and the resident in-memory arrays. Three foundation types flow -through every layer beneath it: `VectorRecordKey` (which record this is), -`VectorPayload` (the raw typed bytes of one vector), and `DenseHit` (one -scored search result). All three are shared, additive-only types — no -search engine defines its own private version of them. - -Underneath `VectorStore` sits the `DenseIndex` protocol, a pluggable engine -seam. Three concrete engines implement it: `BruteForceIndex` (the always- -correct binary linear scan and the conformance reference), `MIHIndex` (the -sub-linear binary search gated against `BruteForceIndex`), and -`FloatBruteForceIndex` (the float-lane linear scan, built once per model). -`VectorStore` decides which binary engine is active by comparing the live -fingerprint count against `mihThreshold`, and swaps between them without -rebuilding either — both are always kept in sync with every write. - -`ResidentArrayStore` manages the optional `.vec` sidecar file: a packed, -fixed-format binary cache of the in-memory array. It is a regenerable -cache, never a second source of truth — if it is missing, stale, or -corrupted, VectorStore rebuilds it from the `vectors` table, the only -durable source. +`vectors` table, through a PersistenceKit `Storage` backend such as +SQLite. It also owns the resident in-memory arrays. Three foundation +types flow through every layer beneath it. `VectorRecordKey` names which +record this is. `VectorPayload` holds the raw typed bytes of one vector. +`DenseHit` is one scored search result. All three are shared, +additive-only types. No search engine defines its own private version of +them. + +Underneath `VectorStore` sits the `DenseIndex` protocol, a pluggable +engine seam. Three concrete engines implement it. `BruteForceIndex` is +the always-correct binary linear scan and the conformance reference. +`MIHIndex` is the sub-linear binary search, gated against +`BruteForceIndex`. `FloatBruteForceIndex` is the float-lane linear scan, +built once per model. `VectorStore` decides which binary engine is +active. It compares the live fingerprint count against `mihThreshold`. +It swaps between the two engines without rebuilding either one, because +both are always kept in sync with every write. + +`ResidentArrayStore` manages the optional `.vec` sidecar file. This file +is a packed, fixed-format binary cache of the in-memory array. It is a +regenerable cache, and never a second source of truth. The sidecar might +be missing, stale, or corrupted. In that case, `VectorStore` rebuilds +the array from the `vectors` table. That table is the only durable +source. ## What Ships in the Package -The package ships the Swift sources listed above, a mirrored Rust -implementation in `rust/`, and no bundled data artifacts — VectorKit has -no fixed reference tables of its own; everything it stores comes from -caller-supplied text and caller-supplied embedding models. +The package ships the Swift sources listed above. It also ships a +mirrored Rust implementation in `rust/`. It ships no bundled data +artifacts. VectorKit has no fixed reference tables of its own. +Everything it stores comes from caller-supplied text and caller-supplied +embedding models.