From 9a91d8ae37b7079789d573fa5e8769df194d2760 Mon Sep 17 00:00:00 2001 From: Fable Date: Sat, 4 Jul 2026 16:06:52 -0500 Subject: [PATCH 1/2] =?UTF-8?q?docs(sdk):=20package=20docs=20fan-out=20?= =?UTF-8?q?=E2=80=94=20OVERVIEW,=20DETAILS,=20AGENT=5FMAP,=20topology=20pe?= =?UTF-8?q?r=20package?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3-4 of the Moot-SDK documentation initiative. Voice: Turabian at 8th grade per the LatticeLib pilot exemplar. Front-matter pins per-repo authored_commit and per-source git blob hashes for docs-scrub. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Pzkqg3bpbbhfuWRsYomJ3o --- packages/libs/EngramLib/docs/AGENT_MAP.md | 62 + packages/libs/EngramLib/docs/DETAILS.md | 180 +++ .../libs/EngramLib/docs/INTERFACE_DOCTRINE.md | 79 -- packages/libs/EngramLib/docs/OVERVIEW.md | 121 ++ .../libs/IntellectusLib/docs/AGENT_MAP.md | 85 ++ packages/libs/IntellectusLib/docs/DETAILS.md | 246 ++++ packages/libs/IntellectusLib/docs/OVERVIEW.md | 121 ++ .../libs/IntellectusLib/docs/topology.svg | 91 ++ .../libs/SubstrateKernel/docs/AGENT_MAP.md | 145 +++ packages/libs/SubstrateKernel/docs/DETAILS.md | 344 +++++ .../libs/SubstrateKernel/docs/OVERVIEW.md | 170 +++ .../libs/SubstrateKernel/docs/topology.svg | 127 ++ packages/libs/SubstrateLib/docs/AGENT_MAP.md | 142 +++ packages/libs/SubstrateLib/docs/DETAILS.md | 417 ++++++ packages/libs/SubstrateLib/docs/OVERVIEW.md | 181 +++ packages/libs/SubstrateLib/docs/topology.svg | 99 ++ packages/libs/SubstrateML/docs/AGENT_MAP.md | 179 +++ packages/libs/SubstrateML/docs/DETAILS.md | 1122 +++++++++++++++++ packages/libs/SubstrateML/docs/OVERVIEW.md | 222 ++++ packages/libs/SubstrateML/docs/topology.svg | 162 +++ .../libs/SubstrateTypes/docs/AGENT_MAP.md | 292 +++++ packages/libs/SubstrateTypes/docs/DETAILS.md | 821 ++++++++++++ packages/libs/SubstrateTypes/docs/OVERVIEW.md | 214 ++++ .../libs/SubstrateTypes/docs/topology.svg | 160 +++ 24 files changed, 5703 insertions(+), 79 deletions(-) create mode 100644 packages/libs/EngramLib/docs/AGENT_MAP.md create mode 100644 packages/libs/EngramLib/docs/DETAILS.md delete mode 100644 packages/libs/EngramLib/docs/INTERFACE_DOCTRINE.md create mode 100644 packages/libs/EngramLib/docs/OVERVIEW.md create mode 100644 packages/libs/IntellectusLib/docs/AGENT_MAP.md create mode 100644 packages/libs/IntellectusLib/docs/DETAILS.md create mode 100644 packages/libs/IntellectusLib/docs/OVERVIEW.md create mode 100644 packages/libs/IntellectusLib/docs/topology.svg create mode 100644 packages/libs/SubstrateKernel/docs/AGENT_MAP.md create mode 100644 packages/libs/SubstrateKernel/docs/DETAILS.md create mode 100644 packages/libs/SubstrateKernel/docs/OVERVIEW.md create mode 100644 packages/libs/SubstrateKernel/docs/topology.svg create mode 100644 packages/libs/SubstrateLib/docs/AGENT_MAP.md create mode 100644 packages/libs/SubstrateLib/docs/DETAILS.md create mode 100644 packages/libs/SubstrateLib/docs/OVERVIEW.md create mode 100644 packages/libs/SubstrateLib/docs/topology.svg create mode 100644 packages/libs/SubstrateML/docs/AGENT_MAP.md create mode 100644 packages/libs/SubstrateML/docs/DETAILS.md create mode 100644 packages/libs/SubstrateML/docs/OVERVIEW.md create mode 100644 packages/libs/SubstrateML/docs/topology.svg create mode 100644 packages/libs/SubstrateTypes/docs/AGENT_MAP.md create mode 100644 packages/libs/SubstrateTypes/docs/DETAILS.md create mode 100644 packages/libs/SubstrateTypes/docs/OVERVIEW.md create mode 100644 packages/libs/SubstrateTypes/docs/topology.svg diff --git a/packages/libs/EngramLib/docs/AGENT_MAP.md b/packages/libs/EngramLib/docs/AGENT_MAP.md new file mode 100644 index 0000000..028777d --- /dev/null +++ b/packages/libs/EngramLib/docs/AGENT_MAP.md @@ -0,0 +1,62 @@ +--- +doc: AGENT_MAP +package: EngramLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/EngramLib/EngramLib.swift + blob: b7f236bfaa193967daf5c5d6061ef4591207ed22 + - path: Sources/EngramLib/Match.swift + blob: 5c61fd65888a795c703ab9f52022dffeae9c7595 +--- + +# AGENT_MAP — EngramLib + +PURPOSE: product-facing similarity/retrieval/aggregation API over 256-bit engrams (`Fingerprint256`). Thin wrapper that hides substrate kernel selection: distance, batch-distance, top-K nearest, radius filter, OR-union. Two files total; no internal multi-stage pipeline. + +DEPS: imports SubstrateTypes (`Fingerprint256`, aliased `Engram`), SubstrateKernel (`SubstrateKernel` protocol, `PortableKernel.kernelForCurrentPlatform()`). Imported by (within SDK): VectorKit, CorpusKit (kits, moot-memory repo — kits compose libs, never the reverse). Rust port in `rust/` (`engram_lib` crate) mirrors the full API 1:1 against `substrate-types`/`substrate-kernel` Rust crates; conformance is behavioral parity checked by mirrored test suites (`rust/tests/engram_lib_tests.rs` vs `Tests/EngramLibTests/`), not a shared fixture file. + +ENTRY POINTS (most callers need only these): +- EngramLib.swift:77 `EngramLib.distance(_ a: Engram, _ b: Engram) -> Int` — Hamming distance, 0...256 +- EngramLib.swift:99 `EngramLib.findNearest(probe:in:k:) -> [Match]` — top-K nearest, distance-asc / index-asc tiebreak +- EngramLib.swift:216 `EngramLib.session() -> Session` — reusable kernel handle for hot loops + +## Symbol Table + +### Engram type — EngramLib.swift +- :43 `public typealias Engram = Fingerprint256` — substrate type, product-facing name; do not add a wrapping struct +- :45-53 `extension Engram { init(blocks b0:_ b1:_ b2:_ b3:) }` — convenience 4-block constructor +- :33 `internal let _engramLibCachedKernel: any SubstrateKernel` — resolved ONCE at module load via `PortableKernel.kernelForCurrentPlatform()`; every call path reads this, never re-resolves + +### Static API — enum EngramLib (EngramLib.swift:71) +- :77 `distance(_:_:) -> Int` — single-pair Hamming distance +- :84 `distances(probe:candidates:) -> [Int]` — batched, same order/count as `candidates`; `[]` on empty input +- :99 `findNearest(probe:in:k:) -> [Match]` — `[]` if `k <= 0` or candidates empty; clamps to `min(k, candidates.count)` +- :112 `findNearest(probe:in:) -> Match?` — k=1 convenience; `nil` on empty candidates +- :124 `findWithin(probe:in:maxDistance:) -> [Match]` — `[]` if candidates empty OR `maxDistance < 0`; inclusive upper bound +- :149 `union(_ engrams: [Engram]) -> Engram` — bitwise OR-reduce; zero engram is the empty-input identity +- :154 `union(_ a:_ b:) -> Engram` — pairwise OR convenience, delegates to `Fingerprint256.union(_:)` +- :222 `private static func kernel() -> any SubstrateKernel` — single choke point back to the cached kernel + +### Session — struct EngramLib.Session (EngramLib.swift:166), Sendable +- :169 `init()` — captures `_engramLibCachedKernel` +- :173/:177/:184/:193/:210 `distance` / `distances` / `findNearest` / `findWithin` / `union` — instance mirrors of the static functions above, byte-identical results, no new behavior +- :216 `EngramLib.session() -> Session` — factory, equivalent to `Session()` + +### Result type — Match.swift +- :14 `public struct Match: Hashable, Sendable, Codable` — `index: Int` (position in caller's candidate array), `distance: Int` (Hamming distance from probe) +- :18 `init(index:distance:)` — public memberwise init +- :24-28 `extension Match: Comparable` — `<` compares `distance` first, `index` second; this IS the sort/tiebreak contract used by `findNearest`/`findWithin` + +## INVARIANTS / GOTCHAS + +- Kernel is resolved exactly once per process (`_engramLibCachedKernel`, EngramLib.swift:33), at module load, not per call. Do not reintroduce per-call kernel resolution — it was removed as a measured, needless dispatch cost (see file header comment, Phase 4.3 / decision 2026-05-28 §6.4.3). +- DO NOT REIMPLEMENT SUBSTRATE MATH (EngramLib.swift:14-24 banner comment). Every primitive this library exposes — Hamming distance, batch distance, top-K, OR-reduce — already exists in SubstrateTypes/SubstrateKernel/SubstrateML; EngramLib only forwards to it. +- Tie-break rule is fixed and load-bearing: distance ascending, then candidate index ascending (`Match.<`, Match.swift:25-27). `findNearest` and `findWithin` both depend on this for deterministic, reproducible output — same probe + same candidate list + same order = same result, always. +- All empty/degenerate-input guards return empty collections or `nil`, never throw and never crash: empty candidates → `[]`/`nil`; `k <= 0` → `[]`; `maxDistance < 0` → `[]`; empty `union` input → `Engram.zero` (the OR identity). +- `Session` and the static functions are result-identical by construction — both bottom out in the same cached kernel. `Session` exists only to save the small per-call cost of re-touching the static cache in a hot loop; it is not a different code path. +- `Engram` is a type alias, not a wrapper type. Treat the underlying representation (`Fingerprint256`, four `UInt64` blocks) as substrate-owned; construct via `Engram(blocks:_:_:_:)` or substrate constructors, not by touching blocks directly from product code. +- Whole library is stateless and thread-safe (file header, EngramLib.swift:7-10): every static method creates no persistent state; `Session` is `Sendable` and safe to share across tasks. +- Rust and Swift legs must stay behaviorally identical: same guard conditions, same tie-break order, same identity values. There is no shared byte-fixture gate here (contrast LatticeLib's conformance JSON) — parity is enforced by keeping `rust/tests/engram_lib_tests.rs` and `Tests/EngramLibTests/` mirrored. Changing a guard or ordering rule in one leg without the other is a silent cross-platform divergence. +- Package has no pinned artifacts, no build-time-only code, and no privacy seams — unlike larger sibling libs (e.g. LatticeLib), there is nothing here gated by an environment variable or a resource bundle. diff --git a/packages/libs/EngramLib/docs/DETAILS.md b/packages/libs/EngramLib/docs/DETAILS.md new file mode 100644 index 0000000..6617720 --- /dev/null +++ b/packages/libs/EngramLib/docs/DETAILS.md @@ -0,0 +1,180 @@ +--- +doc: DETAILS +package: EngramLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/EngramLib/EngramLib.swift + blob: b7f236bfaa193967daf5c5d6061ef4591207ed22 + - path: Sources/EngramLib/Match.swift + blob: 5c61fd65888a795c703ab9f52022dffeae9c7595 +--- + +# EngramLib Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. The package has two files: the +public API surface and the result type it returns. + +## EngramLib.swift + +This file provides the entire public surface of the library: the `Engram` +type alias, the `EngramLib` enum with its static comparison and retrieval +functions, and the `Session` type for callers that run many comparisons in +a loop. + +### How It Works + +The file opens with a type alias, `public typealias Engram = Fingerprint256`. +`Fingerprint256` is a 256-bit code defined in the substrate's +`SubstrateTypes` package: four 64-bit blocks that together form one +fixed-size fingerprint. EngramLib does not define its own bit +representation. It reuses the substrate's, under a product-facing name. +This indirection matters for one reason: if a future version of the +substrate widens the representation, product code that only ever wrote +`Engram` never has to change. An extension adds `Engram.init(blocks:_:_:_:)`, +a convenience initializer that takes the four blocks as separate arguments, +so callers can write `Engram(blocks: 0xDEAD, 0xBEEF, 0xCAFE, 0xBABE)` +instead of naming the underlying block parameters. + +Every comparison in the file ultimately calls a kernel: an object, supplied +by the substrate's `SubstrateKernel` package, that knows how to run bit +operations efficiently on the current device. The file resolves this +kernel exactly once, at module load time, into the private constant +`_engramLibCachedKernel`. Every static function then reads that same +cached value instead of asking the substrate to choose again. The file's +own comment explains why: the kernel choice does not change while the +program runs, so resolving it on every call was pure overhead, paid on +every single similarity check. + +### Function Coverage + +`EngramLib.distance(_:_:)` returns the Hamming distance between two +engrams. Hamming distance counts the bit positions where two equal-length +codes differ. Identical engrams return 0; engrams that are exact bit +inverses of each other return 256, the width of the fingerprint. This +function matters because it is the single building block every other +comparison in the file is built from. + +`EngramLib.distances(probe:candidates:)` computes the distance from one +probe engram to every member of a candidate list, returning an array in +the same order as the input. It matters because computing distances one +at a time in a loop would call into the kernel once per candidate; this +function hands the whole batch to the kernel in one call, so the kernel +can process it more efficiently. It returns an empty array immediately +when the candidate list is empty, avoiding any wasted work. + +`EngramLib.findNearest(probe:in:k:)` returns the `k` candidates closest to +the probe, sorted by distance from nearest to farthest, with ties broken +by the candidate's position in the input list. This is the library's core +retrieval function: it answers "which memories are most like this one," +bounded to a fixed count. It returns an empty array when `k` is zero or +negative, or when the candidate list is empty, rather than treating either +case as an error. When `k` is larger than the candidate count, it returns +every candidate rather than failing. + +`EngramLib.findNearest(probe:in:)`, the two-argument overload, is a +convenience for the common case of wanting only the single closest match. +It calls the three-argument version with `k` fixed to 1 and returns the +first result, or `nil` when the candidate list is empty. It matters +because "find the closest one" is common enough to deserve its own name, +rather than making every caller unwrap a one-element array. + +`EngramLib.findWithin(probe:in:maxDistance:)` returns every candidate +whose distance from the probe is at most `maxDistance`, sorted the same +way as `findNearest`. It matters because some callers do not know in +advance how many results they want; they want everything within a +similarity radius, however many candidates that turns out to include. It +computes the full distance array first, filters it, and sorts only the +matches that passed the filter, so the sort never touches a candidate it +is about to discard. It also guards against a negative `maxDistance`, +which cannot correspond to any real distance and would otherwise silently +match nothing in a confusing way. + +`EngramLib.union(_:)` performs a bitwise OR across a list of engrams: the +result has a 1-bit at every position where at least one input engram had +one. It matters because a memory system often wants one summary engram +for a whole group — a cohort of related memories, or the union of a +topic's structural features — rather than comparing every group member +individually. An empty input returns the zero engram, which is the +correct identity value for OR: combining it with anything leaves that +thing unchanged. + +`EngramLib.union(_:_:)`, the two-argument overload, ORs exactly two +engrams together. It exists as a lightweight convenience for the common +pairwise case, so a caller merging two engrams does not have to build a +two-element array just to call the list version. + +`EngramLib.Session` is a struct that holds one kernel reference and +exposes `distance`, `distances`, `findNearest`, `findWithin`, and `union` +as instance methods with the same behavior as their static counterparts. +It matters for one reason: reuse. The static functions already share one +cached kernel internally, so a `Session` does not unlock any new +capability; it exists so that call sites processing thousands of +comparisons in a tight loop can hold one reference and avoid the small, +repeated cost of reaching back into the static cache on every call. A +session is `Sendable`, so the same session can be shared safely across +concurrent tasks. `EngramLib.session()` is a static factory that returns a +new `Session`; it is equivalent to calling `Session()` directly and exists +purely so callers do not need to know the type's name to construct one. + +The private function `kernel()` is the single choke point every static +function in the enum calls through to reach the cached kernel. Routing +every call through one small function, rather than reading the module +constant directly everywhere, keeps the indirection in one place should +the caching strategy ever need to change. + +## Match.swift + +This file provides `Match`, the value every retrieval function in +`EngramLib.swift` returns. + +### How It Works + +A `Match` is a pair of two integers: `index`, the position of the matched +candidate in the caller's original input array, and `distance`, the +Hamming distance from the probe to that candidate. The type carries no +reference to the engram itself, only its position and its distance, +because the caller already has the original candidate array and can look +the engram up by index if needed. Keeping the type this small makes it +cheap to create in bulk during a retrieval scan. + +`Match` conforms to `Hashable`, `Sendable`, and `Codable`, so it can be +stored in sets, passed safely across concurrent code, and serialized for +persistence or logging without any extra work. + +### Function Coverage + +`Match.init(index:distance:)` is the memberwise initializer. It is public +so that code outside the package — test suites, or a caller assembling +results from its own logic — can construct a `Match` directly rather than +only receiving one from an `EngramLib` function. + +The `Comparable` conformance, added in a small extension, defines the +ordering that every retrieval function in `EngramLib.swift` relies on: +distance ascending first, and for two matches at equal distance, index +ascending second. This ordering is what makes `findNearest` and +`findWithin` deterministic. Given the same probe and the same candidate +list, the result is always sorted the same way, because the tie-break +rule never depends on anything but the candidate's fixed position in the +input. + +## Rust Port and Conformance + +The `rust/` directory contains a second implementation of the same API, +in two files: `lib.rs` defines the `EngramLib` struct with its associated +functions (`distance`, `find_nearest`, `find_within`, `union`, and the +rest) plus the `Session` type, and `matchx.rs` defines the `Match` struct +with its `Ord` implementation. The function names differ only in the ways +that Swift and Rust naming conventions differ — `findNearest` in Swift is +`find_nearest` in Rust, `findNearest(probe:in:)` becomes +`find_nearest_one` because Rust has no argument-label overloading — but +the behavior, including every empty-input guard and every tie-break rule, +matches the Swift implementation function for function. Both legs depend +on their language's version of `SubstrateTypes` and `SubstrateKernel` and +route their bit math through the same kernel abstraction, so a Swift +caller and a Rust caller comparing the same two engrams compute the same +distance. `rust/tests/engram_lib_tests.rs` mirrors the Swift test suite +in `Tests/EngramLibTests/`; when you change the behavior of one leg, +update the other and its tests together. diff --git a/packages/libs/EngramLib/docs/INTERFACE_DOCTRINE.md b/packages/libs/EngramLib/docs/INTERFACE_DOCTRINE.md deleted file mode 100644 index d666f58..0000000 --- a/packages/libs/EngramLib/docs/INTERFACE_DOCTRINE.md +++ /dev/null @@ -1,79 +0,0 @@ -# EngramLib Interface Doctrine - -For coding agents using EngramLib in product code or downstream kits. - -## 1. Always go through the public API - -Call `EngramLib.distance`, `EngramLib.findNearest`, `EngramLib.findWithin`, `EngramLib.union`, or `EngramLib.session()`. Never import SubstrateLib directly from product code just to call a kernel method. The dispatcher and kernel selection are internal; consumers do not see them. - -```swift -// CORRECT -let matches = EngramLib.findNearest(probe: query, in: estate, k: 10) - -// WRONG -import SubstrateLib -let raw = PortableKernel.kernelForCurrentPlatform().hammingTopK(...) -``` - -If you find yourself reaching for SubstrateLib directly from a non-substrate kit, file a decision record proposing a new EngramLib surface for that need. - -## 2. Treat Engram as opaque - -The `Engram` typealias maps to `SubstrateLib.Fingerprint256` today. Future substrate versions may use a different representation (wider fingerprint, different block layout). Product code that constructs engrams via `Engram(blocks: b0, b1, b2, b3)` and treats them as opaque survives the change; code that reaches into `.block0`, `.block1` directly does not. - -```swift -// CORRECT -let e = Engram(blocks: 0xDEAD, 0xBEEF, 0xCAFE, 0xBABE) -let d = EngramLib.distance(e1, e2) - -// FRAGILE: depends on the underlying representation -let e: Engram = something -let firstBlock = e.block0 -``` - -If you genuinely need the block layout (federation handshakes, on-wire encoding), do that work inside a substrate-aware kit, not in product code. - -## 3. Use Session for hot loops - -The stateless static methods (`EngramLib.distance`, `EngramLib.findNearest`, etc.) construct a kernel per call. For hundreds or thousands of operations in a hot path, prefer `EngramLib.session()`: the session holds one kernel for reuse. - -```swift -let session = EngramLib.session() -for q in queries { - let matches = session.findNearest(probe: q, in: estate, k: 10) - process(matches) -} -``` - -`EngramLib.Session` is `Sendable` and safe to share across tasks. - -## 4. Match ordering is stable - -`findNearest` and `findWithin` return matches ordered by distance ascending, with ties broken by candidate index ascending. This ordering is deterministic across kernel implementations (NEON, BNNS, Metal, SIMD, scalar). Tests can assume it; product code can rely on it for UI rendering. - -## 5. Distance semantics are Hamming - -All distance methods return Hamming distance, 0...256. Zero means identical engrams; 256 means bit-inverse. Cosine, Euclidean, Jaccard, and Hyperplane-family-aware distances live in VectorKit (mission 6) and federation-aware code (SubstrateLib + ConvergenceKit-Federation). EngramLib's stable contract is Hamming. - -## 6. Union semantics are bitwise OR - -`EngramLib.union(_ engrams:)` returns the bitwise OR of all inputs. The result has a 1-bit at every position where at least one input had a 1-bit. Empty input returns the zero engram. This is the substrate's standard cohort/union operator (paper section 5.3); the same operation backs federation OR-reductions. - -## 7. Empty inputs return empty outputs - -`findNearest(probe:in:[], k:)` returns `[]`. `findWithin(probe:in:[], maxDistance:)` returns `[]`. `distances(probe:candidates: [])` returns `[]`. `union([])` returns `Engram.zero`. No `nil`, no throw, no error. Product code can call these with empty collections safely. - -## 8. When you need something EngramLib does not expose - -File a decision record. Examples of valid additions: - -- A different distance metric (cosine over engrams, hyperplane-family-aware) -- Approximate nearest neighbor (LSH index, HNSW over engrams) -- Streaming top-K over a sequence rather than an array -- Bit-pattern queries (find engrams with all bits in a mask set) - -Examples of invalid additions: - -- Exposing kernel selection (`EngramLib.useMetalKernel`): defeats the dispatcher -- Returning raw substrate types (`EngramLib.rawKernel()`): leaks abstraction -- Mutating-shape methods (engrams are values; product code never mutates them) diff --git a/packages/libs/EngramLib/docs/OVERVIEW.md b/packages/libs/EngramLib/docs/OVERVIEW.md new file mode 100644 index 0000000..933c956 --- /dev/null +++ b/packages/libs/EngramLib/docs/OVERVIEW.md @@ -0,0 +1,121 @@ +--- +doc: OVERVIEW +package: EngramLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/EngramLib/EngramLib.swift + blob: b7f236bfaa193967daf5c5d6061ef4591207ed22 + - path: Sources/EngramLib/Match.swift + blob: 5c61fd65888a795c703ab9f52022dffeae9c7595 +--- + +# EngramLib Overview + +## What This Library Does + +EngramLib compares engrams. An engram is a 256-bit fingerprint: a short, +fixed-size code computed from a piece of content. Similar content produces +similar fingerprints, so comparing two engrams is fast even though the +content behind them may be long. EngramLib answers three kinds of question +about a set of engrams: how far apart are two of them, which ones are +nearest to a given engram, and how do many of them combine into one summary +engram. + +MOOTx01 is an on-device AI memory system. It stores what an AI observes over +time and helps the AI recall it later. Every stored memory can carry an +engram that summarizes its structure. When the system needs to find +memories that resemble a new one, it does not compare the memories +directly. It compares their engrams instead, using the distance and +nearest-neighbor functions this library provides. + +## The Problem It Solves + +Comparing bit patterns sounds simple, but doing it correctly and quickly +across an entire memory store is not. The building blocks already exist in +a lower-level package called the substrate: routines for measuring the +distance between two 256-bit codes, for scanning many codes to find the +closest ones, and for combining codes with a bitwise OR. The substrate also +picks the fastest way to run these routines on the current device, choosing +among a plain reference implementation and hardware-accelerated versions +for different chip families. + +Product code should not have to know about any of that. A consumer that +wants to find the ten memories most similar to a probe engram should not +have to pick a chip-specific routine, manage a kernel object, or reason +about which hardware path is active. EngramLib is a thin, stable layer over +the substrate that hides those choices. It exposes a small set of static +functions with plain names — `distance`, `findNearest`, `findWithin`, +`union` — and it always picks the best available implementation for the +device it runs on. + +## How It Works + +Every public function in EngramLib forwards its work to a kernel, an +object that knows how to run the underlying bit operations on the current +device. EngramLib selects a kernel once, when the module first loads, and +reuses it for every call. Consumers never see the kernel or choose one +themselves. + +`EngramLib.distance(_:_:)` reports the Hamming distance between two +engrams. Hamming distance is the number of bit positions where two +equal-length codes differ; a smaller number means the two engrams — and +likely the content they summarize — are more alike. `EngramLib.distances` +computes that same distance from one probe engram to many candidates in a +single call, which is faster than calling `distance` in a loop because the +kernel can batch the work. + +`EngramLib.findNearest` and `EngramLib.findWithin` build on that +distance measurement to answer retrieval questions. `findNearest` returns +the closest matches to a probe, ranked by distance, for use when a caller +wants a fixed number of results — the ten memories most like this one, for +example. `findWithin` instead returns every candidate inside a distance +radius, for use when a caller wants everything similar enough, however many +that turns out to be. Both break ties the same way: when two candidates +sit at equal distance, the one that appeared earlier in the input list +comes first. This tie-break rule matters because it makes the result +reproducible. The same probe against the same candidate list always +produces the same ranking, on every run. + +`EngramLib.union` combines many engrams into one, bit by bit, keeping a +bit set in the result wherever any input had it set. This is useful for +building a single summary engram for a group of memories — a cohort, a +conversation, a topic cluster — that carries the structural features of +every member. + +Most calls create a fresh, momentary link to the shared kernel and are +done. For code that runs the same kind of comparison many thousands of +times in a loop, EngramLib also offers a `Session`: a small object that +holds the kernel once and exposes the same functions as instance methods. +A session produces identical results to the static functions; it exists +only to save the small cost of re-resolving the kernel on every call. + +## How the Pieces Fit + +The package is deliberately small: one file defines the public API, and one +file defines its result type. `EngramLib.swift` contains the `EngramLib` +enum with its static functions, the `Session` type for reuse in hot loops, +and the `Engram` type alias itself. `Match.swift` contains `Match`, the +small value returned by every retrieval function, holding the position of +a candidate in its input list and its distance from the probe. + +`Engram` is a public alias for `Fingerprint256`, a type owned by the +substrate's `SubstrateTypes` package. EngramLib does not define its own +fingerprint representation; it borrows the substrate's and gives it a +product-facing name, so that a future change to the underlying +representation does not force every consumer to rename their variables. +The kernel that does the actual bit math comes from `SubstrateKernel`, +the substrate's other half. Because the whole library is two short files +wrapping a single external kernel call, there is no internal topology +worth diagramming; every function's story is "validate the input, then +call the kernel." + +## What Ships in the Package + +The package ships the two Swift source files and their test suites, plus a +Rust port in `rust/` that mirrors the Swift API function for function: +`EngramLib::distance`, `EngramLib::find_nearest`, `EngramLib::union`, and +the rest. Both legs depend on their language's version of the substrate +packages and forward to the same kernel logic, so a Swift caller and a Rust +caller comparing the same two engrams get the same distance. diff --git a/packages/libs/IntellectusLib/docs/AGENT_MAP.md b/packages/libs/IntellectusLib/docs/AGENT_MAP.md new file mode 100644 index 0000000..6ee11a2 --- /dev/null +++ b/packages/libs/IntellectusLib/docs/AGENT_MAP.md @@ -0,0 +1,85 @@ +--- +doc: AGENT_MAP +package: IntellectusLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/IntellectusLib/Intellectus.swift + blob: 47d1dbe60d0ed880eaf657986ef9371f66423fda + - path: Sources/IntellectusLib/RecentWindowSink.swift + blob: 275f059bb1c0b5ebea68c7d32d2e6bb5ad5ed034 + - path: Sources/IntellectusLib/StatSample.swift + blob: 5a63d181680c35a1b79279f83651f8cb62dcd9b7 + - path: Sources/IntellectusLib/StatsSink.swift + blob: 60dc64a00370e491273c2328409a9d56fc46463b +--- + +# AGENT_MAP — IntellectusLib + +PURPOSE: substrate self-report telemetry faculty. Zero-dependency LEAF/floor library. Caller → `Intellectus.report(sample)` → lock-free enabled gate → (if on) installed `StatsSink.receive(_:)`. Off-path = one atomic load + branch, payload never built. + +DEPS: imports Foundation (NSLock; StatSample/RecentWindowSink/Intellectus), Synchronization (Atomic; Intellectus.swift only). Zero repo dependencies — this IS the floor of the dependency tree (SubstrateTypes/SubstrateKernel/SubstrateLib/SubstrateML are meant to depend on this in a later mission per Package.swift header; not yet wired as of this commit). Imported by: not enumerated in Package.swift/product graph as of this commit; source-comment call sites (forward references, not verified imports) — `AriaResident.installManagerTelemetry` / `AriaResident.runResidentDaemon` (StatsSink.swift, Intellectus.swift), `LocusKit.EstateVerbs`, `NeuronKit.EstateDreamingSink`, `ObserverSink.PersistenceStatsSink` (StatSample.swift). Rust port in `rust/` mirrors every file 1:1 (sample.rs/sink.rs/window.rs/holder.rs/global.rs/lib.rs); NO shared conformance fixtures — StatSample is opaque caller-supplied data, not a computed/derived value, so there is nothing for the two legs to agree on byte-for-byte. + +ENTRY POINTS (most callers need only these): +- Intellectus.swift:238 `Intellectus.report(_ make: @autoclosure () -> StatSample)` — the only call substrate hot-path code makes; off when disabled, no payload build +- Intellectus.swift:185 `Intellectus.install(sink: any StatsSink)` — host wires a real sink, once, at startup +- Intellectus.swift:205 `Intellectus.setEnabled(_ enabled: Bool)` — host/daemon flips the gate + +## Symbol Table + +### Sample type — StatSample.swift +- :32 `enum StatSample: Sendable` — the telemetry datum; value type, two cases +- :43 `case metric(name:value:tags:ts:)` — named Double measurement + String:String tags +- :70 `case event(kind:nounType:rowID:estate:ts:)` — substrate lifecycle transition; nounType is Int (caller casts NounType→Int to keep this crate dependency-free) +- :86 `enum EventKind: String, Sendable, Hashable, CaseIterable` — `.capture` (:87) | `.think` (:88); rawValue is the wire string +- :96 `StatSample.ts: Double` (extension) — timestamp regardless of case + +### Receiver contract — StatsSink.swift +- :36 `protocol StatsSink: Sendable` — one requirement +- :47 `receive(_ sample: StatSample)` — must be non-blocking/inexpensive; not self-enforced, caller (Intellectus) owns the enabled gate, sink does not re-check it +- :60 `struct NoOpSink: StatsSink` — default installed sink; discards, O(1), no allocation +- :63 `NoOpSink.shared` — shared singleton instance +- :69 `NoOpSink.receive(_:)` — intentional no-op + +### Bundled sink — RecentWindowSink.swift +- :63 `final class RecentWindowSink: StatsSink, @unchecked Sendable` — bounded ring buffer + optional forward decorator +- :69 `capacity: Int` — fixed at init, clamped to min 1 +- :72 `forward: (any StatsSink)?` (private) — decorator target; nil = window-only +- :76 `lock = NSLock()` (private) — guards ring/head/filled/_totalReceived only +- :80-86 `ring`/`head`/`filled`/`_totalReceived` (private) — backing storage; _totalReceived counts every receive() ever, ignoring eviction +- :97 `init(capacity:forward:)` — capacity clamped via `max(1, capacity)` +- :114 `receive(_ sample:)` — O(1) ring write under lock (oldest slot overwritten when full); forward call happens AFTER lock release +- :133 `snapshot() -> [StatSample]` — point-in-time copy, oldest-first; safe to iterate lock-free +- :148 `count: Int` — current retained count, 0...capacity +- :157 `totalReceived: Int` — monotonic lifetime receive count; 0 = "nothing has ever crossed the gate," distinct from count==0 + +### Global facade + holder — Intellectus.swift +- :44 `final class _IntellectusHolder: @unchecked Sendable` (internal, not public) — all mutable state lives here; `Intellectus` enum is a thin wrapper over the singleton +- :56 `_enabled: Atomic` (private) — lock-free gate, default false +- :62 `_sinkLock = NSLock()` (private) — guards `_sink` only; acquired only on the on-path (enabled) and during install() +- :67 `_sink: any StatsSink` (private, var) — starts as `NoOpSink.shared` +- :77 `install(sink:)` — replace sink under lock; rare/write-path call +- :91 `setEnabled(_:)` — `.releasing` store +- :100 `isEnabled: Bool` — `.acquiring` load +- :118 `report(_ make: @autoclosure () -> StatSample)` — OFF-path: single atomic load(.acquiring) + branch, autoclosure never forced. ON-path: snapshot sink under lock, release lock, THEN evaluate autoclosure, THEN `sink.receive(...)` — payload build and sink call both happen outside the lock +- :141 `let _intellectus = _IntellectusHolder()` — module-private global singleton; the only instance the public API touches +- :172 `enum Intellectus` — public, caseless namespace; all members static +- :185 `Intellectus.install(sink:)` — forwards to `_intellectus.install` +- :205 `Intellectus.setEnabled(_:)` — forwards to `_intellectus.setEnabled`; driven in production by `AriaResident.runResidentDaemon`'s poll loop reading the stats-store monitoring flag (per doc comment; that call site is outside this package) +- :210 `Intellectus.isEnabled: Bool` — forwards to `_intellectus.isEnabled` +- :238 `Intellectus.report(_:)` — public entry; re-declares `@autoclosure` so short-circuit holds end-to-end from call site to atomic check + +## INVARIANTS / GOTCHAS + +- OFF-PATH CONTRACT: when `Intellectus.isEnabled` is false, the `@autoclosure` argument to `report(_:)` is NEVER evaluated — no allocation, no lock, no sink call. This is enforced by ordering the `guard _enabled.load(...)` check BEFORE any reference to `make`. Do not restructure `report(_:)` in a way that forces the closure before the gate check. +- Atomic ordering pairing is load-bearing, not decorative: `setEnabled` stores with `.releasing`, `isEnabled`/the gate load with `.acquiring`. This is what guarantees a `setEnabled(true)` on one thread makes state written before it (e.g., a just-completed `install(sink:)`) visible to `report(_:)` on another thread. Do not change either ordering independently — they are a matched pair, mirrored by Rust's `Ordering::Release`/`Ordering::Acquire` in holder.rs. (Rust's own inline comment on `_enabled` load calls it `Relaxed`-sufficient in one place but the code actually uses `Ordering::Acquire`/`Ordering::Release` — treat the Swift acquire/release pairing as the authoritative contract.) +- Sink lock discipline: `_sinkLock` / `RecentWindowSink.lock` are held ONLY for the O(1) state mutation (snapshot/copy the sink reference, or write one ring slot). The actual `sink.receive(...)` / forward call ALWAYS runs after the lock is released. This is SPEC I-7 in the source comments. Breaking this (calling receive under the lock) risks deadlock if a sink's own `receive` re-enters this library or takes a lock a caller also holds. +- `RecentWindowSink` bound is contractual (SPEC I-8 in source comments): memory is O(capacity) regardless of how many samples are ever received. Do not add unbounded accumulation (e.g., a growing log) to this type. +- `RecentWindowSink.totalReceived` counts EVERY direct `receive(_:)` call, including calls made outside the `Intellectus` gate (e.g., in tests that call `window.receive(...)` directly). It is not scoped to gated traffic only. +- `NoOpSink` is the sink installed before any `install(sink:)` call. Because the default `isEnabled` is false, `NoOpSink.receive` is not expected to run in default configuration — but it must remain safe to call, because a host could call `setEnabled(true)` before `install(sink:)`. +- Timestamps are ALWAYS caller-supplied (`Double` epoch seconds). No file in this package reads a clock. Do not add a `Date()`/clock read inside `IntellectusLib` — determinism for replay/testing depends on this. +- Zero repo dependencies is a structural constraint, not a preference: Package.swift's header comment explicitly forbids importing SubstrateTypes/SubstrateKernel or any other repo target from this package, because this package is meant to become the dependency floor those targets build on. Adding a repo import here would create the exact layering cycle the package exists to avoid. +- `EventKind` is a two-case, `CaseIterable`, `String`-backed enum (`.capture`/`.think`). The test suite pins `allCases.count == 2` — adding a case is a breaking, deliberate surface change, not a silent addition. +- Platform floor is macOS 26 / iOS 26 specifically because `Synchronization.Atomic` needs it per project policy; do not lower the floor without re-deriving the atomic gate's performance argument. +- Rust parity is structural (file-for-file, function-for-function) but NOT byte-fixture-gated like LatticeLib: there is no `rust/tests/fixtures/*.json` here, because `StatSample` values are opaque caller data, not values this library derives from an input. Conformance is judged by matching test suite sections (see rust/tests/intellectus_lib_tests.rs vs. Tests/IntellectusLibTests/IntellectusLibTests.swift §1–§8), not by fixture replay. diff --git a/packages/libs/IntellectusLib/docs/DETAILS.md b/packages/libs/IntellectusLib/docs/DETAILS.md new file mode 100644 index 0000000..87b99e3 --- /dev/null +++ b/packages/libs/IntellectusLib/docs/DETAILS.md @@ -0,0 +1,246 @@ +--- +doc: DETAILS +package: IntellectusLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/IntellectusLib/Intellectus.swift + blob: 47d1dbe60d0ed880eaf657986ef9371f66423fda + - path: Sources/IntellectusLib/RecentWindowSink.swift + blob: 275f059bb1c0b5ebea68c7d32d2e6bb5ad5ed034 + - path: Sources/IntellectusLib/StatSample.swift + blob: 5a63d181680c35a1b79279f83651f8cb62dcd9b7 + - path: Sources/IntellectusLib/StatsSink.swift + blob: 60dc64a00370e491273c2328409a9d56fc46463b +--- + +# IntellectusLib Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in the order data +flows through the library: the sample type, the receiver contract, the one +bundled receiver, and finally the global facade that ties everything +together for callers. + +## StatSample.swift + +This file provides `StatSample`, the telemetry datum that flows through the +whole library, and `EventKind`, a small enum used by one of its two cases. + +`StatSample` is a value type — an enum with associated values, not a class — +and it conforms to `Sendable`, meaning the Swift compiler checks that it is +safe to pass between threads or concurrent tasks without extra +synchronization. Value types copy instead of sharing memory, which is what +makes that safety check possible: no two threads can race on the same +storage. + +The enum has two cases. `.metric` carries a dot-separated name (such as +`"locus.capture.latency_ms"`), a floating-point value, an optional +dictionary of string tags for context (such as `["kit": "LocusKit"]`), and a +timestamp. `.event` carries a lifecycle event: an `EventKind` (`.capture` or +`.think`), the integer form of a substrate row's noun type, the row's +identifier string, the estate identifier string, and a timestamp. `EventKind` +distinguishes a caller-driven write (`.capture`) from a substrate-driven +autonomous transition (`.think`) — the two broad classes of activity the +substrate's Brain layer can trigger. + +Every timestamp in the library is a caller-supplied number of seconds since +the epoch, never a value read from the system clock inside this package. This +matters for determinism: a caller that replays the same sequence of events +gets the same timestamps back, and tests can construct samples with fixed, +predictable times instead of racing against the real clock. + +`ts`, a computed property added in an extension, returns the timestamp +regardless of which case the sample is. This spares every caller that only +cares about ordering samples in time from writing a `switch` over both cases. + +## StatsSink.swift + +This file provides `StatsSink`, the protocol every receiver of telemetry +samples must conform to, and `NoOpSink`, the receiver installed before any +host supplies its own. + +A protocol in Swift is a contract: it lists the methods and properties a +conforming type must supply, without dictating how. `StatsSink` requires +exactly one method, `receive(_:)`, and requires conformers to be `Sendable`, +because the one sink installed at any moment is held in global state and +called from whatever thread happens to be reporting a sample at the time. + +`receive(_:)` delivers one `StatSample` to the sink. The file documents an +expectation, not an enforced rule: implementations should return quickly and +should not block. Anything slow — writing to a database, sending data over a +network — belongs in the sink's own background work, scheduled from inside +`receive(_:)`, not run synchronously inside it. A slow sink would otherwise +add latency to whatever substrate code path triggered the sample. + +`NoOpSink` is the simplest possible conformer: `receive(_:)` does nothing and +returns immediately. It exists so the global holder in `Intellectus.swift` +always has a valid sink installed, even before a host calls `install(sink:)`. +Because monitoring is off by default, `NoOpSink.receive(_:)` is not expected +to run in ordinary use — the enabled gate in `Intellectus.swift` stops a +sample before it ever reaches a sink — but the no-op is still the correct, +safe answer if a host enables monitoring before installing a real sink. + +## RecentWindowSink.swift + +This file provides `RecentWindowSink`, the one concrete `StatsSink` +implementation the library ships: a fixed-size, in-memory buffer of the most +recently received samples. + +The buffer is what programmers call a ring buffer, or FIFO ring (first in, +first out): a fixed-length array where new entries overwrite the oldest slot +once the array is full, so the memory used never grows past the buffer's +capacity no matter how many samples arrive over the buffer's lifetime. This +bound is a deliberate contract (documented in the file as SPEC I-8): a +component that is supposed to help observe the system must not itself become +a source of unbounded memory growth. + +`RecentWindowSink` optionally wraps a second sink, called the forward sink. +Every sample the window records is, after recording, also handed to the +forward sink if one was supplied at construction. This decorator arrangement +— one sink wrapping another — lets a host keep a live, readable window of +recent activity and simultaneously persist every sample to durable storage, +using a single sink installed with `Intellectus.install(sink:)`. + +`init(capacity:forward:)` builds the sink. The requested capacity is clamped +to a minimum of one: a zero or negative capacity would make the window +useless, so the initializer silently corrects it rather than crashing or +throwing, treating the input as a caller mistake rather than a fatal +condition. + +`receive(_:)` is the `StatsSink` method. It takes an internal lock, writes the +sample into the ring buffer's next slot, advances the write position, and +counts the sample toward a running total, then releases the lock before +calling the forward sink. Running the forward sink outside the lock matters +for the same reason `Intellectus.report(_:)` calls its sink outside its own +lock: a forward sink that itself needs to take a lock — its own, or even this +same one, if it happened to be reinstalled recursively — cannot deadlock +against this method, because this method is not holding anything by the time +the forward sink runs. + +`snapshot()` returns a copy of the buffer's current contents, oldest sample +first. Because it is a copy taken while holding the lock only long enough to +read the buffer, a caller can freely inspect or iterate the returned array +without any risk of it changing underneath them, and without blocking +further calls to `receive(_:)` for any longer than the copy itself takes. + +`count` reports how many samples the buffer currently holds, which is at most +its capacity. `totalReceived` reports how many samples have ever been handed +to `receive(_:)`, including ones already evicted from the buffer. The +distinction matters for a host reading the window: a `count` of zero could +mean either "monitoring just started" or "monitoring has been off the whole +time," and `totalReceived` tells them apart — a nonzero total with an empty +buffer would be a contradiction that never actually occurs, but a zero total +is the library's explicit signal that nothing has crossed the gate yet. + +## Intellectus.swift + +This file provides the library's single entry point, the `Intellectus` +public enum, together with its private backing implementation, +`_IntellectusHolder`. + +`Intellectus` is what the file calls a stateless namespace: an enum with no +cases, used purely to group related static functions under one name, because +Swift has no notion of free-standing module-level functions. Every piece of +mutable state — the installed sink, and the on/off flag — lives instead in a +single global instance of `_IntellectusHolder`, a class not exposed outside +the package. This split keeps the public surface simple (four static +members) while keeping the actual synchronization logic in one well-tested +place. + +The on/off flag is stored as a `Synchronization.Atomic`, a type +supplied by the platform that lets a single boolean be read and written +directly by the processor without an operating-system lock. The file +explains the reasoning in a comment: an uncontended `NSLock` still costs +roughly six nanoseconds to acquire and release on Apple Silicon, while a +single atomic read costs roughly one nanosecond and can sometimes be +optimized away entirely inside a tight loop. Since the flag is checked on +every single call to `report(_:)` — the library's hottest path by far — this +difference compounds across a busy substrate. + +Reading and writing the flag use matched atomic orderings, `.acquiring` for +reads and `.releasing` for writes. These orderings are a promise to the +compiler and processor: once a read observes the flag as `true`, it is also +guaranteed to observe any other data a writer set up before flipping the flag +to `true`. Without that promise, a thread could theoretically see the flag +turn on before it saw the sink that was installed alongside it, even though +program order says the sink was installed first. + +The installed sink, in contrast, is protected by an ordinary `NSLock`, because +an "any `StatsSink`" value — the type-erased container that lets any +conforming type be stored — cannot be swapped as a single atomic operation +the way a boolean can. The lock is held only long enough to copy the sink +reference out; the sink's `receive(_:)` method always runs after the lock has +already been released, matching the same outside-the-lock discipline +`RecentWindowSink.receive(_:)` uses for its own forward sink. + +`_IntellectusHolder.install(sink:)` replaces the currently installed sink. +Because it is expected to run rarely — typically once, at process startup — +it can afford the ordinary lock; a report already in flight on another +thread finishes with whichever sink it captured, and the very next report +sees the new one. + +`_IntellectusHolder.setEnabled(_:)` flips the atomic flag. `isEnabled` reads +it back. Both are simple one-line wrappers, but their simplicity is the +point: this is the only place the ordering guarantees above need to be +stated and reasoned about. + +`_IntellectusHolder.report(_:)` is the function every other function in this +file exists to support quickly. It takes its sample-building argument as an +`@autoclosure`, a Swift feature that wraps an expression in a closure +automatically at the call site, so the expression is not evaluated as soon as +it is written — only when and if the closure is actually called. `report(_:)` +reads the atomic flag first; if it is `false`, the function returns +immediately, and the argument expression is never evaluated at all, which is +exactly the "off-path" cost the whole file is designed around: one atomic +load and a branch, nothing more. Only when the flag is `true` does the +function take the sink lock, copy the sink reference, release the lock, then +finally evaluate the sample-building expression and hand the result to the +sink. + +`Intellectus.install(sink:)`, `Intellectus.setEnabled(_:)`, and +`Intellectus.isEnabled` are thin public wrappers over the matching +`_IntellectusHolder` members, forwarding to the single global instance, +`_intellectus`. `Intellectus.report(_:)` does the same for the reporting +path, and repeats the `@autoclosure` on its own parameter so the +short-circuit behavior holds all the way from a substrate call site down to +the atomic check, with no intermediate function ever forcing early +evaluation of the sample expression. + +## Rust Port and Conformance + +The `rust/` directory contains the second leg of the library, structured to +mirror the Swift files one for one: `sample.rs` matches `StatSample.swift`, +`sink.rs` matches `StatsSink.swift`, `window.rs` matches +`RecentWindowSink.swift`, and `holder.rs` together with `global.rs` matches +`Intellectus.swift`. `lib.rs` assembles the crate's public surface and adds +one addition the Swift side does not need: a `report!` macro, which gives +Rust callers the same short-circuit evaluation that Swift gets for free from +`@autoclosure` — the macro expands to an `if` check around the sample +expression, so the expression is only compiled into a branch that runs when +monitoring is enabled. + +The two legs use different low-level tools for the same guarantees. Swift +uses `Synchronization.Atomic` with `.acquiring`/`.releasing` orderings; +Rust uses `std::sync::atomic::AtomicBool` with `Ordering::Acquire` and +`Ordering::Release` — the same memory-ordering concept under the standard +library's own name. Swift protects the installed sink with `NSLock`; Rust +protects it with `std::sync::Mutex`, wrapping the sink in an `Arc` (Rust's +shared, reference-counted pointer) so it can be cloned out from under the +lock the same way Swift copies the "any `StatsSink`" existential out from +under its own lock. `RecentWindowSink`'s ring buffer is a fixed-size Swift +array with a manually tracked head position on the Swift side, and a +`VecDeque` — a double-ended queue supplied by Rust's standard library — on +the Rust side; both enforce the same bound and the same oldest-first eviction +rule. + +The package's `Tests/IntellectusLibTests/IntellectusLibTests.swift` suite and +the Rust suite in `rust/tests/intellectus_lib_tests.rs` are organized around +matching sections — gating behavior, the no-op sink, thread safety, the +sample type, `EventKind`, a performance smoke check, and the recent-window +sink — so a change in one leg has a direct counterpart to check in the other, +even though, unlike LatticeLib, this package has no shared byte-for-byte +conformance fixtures: nothing here produces a value that two platforms need +to agree on, since a `StatSample` is opaque data the caller constructs, not a +value this library computes from an input. diff --git a/packages/libs/IntellectusLib/docs/OVERVIEW.md b/packages/libs/IntellectusLib/docs/OVERVIEW.md new file mode 100644 index 0000000..e42fd92 --- /dev/null +++ b/packages/libs/IntellectusLib/docs/OVERVIEW.md @@ -0,0 +1,121 @@ +--- +doc: OVERVIEW +package: IntellectusLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/IntellectusLib/Intellectus.swift + blob: 47d1dbe60d0ed880eaf657986ef9371f66423fda + - path: Sources/IntellectusLib/RecentWindowSink.swift + blob: 275f059bb1c0b5ebea68c7d32d2e6bb5ad5ed034 + - path: Sources/IntellectusLib/StatSample.swift + blob: 5a63d181680c35a1b79279f83651f8cb62dcd9b7 + - path: Sources/IntellectusLib/StatsSink.swift + blob: 60dc64a00370e491273c2328409a9d56fc46463b +--- + +# IntellectusLib Overview + +## What This Library Does + +IntellectusLib lets any part of the MOOTx01 substrate report a small fact +about itself while it runs: how long an operation took, how many times +something happened, or that a particular lifecycle event occurred. It calls +this small fact a telemetry sample. A telemetry sample is either a named +measurement, such as "capture latency was 42.5 milliseconds," or a record of +a lifecycle transition, such as "row X was captured in estate Y." + +The library does not decide what to do with a sample. It hands each sample to +a receiver, called a sink, that the host program installs. A host that wants +to persist samples to disk installs a sink that writes to a database. A host +that only wants a live look at recent activity installs a sink that keeps the +last few hundred samples in memory. IntellectusLib supplies one such sink, +`RecentWindowSink`, and a protocol, `StatsSink`, so any host can supply its +own. + +## The Problem It Solves + +Substrate code runs on a hot path: the same functions execute many times per +second while a user captures memories or the system dreams through them. +Measuring that code is valuable, but a always-on measurement system is not +free. Building a telemetry payload costs time and memory even if nothing +reads the result afterward. + +Most of the time, no one is watching. A production device runs with +monitoring off by default. IntellectusLib is built around one guarantee: when +monitoring is off, reporting a sample costs almost nothing — a single memory +read and a comparison, nothing else. The library never builds a payload, never +takes a lock, and never calls a sink unless a host has explicitly turned +monitoring on. This guarantee is what lets substrate code call +`Intellectus.report(...)` freely, everywhere, without worrying about the cost +of doing so. + +The library also has to work correctly when many threads report samples at +once, since the substrate itself runs concurrently. Turning monitoring on or +off, or swapping in a new sink, must never corrupt a sample that is in flight +on another thread. + +IntellectusLib keeps this promise in two independent implementations. A Swift +leg serves Apple platforms, and a Rust leg (in `rust/`) mirrors it, section for +section, function for function, for hosts that need a Rust build of the +substrate. Both legs use the same design: a lock-free flag for the on/off +gate, and a lock held only briefly around the installed sink. + +## How It Works + +Reporting a sample happens in one call: `Intellectus.report(...)`. That call +takes an argument built lazily — Swift's `@autoclosure` — meaning the +expression that builds the sample is not evaluated until `report` decides it +is needed. `report` checks a single on/off flag first. If monitoring is off, +the function returns immediately, and the sample-building expression never +runs at all. Nothing is allocated, and no sink is touched. + +If monitoring is on, `report` reads the currently installed sink, briefly +holding a lock so it cannot read a sink in the middle of being replaced, then +releases the lock, builds the sample, and calls the sink's `receive(_:)` +method. The sink runs outside the lock, so a slow or reentrant sink cannot +block a concurrent call that only wants to install a new sink. + +A host controls the gate with `Intellectus.setEnabled(_:)` and installs a +receiver with `Intellectus.install(sink:)`. Both are meant to be called +rarely — typically once, at startup, or when an operator toggles monitoring +on or off. The library ships one ready-to-use sink, `RecentWindowSink`, which +keeps a fixed-size, oldest-evicted-first buffer of the most recent samples in +memory. A host can read that buffer back at any time to see recent activity, +which is useful for a live dashboard or a diagnostic command. `RecentWindowSink` +can also wrap a second sink and forward every sample to it after recording it, +so a host can keep a live window and write to durable storage from a single +installed sink. + +## How the Pieces Fit + +Figure 1 shows the library's topology — its major parts and how a sample +moves through them. + +![Figure 1. Topology of IntellectusLib](topology.svg) + +*Figure 1. Topology of IntellectusLib. A caller's report flows through the +enabled gate; when open, the sample reaches the installed sink, which may be +the bundled `RecentWindowSink` forwarding on to a second, host-supplied sink. +The dashed region marks the parts a host, not the library, supplies.* + +`StatSample.swift` defines the data that flows through the system: the +`StatSample` enum and its two cases, `.metric` and `.event`. `StatsSink.swift` +defines the receiver contract, `StatsSink`, and the default do-nothing +receiver, `NoOpSink`, which is installed before any host calls `install(sink:)`. +`RecentWindowSink.swift` provides the one concrete sink the library ships. +`Intellectus.swift` ties the pieces together: it holds the current sink and +the on/off flag, and exposes the `Intellectus` enum as the single public entry +point that the rest of the substrate calls. + +## What Ships in the Package + +The package ships four Swift source files and a matching Rust port in +`rust/`. It has no dependency on any other library in the repository — not +even the lowest substrate libraries — so that those libraries can, in turn, +depend on IntellectusLib without creating a dependency cycle. It depends only +on Foundation and on Swift's `Synchronization` module, both supplied by the +platform. The platform floor is macOS 26 and iOS 26, chosen because that is +the first OS release where `Synchronization.Atomic` is available without a +compatibility fallback. diff --git a/packages/libs/IntellectusLib/docs/topology.svg b/packages/libs/IntellectusLib/docs/topology.svg new file mode 100644 index 0000000..a935eda --- /dev/null +++ b/packages/libs/IntellectusLib/docs/topology.svg @@ -0,0 +1,91 @@ + + + + + + + + + + + + + IntellectusLib: sample in, gated delivery out + + + + Caller + Intellectus.report(...) + + + + Enabled gate + Atomic<Bool> load + off = autoclosure never runs + + + + disabled (default) + return — no sink call + + + + _IntellectusHolder + NSLock snapshot of + installed sink, then unlock + + + + host-installed StatsSink (protocol boundary) + + + NoOpSink + default, discards + + + Custom sink + host-supplied + + + RecentWindowSink + bundled: fixed-capacity ring buffer, + oldest evicted; snapshot() / count / totalReceived + + + forward sink + optional, host durable store + + + + + + + + + + sample.receive(_:) — outside the lock + diff --git a/packages/libs/SubstrateKernel/docs/AGENT_MAP.md b/packages/libs/SubstrateKernel/docs/AGENT_MAP.md new file mode 100644 index 0000000..e72f061 --- /dev/null +++ b/packages/libs/SubstrateKernel/docs/AGENT_MAP.md @@ -0,0 +1,145 @@ +--- +doc: AGENT_MAP +package: SubstrateKernel +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateKernel/BitField.swift + blob: 755ce095e0db4d4d4f5f01cc3568cb8403e5952f + - path: Sources/SubstrateKernel/FloatVecOps.swift + blob: b1c4593a99fe078b19ad0058a2e76e9949abd524 + - path: Sources/SubstrateKernel/HammingNN.swift + blob: c810cb5ada7b0c1cdc7789791552478b19bdad07 + - path: Sources/SubstrateKernel/HKDF.swift + blob: 53cd1dbef1ea0b9a8c932cdcfc303b50eb49adba + - path: Sources/SubstrateKernel/PortableKernel-Metal.swift + blob: 4c9837448547cb44fde52a08b8e639876d23f150 + - path: Sources/SubstrateKernel/PortableKernel-NEON.swift + blob: 996063a101bcd76d2ea965b58072eac2141adf98 + - path: Sources/SubstrateKernel/PortableKernel-SIMD.swift + blob: 88caebd2a66c6541f9dff3aff7555dabc557ee5f + - path: Sources/SubstrateKernel/PortableKernel.swift + blob: 5279dadeb909d4d8980f2fc3a3100caf7d9066d3 + - path: Sources/SubstrateKernel/SHA256.swift + blob: 143a3530a8dee15446fa5711022638e0d61219b2 +--- + +# AGENT_MAP — SubstrateKernel + +PURPOSE: layer 2 of the four-package substrate split (SubstrateTypes → SubstrateKernel → SubstrateML → SubstrateLib). Bandwidth-bound bit ops over `Fingerprint256`: Hamming distance/top-K, OR-reduce, SimHash, count-fold, popcount; plus BitField (packed-bitmap field extract/write), SHA-256 content hashing, HKDF-SHA256 key derivation, and canonical scalar float-vector ops. No pinned data artifacts — behavior is pure algorithm, not versioned reference data. + +DEPS: imports SubstrateTypes (Fingerprint256, HyperplaneFamily, CountVector256, FloatSimHashPlanes, SimHash, Hamming, BlockMask), IntellectusLib (Intellectus.report telemetry — zero-dep leaf), Foundation, `simd` (Apple + aarch64 Linux), Metal (Apple only, `#if canImport(Metal)`). Imported by: SubstrateML (depends on SubstrateTypes + SubstrateKernel per dependency graph); hot-path consumers LocusKit, CorpusKit, GeniusLocusKit, EngramLib; SubstrateLib's AuditGate consumes only BitField + SHA256 (not the kernel dispatch layer). Rust port in rust/ mirrors kernel.rs/bit_field.rs/hamming_nn.rs/sha256.rs/hkdf.rs/float_vec_ops.rs function-for-function; kernel_simd.rs (nightly-gated `simd-nightly` feature) covers NEON's ground so there is no neon.rs; kernel_avx512.rs (x86_64-only, DARK path, P11-VAL-015) has no Swift counterpart; Metal has no Rust counterpart (explicit platform waiver, no Linux/Windows GPU path). + +ENTRY POINTS (most callers need only these): +- PortableKernel.swift:313 `PortableKernel.kernelForCurrentPlatform() -> SubstrateKernel` — auto-selects SimdKernel (arm64) or ScalarKernel (else); never selects Metal +- PortableKernel.swift:368 `PortableKernel.kernel(of: KernelKind) -> SubstrateKernel` — explicit backend selection (tests, conformance harness, opt-in GPU/NEON) +- BitField.swift:62/86 `BitField.extractField/writeField(...)` — packed-bitmap field access, the F18 centralization surface +- SHA256.swift:30 `SHA256.hash(_ bytes:) -> [UInt8]` — FIPS 180-4 content hash +- HKDF.swift:54 `GrantHKDF.deriveKey(...)` — RFC 5869 HKDF-SHA256 grant scope-key derivation + +## Symbol Table + +### Protocol + reference kernel — PortableKernel.swift +- :49 `protocol SubstrateKernel: Sendable` — every backend conforms; all methods must be bit-identical across conformers +- :54 `kind: KernelKind` — runtime introspection; default `.scalar` (extension :137), every concrete kernel overrides +- :57 `popcount64(_:) -> Int` — 64-bit population count +- :60 `hammingDistance256(_:_:) -> Int` — bit-differences between two Fingerprint256 +- :64 `orReduce256(_:) -> Fingerprint256` — bitwise OR fold; identity = zero +- :69 `hammingTopK(probe:candidates:k:) -> [(index:Int,distance:Int)]` — K nearest by Hamming distance, ties → lower index +- :76 `simhashCompute(subhashes:families:) -> Fingerprint256` — 4-subhash → Fingerprint256 projection +- :91/:97/:102 `hammingDistanceBatch` / `simhashBlockBatch` / `orReduceBatch` — batched variants; protocol-extension defaults are naive loops (:139/:144/:157), always correct, backends may override for speed +- :112/:116 `countFold256` / `countFoldBatch` — per-bit-position set counts across a cohort; default (:166) = `CountVector256.fold`; OR-reduce is this fold's saturating-at-one degenerate case +- :127 `floatSimHashProject(vector:planes:) -> Fingerprint256` — float analogue of simhashCompute; default impl (:183) is the SCALAR ORACLE for this op — fixed left-to-right accumulation order (FP `+` non-associative, order IS the contract); precondition on `planes.dim == vector.count` +- :222 `struct ScalarKernel: SubstrateKernel` — THE ORACLE; every other backend's output must match this bit-for-bit for the same input +- :233 `hammingDistance256` — `a.zip4(b, ^).popcount()` (SubstrateTypes combinator) +- :239 `orReduce256` — `Fingerprint256.reduce4(fingerprints, |)` +- :245 `hammingTopK` — bounded max-heap O(N log k); tie-break lower-index-wins is the cross-backend contract +- :287 `enum KernelKind` — scalar|simd|neon|metal|avx512|avx2 (avx512/avx2 always fall to scalar in THIS Swift build — no Swift AVX implementation exists) +- :296 `enum PortableKernel` — dispatcher namespace +- :313 `kernelForCurrentPlatform()` — arm64→SimdKernel else ScalarKernel; COMPILE-TIME choice, no runtime CPU-feature probing (contrast Rust's Avx512 CPUID guard, which DOES probe at runtime because that backend is unsafe without it); emits `substrate.kernel.backend_selected` telemetry, autoclosure skipped (no clock read) when monitoring off +- :355 `currentArchTag` — static string ("arm64"/"x86_64"/"other"), must track the `#if arch()` predicate above it +- :368 `kernel(of:)` — explicit selector; `.metal` falls back to ScalarKernel if `MetalKernel()` init fails (no GPU); `.avx512`/`.avx2` always → ScalarKernel +- :406 `assertEqual(lhs:rhs:probe:candidates:k:)` — hammingTopK-based bit-identity check; conformance-suite building block +- :430 `ScalarKernelScored` / :440 `ScalarKernelMaxHeap` (internal) — (distance, index) max-heap shared plumbing; root = worst retained + +### SIMD backend — PortableKernel-SIMD.swift +- :39 `struct SimdKernel: SubstrateKernel` — selected by kernelForCurrentPlatform() on arm64; `import simd` → NEON codegen +- :65 `hammingDistance256` — SIMD4 XOR + per-lane nonzeroBitCount, summed +- :114 `hammingTopK` — sorted-ladder maintenance (NOT a heap); identical sorted output + tie-break to ScalarKernel +- :214 `simhashBlockBatch` — uses :233 `PackedFamily` (SoA repack of 64 hyperplanes into 16 groups of 4) built ONCE per batch, amortized across inputs via :275 `simhashBlockSIMD` +- :376 `countFold256` — bit-sliced vertical binary counter across SIMD4 lanes; O(fingerprints × ~2 ops) vs scalar's bit-by-bit walk; conformance-gated against `CountVector256.fold` + +### NEON backend — PortableKernel-NEON.swift +- :38 `struct NeonKernel: SubstrateKernel` — NOT auto-selected; reachable only via `PortableKernel.kernel(of: .neon)`; exists to be BENCHMARKED against SimdKernel, not to replace it +- :63 `toBytes(_:) -> SIMD32` (static) — reframes Fingerprint256 at byte level via single 256-bit load (not byte-by-byte copy) +- :87 `popcountBytes(_:)` (static) — 32 scalar nonzeroBitCount calls; Swift does NOT auto-vectorize this into one `cnt.16b` (measured gap, see file header) +- :104/:113/:128 `hammingDistance256` / `hammingDistanceBatch` / `hammingTopK` — byte-level XOR + popcount + horizontal sum; hammingTopK uses same ladder pattern as SimdKernel +- :170/:182/:186 `orReduce256` / `orReduceBatch` / `simhashCompute` — NOT re-framed at byte level; identical to SimdKernel's implementation (byte reframing judged not to help these ops) + +### Metal backend — PortableKernel-Metal.swift (Apple-only, `#if canImport(Metal)`) +- :67 `MetalBufferPool` (fileprivate, `@unchecked Sendable`) — persistent GPU buffers sized for `maxN`; caller-enforced per-call synchronization, harness is single-threaded +- :110 `struct MetalKernel: SubstrateKernel` — NOT auto-selected; reachable only via `PortableKernel.kernel(of: .metal)` +- :128 `defaultMaxN = 100_000` — buffer-pool sizing default (~3.6 MB), tuned to dreaming-daemon batch-index scale +- :134 `init?(maxN:)` — returns nil on ANY setup failure (no GPU, shader compile fail, pipeline fail); PortableKernel.kernel(of:) treats nil as "fall back to ScalarKernel", not an error +- :172 `popcount64` / :182 `hammingDistance256` — inherited scalar semantics; per-pair GPU dispatch not worth its ~10–30µs overhead +- :203 `hammingDistanceBatch` — :210 dispatches :221 `dispatchWithPool` (N ≤ maxN, reuse buffers) or :283 `dispatchWithFreshBuffers` (N > maxN, allocate per-call) +- :418 `scalarFallback` (private) — CPU fallback on ANY Metal failure (buffer alloc, command-buffer creation, GPU execution error) — never crashes or returns a wrong/partial result +- :364 `hammingTopK` — GPU batched distance + CPU heap-of-K selection (GPU-side top-K selection judged not worth it for small K) +- :440 `shaderSource` (private static) — embedded Metal shader source (`hamming_distance_kernel`); must be kept byte-for-byte in sync with the canonical `.metal` file or the conformance gate's CRC check fails + +### Standalone top-K reference — HammingNN.swift +- :32 `struct HammingNNHit: Hashable, Sendable` — (rowID: UUID, distance: Int) +- :51 `enum HammingNN` +- :53 `topK(anchor:candidates:k:blocks:) -> [HammingNNHit]` — NOT part of the SubstrateKernel/PortableKernel dispatch; operates over any `(UUID, Fingerprint256)` sequence, not a dense array; supports partial-block comparison via `blocks: BlockMask` (e.g. topic-only similarity); requires `k > 0` +- tie-break: rowID.uuidString ascending (DIFFERENT key from PortableKernel backends' index-ascending — do not conflate the two top-K contracts) +- :94/:101/:114 `heapLarger` / `heapifyUp` / `heapifyDown` (private) — bounded max-heap, same shape as ScalarKernelMaxHeap but keyed on UUID string not array index + +### Packed-bitmap fields — BitField.swift +- :45 `enum BitField` — F18 atomic-centralization surface; kits must NOT open-code bit-shift/mask logic +- :62 `extractField(_:shift:width:) -> Int64` — precondition: 0≤shift, 1≤width≤64, shift+width≤64 +- :86 `writeField(_:into:shift:width:) -> Int64` — same preconditions; value bits outside `width` are SILENTLY truncated (documented contract, not a bug) +- :135 `maskedEquals(_:mask:expected:) -> Bool` — one-step `(bitmap & mask) == expected`; caller invariant: expected must already be masked/aligned +- :153/:164 `extractFlag` / `writeFlag` — single-bit case; precondition 0≤bit<64 +- :181 `popcount(_:) -> Int` — via `UInt64(bitPattern:).nonzeroBitCount` +- :188 `hammingDistance(_:_:) -> Int` — `popcount(a ^ b)`, single-Int64 version (contrast Fingerprint256's 256-bit version elsewhere) +- :200 `xorFold(_:) -> Int64` — running XOR reduce; empty → 0 + +### Content hashing — SHA256.swift +- :27 `enum SHA256` — FIPS 180-4, dependency-free +- :30 `hash(_ bytes: [UInt8]) -> [UInt8]` — 32-byte digest; ported VERBATIM from GeniusLocusKit's prior in-kit copy so centralization preserves every existing content ID byte-for-byte +- :119 `rotr` (private) — right-rotate helper + +### Key derivation — HKDF.swift +- :37 `enum GrantHKDF` — RFC 5869 HKDF-SHA256, built entirely over this package's own SHA256 (no CryptoKit) for cross-platform byte-identical key derivation +- :54 `deriveKey(inputKeyMaterial:salt:info:outputByteCount:) -> [UInt8]` — public entry; `outputByteCount` ≤ 8160 (32×255) +- :69 `extract(salt:ikm:)` — RFC 5869 §2.2, `hmac(key: salt, data: ikm)` +- :76 `expand(prk:info:length:)` — RFC 5869 §2.3, chained HMAC with counter byte +- :101 `hmac(key:data:) -> [UInt8]` — public (not just internal) because SubstrateLib's KeyedCommitment API reuses this exact construction; RFC 2104, keys > 64 bytes pre-hashed + +### Scalar float-vector oracle — FloatVecOps.swift +- :62 `enum FloatVecOps` — canonical IEEE-754 scalar reference; every faster backend (Accelerate/BLAS/etc.) MUST match bit-for-bit +- :72 `l2Norm(_:) -> Float` — sqrt(sum(x*x)); empty → 0.0 +- :99 `l2Normalize(_:) -> [Float]` — zero-vector PASSTHROUGH (not NaN, not a panic) — the "no information" signal that projects to Engram.zero via FloatSimHash +- :121 `dot(_:_:) -> Float` — precondition a.count == b.count +- :147 `cosine(_:_:) -> Float` — DEFINED ONLY for pre-normalized unit vectors (caller must call l2Normalize first); debug-only assert checks norm ≈ 1.0 within 1e-5, compiled away in release + +## INVARIANTS / GOTCHAS + +- CONFORMANCE IS THE CONTRACT. Every `SubstrateKernel` backend (Scalar/Simd/Neon/Metal, both legs) must produce bit-identical output to `ScalarKernel` for the same input. A divergence is a bug in the backend, never grounds to change the scalar oracle. +- `FloatVecOps` is a second, independent oracle for decimal-vector math (l2Norm/l2Normalize/dot/cosine); same bit-identical-to-scalar rule applies to any faster override elsewhere in the substrate. +- TWO DIFFERENT top-K contracts coexist and must not be conflated: `PortableKernel` backends' `hammingTopK` (dense array input, ties → lower ARRAY INDEX) vs. `HammingNN.topK` (candidate-sequence input, ties → lower `rowID.uuidString`, supports partial-block comparison via `BlockMask`). +- `kernelForCurrentPlatform()` is a COMPILE-TIME choice (`#if arch(arm64)`), not a runtime CPU-feature probe. It never selects `.metal`; Metal requires explicit `PortableKernel.kernel(of: .metal)`. `.avx512`/`.avx2` have no Swift implementation and always resolve to `ScalarKernel` in this build. +- Rust's AVX-512 backend (`kernel_avx512.rs`) is the opposite of the Swift dispatcher: it DOES check `is_x86_feature_detected!("avx512f")` && `("avx512vpopcntdq")` at runtime before every unsafe intrinsic call, because skipping that check is undefined behavior (SIGILL) on CPUs lacking those features. This guard was present from the initial commit and was hardened/test-covered in the 2026-06-28 security review (`tests/avx512_hamming_conformance.rs::cpuid_guard_prevents_intrinsics_without_feature`). `for_current_platform()` never auto-selects it regardless (dark path, P11-VAL-015 gates enable). +- `MetalKernel.init?` returning `nil` is a NORMAL, expected outcome (no GPU / headless CI / virtualization), not an error to surface — callers fall back to scalar. +- `MetalKernel.hammingDistanceBatch` falls back to `scalarFallback` on ANY Metal failure (buffer alloc, command-buffer/encoder creation, GPU execution error) — never crashes, never returns a partial/wrong result. +- `MetalKernel.defaultMaxN` = 100,000: batches at or under this reuse the persistent `MetalBufferPool`; batches over it pay full per-call buffer allocation via `dispatchWithFreshBuffers`. +- `MetalKernel.shaderSource` is a hand-maintained string copy of the canonical `.metal` shader file; the conformance gate's CRC check is the only thing that catches drift between them — keep them in sync manually. +- `BitField.writeField`'s value truncation (bits of `value` outside `width` are silently dropped) is an intentional contract, not a bug — matches packed-row semantics where the field's width defines its value space. +- `SHA256.hash` was ported VERBATIM from GeniusLocusKit specifically to preserve existing audit-log content IDs byte-for-byte; do not "clean up" the implementation without checking downstream ID stability. +- `HKDF`/`GrantHKDF` uses ONLY the in-repo `SHA256`, never CryptoKit — required for the Swift↔Rust byte-identical key-derivation guarantee (PAR-4-GL1). +- `GrantHKDF.hmac` is public (not internal) because `SubstrateLib`'s KeyedCommitment API depends on reusing this exact HMAC-SHA256 construction; do not narrow its access without checking that caller. +- No pinned/versioned data artifacts ship in this package (contrast e.g. LatticeLib's Resources/ lexicon+frame+signatures) — everything here is pure algorithm; conformance is enforced by test fixtures, not artifact versioning. +- `AuditGate` (in `SubstrateLib`, NOT in this package) consumes only `BitField` + `SHA256` from here — it does not touch the kernel-dispatch layer at all. +- No NEON or Metal Rust files exist by design: `kernel_simd.rs` (nightly `simd-nightly` feature) covers NEON's aarch64 ground on the Rust side; Metal has no cross-platform equivalent and is an explicit, documented platform waiver. +- Telemetry (`Intellectus.report` / `substrate.kernel.backend_selected`) is off by default; when off, the reporting autoclosure is never evaluated and no clock (`Date()`) is read — selection has zero overhead in the default configuration. diff --git a/packages/libs/SubstrateKernel/docs/DETAILS.md b/packages/libs/SubstrateKernel/docs/DETAILS.md new file mode 100644 index 0000000..3b89114 --- /dev/null +++ b/packages/libs/SubstrateKernel/docs/DETAILS.md @@ -0,0 +1,344 @@ +--- +doc: DETAILS +package: SubstrateKernel +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateKernel/BitField.swift + blob: 755ce095e0db4d4d4f5f01cc3568cb8403e5952f + - path: Sources/SubstrateKernel/FloatVecOps.swift + blob: b1c4593a99fe078b19ad0058a2e76e9949abd524 + - path: Sources/SubstrateKernel/HammingNN.swift + blob: c810cb5ada7b0c1cdc7789791552478b19bdad07 + - path: Sources/SubstrateKernel/HKDF.swift + blob: 53cd1dbef1ea0b9a8c932cdcfc303b50eb49adba + - path: Sources/SubstrateKernel/PortableKernel-Metal.swift + blob: 4c9837448547cb44fde52a08b8e639876d23f150 + - path: Sources/SubstrateKernel/PortableKernel-NEON.swift + blob: 996063a101bcd76d2ea965b58072eac2141adf98 + - path: Sources/SubstrateKernel/PortableKernel-SIMD.swift + blob: 88caebd2a66c6541f9dff3aff7555dabc557ee5f + - path: Sources/SubstrateKernel/PortableKernel.swift + blob: 5279dadeb909d4d8980f2fc3a3100caf7d9066d3 + - path: Sources/SubstrateKernel/SHA256.swift + blob: 143a3530a8dee15446fa5711022638e0d61219b2 +--- + +# SubstrateKernel Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in this +order: the shared protocol and its reference implementation, the three +specialized hardware backends, the standalone top-K primitive, and +finally the three independent utilities — bit fields, hashing, and +float-vector math. + +## PortableKernel.swift + +This file provides the `SubstrateKernel` protocol, the `ScalarKernel` +reference implementation, and the `PortableKernel` dispatcher that +hands callers a concrete kernel. + +The protocol declares every operation a kernel must supply: +`popcount64` (count the 1-bits in a 64-bit word), `hammingDistance256` +(count differing bits between two fingerprints), `orReduce256` +(bitwise-OR a group of fingerprints into one), `hammingTopK` (find the +K candidates nearest a probe fingerprint), `simhashCompute` (fold a +set of 64-bit numbers into a fingerprint), `countFold256` (tally, per +bit position, how many fingerprints in a group have that bit set), and +`floatSimHashProject` (the same folding operation starting from a +vector of decimal numbers instead of integers). A protocol extension +supplies default, correct-but-unoptimized implementations of the +batched variants of each operation, built as simple loops over the +single-item version; a backend that has a faster batched strategy +overrides the default, but every backend is correct even before it +does. + +`ScalarKernel` implements every required method as a straightforward +loop with no special hardware instructions. It is deliberately the +simplest possible correct implementation, because it is the oracle: a +backend is defined as correct when its output for a given input +matches `ScalarKernel`'s output for that same input. Two of its +methods matter beyond their obvious job. `hammingTopK` maintains a +bounded max-heap of the K best candidates seen so far, evicting the +current worst kept candidate whenever a better one arrives, which +keeps the cost proportional to N times the log of K rather than N +times the log of N; ties between equal distances are broken toward the +candidate with the lower index, and every other backend must match +that exact tie-break. `floatSimHashProject`'s default body (defined in +the protocol extension, inherited by every backend that does not +override it) is itself the reference for that operation: for each of +256 hyperplanes it accumulates a signed sum over the input vector in a +fixed left-to-right order, because floating-point addition is not +associative and the order is therefore part of the contract, not an +implementation detail. + +`PortableKernel.kernelForCurrentPlatform()` is the entry point most +callers use. On 64-bit ARM (`arm64`, which covers Apple Silicon and +ARM64 Linux) it returns `SimdKernel`; everywhere else it returns +`ScalarKernel`. The choice is fixed at compile time for a given +platform — there is no runtime hardware probing here, unlike the +Rust port's AVX-512 backend, which checks the processor's feature +flags at runtime because that backend is unsafe to run without them. +Every call reports which backend it chose through IntellectusLib's +telemetry system, but only when telemetry is enabled; when it is off, +the reporting closure is never evaluated and no clock is read, so the +selection has no cost in normal operation. `PortableKernel.kernel(of:)` +is the explicit-selection counterpart used by tests and by any caller +that wants a specific backend regardless of platform; requesting +`.metal` falls back to `ScalarKernel` if no GPU is available, and +requesting `.avx512` or `.avx2` always falls back to `ScalarKernel` in +this Swift build, since neither has a Swift implementation. +`PortableKernel.assertEqual` runs `hammingTopK` on two kernels with the +same inputs and reports whether their results match — the building +block the conformance test suite uses to check every backend against +the oracle. + +## PortableKernel-SIMD.swift + +This file provides `SimdKernel`, the backend `kernelForCurrentPlatform` +selects on 64-bit ARM. + +`SimdKernel` uses Swift's `simd` module, which represents four 64-bit +numbers as one `SIMD4` value and lets ordinary operators like +XOR and OR act on all four at once. The Swift compiler turns those +operators into single ARM NEON instructions rather than four separate +ones. `hammingDistance256` XORs two fingerprints as `SIMD4` +values, then counts the 1-bits in each of the four resulting 64-bit +lanes and adds the four counts together. `hammingTopK` uses a +different strategy from `ScalarKernel`'s heap: a small sorted array of +the K best candidates found so far ("ladder"), shifted to make room +for a new entry, which the file's comments note is faster than a heap +for the small K values this package expects, but produces the +identical sorted output and tie-break. + +`simhashBlockBatch` computes one 64-bit SimHash block for many input +vectors against the same hyperplane family — a fixed set of 64 +reference directions used to decide, for each bit of the output, which +side of that direction the input vector falls on. Rather than compute +that decision one hyperplane at a time for every input, `PackedFamily` +rearranges the family's 64 hyperplanes once, into groups of four, so +the per-input inner loop can process four hyperplanes together with +the same `SIMD4` trick used for Hamming distance. Building this +packed layout costs a little extra time up front, which is worthwhile +because that cost is paid once and the savings apply to every input in +the batch. + +`countFold256` computes, for a group of fingerprints, how many of them +have each of the 256 bit positions set. Rather than count each +fingerprint's contribution to each bit position one at a time, +`SimdKernel` keeps a growing binary counter for each bit position, held +across `SIMD4` lanes, and folds each new fingerprint in with +two bitwise operations: the counter's next bit is the counter XOR the +fingerprint, and the amount to carry into the following bit is the +counter AND the fingerprint. This produces the same result as adding +one for every set bit, but does the addition four fingerprint-blocks +at a time instead of bit by bit. + +## PortableKernel-NEON.swift + +This file provides `NeonKernel`, an alternate ARM implementation kept +side by side with `SimdKernel` to test a different way of expressing +the same computation. + +`SimdKernel` counts bits four 64-bit words at a time. `NeonKernel` +instead treats a fingerprint as 32 individual bytes and counts bits one +byte at a time before adding all 32 counts together. The file's +comments record the reasoning: processors often have an instruction +that counts bits across a whole row of bytes in one step, and the +hope was that framing the computation at the byte level would let the +compiler use that instruction more effectively than the word-level +framing does. Whether it actually helps is a question the package +answers by measurement, not assumption — a companion benchmark +compares the two — and `NeonKernel` exists so that measurement has +something concrete to measure. It is never chosen automatically; +callers reach it only through `PortableKernel.kernel(of: .neon)`. + +Every method mirrors its `SimdKernel` counterpart at the byte level +instead of the word level: `hammingDistance256` and +`hammingDistanceBatch` XOR two byte-vectors and sum per-byte bit +counts, and `hammingTopK` uses the same ladder-based top-K selection +as `SimdKernel`. `orReduce256`, `orReduceBatch`, and `simhashCompute` +are not re-framed at the byte level, because the file's comments judge +that the byte-level trick that helps Hamming distance does not help +those operations, so they use the same implementation `SimdKernel` +uses. + +## PortableKernel-Metal.swift + +This file provides `MetalKernel`, which runs the batched Hamming +distance computation on the graphics processor through Apple's Metal +framework, and which is available only on Apple platforms. + +Dispatching work to the GPU has a fixed cost per call — tens of +microseconds to set up buffers and hand off the computation — that +does not shrink as the batch grows. That fixed cost makes GPU dispatch +a poor choice for comparing one pair of fingerprints, so `MetalKernel` +does not override `hammingDistance256`; it inherits the scalar version +for single pairs and only specializes the batched form, +`hammingDistanceBatch`, where the GPU's throughput advantage outweighs +its setup cost. `MetalBufferPool` removes a second fixed cost: rather +than allocate a fresh block of GPU-visible memory on every call, the +kernel allocates its buffers once, sized for up to `defaultMaxN` +(100,000) candidates, and reuses them on every call within that size; +a batch larger than that falls back to `dispatchWithFreshBuffers`, +which allocates for that call only. + +`init?(maxN:)` builds the one-time Metal state: it asks the system for +its default GPU, compiles the compute shader embedded in +`shaderSource`, and builds the buffer pool. Any step failing — +because there is no GPU (a virtualized environment, for instance) or +the shader fails to compile — makes the initializer return `nil`, +which is a normal outcome, not an error condition: `PortableKernel` +falls back to `ScalarKernel` whenever `MetalKernel`'s initializer +returns `nil`. `hammingDistanceBatch` writes the probe and candidate +fingerprints into GPU-visible memory, dispatches one GPU thread per +candidate, waits for the GPU to finish, and reads the resulting +distances back; if any Metal call along that path fails, `scalarFallback` +computes the same batch on the CPU instead, so a transient GPU problem +degrades performance rather than producing a wrong or missing answer. +`hammingTopK` computes distances through the GPU path above, then +selects the K smallest with the same CPU-side max-heap `ScalarKernel` +uses, because GPU-side selection of a small K from a large result set +is not worth its complexity. + +## HammingNN.swift + +This file provides `HammingNN`, a standalone top-K search over +fingerprints that does not go through the `PortableKernel` protocol at +all. + +Where `PortableKernel`'s backends operate on a dense array of +candidates and select the fastest hardware path, `HammingNN.topK` +operates over any sequence of `(rowID, fingerprint)` pairs and is meant +as a simple, general-purpose reference a caller can use directly +without choosing a backend. It also supports comparing only a subset +of a fingerprint's four blocks through its `blocks` parameter — useful +when a caller wants similarity on, say, only the topic-related portion +of a fingerprint rather than the whole thing. Internally it maintains +the same kind of bounded max-heap `ScalarKernel` uses, and it breaks +ties the same way in spirit — deterministically — though by the row's +UUID string rather than by array index, since this function's +candidates do not come with array positions of their own. `topK` +requires `k` to be positive and returns the results sorted by distance, +then by UUID string, so that two runs over the same data always +produce the same order. + +## BitField.swift + +This file provides `BitField`, a set of pure functions for reading and +writing fixed-width fields packed inside a 64-bit integer. + +MOOTx01 packs several small values — flags, small counters, short +codes — into single 64-bit "bitmap" words, because doing so keeps +those words compact and fast to compare. Every package that needs to +read or write one of those packed fields is expected to call through +this file rather than write its own bit-shifting logic, so that a +change to how a field is laid out only has to be made once. +`extractField(_:shift:width:)` reads a field of `width` bits starting +at bit position `shift`; `writeField(_:into:shift:width:)` writes a new +value into that same position while leaving every other bit +untouched. `extractFlag` and `writeFlag` are the single-bit special +case of the same idea. `maskedEquals` tests whether a bitmap's masked +bits equal an expected value in one step, which is useful when a +caller only wants a yes-or-no answer and does not need the field's +actual value. `popcount`, `hammingDistance`, and `xorFold` round out +the file with the same bit-counting and bit-comparison operations the +kernel layer uses, but defined here over plain 64-bit integers instead +of 256-bit fingerprints, for callers working with a single packed +word rather than a whole fingerprint. + +## SHA256.swift + +This file provides `SHA256`, a complete implementation of the +SHA-256 cryptographic hash algorithm with no dependency on any +system library. + +MOOTx01's audit log identifies each entry by a hash of its exact +contents, so that two replicas that received the same entry compute +the same identifier and can recognize it as the same entry without +comparing the whole thing. That only works if every device computes +the hash the same way, which is why this file implements the standard +itself instead of relying on a platform library that could change +behavior between operating system versions. The implementation was +carried over unchanged from an existing in-kit copy specifically so +that centralizing it here would not change any identifier a caller +had already computed and stored. `SHA256.hash(_:)` is the entire +public surface: it takes a list of bytes and returns the 32-byte +digest, following the standard's padding and compression steps +exactly. + +## HKDF.swift + +This file provides `GrantHKDF`, an implementation of HKDF-SHA256 (a +standard method for deriving new cryptographic keys from existing key +material), built entirely on top of this package's own `SHA256`. + +MOOTx01's grant system needs to derive a key specific to one sharing +grant from an estate's underlying cryptographic identity, without +exposing that identity directly. Building this derivation on the +in-repo `SHA256`, rather than on a system cryptography library, means +a key derived on one platform is guaranteed to be identical to the +same derivation run on another platform, which is essential when two +devices must independently arrive at the same derived key. +`GrantHKDF.deriveKey(inputKeyMaterial:salt:info:outputByteCount:)` is +the public entry point; it runs the standard's two steps, extract and +expand, in sequence. `extract` and `expand` implement those two steps +directly from the specification. `hmac(key:data:)` implements the +keyed-hash construction both steps depend on and is itself public, +because SubstrateLib's commitment-signing feature needs the same +HMAC-SHA256 construction over its own data and this avoids a second +implementation existing anywhere in the substrate. + +## FloatVecOps.swift + +This file provides `FloatVecOps`, the canonical scalar implementations +of four vector operations: length, normalization, dot product, and +cosine similarity, all over ordinary lists of decimal numbers. + +Just as `ScalarKernel` is the oracle for fingerprint operations, +`FloatVecOps` is the oracle for decimal-vector operations: any faster +implementation elsewhere in the substrate — using a platform math +library, for instance — is required to match these functions bit for +bit, not merely approximately. `l2Norm(_:)` computes a vector's length +by summing the square of each element and taking the square root. +`l2Normalize(_:)` rescales a vector to length one, which is the form +most similarity comparisons expect; a vector of all zeros has no +meaningful direction to rescale to, so the function returns it +unchanged rather than dividing by zero, and that unchanged zero vector +is itself a meaningful signal elsewhere in the substrate for "no +information available." `dot(_:_:)` sums the products of corresponding +elements from two equal-length vectors. `cosine(_:_:)` measures the +angle between two vectors, but only gives a meaningful answer when +both inputs are already unit-length; rather than normalize them itself +on every call, which would be wasted work when a caller normalizes +once and compares many times, it requires the caller to normalize +first and, in debug builds only, checks that the caller did so. + +## Rust Port and Conformance + +The `rust/` directory contains the second leg of the library: `kernel.rs` +mirrors `PortableKernel.swift`'s protocol, `ScalarKernel`, and +dispatcher; `bit_field.rs`, `hamming_nn.rs`, `sha256.rs`, `hkdf.rs`, and +`float_vec_ops.rs` mirror their same-named Swift files function for +function. Two Swift backends have no Rust counterpart. `NeonKernel` +does not need one, because Rust's own portable-SIMD path +(`kernel_simd.rs`, gated behind the nightly-only `simd-nightly` Cargo +feature) already covers the same aarch64 targets. `MetalKernel` cannot +have one: Metal is an Apple-only framework, and the Rust port's +documentation records this explicitly as a platform waiver rather than +a gap. The Rust leg also ships one backend the Swift leg lacks +entirely: `kernel_avx512.rs`, an AVX-512 implementation for x86-64 +processors. It compiles and is reachable through explicit selection, +but `PortableKernel::for_current_platform()` never chooses it — it +stays a dark path until a future performance study proves it belongs +in the default path on real AVX-512 hardware. Because an unguarded call +into its processor-specific instructions would crash on hardware that +lacks them, every one of its entry points checks the processor's +actual feature flags at runtime before making that call, a guard +verified in a 2026-06-28 security review and covered by a dedicated +cross-platform test. Both legs share one conformance obligation: +whichever backend a caller selects, on either leg, its output must +match the scalar reference bit for bit, and the test suites for both +legs enforce this on every change. diff --git a/packages/libs/SubstrateKernel/docs/OVERVIEW.md b/packages/libs/SubstrateKernel/docs/OVERVIEW.md new file mode 100644 index 0000000..8dc6a85 --- /dev/null +++ b/packages/libs/SubstrateKernel/docs/OVERVIEW.md @@ -0,0 +1,170 @@ +--- +doc: OVERVIEW +package: SubstrateKernel +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateKernel/BitField.swift + blob: 755ce095e0db4d4d4f5f01cc3568cb8403e5952f + - path: Sources/SubstrateKernel/FloatVecOps.swift + blob: b1c4593a99fe078b19ad0058a2e76e9949abd524 + - path: Sources/SubstrateKernel/HammingNN.swift + blob: c810cb5ada7b0c1cdc7789791552478b19bdad07 + - path: Sources/SubstrateKernel/HKDF.swift + blob: 53cd1dbef1ea0b9a8c932cdcfc303b50eb49adba + - path: Sources/SubstrateKernel/PortableKernel-Metal.swift + blob: 4c9837448547cb44fde52a08b8e639876d23f150 + - path: Sources/SubstrateKernel/PortableKernel-NEON.swift + blob: 996063a101bcd76d2ea965b58072eac2141adf98 + - path: Sources/SubstrateKernel/PortableKernel-SIMD.swift + blob: 88caebd2a66c6541f9dff3aff7555dabc557ee5f + - path: Sources/SubstrateKernel/PortableKernel.swift + blob: 5279dadeb909d4d8980f2fc3a3100caf7d9066d3 + - path: Sources/SubstrateKernel/SHA256.swift + blob: 143a3530a8dee15446fa5711022638e0d61219b2 +--- + +# SubstrateKernel Overview + +## What This Library Does + +SubstrateKernel computes the small set of bit-level and hash +operations that MOOTx01 runs on every memory it stores. It compares +fingerprints, extracts values from packed bitmaps, and computes +cryptographic hashes. A fingerprint is a short fixed-size code +computed from a piece of content; similar content produces similar +fingerprints, so the system compares things quickly without reading +them in full. This package works with a specific fingerprint shape, +`Fingerprint256`, a 256-bit code held as four 64-bit blocks. + +Comparing two fingerprints means counting how many bit positions +differ between them. That count is called Hamming distance: the +number of positions where two equal-length codes differ. A smaller +Hamming distance means the two fingerprints, and the content behind +them, are more similar. Finding the memories most similar to a query +means finding the fingerprints with the smallest Hamming distance to +a probe fingerprint. This is the single most frequent operation in +MOOTx01's recall path, so SubstrateKernel gives it several +implementations, each tuned for a different kind of hardware. + +SubstrateKernel is layer 2 of a four-package substrate split. Layer 1, +SubstrateTypes, defines pure data types with no logic, including +`Fingerprint256` itself. Layer 2, this package, computes over those +types. Layer 3, SubstrateML, builds learning and graph algorithms on +top of layer 2. Layer 4, SubstrateLib, orchestrates the whole +substrate for its callers. A package that composes libraries into a +larger subsystem is called a kit; kits depend on libraries like this +one, never the reverse. Four kits currently depend on SubstrateKernel +directly for their hot paths: LocusKit, CorpusKit, GeniusLocusKit, and +EngramLib. + +## The Problem It Solves + +Two devices must agree on how similar two fingerprints are. MOOTx01 +estates — one user's complete memory store — can federate, meaning +separate devices share and compare memories. If one device's Hamming +distance for a pair of fingerprints disagreed with another device's +distance for the same pair, shared recall would produce different +answers on different hardware. Every operation in this package must +therefore produce the same output for the same input, on every +platform, every time. This package calls that guarantee its +conformance contract, and it treats one implementation, `ScalarKernel`, +as the oracle every other implementation must match bit for bit. + +Meeting that contract while also running fast is harder than meeting +it alone. A scan over a million fingerprints, at 32 bytes each, moves +32 megabytes through memory for a single query. That volume of +straight-line bit manipulation is bandwidth-bound: its cost is +dominated by how fast data moves through the processor, not by how +complex the arithmetic is. Different processors move bandwidth-bound +data at different speeds depending on which instructions they use, so +one fixed implementation cannot be fastest everywhere. SubstrateKernel +resolves this by defining one interface, several interchangeable +implementations behind it, and a single reference implementation that +every one of them is checked against. + +A second, smaller problem is duplication. Bit-field extraction from a +packed bitmap, and content hashing for the audit log, are needed by +many packages upstream of SubstrateKernel. Left unmanaged, each +package would write its own version, and a change to the bit layout +or the hash algorithm would require updating every copy. SubstrateKernel +centralizes both so a single implementation serves every caller. + +## How It Works + +The `SubstrateKernel` protocol declares the operations every backend +must supply: Hamming distance between two fingerprints, top-K nearest +neighbor search, bitwise OR-reduction across a group of fingerprints, +SimHash projection (folding a set of numbers into a fingerprint), and +a handful of batched variants of each. `ScalarKernel` implements the +protocol with a plain loop over each fingerprint's four 64-bit blocks. +It is always available, on every platform, and it is the oracle: any +other implementation that disagrees with it on any input is, by +definition, wrong. + +Three more implementations specialize the same protocol for particular +hardware. `SimdKernel` uses Swift's portable `simd` module, which the +compiler turns into ARM NEON instructions on Apple Silicon. `NeonKernel` +frames the same computation a different way, at the level of +individual bytes rather than 64-bit words, to test whether that shape +compiles to tighter machine code. `MetalKernel` dispatches the batched +Hamming distance computation to the GPU through Apple's Metal +framework, which pays a fixed per-call setup cost but scales well past +roughly one hundred thousand candidates. A caller selects an +implementation through `PortableKernel`, either automatically for the +current platform or explicitly by name for testing. + +Two more file pairs round out the package. `BitField` extracts and +writes fixed-width fields inside a 64-bit packed bitmap — the encoding +MOOTx01 uses to pack several small values into one machine word. +`SHA256` computes a standard cryptographic hash, used both to give +each audit-log entry a unique, content-derived identifier and, through +`HKDF`, to derive keys for the estate's cryptographic grants. A last +file, `FloatVecOps`, defines the canonical floating-point vector +operations — length, normalization, dot product, cosine similarity — +that any faster backend for those operations must also match bit for +bit. A final file, `HammingNN`, offers a simpler, general-purpose top-K +search over any sequence of candidates, independent of the +`PortableKernel` backends, for callers that do not need backend +selection. + +## How the Pieces Fit + +Figure 1 shows the library's topology — its major parts and how data +moves between them. + +![Figure 1. Topology of SubstrateKernel](topology.svg) + +*Figure 1. Topology of SubstrateKernel. A caller asks `PortableKernel` +for a kernel; the dispatcher hands back one of four interchangeable +implementations of the `SubstrateKernel` protocol, all conformance- +gated against the `ScalarKernel` oracle. `HammingNN`, `BitField`, +`SHA256`, `HKDF`, and `FloatVecOps` are independent primitives that do +not go through the dispatcher.* + +`PortableKernel.kernelForCurrentPlatform()` picks `SimdKernel` on +64-bit ARM and falls back to `ScalarKernel` everywhere else; it never +selects `MetalKernel`; a caller wanting the GPU path must ask for it +by name with `PortableKernel.kernel(of: .metal)`. `NeonKernel` is +likewise available only by explicit request, since it exists to be +measured against `SimdKernel`, not to replace it automatically. +Whichever implementation runs, its output for a given input is +required to match `ScalarKernel`'s output for that same input; a +conformance test suite enforces this for every backend, on both the +Swift and Rust legs. + +## What Ships in the Package + +The package ships nine Swift source files and no pinned data +artifacts — unlike some sibling libraries, SubstrateKernel's behavior +depends only on its algorithms, not on any versioned reference data. +It also ships a Rust port in `rust/`, mirroring every file except the +two that have no meaningful Rust equivalent: `NeonKernel`, because +Rust's portable-SIMD path already covers the same ground on the +relevant targets, and `MetalKernel`, because Metal is an Apple-only +framework with no Linux or Windows counterpart. The Rust leg adds one +backend the Swift leg does not have — an AVX-512 implementation for +x86-64 processors — but keeps it dark: built, tested, and reachable +only by explicit request, never chosen automatically, until a future +performance study proves it belongs in the default path. diff --git a/packages/libs/SubstrateKernel/docs/topology.svg b/packages/libs/SubstrateKernel/docs/topology.svg new file mode 100644 index 0000000..df905ef --- /dev/null +++ b/packages/libs/SubstrateKernel/docs/topology.svg @@ -0,0 +1,127 @@ + + + + + + + + + + + + + SubstrateKernel: kernel dispatch plus independent primitives + + + + Caller + e.g. LocusKit + + + + PortableKernel + kernelForCurrentPlatform / kernel(of:) + + + + SubstrateKernel protocol — every backend bit-identical to Scalar + + + ScalarKernel + the oracle + + + SimdKernel + arm64 default + + + NeonKernel + opt-in, benchmark + + + MetalKernel + opt-in, GPU + + + Hamming / OR-reduce / + SimHash / count-fold + result: identical across backends + + + + + + + + + + + + + Independent primitives — no dispatcher, called directly + + + HammingNN.topK + candidate-iterator top-K, + rowID-string tie-break + + + BitField + packed-bitmap + field extract/write + + + FloatVecOps + scalar oracle: + norm / dot / cosine + + + SHA256 + FIPS 180-4 + content hash + + + GrantHKDF + RFC 5869 + key derivation + + + + Every primitive above is called directly by its consumer; none passes through PortableKernel. + Audit-log content IDs (SubstrateLib's AuditGate) consume only BitField + SHA256 from this package. + Metal is Apple-only; the Rust port has no Metal leg (platform waiver) and adds a dark AVX-512 backend Swift lacks. + diff --git a/packages/libs/SubstrateLib/docs/AGENT_MAP.md b/packages/libs/SubstrateLib/docs/AGENT_MAP.md new file mode 100644 index 0000000..b9cef3c --- /dev/null +++ b/packages/libs/SubstrateLib/docs/AGENT_MAP.md @@ -0,0 +1,142 @@ +--- +doc: AGENT_MAP +package: SubstrateLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateLib/AuditGate.swift + blob: 1f5ea748a82d21756518ed2dc4dda9e0a2ceed4a + - path: Sources/SubstrateLib/KeyedCommitment.swift + blob: 2b93f766914a2261cc4d63067469a0609157665f + - path: Sources/SubstrateLib/KeyedCommitmentAudit.swift + blob: afdac27147578c4b0cd2324e3db861812f89097a + - path: Sources/SubstrateLib/MerkleHash.swift + blob: 2a7e1b957d0f38633ee9048108fb9b59f8ee5648 + - path: Sources/SubstrateLib/RowStateAutomaton.swift + blob: b3111584e1495a41d66e51de620e27e55f31d4cd + - path: Sources/SubstrateLib/SubstrateLibTelemetry.swift + blob: b76dd8280311aeb67d8f5499108a0c0b080dc787 + - path: Sources/SubstrateLib/Verbs.swift + blob: b3f6b0db6b099d6932f7e57c017637193e2a4be6 +--- + +# AGENT_MAP — SubstrateLib + +PURPOSE: orchestration layer of the substrate stack. Single write gate (AuditGate.admit) + row-state automaton (legal transitions, I-22/S-1/S-2/S-4 forbidden combinations) + reference impl of the nine substrate verbs (Substrate struct) + Merkle content-hash pipeline + keyed-commitment pipeline for expunge provenance. Value types (Row, RowState, NounType, LatticeAnchor, AuditEvent, HLC, MatrixF/O/T, Fingerprint256) live in SubstrateTypes. + +DEPS: imports SubstrateTypes (value types), SubstrateKernel (BitField, SHA256, GrantHKDF.hmac), SubstrateML (transitively via Package.swift, not used directly in these 7 files), IntellectusLib (Intellectus.report telemetry sink). Imported by: LocusKit (AuditGate.admit is the sole bitmap write path), other kits per DECISION_KIT_GRAPH_REFACTOR_2026-05-19. Sibling packages in the four-way split: SubstrateTypes (data), SubstrateKernel (hot-path kernels), SubstrateML (cold-path/ML). Rust port in rust/src/ mirrors all 7 files 1:1; conformance fixtures in rust/tests/ + Tests/SubstrateLibConformanceTests/ gate byte-identity. + +ENTRY POINTS (most callers need only these): +- AuditGate.swift:317 `AuditGate.admit(estateUuid:rowId:nounType:verb:prior:priorLatticeAnchor:writes:afterLatticeAnchor:vocabulary:hlc:actor:) -> Result` — THE single production write gate for row bitmaps +- AuditGate.swift:222 `VocabularyValidator.freeze(union:) -> Result` — one-time vocabulary construction, required before any Vocabulary exists +- RowStateAutomaton.swift:213 `RowStateAutomaton.validate(from:on:targetingFields:) throws -> RowState` — the constitutional mutation gate (legality + I-22/S-1/S-2/S-4) +- Verbs.swift:76 `struct Substrate` — in-memory reference impl of the nine verbs; scalar oracle for production storage engines + +## Symbol Table + +### Row-state automaton — RowStateAutomaton.swift +- :50 `enum RowStateAutomaton` — namespace; RowState/RowVerb/RowStateError types live in SubstrateTypes, imported here +- :72 `canTransition(from:to:viaVerb:) -> Bool` — §10 STRING-verb vocabulary (legacy); looks up :84 `verbTable` (fileprivate `[VerbKey: RowState]`); used by Verbs.swift mutate/withdraw +- :78 `fileprivate struct VerbKey` — (RowState, String) composite key for verbTable +- :127 `static let transitions: [TransitionKey: RowState]` — CANONICAL enum-keyed table (§9 lifecycle vocabulary); absence from map = illegal +- :204 `transition(from:on:) -> RowState?` — single lookup against `transitions` +- :213 `validate(from:on:targetingFields:) throws -> RowState` — transition() + ForbiddenCombinations.check(); THE gate; bypassing is forbidden (v0.36 C1 resolution) +- :227 `struct TransitionKey` — (from: RowState, verb: RowVerb) +- :240 `struct BitmapFields` — adjective/operational/provenance UInt64 triple +- :260 `enum ForbiddenCombinations` +- :270 `check(state:fields:) throws` — I-22 (secret+exportable forbidden), S-1 (accepted⇒trust≥canonical), S-2 (defensive state-encoding check), S-4 (accepted⇒sensitivity≤elevated); S-5 (tombstone completion flag) NOT YET IMPLEMENTED (queued, see comment at file end) + +### Write gate — AuditGate.swift +- :51 `struct FieldSlot: Hashable, Sendable` — one declared field-slot: column/shift/width/label/legalValues +- :52 `enum Column` — .adjective | .operational | .provenance +- :72 `capacity: Int64` — 1< Bool` — width + enumerated-legal-value check +- :106 `enum AuditState` / :113 `AuditSensitivity` / :118 `AuditExportability` / :123 `AuditTrust` — SubstrateLib-local mirrors of LocusKit's Adjectives.swift enums (raw values MUST match; can't import upward); parity enforced by GuardianPairParityTests +- :134 `struct Vocabulary: Sendable` +- :145 `static let basis: Set` — universal fields: state(0,w6) sensitivity(6,w6) exportability(12,w6) trust(18,w6) flags(24,w3, bits 24/25/26 = state_extension/lineage_clustering/dreaming_recalc_required); legalValues derived from allCases.map{rawValue} +- :188 `union: Set` — per-instance consumer fields, frozen +- :192 `fileprivate init(validatedUnion:)` — ONLY VocabularyValidator.freeze can construct +- :196 `slot(for:) -> FieldSlot?` — basis-first lookup, then union +- :206 `enum VocabularyError` — .overlap / .basisCollision / .malformedWidth / .valueExceedsWidth +- :222 `enum VocabularyValidator` +- :224 `freeze(union:) -> Result` — 3-step validation (width-sane → no basis collision → no union-union overlap); run ONCE before any data exists +- :258 `struct FieldWrite: Sendable` — (slot, value) one consumer-requested change +- :269 `enum GateViolation: Error, Sendable` — .undeclaredField / .illegalValue / .basisViolation(Error) / .stateInconsistentWithVerb +- :292 `description: String` — English-only at ARIA boundary; NO Swift case-name leakage; carries the "illegal state transition: ..." sentinel the describe_gate_rejection parser matches +- :311 `enum AuditGate` +- :317 `admit(...) -> Result` — see ENTRY POINTS; 4 steps: vocab+value gate → read-modify-write (preserves unaddressed bits) → basis gate (state must equal verb's authorized output) → contentID + AuditEvent +- :421 `static func contentID(estateUuid:rowId:hlc:verb:after:afterAnchor:) -> UUID` — SHA-256(stable byte encoding) folded to UUID; NAME-keyed (not ordinal), vocabulary-set-independent; also called directly by Verbs.swift's appendAudit so both paths produce identical event IDs + +### Verb orchestration — Verbs.swift +- :48 `typealias RowId = UUID` — wire-compatible with Rust's RowId(u128) +- :55 `enum SubstrateError: Error, Equatable` — invalidStateTransition / missingLatticeAnchor / invalidNounType / rowNotFound / forbiddenStateCombination / alreadyTombstoned / proposalRequired / nonProposalCannotUseProposalVerb +- :76 `struct Substrate` — see ENTRY POINTS; fields :77-84 (estateUuid, rows, auditEvents(G-Set), matrixF/O/T, hlc, rowCountActive) +- :86 `init(estateUuid:hlc:)` +- :110 `capture(nounType:adjectiveBitmap:operationalBitmap:provenanceBitmap:latticeAnchor:fingerprint:lineageId:content:actor:ts:) -> Result` — ONLY verb that originates a row w/o prior state; proposal⇒pending else active; rejects null latticeAnchor +- :180 `reanchor(rowId:newLatticeAnchor:actor:ts:) -> Result<(), SubstrateError>` — lattice-link change only, bitmaps untouched +- :213 `enum MutationKind: String` — confirm/reject/contest/supersede/automated_confirm/decay/expire/lineage_advance/actuator_confirm +- :222 `mutate(rowId:mutationKind:newAdjectiveBitmap:newOperationalBitmap:newProvenanceBitmap:actor:ts:) -> Result<(), SubstrateError>` — canTransition() (§10 string verbs) + isLegalRowState() gate; general state+field change +- :304 `withdraw(rowId:actor:ts:) -> Result<(), SubstrateError>` — → .withdrawn; setStateField raw=18 +- :365 `expunge(rowId:reason:actor:ts:) -> Result<(), SubstrateError>` — → .tombstoned, content=nil; REFUSES .accepted rows (S-3, audit-grade survive intact) before any other check +- :425 `recall(matching:asOf:ts:) -> [Row]` — READ-ONLY; no mutation, no audit event; asOf reconstructs via audit-log HLC filter +- :457 `propose(...) -> Result` — capture(nounType: .proposal, ...) +- :484 `associate(rowA:rowB:signalSourcesBitset:weight:...) -> Result` — capture(nounType: .association, ...); `weight` param ACCEPTED THEN DISCARDED (vestigial, reserved for pre-2.0 gauntlet experiment) — do not silently wire it up or silently drop the parameter +- :531 `learn(...) -> Result` — capture(nounType: .learnedReference, ...) +- :555 `private appendAudit(verb:rowId:before:after:beforeAnchor:afterAnchor:actor:)` — advances hlc; calls AuditGate.contentID (same fn as the gate path) so both paths produce identical event IDs +- :596 `private isLegalRowState(state:adjective:operational:) -> SubstrateError?` — wraps ForbiddenCombinations.check; provenance passed as 0 (unused by check) +- :615/:624/:632/:637 private bit-twiddling wrappers around SubstrateTypes.RowBitmaps (rowHasBit, extractFieldValues, extractState, setStateField) + +### Merkle hash pipeline — MerkleHash.swift +- :31 `struct MerkleVectorInput: Sendable` — modelID, vectorIndex, floats (Float32 LE) +- :51 `enum MerkleHash` +- :76 `leaf(drawerId:content:vectors:) -> ContentHash` — v2 format: vector identity (modelID+vectorIndex) written BEFORE float payload (v1 gap = WS2-F4 vector-substitution finding, fixed 2026-06-28) +- :101 `interior(childHashes: [(UUID, ContentHash)]) -> MerkleRoot` — sorted by UUID BE bytes; empty ⇒ MerkleRoot.empty +- :130 `interior(childRoots: [(UUID, MerkleRoot)]) -> MerkleRoot` — same as above, wing/estate-level roll-up (children already MerkleRoots) +- :157 `tombstone(drawerId:) -> ContentHash` — domain tag only + drawer id; no content/vectors (already destroyed) +- :178 `static func canonicalLeafBytes(drawerId:content:vectors:domainTag:) -> [UInt8]` — SHARED encoding with KeyedCommitment.commit; domainTag is the only difference between leaf hash and commitment preimage +- :226/:230/:236 private byte-encoding helpers (uuidBytes, appendU64BE, appendU32BE) + +### Keyed commitment — KeyedCommitment.swift +- :26 `struct KeyedCommitmentValue: Hashable, Sendable, Codable` — hmacBytes (32B, precondition-enforced) + keyVersion +- :40 `hexString: String` +- :49 `enum KeyedCommitment` +- :66 `commit(key:keyVersion:drawerId:content:vectors:) -> KeyedCommitmentValue` — MerkleHash.canonicalLeafBytes(domainTag: .commitment=0x03) → GrantHKDF.hmac (SubstrateKernel, no new HMAC impl) + +### Commitment audit log — KeyedCommitmentAudit.swift +- :27 `struct KeyedCommitmentAuditEntry: Hashable, Sendable` — id (content-hash), drawerId, commitment, tombstoneHLC, reason +- :40 `init(drawerId:commitment:tombstoneHLC:reason:)` — computes :62 `private static computeID(...) -> [UInt8]` (SHA-256 over identifying fields) at construction +- :92 `struct CommitmentAuditLog: Sendable` — G-Set keyed by content hash +- :103 `add(_:)` — idempotent insert +- :108 `merge(_:)` — CRDT join = set union +- :114 `count: Int` +- :117 `orderedEntries: [KeyedCommitmentAuditEntry]` — sorted by tombstoneHLC +- :122 `entries(forDrawer:) -> [KeyedCommitmentAuditEntry]` + +### Telemetry — SubstrateLibTelemetry.swift +- :78 `enum SubstrateLibMetric` — string-constant namespace, `substratelib.*` metrics +- :83/:86 auditGateAdmitCount / auditGateRejectCount +- :91/:94/:97/:100/:103/:106 verbCaptureCount / verbMutateCount / verbWithdrawCount / verbExpungeCount / verbRecallCount / verbReanchorCount +- :111/:114 writeGateAdmittedCount / writeGateRejectedCount +- :132 `emitAuditGateAdmit(nounTypeRaw:ts:)`, :152 `emitAuditGateReject(violationName:ts:)` +- :170 `emitWriteGateAdmitted(verb:ts:)`, :187 `emitWriteGateRejected(verb:reason:ts:)` +- :204/:219/:233/:247/:266/:280 `emitVerb{Capture,Mutate,Withdraw,Expunge,Recall,Reanchor}Count(...)` — one per Verbs.swift verb +- ALL emit fns `@inline(__always)`; wrap `Intellectus.report(...)`; disabled-path cost = one atomic load + branch, zero allocation + +## INVARIANTS / GOTCHAS + +- DETERMINISM IS THE CONTRACT (same as LatticeLib). No file in this package may read a clock. Every `ts` parameter is caller-supplied epoch seconds; monitoring on/off must never change functional output. Rust mirrors every algorithm; conformance fixtures in rust/tests/ + Tests/SubstrateLibConformanceTests/ gate byte-identity — run both suites on any change to audit_gate/merkle_hash/keyed_commitment/row_state/verbs. +- AuditGate and Substrate do NOT call each other. Two independent consumers of the same RowStateAutomaton rules, not a pipeline. Do not wire one to call the other — that would let a fast-path bug hide behind a passing reference-path test. +- AuditGate.admit and Verbs.appendAudit call the SAME contentID function (AuditGate.contentID) so both paths produce byte-identical event IDs for the same logical event. If you change the byte layout in contentID, both paths and the Rust mirror change together. +- Content IDs (AuditGate.contentID, KeyedCommitmentAuditEntry.computeID) are NAME/CONTENT-keyed, never random UUIDs. This is what lets a G-Set (grow-only set) dedupe on federation merge. Do not reintroduce `UUID()` random defaults on any audit-adjacent identity. +- S-3 (cookbook §9.5.3): accepted rows CANNOT transition to tombstoned. Enforced in TWO places independently: RowStateAutomaton.transitions (no TransitionKey(.accepted, .tombstone) entry) AND Verbs.expunge's explicit early-return guard. Do not remove either; they are a deliberate belt-and-suspenders pair. +- S-5 (tombstone ⇒ expunge_completed_flag) is NOT YET IMPLEMENTED in ForbiddenCombinations.check — queued pending F17 cookbook location decision (adjective bit 24 or operational reserved bits). A previous incorrect implementation (zeroing bitmaps on tombstone) was removed 2026-05-27; do not reintroduce that specific check. +- AuditState/AuditSensitivity/AuditExportability/AuditTrust (AuditGate.swift) are LOCAL COPIES of LocusKit/Adjectives.swift enums — raw values must match exactly (SubstrateLib cannot import LocusKit; dependency graph points the other way). GuardianPairParityTests enforces this; the `@guardian-pair` comment annotations mark every duplicated literal (also present in RowStateAutomaton.swift's ForbiddenCombinations.check: 48/32/3/16 = secret/public/canonical/elevated raw values). +- Vocabulary can ONLY be constructed via VocabularyValidator.freeze (fileprivate init). Never bypass with a literal struct construction — that skips overlap/collision/width validation entirely. +- FieldSlot.legalValues empty ⇒ "any value that fits width" (used for state, governed by the automaton instead of an enumerated set... actually state DOES enumerate via AuditState.allCases; empty-legalValues is used only by the flags field). Do not conflate "empty" with "no restriction beyond width" vs "enumerated" — check the specific basis slot. +- MerkleHash leaf format is v2 (vector identity before float payload). v1 had a vector-substitution security gap (WS2-F4, fixed 2026-06-28). KeyedCommitment.commit shares this exact byte encoding via canonicalLeafBytes — a leaf-format change without updating both Rust and Swift, and both the hash and commitment call sites, reopens or diverges the fix. +- MerkleHash.interior sorts children by UUID ascending (lexicographic 16-byte BE) — required for roll-up to be write-order-independent. Do not sort by insertion order or hash value. +- Verbs.associate's `weight` parameter is accepted and explicitly discarded (`_ = weight`) — vestigial, reserved for a future experiment. Do not silently start persisting it without a matching bitmap/column change, and do not remove the parameter from the signature without checking callers. +- Package.swift depends on SubstrateTypes + SubstrateKernel + SubstrateML + IntellectusLib directly (not transitively) per DECISION_LIFT_PACKAGE_SWIFT_RULE_2026-05-28 — IntellectusLib sits below SubstrateLib in the topology (depends on nothing in-repo), so this is not a layering violation. +- No pinned data artifacts ship in this package (unlike LatticeLib) — every guarantee here is computational (fixed table, fixed byte encoding, cryptographic hash), not reference-data-driven. diff --git a/packages/libs/SubstrateLib/docs/DETAILS.md b/packages/libs/SubstrateLib/docs/DETAILS.md new file mode 100644 index 0000000..872f5a9 --- /dev/null +++ b/packages/libs/SubstrateLib/docs/DETAILS.md @@ -0,0 +1,417 @@ +--- +doc: DETAILS +package: SubstrateLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateLib/AuditGate.swift + blob: 1f5ea748a82d21756518ed2dc4dda9e0a2ceed4a + - path: Sources/SubstrateLib/KeyedCommitment.swift + blob: 2b93f766914a2261cc4d63067469a0609157665f + - path: Sources/SubstrateLib/KeyedCommitmentAudit.swift + blob: afdac27147578c4b0cd2324e3db861812f89097a + - path: Sources/SubstrateLib/MerkleHash.swift + blob: 2a7e1b957d0f38633ee9048108fb9b59f8ee5648 + - path: Sources/SubstrateLib/RowStateAutomaton.swift + blob: b3111584e1495a41d66e51de620e27e55f31d4cd + - path: Sources/SubstrateLib/SubstrateLibTelemetry.swift + blob: b76dd8280311aeb67d8f5499108a0c0b080dc787 + - path: Sources/SubstrateLib/Verbs.swift + blob: b3f6b0db6b099d6932f7e57c017637193e2a4be6 +--- + +# SubstrateLib Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in dependency +order: the row-state automaton first, because both write paths consult it; +then the write gate and the reference verbs, the two paths that consult +it; then the content-integrity pipeline, which is independent of both; and +finally telemetry, which every other file calls into. + +A note on vocabulary before the walkthrough: a row is one stored memory. A +bitmap is a 64-bit integer that packs several small status fields at fixed +bit positions, so one integer can carry many independent facts cheaply. A +verb is one of the fixed set of actions a row can undergo — capture, +mutate, withdraw, and so on. An audit event is the permanent record of one +verb having happened to one row. + +## RowStateAutomaton.swift + +This file provides the row-state automaton: the transition table that +says which lifecycle-state changes are legal, and the forbidden-combination +checks that catch a legal transition landing on an illegal field +combination. + +The data types the automaton operates on — `RowState` (ten states), +`RowVerb` (twelve verbs), and `RowStateError` — are defined in the sibling +package SubstrateTypes and imported here. Only the compute stays in +SubstrateLib: the table, the validation function, and the forbidden- +combination rules. `RowState`'s ten values are deliberately scale-gapped +into three numeric clusters — active (0–3), historical (16–19), and +terminal (32–33) — so a caller can classify a stored raw value's cluster +with one shift-and-mask, without decoding the full enum. + +`RowStateAutomaton.canTransition(from:to:viaVerb:)` answers a yes-or-no +question for one older vocabulary of verb names (contest, supersede, +expunge, and others) inherited from an earlier design phase. It looks the +pair up in a private table, `verbTable`, and compares the result to the +requested target state. It exists because `Verbs.swift`'s `mutate` and +`withdraw` still speak this vocabulary; new call sites should prefer the +canonical function below instead. + +`RowStateAutomaton.transitions` is the canonical table: a dictionary +keyed by `(from state, verb)`, using the `RowVerb` enum rather than raw +strings. Any pair absent from the table is illegal by construction — there +is no default case to fall through. `RowStateAutomaton.transition(from:on:)` +looks up one pair and returns the next state, or `nil` if the pair is +absent. `RowStateAutomaton.validate(from:on:targetingFields:)` is the +single mutation gate the design calls constitutional: it requires a legal +transition, then also runs the target's bitmap fields through +`ForbiddenCombinations.check`, and only returns the next state if both +pass. An earlier design defect let one code path skip this second check +and write field combinations the table alone would not catch; every +mutation path in this package and in `AuditGate` now routes through one +of these two checks before a write is considered final. + +`ForbiddenCombinations.check(state:fields:)` enforces four safety rules +that the transition table cannot express, because they depend on more than +one field at once: + +- A row marked secret can never also be marked exportable, regardless of + its lifecycle state. Sensitivity and exportability are stored as + separate 6-bit fields in the same bitmap; nothing about the storage + format alone prevents both being set to conflicting extremes, so this + rule is checked explicitly on every write. +- An accepted row — one a person or process explicitly confirmed, treated + as audit-grade — must carry a trust level of at least "canonical." + Accepted status is a promise about reliability, and a low trust value + would make that promise false. +- A withdrawn or rejected row must actually encode the matching state + value in its bitmap, a defensive check against corrupted input rather + than a rule that a caller could realistically violate through the API. +- An accepted row's sensitivity may not exceed "elevated," because a + higher sensitivity would make an audit-grade row too restricted to + serve the purpose accepted status is meant to guarantee. + +A fifth rule the design calls for — that a tombstoned row must carry an +"expunge completed" flag — is deliberately not yet implemented here; the +comment in the source explains that an earlier attempt enforced the wrong +invariant (zeroing bitmap fields that the design says must survive +expunge) and was removed until the correct flag location is settled +elsewhere in the bitmap layout. + +`BitmapFields` and `TransitionKey` are the small carrier types the +functions above operate on: three raw 64-bit integers, and a +`(state, verb)` composite key, respectively. + +## AuditGate.swift + +This file provides the production write gate, `AuditGate.admit`, and the +vocabulary machinery that arms it. This is the interface a storage kit — +LocusKit, in the current design — calls on every bitmap mutation. + +A write gate is only as strong as the vocabulary it enforces, so this file +first defines what a legal field looks like. `FieldSlot` describes one +declared field: which of three columns it lives in (adjective, +operational, or provenance), its bit position and width within that +column, a human label, and its set of legal values. An empty legal-value +set means "any value that fits the width," which basis fields use because +the automaton, not an enumerated list, governs their combinations. +`FieldSlot.admits(value:)` is the per-field check: the value must fit the +width, and if the field enumerates specific legal values, the value must +be one of them. + +`Vocabulary.basis` is the fixed set of fields every SubstrateLib instance +and every federation peer must agree on: state, sensitivity, exportability, +and trust, each a 6-bit field, plus a 3-bit flags field for miscellaneous +markers. Their legal-value sets are derived automatically from four local +enums — `AuditState`, `AuditSensitivity`, `AuditExportability`, +`AuditTrust` — so adding a case to one of those enums extends the gate's +vocabulary without a second, easy-to-forget update elsewhere. These four +enums duplicate raw values defined independently in a higher-level package +(LocusKit), because SubstrateLib sits below that package in the +dependency graph and cannot import its types; a separate test suite +(`GuardianPairParityTests`) checks that the two definitions never drift +apart. + +Beyond the fixed basis, each SubstrateLib instance may register its own +additional fields — the union. `VocabularyValidator.freeze(union:)` is +the one place a union is accepted: it checks that every proposed field's +width and enumerated values fit within 64 bits, that no proposed field's +bits collide with a basis field's bits, and that no two proposed fields +collide with each other. Only a union that passes all three checks becomes +a `Vocabulary`; the type's initializer is `fileprivate`, so an unvalidated +vocabulary cannot reach the gate by any other path. This validation runs +once, before any row exists, which matters because a corrupt vocabulary +discovered after data has accumulated would be far harder to recover from +than one rejected up front. + +`AuditGate.admit(estateUuid:rowId:nounType:verb:prior:priorLatticeAnchor:writes:afterLatticeAnchor:vocabulary:hlc:actor:)` +is the gate itself, and it runs in four steps. First, every field in the +caller's write list must resolve to a declared slot in the vocabulary, and +its value must be admissible; an undeclared field or an out-of-range value +is rejected immediately, before any bitmap changes. Second, the gate +performs a read-modify-write: it starts from the row's prior bitmaps (or +zero, for a new row), writes only the addressed bits, and leaves every +other bit exactly as it was — a consumer that owns one field can never +accidentally clobber a field it does not know about. Third, the gate +checks the resulting state against what the supplied verb was allowed to +produce: for a brand-new row, the verb must be `capture` and the encoded +state must be a legal starting state; for an existing row, the verb's +transition must match the state the write actually encodes, and the +forbidden-combination rules must hold. Fourth, and only if every prior +step succeeded, the gate computes a deterministic content identifier and +returns one canonical `AuditEvent`. + +`AuditGate.contentID(estateUuid:rowId:hlc:verb:after:afterAnchor:)` +computes that identifier: it serializes the identifying fields into a +fixed byte layout, hashes them with SHA-256, and folds the first sixteen +bytes of the digest into a UUID. Because the identifier is a pure function +of the event's content — not a random value — two replicas that compute +the same logical event independently produce the same identifier. That +sameness is what lets a grow-only set deduplicate the event automatically +when replicas merge, which is the property federation depends on. + +`GateViolation` is the gate's error type, covering an undeclared field, an +illegal value, a basis violation (wrapping the underlying +`RowStateAutomaton` error), and a state that does not match its verb. Its +`description` property is written to produce plain English at the +system's outer boundary — no Swift type or case name ever appears in +user-visible text — including a specific sentinel phrase, +"illegal state transition: ...", that a downstream parser matches on. + +## Verbs.swift + +This file provides the nine substrate verbs and `Substrate`, the in-memory +reference implementation of all of them. This is the scalar oracle: +production storage engines implement the same nine verbs against real +persistence, and their behavior is checked against this reference. + +`RowId` is a typealias for `UUID`, kept as a distinct name so call sites +read in the substrate's own vocabulary while staying a plain `UUID` under +the hood; the Rust port spells the same identifier as a `u128` newtype, +and the two encodings are byte-identical. + +`Substrate` bundles everything one estate's in-memory reference state +needs: a dictionary of rows, an append-only list of audit events (treated +as a grow-only set), three summary matrices (`matrixF`, `matrixO`, +`matrixT`, defined in SubstrateTypes) that track aggregate statistics +across all rows without re-scanning them, a hybrid logical clock (`hlc`) +that orders events across replicas without relying on wall time, and a +running count of non-tombstoned rows. + +Each verb method follows the same shape: check preconditions against the +row-state automaton, apply the bitmap change, update the two matrices with +a symmetric delta (subtract the row's old contribution, add its new one), +append one audit event, and emit one telemetry count. Timestamps arrive as +a caller-supplied `ts` parameter defaulting to `0.0`, never read from a +clock inside the method, which keeps the verb's functional output +independent of when it happens to run. + +`capture(nounType:adjectiveBitmap:operationalBitmap:provenanceBitmap:latticeAnchor:fingerprint:lineageId:content:actor:ts:)` +creates a new row. It matters because it is the only verb that can +originate a row without a prior state to transition from: a proposal noun +type starts `pending`, awaiting confirmation, and every other noun type +starts `active`. A missing lattice anchor is rejected outright, because +every row must be locatable in the classification lattice from the moment +it exists. + +`reanchor(rowId:newLatticeAnchor:actor:ts:)` changes only a row's link +into the classification lattice, leaving its bitmaps untouched. It exists +separately from `mutate` because reclassifying a memory is a different +kind of change from changing its lifecycle state or its status fields. + +`mutate(rowId:mutationKind:newAdjectiveBitmap:newOperationalBitmap:newProvenanceBitmap:actor:ts:)` +is the general-purpose state-and-field change, covering confirm, reject, +contest, supersede, decay, expire, and the actuator/automated confirm +variants. It matters because it is where the automaton's transition check +and the forbidden-combination check both gate a single call: an illegal +transition or an illegal resulting combination is rejected before any row +or matrix is touched. + +`withdraw(rowId:actor:ts:)` retracts a row by moving it to the +`withdrawn` state. It matters as a distinct verb, rather than a case of +`mutate`, because withdrawal is available from more starting states than +most other transitions and is a common enough action to warrant its own +simple call shape. + +`expunge(rowId:reason:actor:ts:)` is the legal-compliance hard delete: it +moves a row to `tombstoned` and sets its content to `nil`, destroying the +verbatim payload. It matters that `expunge` explicitly refuses an +`accepted` row before doing anything else — an audit-grade row must +survive intact, by design, and no combination of inputs can route around +that refusal. + +`recall(matching:asOf:ts:)` is the only read-only verb: it filters rows +by a caller-supplied predicate, optionally reconstructing the row set as +of a past point in the hybrid logical clock by intersecting with the +audit log. It matters that recall never mutates and never appends an audit +event — a memory system that logged every read as if it were a write +would make querying create liability. + +`propose`, `associate`, and `learn` are thin verbs that each delegate to +`capture` with a fixed `nounType` — proposal, association, and learned +reference, respectively. They matter as distinct entry points because each +represents a semantically different reason a row came to exist, even +though the mechanics underneath are identical. `associate` additionally +accepts a computed `weight` parameter that it deliberately discards after +accepting it: the parameter is reserved for a future experiment, and the +source documents that the value is accepted and thrown away today rather +than silently invented if it were missing. + +The four private helpers at the bottom of the file — +`appendAudit`, `isLegalRowState`, `rowHasBit` / +`extractFieldValues` / `extractState` / `setStateField` — are internal +plumbing: `appendAudit` advances the clock and builds one `AuditEvent` +using the same content-identifier function `AuditGate` uses, so an event +produced by either path is identical in shape; `isLegalRowState` is the +verb layer's call into `ForbiddenCombinations.check`; the rest are small +bit-manipulation wrappers around the shared `RowBitmaps` type in +SubstrateTypes. + +## MerkleHash.swift + +This file provides the public hash pipeline for the Merkle-style +content-integrity tree: three functions that turn a row's content, a set +of child hashes, or a tombstoned row into one deterministic SHA-256 value. + +A Merkle tree proves that content has not changed by comparing hashes +instead of comparing the content itself, and by letting a parent's hash +summarize all of its children's hashes. `MerkleDomain` tags (defined in +SubstrateTypes) — a leaf tag, an interior tag, and a tombstone tag — are +prepended to each function's input before hashing, so a hash computed one +way can never be mistaken for, or substituted as, a hash computed another +way. + +`MerkleHash.leaf(drawerId:content:vectors:)` hashes one memory's content +and its vector embeddings into a `ContentHash`. It matters because its +byte encoding is versioned: an earlier version wrote each vector's +identifying information (which embedding model produced it, and which +slot in a multi-vector row) only as a sort key, not into the hashed bytes +themselves, which meant an attacker could substitute one model's vector +for another's without changing the hash — a security defect the source +calls out by name (finding WS2-F4). The current version writes vector +identity into the hash input before the float payload, closing that gap. + +`MerkleHash.interior(childHashes:)` and `interior(childRoots:)` roll up a +node's children into one parent hash, sorting children by UUID first so +the result never depends on the order children happened to be written or +iterated in. Both return `MerkleRoot.empty` for a childless node rather +than hashing nothing, which keeps the empty case an explicit, named value +instead of an arbitrary hash of zero bytes. + +`MerkleHash.tombstone(drawerId:)` hashes an expunged row's identity alone, +with no content and no vectors, because expunge has already destroyed +both. It matters that the tombstone hash still exists at all: it lets the +Merkle tree continue to prove that a specific row occupied a specific +position, even after everything about that row except its identity is +gone. + +`canonicalLeafBytes(drawerId:content:vectors:domainTag:)` is the shared +byte-encoding function both this file and `KeyedCommitment.swift` call. +It matters that it is shared rather than duplicated: the content hash and +the keyed commitment are deliberately computed from the exact same +preimage, differing only in the domain tag and the algorithm applied to +it (a plain hash versus a keyed HMAC), which is what lets the two values +be compared meaningfully as two views of the same underlying commitment +to content. + +## KeyedCommitment.swift + +This file provides the public keyed-commitment API used at expunge time: +a way to prove a payload existed without keeping anything a reader could +reconstruct the payload from. + +`KeyedCommitmentValue` carries the output: 32 raw HMAC-SHA256 bytes plus +the key version that produced them. Recording the key version matters +because estate keys rotate over time, and a commitment made under an +older key must remain verifiable against that older key even after +rotation; without the version, a rotated key would silently invalidate +every commitment made before it. + +`KeyedCommitment.commit(key:keyVersion:drawerId:content:vectors:)` builds +the commitment. It reuses `MerkleHash.canonicalLeafBytes`, the exact same +byte encoding `MerkleHash.leaf` uses, but with the commitment domain tag +instead of the leaf tag, and then runs that byte sequence through an +existing HMAC-SHA256 implementation from the sibling package +SubstrateKernel rather than a new one written for this purpose. The domain +separation matters for a specific reason: it guarantees a commitment and a +plain content hash of the same underlying payload are always different +values, so one can never be mistaken for, or substituted as, the other. + +## KeyedCommitmentAudit.swift + +This file provides the audit-log entry type and append-only container for +keyed commitments made at expunge time — a record distinct from the +ordinary state-transition audit event, because it records that a +commitment was made, not that a row's state changed. + +`KeyedCommitmentAuditEntry` bundles the drawer identity, the commitment +value, the hybrid logical clock time the tombstone was applied, and a +human-readable reason. Its identifier, computed by +`computeID(drawerId:commitment:tombstoneHLC:reason:)`, is a SHA-256 hash +over the same identifying fields, which matters for the same reason +`AuditGate.contentID` matters elsewhere in this package: two replicas +that independently record the same logical commitment compute the same +identifier, so a grow-only set deduplicates them without extra +coordination. + +`CommitmentAuditLog` is that grow-only set: entries are keyed by their +content hash in a dictionary, `add(_:)` inserts one entry idempotently, +and `merge(_:)` unions two logs together — the CRDT join that lets two +replicas' commitment logs converge regardless of the order in which +entries arrive. `orderedEntries` and `entries(forDrawer:)` give callers a +stable, HLC-ordered view for display or audit review. + +## SubstrateLibTelemetry.swift + +This file provides the metric name catalogue and the emit functions every +other file in this package calls into when reporting activity. + +SubstrateLib's own comment calls it the "determinism floor" of the +substrate stack: nothing in this package may read a clock or change its +functional output depending on whether monitoring is enabled. This file +is where that promise is implemented, not merely stated. Every emit +function wraps its call in `Intellectus.report(...)` (from the sibling +package IntellectusLib), which — when monitoring is disabled, the +default — costs exactly one lock-free flag check and a branch: no metric +object is built, no lock is taken, nothing is allocated. Every timestamp +the emit functions accept is a parameter supplied by the caller, never +read from a clock inside this file, so the substrate's behavior is +byte-identical whether monitoring is on or off. + +`SubstrateLibMetric` is a namespace of string constants — one per metric +name — so the complete list of everything this package can report is +visible in one place, and a typo in a metric name is caught at compile +time rather than showing up as a silently missing metric in production. + +The eight `emit...` functions each correspond to one point in the gate or +verb flow: `emitAuditGateAdmit` and `emitAuditGateReject` fire from +`AuditGate.admit`'s two outcomes; `emitWriteGateAdmitted` and +`emitWriteGateRejected` fire from the same gate but tagged by verb and +rejection reason, for operators who want to break down gate traffic +differently than by outcome alone; and one function per verb — +`emitVerbCaptureCount`, `emitVerbMutateCount`, `emitVerbWithdrawCount`, +`emitVerbExpungeCount`, `emitVerbRecallCount`, `emitVerbReanchorCount` — +fires from the matching method in `Verbs.swift`. Each is marked +`@inline(__always)` so the disabled-path cost stays a single flag check +with no function-call overhead added on top. + +## Rust Port and Conformance + +The `rust/` directory contains the second leg of the library: seven +source files mirroring the Swift implementations one for one — +`audit_gate.rs`, `keyed_commitment.rs`, `merkle_hash.rs`, `row_state.rs`, +`verbs.rs`, and `substrate_lib_telemetry.rs`, plus `lib.rs` as the crate +root. Rust has no `CaseIterable`, so the adjective-vocabulary values that +the Swift leg derives from enum cases are instead typed constant slices in +`audit_gate.rs`, kept in step with the Swift enums by convention and +checked by the same cross-layer parity tests. The two legs share +conformance fixtures in `rust/tests/` and +`Tests/SubstrateLibConformanceTests/` — recorded input-output pairs for +the audit-log fold, bitmap field constants, the SimHash-derived float +comparisons, and the wire format — that both implementations must +reproduce byte for byte. When you change either leg, run both test suites; +the fixtures are the contract. diff --git a/packages/libs/SubstrateLib/docs/OVERVIEW.md b/packages/libs/SubstrateLib/docs/OVERVIEW.md new file mode 100644 index 0000000..28636c4 --- /dev/null +++ b/packages/libs/SubstrateLib/docs/OVERVIEW.md @@ -0,0 +1,181 @@ +--- +doc: OVERVIEW +package: SubstrateLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateLib/AuditGate.swift + blob: 1f5ea748a82d21756518ed2dc4dda9e0a2ceed4a + - path: Sources/SubstrateLib/KeyedCommitment.swift + blob: 2b93f766914a2261cc4d63067469a0609157665f + - path: Sources/SubstrateLib/KeyedCommitmentAudit.swift + blob: afdac27147578c4b0cd2324e3db861812f89097a + - path: Sources/SubstrateLib/MerkleHash.swift + blob: 2a7e1b957d0f38633ee9048108fb9b59f8ee5648 + - path: Sources/SubstrateLib/RowStateAutomaton.swift + blob: b3111584e1495a41d66e51de620e27e55f31d4cd + - path: Sources/SubstrateLib/SubstrateLibTelemetry.swift + blob: b76dd8280311aeb67d8f5499108a0c0b080dc787 + - path: Sources/SubstrateLib/Verbs.swift + blob: b3f6b0db6b099d6932f7e57c017637193e2a4be6 +--- + +# SubstrateLib Overview + +## What This Library Does + +SubstrateLib is the orchestration layer of MOOTx01's storage substrate — the +part that decides what a legal change to a memory looks like, and refuses +every other kind. MOOTx01 is an on-device AI memory system. It stores what +an AI observes over time and helps the AI recall it later. Each stored +memory is one row: an entry with a lifecycle state, a link into the +classification lattice built by LatticeLib, and three 64-bit integers +called bitmaps that pack many small status fields — sensitivity, trust, +exportability, and more — into a compact wire form. + +SubstrateLib supplies three things a row needs to stay trustworthy. First, +a single write gate, `AuditGate.admit`, that every bitmap mutation must +pass through. Second, a row-state automaton that says which of ten +lifecycle states a row may hold and which state changes are legal. Third, +a reference implementation of the nine verbs — the fixed vocabulary of +actions a row can undergo, such as capture, mutate, and withdraw — against +which production storage kits can check their own faster implementations. + +Two supporting pipelines round out the package. A Merkle hash pipeline +computes deterministic content hashes so the system can prove a memory's +content is unchanged without re-reading the whole memory store. A +keyed-commitment pipeline proves that a piece of content once existed, +after the content itself has been destroyed to satisfy a deletion request. + +The value types SubstrateLib operates on — `Row`, `RowState`, `NounType`, +`LatticeAnchor`, `AuditEvent` — live in the sibling package SubstrateTypes. +SubstrateLib imports them and adds the logic: the rules, the gate, and the +verbs. + +## The Problem It Solves + +MOOTx01 estates can federate, which means separate devices exchange and +merge memories. Merging only works cleanly if two devices that record the +same logical event compute the same event identifier. SubstrateLib gives +every audit event a deterministic content identifier: a SHA-256 hash of +the event's stable fields, folded into a UUID, rather than a random one. +Two replicas that reach the same write independently produce the same +identifier, so a grow-only set — a data structure that can gain entries +but never lose them, often called a G-Set — simply keeps one copy. A +random identifier would defeat this: the same logical event would look +like two different events on two devices, and federation would double-count +it. + +A second problem is structural corruption. A row's status lives in three +64-bit integers, with several unrelated fields packed into each one at +fixed bit positions. Nothing at the type level stops a careless write from +touching the wrong bits, writing a value wider than its field, or moving a +row into a combination the design forbids — for example, marking a secret +memory as exportable. SubstrateLib closes this gap with one gate that every +write must pass. A write either lands as a legal value in a declared field, +producing a legal state and a legal combination of fields, or the gate +refuses it. Corruption of this kind becomes structurally unrepresentable, +not merely discouraged by convention. + +A third problem is privacy against accountability. When a memory is +expunged, MOOTx01 destroys its content for good — a legal-compliance +requirement. But an audit trail that simply drops the event looks the same +whether nothing happened or something was hidden; from the outside, an +absence proves nothing. SubstrateLib's keyed-commitment pipeline resolves +this: it commits, cryptographically, to the fact that a payload existed, +using a secret key the estate holds, without retaining anything the +payload could be reconstructed from. + +The mathematics in this package is conformance-gated. A Rust port in +`rust/` mirrors every algorithm, and shared test fixtures require both +legs to produce byte-identical output, so a device running the Swift leg +and a device running the Rust leg agree on every hash, every content +identifier, and every gate decision. + +## How It Works + +The write gate and the verb reference sit side by side, both built on the +same row-state automaton, but they serve different callers. + +`AuditGate.admit` is the gate production kits call directly. A caller +supplies the row's identity, the prior snapshot of its bitmaps, and only +the field values it owns. The gate looks up each written field in the +instance's admitted vocabulary — the substrate's universal fields (state, +sensitivity, exportability, trust) plus whatever fields the calling kit +registered for itself — and rejects any field that is undeclared or any +value that is out of range or not in the field's legal set. It then reads +the prior bitmaps, writes only the addressed bits, and leaves everything +else untouched: a read-modify-write, never a blind overwrite. Before +returning, it re-derives the encoded state and checks that state against +what the supplied verb was allowed to produce, and checks the whole result +against the automaton's forbidden-combination rules. Only then does it +compute the deterministic content identifier and emit one canonical audit +event. + +The row-state automaton itself is a fixed transition table: for each +(state, verb) pair, either a next state is defined, or the transition is +illegal and the automaton says so. Layered on top of the table is a second +check, the forbidden-combination rules, because a transition can be legal +in isolation yet still produce a field combination the design never +allows — a secret memory that is also exportable, for instance. Both +checks together are what "legal state" means in this package. + +`Substrate`, the reference verb implementation, is a second, independent +route to the same guarantees. It is an in-memory struct that implements +all nine verbs end to end: it validates a mutation against the automaton +directly, applies the bitmap change, updates two summary structures called +matrices that track how many rows carry each field value, and appends an +audit event, all inside one call. Production storage engines — a SQLite +tail, a memory-mapped bit-slice tensor — reimplement these same nine verbs +against real persistence; `Substrate` is the scalar oracle they are tested +against, not itself the production path. + +The two supporting pipelines are simpler and pure. The Merkle hash +pipeline builds a fixed byte encoding of a memory's content and vector +embeddings, tags it with a domain byte so a leaf hash can never be +mistaken for an interior or tombstone hash, and runs it through SHA-256. +The keyed-commitment pipeline reuses that same byte encoding with a +different domain tag, then runs it through a keyed HMAC instead of a +plain hash, binding the commitment to a secret only the estate holds. + +Every one of these code paths — the gate, the verbs, both hash pipelines — +can report activity to an injected telemetry sink, but none of them ever +reads a clock or changes its output because monitoring is on. Timestamps +arrive as a caller-supplied parameter, and when the sink is disabled, the +entire telemetry call collapses to one lock-free flag check. + +## How the Pieces Fit + +Figure 1 shows the library's topology — its major parts and how a write +moves through them. + +![Figure 1. Topology of SubstrateLib](topology.svg) + +*Figure 1. Topology of SubstrateLib. Two independent write paths — the +production `AuditGate` and the reference `Substrate` verbs — both consult +the shared row-state automaton. A separate, unrelated pipeline computes +content hashes and keyed commitments for the integrity and privacy +guarantees. Both paths report to the telemetry sink, shown as a dashed +external region because it lives in the sibling package IntellectusLib.* + +`AuditGate` and `Substrate` do not call one another. They are two +consumers of the same rules, not a pipeline where one feeds the other. A +kit such as LocusKit calls `AuditGate.admit` directly on every bitmap +write; `Substrate` exists so that alternative or future storage engines +have a known-correct in-memory reference to test against. Keeping them +independent, rather than layering one on the other, is deliberate: it +guarantees that a bug in the fast path cannot hide behind a passing +reference-path test, because the two paths share only the rules, not the +code. + +## What Ships in the Package + +The package ships the seven Swift sources listed above and the Rust port +in `rust/`, which mirrors each of them file for file. It ships no pinned +data artifacts of its own — no bundled JSON, no trained models — because +SubstrateLib's guarantees are entirely computational: a fixed transition +table, a fixed byte encoding, and a cryptographic hash, all deterministic +by construction rather than by shipped reference data. Conformance +fixtures live in `Tests/SubstrateLibConformanceTests/` and +`rust/tests/`, and both suites must pass before either leg changes ship. diff --git a/packages/libs/SubstrateLib/docs/topology.svg b/packages/libs/SubstrateLib/docs/topology.svg new file mode 100644 index 0000000..8ba0278 --- /dev/null +++ b/packages/libs/SubstrateLib/docs/topology.svg @@ -0,0 +1,99 @@ + + + + + + + + + + + + + SubstrateLib: two write paths, one automaton + + + + AuditGate + admit() — production write gate + + + Substrate + nine verbs — reference impl + + + + RowStateAutomaton + transitions + ForbiddenCombinations + + + + + consults + consults + + + + Content-integrity pipeline (independent of AuditGate and Substrate) + + + MerkleHash + leaf · interior · tombstone + + + KeyedCommitment + HMAC over shared leaf bytes + + + CommitmentAuditLog + G-Set, expunge provenance + + + shared encoding + + records + + + + Telemetry sink (external — IntellectusLib) + + + Intellectus.report + off-path: one atomic load, no alloc + + + + + + diff --git a/packages/libs/SubstrateML/docs/AGENT_MAP.md b/packages/libs/SubstrateML/docs/AGENT_MAP.md new file mode 100644 index 0000000..41753bb --- /dev/null +++ b/packages/libs/SubstrateML/docs/AGENT_MAP.md @@ -0,0 +1,179 @@ +--- +doc: AGENT_MAP +package: SubstrateML +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateML/ActionOutcomeMatrix.swift + blob: 612ee840126c72dd0d07505147ba2991ac3bb0b9 + - path: Sources/SubstrateML/AnomalyDetection.swift + blob: ff7d227353ebfc69cf01b5c468b2612d307b8632 + - path: Sources/SubstrateML/AprioriMining.swift + blob: 189a4408f3c6e600a538a6cfc560d24688cbd2af + - path: Sources/SubstrateML/AssociationRuleMining.swift + blob: 9f20faa808883c065412face0063c7682eabbfd6 + - path: Sources/SubstrateML/AuditLogFold.swift + blob: 7423db7764c628cf5fd2aab0e9bb2d98d8c687bd + - path: Sources/SubstrateML/BradleyTerry.swift + blob: 63d335ee50ccee803f0508eb6f3135a29b6545db + - path: Sources/SubstrateML/CommunityDetection.swift + blob: bfba8e0255baeb66d4c79ccfaf34cd11e21a5ebd + - path: Sources/SubstrateML/CompositeDistance.swift + blob: a5f94c631d89c92d45b60e9da43be7fe6bdd19a2 + - path: Sources/SubstrateML/ConceptImplications.swift + blob: 1ac6ccbb70665532f135d2813f260cbbdad21716 + - path: Sources/SubstrateML/DeltaFeatureExtractor.swift + blob: 1ca396968e5e565096dbebb059b45b8fc3e5719f + - path: Sources/SubstrateML/DistillationPipeline.swift + blob: 82ce8bfaaa2e7a92077510ef239a6997a585c888 + - path: Sources/SubstrateML/DistillationScorer.swift + blob: 2cdc6dfe121080d432859d17487f2c8d15fd2910 + - path: Sources/SubstrateML/DPORReduction.swift + blob: 3d3c79e8ce7694b8308bb1c970e86a24de0446b0 + - path: Sources/SubstrateML/EigenvalueCentrality.swift + blob: f427430c34add289231520ba19ae4781316cbd13 + - path: Sources/SubstrateML/FeatureExtractors.swift + blob: 7e6d388a588ff7bec04be631948387a086a253ed + - path: Sources/SubstrateML/FFT.swift + blob: fdfaceeb902428149ead96107ed69c308bca0fff + - path: Sources/SubstrateML/FloatSimHash.swift + blob: 39cd2c4c36492c7cbaf55d6d22a2f32e273cfbae + - path: Sources/SubstrateML/FormalConceptAnalysis.swift + blob: 9cdf9613b30bf2e3a9a85088489c0e2b683a8a77 + - path: Sources/SubstrateML/InformationTheory.swift + blob: f7ba2a27904ceb19eef0fb5b06d1c5aad8b3894b + - path: Sources/SubstrateML/JacobiSVD.swift + blob: fcd924ff0a8409f224b56e7229f59659a6b5f51e + - path: Sources/SubstrateML/LatticeDistance.swift + blob: a7b96a376dfba0c815e4dc91962da1ccaeb2ac1d + - path: Sources/SubstrateML/LLMCalibrationCurve.swift + blob: 4e01882a65570ccce72e2789c23e95c46b6399a7 + - path: Sources/SubstrateML/MatrixDecay.swift + blob: b9595064a54b79f8ddee343385af4b8ebdc7f3e5 + - path: Sources/SubstrateML/MomentSummary.swift + blob: e8c07841c8c3440deb0375500ebcd35a6f116bb4 + - path: Sources/SubstrateML/NMFAlternatingLeastSquares.swift + blob: d64aadc8db165cc3b9f51b4fb1c644a6d6a4ca3c + - path: Sources/SubstrateML/NMFDoubleFrobeniusSquared.swift + blob: 0a725d72b70120828cb5911de594b0278ca450e5 + - path: Sources/SubstrateML/PairingHandshake.swift + blob: cb608882478ec57a6e6ba1f7728646ff3ea610f1 + - path: Sources/SubstrateML/PartialStateRecall.swift + blob: ad7b42bc25f9c4f283d75593d2c42cd546ccf074 + - path: Sources/SubstrateML/RandomWalks.swift + blob: 99f40b1685a876d81f84ca38b4857e5c6ba4e1dd + - path: Sources/SubstrateML/RowAttributeView.swift + blob: f63cf544a87adde435bede527531d1a9414dad03 + - path: Sources/SubstrateML/Sampling.swift + blob: 84fcdcade559229003688a7fc151f5c0a6dac6e6 + - path: Sources/SubstrateML/ShingleSimilarity.swift + blob: bf9dbaa5b0d5a5ee4ec5dfe42a5d65c85bdd0326 + - path: Sources/SubstrateML/TemporalCausalityFold.swift + blob: d99d7358893eeae0a442e0bd4e52f8a9b1cd7f5e + - path: Sources/SubstrateML/TemporalCompression.swift + blob: f34dde022faadd40058e35b62da12c8985763e59 + - path: Sources/SubstrateML/TierAscendingQuery.swift + blob: 8681cbdae9af7facfe7792392dcee15c8f8fe0aa + - path: Sources/SubstrateML/TierContributionFingerprint.swift + blob: bb56b7a82532f8c324168bd81edff6bf68fb33e9 + - path: Sources/SubstrateML/TypedDecayWeighting.swift + blob: 82412f392848c378b5b28221bbddade9f03bf1bf + - path: Sources/SubstrateML/VizGraphSignals.swift + blob: 4f90110f2a19cb92f3d45baf3bc4d82ac9c3dc65 +--- + +# AGENT_MAP — SubstrateML + +PURPOSE: layer-3 cold-path/dreaming algorithm library of the MOOTx01 substrate — learning (decay, Bradley-Terry, calibration, NMF), graph analytics (Louvain, centrality, walks, SVD, FFT, anomaly), pattern mining (association rules, Apriori, FCA, D-G implications, temporal causality), distillation (5-stage cluster→factoid), fingerprint/distance math (FloatSimHash, composite/lattice/partial distances, moment/temporal summaries), and federation privacy math (pairing, tier contribution, tier query, DP OR-reduce). Pure functions + value types only; no storage, no clocks, no hidden state. + +DEPS: imports SubstrateTypes (Fingerprint256, HLC, RowId, MatrixO/F/C, SplitMix64, RecallResult…), SubstrateKernel (PortableKernel dispatch: float SimHash projection, orReduce256), IntellectusLib (telemetry; off-path = one atomic-bool load), Foundation. Imported by (per Package.swift/README): LocusKit, CognitionKit, GeniusLocusKit, NeuronKit, dreaming-daemon paths; LatticeLib (moot-semantics) uses EigenvalueCentrality for LexRank; CorpusKit LsaProvider uses JacobiSVD. Within this repo only tests + SubstrateLib's temporary `@_exported` re-export reference it (four-package split mid-migration). Rust port `rust/` = crate `substrate-ml` v1.0.0-skeleton, 38 modules 1:1 with Swift files; conformance tests rust/tests/{distillation_conformance,float_simhash_kernel_equivalence,viz_graph_signals_tests}.rs + shared JSON vectors in the engineering test harness. + +ENTRY POINTS (no single facade; per-family): +- DistillationPipeline.swift:277 `DistillationPipeline.run(input:extractFeatures:intraItem:) -> DistillationOutput` — full 5-stage distillation +- CommunityDetection.swift:152 `CommunityDetection.detectFull(adjacency:maxLevels:maxPasses:resolution:estate:ts:) -> [Int]` — multi-level Louvain +- EigenvalueCentrality.swift:87 `EigenvalueCentrality.compute(adjacency:maxIterations:tolerance:estate:ts:) -> [Double]` — keystone scores +- NMFAlternatingLeastSquares.swift:81 `factorize(V:rank:...) -> NMFFactorization` — latent themes +- FloatSimHash.swift:42 `FloatSimHash.project(vector:seed:) -> Fingerprint256` — embedding → fingerprint +- CompositeDistance.swift:50 `CompositeDistance.distance(latticeDistance:fingerprintHammingDistance:...)` — primary recall metric +- AuditLogFold.swift:101 `AuditLogFold.projectAll(events:asOf:nounTypeFor:)` — CRDT state reconstruction +- AprioriMining.swift:167 `AprioriMining.mine(rows:thresholds:) -> [AprioriRule]` — multi-antecedent rules + +## Symbol Table + +### Telemetry names — VizGraphSignals.swift +- :65 `enum VizGraphSignals` — canonical metric-name constants; one authoritative file prevents orphaned metrics +- :69 `communityAssignment = "community.assignment"` · :73 `centralityScore = "centrality.score"` · :77 `nmfFactor = "nmf.factor"` · :82 `anomalyFlag = "anomaly.flag"` · :86 `edgeDecayedWeight = "edge.decayed_weight"` + +### Fingerprint + distance math +- FloatSimHash.swift:30 `enum FloatSimHash`; :42 `project(vector:seed:) -> Fingerprint256` — 256-hyperplane signed projection; empty vector → zero fp; :76 `planes(seed:dim:) -> FloatSimHashPlanes` — deterministic ±1 plane set, cacheable per (seed,dim); :61 private cached kernel (`PortableKernel.kernelForCurrentPlatform()`, selected once) +- CompositeDistance.swift:37 `defaultAlphaLattice = 0.5` / :38 `defaultAlphaFingerprint = 0.5` / :40 `fingerprintTotalBits = 256`; :50 `distance(latticeDistance:fingerprintHammingDistance:alphaLattice:alphaFingerprint:compatibleSeedScope:) -> Double` — preconditions (not clamps); scope-incompatible drops fp term WITHOUT renormalizing (intended) +- LatticeDistance.swift:45 `struct LatticeAnchorStr` (udc + qid; 0 = null); :59 `enum UDCTreeDistance` (:63 `longestCommonPrefixLength`, :83 `distance` clamped [0,1]); :104 `protocol WikidataAdjacencyProvider`; :110 `enum WikidataGraphDistance` (:112 `maxDepth = 4`, :113 `normalizationScale = 3.0`, :118 `shortestPathLength` bounded BFS, :144 `distance` = 1-exp(-len/3), unreachable/null → 1.0); :161 `enum LatticeDistance` (:163/:164 default alphas 0.5, :185 combined `distance(_:_:provider:alphaUDC:alphaQID:)`, :172/:180 legacy hashed-anchor overloads — 0/1 only, compatibility) +- PartialStateRecall.swift:58 `enum PartialStateRecall`; :75 `score(rowFingerprint:anchor:matchBlocks:differBlocks:) -> Double` — matchScore × differScore, blocks ∈ {0,1,2,3} precondition-enforced, empty set → 0; :104 `topK(anchor:rows:matchBlocks:differBlocks:k:)` — linear scan reference; :128 `hammingBlocks(_:_:blocks:)` — block-restricted Hamming +- ShingleSimilarity.swift:53 `enum ShingleSimilarity`; :58 `windowSize = 3`; :69 `shingles(_:) -> Set` — lowercase 3-char window; <3 chars → whole-string shingle; empty → empty set; :96 `similarity(_:_:) -> Float32` — Jaccard; both-empty → 0.0 +- MomentSummary.swift:54 `struct RowLite` (fingerprint + captureHLC, 40 bytes); :70 `summarize(rows:[Row]...)` / :89 `summarize(rows:[RowLite]...)` — OR-reduce of rows matching caller predicate; byte-identical, RowLite ~4-5× faster (conformance CRC 0x6762440b); :100 `capturedDuring(_:_:)` convenience predicate; :106 `orReduce(_:)` raw reduction +- TemporalCompression.swift:30 `enum WindowLevel` hour→year (:43 `nextCoarser`, :48 `approxSeconds` — deliberately approximate bucket divisors); :60 `struct TemporalWindow` (:80 `empty(level:)` identity); :93 `compress(rows:startHLC:endHLC:level:)`; :110 `rollup(windows:to:)` — OR fp union, `&+` counts, min/max bounds; :135 `cascadeRollup(hourWindows:upTo:)` — epoch-seconds integer-division bucketing per level + +### Ingestion shaping +- FeatureExtractors.swift:41 `enum StreamSourceFlag: UInt8`; :50 `struct AmbientSampleRow`; five sample+extractor pairs, each `extract(_:hlc:rowId:) -> AmbientSampleRow`: :82/:102/:109 HealthKit, :136/:158/:165 CoreLocation (+ :190 `quantizeGeohash` internal), :200/:223/:230 EventKit (attendees sorted before hashing), :257/:279/:286 ScreenTime, :308/:333/:340 SystemTelemetry; :363 internal `encodePayload` key-sorted deterministic binary. Pinned UDC anchors: 613.71 / 914 / 65.012.4 / 004.5 / 004.2. Only HealthKit rows carry payload +- AuditLogFold.swift:44 `struct ProjectedRowState` (no fingerprint — recompute from bitmaps+anchor); :79 `projectCurrentState(rowId:nounType:events:)`; :90 `projectStateAt(...asOf:)`; :101 `projectAll(events:asOf:nounTypeFor:)`; :131 private `foldOrdered` — HLC-sorted sequential fold; tombstone stateRaw == 33 (adjectiveBitmap & 0x3F) is STICKY (I-22); fold is permutation-invariant over the event set (I-21 sync convergence) +- RowAttributeView.swift:51 `enum RowAuditValue` (.bitmap/.integer/.null); :70 `struct RowAuditEntry`; :111 `struct RowAttributeView` (:130 init sorts attributes, :140/:151 manual Eq/Hash); :189 `from(auditEntries:) -> [RowAttributeView]` — vocab sorted+capped 64 (6-bit field), latest-write-wins per fieldPath by HLC, bitmap → one attr per set bit, integer → low byte, null → nothing, empty rows dropped; vocab is EPHEMERAL per call (pre-merge batches for stable indices) + +### Learning + decay +- MatrixDecay.swift:39 `struct DecayingMatrix` (values + halfLifeSeconds + lastDecayTimeSeconds; :60 subscript); :84 `MatrixDecay.apply(to:nowSeconds:estate:ts:)` — factor = exp(-dt·ln2/τ), dt ≤ 0 no-op but still emits factor 1.0; saturating Int64 cast for telemetry tag; :148 `decayFactor(elapsedSeconds:halfLifeSeconds:)` pure; :157 `decayAndAdd(...)` decay-then-add; :177/:183/:189 `applyExponentialDecay(to: MatrixF/O/C ...)` — intentional NO-OP reference adapters (real logic in production kit); :199 `Double.ln2 = 0.6931471805599453` (local, matches Rust f64::LN_2); :208 `enum DecayHalfLives` — F 90d, C 180d, O 60d, T 30d, ActionOutcomes 365d, calibration 730d, W_ranking 90d (illustrative; manifest is authoritative) +- ActionOutcomeMatrix.swift:37 `struct ActionOutcomeKey` — 6-bit preconditions (o07/o08), :51 `packed: UInt16`, :55 Comparable; :60 `struct ActionOutcomeCell` (:75 `successRate`, :83 `wilsonLowerBound` 95%, z=1.96); :95 `struct ActionOutcomeMatrix` (:101 `observe(action:outcome:success:at:)` `&+=`, :113 `successRate`, :119 `observationCount`, :139 `topActions(forOutcome:k:minObservations:)` — ranked by Wilson LB, ties count desc then action asc, returns rate+LB+count together, :158 `populatedCellCount`) +- BradleyTerry.swift:31 `struct PreferenceObservation` (winnerID/losers/weight); :53 `struct BradleyTerryEstimator` (`theta: [UUID: Double]` private(set)); :67 `init(learningRate: 0.05, l2: 0.001, theta: [:])`; :78 `observe(_:)` — SGD in log-space, multi-loser SEQUENTIAL against running theta (pinned parity contract); :105 `observeBatch(_:)`; :113 `strength(of:)` = exp(theta); :120 `probability(_:beats:)` = sigmoid(Δtheta). No RNG; warm-restartable; ranking signal only, never truth +- LLMCalibrationCurve.swift:37 `struct LLMCalibrationCurve`; :38 `binCount = 20` / :39 `binWidth = 0.05`; :50 `observe(claimedConfidence:actualOutcome:)` — clamp [0, 0.999999], wrapping counters; :62 `actualRate(in:)`; :72 `midpoint(of:)`; :78 `expectedCalibrationError()`; :92 `brierScore()`; :107 `decay(factor:)` fractional (730d half-life policy, daemon-driven) +- Sampling.swift:57 `enum Sampling`; :75 `sampleNormal(rng: inout SplitMix64)` — Box-Muller COSINE branch only, exactly 2 draws/call, u1 floored at leastNormalMagnitude; :102 `sampleGamma(shape:rng:)` — Marsaglia-Tsang (squeeze 0.0331), shape<1 via Ahrens-Dieter with extra draw BEFORE recursion (pinned order); :157 `sampleBeta(alpha:beta:rng:)` — g1/(g1+g2), alpha-Gamma drawn first, 0.5 degenerate fallback. Conformance vector sampling.json; relies on libm bit-identity + +### Graph analytics +- CommunityDetection.swift:80 `typealias Adjacency = [[(neighbor: Int, weight: Double)]]`; :100 `detect(adjacency:maxPasses:estate:ts:)` — Phase 1 only; :152 `detectFull(adjacency:maxLevels:maxPasses:resolution:estate:ts:)` — full Louvain; resolution gamma default 1.0 = classical (IEEE 1.0*x == x, legacy-safe), NeuronKit callers pass 0.05 (pair-lock escape); :368 `canonicalize(_:)` — first-appearance renumbering; private: :202 totalEdgeWeight (zero threshold 1.0e-30 → identity labels, NO emit), :214 detectCore (gain ties → lowest label asc), :317 condense, :345 emit. Defaults maxPasses 10, maxLevels 10; exactly ONE `community.assignment` emit per call +- EigenvalueCentrality.swift:60 `typealias Adjacency`; :62 `defaultMaxIterations = 100` / :63 `defaultTolerance = 1e-6`; :87 `compute(...) -> [Double]` — AUTHORITY via Aᵀx (`xNext[j] += w*x[i]` for edge i→j; design-council 2026-06-04, conformance-locked; symmetrize for undirected), Perron shift 1.0 per iteration (breaks bipartite ±λ oscillation), zero-norm (<1e-30) → uniform 1/√n fallback, all 3 return paths emit `centrality.score` (value 1.0) +- RandomWalks.swift:36 `typealias Adjacency`; :38 `defaultRestartProb = 0.15`; :43 `walk(adjacency:start:length:restartProb:seed:) -> [Int]` — hard preconditions on kernel validity (indices, finite non-negative weights); walk[0] == start; dead end = implicit restart; :99 `sampleWeighted(_:rng:)` — roulette wheel; total ≤ 0 → uniform; rounding shortfall → last neighbor; :127 `uniform01(_:)` — top 53 bits, Rust bit-match; :141 `walkWithRestart(seed:steps:restartProbability:rngSeed:adjacency:) -> [RowId: Int]` — RowId-space uniform-choice visit counts (recall_exploratory); :172 `struct SplitMix64` (public; canonical constants 0x9E3779B97F4A7C15 / 0xBF58476D1CE4E5B9 / 0x94D049BB133111EB) +- NMFAlternatingLeastSquares.swift:45 `struct NMFFactorization` (W m×k, H k×n, rank, iterations, finalError RMS); :81 `factorize(V:rank:maxIterations:100:tolerance:1e-4:seed:0xDEADBEEFCAFEBABE:estate:ts:)` — Lee-Seung multiplicative updates, eps 1e-9, init = SplitMix64 16-bit draws /0xFFFF, preconditions rectangular/finite/non-negative; flat-buffer unsafe-pointer hot loops (loop nest + reduction order preserved EXACTLY; conformance CRC 0x300bf633; ~100× perf vs bounds-checked); emits `nmf.factor` = finalError; :220 `reconstructionError(V:W:H:)` public RMS; internal flat/nested matMul helpers :247/:288/:309/:329/:353/:370/:388 +- NMFDoubleFrobeniusSquared.swift:54 `struct NMFDoubleFrobeniusSquaredFactorization` (:93 `loadings(forRow:)`); :113 `enum NMFDoubleFrobeniusSquared` — :116 `defaultMaxIterations = 100`, :122 `defaultTolerance = 1e-6`, eps 1e-9; :153 `factorize(o:rows:cols:rank:seed:0xC0FFEE_BABE_BEEF:...)` — f64 scalar, RAW Frobenius² convergence (not RMS), init floored at 1e-3; :302 `struct NMFDoubleFrobeniusSquaredRNG` (SplitMix64 core, 53-bit output, floor 1e-3). PRODUCTION GATE: no consumer may wire to this pending substrate_math_performance benchmark (ruling 2026-06-13); still conformance-gated vs Rust (nmf_double_frobenius_squared.json) +- AnomalyDetection.swift:26 `enum AnomalyDetection`; :31 `zScore(value:mean:stddev:)` (stddev ≤ 0 → 0); :51 `rollingZScore(window:current:estate:ts:)` — emits `anomaly.flag`; :90 `modifiedZScore(value:median:mad:)` — 0.6745 consistency constant, mad ≤ 0 → 0; :110 `rollingModifiedZScore(...)` — IN-PLACE sort (5× perf, ~400KB saved at 100k; CRC 0x6c6fda4d; must match Rust sort_by partial_cmp for non-NaN); :152 `isAnomalous(zScore:threshold: 3.0)` +- JacobiSVD.swift:74 `struct SVDResult` (U, singularValues non-increasing, Vt, rank); :129 `decompose(A:rank:sweeps:) -> SVDResult` — one-sided cyclic Jacobi; sweeps default 30 PINNED (fixed count, not convergence-tested; MUST match across ports); eps 1e-9 skip/zero cutoff; scalar Float32 only, NO SIMD/FMA/Accelerate; sign convention: largest-|entry| of each left vector forced positive; preconditions m ≥ n, rectangular, non-empty; unsafe-buffer sweep loops (perf, indices loop-derived); :368 `jacobiCS(alpha:beta:gamma:)` — pinned expression tree (ζ,t,c,s), DO NOT refactor +- FFT.swift:36 `struct Complex` (:51 `magnitude`, :58 `magnitudeSquared`, :63/:68/:73 +,-,*); :83 `enum FFT` — :99 `forward(real:) -> [Complex]` Cooley-Tukey radix-2 DIT, length MUST be power of two (constitutional; callers zero-pad), :162 `magnitudeSpectrum(real:)`; :169/:180 `bitReverse`/`log2Floor` (@usableFromInline); :201 `struct RhythmResult` (dominantPeriodSeconds nil = "no rhythm", honest); :218 `enum RhythmAnalysis` — :229 `analyze(series:bucketDurationSeconds:)` scans bins 1...N/2, energy excludes DC; :281 `analyze(fingerprints:block:bitPosition:bucketDurationSeconds:)` — bit-series extraction, block 0..3 / bit 0..63 preconditions. Production vDSP path must be BIT-IDENTICAL to this scalar reference +- InformationTheory.swift:38 `entropy(_:)`; :48 `mutualInformation(joint:)`; :75 `klDivergence(_:_:)` — length precondition; skips p>0,q=0 terms ⇒ result is a LOWER BOUND; :88 `crossEntropy(_:_:)`; :99 `jensenShannon(_:_:)` bounded [0,1]; :112 `normalizedMutualInformation(joint:)` — ragged matrix → 0 sentinel. All log2 (bits); 0·log0 = 0 by term-skipping; distribution validity is caller's contract + +### Pattern mining +- AssociationRuleMining.swift:52 `struct Item` (field/value UInt8; :64 `packed: UInt16` = field<<8|value; :68 Comparable); :76 `struct AssociationRule` (single-Item antecedent + 5 metrics); :107 `struct MiningThresholds` (minSupport/minConfidence); :136 `mineAssociationRules(matrix: MatrixO, activeRowCount:thresholds:)` — N injected (matrix has no row count); activeRowCount ≤ 0 → []; :154/:164 internal `AssociationRuleEngine.mine` — two passes: diagonal = single supports, off-diagonal in canonical packed-key order (NO final sort needed — conformance-relied ordering); self-rules skipped; conviction = +inf at confidence 1; k>2 → AprioriMining +- AprioriMining.swift:55 `struct AprioriThresholds` (minSupport/minConfidence/minLift ≥ 1.0 default/maxK default 3, floored ≥ 2); :89 custom `init(from:)` routes through public init so serialized maxK < 2 cannot bypass the clamp; :115 `struct AprioriRule` (antecedent `[Item]` sorted by packed, + evidenceCount); :167 `AprioriMining.mine(rows:thresholds:)` — join on (k-2)-prefix, subset-count, prune; rule extraction iterates itemsets in lex order (dict order undefined); 4-key sort lift↓ confidence↓ evidence↓ lex↑; :310 `mineAprioriRules(rows:thresholds:)` free-function wrapper; private `HashableItemset`/`itemsetLess`/`aprioriJoin` (:325/:340/:366) +- FormalConceptAnalysis.swift:41 `struct FormalAttribute: Comparable` — lex (namespace,key,value) order = determinism backbone; :70 `enum SeedMode` .single/.multi; :90 `struct CoverDelta` / :117 `struct ConceptCoverDeltas` (:140 `covering(concepts:)` — cover edges ≠ implications; O(n²m) bounded); :210 `struct FormalConcept` (extent/intent/support/stability?); :247 `struct FormalContext` — :251 nested `RowID = UInt32` (avoids LocusKit RowID collision), :285 `from(rowAttributeViews:)` (namespace "row"), :298 `init(rows:)`, :334 `extent(of:)`, :343 `intent(of:)`, :354 `closure(of:)`; internal FCABitSet plumbing :366/:380/:386/:665; :415 `struct BoundedConceptMiner` (minSupport clamped ≥1, maxIntentSize, maxConcepts, seedMode, maxSeeds, stabilityBudget 0 default → stability nil, stabilitySeed 0xCAFEBABEDEADBEEF) — :475 `mine(context:)`: one closure per seed, dedup by intent, truncate; sort support↓ intent-size↑ lex; :595 `enum StabilityEstimator` (:616 `estimate(concept:context:budget:seed:)` — Bernoulli-half subsets, per-concept SplitMix64 seed = caller seed XOR FNV(canonical key)) +- ConceptImplications.swift:62 `struct Implication` (premise/conclusion; caller ensures disjoint); :96 `struct ConceptImplications` (implications + isTruncated); :148 `conceptImplications(over:context:maxImplications:maxPremiseSize:)` — Next-Closure pseudo-intent enumeration in increasing size order; maxImplications hard cap sets isTruncated (with look-ahead :358 for honesty); maxPremiseSize skips silently (soundness unaffected); output sort premise-size → lex premise → lex conclusion; empty context → empty untruncated +- TemporalCausalityFold.swift:61 `struct TemporalFieldCoord` (fieldPath + valueRepr stable string); :98 `struct TemporalAuditEntry` (hlc + fieldCoords); :121 `struct FoldResult` (deltas in first-insertion order + newWatermark); :139 `struct TemporalCausalityKey` (source/target/lagBucket); :174 `lagBuckets = [1,2,4,8,16,32,64,128]` — must mirror MatrixTier.lagBuckets exactly; :178 `defaultWindowMinutes = 256`; :191 `maxWindowOccupancy = 512` — bulk-import quadratic guard (2026-07-02 decision), keeps most-recent in-window sources; :202 `lagBucket(forMinutes:)` — smallest boundary ≥ delta, clamp 128; zero-ms → minute 1; :241 `fold(entries:windowMinutes:startWatermark:)` — entries MUST be pre-sorted by HLC (not re-sorted; silent corruption otherwise); hourly batch cadence supersedes weekly + +### Distillation +- TypedDecayWeighting.swift:32 `enum DistillationFeatureType: String` — ENT/REL/TMP/NUM; :40 `decayLambda` = 0.1/0.2/0.5/0.8 (pinned, diffusion-schedule analogy); :52 `enum TypedDecayWeighting`; :65 `weight(featureType:ageInUnits:)` = exp(-λ·max(0,age)); :86 `weightedDocFrequency(featureType:presenceTimestamps:allMemoryTimestamps:referenceDate:timeUnit: 86_400)` — decayed presence weight / total weight; empty or zero-total → 0 +- DeltaFeatureExtractor.swift:29 `enum DeltaType: String` STATIC/CONVERGENT/MONOTONE/OSCILLATING/DIVERGENT; :41 `struct DeltaAnalysis` (deltaType/terminalValue/convergenceScore/slope?/confidence); :86 `analyzeCategorical(sequence:decayLambda: 0.5)` — oscillation = period-2 over LAST 4 obs only (≥4 required); convergence = trailing-run fraction vs λ; :148 `analyzeNumerical(sequence:decayLambda: 0.8)` — diffs all-zero STATIC / one-sign MONOTONE (confidence = λ fixed) / strict alternation OSCILLATING / else DIVERGENT. Pure; zero RNG +- DistillationScorer.swift:16 `struct ExtractedFeature` (:41 init, display defaults to value); :58 `struct DistillationSNR` (readyToDistill iff snr ≥ 2.0); :71 `struct FeatureGraph` (components sorted by weighted df ↓ — dominant first); :94 internal `binaryEntropy(_:)`; :114 `structuralThreshold(M:)` = 2/M (M=1 → 2.0, correctly blocks); :120 `applyStructuralThreshold(features:M:)`; :145 `computeSNR(features:M:)` — episodic floor 1e-6; :174 `computeStructuralScores(features: inout)` σ = df·(1−H(df)); :194 `buildPMIGraph(thresholdFeatures:incidenceMatrix:M:)` — PMI log2 ONLY (Foundation.log2 ⇔ f32::log2 bit-match), positive-PMI edges, DFS components O(n²) ok ≤ ~150; :257 `selectDominantComponent(graph:)`; :277 `computeConfidence(selected:allThreshold:)` = meanDf × coherence ratio +- DistillationPipeline.swift:27 `struct DistillationInput` (memoryContents/timestamps?/clusterID/sourceIDs; computed M); :53 `struct DistillationOutput` (drawerContent/confidence/uncertain/snr/deltaType/succeeded/failureReason/featureFingerprint); :81 `struct DistilledHeader` (:102 `parse(_:)` — nil unless "[DIST|" prefix); :172 `typealias FeatureExtractor = @Sendable (String, DistillationFeatureType) -> [ExtractedFeature]`; :184 `featureSimHashSeed = 0x44495354494C4C41` ("DISTILLA") — conformance-critical, changing invalidates ALL stored fingerprints; :193 `featureHash(_:) -> Fingerprint256`; :221 `queryFingerprint(query:extractFeatures:)` — probe fp, no inference; :240 `defaultExtractor` (capitalization heuristic; tests/default); :277 `run(input:extractFeatures:intraItem: false)` — 5 stages; intraItem skips SNR gate + keeps ALL passing features (no PMI pruning of a single doc); ubiquity re-add at df ≥ (M-1)/M (zero-PMI spine rescue); confidence < 0.4 fail, [0.4,0.7) uncertain; `src=` = sourceIDs.count NOT M; deliberate monolithic body + +### Federation + privacy +- PairingHandshake.swift:37 `struct PairingNonce` — exactly 32 bytes (precondition); :48 `seedWith(estateA:estateB:)` — UUIDs ordered by RAW BYTES, never uuidString (ASCII hex order ≠ byte order; Rust compares [u8;16] — string order would derive incompatible families); :79 `struct PairingRecord` (:87 `isActive`); :114 `struct PairingAuditPayload`; :133 `enum PairingHandshake` — :145 `generateSharedFamily(nonce:estateA:estateB:density:)` (both sides derive identical family, no round trip), :156 `sharedFamilyKey(case:peerEstate:)` = "H_shared__", :168 `buildPairEvent(...)`, :179 `buildUnpairEvent(...)` (unpair RETAINS family for asOf queries); :192 private `combinedFamilyHash`; :209 fileprivate `lexLessOrEqual` raw-byte compare. FNV-1a 0xCBF29CE484222325 / 0x100000001B3 mixers +- TierContributionFingerprint.swift:45 `enum FederationCase: UInt32` household 1 / fleet 2 / industry 3; :51 `struct TierContribution`; :81 `build(estateUUID:case:shareableFingerprints:hlc:)` — kernel-dispatched orReduce256; :95 `encode(_:) -> Data` — 64-byte canonical wire: UUID(16) | case u32 BE | rowCount u32 BE | 4× fp block u64 BE | hlc.packed u64 BE — BE-UNIFORM (earlier LE-fp draft diverged from Rust by 32 bytes; conformance caught it); :117 `decode(_:)` — nil unless exactly 64 bytes + known case; :148/:154/:160/:166 endian helpers. No signing/checksum at this layer (egress signs; DP at aggregator) +- TierAscendingQuery.swift:35 `enum TargetTier` .peer/.fleetAggregate/.industryAggregate; :41 `struct TierAscendingQuery` (originatingEstate/primitiveName/primitiveInput/targetTier/privacyBudget/queryHLC); :61 `struct PeerResponse`; :68 `enum TierAscendingQueryProtocol` — :72 `computeLocal(query:dispatch:)` (injected closure; no CognitionKit dep), :80 `applyDPToContribution(_:budget:rngSeed:)` — Laplace scale 1/ε per score, CI = ±1.96·scale; NOTE Swift forwards `breakdown` unchanged while Rust zeroes it (side-channel; parity nuance), :100 `combine(local:peers:)` — per-RowId score sum, sort score↓ RowId↑, WIDEST CI wins (conservative); :133 `struct PrivacyLedger` (:141 `remaining`, :147 `canConsume`, :152 `consume`, :159 `dailyReset` — manual, no auto-expiry) +- DPORReduction.swift:31 `struct DPParameters` (:36 init — ε 1.0 > 0, δ 1e-9 ∈ [0,1) VALIDATED BUT UNUSED by Laplace (documented), k 3 ≥ 1); :48 `enum DPORReduction`; :54 `reduce(fingerprints:params:rngSeed:) -> Fingerprint256` — per-bit popcount + Laplace(1/ε) noise + k-anonymity threshold; empty → .zero; fresh SplitMix64(rngSeed) inside; :85 internal `laplaceNoise(scale:rng:)` — inverse CDF from top 53 bits (raw >> 11) + +## INVARIANTS / GOTCHAS + +- DETERMINISM IS THE CONTRACT. Swift and Rust legs must agree bit-for-bit on shared conformance vectors. Any change to seeds, draw orders, tie-breaks, loop-nest/reduction order, or pinned constants must be mirrored in `rust/src/` and pass both test suites. SplitMix64 is the only PRNG; all uniform doubles come from the top 53 bits. +- NO CLOCKS, NO STATE, NO I/O anywhere in the package. `ts`, `nowSeconds`, `referenceDate`, watermarks — all caller-supplied. Everything is Sendable value types or pure static functions. +- Pinned seeds: canonical conformance seed 0xCAFEBABEDEADBEEF (FCA stability, temporal-fold vectors); NMF default 0xDEADBEEFCAFEBABE; parked-NMF 0xC0FFEE_BABE_BEEF (historical, deliberate); distillation featureSimHashSeed 0x44495354494C4C41 — changing it orphans every stored distillation fingerprint. +- Pinned constants — do not change without conformance regen: JacobiSVD sweeps 30; FFT power-of-two input; Louvain zero-weight 1e-30 + gamma 1.0 legacy identity; centrality Perron shift 1.0 + tolerance 1e-6 + cap 100; restartProb 0.15; Marsaglia-Tsang 0.0331; anomaly threshold 3.0 + MAD 0.6745 (CRC 0x6c6fda4d); shingle window 3; lag buckets {1..128} + window 256 min + occupancy 512; SNR gate 2.0 + τ 2/M + confidence 0.4/0.7 + ubiquity (M-1)/M; decay λ ENT/REL/TMP/NUM 0.1/0.2/0.5/0.8; half-life table (90/180/60/30/365/730/90 d); calibration 20 × 0.05 bins; Wilson z 1.96; DP defaults ε 1.0, δ 1e-9, k 3; CompositeDistance 256 bits, alphas 0.5/0.5; UDC clamp, BFS depth 4, scale 3.0; NMF eps 1e-9 + tolerance 1e-4 (CRC 0x300bf633); MomentSummary CRC 0x6762440b. +- NMFDoubleFrobeniusSquared is PRODUCTION-GATED: no consumer may wire to it until the substrate_math_performance benchmark passes. Do not "clean it up" into use; do not delete it either (it is the benchmark baseline). +- EigenvalueCentrality is AUTHORITY (Aᵀx), locked by conformance vector. Do not flip to hub semantics. Symmetrize adjacency for undirected use. +- MatrixDecay is constitutional: decay only shrinks; the three typed applyExponentialDecay overloads are intentional no-ops at reference level. dt ≤ 0 is a no-op that still emits factor 1.0. +- AuditLogFold tombstone (stateRaw 33) is sticky forever; fold is event-set permutation-invariant — that IS the sync-convergence proof. Fingerprint deliberately absent from projections. +- TemporalCausalityFold requires pre-sorted input by HLC; violation silently corrupts (no crash). RowAttributeView vocab (≤64 fields) is per-call ephemeral — merge batches first for stable indices. +- PairingHandshake seed ordering compares raw UUID bytes, never uuidString — ASCII hex order can invert byte order and desync from Rust. Nonce exactly 32 bytes. +- TierContribution wire format is BE-uniform 64 bytes (fingerprint blocks BE, not the LE storage form). Unsigned at this layer by design. +- TierAscendingQuery parity nuance: Rust zeroes DistanceBreakdown in DP contributions; Swift forwards it unchanged. Flag before relying on either behavior. +- CompositeDistance with incompatible seed scope DROPS the fingerprint term without renormalizing — smaller distance is intended, not a bug. Preconditions, not clamps, on inputs. +- InformationTheory.klDivergence is a LOWER BOUND when q has zeros where p has mass. NMI returns 0 sentinel on ragged input. All log2. +- Apriori maxK floor 2 is enforced in init AND in custom Decodable (synthesized decode would bypass it). Association mining needs caller-injected N; matrix diagonal doubles as single-item support. +- Hard preconditions (crash, not error) on malformed inputs throughout: RandomWalks kernel validity, NMF domain (rectangular/finite/non-negative), SVD shape (m ≥ n), FFT length, block IDs {0..3}, 6-bit action/outcome keys, 32-byte nonce. +- Unsafe-buffer hot loops (NMF, SVD, moment/anomaly paths) exist for auto-vectorization; all indices are loop-derived, never data-derived. Preserve loop nest and reduction order exactly — Float32 lowest-bit rounding is conformance-visible. +- Telemetry: exactly five signals (VizGraphSignals); one emit per invocation; off-path = single atomic-bool load. Degenerate zero-weight Louvain emits nothing. Only AnomalyDetection/CommunityDetection/EigenvalueCentrality/NMF/MatrixDecay emit. +- Distillation `src=` field counts sourceIDs, not cluster members (differs in intra-item mode). intraItem skips the SNR gate and PMI pruning by design. diff --git a/packages/libs/SubstrateML/docs/DETAILS.md b/packages/libs/SubstrateML/docs/DETAILS.md new file mode 100644 index 0000000..822712f --- /dev/null +++ b/packages/libs/SubstrateML/docs/DETAILS.md @@ -0,0 +1,1122 @@ +--- +doc: DETAILS +package: SubstrateML +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateML/ActionOutcomeMatrix.swift + blob: 612ee840126c72dd0d07505147ba2991ac3bb0b9 + - path: Sources/SubstrateML/AnomalyDetection.swift + blob: ff7d227353ebfc69cf01b5c468b2612d307b8632 + - path: Sources/SubstrateML/AprioriMining.swift + blob: 189a4408f3c6e600a538a6cfc560d24688cbd2af + - path: Sources/SubstrateML/AssociationRuleMining.swift + blob: 9f20faa808883c065412face0063c7682eabbfd6 + - path: Sources/SubstrateML/AuditLogFold.swift + blob: 7423db7764c628cf5fd2aab0e9bb2d98d8c687bd + - path: Sources/SubstrateML/BradleyTerry.swift + blob: 63d335ee50ccee803f0508eb6f3135a29b6545db + - path: Sources/SubstrateML/CommunityDetection.swift + blob: bfba8e0255baeb66d4c79ccfaf34cd11e21a5ebd + - path: Sources/SubstrateML/CompositeDistance.swift + blob: a5f94c631d89c92d45b60e9da43be7fe6bdd19a2 + - path: Sources/SubstrateML/ConceptImplications.swift + blob: 1ac6ccbb70665532f135d2813f260cbbdad21716 + - path: Sources/SubstrateML/DeltaFeatureExtractor.swift + blob: 1ca396968e5e565096dbebb059b45b8fc3e5719f + - path: Sources/SubstrateML/DistillationPipeline.swift + blob: 82ce8bfaaa2e7a92077510ef239a6997a585c888 + - path: Sources/SubstrateML/DistillationScorer.swift + blob: 2cdc6dfe121080d432859d17487f2c8d15fd2910 + - path: Sources/SubstrateML/DPORReduction.swift + blob: 3d3c79e8ce7694b8308bb1c970e86a24de0446b0 + - path: Sources/SubstrateML/EigenvalueCentrality.swift + blob: f427430c34add289231520ba19ae4781316cbd13 + - path: Sources/SubstrateML/FeatureExtractors.swift + blob: 7e6d388a588ff7bec04be631948387a086a253ed + - path: Sources/SubstrateML/FFT.swift + blob: fdfaceeb902428149ead96107ed69c308bca0fff + - path: Sources/SubstrateML/FloatSimHash.swift + blob: 39cd2c4c36492c7cbaf55d6d22a2f32e273cfbae + - path: Sources/SubstrateML/FormalConceptAnalysis.swift + blob: 9cdf9613b30bf2e3a9a85088489c0e2b683a8a77 + - path: Sources/SubstrateML/InformationTheory.swift + blob: f7ba2a27904ceb19eef0fb5b06d1c5aad8b3894b + - path: Sources/SubstrateML/JacobiSVD.swift + blob: fcd924ff0a8409f224b56e7229f59659a6b5f51e + - path: Sources/SubstrateML/LatticeDistance.swift + blob: a7b96a376dfba0c815e4dc91962da1ccaeb2ac1d + - path: Sources/SubstrateML/LLMCalibrationCurve.swift + blob: 4e01882a65570ccce72e2789c23e95c46b6399a7 + - path: Sources/SubstrateML/MatrixDecay.swift + blob: b9595064a54b79f8ddee343385af4b8ebdc7f3e5 + - path: Sources/SubstrateML/MomentSummary.swift + blob: e8c07841c8c3440deb0375500ebcd35a6f116bb4 + - path: Sources/SubstrateML/NMFAlternatingLeastSquares.swift + blob: d64aadc8db165cc3b9f51b4fb1c644a6d6a4ca3c + - path: Sources/SubstrateML/NMFDoubleFrobeniusSquared.swift + blob: 0a725d72b70120828cb5911de594b0278ca450e5 + - path: Sources/SubstrateML/PairingHandshake.swift + blob: cb608882478ec57a6e6ba1f7728646ff3ea610f1 + - path: Sources/SubstrateML/PartialStateRecall.swift + blob: ad7b42bc25f9c4f283d75593d2c42cd546ccf074 + - path: Sources/SubstrateML/RandomWalks.swift + blob: 99f40b1685a876d81f84ca38b4857e5c6ba4e1dd + - path: Sources/SubstrateML/RowAttributeView.swift + blob: f63cf544a87adde435bede527531d1a9414dad03 + - path: Sources/SubstrateML/Sampling.swift + blob: 84fcdcade559229003688a7fc151f5c0a6dac6e6 + - path: Sources/SubstrateML/ShingleSimilarity.swift + blob: bf9dbaa5b0d5a5ee4ec5dfe42a5d65c85bdd0326 + - path: Sources/SubstrateML/TemporalCausalityFold.swift + blob: d99d7358893eeae0a442e0bd4e52f8a9b1cd7f5e + - path: Sources/SubstrateML/TemporalCompression.swift + blob: f34dde022faadd40058e35b62da12c8985763e59 + - path: Sources/SubstrateML/TierAscendingQuery.swift + blob: 8681cbdae9af7facfe7792392dcee15c8f8fe0aa + - path: Sources/SubstrateML/TierContributionFingerprint.swift + blob: bb56b7a82532f8c324168bd81edff6bf68fb33e9 + - path: Sources/SubstrateML/TypedDecayWeighting.swift + blob: 82412f392848c378b5b28221bbddade9f03bf1bf + - path: Sources/SubstrateML/VizGraphSignals.swift + blob: 4f90110f2a19cb92f3d45baf3bc4d82ac9c3dc65 +--- + +# SubstrateML Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear in working-group +order: the telemetry names, the fingerprint and distance math, the +ingestion shapers, the learning and decay machinery, the graph analytics, +the pattern miners, the distillation stages, and finally the federation +and privacy layer. + +## VizGraphSignals.swift + +This file provides the canonical metric names for the package's telemetry +layer: five string constants used as signal names when the five +graph-analytic algorithms report completion. + +Telemetry is optional observation data for the Topology visualization — a +live picture of the memory graph. Each algorithm emits exactly one metric +per invocation through `IntellectusLib`, and only when monitoring is +enabled. When monitoring is off (the default), the emit costs a single +atomic boolean load and a branch: no clock read, no allocation. Keeping +the names in one authoritative file prevents a typo from producing an +orphaned metric that no dashboard ever finds. + +The five constants are `communityAssignment` ("community.assignment", +value is the community count), `centralityScore` ("centrality.score", +value 1.0 as a completion marker), `nmfFactor` ("nmf.factor", value is the +final reconstruction error), `anomalyFlag` ("anomaly.flag", value is the +absolute z-score), and `edgeDecayedWeight` ("edge.decayed_weight", value +is the applied decay factor). The file also records the caller contract: +callers supply the estate tag and the timestamp, because SubstrateML never +reads a clock. + +## FloatSimHash.swift + +This file provides `FloatSimHash`, which projects a dense float embedding +vector — for example a 384-dimension MiniLM or 768-dimension BERT vector +from an external model — into the substrate's canonical 256-bit +fingerprint type, `Fingerprint256`. + +The substrate compares memories by Hamming distance, the number of +differing bit positions between two equal-length codes. External +embeddings are floats, not bits, so they cannot join that comparison +directly. `FloatSimHash` bridges the gap with signed random-hyperplane +projection, the classic SimHash construction: generate 256 pseudo-random +"+1 or -1" direction vectors (hyperplanes) matching the input dimension, +take the sign of each hyperplane's dot product with the input, and pack +the 256 sign bits into four 64-bit blocks. Vectors that point the same way +land on the same side of most hyperplanes, so cosine similarity in float +space is approximately preserved as Hamming closeness in bit space. + +`project(vector:seed:)` is the entry point. The hyperplanes come from a +SplitMix64 generator seeded by the caller, drawn in a pinned order (256 +planes outer, coordinates inner) with a pinned sign convention (low bit 1 +means +1). The same seed always yields the same planes, which matters +because stored fingerprints must stay comparable to freshly computed ones. +Different embedding providers use different seeds, so each provider's +fingerprints live in their own namespace. The dot-product work itself is +dispatched to a `SubstrateKernel` backend, selected once per process, so a +future SIMD kernel can drop in without touching this file. +`planes(seed:dim:)` materializes the hyperplane set for callers, such as +the kernel-equivalence conformance test, that need it directly. + +## CompositeDistance.swift + +This file provides `CompositeDistance.distance`, the substrate's primary +retrieval metric: one number in the range zero to one that says how far +apart two memory rows are. + +The score is a weighted blend of two normalized parts: +`alphaLattice * latticeDistance + alphaFingerprint * (hamming / 256)`. +The lattice part measures topical distance in the classification +hierarchy. The fingerprint part is Hamming distance divided by the fixed +256-bit width. Both weights default to 0.5, but the defaults are only the +starting point; the weights are learned per user over time through +Bradley-Terry feedback (see `BradleyTerry.swift`). + +One edge case is engineered, not accidental. Two fingerprints are only +comparable when they were computed under the same hyperplane family — the +`compatibleSeedScope` argument. That always holds inside one estate, but +across estates it requires a completed pairing handshake. When the scopes +are not compatible, the function keeps only the lattice term and does not +rescale it. The resulting smaller distance is the documented intended +behavior. Inputs are checked with preconditions rather than clamped, so an +out-of-range value fails loudly at the source. + +## LatticeDistance.swift + +This file provides the two conceptual distance axes that feed the +composite metric: tree distance in the UDC classification hierarchy and +graph distance in the Wikidata concept graph. + +`LatticeAnchorStr` pairs a UDC code string with an optional Wikidata Q-ID +(zero means none). `UDCTreeDistance.distance(_:_:)` compares two code +strings by their longest common prefix: the more leading characters two +codes share, the closer their subjects sit in the hierarchy. The raw +formula can reach two, so the result is clamped to the zero-to-one range +the composite metric requires. `WikidataGraphDistance` runs a +breadth-first search over a caller-injected `WikidataAdjacencyProvider` +(the "is-a" and "part-of" edges), bounded at depth 4, and normalizes the +path length with `1 - exp(-length / 3)`. Unreachable concepts, and the +null Q-ID zero, score the maximum 1.0. The graph is directed as given, so +symmetry holds only if the provider supplies reverse edges. + +`LatticeDistance.distance(_:_:provider:alphaUDC:alphaQID:)` blends the two +axes with weights that default to 0.5 each. A legacy overload also accepts +the older integer-hashed anchor type; hashing destroys prefix structure, +so that overload can only answer zero (identical) or one (different) and +is kept purely for compatibility with earlier call sites. + +## PartialStateRecall.swift + +This file provides the "same in one way, different in another" recall +primitive. A row fingerprint has four 64-bit blocks, and different blocks +encode different aspects of a memory. `PartialStateRecall` scores a +candidate row by how well it matches an anchor on one chosen block subset +while differing from it on another — for example, same topic but different +behavioral state. + +`score(rowFingerprint:anchor:matchBlocks:differBlocks:)` computes a +restricted Hamming distance over each block subset. The match score is one +minus the normalized distance on the match blocks; the differ score is the +normalized distance on the differ blocks. The final score is their +product, so a candidate must satisfy both constraints at once: a row +identical to the anchor everywhere scores zero (it fails to differ), and a +row unrelated on the match blocks also scores zero. Block identifiers must +come from the set {0, 1, 2, 3}; a precondition enforces this because an +out-of-range block would silently corrupt the denominator and produce a +subtly wrong score instead of a crash. `topK(anchor:rows:...)` scores every +row in a linear scan and returns the best k — the reference form of the +`recall_partial_match` query. `hammingBlocks(_:_:blocks:)` exposes the +block-restricted Hamming count itself. + +## ShingleSimilarity.swift + +This file provides the substrate's one character-shingle Jaccard +similarity: how alike two strings are, as the overlap of their +three-character substring sets. + +Two consuming kits needed exactly this function for recall de-duplication, +and neither may depend on the other. Following the "one implementation per +substrate atomic" rule, the shared math lives here, below both. +`shingles(_:)` lowercases the input and slides a three-character window +across it, collecting each substring into a set; strings shorter than the +window collapse to a single whole-string shingle, and the empty string +yields the empty set. `similarity(_:_:)` returns intersection size divided +by union size, with "both empty" defined as 0.0 rather than undefined. No +stemming, tokenization, or locale-sensitive transform is applied, which is +what keeps Swift and Rust byte-identical on the content the recall path +actually compares. The window size 3 is pinned: long enough to +discriminate near-duplicates, short enough to survive paraphrase. + +## MomentSummary.swift + +This file provides the moment summary: one 256-bit fingerprint that +stands for everything active during a time window, built by OR-reducing +the fingerprints of the matching rows. + +OR-reduction sets a bit in the output when any input has it set. The +choice of OR gives the summary useful algebra for free: it is +order-independent, idempotent, and monotonic — more rows can only add +bits, never clear them. An empty match yields the all-zero fingerprint. +Recall can then find similar past moments by Hamming search over cached +window summaries. + +What counts as "active during" the window is deliberately not hardcoded. +The cookbook defines three legitimate meanings, so both `summarize` +overloads take the predicate as a caller-supplied closure. One overload +works on full `Row` values; the other works on `RowLite`, a 40-byte +fingerprint-plus-timestamp pair, and runs four to five times faster on +large windows purely because the loop touches less memory. Both paths are +verified byte-identical against a conformance vector (CRC `0x6762440b`). +`capturedDuring(_:_:)` is the common convenience predicate, and +`orReduce(_:)` exposes the raw reduction for callers that already hold a +fingerprint list. + +## TemporalCompression.swift + +This file provides hierarchical time roll-up: hour summaries roll into +days, days into weeks, and onward through months and quarters to years. +Each level is a `TemporalWindow` — a summary fingerprint, a row count, and +start and end timestamps. + +Because the summary operator is OR-reduction, rolling up is associative +and commutative: the result does not depend on the order windows arrive. +`compress(rows:startHLC:endHLC:level:)` builds a base window from raw row +fingerprints. `rollup(windows:to:)` merges finer windows into one coarser +window, summing row counts with overflow-safe addition and taking the +outermost time bounds. `cascadeRollup(hourWindows:upTo:)` automates the +whole ladder: it buckets the current level's windows by which coarser slot +they fall into (integer division of epoch seconds by the coarser level's +approximate duration) and rolls each bucket, one level at a time. + +The `WindowLevel` durations are deliberately approximate — a month is +thirty days, a year 365 — because they serve only as bucket divisors, not +calendar math. Storage and scheduling belong to the caller (the dreaming +daemon); this file is pure computation and never reads a clock. + +## FeatureExtractors.swift + +This file provides the ingestion boundary for ambient signals: five +encoder types that turn raw OS sensor samples into `AmbientSampleRow` +values — fingerprinted, classification-anchored rows ready for storage +and recall. + +Each extractor covers one stream: `HealthKitExtractor` (biometrics), +`CoreLocationExtractor` (position), `EventKitExtractor` (calendar), +`ScreenTimeExtractor` (app usage), and `SystemTelemetryExtractor` (system +load). The OS-specific capture code lives in the host apps; only the pure +encoding lives here. Each `extract` call computes four 64-bit subhashes +from the sample's salient fields, then feeds them with a hyperplane family +into the substrate's SimHash to produce the 256-bit fingerprint. Numeric +fields are quantized to integers before hashing (for example, coordinates +times one million) so floating-point representation can never change a +fingerprint. Attendee lists are sorted before hashing so the same meeting +fingerprints identically regardless of listing order. Each stream carries +a fixed UDC lattice code (for example "613.71" for health, "914" for +location) as its coarse topical anchor. + +Only HealthKit rows carry a `payload` — a small, key-sorted, deterministic +binary encoding for human-readable recall detail. The other streams emit +no payload by design: the fingerprint already carries all recall-relevant +signal, and less stored raw data means less to protect. The FNV-1a hash +constants serve as the file's general 64-bit mixers. + +## AuditLogFold.swift + +This file provides `AuditLogFold`, which reconstructs what a memory row +looked like at any point in time by replaying its audit log. + +The audit log is a grow-only set (a G-Set, a conflict-free replicated data +type): events are only ever added, never edited. Current state is defined +as the fold — the sequential replay — of a row's events sorted by their +hybrid logical clock (HLC) timestamps, and "state as of time T" is the +same fold truncated at T. Because the fold depends only on the set of +events, not their arrival order, two devices that have exchanged the same +events converge to identical state. That property is what makes sync +correct without coordination. + +`projectCurrentState(rowId:nounType:events:)` folds a full history; +`projectStateAt(...)` folds up to a cutoff; `projectAll(events:asOf:...)` +groups a whole log by row and folds each. The result type, +`ProjectedRowState`, carries the row's bitmaps, lattice anchor, tombstone +flag, and last-event time. One rule is deliberately sticky: once the +tombstone marker (state raw value 33) appears, the row stays tombstoned +even if a later, illegal event tries to revive it. The fingerprint is +intentionally absent from the projection; callers recompute it from the +bitmaps and anchor when needed. + +## RowAttributeView.swift + +This file provides the bridge from the audit log to the pattern miners: +`RowAttributeView`, one row's field writes flattened into sorted +`(field, value)` byte pairs — the exact shape Apriori and Formal Concept +Analysis consume. + +`from(auditEntries:)` runs a four-step pipeline. It first builds a +vocabulary of all distinct field paths in the batch, sorted +alphabetically and capped at 64 entries, because the miners' item type +stores the field index in six bits. It then groups entries by row, keeps +only the latest write per field by HLC order (the same last-write-wins +rule the audit fold uses, so the two subsystems never disagree), and +expands each surviving value: a bitmap value becomes one attribute per set +bit, an integer contributes its low byte, and a null contributes nothing. +Rows left with no attributes are dropped. Attributes are sorted within a +view and views are sorted by row identity, so the output is deterministic. + +One subtlety is documented rather than hidden: the vocabulary is ephemeral +to a single call. Two separate calls can assign different field indexes, +so callers who need a stable vocabulary must merge their entry lists +before calling. The input types `RowAuditEntry` and `RowAuditValue` are +defined here, in a lower layer than the kit that owns the real audit +types, precisely so the dependency arrow keeps pointing downward. + +## MatrixDecay.swift + +This file provides the substrate's forgetting mechanism: exponential +half-life decay for every matrix in the matrix tier — the accumulated +statistics tables such as field presence, correlation, co-activation, and +action outcomes. + +The rule is constitutional: decay may only shrink values. New evidence +enters through separate write operations, never through this file. +`DecayingMatrix` is a plain row-major matrix that carries its own +half-life and its last-decay timestamp. `apply(to:nowSeconds:estate:ts:)` +computes the elapsed time, derives one shared factor +`exp(-elapsed * ln2 / halfLife)`, and multiplies every cell by it. A +non-positive elapsed time is a documented no-op — decay never runs +backward — but it still emits its telemetry metric with factor 1.0, so a +watcher can tell "daemon ran, nothing to do" from "daemon never ran." +Using one scalar factor per pass (rather than per-cell clocks) is what +makes decay commute with addition, a property the file states formally. + +`decayFactor(elapsedSeconds:halfLifeSeconds:)` is the pure projection +helper, and `decayAndAdd(...)` is the decay-then-add step the online write +path uses. `DecayHalfLives` records the per-matrix schedule from the +cookbook: 90 days for field presence, 180 for correlation, 60 for +co-activation, 30 for temporal causality, 365 for action outcomes, and +730 for calibration. Two details protect determinism and safety: `ln2` is +a local constant chosen to match Rust's value bit-for-bit, and the +elapsed-seconds cast for the telemetry tag saturates instead of trapping +on absurdly large values. Three `applyExponentialDecay` overloads for the +typed matrices are deliberate no-op adapters at this reference level; the +production per-matrix routine lives in the consuming kit. + +## ActionOutcomeMatrix.swift + +This file provides the action-outcome matrix: the substrate's +reinforcement record of which actions succeed. Each cell, keyed by a +six-bit action kind and a six-bit outcome category, accumulates a success +count, a total count, and the last-update HLC. + +`ActionOutcomeKey` enforces the six-bit ranges with preconditions and +packs both bytes into one `UInt16` ordering key. `observe(...)` increments +a cell with overflow-safe arithmetic. `successRate(...)` and +`observationCount(...)` read cells back. The interesting method is +`topActions(forOutcome:k:minObservations:)`: it ranks actions not by raw +success rate but by the Wilson lower bound — the bottom of a 95 percent +confidence interval around the rate. A cell with two observations and a +perfect record should not outrank a cell with two hundred observations and +a 95 percent record; the Wilson bound encodes exactly that caution. The +method returns rate, bound, and count together so callers see the value +the ranking actually used. Ties break by total count, then by ascending +action, keeping output deterministic. Decay for this matrix (365-day +half-life) is applied lazily by the dreaming daemon through +`MatrixDecay`, not here. + +## BradleyTerry.swift + +This file provides an online Bradley-Terry estimator: it learns a +per-row "strength" score from pairwise preference feedback, and that +strength reorders future recall candidates. + +Every time recall surfaces a candidate set and the user or agent picks +one, that pick is an observation: the winner beat the losers. The +Bradley-Terry model says the probability that row i beats row j is +`w_i / (w_i + w_j)`. The implementation works in log-space (`theta = log +w`), so the win probability becomes a sigmoid of the theta difference and +gradient updates never have to guard positivity. `observe(_:)` performs +one stochastic gradient step per observation with L2 regularization +pulling theta toward zero, which keeps early sparse data from overfitting. +For multi-loser observations the winner's theta is updated sequentially +against each loser, recomputing against the running value — a pinned +ordering that the Rust port mirrors exactly. + +`observeBatch(_:)` applies a day's worth of observations in order. +`strength(of:)` and `probability(_:beats:)` read the model. There is no +randomness anywhere: the same observation sequence always produces +bit-identical theta, and a serialized theta dictionary warm-restarts the +estimator exactly. The defaults (learning rate 0.05, L2 0.001) are tuned +for the ten-to-one-hundred observations per day a personal estate +produces. Strength is a ranking signal only; it never changes what is +stored or filters what exists. + +## LLMCalibrationCurve.swift + +This file provides a twenty-bucket calibration histogram that tracks how +well an LLM's stated confidence matches reality. + +When the cognition tier resolves a probabilistic claim — the model said +"80 percent sure" and the claim turned out true or false — +`observe(claimedConfidence:actualOutcome:)` clamps the confidence into +the range zero to just under one, maps it to one of twenty 0.05-wide +buckets, and bumps that bucket's predicted and hit counters. Counters use +wrapping arithmetic on purpose: this is a long-lived accumulator, and +overflow should wrap rather than crash. + +`actualRate(in:)` reads one bucket's observed hit rate. +`expectedCalibrationError()` measures the weighted average gap between +each bucket's observed rate and its midpoint — the "perfectly calibrated" +diagonal. `brierScore()` is the same with squared gaps. `decay(factor:)` +scales every counter by a fraction so stale calibration evidence fades +smoothly; the dreaming daemon drives it on the 730-day half-life schedule. +The result lets recall annotate model-claimed confidence with how much +that model's confidence has historically been worth. + +## Sampling.swift + +This file provides the three deterministic distribution samplers — +Normal, Gamma, and Beta — that sit under Thompson sampling. Thompson +sampling is a decision strategy that picks among options by drawing from a +belief distribution over each option's payoff; the dreaming daemon uses it +to choose maintenance strategies. The policy lives in the consuming kit; +the sampling math is centralized here so every consumer shares one +implementation. + +All three samplers take the caller's SplitMix64 generator by `inout` +reference, so the random stream advances visibly and reproducibly. +`sampleNormal(rng:)` uses the Box-Muller transform and deliberately keeps +only the cosine branch, consuming exactly two uniform draws per call — +caching the sine branch would desynchronize the stream between ports. +`sampleGamma(shape:rng:)` implements Marsaglia-Tsang rejection sampling +for shape at least one, with the standard squeeze constant 0.0331, and +reduces smaller shapes through the Ahrens-Dieter identity; the reduction's +extra uniform draw happens before the recursive call, an order pinned for +cross-port stream identity. `sampleBeta(alpha:beta:rng:)` derives Beta +from the ratio of two Gamma draws, alpha first. Given the same seed and +inputs, both legs consume the same words in the same order and return the +same values, gated by a shared conformance vector. + +## CommunityDetection.swift + +This file provides Louvain community detection over the estate graph: it +labels every node with a community so related memories cluster together. +Auto-rooming, keystone recall, and the daily dreaming refresh all consume +the partition. + +Louvain works in two phases. Phase 1 repeatedly tries moving each node +into a neighboring community and keeps the move that most improves +modularity — a score that rewards putting well-connected nodes together +relative to chance. Phase 2 condenses each community into a supernode and +repeats on the smaller graph. `detect(...)` runs Phase 1 only; +`detectFull(...)` runs the full multi-level algorithm and is the normal +entry point. + +Two design choices deserve explanation. First, determinism: candidate +communities are evaluated in ascending label order, so score ties always +resolve to the lowest label instead of depending on hash-map iteration +order; a `canonicalize(_:)` step then renumbers labels by first +appearance, so Swift and Rust emit identical partitions. Second, the +resolution parameter: plain Louvain can get "pair-locked" on graphs with +strongly bonded pairs plus weak star edges, unable to escape a local +optimum. The Reichardt-Bornholdt gamma scales only the degree-penalty +term; small gamma (consumers use 0.05) forces pairs to merge into hub +communities, while the default 1.0 reproduces classical Louvain exactly — +IEEE arithmetic guarantees `1.0 * x == x`, so the refactor cannot perturb +legacy results. The implementation is intentionally the compact +O(N·E)-per-pass reference rather than the fastest possible variant. One +`community.assignment` telemetry signal fires per call when monitoring is +on; a degenerate all-zero-weight graph returns identity labels and emits +nothing. + +## EigenvalueCentrality.swift + +This file provides eigenvalue centrality: a per-row authority score +computed by power iteration over the estate graph, cached as each row's +keystone score for `recall_keystone`. + +Power iteration repeatedly multiplies a score vector by the graph's +adjacency and renormalizes; the vector converges to the principal +eigenvector, where a node scores highly when high-scoring nodes connect to +it. The direction matters and is a locked design decision: `compute(...)` +multiplies by the transpose, accumulating at node j the influence of nodes +that point TO j. That is authority ("many rows reference this one"), not +hub ("this row references many"), and it is the right meaning for +cognitive keystones. A conformance vector pins the directed behavior so it +cannot drift back. For undirected centrality, callers symmetrize the +adjacency first. + +Each iteration adds a Perron shift (`xNext += 1.0 * x`), which shifts +every eigenvalue without changing eigenvectors and thereby breaks the +plus-minus oscillation that bipartite graphs such as stars exhibit under +raw power iteration. Iteration stops at convergence (L2 difference below +1e-6) or at the 100-iteration safety cap. A zero-norm vector falls back to +the uniform distribution — an edgeless graph makes every row equally +non-central. All three return paths emit the single `centrality.score` +telemetry signal, tagged with the node count and iterations used. + +## RandomWalks.swift + +This file provides random-walk-with-restart over the memory graph — the +engine behind exploratory recall, which wanders outward from a starting +memory and reports where it keeps landing. + +`walk(adjacency:start:length:restartProb:seed:)` runs one walk over a +densely indexed weighted adjacency list. Each step either restarts at the +start node (probability 0.15 by default) or moves to a neighbor chosen by +roulette-wheel weighted sampling. A dead end — a node with no out-edges — +is an implicit restart rather than an error. The walk is seeded +explicitly: the same graph, start, length, and seed reproduce the same +walk on both legs, because the file carries its own public `SplitMix64` +and a `uniform01(_:)` that derives a double from the top 53 bits exactly +as Rust does. Malformed graphs (out-of-range neighbor, negative or +non-finite weight) fail hard preconditions immediately, because a bad +adjacency cannot represent probabilities and silence would corrupt +results downstream. + +`walkWithRestart(seed:steps:restartProbability:rngSeed:adjacency:)` is the +row-identifier variant: it walks a dictionary adjacency with uniform +neighbor choice and returns visit counts per row, the shape the +exploratory recall recipe consumes. `sampleWeighted(_:rng:)` exposes the +weighted sampler, including its pinned fallbacks (uniform choice when +total weight is zero; last neighbor on cumulative rounding shortfall). + +## NMFAlternatingLeastSquares.swift + +This file provides the canonical non-negative matrix factorization (NMF) +engine. NMF approximates a non-negative matrix V as the product of two +smaller non-negative matrices, V ≈ W × H. Rows of H are latent "themes"; +rows of W say how much each original row loads on each theme. The +substrate runs NMF over its field-presence and co-occurrence matrices to +surface themes for recall and for the Topology theme overlay; the +dreaming daemon reruns it monthly. + +`factorize(V:rank:maxIterations:tolerance:seed:estate:ts:)` uses the +Lee-Seung multiplicative update rules, which keep every entry non-negative +automatically: each iteration rescales H by the ratio of `WᵀV` to `WᵀWH`, +then W by the ratio of `VHᵀ` to `WHHᵀ`, with epsilon 1e-9 in denominators +against division by zero. W and H start as uniform random values from a +SplitMix64 seeded by the caller (default `0xDEADBEEFCAFEBABE`), so +initialization is bit-identical everywhere. Iteration stops when the +root-mean-square reconstruction error stops changing by more than the +tolerance, or at the cap. Entry preconditions enforce a rectangular, +finite, non-negative V, because the Lee-Seung theorem is undefined +otherwise. + +The hot loops run on flat row-major arrays through unsafe buffer pointers. +This is a documented performance necessity: bounds-checked nested-array +subscripts block auto-vectorization and left Swift roughly one hundred +times behind the Rust port on large reindex drains. The loop nest and +reduction order were preserved exactly through that rewrite, so the NMF +conformance vector (CRC `0x300bf633`) remains byte-identical. +`reconstructionError(V:W:H:)` exposes the RMS error on nested arrays, and +the nested-array matrix helpers remain for external callers. On +completion the engine emits the `nmf.factor` telemetry signal carrying +the final error. + +## NMFDoubleFrobeniusSquared.swift + +This file provides a parked, explicitly non-production NMF variant: the +double-precision engine with a raw Frobenius-squared convergence test +that one consumer used before migrating to the canonical engine above. + +It exists for one reason: honesty in benchmarking. The migration decision +requires a future benchmark comparing the old f64/Frobenius-squared +approach against the canonical f32/RMS approach on iteration count, time, +memory, and recall quality. Deleting the old algorithm would make that +comparison impossible, so it is preserved verbatim behind a production +gate: no consumer may wire to this type until the benchmark passes, and +even SIMD acceleration of the f64 path is separately gated. + +The algorithm is the same Lee-Seung scheme in scalar f64, with three +deliberate differences: convergence checks the raw (unnormalized) +Frobenius-squared error rather than RMS; initialization floors every +starting cell at 1e-3, because a cell at exact zero can never recover +under multiplicative updates; and the default seed is its own historical +value (`0xC0FFEE_BABE_BEEF`), distinct from the package's canonical seeds. +`factorize(...)` returns an `NMFDoubleFrobeniusSquaredFactorization` whose +`loadings(forRow:)` yields a row's latent factors. Despite being parked, +the file is still conformance-gated: Swift and Rust must stay +bit-identical on its named vector, so the eventual benchmark measures the +algorithm, not implementation drift. + +## AnomalyDetection.swift + +This file provides statistical anomaly scoring for telemetry streams and +matrix cells: the classic z-score and a robust variant, plus the +threshold check that turns a score into a flag. + +A z-score says how many standard deviations a value sits from the mean of +its baseline window. `zScore(value:mean:stddev:)` is the pure formula; +`rollingZScore(window:current:...)` computes mean and deviation from the +window first. The robust variant, `modifiedZScore(value:median:mad:)`, +replaces mean with median and deviation with the median absolute +deviation, then scales by the standard constant 0.6745 so the same +threshold applies to both variants; outliers already inside the window +cannot drag the baseline the way they drag a mean. +`isAnomalous(zScore:threshold:)` compares the absolute score against the +default threshold 3.0, roughly a one-in-370 false-positive rate on normal +data. + +`rollingModifiedZScore(...)` sorts a mutable copy of the window in place +rather than allocating a sorted copy; at large windows this measured five +times faster, and the behavior is pinned by a CRC marker (`0x6c6fda4d`) +plus a note matching Rust's sort semantics for all non-NaN input. Both +rolling functions emit the `anomaly.flag` telemetry signal with the +absolute score as the value; the timestamp is always caller-supplied. + +## JacobiSVD.swift + +This file provides a deterministic truncated singular value +decomposition (SVD) — the linear algebra that turns a term-document +matrix into low-dimensional semantic embeddings for latent semantic +analysis in CorpusKit. + +SVD factors a matrix into rotation, scaling, and rotation: `A = U S Vᵀ`. +The implementation is one-sided cyclic Jacobi: it repeatedly applies +two-by-two planar rotations to every pair of columns in a fixed cyclic +order, accumulating the rotations into V; after the sweeps, column norms +become the singular values and normalized columns become U. A sign +convention (largest-magnitude entry of each left vector forced positive) +removes the inherent plus-minus ambiguity. + +Every convergence-affecting choice is pinned for cross-port bit-identity, +which is why this exists instead of a call to Accelerate or LAPACK: the +sweep count is fixed (default 30, not convergence-tested), the rotation +order is fixed, all arithmetic is scalar Float32 with no SIMD or fused +multiply-add, and the rotation formula in `jacobiCS(alpha:beta:gamma:)` +keeps a pinned expression tree that must not be refactored. +`decompose(A:rank:sweeps:)` requires at least as many rows as columns and +returns an `SVDResult` (U, non-increasing singular values, Vt, rank). The +sweep loops use unsafe buffer pointers for the same documented +vectorization reason as the NMF engine; every index is derived from loop +bounds, never from data. + +## FFT.swift + +This file provides the discrete Fourier transform and the rhythm analysis +built on it. The substrate uses it to detect periodicity: read one +fingerprint bit across the most recent N time buckets, and the dominant +frequency of that 0/1 series reveals the rhythm of whatever that bit +encodes — circadian patterns in heart-rate bits, weekly cycles in +calendar bits. + +`Complex` is a plain double-precision complex number with the arithmetic +the transform needs. `FFT.forward(real:)` is the Cooley-Tukey radix-2 +algorithm: a bit-reversal permutation followed by log2(N) butterfly +stages. Input length must be a power of two — a constitutional constraint; +callers zero-pad shorter windows. The production Apple-silicon path routes +to the Accelerate framework but must produce bit-identical output to this +scalar reference on the conformance vectors; conformance is output +equality, not speed parity. `magnitudeSpectrum(real:)` maps the spectrum +to magnitudes. + +`RhythmAnalysis.analyze(series:bucketDurationSeconds:)` runs the +transform, finds the strongest bin among the unique positive frequencies +(bin 1 through N/2), and converts it to a dominant period in seconds, +alongside total spectral energy excluding the DC bin. A constant series +has no dominant period and reports `nil` — the cookbook treats "no rhythm" +as an honest answer, not a default. The fingerprint-based overload +extracts the bit series from a list of fingerprints first, with the block +and bit position precondition-checked. + +## InformationTheory.swift + +This file provides the information-theoretic primitives — entropy, mutual +information, KL divergence, cross-entropy, Jensen-Shannon divergence, and +normalized mutual information — used to quantify uncertainty in bitmap +distributions and to detect drift between a recent window and a long-term +baseline. + +All quantities use log base 2, so results are in bits. Entropy measures +how unpredictable a distribution is. Mutual information measures how much +knowing one variable tells you about another. KL divergence measures how +badly a model distribution q explains an observed distribution p. +Everywhere, terms with probability exactly zero are skipped, implementing +the standard `0 · log 0 = 0` limit without ever producing NaN. + +Two honesty notes are documented in the code. When p has mass where q has +none, true KL divergence is infinite; `klDivergence(_:_:)` skips such +terms and its result is therefore a lower bound, which callers must know. +And `normalizedMutualInformation(joint:)` returns the zero sentinel for a +ragged joint matrix rather than silently computing a plausible-looking +wrong value from corrupted marginals. Length mismatches between p and q +are hard preconditions; distribution validity itself (non-negative, +summing to one) is the caller's responsibility by contract. + +## AssociationRuleMining.swift + +This file provides pairwise association-rule mining over `MatrixO`, the +co-occurrence matrix the substrate updates on every capture: it surfaces +rules of the form "when field A has value x, field B tends to have value +y," with the standard support, confidence, lift, conviction, and leverage +metrics. + +The design leans on how `MatrixO` is built. Its diagonal cells already +hold single-item counts, and its off-diagonal cells hold pair counts, so +`mineAssociationRules(matrix:activeRowCount:thresholds:)` needs only two +passes over the stored entries: one to collect single supports from the +diagonal, one to walk the pairs, compute the five metrics, and keep rules +that clear the `MiningThresholds` gates. Self-rules (A implies A) are +skipped as noise, conviction is defined as positive infinity at +confidence one, and the total row count N is injected by the caller +because the matrix itself does not know it. Because the entries iterate +in canonical packed-key order, the output needs no final sort — an +ordering both legs rely on for byte-identical results. + +`Item` is the shared atom: a `(field, value)` byte pair with a packed +16-bit ordering key, reused by the Apriori engine. The internal +`AssociationRuleEngine` holds the actual math so conformance tests can +drive it directly; multi-item antecedents are out of scope here and +belong to `AprioriMining.swift`. + +## AprioriMining.swift + +This file provides the multi-antecedent generalization: the classic +Apriori algorithm over `RowAttributeView` rows, producing rules such as +"{A, B} implies C." At its minimum itemset size it reproduces the pairwise +engine's answers; beyond that it finds combinations the pairwise engine +cannot see. + +`AprioriMining.mine(rows:thresholds:)` follows the textbook outline. Each +row becomes an item set. Frequent single items seed the search; each +level joins frequent (k-1)-item sets that share a lexicographic prefix, +counts candidate support with subset tests, and prunes below minimum +support, until the size cap `maxK` (default 3). Rules are then extracted +from every frequent itemset of size two or more, with each member tried +as the consequent, and filtered by the support, confidence, and lift +gates in `AprioriThresholds`. Requiring lift at least 1.0 suppresses +anti-correlated coincidences. + +Determinism gets specific care because Swift dictionaries iterate in +undefined order: itemsets are processed in canonical lexicographic order, +and the final sort uses a four-key total order (lift, confidence, +evidence count, then lexicographic tie-break), so output is identical +across runs and across languages. One defensive detail: the type's custom +`Decodable` initializer routes through the public initializer so a +serialized `maxK` of zero or one cannot bypass the floor of two and +corrupt the level loop. `mineAprioriRules(rows:thresholds:)` is a free +function wrapper matching the pairwise engine's call style; `AprioriRule` +carries the five metrics plus a raw `evidenceCount` for interpretability. + +## FormalConceptAnalysis.swift + +This file provides bounded Formal Concept Analysis (FCA): it finds the +exact groupings hidden in a table of rows and attributes. A formal +concept is a maximal pair — a set of rows that share exactly a set of +attributes — and the concepts of an estate are the data-driven "kinds of +memory" that emerged from what was actually recorded, as opposed to the +soft themes NMF finds or the graph clusters Louvain finds. + +`FormalAttribute` is one typed attribute (namespace, key, value), ordered +lexicographically — that ordering is the determinism backbone of the whole +file. `FormalContext` stores the row-attribute relation in both directions +as bitsets, so the derivation operators `extent(of:)` (rows carrying all +given attributes), `intent(of:)` (attributes common to all given rows), +and `closure(of:)` reduce to word-wise bit operations. Contexts build from +raw per-row attribute lists or from `RowAttributeView` batches. + +Full concept-lattice enumeration is exponential, so `BoundedConceptMiner` +never attempts it. It seeds from attributes meeting `minSupport` (and +optionally frequent attribute pairs in `.multi` seed mode, capped by +`maxSeeds`), takes exactly one closure per seed, deduplicates by intent, +and truncates at `maxConcepts` — polynomial cost by construction, with a +fully specified output sort. `ConceptCoverDeltas.covering(concepts:)` +computes the direct-neighbor edges of the concept order, the "what +attributes were added" structural lens; the file is explicit that a cover +edge is not a logical implication. `StabilityEstimator.estimate(...)` +approximates how robust a concept is to noise by re-closing random +half-subsets of its extent under a per-concept SplitMix64 stream (caller +seed mixed with a hash of the concept; default seed +`0xCAFEBABEDEADBEEF`), trading exactness for a bounded budget. A zero +budget, the default, leaves stability `nil`. + +## ConceptImplications.swift + +This file provides the Duquenne-Guigues canonical basis: the minimal set +of implications — "every row with attributes X also has attribute Y" — +that hold without exception across a formal context. Where FCA finds the +groupings, this finds the rules the groupings obey, feeding consolidation +and synthesis. + +`conceptImplications(over:context:maxImplications:maxPremiseSize:)` +enumerates candidate premises in strictly increasing size order and tests +each for pseudo-intent status: its closure must add something, and every +smaller pseudo-intent's conclusion must already be inside it. Processing +in size order makes that incremental test equivalent to full minimality +checking. Two independent caps keep an NP-hard enumeration bounded: +`maxImplications` is a hard stop that sets `isTruncated` (with a +look-ahead so the flag is honest when the cap lands mid-run), and +`maxPremiseSize` silently skips larger premises without affecting the +soundness of what is emitted. Every ordering — the attribute universe, +the combination enumeration, and the final sort by premise size then +lexicographic content — is pinned, so both legs emit bit-identical bases. +`Implication` carries the premise and conclusion sets; an empty context +short-circuits to an empty, untruncated result. + +## TemporalCausalityFold.swift + +This file provides the fold that feeds the temporal-causality matrix: it +scans a time-sorted stream of audit entries and counts "field X changed, +then field Y changed N minutes later" pairs, bucketed by lag. + +`fold(entries:windowMinutes:startWatermark:)` processes each entry newer +than the caller's watermark. It evicts buffered entries older than the +window (256 minutes by default), then pairs the new entry against every +remaining buffered entry, mapping each minute delta to one of eight +log-spaced lag buckets — 1, 2, 4, up to 128 minutes, via +`lagBucket(forMinutes:)` — and emitting one delta increment per source +coordinate, target coordinate, and bucket. The result carries the deltas +plus a new watermark, so the hourly batch run resumes where it stopped. + +Two contracts matter. Entries must arrive pre-sorted by HLC; the fold +does not re-sort and would silently compute wrong deltas otherwise. And +the rolling buffer is capped at 512 entries (`maxWindowOccupancy`), +because a bulk import can drop tens of thousands of events into one +window and unbounded pairing is quadratic; keeping only the most recent +in-window entries preserves the near-lag signal the causality matrix +actually cares about. Output deltas are aggregated by key but returned in +first-insertion order, so repeated runs are bit-identical. The lag bucket +table and the occupancy cap are pinned constants mirrored exactly in the +downstream matrix type and the Rust port. The input shapes +(`TemporalFieldCoord`, `TemporalAuditEntry`) are defined locally so the +kit that owns the real audit types stays upstream. + +## TypedDecayWeighting.swift + +This file provides type-aware staleness weighting for distillation +features. Different kinds of extracted facts age at different speeds: a +version number (numerical) goes stale in days, while a domain concept +(entity) stays true across major changes. + +`DistillationFeatureType` names the four categories with their wire tags — +ENT, REL, TMP, NUM — and pins each one's decay rate lambda: 0.1, 0.2, +0.5, and 0.8 respectively, by analogy with frequency bands in diffusion +noise schedules. `weight(featureType:ageInUnits:)` computes the +exponential weight `exp(-lambda * age)`, clamping negative ages to zero +so future-dated observations count as present. One age unit is one day by +convention (86,400 seconds), though callers choose the unit. + +`weightedDocFrequency(featureType:presenceTimestamps:allMemoryTimestamps: +referenceDate:timeUnit:)` computes the decayed document frequency: the +summed weight of the memories where a feature is present, divided by the +summed weight of all memories in the cluster. Recent evidence therefore +counts more than old evidence, at a rate set by the feature's type. This +is the frequency the distillation pipeline uses whenever timestamps are +available. + +## DeltaFeatureExtractor.swift + +This file provides trajectory classification for a single feature's value +sequence across a cluster: has the value stabilized, is it trending, is +it flip-flopping, or is it noise? Its job is rescue — a feature that +fails the recurrence threshold on raw counts may still carry a real +pattern over time, and this classifier finds it. + +`DeltaType` names the five verdicts: STATIC, CONVERGENT, MONOTONE, +OSCILLATING, DIVERGENT. `analyzeCategorical(sequence:decayLambda:)` +handles string-valued features: all-identical values are STATIC; an +A-B-A-B pattern over the last four observations (four required, to avoid +false positives on short sequences) is OSCILLATING; otherwise the length +of the trailing run of the final value, as a fraction of the sequence, +decides CONVERGENT (at least the threshold, default 0.5) or DIVERGENT. +`analyzeNumerical(sequence:decayLambda:)` handles numbers via consecutive +differences: all zero is STATIC, all one sign is MONOTONE (with slope and +a fixed confidence of the type's decay rate, default 0.8), strict sign +alternation is OSCILLATING, anything else DIVERGENT. + +Both paths return a full `DeltaAnalysis` — verdict, terminal value, +convergence score, optional slope, confidence — so the caller always has +a well-formed result. The file is a pure namespace: no state, no clock, +no randomness at all. The oscillation check looks only at the recent +tail on purpose; the question is the current trajectory, not history. + +## DistillationScorer.swift + +This file provides the scoring stages of distillation: the math that +decides whether a cluster of memories is coherent enough to compress and, +if so, which features form the core factoid. + +The pipeline of pure functions runs in order. A structural threshold, +`structuralThreshold(M:)` = 2/M, splits features into structural +(recurring in at least two of the M units) and episodic (one-off); +`applyStructuralThreshold(features:M:)` performs the split. The file's +long comment explains why this is intra-cluster recurrence rather than a +majority vote: majority voting was found to discard real distillable +cores when a single document is split into sentences. +`computeSNR(features:M:)` then gates readiness: the summed frequency of +structural features must be at least twice the episodic sum (the +signal-to-noise ratio, floored at a tiny epsilon against division by +zero). `computeStructuralScores(features:)` assigns each feature a score +rewarding frequency and consistency via binary entropy. + +`buildPMIGraph(thresholdFeatures:incidenceMatrix:M:)` builds the +coherence graph: pointwise mutual information (PMI), in log base 2, +measures whether two features co-occur more than chance; positive-PMI +pairs become edges, and connected components are found by depth-first +search (quadratic, acceptable at the documented cap of about 150 +features) and sorted by descending weighted frequency. +`selectDominantComponent(graph:)` returns the top component — the factoid +core — and `computeConfidence(selected:allThreshold:)` scores it as mean +frequency times a fragmentation penalty (the fraction of threshold +features the core kept). Everything is deterministic and log2-based; the +Rust port's comments stress that log2, not natural log, must match +bit-for-bit. + +## DistillationPipeline.swift + +This file provides the complete five-stage distillation algorithm: a +cluster of raw memory strings in, one condensed factoid out. The output +`DistillationOutput` carries the formatted drawer content +(`"[DIST|conf=…|src=…|snr=…|delta=…] prose"`), the confidence, the SNR, +an optional delta verdict, a success flag, and a structural fingerprint +for Hamming-space lookup. + +`run(input:extractFeatures:intraItem:)` executes the stages. Stage 1 +extracts features per memory per feature type through an injected +`FeatureExtractor` closure — production injects a real entity tagger, +tests use the bundled capitalization-heuristic `defaultExtractor` — and +builds the vocabulary and incidence matrix. Document frequencies come +from `TypedDecayWeighting` when timestamps exist. Stage 2 applies the SNR +gate and the recurrence threshold. Stage 2.5 is the delta pre-pass: +features that failed recurrence are grouped by predicate key and their +value sequences run through `DeltaFeatureExtractor`; converging or +trending features are rescued with their terminal value. Stage 3 builds +the PMI graph and keeps the dominant component, with two deliberate +exceptions: in intra-item mode (one document split into sentences) all +passing features are kept, because PMI pruning would wrongly split a +single coherent document; and a ubiquity fix-up re-adds any feature +present in nearly every member, because a ubiquitous feature has zero PMI +with everything and would otherwise vanish despite being the semantic +spine. Stages 4 and 5 score, format the header, and OR-reduce per-feature +hashes into the fingerprint. + +`featureHash(_:)` maps a feature string to a `Fingerprint256` via +SplitMix64 under the pinned seed `featureSimHashSeed` +(`0x44495354494C4C41`, ASCII "DISTILLA"); changing that seed invalidates +every stored distillation fingerprint, so it is conformance-locked. +`queryFingerprint(query:extractFeatures:)` builds the matching probe +fingerprint at query time with no model inference. `DistilledHeader.parse` +reads the `[DIST|...]` header back out of stored content. Confidence +below 0.4 is failure; between 0.4 and 0.7 the output is marked uncertain. +One formatting subtlety is pinned: the `src=` field counts source drawer +IDs, not cluster members — the two differ in intra-item mode. The `run` +body is deliberately one large function; splitting it would thread the +incidence matrix and vocabulary through every helper signature. + +## PairingHandshake.swift + +This file provides the pairing handshake: the protocol steps that let two +estates establish a shared fingerprint basis so their memories become +comparable for federation. + +Fingerprints are only comparable under the same hyperplane family, and +each estate normally has its own. The handshake fixes that without a +negotiation round trip: after the two estates exchange a 32-byte nonce +out of band, `generateSharedFamily(nonce:estateA:estateB:density:)` lets +each side independently derive the identical shared family, because the +SplitMix64 seed is a pure function of the nonce and the two estate UUIDs. +`sharedFamilyKey(case:peerEstate:)` computes the canonical manifest key +(`H_shared__`) both sides store it under, and +`buildPairEvent(...)` and `buildUnpairEvent(...)` produce the audit +payloads that record the relationship. Dissolving a pairing keeps the +shared family — historical "as of" queries must still work — but stops +further sync. + +The subtlest line in the file is `PairingNonce.seedWith(estateA:estateB:)`: +it orders the two UUIDs by raw byte comparison, never by their hexadecimal +string form. ASCII comparison of hex strings can rank two UUIDs in the +opposite order from their raw bytes, and the Rust leg always compares raw +bytes — a string-ordered Swift would derive a different seed and the two +devices would build incompatible families. The nonce length (exactly 32 +bytes) is precondition-enforced, and the FNV-1a constants do the seed and +family-hash mixing. + +## TierContributionFingerprint.swift + +This file provides the tier contribution: the fixed 64-byte payload an +estate sends up the federation hierarchy, summarizing its shareable rows +as one OR-reduced fingerprint. + +`FederationCase` names the three tiers (household, fleet, industry). +`build(estateUUID:case:shareableFingerprints:hlc:)` OR-reduces the +shareable fingerprints — routed through the platform kernel so federation +work uses the best available SIMD backend while preserving the +commutative, associative, idempotent math — and wraps the result with the +estate UUID, the case, the row count, and the HLC into a +`TierContribution`. + +`encode(_:)` and `decode(_:)` implement the canonical wire format: 16 +UUID bytes, then case and row count as big-endian 32-bit integers, then +the four fingerprint blocks as big-endian 64-bit integers, then the +packed HLC — 64 bytes exactly, big-endian throughout. The comment records +a real caught bug: an earlier draft emitted the fingerprint in its +little-endian storage form and diverged from Rust by 32 bytes until the +cross-language conformance gate flagged it. This layer neither signs nor +checksums; authenticity is a federation-egress concern applied at the +share point, and privacy noise is applied at the aggregator, not by the +contributor. + +## TierAscendingQuery.swift + +This file provides the local-scope steps of the tier-ascending query +protocol: how a recall query travels to peer estates and how the answers +come back combined, without exposing any single estate's data. + +`TierAscendingQuery` models the query itself — the originating estate, +the named recall primitive to run, the target tier, the privacy budget, +and the query HLC. `computeLocal(query:dispatch:)` runs the local +primitive through an injected closure, keeping this package independent +of the cognition layer. On the peer side, +`applyDPToContribution(_:budget:rngSeed:)` adds Laplace noise (scale one +over epsilon, drawn from a seeded SplitMix64) to every score in the +peer's result and attaches a fixed 95 percent confidence interval of plus +or minus 1.96 times the scale, so the requester knows how blurry the +contribution is. `combine(local:peers:)` merges the exact local result +with the noised peer results: scores sum per row, ordering is descending +score with row identity as the deterministic tie-break, and the widest +confidence interval seen wins — a deliberately conservative choice, since +the least certain contribution bounds the certainty of the union. + +`PrivacyLedger` tracks per-peer epsilon and delta consumption against a +daily budget: `canConsume(peer:query:)` gates a query, `consume(...)` +records it, and `dailyReset()` starts a new day. Networking, signing, and +scheduling are explicitly the caller's job. One parity nuance is worth +knowing: the Rust leg zeroes the per-lane distance breakdown in a noised +contribution to close a side channel, while the Swift reference forwards +the breakdown unchanged. + +## DPORReduction.swift + +This file provides the differentially private OR-reduction used at the +aggregation point of federated queries: many estates' fingerprints in, +one aggregate fingerprint out, with mathematical privacy protection +layered in. + +`reduce(fingerprints:params:rngSeed:)` works per bit position. For each +of the 256 positions it counts how many contributors set that bit, adds +Laplace noise with scale one over epsilon (drawn from a SplitMix64 seeded +by the caller, so the noise is reproducible for testing), and sets the +output bit only if the noised count reaches the k-anonymity threshold. +The two mechanisms stack: differential privacy means no observer can tell +whether any one estate contributed, and the k-anonymity floor (default +k=3) means no bit lights up from too few contributors even without +adversarial analysis. Empty input returns the zero fingerprint. + +`DPParameters` pins the defaults from the cookbook: epsilon 1.0, delta +1e-9, k 3, with validity preconditions. One documented asymmetry: delta +is validated and stored for API completeness but is not consumed by the +Laplace mechanism itself. The `laplaceNoise(scale:rng:)` helper derives a +uniform value from the top 53 bits of the raw draw — the same convention +the package's other samplers use — and inverts the Laplace distribution's +cumulative curve. + +## Rust Port and Conformance + +The `rust/` directory contains the second leg of the library: crate +`substrate-ml`, with one module per Swift file — thirty-eight modules from +`anomaly.rs` through `viz_graph_signals.rs`, declared in `rust/src/lib.rs` +with the same layering notes and the same production gate on the parked +NMF variant. The crate depends on `substrate-types`, `substrate-kernel`, +and `intellectus-lib`, mirroring the Swift package manifest exactly. + +Most modules open with an explicit "mirror of" header naming their Swift +source, and the determinism-bearing files repeat the pinned constants, +seeds, draw orders, and tie-break rules line for line. Shared gates prove +the agreement property: `rust/tests/distillation_conformance.rs` locks the +delta extractor, the typed decay weights, `featureHash`, and full +pipeline runs against fixed vectors mirrored in Swift's +`DistillationConformanceTests` ("fix the algorithm, not the vector"); +`rust/tests/float_simhash_kernel_equivalence.rs` proves the kernel +projection reproduces the oracle bit for bit across seeds and dimensions; +and `rust/tests/viz_graph_signals_tests.rs` exercises the telemetry emit +of all five graph algorithms under a process-wide lock. Several +algorithms carry additional shared JSON vectors under the engineering +test harness (sampling, temporal causality fold, NMF, community +detection). When you change either leg, change both and run both suites; +the fixtures are the contract. diff --git a/packages/libs/SubstrateML/docs/OVERVIEW.md b/packages/libs/SubstrateML/docs/OVERVIEW.md new file mode 100644 index 0000000..6955ac7 --- /dev/null +++ b/packages/libs/SubstrateML/docs/OVERVIEW.md @@ -0,0 +1,222 @@ +--- +doc: OVERVIEW +package: SubstrateML +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateML/ActionOutcomeMatrix.swift + blob: 612ee840126c72dd0d07505147ba2991ac3bb0b9 + - path: Sources/SubstrateML/AnomalyDetection.swift + blob: ff7d227353ebfc69cf01b5c468b2612d307b8632 + - path: Sources/SubstrateML/AprioriMining.swift + blob: 189a4408f3c6e600a538a6cfc560d24688cbd2af + - path: Sources/SubstrateML/AssociationRuleMining.swift + blob: 9f20faa808883c065412face0063c7682eabbfd6 + - path: Sources/SubstrateML/AuditLogFold.swift + blob: 7423db7764c628cf5fd2aab0e9bb2d98d8c687bd + - path: Sources/SubstrateML/BradleyTerry.swift + blob: 63d335ee50ccee803f0508eb6f3135a29b6545db + - path: Sources/SubstrateML/CommunityDetection.swift + blob: bfba8e0255baeb66d4c79ccfaf34cd11e21a5ebd + - path: Sources/SubstrateML/CompositeDistance.swift + blob: a5f94c631d89c92d45b60e9da43be7fe6bdd19a2 + - path: Sources/SubstrateML/ConceptImplications.swift + blob: 1ac6ccbb70665532f135d2813f260cbbdad21716 + - path: Sources/SubstrateML/DeltaFeatureExtractor.swift + blob: 1ca396968e5e565096dbebb059b45b8fc3e5719f + - path: Sources/SubstrateML/DistillationPipeline.swift + blob: 82ce8bfaaa2e7a92077510ef239a6997a585c888 + - path: Sources/SubstrateML/DistillationScorer.swift + blob: 2cdc6dfe121080d432859d17487f2c8d15fd2910 + - path: Sources/SubstrateML/DPORReduction.swift + blob: 3d3c79e8ce7694b8308bb1c970e86a24de0446b0 + - path: Sources/SubstrateML/EigenvalueCentrality.swift + blob: f427430c34add289231520ba19ae4781316cbd13 + - path: Sources/SubstrateML/FeatureExtractors.swift + blob: 7e6d388a588ff7bec04be631948387a086a253ed + - path: Sources/SubstrateML/FFT.swift + blob: fdfaceeb902428149ead96107ed69c308bca0fff + - path: Sources/SubstrateML/FloatSimHash.swift + blob: 39cd2c4c36492c7cbaf55d6d22a2f32e273cfbae + - path: Sources/SubstrateML/FormalConceptAnalysis.swift + blob: 9cdf9613b30bf2e3a9a85088489c0e2b683a8a77 + - path: Sources/SubstrateML/InformationTheory.swift + blob: f7ba2a27904ceb19eef0fb5b06d1c5aad8b3894b + - path: Sources/SubstrateML/JacobiSVD.swift + blob: fcd924ff0a8409f224b56e7229f59659a6b5f51e + - path: Sources/SubstrateML/LatticeDistance.swift + blob: a7b96a376dfba0c815e4dc91962da1ccaeb2ac1d + - path: Sources/SubstrateML/LLMCalibrationCurve.swift + blob: 4e01882a65570ccce72e2789c23e95c46b6399a7 + - path: Sources/SubstrateML/MatrixDecay.swift + blob: b9595064a54b79f8ddee343385af4b8ebdc7f3e5 + - path: Sources/SubstrateML/MomentSummary.swift + blob: e8c07841c8c3440deb0375500ebcd35a6f116bb4 + - path: Sources/SubstrateML/NMFAlternatingLeastSquares.swift + blob: d64aadc8db165cc3b9f51b4fb1c644a6d6a4ca3c + - path: Sources/SubstrateML/NMFDoubleFrobeniusSquared.swift + blob: 0a725d72b70120828cb5911de594b0278ca450e5 + - path: Sources/SubstrateML/PairingHandshake.swift + blob: cb608882478ec57a6e6ba1f7728646ff3ea610f1 + - path: Sources/SubstrateML/PartialStateRecall.swift + blob: ad7b42bc25f9c4f283d75593d2c42cd546ccf074 + - path: Sources/SubstrateML/RandomWalks.swift + blob: 99f40b1685a876d81f84ca38b4857e5c6ba4e1dd + - path: Sources/SubstrateML/RowAttributeView.swift + blob: f63cf544a87adde435bede527531d1a9414dad03 + - path: Sources/SubstrateML/Sampling.swift + blob: 84fcdcade559229003688a7fc151f5c0a6dac6e6 + - path: Sources/SubstrateML/ShingleSimilarity.swift + blob: bf9dbaa5b0d5a5ee4ec5dfe42a5d65c85bdd0326 + - path: Sources/SubstrateML/TemporalCausalityFold.swift + blob: d99d7358893eeae0a442e0bd4e52f8a9b1cd7f5e + - path: Sources/SubstrateML/TemporalCompression.swift + blob: f34dde022faadd40058e35b62da12c8985763e59 + - path: Sources/SubstrateML/TierAscendingQuery.swift + blob: 8681cbdae9af7facfe7792392dcee15c8f8fe0aa + - path: Sources/SubstrateML/TierContributionFingerprint.swift + blob: bb56b7a82532f8c324168bd81edff6bf68fb33e9 + - path: Sources/SubstrateML/TypedDecayWeighting.swift + blob: 82412f392848c378b5b28221bbddade9f03bf1bf + - path: Sources/SubstrateML/VizGraphSignals.swift + blob: 4f90110f2a19cb92f3d45baf3bc4d82ac9c3dc65 +--- + +# SubstrateML Overview + +## What This Library Does + +SubstrateML is the learning layer of the MOOTx01 substrate. MOOTx01 is an +on-device AI memory system. It stores what an AI observes over time and +helps the AI recall it later. The substrate is the mathematical foundation +under that store: the types, the bit operations, and the algorithms that +every higher layer builds on. + +The substrate is split into layered packages. `SubstrateTypes` holds pure +data types, such as the 256-bit row fingerprint. `SubstrateKernel` holds +hot-path bit operations that run on every capture. SubstrateML — this +package, layer 3 — holds the cold-path algorithms: the math that learns +structure from stored memories rather than storing them. A memory in this +system is a row: one record with a fingerprint, a classification anchor, +and a set of bitmap fields. SubstrateML never touches storage. Every +function here takes plain values in and returns plain values out. + +Most of these algorithms run during dreaming. Dreaming is the system's +idle-time maintenance cycle: a background daemon that clusters, decays, +summarizes, and re-scores the memory estate while the device is otherwise +quiet. An estate is one user's complete memory store. The algorithms here +find the estate's themes, its communities, its habits, its rules, and its +condensable clusters — and they forget on schedule, so old evidence fades. + +## The Problem It Solves + +An on-device memory system must learn without a cloud. Cloud machine +learning changes without notice, needs a network, and sees private data. +SubstrateML instead ships small, exact, well-bounded reference algorithms +that run entirely on the device. + +It must also learn identically everywhere. MOOTx01 estates can federate, +which means separate devices share and compare results. Two devices that +factor the same matrix or mine the same rules must get the same answer, or +shared recall falls apart. SubstrateML therefore holds one agreement +property across two implementations: a Swift leg for Apple platforms and a +Rust leg (in `rust/`) for everything else. Both legs use the same canonical +pseudo-random number generator (SplitMix64), the same pinned seeds, the +same tie-breaking rules, and the same arithmetic order, so results are +bit-identical. Conformance fixtures — recorded input and output pairs both +legs must reproduce exactly — gate every change. + +Two library-wide rules protect that promise. First, SubstrateML never reads +a clock; every timestamp is passed in by the caller. Second, no algorithm +here holds hidden state; everything is a pure function or a plain value +type, safe to run from any thread. + +## How It Works + +The thirty-eight source files form seven working groups. + +**Fingerprint and distance math.** A fingerprint is a short fixed-size code +computed from content; similar content gives similar fingerprints. +`FloatSimHash` projects float embedding vectors from external models into +the substrate's 256-bit fingerprint form. `CompositeDistance` blends +classification distance and fingerprint distance into the one score recall +ranks by. `LatticeDistance`, `PartialStateRecall`, and `ShingleSimilarity` +supply specialized distances. `MomentSummary` and `TemporalCompression` +OR-reduce many row fingerprints into one signature per time window, so the +system can ask "what was going on during this hour" as a single comparison. + +**Ingestion shaping.** `FeatureExtractors` turns raw ambient sensor samples +(health, location, calendar, screen time, telemetry) into fingerprinted +rows. `AuditLogFold` replays a row's append-only change log to reconstruct +its state at any point in time. `RowAttributeView` reshapes that same log +into flat attribute lists that the pattern miners consume. + +**Learning and decay.** `MatrixDecay` applies exponential half-life decay +to every statistics matrix — the system's forgetting mechanism. +`ActionOutcomeMatrix` tracks which actions succeed. `BradleyTerry` learns +per-row ranking strength from recall feedback. `LLMCalibrationCurve` tracks +how honest an LLM's confidence claims are. `Sampling` provides the +deterministic Normal, Gamma, and Beta samplers under Thompson-sampling +decisions. + +**Graph analytics.** The estate graph connects rows by association. +`CommunityDetection` (Louvain) finds its clusters. `EigenvalueCentrality` +scores each row's authority for keystone recall. `RandomWalks` wanders the +graph for exploratory recall. `NMFAlternatingLeastSquares` factors the +matrices into latent themes. `AnomalyDetection` flags unusual values. +`JacobiSVD` and `FFT` supply the deterministic linear algebra and rhythm +analysis beneath semantic embeddings and periodicity detection. These five +graph algorithms emit telemetry signals, named in `VizGraphSignals`, when +monitoring is enabled. + +**Pattern mining.** `AssociationRuleMining` and `AprioriMining` find +"when A is set, B tends to be set" rules. `FormalConceptAnalysis` and +`ConceptImplications` find exact groupings and always-true implications. +`TemporalCausalityFold` mines "X changed, then Y changed N minutes later" +statistics. `InformationTheory` supplies the entropy and divergence math. + +**Distillation.** Distillation compresses a cluster of related memories +into one condensed factoid. `DistillationScorer` decides whether a cluster +is coherent enough. `DeltaFeatureExtractor` and `TypedDecayWeighting` +handle trends and staleness. `DistillationPipeline` runs the whole +five-stage algorithm and emits the factoid plus its fingerprint. + +**Federation and privacy.** `PairingHandshake` lets two estates derive a +shared fingerprint basis with no extra network round trip. +`TierContributionFingerprint` packs an estate's shareable summary into a +fixed 64-byte wire format. `TierAscendingQuery` and `DPORReduction` add +differential-privacy noise and k-anonymity so aggregate answers never +expose any single estate's contribution. + +## How the Pieces Fit + +Figure 1 shows the library's topology — its major parts and how data moves +between them. + +![Figure 1. Topology of SubstrateML](topology.svg) + +*Figure 1. Topology of SubstrateML. Rows, audit events, and sensor samples +enter on the left. The shaping layer feeds the four algorithm families. +Dashed regions mark the substrate packages below and the telemetry seam; +all outputs return to the calling kits, never to storage.* + +SubstrateML has no single facade. Each algorithm family is its own entry +point, and the consuming kits — LocusKit, CognitionKit, GeniusLocusKit, +NeuronKit, and the dreaming daemon — call the piece they need. A kit is a +larger package that composes libraries into a subsystem; kits depend on +libs, never the reverse. The package depends downward only: on +`SubstrateTypes` for shared types, on `SubstrateKernel` for dispatched bit +kernels, and on `IntellectusLib`, the zero-dependency telemetry leaf. When +monitoring is off (the default), every telemetry emit costs one atomic +boolean load and nothing more. + +## What Ships in the Package + +The package ships the thirty-eight Swift sources, a matching test suite, +and the Rust port in `rust/` (crate `substrate-ml`, one module per Swift +file plus shared conformance tests). There are no bundled data artifacts; +every input is caller-supplied. Determinism is carried instead by pinned +constants — seeds, thresholds, half-lives, and bucket tables recorded in +the sources and locked by conformance vectors — so the same input produces +the same result on every platform, every time. diff --git a/packages/libs/SubstrateML/docs/topology.svg b/packages/libs/SubstrateML/docs/topology.svg new file mode 100644 index 0000000..d90fde6 --- /dev/null +++ b/packages/libs/SubstrateML/docs/topology.svg @@ -0,0 +1,162 @@ + + + + + + + + + + + + + + + + SubstrateML: cold-path learning over the memory estate + + + + Caller inputs + rows · audit events + sensor samples · matrices + + + + Ingestion shaping + FeatureExtractors · AuditLogFold + RowAttributeView + + + + Fingerprint + distance + FloatSimHash · Composite/Lattice + PartialState · Moment/Temporal + + + + Graph analytics + Louvain · Centrality · Walks + NMF · SVD · FFT · Anomaly + + + + Pattern mining + Association · Apriori · FCA + Implications · TemporalFold + + + + Distillation + Scorer · DeltaExtractor + TypedDecay · Pipeline + + + + Learning + decay + MatrixDecay · BradleyTerry + ActionOutcome · Calibration + + + + Federation privacy + Pairing · TierContribution + TierQuery · DP OR-reduce + + + + Consuming kits + LocusKit · CognitionKit · NeuronKit + GeniusLocusKit · dreaming daemon + + + + + + + + + + + + + + + + + + Substrate layers below (types + dispatched kernels) + + SubstrateTypes + Fingerprint256 · HLC · SplitMix64 + + SubstrateKernel + SimHash project · orReduce256 + + + + IntellectusLib telemetry + + VizGraphSignals + community.assignment + centrality.score · nmf.factor + anomaly.flag · edge.decayed_weight + + + + + + + + + diff --git a/packages/libs/SubstrateTypes/docs/AGENT_MAP.md b/packages/libs/SubstrateTypes/docs/AGENT_MAP.md new file mode 100644 index 0000000..ac1d88b --- /dev/null +++ b/packages/libs/SubstrateTypes/docs/AGENT_MAP.md @@ -0,0 +1,292 @@ +--- +doc: AGENT_MAP +package: SubstrateTypes +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateTypes/AsOfCoordinate.swift + blob: b4433864e36b5803533266c97b153a836ee04adb + - path: Sources/SubstrateTypes/AuditEvent.swift + blob: fbfabeb4eaba4bf43b6e45d8c2b772d994a12b7b + - path: Sources/SubstrateTypes/BitwiseArithmetic.swift + blob: a6270b0f7395c17915c54fdc119e854ca5a44077 + - path: Sources/SubstrateTypes/BlockMask.swift + blob: 904c2179be96d5c20472f0f356947e82e9d8cf37 + - path: Sources/SubstrateTypes/ContentHash.swift + blob: 835744792dc191e824e676d8715c96111b4945ff + - path: Sources/SubstrateTypes/CountVector256.swift + blob: edabb34cb02005135d745ffa6da5541835159927 + - path: Sources/SubstrateTypes/Fingerprint256.swift + blob: 465607fe4164bb0dd94009dbbb20d6e766b973a8 + - path: Sources/SubstrateTypes/FloatSimHashPlanes.swift + blob: b5506e7d12fd153b721f4ca7a52a65a6667ea2f6 + - path: Sources/SubstrateTypes/FNV.swift + blob: f3d8874705ad66f513e49ccbf0c714980a1172e8 + - path: Sources/SubstrateTypes/GSetAuditLog.swift + blob: 1a4452e4c8a147d4fac299a3319b0cb4e70cb795 + - path: Sources/SubstrateTypes/Hamming.swift + blob: dc899742f77434910d51bb3cd830cc9603f56f0a + - path: Sources/SubstrateTypes/HLC.swift + blob: a74442251bd542e8f403029ee475c2ac9d7b91b0 + - path: Sources/SubstrateTypes/HyperplaneFamily.swift + blob: f30c4640629116b1952df48790ed6f5ae67e9025 + - path: Sources/SubstrateTypes/LatticeAnchor.swift + blob: 5f3ba24efa5a85109d20f3b61e6acb17e21cb0a5 + - path: Sources/SubstrateTypes/MatrixC.swift + blob: a8612ea7b09e6d4d93a4acbeedbe2b3b91776a74 + - path: Sources/SubstrateTypes/MatrixF.swift + blob: ed6e38a78c654562b04074878bf86686da0172c3 + - path: Sources/SubstrateTypes/MatrixO.swift + blob: c10fa0b14d8abd172503d0375dfab9302da32025 + - path: Sources/SubstrateTypes/MatrixT.swift + blob: cd31ca818be6446f8f1cd98c6b912c640a78c613 + - path: Sources/SubstrateTypes/MerkleDomain.swift + blob: 4e7d1888a04b2e9979e62c58b54ba698712e56d5 + - path: Sources/SubstrateTypes/MerkleRoot.swift + blob: 1979925f4646cdbed8785a43e2f2a14a7f9c0cac + - path: Sources/SubstrateTypes/NounType.swift + blob: 940d6c51329a66849e572a95ff8ae51648784f2f + - path: Sources/SubstrateTypes/ORReduce.swift + blob: 3f9f6e6ed30fb233f48457f95047f274bd271745 + - path: Sources/SubstrateTypes/RecallTypes.swift + blob: 7f01133df8495156fab87f1f4e441fb7867b3434 + - path: Sources/SubstrateTypes/Row.swift + blob: d7cf86f774dbccbbdf9daf1f3a5ed9be29fb24e8 + - path: Sources/SubstrateTypes/RowBitmaps.swift + blob: b879191a7076807c6fcd20e54cf806a639e63388 + - path: Sources/SubstrateTypes/RowState.swift + blob: 3f8c772be1fe1e3b44b90a5ea8681d4ce4326db7 + - path: Sources/SubstrateTypes/SimHash.swift + blob: e32a8b685c3db29408ed9e7fb21da4a7da10c467 + - path: Sources/SubstrateTypes/SnapshotId.swift + blob: b476bad15abecba2dacea225e0a7db7f445be777 + - path: Sources/SubstrateTypes/ThreeDBitTensor.swift + blob: 47b53642dda804afe8fbc0d863dc4ccd11d3ec55 + - path: Sources/SubstrateTypes/TimeRange.swift + blob: e1b7fbed94b6505b596ac5701cfd7c693e7d2425 +--- + +# AGENT_MAP — SubstrateTypes + +PURPOSE: layer 1 of the four-package substrate split. Pure data shapes (Row, +Fingerprint256, LatticeAnchor, HLC, RowState/RowBitmaps, AuditEvent) plus +canonical reference compute logically inseparable from those shapes +(Hamming, SimHash, FNV, ORReduce, BitwiseArithmetic, CountVector256). Zero +transcendentals, zero I/O, zero pinned data artifacts. Consumers needing +only substrate-shape (e.g. ConvergenceKit) depend on this package alone. + +DEPS: imports Foundation only (no intra-SDK imports — this is the dependency +floor). Imported by: SubstrateKernel, SubstrateML, SubstrateLib, EngramLib, +IntellectusLib, and the root app package (Package.swift). SubstrateLib +RETAINS the row-state automaton, the verb mechanics, and the full +multi-row substrate object — those are compute, not shape, and stay one +layer up. Rust port in `rust/src/` mirrors every module 1:1 (see DEPS note +in Rust Port and Conformance, DETAILS.md); no fixture directory — agreement +is verified by matched unit-test vectors in each module, not a shared +fixtures/ dir. + +ENTRY POINTS (most callers need only these): +- Row.swift:23 `struct Row` — the substrate's central data shape +- Fingerprint256.swift:43 `struct Fingerprint256` — 256-bit structural coordinate +- LatticeAnchor.swift:12 `struct LatticeAnchor` — topic coordinate (hash of a lattice code + Q-ID) +- HLC.swift:37 `struct HLC` — ordering coordinate across replicas +- Hamming.swift:37 `Hamming.distance(_:_:blocks:)` — structural-similarity primitive +- GSetAuditLog.swift:134 `struct GSetAuditLog` — CRDT audit trail + +## Symbol Table + +### Row shape and lifecycle +- Row.swift:19 `typealias RowId = UUID` — wire-identical to Rust `RowId(u128)` +- Row.swift:23 `struct Row: Sendable` — id, nounType, state, 3 bitmap columns, fingerprint, latticeAnchor, lineageId?, content? +- NounType.swift:12 `enum NounType: UInt8` — drawer=0/tunnel=1/kgFact=2/diaryEntry=3/proposal=4/association=5/learnedReference=6/ambientSample=7; wire-stable, never renumber +- RowState.swift:35 `enum RowState: UInt8` — 10 scale-gapped states (active=0..accepted=3, superseded=16..expired=19, rejected=32/tombstoned=33) +- RowState.swift:63 `enum RowStateCluster: UInt8` — a=active/becoming, b=superseded/historical, c=terminal +- RowState.swift:98 `RowState.cluster: RowStateCluster` — `(rawValue >> 4) & 0x3`; canonical active/retired partition, never re-derive by hand +- RowState.swift:90 `RowState.activeClusterUpperBoundRaw = 16` — the one boundary constant for storage-layer predicates that can't call `cluster` +- RowState.swift:115 `RowState.cluster(ofRawState:)` — classify from a raw byte, nil if undefined +- RowState.swift:122 `enum RowVerb: String` — 12 verbs the (SubstrateLib) automaton accepts +- RowState.swift:137 `enum RowStateError` — `.illegalTransition(RowState, RowVerb)` / `.violatesInvariant(String)` +- RowBitmaps.swift:37 `struct RowBitmaps: Sendable, Hashable, Codable` — adjective/operational/provenance Int64 columns +- RowBitmaps.swift:65 `RowBitmaps.field(_:) -> UInt8` — 36-field/6-bit uniform-grid view over the 3 named Int64 columns; UInt64 cast before shift (avoids Int64 arithmetic-shift sign bleed) +- RowBitmaps.swift:110 `RowBitmaps.fieldValues() -> [(field,value)]` — feeds MatrixO.applyRow / MatrixT.applyPair +- RowBitmaps.swift:151 `struct BitVector216: Sendable, Hashable` — dense 216-bit view; feeds MatrixF +- RowBitmaps.swift:167 `BitVector216.init(rowBitmaps:)` — real-row path, bits 60-71 always zero +- RowBitmaps.swift:191 `BitVector216.init(presenceBytes:)` — raw 27-byte harness/wire path, can set bits 60-71 + +### Lattice anchor (topic coordinate) +- LatticeAnchor.swift:12 `struct LatticeAnchor: Hashable, Sendable` — udcCode: UInt64, qidPointer: UInt64 (0 = null) +- LatticeAnchor.swift:21 `isNull: Bool` — both halves zero +- LatticeAnchor.swift:41 `LatticeAnchor.udc(_:)` — FNV-1a64 hash of a UDC string, null Q-ID +- LatticeAnchor.swift:50 `LatticeAnchor.udcQid(_:qid:)` — hashes both; empty qidString ⇒ same as `udc(_:)` + +### Time coordinates +- HLC.swift:37 `struct HLC: Hashable, Sendable, Codable, Comparable` — (physicalTime, logicalCount, nodeID), lexicographic order +- HLC.swift:58 `HLC.zero` — sentinel earliest value +- HLC.swift:77 `HLC.advanced()` — bump logicalCount, hold physical/node +- HLC.swift:99 `HLC.packed: UInt64` / :108 `init(packed:)` — lossy 8-byte federation form (40-bit ms ≈ 34yr range) +- HLC.swift:206 `HLC.wireBytes` / :224 `init(wireBytes:)` — lossless 16-byte form +- HLC.swift:125 `struct HLCGenerator: Sendable` — per-replica HLC state machine (Kulkarni et al. 2014) +- HLC.swift:139 `HLCGenerator.send(now:)` — local-event timestamp; strictly monotonic per replica +- HLC.swift:156 `HLCGenerator.receive(remote:now:)` — merge rule on message receipt; underlies GSetAuditLog convergence +- TimeRange.swift:11 `struct TimeRange: Sendable, Equatable` — closed [start,end] HLC interval; init preconditions end >= start +- TimeRange.swift:24 `contains(_:) -> Bool` — inclusive both ends +- AsOfCoordinate.swift:18 `enum AsOfCoordinate: Hashable, Sendable, Codable` — `.present` | `.asOf(HLC)`; enum discriminant prevents zero-HLC ⇔ "now" ambiguity +- SnapshotId.swift:15 `struct SnapshotId: Hashable, Sendable, Codable` — UUID wrapper, type-distinct from RowId/drawer/estate ids + +### Fingerprint construction +- Fingerprint256.swift:43 `struct Fingerprint256: Hashable, Sendable, Codable` — block0..block3: UInt64 (256 bits); block0=Bitmap-LSH, block1=Lattice-LSH, block2=Lineage+Temporal, block3=Channel+Source +- Fingerprint256.swift:59 `Fingerprint256.zero` — OR-identity / null-block value +- Fingerprint256.swift:88 `with(bit:set:) -> Fingerprint256` — value-semantic bit set/clear (replaces old mutating setBit) +- Fingerprint256.swift:120 `union(_:) -> Fingerprint256` — bitwise OR +- Fingerprint256.swift:153 `Fingerprint256.fromBits(_:)` — from [Bool] of 256 +- Fingerprint256.swift:166/178 `wireBytes` / `init(wireBytes:)` — 32-byte LE canonical form; `toBytes()`/`fromBytes(_:)` are throws-free aliases +- Fingerprint256.swift:252 `zip4(_:_:)` — pairwise per-block binary op; the ONE unroll every block-wise op in this package (and Hamming/ORReduce/BitwiseArithmetic) must route through +- Fingerprint256.swift:270 `Fingerprint256.reduce4(_:_:)` — block-wise fold over a fingerprint sequence +- Fingerprint256.swift:281 `map4(_:)` — per-block unary op +- Fingerprint256.swift:296 `popcount() -> Int` — sum of 4 block popcounts +- Fingerprint256.swift:313/331 `zip4Batch` / `map4Batch` — vectorize across row arrays, not across one fingerprint's blocks +- HyperplaneFamily.swift:30 `struct Hyperplane: Sendable, Codable, Equatable` — ±1 plane as (positiveMask, negativeMask) bit-pair; 0/0 = sparse zero entry +- HyperplaneFamily.swift:53 `sign(over:) -> Bool` — popcount(v&pos) > popcount(v&neg); tie ⇒ false +- HyperplaneFamily.swift:68 `struct HyperplaneFamily: Sendable, Codable, Equatable` — exactly 64 planes, fixed blockIndex 0..3 + inputBitLength (192 for block0, 64 for 1-3) +- HyperplaneFamily.swift:95 `HyperplaneFamily.generate(seed:blockIndex:inputBitLength:density:)` — deterministic from 32-byte seed; CONSTITUTIONAL: seeds fixed at estate creation, never rotate +- HyperplaneFamily.swift:147 `canonicalHash() -> UInt64` — FNV-1a over canonical wire form, for pairing-audit labels +- HyperplaneFamily.swift:247 `HyperplaneFamily.blockFamilies(baseSeed:density:)` — canonical 4-family generator; fixes prior bug of reusing 1 seed across blocks + assuming uniform 64-bit width +- HyperplaneFamily.swift:261 `diversifiedSeed(base:blockIndex:)` / :273 `expandSeed64(_:)` — per-block seed derivation (SplitMix64-style) +- FloatSimHashPlanes.swift:30 `struct FloatSimHashPlanes: Sendable, Equatable` — materialized ±1 planes for the FLOAT-input SimHash path; lives here so SubstrateKernel (applies) and SubstrateML (generates) both reach it without a layering inversion +- SimHash.swift:37 `SimHash.block(over:family:) -> UInt64` — one 64-bit block: bit k = sign() +- SimHash.swift:52 `SimHash.fingerprint(bitmapInput:latticeInput:lineageTemporalInput:channelSourceInput:families:)` — assembles all 4 blocks; families order MUST be H_0..H_3 +- SimHash.swift:74 `SimHash.fingerprintBatch(...)` — reference loop; hardware backends must match bit-for-bit +- SimHash.swift:102 `SimHash.fingerprint(fromSubhashes:hyperplanes:)` — 4-subhash convenience path +- SimHashInput (SimHash.swift:123) — :128 `bitmap(...)`, :140 `lattice(...)`, :156 `lineageTemporal(...)`, :178 `channelSource(...)` — fixed bit-offset input-vector assemblers per cookbook §3.2-3.5; offset mismatch ⇒ silently incompatible fingerprints + +### Fingerprint algebra +- Hamming.swift:21 `typealias HammingDistance = Hamming` +- Hamming.swift:37 `Hamming.distance(_:_:blocks:) -> Int` — XOR+popcount; `.all` fast path is `zip4(^).popcount()` +- Hamming.swift:54 `Hamming.similarity(_:_:blocks:) -> Double` — 1 - distance/max, max = 64*blockCount +- BlockMask.swift:26 `struct BlockMask: OptionSet` — .block0/.block1/.block2/.block3, .all, .none; replaced a Set-per-call allocation hot-path +- BlockMask.swift:42 `blockCount: Int` — popcount of rawValue +- BitwiseArithmetic.swift:31 `intersect(_:_:)` — AND (zip4 &) +- BitwiseArithmetic.swift:45 `difference(_:_:)` — XOR (zip4 ^) +- BitwiseArithmetic.swift:63 `prototype(_:) -> Fingerprint256` — cohort majority vote via CountVector256.fold(...).majorityVote() +- BitwiseArithmetic.swift:77 `indirect enum FingerprintBuilder` — literal/intersect/difference/prototypeOf; `.evaluate()` walks the tree +- ORReduce.swift:31 `ORReduce.reduce(_:) -> Fingerprint256` — OR-fold, zero for empty (identity); delegates to `Fingerprint256.reduce4(_:|:)` +- ORReduce.swift:46 `ORReduce.reduce(_:blocks:defaults:)` — block-subset OR, unselected blocks pass through `defaults` +- CountVector256.swift:41 `struct CountVector256: Sendable, Equatable, Codable` — counts:[UInt32] (256), n:UInt32; wrapping arithmetic (`&+=`) +- CountVector256.swift:52 `CountVector256.zero` — fold/merge identity +- CountVector256.swift:75 `accumulate(_:)` — fold one fingerprint (leaf step) +- CountVector256.swift:94/103 `merge(_:)` / `+` — commutative/associative tree-fold step; order-independent +- CountVector256.swift:120 `majorityVote() -> Fingerprint256` — bit set iff `2*count > n` (strict; exact tie does NOT set); n==0 ⇒ zero +- CountVector256.swift:134 `profile() -> [Float]` — per-bit Bernoulli parameter, not stored, recomputed on demand +- CountVector256.swift:148 `CountVector256.fold(_:)` — reference whole-array accumulate; kernel-layer vectorized backends gate against this +- FNV.swift:25 `FNV.hash64(_:) -> UInt64` — offset 0xCBF29CE484222325, prime 0x100000001B3 +- FNV.swift:40 `FNV.hash32(_:) -> UInt32` — INDEPENDENT hash family, not a truncation of hash64 +- FNV.swift:57 `FNV.hash16(_:) -> UInt16` — low-16 fold of hash64 (NOT a from-scratch FNV variant) + +### Integrity hashing +- MerkleDomain.swift:20 `enum MerkleDomain` — leaf=0x00, interior=0x01, tombstone=0x02, commitment=0x03; conformance-frozen forever +- ContentHash.swift:20 `struct ContentHash: Hashable, Sendable, Codable` — 32-byte SHA-256 leaf-payload digest; private storage, `bytes`/`hexString` accessors +- ContentHash.swift:45 `ContentHash.tombstone` — literal bytes (SHA256 of MerkleDomain.tombstone); literal because layer-1 can't import SubstrateKernel's SHA256 +- MerkleRoot.swift:16 `struct MerkleRoot: Hashable, Sendable, Codable` — 32-byte subtree-summary hash; NOT interchangeable with ContentHash (type-distinct on purpose) +- MerkleRoot.swift:41 `MerkleRoot.empty` — literal bytes (SHA256 of MerkleDomain.interior), same layering reason + +### Audit trail (CRDT) +- AuditEvent.swift:15 `struct AuditEvent: Sendable` — eventID+hlc = idempotence key; before/afterBitmaps, before/afterLatticeAnchor, actor, reason? +- AuditEvent.swift:63 `withReason(_:) -> AuditEvent` — attach a reason post hoc (AuditGate.admit itself is reason-less) +- GSetAuditLog.swift:35 `struct AuditEntry: Hashable, Sendable, Codable` — id = 32-byte content hash (SHA-256 over wire fields); dedupe key +- GSetAuditLog.swift:62 `enum AuditVerb: String` — 9 cookbook verbs + migrate/dreamCompact +- GSetAuditLog.swift:95 `enum AuditValue: Hashable, Sendable, Codable` — bitmap/string/fingerprint/integer; HAND-WRITTEN Codable ⇒ externally-tagged `{"bitmap":42}` shape matching Rust serde, NOT Swift's default `{"bitmap":{"_0":42}}` +- GSetAuditLog.swift:134 `struct GSetAuditLog: Sendable, Codable` — internal store keyed by id (O(1) dedupe); WIRE FORM is a sorted array (`{"entries":[...]}`), not the hash map +- GSetAuditLog.swift:180 `add(_:)` — idempotent insert +- GSetAuditLog.swift:187 `merge(_:)` — CRDT join = set union; commutative/associative/idempotent +- GSetAuditLog.swift:198 `orderedEntries` — full HLC-order replay (drives projection) +- GSetAuditLog.swift:204 `entries(forRow:)` — HLC-order, one row (drives row-state automaton in SubstrateLib) +- GSetAuditLog.swift:212 `entries(since:)` — HLC-order, exclusive cutoff (sync delta) + +### Population-statistics matrices +- MatrixF.swift:24 `struct MatrixF: Sendable, Equatable` — 216-cell Int64 field-presence counts; NO decay +- MatrixF.swift:31 `fieldCount`/`bitsPerField`/`cellCount` — ALIASES onto RowBitmaps' constants (must match by definition) +- MatrixF.swift:76 `applyRow(delta:bitVector:)` — +1 capture / -1 expunge / two calls for mutate +- MatrixC.swift:26 `struct MatrixC: Sendable, Equatable` — 216-cell Float32 marginal probabilities; derived only, NO decay +- MatrixC.swift:67 `MatrixC.derive(from:nRows:)` — Int64→Double→Float32 cast order is the cross-language bit-identity contract; nRows==0 ⇒ all zero +- MatrixO.swift:38 `struct CooccurrenceKey: Hashable, Comparable, Sendable` — packed UInt32 (fieldI|valueI|fieldJ|valueJ, 8 bits each) +- MatrixO.swift:70 `struct MatrixO: Sendable, Equatable` — sorted sparse cell list (NOT a Dictionary — deterministic iteration/serialization); decay half-life 365 days (applied elsewhere, by MatrixDecay) +- MatrixO.swift:165 `applyRow(delta:fieldValues:)` — ALL ordered pairs incl. i==j; NOT required symmetric in storage (unordered pair ⇒ 2 cells) +- MatrixT.swift:43 `struct CausalityKey: Hashable, Comparable, Sendable` — packed UInt64, adds lagBucket (0..7) to CooccurrenceKey's shape +- MatrixT.swift:84 `struct MatrixT: Sendable, Equatable` — ASYMMETRIC by design (source→target ≠ target→source); decay half-life 90 days +- MatrixT.swift:89 `bucketEdgesMinutes = [1,2,4,8,16,32,64,128]` — log-spaced lag buckets +- MatrixT.swift:99 `MatrixT.lagBucket(forMinutes:)` — nil outside [1,256); applyPair is then a no-op +- ThreeDBitTensor.swift:36 `struct ThreeDBitTensor: Sendable` — 6 bit-slices (one per bit position), row-major within each slice; hot-path layout, ~27MiB at 1M rows +- ThreeDBitTensor.swift:71 `setValue(row:field:value:)` — precondition value < 64 (6-bit width, I-6) +- ThreeDBitTensor.swift:107 `scanFieldEquals(field:value:) -> [UInt8]` — O(rowCount/8) per bit-slice, NOT yet word-vectorized +- ThreeDBitTensor.swift:140 `reserveCapacity(_:)` — grow-only, no-op if not larger, preserves existing data + +### Recall wire vocabulary +- RecallTypes.swift:55 `struct RecallScore: Equatable, Sendable` — (RowId, Float32); score meaning is PER-PRIMITIVE, normalize before combining +- RecallTypes.swift:69 `struct DistanceBreakdown: Equatable, Sendable` — lattice/fingerprint/temporal/bitmap contributions, each [0,1] +- RecallTypes.swift:89 `struct RecallResult: Sendable` — rows + breakdown + confidenceInterval? + primitiveName (drives RRF/MMR composition) +- RecallTypes.swift:112 `struct RowProjection: Sendable` — rowId/captureHLC/fingerprint/lattice/bitmaps/rowState; DELIBERATELY omits verbatim content + rung-2 metadata + +## INVARIANTS / GOTCHAS + +- LAYERING FLOOR: this package imports Foundation only. Never add an + intra-SDK import here — that is what would invert the four-package split + (SubstrateTypes → SubstrateKernel → SubstrateML, with SubstrateLib + orchestrating all three). `ContentHash.tombstone` and `MerkleRoot.empty` + are byte literals, NOT computed SHA-256 calls, for exactly this reason; + a SubstrateKernel bridge test verifies each literal against a live hash. +- `Fingerprint256.zip4(_:_:)` is the ONE place the four-block unroll is + expressed. `Hamming.distance`, `ORReduce.reduce`, and + `BitwiseArithmetic.intersect/difference` all delegate to it. Do not + reintroduce a hand-unrolled block0/block1/block2/block3 loop anywhere else. +- `RowBitmaps.field(_:)` casts to `UInt64` before shifting. Do not shift the + raw `Int64` — arithmetic (sign-extending) right shift on a negative Int64 + bleeds the sign bit into the field value. Also guards `shift >= 64` + explicitly; Swift masks shift amounts by 63 and would otherwise compute + the wrong result for fields 11 (shift=66) silently. +- `RowState`'s ten raw values are scale-gapped (0-3 / 16-19 / 32-33) on + purpose so `cluster` is `(raw >> 4) & 0x3`. Never classify active-vs- + retired by a hand-rolled numeric boundary except via the one named + constant `activeClusterUpperBoundRaw`, which exists only for storage + predicates that cannot call `cluster(ofRawState:)`. +- `HyperplaneFamily` seeds are CONSTITUTIONAL: fixed at estate creation, + never rotated. Rotating changes every fingerprint the estate has ever + produced and breaks CRDT convergence across replicas. `blockFamilies` + is the only sanctioned way to derive all four block families from one + base seed — do not reuse one un-diversified seed across blocks. +- `CountVector256.majorityVote()` uses STRICT `2*count > n`; an exact tie + does not set the bit. This convention is identical across every kernel + backend and both language ports — do not change the inequality direction. +- `GSetAuditLog`'s wire format is a sorted array (`{"entries":[...]}`), not + its internal `[id: AuditEntry]` dictionary. `AuditValue`'s Codable is + hand-written to match Rust serde's externally-tagged shape. Do not let + either fall back to Swift's default synthesized Codable. +- `MatrixF`/`MatrixC` have NO decay; `MatrixO` decays at 365-day half-life; + `MatrixT` decays at 90-day half-life. Decay application itself lives + outside this package (MatrixDecay); these types only hold and update raw + counts/derived marginals. +- `MatrixC.derive(from:nRows:)` casts Int64 → Double → Float32, in that + order, on both language ports. This is the cross-language bit-identity + contract for the correlation matrix; do not shortcut through Float32 + division directly. +- `MatrixT` is asymmetric by design: `(i,j)` and `(j,i)` are independent + cells. `MatrixO` is conceptually symmetric but stores both directions + from `applyRow`'s ordered-pair iteration (including the diagonal, i==j). +- `RowId` is a bare `typealias` for `UUID` (wire-identical to Rust's + `RowId(u128)`); `SnapshotId` is a real wrapper struct on purpose, so it is + NOT interchangeable with a bare UUID at the type level. Do not "simplify" + `SnapshotId` into another typealias. +- `ContentHash` and `MerkleRoot` are both 32-byte hashes but are distinct, + non-interchangeable types (payload digest vs. subtree summary). Same + pattern for `RecallScore.score` — meaning is per-primitive, never compare + raw scores across primitives without normalizing first. +- No pinned data artifacts ship in this package (contrast with LatticeLib). + Every value is either caller-supplied (a seed, a row's fields) or a pure + function of caller-supplied input. Reproducibility rests entirely on the + arithmetic being pinned and mirrored in `rust/src/`, checked by matched + unit-test vectors per module, not a shared fixtures directory. +- `Row`, `AuditEvent`, `RowState`/`RowVerb`/`RowStateError`, `NounType`, + and `LatticeAnchor` are pure data only. The row-state automaton + (transition table + validation), the verb implementations, and the + full multi-row substrate object all remain in SubstrateLib. Do not + add behavior to these types here. diff --git a/packages/libs/SubstrateTypes/docs/DETAILS.md b/packages/libs/SubstrateTypes/docs/DETAILS.md new file mode 100644 index 0000000..e77a9fc --- /dev/null +++ b/packages/libs/SubstrateTypes/docs/DETAILS.md @@ -0,0 +1,821 @@ +--- +doc: DETAILS +package: SubstrateTypes +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateTypes/AsOfCoordinate.swift + blob: b4433864e36b5803533266c97b153a836ee04adb + - path: Sources/SubstrateTypes/AuditEvent.swift + blob: fbfabeb4eaba4bf43b6e45d8c2b772d994a12b7b + - path: Sources/SubstrateTypes/BitwiseArithmetic.swift + blob: a6270b0f7395c17915c54fdc119e854ca5a44077 + - path: Sources/SubstrateTypes/BlockMask.swift + blob: 904c2179be96d5c20472f0f356947e82e9d8cf37 + - path: Sources/SubstrateTypes/ContentHash.swift + blob: 835744792dc191e824e676d8715c96111b4945ff + - path: Sources/SubstrateTypes/CountVector256.swift + blob: edabb34cb02005135d745ffa6da5541835159927 + - path: Sources/SubstrateTypes/Fingerprint256.swift + blob: 465607fe4164bb0dd94009dbbb20d6e766b973a8 + - path: Sources/SubstrateTypes/FloatSimHashPlanes.swift + blob: b5506e7d12fd153b721f4ca7a52a65a6667ea2f6 + - path: Sources/SubstrateTypes/FNV.swift + blob: f3d8874705ad66f513e49ccbf0c714980a1172e8 + - path: Sources/SubstrateTypes/GSetAuditLog.swift + blob: 1a4452e4c8a147d4fac299a3319b0cb4e70cb795 + - path: Sources/SubstrateTypes/Hamming.swift + blob: dc899742f77434910d51bb3cd830cc9603f56f0a + - path: Sources/SubstrateTypes/HLC.swift + blob: a74442251bd542e8f403029ee475c2ac9d7b91b0 + - path: Sources/SubstrateTypes/HyperplaneFamily.swift + blob: f30c4640629116b1952df48790ed6f5ae67e9025 + - path: Sources/SubstrateTypes/LatticeAnchor.swift + blob: 5f3ba24efa5a85109d20f3b61e6acb17e21cb0a5 + - path: Sources/SubstrateTypes/MatrixC.swift + blob: a8612ea7b09e6d4d93a4acbeedbe2b3b91776a74 + - path: Sources/SubstrateTypes/MatrixF.swift + blob: ed6e38a78c654562b04074878bf86686da0172c3 + - path: Sources/SubstrateTypes/MatrixO.swift + blob: c10fa0b14d8abd172503d0375dfab9302da32025 + - path: Sources/SubstrateTypes/MatrixT.swift + blob: cd31ca818be6446f8f1cd98c6b912c640a78c613 + - path: Sources/SubstrateTypes/MerkleDomain.swift + blob: 4e7d1888a04b2e9979e62c58b54ba698712e56d5 + - path: Sources/SubstrateTypes/MerkleRoot.swift + blob: 1979925f4646cdbed8785a43e2f2a14a7f9c0cac + - path: Sources/SubstrateTypes/NounType.swift + blob: 940d6c51329a66849e572a95ff8ae51648784f2f + - path: Sources/SubstrateTypes/ORReduce.swift + blob: 3f9f6e6ed30fb233f48457f95047f274bd271745 + - path: Sources/SubstrateTypes/RecallTypes.swift + blob: 7f01133df8495156fab87f1f4e441fb7867b3434 + - path: Sources/SubstrateTypes/Row.swift + blob: d7cf86f774dbccbbdf9daf1f3a5ed9be29fb24e8 + - path: Sources/SubstrateTypes/RowBitmaps.swift + blob: b879191a7076807c6fcd20e54cf806a639e63388 + - path: Sources/SubstrateTypes/RowState.swift + blob: 3f8c772be1fe1e3b44b90a5ea8681d4ce4326db7 + - path: Sources/SubstrateTypes/SimHash.swift + blob: e32a8b685c3db29408ed9e7fb21da4a7da10c467 + - path: Sources/SubstrateTypes/SnapshotId.swift + blob: b476bad15abecba2dacea225e0a7db7f445be777 + - path: Sources/SubstrateTypes/ThreeDBitTensor.swift + blob: 47b53642dda804afe8fbc0d863dc4ccd11d3ec55 + - path: Sources/SubstrateTypes/TimeRange.swift + blob: e1b7fbed94b6505b596ac5701cfd7c693e7d2425 +--- + +# SubstrateTypes Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in conceptual +order: the row shape and its lifecycle, the two coordinate systems a row +carries (lattice anchor and time), the fingerprint construction pipeline and +its algebra, integrity hashing, the audit trail, the population-statistics +matrices, and finally the recall wire vocabulary. + +## NounType.swift + +This file provides `NounType`, an eight-case enum naming the kind of thing a +row holds: a drawer, a tunnel, a knowledge-graph fact, a diary entry, a +proposal, an association, a learned reference, or an ambient sample. + +The enum is backed by `UInt8` raw values (0 through 7) rather than a string, +because the noun type is stored in a compact wire format alongside the rest +of a row. The file's opening comment states the values are wire-stable and +must never be renumbered: a stored row's noun type is read back by its raw +number, so changing a number would silently reclassify every row already on +disk. + +## RowState.swift + +This file provides the three pure-data types that describe a row's +lifecycle: `RowState` (the ten states a row can be in), `RowVerb` (the twelve +mutations the row-state automaton accepts), and `RowStateError` (the typed +failure a rejected transition returns). The automaton that enforces which +transitions are legal is compute, not data, so it remains one layer up, in +SubstrateLib; this file only names the states and verbs the automaton +operates on. + +The ten states use deliberately spaced raw values — 0 to 3, then a gap, then +16 to 19, another gap, then 32 and 33 — rather than a plain 0-to-9 count. +This spacing lets any consumer recover a state's lifecycle cluster with one +shift-and-mask operation instead of a lookup table. `RowState.cluster` reads +`(rawValue >> 4) & 0x3` to classify a state as cluster A (active or +becoming: active, pending, contested, accepted), cluster B (superseded or +historical: superseded, decayed, withdrawn, expired — retired but +revivable), or cluster C (terminal: rejected, tombstoned — retired and +final). `RowState.isActiveCluster` is the convenience most callers want: a +single true-or-false answer to "is this row currently believed." + +`RowState.cluster(ofRawState:)` performs the same classification starting +from a raw `UInt8` rather than a typed `RowState`, for code that has only +read a bare byte out of a persisted bitmap and has not decoded it yet. +`RowState.activeClusterUpperBoundRaw` names the one boundary value (16) a +storage-layer predicate can compare against directly — for example, in a SQL +`WHERE` clause — when it cannot call a Swift function; the file is explicit +that any code holding a decoded `RowState` should prefer `cluster` over a +hand-rolled numeric comparison, because a future state added inside one of +the gaps would silently misclassify under a magic-number boundary but not +under the shift-and-mask. + +`RowState.description` and `RowStateError.description` supply the exact +lowercase English strings ("active", "pending", and so on, and the compact +"active --reject-->" form) that downstream parsers depend on for parity with +the Rust port's `Display` implementation. Both are written as explicit +`switch` statements rather than the default `String(describing:)`, because +calling that default from inside a type's own `description` recurses forever. + +## RowBitmaps.swift + +This file provides `RowBitmaps`, the named layout over a row's three bitmap +columns, and `BitVector216`, a dense 216-bit view over that same data for +consumers that want to treat it as a flat grid. + +A row carries three `Int64` columns — adjective, operational, and +provenance — and each column packs several independently named fields at +specific bit ranges. For some purposes (updating the population-statistics +matrices), it is easier to think of these three columns as one uniform grid +of thirty-six fields of six bits each, even though the real fields are not +laid out that uniformly. `RowBitmaps.field(_:)` performs this translation: +given a field index from 0 to 35, it locates the right column and bit +offset and returns that field's six-bit value. The function carries two +defensive casts the comment explains in detail: a shift-amount guard, +because Swift silently wraps an out-of-range shift instead of returning +zero, and a cast to `UInt64` before shifting, because right-shifting a +negative `Int64` sign-extends and would leak the sign bit into a field's +value. + +`RowBitmaps.bit(field:bit:)` reads one bit out of a field's six-bit value. +`RowBitmaps.fieldValues()` returns all thirty-six (field, value) pairs in +order, which is what `MatrixO.applyRow` and `MatrixT.applyPair` consume to +update the co-occurrence and causality matrices. `RowBitmaps.bitVector()` +builds a `BitVector216` from a live row. + +`BitVector216` has two constructors for two different sources of the same +216-bit shape. `init(rowBitmaps:)` builds it from a real row through +`RowBitmaps.field(_:)`; because real rows never set bits above position 71 +of the abstract grid, this path is safe for updating `MatrixF` from row +capture and expunge events. `init(presenceBytes:)` builds it directly from a +raw 27-byte pattern that may set any of the 216 bits, including ones no real +`RowBitmaps` value can represent — the shape a conformance-test harness +vector uses. `bit(at:)` and `bit(field:bit:)` read one bit by absolute index +or by (field, bit) coordinate. + +## Row.swift + +This file provides `Row`, the substrate's central data shape, and `RowId`, +a type alias for `UUID` used wherever the substrate vocabulary calls for a +row identifier. + +`RowId` is a plain alias rather than a wrapper type because a `UUID`'s +16-byte big-endian wire form is already byte-identical to the Rust port's +`RowId(u128)` newtype; no translation step is needed at the boundary, and +callers can use `UUID`'s existing API directly. + +`Row` itself is a value type holding one row's complete current state: its +identifier, noun type, lifecycle state, three bitmap columns, fingerprint, +lattice anchor, and two optional fields — a lineage identifier, linking a +row to the row it was derived from, and the verbatim content. The file is +explicit that `Row` is pure data: no logic and no I/O. The full substrate +object that bundles many rows together with the audit log and the +statistics matrices, and the verb functions that mutate a row's state, +remain in SubstrateLib. `Row` only says what a row looks like at rest. + +## LatticeAnchor.swift + +This file provides `LatticeAnchor`, the sixteen-byte reference a row carries +to its position in the classification lattice: an eight-byte hash of a +lattice code plus an eight-byte hash of a concept identity (a Q-ID), or zero +for either half when absent. + +A lattice anchor does not store the lattice code text itself — it stores a +deterministic hash of it. `LatticeAnchor.fnv1a64(_:)` is the private FNV-1a +64-bit hash used for this purpose; because it is the exact FNV-1a algorithm +(matching `FNV.hash64` elsewhere in the package), two callers hashing the +same code string always agree, and the result matches the Rust port bit for +bit. + +`LatticeAnchor.udc(_:)` builds an anchor from a lattice code string alone, +with a null Q-ID pointer. `LatticeAnchor.udcQid(_:qid:)` builds an anchor +that carries both the lattice code and a specific concept identity; the file +explains why this second form matters: without it, every row about a broad +subject collapses onto one anchor value, losing the distinction between +different things classified under the same broad code. An empty Q-ID string +yields the same null pointer as `udc(_:)`, so the two constructors agree +when no concept identity is available. `LatticeAnchor.isNull` reports +whether both halves are zero — the anchor equivalent of "unclassified." + +## HLC.swift + +This file provides `HLC`, the Hybrid Logical Clock timestamp that orders +events across replicas of an estate without requiring synchronized wall +clocks, and `HLCGenerator`, the per-replica state machine that produces +HLC values. + +An `HLC` is three fields compared in order: `physicalTime` (milliseconds +since the Unix epoch), `logicalCount` (a counter that advances when +physical time does not), and `nodeID` (a per-replica tiebreaker). `HLC.<` +compares these three fields lexicographically, which gives any two HLC +values an unambiguous order even when two replicas' clocks briefly agree or +run backward relative to each other. `HLC.zero` is the earliest possible +value, used as a sentinel. `HLC.advanced()` returns a copy with the logical +counter incremented, holding physical time and node fixed. + +`HLCGenerator.send(now:)` produces the timestamp for a locally originated +event: if the wall clock has moved forward since the last call, it adopts +the new time and resets the logical counter to zero; otherwise it holds the +physical time and bumps the logical counter, guaranteeing every timestamp +this replica emits is strictly greater than the last. `HLCGenerator.receive(remote:now:)` +implements the HLC paper's merge rule for an event arriving from another +replica: it takes the maximum of the local clock, the remote clock, and the +current wall time, and sets the logical counter according to which of the +three inputs was tied for that maximum. This is the operation that lets two +replicas exchange audit events and converge on one shared sense of "what +happened before what," which the convergence proof in `GSetAuditLog.swift` +depends on. + +`HLC.packed` and `init(packed:)` convert to and from a lossy 8-byte form +used by federation tier-contribution messages, trading precision (40 bits of +milliseconds, roughly 34 years of range) for a smaller wire size. `HLC.wireBytes` +and `init(wireBytes:)` are the lossless 16-byte round trip used elsewhere. +The file also carries a `nodeId`-spelled initializer overload, kept only so +older call sites written against that casing keep compiling unchanged. + +## TimeRange.swift + +This file provides `TimeRange`, a closed interval `[start, end]` of two HLC +values. + +`TimeRange.init(start:end:)` rejects an interval whose end precedes its +start with a precondition, because an inverted range has no sensible +meaning for any caller. `TimeRange.contains(_:)` reports whether a given HLC +falls within the closed interval, inclusive of both endpoints — the +predicate a caller uses to test "did this event happen during this window." +The file notes that the higher-level primitive consuming these windows, +`MomentSummary`, stays in SubstrateLib; this type is only the interval +shape. + +## AsOfCoordinate.swift + +This file provides `AsOfCoordinate`, a two-case enum that says whether a +read should return the live state (`.present`) or the state as of a +specific HLC (`.asOf(hlc)`). + +The file explains the bug this type prevents: if "present" were represented +by a sentinel HLC value such as zero, a caller could not distinguish "give +me now" from "give me the state at the dawn of time." Making the choice an +explicit enum discriminant removes that ambiguity by construction. The +custom `Codable` conformance encodes the case as an explicit `"kind"` field +(`"present"` or `"asOf"`) plus, for the second case, the HLC itself, rather +than relying on Swift's default enum encoding, so the wire form stays stable +and self-describing across the Swift and Rust legs. + +## SnapshotId.swift + +This file provides `SnapshotId`, a `UUID` wrapper identifying one +point-in-time snapshot of an estate. + +Unlike `RowId`, which is a bare type alias, `SnapshotId` is a distinct +struct. The file states the reason: a snapshot identifier must not be +substitutable for a drawer, node, or estate identifier at the type level, +even though all of these are UUIDs underneath. `init(uuidString:)` and the +`uuidString` accessor round-trip through the standard UUID string form; +`Codable` conformance encodes and decodes that same string form, throwing +`SnapshotIdError.invalidUUID` on a malformed string rather than crashing. + +## BlockMask.swift + +This file provides `BlockMask`, an `OptionSet` selecting any subset of the +fingerprint's four 64-bit blocks. + +Before this type existed, callers passed a `Set` to select blocks, +which allocated memory on every call — a measured hot-path cost in nearest- +neighbor search over large row collections. `BlockMask` replaces that with +four fixed bit flags (`.block0` through `.block3`), an `.all` convenience +covering every block, and an empty `.none`. Because it is a plain integer +under the hood, membership tests are branchless and allocation-free. +`BlockMask.blockCount` counts how many blocks are selected (0 through 4), +which `Hamming.similarity` uses to compute the correct maximum-distance +denominator for a partial-block comparison. + +## Fingerprint256.swift + +This file provides `Fingerprint256`, the substrate's 256-bit structural +fingerprint, along with the pure combinators that other files in this +package build on. + +A fingerprint is four independent 64-bit blocks — `block0` through +`block3` — each a SimHash over a different aspect of a row: its bitmaps, +its lattice position, its lineage and timing, and its channel and source. +The file is careful to distinguish a fingerprint (the coordinate for +structural similarity) from a lattice anchor (the coordinate for topic +similarity); the two answer different questions and neither substitutes for +the other. + +`Fingerprint256.bit(at:)` and `Fingerprint256.with(bit:set:)` give bit-level +read and copy-on-write access across the full 256-bit span, translating a +flat index into the right block automatically. `with(bit:set:)` returns a +new value rather than mutating in place, matching the package's general +preference for value semantics over in-place mutation. `Fingerprint256.union(_:)` +computes the bitwise OR of two fingerprints — set-union over the bits each +one has set. `Fingerprint256.fromBits(_:)` builds a fingerprint from a +256-element array of booleans, mainly useful for tests and for converting +from an external bit representation. `wireBytes` and `init(wireBytes:)` +convert to and from the canonical 32-byte little-endian wire form; +`toBytes()` and `fromBytes(_:)` are thin aliases kept for parity with call +sites that expect a throws-free, optional-returning variant. + +The file's closing section defines a small combinator layer: +`zip4(_:_:)` applies a binary operation to each of the four block pairs +between two fingerprints in one call, `map4(_:)` applies a unary operation +to all four blocks of one fingerprint, `reduce4(_:_:)` folds a whole +sequence of fingerprints block-wise with a binary operation, and +`popcount()` counts set bits across all four blocks. These four functions +exist because the same "unroll across four blocks" pattern previously +appeared, hand-written, in seven or more places across the codebase — this +package's `ORReduce`, `BitwiseArithmetic`, and `Hamming` among them. +Expressing the unroll once here, and asking every other block-wise operation +to call it, means a future change to how blocks are combined only has one +call site to change, with no risk that one of the seven repeated copies +falls out of sync with the rest. `zip4Batch(_:_:_:)` and `map4Batch(_:_:)` +extend the same idea across an array of fingerprints, for callers vectorizing +across many rows rather than across one fingerprint's four blocks. + +## HyperplaneFamily.swift + +This file provides `Hyperplane`, one ±1-valued dividing plane used by +SimHash, and `HyperplaneFamily`, the fixed set of sixty-four such planes +that produce one fingerprint block. + +A hyperplane of length *n* would normally be an array of *n* values, each ++1, −1, or 0. Because binary inputs make this arithmetic reduce to counting +set bits, `Hyperplane` instead stores two bitmasks — `positiveMask` and +`negativeMask` — with a 1 wherever the plane is +1 or −1 respectively, and 0 +in both where the plane's value is 0 at that position (a "sparse +hyperplane"). `Hyperplane.sign(over:)` computes whether the plane's dot +product with a binary input vector is strictly positive, by comparing the +popcount of the input ANDed with each mask; a tied dot product resolves to +false, which the file notes is vanishingly rare in practice because the +per-block input has enough bits that an exact tie is unlikely. + +`HyperplaneFamily.generate(seed:blockIndex:inputBitLength:density:)` builds +a family of sixty-four hyperplanes deterministically from a 32-byte seed +using a private, non-cryptographic pseudo-random generator +(`HyperplanePRNG`). Determinism here is load-bearing: the file states that +hyperplane seeds are fixed once, at estate creation, and never rotate, +because rotating them would change every fingerprint an estate has ever +computed and break cross-replica sync. The `density` parameter controls +what fraction of a plane's positions are active (nonzero) rather than zero; +at density 1.0 every position is active, and the code takes a direct path +around a floating-point computation that would otherwise round up past +`UInt64.max` and trap. + +`HyperplaneFamily.canonicalHash()` computes a stable 64-bit fingerprint of +a whole family's content by folding every mask word through FNV-1a, used to +label a pairing arrangement in an audit trail without storing the full +family. `HyperplaneFamily.blockFamilies(baseSeed:density:)` is the single +routine that builds all four block families — for block 0 (192 bits) and +blocks 1 through 3 (64 bits each) — from one base seed, fixing two mistakes +an earlier, separately-derived generation path had: reusing one seed across +all four blocks (which collapsed them into one projection instead of four +independent ones) and assuming a uniform 64-bit width for every block +(block 0 is 192 bits). `diversifiedSeed(base:blockIndex:)` mixes a block +index into the base seed so each block gets an independent 32-byte seed, and +`expandSeed64(_:)` stretches a 64-bit seed back out to 32 bytes through four +rounds of a SplitMix64-style mixing function. + +## FloatSimHashPlanes.swift + +This file provides `FloatSimHashPlanes`, the materialized hyperplane set for +the floating-point-input variant of SimHash — the float analogue of +`HyperplaneFamily`, which serves the binary-input variant. + +The file explains why this type lives in SubstrateTypes rather than in +either of the two packages that use it: `SubstrateKernel` applies these +planes (pure signed-sum-and-sign arithmetic, no randomness) and +`SubstrateML` generates them from a seed (using the SplitMix64 random +generator). Placing the shared data shape at the lowest layer lets both +higher packages depend on it without either depending on the other, which +would otherwise invert the SDK's layering. `FloatSimHashPlanes` packs +`256 * dim` sign bits — one per (hyperplane, coordinate) pair — into a flat +array of 64-bit words, row-major by hyperplane, matching the draw order the +generator produces them in. Because the planes are immutable once generated +for a given seed and dimensionality, a caller materializes one instance per +(seed, dimension) pair and reuses it across every projection under that seed. + +## SimHash.swift + +This file provides `SimHash`, the construction that turns an input vector +and a hyperplane family into one 64-bit fingerprint block, plus +`SimHashInput`, the set of helpers that assemble the four blocks' input +vectors from a row's actual fields. + +`SimHash.block(over:family:)` is the core operation: for each of the +family's sixty-four hyperplanes, it sets one output bit if the hyperplane's +dot product with the input vector is positive. `SimHash.fingerprint(bitmapInput:latticeInput:lineageTemporalInput:channelSourceInput:families:)` +calls this once per block and assembles the four results into a complete +`Fingerprint256`. `SimHash.fingerprintBatch(...)` is the same computation +repeated across many rows, returning results in input order; the file notes +that this reference loop is what a hardware-accelerated backend elsewhere in +the SDK must reproduce bit for bit, even though that backend vectorizes the +popcount work across many rows at once for speed. +`SimHash.fingerprint(fromSubhashes:hyperplanes:)` is a convenience path for +callers that have already reduced each block's input down to one 64-bit +subhash rather than a full input vector. + +`SimHashInput.bitmap(adjective:operational:provenance:)` concatenates a +row's three bitmap columns into block 0's 192-bit input. +`SimHashInput.lattice(udcPrefixHash:qidDirectHash:qidClosureHash:)` packs +three hash components into block 1's 64-bit input at fixed bit offsets. +`SimHashInput.lineageTemporal(...)` and `SimHashInput.channelSource(...)` +do the same for blocks 2 and 3. Each function's bit-offset layout is fixed +by the cookbook specification the file cites; a mismatched offset here would +silently produce fingerprints that no longer compare correctly against ones +built the standard way, so these helpers exist precisely to keep every +caller assembling the same layout. + +## Hamming.swift + +This file provides `Hamming`, the substrate's primary structural-similarity +measurement: how many bit positions differ between two fingerprints. + +`Hamming.distance(_:_:blocks:)` computes this by XORing the two +fingerprints and counting the set bits in the result — `a.zip4(b, ^).popcount()` +on the common case of all four blocks, or a per-block loop when the caller +restricts the comparison to a `BlockMask` subset. Restricting to a subset +answers a more specific question, such as "how similar are these two rows +in lattice position alone, ignoring timing and provenance." `Hamming.similarity(_:_:blocks:)` +converts a distance into a `[0, 1]` score, where 1.0 means identical and 0.0 +means maximally distant over the selected blocks. `HammingDistance` is a +type alias re-exporting `Hamming` under the name some call sites use. + +## BitwiseArithmetic.swift + +This file provides three fingerprint operations beyond OR-reduction — +intersection, symmetric difference, and a cohort prototype — plus +`FingerprintBuilder`, a small expression type for composing them +declaratively. + +`BitwiseArithmetic.intersect(_:_:)` computes the bitwise AND of two +fingerprints: the bits both rows share. `BitwiseArithmetic.difference(_:_:)` +computes the bitwise XOR: the bits where the two disagree. Both delegate to +`Fingerprint256.zip4(_:_:)`. `BitwiseArithmetic.prototype(_:)` computes the +bit-for-bit majority vote across a whole cohort of fingerprints — the +fingerprint of a "typical" member — by folding the cohort into a +`CountVector256` and reading off its majority-vote view; the file notes this +replaced an earlier hand-written per-bit counting loop with an equivalent +call into the one canonical cohort-fold primitive. + +`FingerprintBuilder` is an indirect enum with four cases — a literal +fingerprint, an intersection of two sub-expressions, a difference of two +sub-expressions, or the prototype of a fingerprint list — and one method, +`evaluate()`, that walks the tree and computes the result. This lets a +caller build up a compound fingerprint query as data (for example, +"the intersection of Bob's typical pattern and Amelia's typical pattern") +and evaluate it once, rather than nesting direct function calls. + +## ORReduce.swift + +This file provides `ORReduce`, the substrate's universal aggregation +primitive over collections of fingerprints. + +`ORReduce.reduce(_:)` computes the bitwise OR across a whole sequence of +fingerprints, returning `Fingerprint256.zero` for an empty sequence (OR's +identity element); it delegates to `Fingerprint256.reduce4(_:|:)`. +`ORReduce.reduce(_:blocks:defaults:)` restricts the reduction to specific +blocks, filling any block outside the selection from a supplied default — +useful when a caller wants only the topic-block aggregate across a cohort, +say, without touching the other three blocks. The file explains why OR +reduction matters beyond simple aggregation: because it is commutative, +associative, and idempotent, it works as the join operator for a CRDT of +fingerprints — replicas can merge any number of contributions, in any +order, with any duplication, and converge on the same result. It is also +a privacy mechanism: once several fingerprints are OR-reduced together, a +reader can see which structural patterns appear somewhere in the group but +cannot recover which specific contributor set which bit. + +## CountVector256.swift + +This file provides `CountVector256`, the per-bit counting accumulator that +underlies cohort-level fingerprint statistics, including the majority-vote +prototype `BitwiseArithmetic.prototype(_:)` reads from it. + +The file's opening comment makes the key design argument: a majority-vote +summary does not compose. If node A's majority is computed from its +children, and node B's majority from its children, the majority of "A's +majority and B's majority" is not generally the same as the true majority +over every leaf under both A and B combined. A count vector does compose: +counts add, member totals add, and the sum is exact regardless of the order +members are folded in. So the count vector, not the majority-vote +fingerprint, is the object worth storing at every level of a tree; the +majority-vote fingerprint is a read-time view computed from it whenever a +caller actually needs a Hamming-comparable engram. + +`CountVector256.accumulate(_:)` folds one fingerprint into the vector: for +every set bit, its counter increases by one, and the member count increases +by one. `CountVector256.merge(_:)` and its operator form `+` combine two +count vectors by adding element-wise — the tree-fold step, safe to apply to +a node's children in any order because addition is commutative and +associative. `CountVector256.majorityVote()` reads off the fingerprint whose +bit `j` is set exactly when a strict majority of accumulated members had bit +`j` set (`2 * count > n`; an exact tie at half does not set the bit, a +convention the file calls out as identical across every kernel backend and +both language ports). `CountVector256.profile()` returns the 256 +per-bit probabilities instead of a threshold, for callers that want the +Bernoulli parameter of each bit rather than a hard yes-or-no answer. The +static `CountVector256.fold(_:)` is the reference implementation of folding +a whole array of fingerprints at once, which a vectorized kernel-layer +backend is required to match exactly. + +## FNV.swift + +This file provides `FNV`, the Fowler–Noll–Vo 1a string hash family the +substrate uses everywhere a deterministic string hash is required: drawer +fingerprints, manifest-derived identifiers, and deterministic tokenization. + +`FNV.hash64(_:)` and `FNV.hash32(_:)` are two independent FNV-1a hashes with +different offset bases and primes — the file is explicit that the 32-bit +result is not a truncation of the 64-bit one, but a wholly separate +computation over the same input bytes. `FNV.hash16(_:)` is different in +kind: it is the low 16 bits of `hash64(_:)`, not a from-scratch FNV-1a +variant (FNV-1a has no official 16-bit definition), used where the substrate +needs a compact prefix hash such as a lattice sub-hash. + +## MerkleDomain.swift + +This file provides `MerkleDomain`, the four one-byte tags prepended before +hashing to keep different kinds of nodes in the Merkle content-integrity +tree from ever colliding: `leaf` (a single drawer's content and vectors), +`interior` (a parent summarizing its children), `tombstone` (an expunged +payload), and `commitment` (a keyed HMAC-SHA256 commitment). Domain +separation means a leaf hash and an interior hash can never be mistaken for +each other even if their underlying bytes happened to coincide, because +each was computed with a different one-byte prefix. The file states these +four values are frozen and must match exactly between the Swift and Rust +legs, permanently. + +## ContentHash.swift + +This file provides `ContentHash`, the 32-byte SHA-256 digest over one leaf +payload — a drawer's content plus its vectors. + +The type stores its 32 bytes privately, exposed only through the `bytes` +accessor, so the fixed size is enforced by construction rather than by +convention. `ContentHash.tombstone` is a named constant: the SHA-256 hash of +the bare tombstone domain tag byte, used as the sentinel content hash for an +expunged (hard-deleted) drawer. The file explains why this constant is a +literal byte array rather than a value computed at runtime: SubstrateTypes, +as the lowest layer, cannot import SubstrateKernel (which owns the SHA-256 +implementation) without inverting the SDK's dependency direction, so the +precomputed literal stands in, checked against the real computation by a +bridge test in SubstrateKernel. `hexString` and `description` render the +bytes as lowercase hex; the custom `Codable` conformance encodes and decodes +that same hex string, validating its length and character set on decode +rather than trusting external input. The file is explicit that `ContentHash` +is not interchangeable with `MerkleRoot`, even though both are 32-byte +hashes: one is a payload digest, the other a subtree summary, and the type +system keeps them from being swapped by mistake. + +## MerkleRoot.swift + +This file provides `MerkleRoot`, the 32-byte hash summarizing an interior +node's children in the Merkle content-integrity tree — the counterpart to +`ContentHash`'s single-payload digest. + +Its shape mirrors `ContentHash` closely: private byte storage, a `bytes` +accessor, hex rendering, and the same style of custom `Codable` +implementation. `MerkleRoot.empty` is the named constant for the hash of a +node with no live children — the SHA-256 of the bare interior domain tag — +again stored as a literal for the same layering reason `ContentHash.tombstone` +is. Keeping `MerkleRoot` and `ContentHash` as two distinct types, rather than +one hash type used for both purposes, means a function that expects a +subtree summary cannot accidentally be handed a single payload's digest; the +compiler catches the mistake instead of a runtime bug surfacing later. + +## AuditEvent.swift + +This file provides `AuditEvent`, a single recorded mutation of a row: the +before-and-after bitmap and lattice-anchor state, which verb performed it, +who performed it, and when. + +`AuditEvent.eventID`, paired with `hlc`, gives every event a compound key +that makes replaying the same event twice — for example, after a sync +retry — a safe no-op rather than a duplicate. `beforeBitmaps` is optional +because a row's very first event, its capture, has no prior state to record. +`reason` is an optional human-readable explanation threaded from the call +site that performed the mutation; the file notes it is populated for +explicit actions such as an expunge but left `nil` for the great majority of +routine mutations. `AuditEvent.withReason(_:)` returns a copy of an event +with a caller-supplied reason attached, used because the structural +validator that first produces an event (`AuditGate.admit`, in SubstrateLib) +is a pure checker with no concept of "why," so the reason is layered on +afterward by the verb that called it. + +## GSetAuditLog.swift + +This file provides `GSetAuditLog`, the substrate's append-only audit log, +and its two supporting types, `AuditEntry` (one immutable log row) and +`AuditValue` (a typed field value inside an entry). + +The log is the substrate's source of truth: a row's visible current state is +a projection computed by replaying its log entries in order, never stored +independently of the log. `AuditEntry.id` is a 32-byte SHA-256 content hash +computed over the entry's other fields, which gives the log a natural way to +deduplicate: two replicas that independently record the same logical +mutation compute the same id and, once merged, keep only one copy. +`AuditValue` is an enum covering the four kinds of value a field change can +carry — a bitmap, a string, a fingerprint, or an integer — with a hand-written +`Codable` conformance that encodes each case as a single-key JSON object +(`{"bitmap": 42}`) rather than Swift's default synthesized shape +(`{"bitmap": {"_0": 42}}`), so the wire format matches what the Rust port's +serde derive produces natively and both legs agree on one JSON shape. + +`GSetAuditLog` itself stores its entries keyed by content hash internally, +for O(1) deduplication, but its `Codable` conformance serializes them as a +plain array sorted by id, because the conceptually correct wire +representation of a grow-only set is a set, not a hash map — the file +explains this distinction explicitly to justify writing custom `Codable` +rather than relying on the default. `GSetAuditLog.add(_:)` inserts one +entry, a no-op if the same id is already present. `GSetAuditLog.merge(_:)` +is the CRDT join: the union of two logs' entries, correct regardless of +which replica calls it or in what order two logs are merged. +`GSetAuditLog.orderedEntries` replays every entry in HLC order, the +sequence a projection applies to compute visible state. +`GSetAuditLog.entries(forRow:)` and `GSetAuditLog.entries(since:)` scope +that same ordered replay to one row or to everything after a cutoff, +serving the row-state automaton and the sync protocol respectively. The +file's closing comment sketches why this design converges: G-Set merge is +set union, set union is commutative, associative, and idempotent, and HLC +gives any two entries an unambiguous order to replay them in — so any two +replicas that have exchanged all of each other's entries compute identical +visible state. + +## MatrixF.swift + +This file provides `MatrixF`, the field-presence matrix: for every +(field, bit) pair, how many rows in the estate currently have that bit set. + +`MatrixF` stores 216 `Int64` counts in one flat array, indexed by +`field * bitsPerField + bit`; `MatrixF.cellIndex(field:bit:)` computes that +index with bounds checking. Its layout constants (`fieldCount`, +`bitsPerField`, `cellCount`) are aliases onto the identical constants in +`RowBitmaps`, kept as aliases rather than a second independent definition so +the two types cannot silently drift out of agreement about the shape of the +216-cell grid they both describe. `MatrixF.applyRow(delta:bitVector:)` +updates every cell whose bit is set in a `BitVector216`, adding `delta` — +positive one on a row's capture, negative one on its expunge, or the pair of +calls needed to represent a mutation as a removal followed by an addition. +`MatrixF.totalCount` sums every cell as a sanity check. +`MatrixF.writeWire(into:)` and `MatrixF.readWire(_:)` serialize the matrix +to and from its canonical 1,728-byte little-endian wire form (216 cells of 8 +bytes each). + +## MatrixC.swift + +This file provides `MatrixC`, the correlation matrix: the marginal +probability of each (field, bit) pair, derived from `MatrixF` rather than +updated independently. + +`MatrixC` shares `MatrixF`'s 216-cell shape but stores `Float` values in +`[0, 1]` instead of raw counts. `MatrixC.derive(from:nRows:)` is its only +way to change: it divides each of `MatrixF`'s counts by the total row count, +routing the division through `Double` before narrowing back to `Float32`, +which the file notes is necessary for cross-language agreement — both the +Swift and Rust legs perform the identical cast-divide-cast sequence, and +IEEE-754 guarantees the same bit pattern results on both. When there are no +rows at all, every cell is zero rather than an undefined division result. +`MatrixC.writeWire(into:)` and `MatrixC.readWire(_:)` serialize to and from +an 864-byte wire form (216 cells of 4 bytes each, `Float32` bit patterns). + +## MatrixO.swift + +This file provides `MatrixO`, the co-occurrence matrix: how often each pair +of (field, value) settings appears together across rows, and +`CooccurrenceKey`, the packed key identifying one such pair. + +Because most of the roughly 46,000 possible (field, value, field, value) +combinations never occur in a typical estate, `MatrixO` stores only nonzero +cells, as a list sorted by `CooccurrenceKey.packed` rather than as a dense +array or an unordered dictionary — sorted order makes both iteration and +serialization deterministic across languages, which an unordered dictionary +could not guarantee. `CooccurrenceKey.packed` folds all four components +into one `UInt32` for compact storage and for the ordering comparison. +`MatrixO.count(_:)` and `MatrixO.increment(_:by:)` use binary search over +the sorted entries to read or update one cell in logarithmic time, and +`increment` removes a cell entirely if its count returns to zero, keeping +the canonical form free of dead entries. `MatrixO.applyRow(delta:fieldValues:)` +updates the matrix for one row: it iterates every ordered pair of the row's +(field, value) settings, including a field paired with itself, and +increments each pair's cell by `delta`. The file notes explicitly that the +matrix is not required to be symmetric in storage even though co-occurrence +is conceptually symmetric — each unordered pair contributes to two distinct +cells, (i, j) and (j, i) — and that an implementation which stores only one +direction and infers the other is a valid optimization, but the reference +here stores both for clarity. `MatrixO.writeWire(into:)` and +`MatrixO.readWire(_:)` define the canonical wire form: an entry count +followed by each entry's packed key and count. + +## MatrixT.swift + +This file provides `MatrixT`, the temporal causality matrix: how often a row +with one (field, value) setting is followed, within a bounded time window, +by a row with another (field, value) setting — the substrate's tool for +distinguishing "these things happen together" from "one of these things +tends to precede the other." + +`CausalityKey` extends `CooccurrenceKey`'s idea with a fifth component, a +lag bucket, and packs all five into a `UInt64`. `MatrixT.lagBucket(forMinutes:)` +converts a raw time difference in minutes into one of eight log-spaced +buckets — 1, 2, 4, 8, 16, 32, 64, and 128 minutes as lower bounds — returning +`nil` for a difference outside the supported 1-to-256-minute window, in +which case no update to the matrix occurs at all. The file states this +matrix is explicitly asymmetric: the cell for "(field i, value i) preceding +(field j, value j)" is tracked completely separately from the reverse +direction, unlike `MatrixO`'s conceptually symmetric co-occurrence. Storage, +lookup (`count(_:)`), update (`increment(_:by:)`), and wire serialization +follow the identical sorted-array, binary-search pattern `MatrixO` uses. +`MatrixT.applyPair(delta:rowAFieldValues:rowBFieldValues:lagMinutes:)` is the +row-pair version of `MatrixO.applyRow`: given that row A precedes row B by a +known number of minutes, it increments the cell for every combination of A's +field values against B's field values, once a valid lag bucket exists for +that separation. + +## ThreeDBitTensor.swift + +This file provides `ThreeDBitTensor`, the dense bit-sliced storage layout +that answers "which rows have field X set to value Y" quickly across up to +a million rows. + +Rather than storing one 216-bit block per row (a row-major layout), the +tensor keeps six separate bit-slices, one per bit position 0 through 5 of a +field's six-bit value; each slice packs one bit per (row, field) pair. This +"bit-sliced" arrangement is what makes `scanFieldEquals(field:value:)` fast: +testing whether a whole batch of rows matches a target value reduces to +scanning six flat byte buffers rather than reading and unpacking a +six-bit value out of every row individually. `valueAt(row:field:)` and +`setValue(row:field:value:)` provide ordinary cell-level read and write, +each internally looping over the six bit-slices; `setValue` enforces the +six-bit width invariant with a precondition. `bitSet(row:field:bit:)` and +`setBit(row:field:bit:on:)` are the underlying single-bit primitives both +higher-level functions call. `scanFieldEquals(field:value:)` returns a byte +mask marking every row matching the target value; `enumerateMatches(_:)` +turns that mask into a plain array of matching row indices. +`reserveCapacity(_:)` grows the tensor to hold more rows, extending each +slice with zero bytes and leaving all existing data untouched — a no-op if +the requested size is not larger than the current one. The file notes the +current scan implementation loops row by row within each bit-slice pass +rather than operating on whole 64-bit words at once, which is a known +optimization opportunity, not yet taken, for a future vectorized version. + +## RecallTypes.swift + +This file provides the shared wire vocabulary that every recall primitive +and every federation query returns: `RecallScore`, `DistanceBreakdown`, +`RecallResult`, and `RowProjection`. + +These four types live in SubstrateTypes, rather than in whichever library +happens to implement a particular recall strategy, because federation needs +them too — a federated query's response is shaped exactly like a local +recall result — and because defining them once here means no two libraries +can quietly redefine the same shape and drift apart. `RecallScore` pairs one +`RowId` with one `Float32` score; the file is explicit that the score's +meaning is specific to whichever recall primitive produced it (cosine +similarity for vector recall, inverted Hamming distance for fingerprint +recall, BM25 for text recall), so any code that combines scores from +different primitives must normalize first. `DistanceBreakdown` reports how +much each of four components — lattice, fingerprint, temporal, bitmap — +contributed to a match, each normalized to `[0, 1]`, both to explain "why +this matched" to a caller and to weight Reciprocal Rank Fusion when +combining several recall strategies' results. `RecallResult` bundles a +ranked list of scores with an optional breakdown, an optional confidence +interval, and the name of the primitive that produced it, so a composition +step downstream knows which combination rule to apply. `RowProjection` is +the minimal slice of a row that a recall primitive actually needs to rank +candidates: its identifier, capture time, fingerprint, lattice anchor, +bitmaps, and lifecycle state, deliberately omitting the row's verbatim +content and any metadata beyond the bitmaps. The file explains this +omission is intentional — ranking operates on structure, and the heavier +verbatim content is fetched separately, only for the rows that survive +ranking. + +## Rust Port and Conformance + +The `rust/` directory mirrors this package one module per Swift file: +`rust/src/row.rs` for `Row.swift`, `rust/src/fingerprint256.rs` for +`Fingerprint256.swift`, and so on through all thirty files, with +`rust/src/lib.rs` re-exporting the primary types at the crate root the same +way this package's types are used directly by Swift callers. There is no +separate conformance-fixture directory analogous to LatticeLib's +`rust/tests/fixtures/`; because SubstrateTypes ships no pinned data +artifacts, agreement between the two legs is instead verified by shared +test vectors embedded directly in each module's own test block (for +example, the illustrative Hamming and Fingerprint256 test vectors quoted in +their Swift source comments), checked against the equivalent Rust unit +tests. When a function in this package changes — most sensitively the +fingerprint combinators, the Hamming and SimHash arithmetic, and the FNV +hash — the corresponding Rust module must change identically, because every +one of these functions is a cross-platform agreement contract, not an +implementation detail local to one language. diff --git a/packages/libs/SubstrateTypes/docs/OVERVIEW.md b/packages/libs/SubstrateTypes/docs/OVERVIEW.md new file mode 100644 index 0000000..2710de4 --- /dev/null +++ b/packages/libs/SubstrateTypes/docs/OVERVIEW.md @@ -0,0 +1,214 @@ +--- +doc: OVERVIEW +package: SubstrateTypes +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateTypes/AsOfCoordinate.swift + blob: b4433864e36b5803533266c97b153a836ee04adb + - path: Sources/SubstrateTypes/AuditEvent.swift + blob: fbfabeb4eaba4bf43b6e45d8c2b772d994a12b7b + - path: Sources/SubstrateTypes/BitwiseArithmetic.swift + blob: a6270b0f7395c17915c54fdc119e854ca5a44077 + - path: Sources/SubstrateTypes/BlockMask.swift + blob: 904c2179be96d5c20472f0f356947e82e9d8cf37 + - path: Sources/SubstrateTypes/ContentHash.swift + blob: 835744792dc191e824e676d8715c96111b4945ff + - path: Sources/SubstrateTypes/CountVector256.swift + blob: edabb34cb02005135d745ffa6da5541835159927 + - path: Sources/SubstrateTypes/Fingerprint256.swift + blob: 465607fe4164bb0dd94009dbbb20d6e766b973a8 + - path: Sources/SubstrateTypes/FloatSimHashPlanes.swift + blob: b5506e7d12fd153b721f4ca7a52a65a6667ea2f6 + - path: Sources/SubstrateTypes/FNV.swift + blob: f3d8874705ad66f513e49ccbf0c714980a1172e8 + - path: Sources/SubstrateTypes/GSetAuditLog.swift + blob: 1a4452e4c8a147d4fac299a3319b0cb4e70cb795 + - path: Sources/SubstrateTypes/Hamming.swift + blob: dc899742f77434910d51bb3cd830cc9603f56f0a + - path: Sources/SubstrateTypes/HLC.swift + blob: a74442251bd542e8f403029ee475c2ac9d7b91b0 + - path: Sources/SubstrateTypes/HyperplaneFamily.swift + blob: f30c4640629116b1952df48790ed6f5ae67e9025 + - path: Sources/SubstrateTypes/LatticeAnchor.swift + blob: 5f3ba24efa5a85109d20f3b61e6acb17e21cb0a5 + - path: Sources/SubstrateTypes/MatrixC.swift + blob: a8612ea7b09e6d4d93a4acbeedbe2b3b91776a74 + - path: Sources/SubstrateTypes/MatrixF.swift + blob: ed6e38a78c654562b04074878bf86686da0172c3 + - path: Sources/SubstrateTypes/MatrixO.swift + blob: c10fa0b14d8abd172503d0375dfab9302da32025 + - path: Sources/SubstrateTypes/MatrixT.swift + blob: cd31ca818be6446f8f1cd98c6b912c640a78c613 + - path: Sources/SubstrateTypes/MerkleDomain.swift + blob: 4e7d1888a04b2e9979e62c58b54ba698712e56d5 + - path: Sources/SubstrateTypes/MerkleRoot.swift + blob: 1979925f4646cdbed8785a43e2f2a14a7f9c0cac + - path: Sources/SubstrateTypes/NounType.swift + blob: 940d6c51329a66849e572a95ff8ae51648784f2f + - path: Sources/SubstrateTypes/ORReduce.swift + blob: 3f9f6e6ed30fb233f48457f95047f274bd271745 + - path: Sources/SubstrateTypes/RecallTypes.swift + blob: 7f01133df8495156fab87f1f4e441fb7867b3434 + - path: Sources/SubstrateTypes/Row.swift + blob: d7cf86f774dbccbbdf9daf1f3a5ed9be29fb24e8 + - path: Sources/SubstrateTypes/RowBitmaps.swift + blob: b879191a7076807c6fcd20e54cf806a639e63388 + - path: Sources/SubstrateTypes/RowState.swift + blob: 3f8c772be1fe1e3b44b90a5ea8681d4ce4326db7 + - path: Sources/SubstrateTypes/SimHash.swift + blob: e32a8b685c3db29408ed9e7fb21da4a7da10c467 + - path: Sources/SubstrateTypes/SnapshotId.swift + blob: b476bad15abecba2dacea225e0a7db7f445be777 + - path: Sources/SubstrateTypes/ThreeDBitTensor.swift + blob: 47b53642dda804afe8fbc0d863dc4ccd11d3ec55 + - path: Sources/SubstrateTypes/TimeRange.swift + blob: e1b7fbed94b6505b596ac5701cfd7c693e7d2425 +--- + +# SubstrateTypes Overview + +## What This Library Does + +SubstrateTypes defines the smallest shared unit of information that MOOTx01 +stores: a row. A row is one recorded observation — a diary entry, a fact +pulled from a knowledge graph, a proposal, a sample of ambient context — held +in one common shape so every other library can read it and write it the same +way. SubstrateTypes is where that shape, and the handful of coordinate +systems attached to it, are defined once for the whole SDK. + +Two coordinate systems ride on every row. A fingerprint is a 256-bit code +that captures structural similarity: rows that look alike in shape produce +fingerprints that share many bits. A lattice anchor is a compact reference to +a lattice code, the classification code that LatticeLib's FDC engine assigns; +rows about the same subject carry related anchors. A third value, the Hybrid +Logical Clock (HLC), timestamps every change to a row so that two devices +holding copies of the same estate can agree on the order of events without +trusting either device's wall clock. + +Around these three coordinate systems sit a small set of pure, deterministic +functions: Hamming distance and bitwise arithmetic over fingerprints, +SimHash construction from hyperplane families, FNV string hashing, and +OR-reduction for combining many fingerprints into one. These functions are +"canonical reference compute" — not heavy algorithms, but the exact +arithmetic that every consumer of the shapes must reproduce identically, on +every platform, forever. + +## The Problem It Solves + +MOOTx01's higher libraries — the ones that rank memories, detect drift, or +sync an estate across a phone and a laptop — all need to talk about the same +row shape. Without one shared definition, each library would invent its own +version of a fingerprint or a timestamp, and the definitions would drift +apart. A comparison between two rows would then depend on which library +produced them, which breaks every guarantee the system makes about +comparing memories. + +SubstrateTypes solves this by being the single, dependency-free home for +these shapes. It ships as the lowest of four packages that together make up +the substrate: SubstrateTypes (this package, pure shape and zero compute +beyond canonical reference arithmetic), SubstrateKernel (dispatch to +hardware-accelerated backends), SubstrateML (learned models), and +SubstrateLib (the orchestration layer, which still owns the row-state +automaton and the verb mechanics that operate the shapes). SubstrateTypes +depends on none of the other three. A library that only needs to describe a +row's shape — for example, ConvergenceKit, which serializes rows for +CloudKit — depends on SubstrateTypes alone and pulls in no compute it does +not need. + +A second problem the package solves is cross-platform agreement. MOOTx01 +estates can federate, so a Swift-based Apple client and a Rust-based +service must compute the identical Hamming distance, the identical +fingerprint, and the identical hash for the same input. The package ships a +Rust port in `rust/` that mirrors every type and function here, so both legs +of the SDK share one arithmetic contract. + +## How It Works + +Every row (`Row`) carries a fixed set of fields: an identifier, a noun type +(what kind of thing it is — a diary entry, a proposal, and so on), a +lifecycle state, three bitmap columns of adjective, operational, and +provenance flags, a fingerprint, a lattice anchor, and optional lineage and +content references. `RowState` and `RowBitmaps` describe the lifecycle and +the bitmap layout in more detail; both are pure data, so the automaton that +enforces which state transitions are legal lives one layer up, in +SubstrateLib. + +The fingerprint is built by SimHash, a locality-sensitive hash: unlike a +cryptographic hash, where a one-bit change in the input scrambles the whole +output, SimHash produces outputs where similar inputs share many bits. Each +of the fingerprint's four 64-bit blocks is computed by comparing an input +vector against sixty-four random hyperplanes — flat dividing surfaces fixed +at estate creation — recorded in a `HyperplaneFamily`. Once built, +fingerprints are compared and combined by a small algebra: `Hamming.distance` +counts differing bits, `BitwiseArithmetic` computes intersection and +symmetric difference, `ORReduce` folds many fingerprints into one that keeps +their shared structure while losing which fingerprint contributed which bit, +and `CountVector256` accumulates per-bit counts across a whole cohort so a +majority-vote fingerprint, or any other statistic, can be read off later. + +Every change to a row is recorded, never overwritten. `AuditEvent` is one +recorded mutation; `GSetAuditLog` is a grow-only set of such events, a +conflict-free replicated data type (CRDT). Two replicas of an estate can +exchange their audit logs in any order, merge by taking the union of +entries, and land on the same result, because set union does not care about +order. The `HLC` timestamp on every event gives the log a total order to +replay, and `TimeRange` and `AsOfCoordinate` let a caller ask what a row +looked like at a past point in that order. + +A separate family of types tracks population-level statistics across an +entire estate rather than one row at a time. `MatrixF` counts how often each +bitmap bit is set across all rows; `MatrixC` derives the marginal probability +of each bit from `MatrixF`; `MatrixO` counts how often pairs of bitmap values +co-occur; `MatrixT` counts how often one bitmap value precedes another within +a time window, which is the substrate's tool for telling correlation apart +from likely causation. `ThreeDBitTensor` is the dense, bit-sliced storage +layout that answers "which rows have field X set to value Y" quickly across +a million rows. + +A last family of types protects data integrity and supports recall. +`ContentHash` and `MerkleRoot` are two distinct fixed-size hashes — +one per leaf payload, one per subtree of children — kept as separate types so +the compiler rejects any code that confuses them; `MerkleDomain` supplies the +one-byte tags that keep leaf, interior, and tombstone hashes from ever +colliding. `RecallTypes` defines the wire vocabulary — `RecallScore`, +`RecallResult`, `DistanceBreakdown`, `RowProjection` — that every recall +primitive and every federation query returns, so composing results from +different recall strategies never requires an ad hoc translation step. + +## How the Pieces Fit + +Figure 1 shows how the major types connect: a row's own fields, the +fingerprint construction pipeline, the fingerprint algebra, the +population-statistics matrices, the audit trail, and the integrity and +recall vocabularies that read from a row without holding a reference to it. + +![Figure 1. Topology of SubstrateTypes](topology.svg) + +*Figure 1. Topology of SubstrateTypes. A row carries a fingerprint and a +lattice anchor. Hyperplane families and SimHash build the fingerprint; the +fingerprint algebra (Hamming distance, bitwise arithmetic, OR-reduction, +count-vectors) operates on it afterward. The row's bitmaps feed the +population-statistics matrices. Every mutation is recorded as an audit event +under Hybrid-Logical-Clock ordering in the grow-only audit log. Integrity +hashes and the recall wire vocabulary both read from row data without +belonging to the row shape itself.* + +Nothing in this package reaches upward. The row-state automaton, the verb +implementations that call it, and the full substrate object that bundles +rows with an audit log and the statistics matrices all remain in +SubstrateLib. SubstrateTypes supplies the vocabulary; SubstrateLib supplies +the behavior. + +## What Ships in the Package + +The package ships thirty Swift source files under `Sources/SubstrateTypes/` +and a mirrored Rust crate under `rust/src/`, one Rust module per Swift file +with matching names. There are no pinned data artifacts in this package — +unlike a library such as LatticeLib, SubstrateTypes carries no bundled +reference tables, because every value here is either supplied by a caller +(a hyperplane seed, a row's own fields) or computed from caller-supplied +input by a pure function (a Hamming distance, a SimHash block, an FNV hash). +Reproducibility rests on the arithmetic itself being pinned, verified by +conformance tests that check the Swift and Rust legs agree bit for bit. diff --git a/packages/libs/SubstrateTypes/docs/topology.svg b/packages/libs/SubstrateTypes/docs/topology.svg new file mode 100644 index 0000000..455804d --- /dev/null +++ b/packages/libs/SubstrateTypes/docs/topology.svg @@ -0,0 +1,160 @@ + + + + + + + + + + + + + SubstrateTypes: row shape, coordinates, and the views derived from them + + + + HyperplaneFamily + 64 fixed ±1 planes + + + SimHash + sign(<v,h>) per block + + + Fingerprint256 + 4×64-bit structural code + + + Fingerprint algebra + Hamming · Bitwise · OR · CountVector + + + + + + + + LatticeAnchor + topic coordinate + + + Row + the substrate unit + + + Row lifecycle + layout + NounType · RowState · RowBitmaps + + + Temporal coordinates + HLC · TimeRange · AsOfCoordinate + + + + row.fingerprint + + + row.latticeAnchor + + + row.state / .bitmaps + + + + Estate-level views (derived from many rows, not stored on one row) + + + Population matrices + MatrixF · MatrixC · MatrixO · MatrixT + + + Audit trail (CRDT) + AuditEvent · GSetAuditLog + + + Integrity hashes + ContentHash · MerkleRoot + + + Recall vocabulary + RecallResult · RowProjection + + + + + + + + + + + composes / produces + + read by / derived from + From be2b292cfd1f5b11c8b4d92016265d6fdcba6226 Mon Sep 17 00:00:00 2001 From: Fable Date: Sat, 4 Jul 2026 22:14:59 -0500 Subject: [PATCH 2/2] =?UTF-8?q?docs(sdk):=20rewrite=20prose=20to=20the=20f?= =?UTF-8?q?our=20immutable=20laws=20=E2=80=94=20FK=20<=3D=208.0,=20no=20em?= =?UTF-8?q?-dashes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full rewrite of OVERVIEW.md and DETAILS.md per package: Turabian/Chicago, reading level 8.0 or less, no sentence over 2 commas outside simple lists, no 'not X, but Y' or 'it is not X, it is Y' forms. AGENT_MAP and topology.svg scrubbed of em-dashes only. Front matter unchanged. Gated by scripts/docs-style-lint.py (mootx01-ee): 65/65 pass; docs-scrub clean. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Pzkqg3bpbbhfuWRsYomJ3o --- packages/libs/EngramLib/docs/AGENT_MAP.md | 70 +- packages/libs/EngramLib/docs/DETAILS.md | 244 +-- packages/libs/EngramLib/docs/OVERVIEW.md | 156 +- .../libs/IntellectusLib/docs/AGENT_MAP.md | 96 +- packages/libs/IntellectusLib/docs/DETAILS.md | 347 +-- packages/libs/IntellectusLib/docs/OVERVIEW.md | 178 +- .../libs/IntellectusLib/docs/topology.svg | 4 +- .../libs/SubstrateKernel/docs/AGENT_MAP.md | 196 +- packages/libs/SubstrateKernel/docs/DETAILS.md | 513 ++--- .../libs/SubstrateKernel/docs/OVERVIEW.md | 241 ++- .../libs/SubstrateKernel/docs/topology.svg | 4 +- packages/libs/SubstrateLib/docs/AGENT_MAP.md | 160 +- packages/libs/SubstrateLib/docs/DETAILS.md | 480 +---- packages/libs/SubstrateLib/docs/OVERVIEW.md | 215 +- packages/libs/SubstrateLib/docs/topology.svg | 6 +- packages/libs/SubstrateML/docs/AGENT_MAP.md | 112 +- packages/libs/SubstrateML/docs/DETAILS.md | 1904 +++++++++-------- packages/libs/SubstrateML/docs/OVERVIEW.md | 255 ++- .../libs/SubstrateTypes/docs/AGENT_MAP.md | 254 +-- packages/libs/SubstrateTypes/docs/DETAILS.md | 1269 +++++------ packages/libs/SubstrateTypes/docs/OVERVIEW.md | 241 ++- 21 files changed, 3439 insertions(+), 3506 deletions(-) diff --git a/packages/libs/EngramLib/docs/AGENT_MAP.md b/packages/libs/EngramLib/docs/AGENT_MAP.md index 028777d..0e539e6 100644 --- a/packages/libs/EngramLib/docs/AGENT_MAP.md +++ b/packages/libs/EngramLib/docs/AGENT_MAP.md @@ -11,52 +11,52 @@ sources: blob: 5c61fd65888a795c703ab9f52022dffeae9c7595 --- -# AGENT_MAP — EngramLib +# AGENT_MAP: EngramLib PURPOSE: product-facing similarity/retrieval/aggregation API over 256-bit engrams (`Fingerprint256`). Thin wrapper that hides substrate kernel selection: distance, batch-distance, top-K nearest, radius filter, OR-union. Two files total; no internal multi-stage pipeline. -DEPS: imports SubstrateTypes (`Fingerprint256`, aliased `Engram`), SubstrateKernel (`SubstrateKernel` protocol, `PortableKernel.kernelForCurrentPlatform()`). Imported by (within SDK): VectorKit, CorpusKit (kits, moot-memory repo — kits compose libs, never the reverse). Rust port in `rust/` (`engram_lib` crate) mirrors the full API 1:1 against `substrate-types`/`substrate-kernel` Rust crates; conformance is behavioral parity checked by mirrored test suites (`rust/tests/engram_lib_tests.rs` vs `Tests/EngramLibTests/`), not a shared fixture file. +DEPS: imports SubstrateTypes (`Fingerprint256`, aliased `Engram`), SubstrateKernel (`SubstrateKernel` protocol, `PortableKernel.kernelForCurrentPlatform()`). Imported by (within SDK): VectorKit, CorpusKit (kits, moot-memory repo: kits compose libs, never the reverse). Rust port in `rust/` (`engram_lib` crate) mirrors the full API 1:1 against `substrate-types`/`substrate-kernel` Rust crates; conformance is behavioral parity checked by mirrored test suites (`rust/tests/engram_lib_tests.rs` vs `Tests/EngramLibTests/`), not a shared fixture file. ENTRY POINTS (most callers need only these): -- EngramLib.swift:77 `EngramLib.distance(_ a: Engram, _ b: Engram) -> Int` — Hamming distance, 0...256 -- EngramLib.swift:99 `EngramLib.findNearest(probe:in:k:) -> [Match]` — top-K nearest, distance-asc / index-asc tiebreak -- EngramLib.swift:216 `EngramLib.session() -> Session` — reusable kernel handle for hot loops +- EngramLib.swift:77 `EngramLib.distance(_ a: Engram, _ b: Engram) -> Int` | Hamming distance, 0...256 +- EngramLib.swift:99 `EngramLib.findNearest(probe:in:k:) -> [Match]` | top-K nearest, distance-asc / index-asc tiebreak +- EngramLib.swift:216 `EngramLib.session() -> Session` | reusable kernel handle for hot loops ## Symbol Table -### Engram type — EngramLib.swift -- :43 `public typealias Engram = Fingerprint256` — substrate type, product-facing name; do not add a wrapping struct -- :45-53 `extension Engram { init(blocks b0:_ b1:_ b2:_ b3:) }` — convenience 4-block constructor -- :33 `internal let _engramLibCachedKernel: any SubstrateKernel` — resolved ONCE at module load via `PortableKernel.kernelForCurrentPlatform()`; every call path reads this, never re-resolves - -### Static API — enum EngramLib (EngramLib.swift:71) -- :77 `distance(_:_:) -> Int` — single-pair Hamming distance -- :84 `distances(probe:candidates:) -> [Int]` — batched, same order/count as `candidates`; `[]` on empty input -- :99 `findNearest(probe:in:k:) -> [Match]` — `[]` if `k <= 0` or candidates empty; clamps to `min(k, candidates.count)` -- :112 `findNearest(probe:in:) -> Match?` — k=1 convenience; `nil` on empty candidates -- :124 `findWithin(probe:in:maxDistance:) -> [Match]` — `[]` if candidates empty OR `maxDistance < 0`; inclusive upper bound -- :149 `union(_ engrams: [Engram]) -> Engram` — bitwise OR-reduce; zero engram is the empty-input identity -- :154 `union(_ a:_ b:) -> Engram` — pairwise OR convenience, delegates to `Fingerprint256.union(_:)` -- :222 `private static func kernel() -> any SubstrateKernel` — single choke point back to the cached kernel - -### Session — struct EngramLib.Session (EngramLib.swift:166), Sendable -- :169 `init()` — captures `_engramLibCachedKernel` -- :173/:177/:184/:193/:210 `distance` / `distances` / `findNearest` / `findWithin` / `union` — instance mirrors of the static functions above, byte-identical results, no new behavior -- :216 `EngramLib.session() -> Session` — factory, equivalent to `Session()` - -### Result type — Match.swift -- :14 `public struct Match: Hashable, Sendable, Codable` — `index: Int` (position in caller's candidate array), `distance: Int` (Hamming distance from probe) -- :18 `init(index:distance:)` — public memberwise init -- :24-28 `extension Match: Comparable` — `<` compares `distance` first, `index` second; this IS the sort/tiebreak contract used by `findNearest`/`findWithin` +### Engram type: EngramLib.swift +- :43 `public typealias Engram = Fingerprint256` | substrate type, product-facing name; do not add a wrapping struct +- :45-53 `extension Engram { init(blocks b0:_ b1:_ b2:_ b3:) }` | convenience 4-block constructor +- :33 `internal let _engramLibCachedKernel: any SubstrateKernel` | resolved ONCE at module load via `PortableKernel.kernelForCurrentPlatform()`; every call path reads this, never re-resolves + +### Static API: enum EngramLib (EngramLib.swift:71) +- :77 `distance(_:_:) -> Int` | single-pair Hamming distance +- :84 `distances(probe:candidates:) -> [Int]` | batched, same order/count as `candidates`; `[]` on empty input +- :99 `findNearest(probe:in:k:) -> [Match]` | `[]` if `k <= 0` or candidates empty; clamps to `min(k, candidates.count)` +- :112 `findNearest(probe:in:) -> Match?` | k=1 convenience; `nil` on empty candidates +- :124 `findWithin(probe:in:maxDistance:) -> [Match]` | `[]` if candidates empty OR `maxDistance < 0`; inclusive upper bound +- :149 `union(_ engrams: [Engram]) -> Engram` | bitwise OR-reduce; zero engram is the empty-input identity +- :154 `union(_ a:_ b:) -> Engram` | pairwise OR convenience, delegates to `Fingerprint256.union(_:)` +- :222 `private static func kernel() -> any SubstrateKernel` | single choke point back to the cached kernel + +### Session: struct EngramLib.Session (EngramLib.swift:166), Sendable +- :169 `init()` | captures `_engramLibCachedKernel` +- :173/:177/:184/:193/:210 `distance` / `distances` / `findNearest` / `findWithin` / `union` | instance mirrors of the static functions above, byte-identical results, no new behavior +- :216 `EngramLib.session() -> Session` | factory, equivalent to `Session()` + +### Result type: Match.swift +- :14 `public struct Match: Hashable, Sendable, Codable` | `index: Int` (position in caller's candidate array), `distance: Int` (Hamming distance from probe) +- :18 `init(index:distance:)` | public memberwise init +- :24-28 `extension Match: Comparable` | `<` compares `distance` first, `index` second; this IS the sort/tiebreak contract used by `findNearest`/`findWithin` ## INVARIANTS / GOTCHAS -- Kernel is resolved exactly once per process (`_engramLibCachedKernel`, EngramLib.swift:33), at module load, not per call. Do not reintroduce per-call kernel resolution — it was removed as a measured, needless dispatch cost (see file header comment, Phase 4.3 / decision 2026-05-28 §6.4.3). -- DO NOT REIMPLEMENT SUBSTRATE MATH (EngramLib.swift:14-24 banner comment). Every primitive this library exposes — Hamming distance, batch distance, top-K, OR-reduce — already exists in SubstrateTypes/SubstrateKernel/SubstrateML; EngramLib only forwards to it. -- Tie-break rule is fixed and load-bearing: distance ascending, then candidate index ascending (`Match.<`, Match.swift:25-27). `findNearest` and `findWithin` both depend on this for deterministic, reproducible output — same probe + same candidate list + same order = same result, always. +- Kernel is resolved exactly once per process (`_engramLibCachedKernel`, EngramLib.swift:33), at module load, not per call. Do not reintroduce per-call kernel resolution: it was removed as a measured, needless dispatch cost (see file header comment, Phase 4.3 / decision 2026-05-28 §6.4.3). +- DO NOT REIMPLEMENT SUBSTRATE MATH (EngramLib.swift:14-24 banner comment). Every primitive this library exposes, Hamming distance, batch distance, top-K, OR-reduce, already exists in SubstrateTypes/SubstrateKernel/SubstrateML; EngramLib only forwards to it. +- Tie-break rule is fixed and load-bearing: distance ascending, then candidate index ascending (`Match.<`, Match.swift:25-27). `findNearest` and `findWithin` both depend on this for deterministic, reproducible output: same probe + same candidate list + same order = same result, always. - All empty/degenerate-input guards return empty collections or `nil`, never throw and never crash: empty candidates → `[]`/`nil`; `k <= 0` → `[]`; `maxDistance < 0` → `[]`; empty `union` input → `Engram.zero` (the OR identity). -- `Session` and the static functions are result-identical by construction — both bottom out in the same cached kernel. `Session` exists only to save the small per-call cost of re-touching the static cache in a hot loop; it is not a different code path. +- `Session` and the static functions are result-identical by construction: both bottom out in the same cached kernel. `Session` exists only to save the small per-call cost of re-touching the static cache in a hot loop; it is not a different code path. - `Engram` is a type alias, not a wrapper type. Treat the underlying representation (`Fingerprint256`, four `UInt64` blocks) as substrate-owned; construct via `Engram(blocks:_:_:_:)` or substrate constructors, not by touching blocks directly from product code. - Whole library is stateless and thread-safe (file header, EngramLib.swift:7-10): every static method creates no persistent state; `Session` is `Sendable` and safe to share across tasks. -- Rust and Swift legs must stay behaviorally identical: same guard conditions, same tie-break order, same identity values. There is no shared byte-fixture gate here (contrast LatticeLib's conformance JSON) — parity is enforced by keeping `rust/tests/engram_lib_tests.rs` and `Tests/EngramLibTests/` mirrored. Changing a guard or ordering rule in one leg without the other is a silent cross-platform divergence. -- Package has no pinned artifacts, no build-time-only code, and no privacy seams — unlike larger sibling libs (e.g. LatticeLib), there is nothing here gated by an environment variable or a resource bundle. +- Rust and Swift legs must stay behaviorally identical: same guard conditions, same tie-break order, same identity values. There is no shared byte-fixture gate here (contrast LatticeLib's conformance JSON): parity is enforced by keeping `rust/tests/engram_lib_tests.rs` and `Tests/EngramLibTests/` mirrored. Changing a guard or ordering rule in one leg without the other is a silent cross-platform divergence. +- Package has no pinned artifacts, no build-time-only code, and no privacy seams: unlike larger sibling libs (e.g. LatticeLib), there is nothing here gated by an environment variable or a resource bundle. diff --git a/packages/libs/EngramLib/docs/DETAILS.md b/packages/libs/EngramLib/docs/DETAILS.md index 6617720..97272b1 100644 --- a/packages/libs/EngramLib/docs/DETAILS.md +++ b/packages/libs/EngramLib/docs/DETAILS.md @@ -15,115 +15,114 @@ sources: This document walks through every source file in the package. Read `OVERVIEW.md` first for the big picture. The package has two files: the -public API surface and the result type it returns. +public API surface, and the result type it returns. ## EngramLib.swift -This file provides the entire public surface of the library: the `Engram` -type alias, the `EngramLib` enum with its static comparison and retrieval -functions, and the `Session` type for callers that run many comparisons in -a loop. +This file holds the entire public surface of the library. It defines the +`Engram` type alias, the `EngramLib` enum with its static comparison and +retrieval functions, and the `Session` type. Callers that run many +comparisons in a loop use `Session`. ### How It Works -The file opens with a type alias, `public typealias Engram = Fingerprint256`. -`Fingerprint256` is a 256-bit code defined in the substrate's -`SubstrateTypes` package: four 64-bit blocks that together form one -fixed-size fingerprint. EngramLib does not define its own bit -representation. It reuses the substrate's, under a product-facing name. -This indirection matters for one reason: if a future version of the -substrate widens the representation, product code that only ever wrote -`Engram` never has to change. An extension adds `Engram.init(blocks:_:_:_:)`, -a convenience initializer that takes the four blocks as separate arguments, -so callers can write `Engram(blocks: 0xDEAD, 0xBEEF, 0xCAFE, 0xBABE)` -instead of naming the underlying block parameters. - -Every comparison in the file ultimately calls a kernel: an object, supplied -by the substrate's `SubstrateKernel` package, that knows how to run bit -operations efficiently on the current device. The file resolves this -kernel exactly once, at module load time, into the private constant -`_engramLibCachedKernel`. Every static function then reads that same -cached value instead of asking the substrate to choose again. The file's -own comment explains why: the kernel choice does not change while the -program runs, so resolving it on every call was pure overhead, paid on -every single similarity check. +The file opens with a type alias: `public typealias Engram = +Fingerprint256`. `Fingerprint256` is a 256-bit code defined in the +substrate's `SubstrateTypes` package. It packs four 64-bit blocks into one +fixed-size fingerprint. EngramLib does not define its own bit shape. It +reuses the substrate's shape, under a product-facing name. This choice +matters for one reason. If a future substrate version widens the shape, +product code that only ever wrote `Engram` never has to change. An +extension adds `Engram.init(blocks:_:_:_:)`. This convenience initializer +takes the four blocks as separate arguments. Callers can write +`Engram(blocks: 0xDEAD, 0xBEEF, 0xCAFE, 0xBABE)` instead of naming the +underlying block parameters. + +Every comparison in the file calls a kernel in the end. A kernel is an +object, supplied by the substrate's `SubstrateKernel` package, that knows +how to run bit operations well on the current device. The file resolves +this kernel exactly once, at module load time. It stores that kernel in +the private constant `_engramLibCachedKernel`. Every static function then +reads that same cached value, instead of asking the substrate to choose +again. The file's own comment explains why. The kernel choice does not +change while the program runs. Resolving it on every call was pure +overhead, paid on every single check. ### Function Coverage `EngramLib.distance(_:_:)` returns the Hamming distance between two engrams. Hamming distance counts the bit positions where two equal-length -codes differ. Identical engrams return 0; engrams that are exact bit -inverses of each other return 256, the width of the fingerprint. This -function matters because it is the single building block every other -comparison in the file is built from. +codes differ. Identical engrams return zero. Engrams that are exact bit +inverses of each other return 256, the full width of the fingerprint. +This function matters because every other comparison in the file builds +from it. `EngramLib.distances(probe:candidates:)` computes the distance from one -probe engram to every member of a candidate list, returning an array in -the same order as the input. It matters because computing distances one -at a time in a loop would call into the kernel once per candidate; this -function hands the whole batch to the kernel in one call, so the kernel -can process it more efficiently. It returns an empty array immediately -when the candidate list is empty, avoiding any wasted work. - -`EngramLib.findNearest(probe:in:k:)` returns the `k` candidates closest to -the probe, sorted by distance from nearest to farthest, with ties broken -by the candidate's position in the input list. This is the library's core -retrieval function: it answers "which memories are most like this one," -bounded to a fixed count. It returns an empty array when `k` is zero or -negative, or when the candidate list is empty, rather than treating either -case as an error. When `k` is larger than the candidate count, it returns -every candidate rather than failing. - -`EngramLib.findNearest(probe:in:)`, the two-argument overload, is a +probe engram to every candidate in a list. It returns an array in the +same order as the input. This matters because computing distances one at +a time, in a loop, would call into the kernel once per candidate. This +function hands the whole batch to the kernel in one call. The kernel then +processes it more efficiently. It returns an empty array right away when +the candidate list is empty, so no work goes to waste. + +`EngramLib.findNearest(probe:in:k:)` returns the `k` candidates closest +to the probe. It sorts them by distance, from nearest to farthest. Ties +break by the candidate's position in the input list. This is the +library's core retrieval function. It answers which memories look most +like this one, bounded to a fixed count. It returns an empty array when +`k` is zero or negative, or when the candidate list is empty. Neither +case counts as an error. When `k` exceeds the candidate count, the +function returns every candidate rather than failing. + +`EngramLib.findNearest(probe:in:)` is the two-argument overload. It is a convenience for the common case of wanting only the single closest match. -It calls the three-argument version with `k` fixed to 1 and returns the -first result, or `nil` when the candidate list is empty. It matters -because "find the closest one" is common enough to deserve its own name, -rather than making every caller unwrap a one-element array. +It calls the three-argument version with `k` fixed to one. It returns the +first result, or `nil` when the candidate list is empty. This overload +matters because finding the closest one is common enough to earn its own +name. Callers should not have to unwrap a one-element array by hand. `EngramLib.findWithin(probe:in:maxDistance:)` returns every candidate -whose distance from the probe is at most `maxDistance`, sorted the same -way as `findNearest`. It matters because some callers do not know in -advance how many results they want; they want everything within a -similarity radius, however many candidates that turns out to include. It -computes the full distance array first, filters it, and sorts only the -matches that passed the filter, so the sort never touches a candidate it -is about to discard. It also guards against a negative `maxDistance`, -which cannot correspond to any real distance and would otherwise silently -match nothing in a confusing way. - -`EngramLib.union(_:)` performs a bitwise OR across a list of engrams: the -result has a 1-bit at every position where at least one input engram had -one. It matters because a memory system often wants one summary engram -for a whole group — a cohort of related memories, or the union of a -topic's structural features — rather than comparing every group member -individually. An empty input returns the zero engram, which is the -correct identity value for OR: combining it with anything leaves that -thing unchanged. - -`EngramLib.union(_:_:)`, the two-argument overload, ORs exactly two -engrams together. It exists as a lightweight convenience for the common -pairwise case, so a caller merging two engrams does not have to build a +whose distance from the probe sits at or under `maxDistance`. It sorts +results the same way as `findNearest`. This matters because some callers +do not know in advance how many results they want. They want everything +within a similarity radius, however many candidates that turns out to +include. The function builds the full distance array first, then filters +it. It sorts only the matches that passed the filter. So the sort never +touches a candidate it is about to drop. The function also guards against +a negative `maxDistance`. A negative value cannot match any real +distance. Without the guard, it would silently match nothing, which +would confuse a caller. + +`EngramLib.union(_:)` runs a bitwise OR across a list of engrams. The +result carries a one-bit at every position where at least one input +engram had one. This matters because a memory system often wants one +summary engram for a whole group, such as a cohort of related memories, +or the union of a topic's structural traits. An empty input returns the +zero engram. That is the correct identity value for OR, since combining +it with anything leaves that thing unchanged. + +`EngramLib.union(_:_:)` is the two-argument overload. It ORs exactly two +engrams together. It exists as a light convenience for the common +pairwise case. A caller merging two engrams does not have to build a two-element array just to call the list version. -`EngramLib.Session` is a struct that holds one kernel reference and +`EngramLib.Session` is a struct that holds one kernel reference. It exposes `distance`, `distances`, `findNearest`, `findWithin`, and `union` -as instance methods with the same behavior as their static counterparts. +as instance methods. Each mirrors the behavior of its static counterpart. It matters for one reason: reuse. The static functions already share one -cached kernel internally, so a `Session` does not unlock any new -capability; it exists so that call sites processing thousands of -comparisons in a tight loop can hold one reference and avoid the small, -repeated cost of reaching back into the static cache on every call. A -session is `Sendable`, so the same session can be shared safely across -concurrent tasks. `EngramLib.session()` is a static factory that returns a -new `Session`; it is equivalent to calling `Session()` directly and exists -purely so callers do not need to know the type's name to construct one. +cached kernel inside, so a `Session` unlocks no new capability. It exists +so that call sites processing thousands of comparisons in a tight loop +can hold one reference. That reference skips the small, repeated cost of +reaching back into the static cache on every call. A session is +`Sendable`. So the same session can be shared safely across concurrent +tasks. `EngramLib.session()` is a static factory that returns a new +`Session`. It is equivalent to calling `Session()` directly. It exists so +callers do not need to know the type's name to build one. The private function `kernel()` is the single choke point every static -function in the enum calls through to reach the cached kernel. Routing -every call through one small function, rather than reading the module -constant directly everywhere, keeps the indirection in one place should -the caching strategy ever need to change. +function calls through to reach the cached kernel. Routing every call +through one small function keeps that indirection in one place. This +matters should the caching strategy ever need to change. ## Match.swift @@ -132,49 +131,50 @@ This file provides `Match`, the value every retrieval function in ### How It Works -A `Match` is a pair of two integers: `index`, the position of the matched -candidate in the caller's original input array, and `distance`, the +A `Match` is a pair of two integers. `index` is the position of the +matched candidate in the caller's original input array. `distance` is the Hamming distance from the probe to that candidate. The type carries no reference to the engram itself, only its position and its distance, -because the caller already has the original candidate array and can look -the engram up by index if needed. Keeping the type this small makes it -cheap to create in bulk during a retrieval scan. +because the caller already holds the original candidate array. The caller +can look the engram up by index if needed. Keeping the type this small +makes it cheap to build in bulk during a retrieval scan. -`Match` conforms to `Hashable`, `Sendable`, and `Codable`, so it can be -stored in sets, passed safely across concurrent code, and serialized for -persistence or logging without any extra work. +`Match` conforms to `Hashable`, `Sendable`, and `Codable`. It can be +stored in sets. It can be passed safely across concurrent code. It can be +serialized for storage or logging, without any extra work. ### Function Coverage `Match.init(index:distance:)` is the memberwise initializer. It is public -so that code outside the package — test suites, or a caller assembling -results from its own logic — can construct a `Match` directly rather than -only receiving one from an `EngramLib` function. - -The `Comparable` conformance, added in a small extension, defines the -ordering that every retrieval function in `EngramLib.swift` relies on: -distance ascending first, and for two matches at equal distance, index -ascending second. This ordering is what makes `findNearest` and -`findWithin` deterministic. Given the same probe and the same candidate -list, the result is always sorted the same way, because the tie-break -rule never depends on anything but the candidate's fixed position in the -input. +so that code outside the package can build a `Match` directly. This +covers test suites, or a caller assembling results from its own logic, +rather than only receiving a `Match` from an `EngramLib` function. + +The `Comparable` conformance sits in a small extension. It defines the +order that every retrieval function in `EngramLib.swift` relies on. +Distance ascends first. For two matches at equal distance, index ascends +second. + +This order is what makes `findNearest` and `findWithin` deterministic. +Given the same probe and the same candidate list, the result always +sorts the same way. The tie-break rule never depends on anything but the +candidate's fixed position in the input. ## Rust Port and Conformance -The `rust/` directory contains a second implementation of the same API, -in two files: `lib.rs` defines the `EngramLib` struct with its associated +The `rust/` directory holds a second implementation of the same API, in +two files. `lib.rs` defines the `EngramLib` struct with its associated functions (`distance`, `find_nearest`, `find_within`, `union`, and the -rest) plus the `Session` type, and `matchx.rs` defines the `Match` struct -with its `Ord` implementation. The function names differ only in the ways -that Swift and Rust naming conventions differ — `findNearest` in Swift is -`find_nearest` in Rust, `findNearest(probe:in:)` becomes -`find_nearest_one` because Rust has no argument-label overloading — but -the behavior, including every empty-input guard and every tie-break rule, -matches the Swift implementation function for function. Both legs depend -on their language's version of `SubstrateTypes` and `SubstrateKernel` and -route their bit math through the same kernel abstraction, so a Swift -caller and a Rust caller comparing the same two engrams compute the same -distance. `rust/tests/engram_lib_tests.rs` mirrors the Swift test suite -in `Tests/EngramLibTests/`; when you change the behavior of one leg, -update the other and its tests together. +rest), plus the `Session` type. `matchx.rs` defines the `Match` struct +with its `Ord` implementation. The function names differ only in the +ways Swift and Rust naming conventions differ. `findNearest` in Swift is +`find_nearest` in Rust. `findNearest(probe:in:)` becomes +`find_nearest_one`, since Rust has no argument-label overloading. The +behavior matches the Swift version function for function, including +every empty-input guard and every tie-break rule. Both legs depend on +their own language's version of `SubstrateTypes` and `SubstrateKernel`. +Both route their bit math through the same kernel abstraction. So a +Swift caller and a Rust caller, comparing the same two engrams, compute +the same distance. `rust/tests/engram_lib_tests.rs` mirrors the Swift +test suite in `Tests/EngramLibTests/`. A change to one leg's behavior +requires the same change to the other leg, and to its tests. diff --git a/packages/libs/EngramLib/docs/OVERVIEW.md b/packages/libs/EngramLib/docs/OVERVIEW.md index 933c956..380afd4 100644 --- a/packages/libs/EngramLib/docs/OVERVIEW.md +++ b/packages/libs/EngramLib/docs/OVERVIEW.md @@ -15,107 +15,109 @@ sources: ## What This Library Does -EngramLib compares engrams. An engram is a 256-bit fingerprint: a short, +EngramLib compares engrams. An engram is a fingerprint. It is a short, fixed-size code computed from a piece of content. Similar content produces -similar fingerprints, so comparing two engrams is fast even though the -content behind them may be long. EngramLib answers three kinds of question -about a set of engrams: how far apart are two of them, which ones are -nearest to a given engram, and how do many of them combine into one summary -engram. - -MOOTx01 is an on-device AI memory system. It stores what an AI observes over -time and helps the AI recall it later. Every stored memory can carry an -engram that summarizes its structure. When the system needs to find +similar fingerprints. So comparing two engrams is fast, even when the +content behind them is long. EngramLib answers three questions about a set +of engrams. How far apart are two of them? Which ones sit nearest to a +given engram? How do many of them combine into one summary engram? + +MOOTx01 is an on-device AI memory system. It stores what an AI observes +over time. It helps the AI recall that memory later. Every stored memory +can carry an engram that sums up its structure. When the system looks for memories that resemble a new one, it does not compare the memories -directly. It compares their engrams instead, using the distance and +directly. It compares their engrams instead. It uses the distance and nearest-neighbor functions this library provides. ## The Problem It Solves -Comparing bit patterns sounds simple, but doing it correctly and quickly -across an entire memory store is not. The building blocks already exist in -a lower-level package called the substrate: routines for measuring the -distance between two 256-bit codes, for scanning many codes to find the -closest ones, and for combining codes with a bitwise OR. The substrate also -picks the fastest way to run these routines on the current device, choosing -among a plain reference implementation and hardware-accelerated versions -for different chip families. - -Product code should not have to know about any of that. A consumer that -wants to find the ten memories most similar to a probe engram should not -have to pick a chip-specific routine, manage a kernel object, or reason -about which hardware path is active. EngramLib is a thin, stable layer over -the substrate that hides those choices. It exposes a small set of static -functions with plain names — `distance`, `findNearest`, `findWithin`, -`union` — and it always picks the best available implementation for the -device it runs on. +Comparing bit patterns sounds simple. Doing it well and fast, across a +whole memory store, is not simple. The building blocks already exist in a +lower-level package called the substrate. The substrate holds routines +that measure the distance between two 256-bit codes. It holds routines +that scan many codes to find the closest ones. It holds routines that +combine codes with a bitwise OR. The substrate also picks the fastest way +to run these routines on the current device. It chooses among a plain +reference version and hardware-tuned versions for different chip +families. + +Product code should not need to know any of that. Picture a consumer that +wants the ten memories most like a probe engram. That consumer should not +have to pick a chip-specific routine. It should not have to manage a +kernel object. It should not have to reason about which hardware path is +active. EngramLib is a thin, stable layer over the substrate that hides +those choices. It exposes a small set of static functions with plain +names: `distance`, `findNearest`, `findWithin`, `union`. It always picks +the best version for the device it runs on. ## How It Works -Every public function in EngramLib forwards its work to a kernel, an -object that knows how to run the underlying bit operations on the current -device. EngramLib selects a kernel once, when the module first loads, and -reuses it for every call. Consumers never see the kernel or choose one -themselves. +Every public function in EngramLib passes its work to a kernel. A kernel +is an object that knows how to run bit operations on the current device. +EngramLib picks a kernel once, when the module first loads. It reuses +that kernel for every call after that. Consumers never see the kernel. +They never pick one themselves. `EngramLib.distance(_:_:)` reports the Hamming distance between two -engrams. Hamming distance is the number of bit positions where two -equal-length codes differ; a smaller number means the two engrams — and -likely the content they summarize — are more alike. `EngramLib.distances` +engrams. Hamming distance counts the bit positions where two equal-length +codes differ. A smaller number means the two engrams are more alike. The +content they sum up likely is more alike too. `EngramLib.distances` computes that same distance from one probe engram to many candidates in a -single call, which is faster than calling `distance` in a loop because the +single call. This is faster than calling `distance` in a loop, because the kernel can batch the work. -`EngramLib.findNearest` and `EngramLib.findWithin` build on that -distance measurement to answer retrieval questions. `findNearest` returns -the closest matches to a probe, ranked by distance, for use when a caller -wants a fixed number of results — the ten memories most like this one, for -example. `findWithin` instead returns every candidate inside a distance -radius, for use when a caller wants everything similar enough, however many -that turns out to be. Both break ties the same way: when two candidates -sit at equal distance, the one that appeared earlier in the input list -comes first. This tie-break rule matters because it makes the result +`EngramLib.findNearest` and `EngramLib.findWithin` build on that distance +measure to answer retrieval questions. `findNearest` returns the closest +matches to a probe, ranked by distance. Use it when a caller wants a fixed +number of results, such as the ten memories most like this one. `findWithin` +instead returns every candidate inside a distance radius. Use it when a +caller wants everything similar enough, however many results that turns +out to be. Both functions break ties the same way. When two candidates sit +at equal distance, the one that appeared earlier in the input list comes +first. This tie-break rule matters, because it makes the result reproducible. The same probe against the same candidate list always produces the same ranking, on every run. -`EngramLib.union` combines many engrams into one, bit by bit, keeping a -bit set in the result wherever any input had it set. This is useful for -building a single summary engram for a group of memories — a cohort, a -conversation, a topic cluster — that carries the structural features of -every member. +`EngramLib.union` combines many engrams into one, bit by bit. It keeps a +bit set in the result wherever any input had that bit set. This helps +build one summary engram for a group of memories, such as a cohort, a +conversation, or a topic cluster. That summary engram carries the +structural traits of every member. -Most calls create a fresh, momentary link to the shared kernel and are -done. For code that runs the same kind of comparison many thousands of -times in a loop, EngramLib also offers a `Session`: a small object that -holds the kernel once and exposes the same functions as instance methods. -A session produces identical results to the static functions; it exists -only to save the small cost of re-resolving the kernel on every call. +Most calls create a brief link to the shared kernel and finish. Some code +runs the same kind of comparison many thousands of times in a loop. For +that code, EngramLib also offers a `Session`. A session is a small object +that holds the kernel once. It exposes the same functions as instance +methods. A session returns the same results as the static functions. It +exists only to skip the small cost of finding the kernel again on every +call. ## How the Pieces Fit -The package is deliberately small: one file defines the public API, and one -file defines its result type. `EngramLib.swift` contains the `EngramLib` -enum with its static functions, the `Session` type for reuse in hot loops, -and the `Engram` type alias itself. `Match.swift` contains `Match`, the -small value returned by every retrieval function, holding the position of -a candidate in its input list and its distance from the probe. +The package stays small on purpose. One file defines the public API. One +file defines its result type. `EngramLib.swift` holds the `EngramLib` +enum with its static functions, the `Session` type for reuse in hot +loops, and the `Engram` type alias itself. `Match.swift` holds `Match`. +`Match` is the small value every retrieval function returns. It carries +the position of a candidate in its input list and its distance from the +probe. `Engram` is a public alias for `Fingerprint256`, a type owned by the substrate's `SubstrateTypes` package. EngramLib does not define its own -fingerprint representation; it borrows the substrate's and gives it a -product-facing name, so that a future change to the underlying -representation does not force every consumer to rename their variables. -The kernel that does the actual bit math comes from `SubstrateKernel`, -the substrate's other half. Because the whole library is two short files -wrapping a single external kernel call, there is no internal topology -worth diagramming; every function's story is "validate the input, then -call the kernel." +fingerprint shape. It borrows the substrate's shape and gives it a +product-facing name. A future change to that shape then does not force +every consumer to rename their variables. The kernel that does the actual +bit math comes from `SubstrateKernel`, the substrate's other half. The +whole library is two short files that wrap a single external kernel call. +So there is no internal layout worth a diagram. Every function tells the +same story: check the input, then call the kernel. ## What Ships in the Package -The package ships the two Swift source files and their test suites, plus a -Rust port in `rust/` that mirrors the Swift API function for function: -`EngramLib::distance`, `EngramLib::find_nearest`, `EngramLib::union`, and -the rest. Both legs depend on their language's version of the substrate -packages and forward to the same kernel logic, so a Swift caller and a Rust -caller comparing the same two engrams get the same distance. +The package ships the two Swift source files and their test suites. It +also ships a Rust port in `rust/` that mirrors the Swift API function for +function: `EngramLib::distance`, `EngramLib::find_nearest`, +`EngramLib::union`, and the rest. Both legs depend on their own +language's version of the substrate packages. Both forward to the same +kernel logic. So a Swift caller and a Rust caller, comparing the same two +engrams, get the same distance. diff --git a/packages/libs/IntellectusLib/docs/AGENT_MAP.md b/packages/libs/IntellectusLib/docs/AGENT_MAP.md index 6ee11a2..e9ea444 100644 --- a/packages/libs/IntellectusLib/docs/AGENT_MAP.md +++ b/packages/libs/IntellectusLib/docs/AGENT_MAP.md @@ -15,71 +15,71 @@ sources: blob: 60dc64a00370e491273c2328409a9d56fc46463b --- -# AGENT_MAP — IntellectusLib +# AGENT_MAP: IntellectusLib PURPOSE: substrate self-report telemetry faculty. Zero-dependency LEAF/floor library. Caller → `Intellectus.report(sample)` → lock-free enabled gate → (if on) installed `StatsSink.receive(_:)`. Off-path = one atomic load + branch, payload never built. -DEPS: imports Foundation (NSLock; StatSample/RecentWindowSink/Intellectus), Synchronization (Atomic; Intellectus.swift only). Zero repo dependencies — this IS the floor of the dependency tree (SubstrateTypes/SubstrateKernel/SubstrateLib/SubstrateML are meant to depend on this in a later mission per Package.swift header; not yet wired as of this commit). Imported by: not enumerated in Package.swift/product graph as of this commit; source-comment call sites (forward references, not verified imports) — `AriaResident.installManagerTelemetry` / `AriaResident.runResidentDaemon` (StatsSink.swift, Intellectus.swift), `LocusKit.EstateVerbs`, `NeuronKit.EstateDreamingSink`, `ObserverSink.PersistenceStatsSink` (StatSample.swift). Rust port in `rust/` mirrors every file 1:1 (sample.rs/sink.rs/window.rs/holder.rs/global.rs/lib.rs); NO shared conformance fixtures — StatSample is opaque caller-supplied data, not a computed/derived value, so there is nothing for the two legs to agree on byte-for-byte. +DEPS: imports Foundation (NSLock; StatSample/RecentWindowSink/Intellectus), Synchronization (Atomic; Intellectus.swift only). Zero repo dependencies: this IS the floor of the dependency tree (SubstrateTypes/SubstrateKernel/SubstrateLib/SubstrateML are meant to depend on this in a later mission per Package.swift header; not yet wired as of this commit). Imported by: not enumerated in Package.swift/product graph as of this commit; source-comment call sites (forward references, not verified imports): `AriaResident.installManagerTelemetry` / `AriaResident.runResidentDaemon` (StatsSink.swift, Intellectus.swift), `LocusKit.EstateVerbs`, `NeuronKit.EstateDreamingSink`, `ObserverSink.PersistenceStatsSink` (StatSample.swift). Rust port in `rust/` mirrors every file 1:1 (sample.rs/sink.rs/window.rs/holder.rs/global.rs/lib.rs); NO shared conformance fixtures: StatSample is opaque caller-supplied data, not a computed/derived value, so there is nothing for the two legs to agree on byte-for-byte. ENTRY POINTS (most callers need only these): -- Intellectus.swift:238 `Intellectus.report(_ make: @autoclosure () -> StatSample)` — the only call substrate hot-path code makes; off when disabled, no payload build -- Intellectus.swift:185 `Intellectus.install(sink: any StatsSink)` — host wires a real sink, once, at startup -- Intellectus.swift:205 `Intellectus.setEnabled(_ enabled: Bool)` — host/daemon flips the gate +- Intellectus.swift:238 `Intellectus.report(_ make: @autoclosure () -> StatSample)`: the only call substrate hot-path code makes; off when disabled, no payload build +- Intellectus.swift:185 `Intellectus.install(sink: any StatsSink)`: host wires a real sink, once, at startup +- Intellectus.swift:205 `Intellectus.setEnabled(_ enabled: Bool)`: host/daemon flips the gate ## Symbol Table -### Sample type — StatSample.swift -- :32 `enum StatSample: Sendable` — the telemetry datum; value type, two cases -- :43 `case metric(name:value:tags:ts:)` — named Double measurement + String:String tags -- :70 `case event(kind:nounType:rowID:estate:ts:)` — substrate lifecycle transition; nounType is Int (caller casts NounType→Int to keep this crate dependency-free) -- :86 `enum EventKind: String, Sendable, Hashable, CaseIterable` — `.capture` (:87) | `.think` (:88); rawValue is the wire string -- :96 `StatSample.ts: Double` (extension) — timestamp regardless of case +### Sample type: StatSample.swift +- :32 `enum StatSample: Sendable`: the telemetry datum; value type, two cases +- :43 `case metric(name:value:tags:ts:)`: named Double measurement + String:String tags +- :70 `case event(kind:nounType:rowID:estate:ts:)`: substrate lifecycle transition; nounType is Int (caller casts NounType→Int to keep this crate dependency-free) +- :86 `enum EventKind: String, Sendable, Hashable, CaseIterable`: `.capture` (:87) | `.think` (:88); rawValue is the wire string +- :96 `StatSample.ts: Double` (extension): timestamp regardless of case -### Receiver contract — StatsSink.swift -- :36 `protocol StatsSink: Sendable` — one requirement -- :47 `receive(_ sample: StatSample)` — must be non-blocking/inexpensive; not self-enforced, caller (Intellectus) owns the enabled gate, sink does not re-check it -- :60 `struct NoOpSink: StatsSink` — default installed sink; discards, O(1), no allocation -- :63 `NoOpSink.shared` — shared singleton instance -- :69 `NoOpSink.receive(_:)` — intentional no-op +### Receiver contract: StatsSink.swift +- :36 `protocol StatsSink: Sendable`: one requirement +- :47 `receive(_ sample: StatSample)`: must be non-blocking/inexpensive; not self-enforced, caller (Intellectus) owns the enabled gate, sink does not re-check it +- :60 `struct NoOpSink: StatsSink`: default installed sink; discards, O(1), no allocation +- :63 `NoOpSink.shared`: shared singleton instance +- :69 `NoOpSink.receive(_:)`: intentional no-op -### Bundled sink — RecentWindowSink.swift -- :63 `final class RecentWindowSink: StatsSink, @unchecked Sendable` — bounded ring buffer + optional forward decorator -- :69 `capacity: Int` — fixed at init, clamped to min 1 -- :72 `forward: (any StatsSink)?` (private) — decorator target; nil = window-only -- :76 `lock = NSLock()` (private) — guards ring/head/filled/_totalReceived only -- :80-86 `ring`/`head`/`filled`/`_totalReceived` (private) — backing storage; _totalReceived counts every receive() ever, ignoring eviction -- :97 `init(capacity:forward:)` — capacity clamped via `max(1, capacity)` -- :114 `receive(_ sample:)` — O(1) ring write under lock (oldest slot overwritten when full); forward call happens AFTER lock release -- :133 `snapshot() -> [StatSample]` — point-in-time copy, oldest-first; safe to iterate lock-free -- :148 `count: Int` — current retained count, 0...capacity -- :157 `totalReceived: Int` — monotonic lifetime receive count; 0 = "nothing has ever crossed the gate," distinct from count==0 +### Bundled sink: RecentWindowSink.swift +- :63 `final class RecentWindowSink: StatsSink, @unchecked Sendable`: bounded ring buffer + optional forward decorator +- :69 `capacity: Int`: fixed at init, clamped to min 1 +- :72 `forward: (any StatsSink)?` (private): decorator target; nil = window-only +- :76 `lock = NSLock()` (private): guards ring/head/filled/_totalReceived only +- :80-86 `ring`/`head`/`filled`/`_totalReceived` (private): backing storage; _totalReceived counts every receive() ever, ignoring eviction +- :97 `init(capacity:forward:)`: capacity clamped via `max(1, capacity)` +- :114 `receive(_ sample:)`: O(1) ring write under lock (oldest slot overwritten when full); forward call happens AFTER lock release +- :133 `snapshot() -> [StatSample]`: point-in-time copy, oldest-first; safe to iterate lock-free +- :148 `count: Int`: current retained count, 0...capacity +- :157 `totalReceived: Int`: monotonic lifetime receive count; 0 = "nothing has ever crossed the gate," distinct from count==0 -### Global facade + holder — Intellectus.swift -- :44 `final class _IntellectusHolder: @unchecked Sendable` (internal, not public) — all mutable state lives here; `Intellectus` enum is a thin wrapper over the singleton -- :56 `_enabled: Atomic` (private) — lock-free gate, default false -- :62 `_sinkLock = NSLock()` (private) — guards `_sink` only; acquired only on the on-path (enabled) and during install() -- :67 `_sink: any StatsSink` (private, var) — starts as `NoOpSink.shared` -- :77 `install(sink:)` — replace sink under lock; rare/write-path call -- :91 `setEnabled(_:)` — `.releasing` store -- :100 `isEnabled: Bool` — `.acquiring` load -- :118 `report(_ make: @autoclosure () -> StatSample)` — OFF-path: single atomic load(.acquiring) + branch, autoclosure never forced. ON-path: snapshot sink under lock, release lock, THEN evaluate autoclosure, THEN `sink.receive(...)` — payload build and sink call both happen outside the lock -- :141 `let _intellectus = _IntellectusHolder()` — module-private global singleton; the only instance the public API touches -- :172 `enum Intellectus` — public, caseless namespace; all members static -- :185 `Intellectus.install(sink:)` — forwards to `_intellectus.install` -- :205 `Intellectus.setEnabled(_:)` — forwards to `_intellectus.setEnabled`; driven in production by `AriaResident.runResidentDaemon`'s poll loop reading the stats-store monitoring flag (per doc comment; that call site is outside this package) -- :210 `Intellectus.isEnabled: Bool` — forwards to `_intellectus.isEnabled` -- :238 `Intellectus.report(_:)` — public entry; re-declares `@autoclosure` so short-circuit holds end-to-end from call site to atomic check +### Global facade + holder: Intellectus.swift +- :44 `final class _IntellectusHolder: @unchecked Sendable` (internal, not public): all mutable state lives here; `Intellectus` enum is a thin wrapper over the singleton +- :56 `_enabled: Atomic` (private): lock-free gate, default false +- :62 `_sinkLock = NSLock()` (private): guards `_sink` only; acquired only on the on-path (enabled) and during install() +- :67 `_sink: any StatsSink` (private, var): starts as `NoOpSink.shared` +- :77 `install(sink:)`: replace sink under lock; rare/write-path call +- :91 `setEnabled(_:)`: `.releasing` store +- :100 `isEnabled: Bool`: `.acquiring` load +- :118 `report(_ make: @autoclosure () -> StatSample)`: OFF-path: single atomic load(.acquiring) + branch, autoclosure never forced. ON-path: snapshot sink under lock, release lock, THEN evaluate autoclosure, THEN `sink.receive(...)`: payload build and sink call both happen outside the lock +- :141 `let _intellectus = _IntellectusHolder()`: module-private global singleton; the only instance the public API touches +- :172 `enum Intellectus`: public, caseless namespace; all members static +- :185 `Intellectus.install(sink:)`: forwards to `_intellectus.install` +- :205 `Intellectus.setEnabled(_:)`: forwards to `_intellectus.setEnabled`; driven in production by `AriaResident.runResidentDaemon`'s poll loop reading the stats-store monitoring flag (per doc comment; that call site is outside this package) +- :210 `Intellectus.isEnabled: Bool`: forwards to `_intellectus.isEnabled` +- :238 `Intellectus.report(_:)`: public entry; re-declares `@autoclosure` so short-circuit holds end-to-end from call site to atomic check ## INVARIANTS / GOTCHAS -- OFF-PATH CONTRACT: when `Intellectus.isEnabled` is false, the `@autoclosure` argument to `report(_:)` is NEVER evaluated — no allocation, no lock, no sink call. This is enforced by ordering the `guard _enabled.load(...)` check BEFORE any reference to `make`. Do not restructure `report(_:)` in a way that forces the closure before the gate check. -- Atomic ordering pairing is load-bearing, not decorative: `setEnabled` stores with `.releasing`, `isEnabled`/the gate load with `.acquiring`. This is what guarantees a `setEnabled(true)` on one thread makes state written before it (e.g., a just-completed `install(sink:)`) visible to `report(_:)` on another thread. Do not change either ordering independently — they are a matched pair, mirrored by Rust's `Ordering::Release`/`Ordering::Acquire` in holder.rs. (Rust's own inline comment on `_enabled` load calls it `Relaxed`-sufficient in one place but the code actually uses `Ordering::Acquire`/`Ordering::Release` — treat the Swift acquire/release pairing as the authoritative contract.) +- OFF-PATH CONTRACT: when `Intellectus.isEnabled` is false, the `@autoclosure` argument to `report(_:)` is NEVER evaluated: no allocation, no lock, no sink call. This is enforced by ordering the `guard _enabled.load(...)` check BEFORE any reference to `make`. Do not restructure `report(_:)` in a way that forces the closure before the gate check. +- Atomic ordering pairing is load-bearing, not decorative: `setEnabled` stores with `.releasing`, `isEnabled`/the gate load with `.acquiring`. This is what guarantees a `setEnabled(true)` on one thread makes state written before it (e.g., a just-completed `install(sink:)`) visible to `report(_:)` on another thread. Do not change either ordering independently: they are a matched pair, mirrored by Rust's `Ordering::Release`/`Ordering::Acquire` in holder.rs. (Rust's own inline comment on `_enabled` load calls it `Relaxed`-sufficient in one place but the code actually uses `Ordering::Acquire`/`Ordering::Release`: treat the Swift acquire/release pairing as the authoritative contract.) - Sink lock discipline: `_sinkLock` / `RecentWindowSink.lock` are held ONLY for the O(1) state mutation (snapshot/copy the sink reference, or write one ring slot). The actual `sink.receive(...)` / forward call ALWAYS runs after the lock is released. This is SPEC I-7 in the source comments. Breaking this (calling receive under the lock) risks deadlock if a sink's own `receive` re-enters this library or takes a lock a caller also holds. - `RecentWindowSink` bound is contractual (SPEC I-8 in source comments): memory is O(capacity) regardless of how many samples are ever received. Do not add unbounded accumulation (e.g., a growing log) to this type. - `RecentWindowSink.totalReceived` counts EVERY direct `receive(_:)` call, including calls made outside the `Intellectus` gate (e.g., in tests that call `window.receive(...)` directly). It is not scoped to gated traffic only. -- `NoOpSink` is the sink installed before any `install(sink:)` call. Because the default `isEnabled` is false, `NoOpSink.receive` is not expected to run in default configuration — but it must remain safe to call, because a host could call `setEnabled(true)` before `install(sink:)`. -- Timestamps are ALWAYS caller-supplied (`Double` epoch seconds). No file in this package reads a clock. Do not add a `Date()`/clock read inside `IntellectusLib` — determinism for replay/testing depends on this. +- `NoOpSink` is the sink installed before any `install(sink:)` call. Because the default `isEnabled` is false, `NoOpSink.receive` is not expected to run in default configuration: but it must remain safe to call, because a host could call `setEnabled(true)` before `install(sink:)`. +- Timestamps are ALWAYS caller-supplied (`Double` epoch seconds). No file in this package reads a clock. Do not add a `Date()`/clock read inside `IntellectusLib`: determinism for replay/testing depends on this. - Zero repo dependencies is a structural constraint, not a preference: Package.swift's header comment explicitly forbids importing SubstrateTypes/SubstrateKernel or any other repo target from this package, because this package is meant to become the dependency floor those targets build on. Adding a repo import here would create the exact layering cycle the package exists to avoid. -- `EventKind` is a two-case, `CaseIterable`, `String`-backed enum (`.capture`/`.think`). The test suite pins `allCases.count == 2` — adding a case is a breaking, deliberate surface change, not a silent addition. +- `EventKind` is a two-case, `CaseIterable`, `String`-backed enum (`.capture`/`.think`). The test suite pins `allCases.count == 2`: adding a case is a breaking, deliberate surface change, not a silent addition. - Platform floor is macOS 26 / iOS 26 specifically because `Synchronization.Atomic` needs it per project policy; do not lower the floor without re-deriving the atomic gate's performance argument. - Rust parity is structural (file-for-file, function-for-function) but NOT byte-fixture-gated like LatticeLib: there is no `rust/tests/fixtures/*.json` here, because `StatSample` values are opaque caller data, not values this library derives from an input. Conformance is judged by matching test suite sections (see rust/tests/intellectus_lib_tests.rs vs. Tests/IntellectusLibTests/IntellectusLibTests.swift §1–§8), not by fixture replay. diff --git a/packages/libs/IntellectusLib/docs/DETAILS.md b/packages/libs/IntellectusLib/docs/DETAILS.md index 87b99e3..c99cc26 100644 --- a/packages/libs/IntellectusLib/docs/DETAILS.md +++ b/packages/libs/IntellectusLib/docs/DETAILS.md @@ -19,228 +19,237 @@ sources: This document walks through every source file in the package. Read `OVERVIEW.md` first for the big picture. Files appear here in the order data -flows through the library: the sample type, the receiver contract, the one -bundled receiver, and finally the global facade that ties everything +flows through the library: the sample type, the receiver contract, and the +one bundled receiver. The global facade comes last. It ties everything together for callers. ## StatSample.swift This file provides `StatSample`, the telemetry datum that flows through the -whole library, and `EventKind`, a small enum used by one of its two cases. - -`StatSample` is a value type — an enum with associated values, not a class — -and it conforms to `Sendable`, meaning the Swift compiler checks that it is -safe to pass between threads or concurrent tasks without extra -synchronization. Value types copy instead of sharing memory, which is what -makes that safety check possible: no two threads can race on the same -storage. - -The enum has two cases. `.metric` carries a dot-separated name (such as -`"locus.capture.latency_ms"`), a floating-point value, an optional -dictionary of string tags for context (such as `["kit": "LocusKit"]`), and a -timestamp. `.event` carries a lifecycle event: an `EventKind` (`.capture` or -`.think`), the integer form of a substrate row's noun type, the row's -identifier string, the estate identifier string, and a timestamp. `EventKind` -distinguishes a caller-driven write (`.capture`) from a substrate-driven -autonomous transition (`.think`) — the two broad classes of activity the -substrate's Brain layer can trigger. +whole library. It also provides `EventKind`, a small enum used by one of +`StatSample`'s two cases. + +`StatSample` is a value type. It is an enum with associated values, not a +class. It conforms to `Sendable`. That means the Swift compiler checks that +it is safe to pass between threads or concurrent tasks without extra +synchronization. Value types copy instead of sharing memory. That copying is +what makes the safety check possible, since no two threads can then race on +the same storage. + +The enum has two cases. `.metric` carries a dot-separated name, such as +`"locus.capture.latency_ms"`. It also carries a floating-point value, an +optional dictionary of string tags for context, and a timestamp. An example +tag dictionary looks like `["kit": "LocusKit"]`. `.event` carries a +lifecycle event instead. It carries an `EventKind`, either `.capture` or +`.think`. It also carries the integer form of a substrate row's noun type, +the row's identifier string, the estate identifier string, and a timestamp. +`EventKind` distinguishes two broad classes of activity. `.capture` marks a +caller-driven write. `.think` marks a substrate-driven autonomous +transition. Both classes can be triggered by the substrate's Brain layer. Every timestamp in the library is a caller-supplied number of seconds since -the epoch, never a value read from the system clock inside this package. This -matters for determinism: a caller that replays the same sequence of events -gets the same timestamps back, and tests can construct samples with fixed, -predictable times instead of racing against the real clock. +the epoch. No timestamp is ever read from the system clock inside this +package. This matters for determinism. A caller that replays the same +sequence of events gets the same timestamps back. Tests can construct +samples with fixed, predictable times instead of racing against the real +clock. -`ts`, a computed property added in an extension, returns the timestamp +`ts` is a computed property added in an extension. It returns the timestamp regardless of which case the sample is. This spares every caller that only -cares about ordering samples in time from writing a `switch` over both cases. +cares about ordering samples in time. Such a caller need not write a +`switch` over both cases. ## StatsSink.swift This file provides `StatsSink`, the protocol every receiver of telemetry -samples must conform to, and `NoOpSink`, the receiver installed before any -host supplies its own. +samples must conform to. It also provides `NoOpSink`, the receiver installed +before any host supplies its own. -A protocol in Swift is a contract: it lists the methods and properties a +A protocol in Swift is a contract. It lists the methods and properties a conforming type must supply, without dictating how. `StatsSink` requires -exactly one method, `receive(_:)`, and requires conformers to be `Sendable`, -because the one sink installed at any moment is held in global state and -called from whatever thread happens to be reporting a sample at the time. +exactly one method, `receive(_:)`. It also requires conformers to be +`Sendable`, because the one sink installed at any moment lives in global +state. That sink is called from whatever thread happens to be reporting a +sample at the time. `receive(_:)` delivers one `StatSample` to the sink. The file documents an -expectation, not an enforced rule: implementations should return quickly and -should not block. Anything slow — writing to a database, sending data over a -network — belongs in the sink's own background work, scheduled from inside -`receive(_:)`, not run synchronously inside it. A slow sink would otherwise -add latency to whatever substrate code path triggered the sample. - -`NoOpSink` is the simplest possible conformer: `receive(_:)` does nothing and -returns immediately. It exists so the global holder in `Intellectus.swift` -always has a valid sink installed, even before a host calls `install(sink:)`. -Because monitoring is off by default, `NoOpSink.receive(_:)` is not expected -to run in ordinary use — the enabled gate in `Intellectus.swift` stops a -sample before it ever reaches a sink — but the no-op is still the correct, -safe answer if a host enables monitoring before installing a real sink. +expectation rather than an enforced rule. Implementations should return +quickly. They should not block. Anything slow, such as writing to a database +or sending data over a network, belongs in the sink's own background work. +That background work should be scheduled from inside `receive(_:)`, not run +synchronously inside it. A slow sink would otherwise add latency to whatever +substrate code path triggered the sample. + +`NoOpSink` is the simplest possible conformer. Its `receive(_:)` does +nothing and returns immediately. It exists so the global holder in +`Intellectus.swift` always has a valid sink installed. This holds even +before a host calls `install(sink:)`. Because monitoring is off by default, +`NoOpSink.receive(_:)` is not expected to run in ordinary use. The enabled +gate in `Intellectus.swift` stops a sample before it ever reaches a sink. +Still, the no-op is the correct, safe answer if a host enables monitoring +before installing a real sink. ## RecentWindowSink.swift This file provides `RecentWindowSink`, the one concrete `StatsSink` -implementation the library ships: a fixed-size, in-memory buffer of the most -recently received samples. +implementation the library ships. It is a fixed-size, in-memory buffer of +the most recently received samples. -The buffer is what programmers call a ring buffer, or FIFO ring (first in, -first out): a fixed-length array where new entries overwrite the oldest slot -once the array is full, so the memory used never grows past the buffer's -capacity no matter how many samples arrive over the buffer's lifetime. This -bound is a deliberate contract (documented in the file as SPEC I-8): a -component that is supposed to help observe the system must not itself become -a source of unbounded memory growth. +The buffer is what programmers call a ring buffer, or FIFO ring. FIFO means +first in, first out. It is a fixed-length array. New entries overwrite the +oldest slot once the array is full. So the memory used never grows past the +buffer's capacity, no matter how many samples arrive over the buffer's +lifetime. This bound is a deliberate contract, documented in the file as +SPEC I-8. A component that helps observe the system must not itself become a +source of unbounded memory growth. `RecentWindowSink` optionally wraps a second sink, called the forward sink. Every sample the window records is, after recording, also handed to the -forward sink if one was supplied at construction. This decorator arrangement -— one sink wrapping another — lets a host keep a live, readable window of -recent activity and simultaneously persist every sample to durable storage, -using a single sink installed with `Intellectus.install(sink:)`. +forward sink if one was supplied at construction. This is a decorator +arrangement, one sink wrapping another. It lets a host keep a live, readable +window of recent activity. At the same time, the host can persist every +sample to durable storage. Both effects come from a single sink installed +with `Intellectus.install(sink:)`. `init(capacity:forward:)` builds the sink. The requested capacity is clamped -to a minimum of one: a zero or negative capacity would make the window -useless, so the initializer silently corrects it rather than crashing or -throwing, treating the input as a caller mistake rather than a fatal -condition. - -`receive(_:)` is the `StatsSink` method. It takes an internal lock, writes the -sample into the ring buffer's next slot, advances the write position, and -counts the sample toward a running total, then releases the lock before -calling the forward sink. Running the forward sink outside the lock matters -for the same reason `Intellectus.report(_:)` calls its sink outside its own -lock: a forward sink that itself needs to take a lock — its own, or even this -same one, if it happened to be reinstalled recursively — cannot deadlock -against this method, because this method is not holding anything by the time -the forward sink runs. +to a minimum of one. A zero or negative capacity would make the window +useless. So the initializer silently corrects it rather than crashing or +throwing. It treats the input as a caller mistake, not a fatal condition. + +`receive(_:)` is the `StatsSink` method. It takes an internal lock. It writes +the sample into the ring buffer's next slot. It advances the write position. +It counts the sample toward a running total. Then it releases the lock +before calling the forward sink. Running the forward sink outside the lock +matters for the same reason `Intellectus.report(_:)` calls its sink outside +its own lock. A forward sink might itself need to take a lock, its own or +even this same one if reinstalled recursively. That sink cannot deadlock +against this method, because this method holds nothing by the time the +forward sink runs. `snapshot()` returns a copy of the buffer's current contents, oldest sample -first. Because it is a copy taken while holding the lock only long enough to -read the buffer, a caller can freely inspect or iterate the returned array -without any risk of it changing underneath them, and without blocking -further calls to `receive(_:)` for any longer than the copy itself takes. +first. It is a copy taken while holding the lock only long enough to read +the buffer. So a caller can freely inspect or iterate the returned array. No +risk exists of it changing underneath them. No call blocks further calls to +`receive(_:)` for any longer than the copy itself takes. -`count` reports how many samples the buffer currently holds, which is at most +`count` reports how many samples the buffer currently holds. This is at most its capacity. `totalReceived` reports how many samples have ever been handed to `receive(_:)`, including ones already evicted from the buffer. The -distinction matters for a host reading the window: a `count` of zero could -mean either "monitoring just started" or "monitoring has been off the whole -time," and `totalReceived` tells them apart — a nonzero total with an empty -buffer would be a contradiction that never actually occurs, but a zero total -is the library's explicit signal that nothing has crossed the gate yet. +distinction matters for a host reading the window. A `count` of zero could +mean monitoring just started. Or it could mean monitoring has been off the +whole time. `totalReceived` tells the two apart. A nonzero total with an +empty buffer would be a contradiction that never actually occurs. A zero +total is the library's explicit signal that nothing has crossed the gate +yet. ## Intellectus.swift This file provides the library's single entry point, the `Intellectus` -public enum, together with its private backing implementation, +public enum. It also provides its private backing implementation, `_IntellectusHolder`. -`Intellectus` is what the file calls a stateless namespace: an enum with no -cases, used purely to group related static functions under one name, because -Swift has no notion of free-standing module-level functions. Every piece of -mutable state — the installed sink, and the on/off flag — lives instead in a -single global instance of `_IntellectusHolder`, a class not exposed outside -the package. This split keeps the public surface simple (four static -members) while keeping the actual synchronization logic in one well-tested -place. - -The on/off flag is stored as a `Synchronization.Atomic`, a type -supplied by the platform that lets a single boolean be read and written -directly by the processor without an operating-system lock. The file -explains the reasoning in a comment: an uncontended `NSLock` still costs -roughly six nanoseconds to acquire and release on Apple Silicon, while a -single atomic read costs roughly one nanosecond and can sometimes be -optimized away entirely inside a tight loop. Since the flag is checked on -every single call to `report(_:)` — the library's hottest path by far — this -difference compounds across a busy substrate. - -Reading and writing the flag use matched atomic orderings, `.acquiring` for -reads and `.releasing` for writes. These orderings are a promise to the -compiler and processor: once a read observes the flag as `true`, it is also -guaranteed to observe any other data a writer set up before flipping the flag -to `true`. Without that promise, a thread could theoretically see the flag -turn on before it saw the sink that was installed alongside it, even though -program order says the sink was installed first. - -The installed sink, in contrast, is protected by an ordinary `NSLock`, because -an "any `StatsSink`" value — the type-erased container that lets any -conforming type be stored — cannot be swapped as a single atomic operation -the way a boolean can. The lock is held only long enough to copy the sink -reference out; the sink's `receive(_:)` method always runs after the lock has -already been released, matching the same outside-the-lock discipline +`Intellectus` is what the file calls a stateless namespace. It is an enum +with no cases, used purely to group related static functions under one +name, because Swift has no notion of free-standing module-level functions. +Every piece of mutable state lives instead in a single global instance of +`_IntellectusHolder`, a class not exposed outside the package. That state +includes the installed sink and the on/off flag. This split keeps the +public surface simple, at four static members. It also keeps the actual +synchronization logic in one well-tested place. + +The on/off flag is stored as a `Synchronization.Atomic`. This type is +supplied by the platform. It lets a single boolean be read and written +directly by the processor, without an operating-system lock. The file +explains the reasoning in a comment. An uncontended `NSLock` still costs +roughly six nanoseconds to acquire and release on Apple Silicon. A single +atomic read costs roughly one nanosecond. It can sometimes be optimized away +entirely inside a tight loop. Since the flag is checked on every single call +to `report(_:)`, the library's hottest path by far, this difference compounds +across a busy substrate. + +Reading and writing the flag use matched atomic orderings. Reads use +`.acquiring`. Writes use `.releasing`. These orderings are a promise to the +compiler and processor. Once a read observes the flag as `true`, it is also +guaranteed to observe any other data a writer set up before flipping the +flag to `true`. Without that promise, a thread could theoretically see the +flag turn on before it saw the sink installed alongside it. This could +happen even though program order says the sink was installed first. + +The installed sink, by contrast, is protected by an ordinary `NSLock`. An +"any `StatsSink`" value is a type-erased container that lets any conforming +type be stored. It cannot be swapped as a single atomic operation the way a +boolean can. The lock is held only long enough to copy the sink reference +out. The sink's `receive(_:)` method always runs after the lock has already +been released. This matches the same outside-the-lock discipline `RecentWindowSink.receive(_:)` uses for its own forward sink. `_IntellectusHolder.install(sink:)` replaces the currently installed sink. -Because it is expected to run rarely — typically once, at process startup — -it can afford the ordinary lock; a report already in flight on another -thread finishes with whichever sink it captured, and the very next report -sees the new one. +It is expected to run rarely, typically once, at process startup. So it can +afford the ordinary lock. A report already in flight on another thread +finishes with whichever sink it captured. The very next report sees the new +one. `_IntellectusHolder.setEnabled(_:)` flips the atomic flag. `isEnabled` reads -it back. Both are simple one-line wrappers, but their simplicity is the -point: this is the only place the ordering guarantees above need to be +it back. Both are simple one-line wrappers. Their simplicity is the point, +though: this is the only place the ordering guarantees above need to be stated and reasoned about. `_IntellectusHolder.report(_:)` is the function every other function in this file exists to support quickly. It takes its sample-building argument as an -`@autoclosure`, a Swift feature that wraps an expression in a closure -automatically at the call site, so the expression is not evaluated as soon as -it is written — only when and if the closure is actually called. `report(_:)` -reads the atomic flag first; if it is `false`, the function returns -immediately, and the argument expression is never evaluated at all, which is -exactly the "off-path" cost the whole file is designed around: one atomic +`@autoclosure`. This Swift feature wraps an expression in a closure +automatically at the call site. So the expression is not evaluated as soon +as it is written, only when and if the closure is actually called. +`report(_:)` reads the atomic flag first. If it is `false`, the function +returns immediately. The argument expression is never evaluated at all. This +is exactly the off-path cost the whole file is designed around: one atomic load and a branch, nothing more. Only when the flag is `true` does the -function take the sink lock, copy the sink reference, release the lock, then -finally evaluate the sample-building expression and hand the result to the -sink. +function take the sink lock. It copies the sink reference, releases the +lock, then finally evaluates the sample-building expression and hands the +result to the sink. `Intellectus.install(sink:)`, `Intellectus.setEnabled(_:)`, and -`Intellectus.isEnabled` are thin public wrappers over the matching -`_IntellectusHolder` members, forwarding to the single global instance, +`Intellectus.isEnabled` are thin public wrappers. Each forwards to the +matching `_IntellectusHolder` member, on the single global instance, `_intellectus`. `Intellectus.report(_:)` does the same for the reporting -path, and repeats the `@autoclosure` on its own parameter so the -short-circuit behavior holds all the way from a substrate call site down to -the atomic check, with no intermediate function ever forcing early -evaluation of the sample expression. +path. It repeats the `@autoclosure` on its own parameter. This lets the +short-circuit behavior hold all the way from a substrate call site down to +the atomic check. No intermediate function ever forces early evaluation of +the sample expression. ## Rust Port and Conformance -The `rust/` directory contains the second leg of the library, structured to -mirror the Swift files one for one: `sample.rs` matches `StatSample.swift`, -`sink.rs` matches `StatsSink.swift`, `window.rs` matches -`RecentWindowSink.swift`, and `holder.rs` together with `global.rs` matches -`Intellectus.swift`. `lib.rs` assembles the crate's public surface and adds -one addition the Swift side does not need: a `report!` macro, which gives -Rust callers the same short-circuit evaluation that Swift gets for free from -`@autoclosure` — the macro expands to an `if` check around the sample -expression, so the expression is only compiled into a branch that runs when -monitoring is enabled. +The `rust/` directory contains the second leg of the library. It is +structured to mirror the Swift files one for one. `sample.rs` matches +`StatSample.swift`. `sink.rs` matches `StatsSink.swift`. `window.rs` matches +`RecentWindowSink.swift`. `holder.rs`, together with `global.rs`, matches +`Intellectus.swift`. `lib.rs` assembles the crate's public surface. It adds +one addition the Swift side does not need: a `report!` macro. This macro +gives Rust callers the same short-circuit evaluation that Swift gets for +free from `@autoclosure`. The macro expands to an `if` check around the +sample expression. So the expression compiles into a branch that only runs +when monitoring is enabled. The two legs use different low-level tools for the same guarantees. Swift -uses `Synchronization.Atomic` with `.acquiring`/`.releasing` orderings; -Rust uses `std::sync::atomic::AtomicBool` with `Ordering::Acquire` and -`Ordering::Release` — the same memory-ordering concept under the standard -library's own name. Swift protects the installed sink with `NSLock`; Rust -protects it with `std::sync::Mutex`, wrapping the sink in an `Arc` (Rust's -shared, reference-counted pointer) so it can be cloned out from under the -lock the same way Swift copies the "any `StatsSink`" existential out from -under its own lock. `RecentWindowSink`'s ring buffer is a fixed-size Swift -array with a manually tracked head position on the Swift side, and a -`VecDeque` — a double-ended queue supplied by Rust's standard library — on -the Rust side; both enforce the same bound and the same oldest-first eviction -rule. - -The package's `Tests/IntellectusLibTests/IntellectusLibTests.swift` suite and -the Rust suite in `rust/tests/intellectus_lib_tests.rs` are organized around -matching sections — gating behavior, the no-op sink, thread safety, the -sample type, `EventKind`, a performance smoke check, and the recent-window -sink — so a change in one leg has a direct counterpart to check in the other, -even though, unlike LatticeLib, this package has no shared byte-for-byte -conformance fixtures: nothing here produces a value that two platforms need -to agree on, since a `StatSample` is opaque data the caller constructs, not a -value this library computes from an input. +uses `Synchronization.Atomic` with `.acquiring` and `.releasing` +orderings. Rust uses `std::sync::atomic::AtomicBool` with `Ordering::Acquire` +and `Ordering::Release`, the same memory-ordering concept under the standard +library's own name. Swift protects the installed sink with `NSLock`. Rust +protects it with `std::sync::Mutex`, wrapping the sink in an `Arc`. That is +Rust's shared, reference-counted pointer. It lets the sink be cloned out +from under the lock, the same way Swift copies the "any `StatsSink`" +existential out from under its own lock. `RecentWindowSink`'s ring buffer is +a fixed-size Swift array on the Swift side, with a manually tracked head +position. On the Rust side, it is a `VecDeque`, a double-ended queue +supplied by Rust's standard library. Both sides enforce the same bound and +the same oldest-first eviction rule. + +The package's `Tests/IntellectusLibTests/IntellectusLibTests.swift` suite +and the Rust suite in `rust/tests/intellectus_lib_tests.rs` are organized +around matching sections. Those sections cover gating behavior, the no-op +sink, thread safety, the sample type, `EventKind`, a performance smoke +check, and the recent-window sink. So a change in one leg has a direct +counterpart to check in the other. This differs from LatticeLib in one way, +though: this package has no shared byte-for-byte conformance fixtures. +Nothing here produces a value that two platforms need to agree on, since a +`StatSample` is opaque data the caller constructs. It is not a value this +library computes from an input. diff --git a/packages/libs/IntellectusLib/docs/OVERVIEW.md b/packages/libs/IntellectusLib/docs/OVERVIEW.md index e42fd92..e18a078 100644 --- a/packages/libs/IntellectusLib/docs/OVERVIEW.md +++ b/packages/libs/IntellectusLib/docs/OVERVIEW.md @@ -20,102 +20,118 @@ sources: ## What This Library Does IntellectusLib lets any part of the MOOTx01 substrate report a small fact -about itself while it runs: how long an operation took, how many times -something happened, or that a particular lifecycle event occurred. It calls -this small fact a telemetry sample. A telemetry sample is either a named -measurement, such as "capture latency was 42.5 milliseconds," or a record of -a lifecycle transition, such as "row X was captured in estate Y." - -The library does not decide what to do with a sample. It hands each sample to -a receiver, called a sink, that the host program installs. A host that wants -to persist samples to disk installs a sink that writes to a database. A host -that only wants a live look at recent activity installs a sink that keeps the -last few hundred samples in memory. IntellectusLib supplies one such sink, -`RecentWindowSink`, and a protocol, `StatsSink`, so any host can supply its -own. +about itself. The fact can be a number, such as a latency in milliseconds. +Or the fact can be a lifecycle event, such as a capture. The library calls +this small fact a telemetry sample. + +A telemetry sample takes one of two forms. The first form is a named +measurement. An example is "capture latency was 42.5 milliseconds." The +second form is a record of a lifecycle transition. An example is "row X was +captured in estate Y." + +The library does not decide what to do with a sample. It hands each sample +to a receiver called a sink. The host program installs that sink. A host +that wants to persist samples to disk installs a sink that writes to a +database. A host that only wants a live look at recent activity installs a +different kind of sink. That sink keeps the last few hundred samples in +memory. + +IntellectusLib supplies one such sink, `RecentWindowSink`. It also supplies +a protocol, `StatsSink`, so any host can build its own sink. ## The Problem It Solves -Substrate code runs on a hot path: the same functions execute many times per -second while a user captures memories or the system dreams through them. -Measuring that code is valuable, but a always-on measurement system is not -free. Building a telemetry payload costs time and memory even if nothing -reads the result afterward. +Substrate code runs on a hot path. The same functions execute many times per +second. This happens while a user captures memories or the system dreams +through them. Measuring that code is valuable. An always-on measurement +system is not free, though. Building a telemetry payload costs time and +memory. That cost applies even if nothing reads the result afterward. Most of the time, no one is watching. A production device runs with -monitoring off by default. IntellectusLib is built around one guarantee: when -monitoring is off, reporting a sample costs almost nothing — a single memory -read and a comparison, nothing else. The library never builds a payload, never -takes a lock, and never calls a sink unless a host has explicitly turned -monitoring on. This guarantee is what lets substrate code call -`Intellectus.report(...)` freely, everywhere, without worrying about the cost -of doing so. - -The library also has to work correctly when many threads report samples at -once, since the substrate itself runs concurrently. Turning monitoring on or -off, or swapping in a new sink, must never corrupt a sample that is in flight -on another thread. - -IntellectusLib keeps this promise in two independent implementations. A Swift -leg serves Apple platforms, and a Rust leg (in `rust/`) mirrors it, section for -section, function for function, for hosts that need a Rust build of the -substrate. Both legs use the same design: a lock-free flag for the on/off -gate, and a lock held only briefly around the installed sink. +monitoring off by default. IntellectusLib is built around one guarantee. +When monitoring is off, reporting a sample costs almost nothing. It costs a +single memory read and one comparison, and nothing more. The library never +builds a payload when monitoring is off. It never takes a lock in that case +either. It never calls a sink unless a host has explicitly turned monitoring +on. This guarantee lets substrate code call `Intellectus.report(...)` freely +and everywhere. A caller never has to worry about the cost of that call. + +The library also must work correctly under concurrency. Many threads can +report samples at the same time, since the substrate itself runs +concurrently. Turning monitoring on or off must never corrupt a sample in +flight. Swapping in a new sink must never corrupt one either. + +IntellectusLib keeps this promise through two independent implementations. A +Swift leg serves Apple platforms. A Rust leg, found in `rust/`, mirrors the +Swift leg section for section. It mirrors the Swift leg function for +function too. Hosts that need a Rust build of the substrate use this leg. +Both legs share one design. Each uses a lock-free flag for the on/off gate. +Each holds a lock only briefly, and only around the installed sink. ## How It Works -Reporting a sample happens in one call: `Intellectus.report(...)`. That call -takes an argument built lazily — Swift's `@autoclosure` — meaning the -expression that builds the sample is not evaluated until `report` decides it -is needed. `report` checks a single on/off flag first. If monitoring is off, -the function returns immediately, and the sample-building expression never -runs at all. Nothing is allocated, and no sink is touched. - -If monitoring is on, `report` reads the currently installed sink, briefly -holding a lock so it cannot read a sink in the middle of being replaced, then -releases the lock, builds the sample, and calls the sink's `receive(_:)` -method. The sink runs outside the lock, so a slow or reentrant sink cannot -block a concurrent call that only wants to install a new sink. - -A host controls the gate with `Intellectus.setEnabled(_:)` and installs a -receiver with `Intellectus.install(sink:)`. Both are meant to be called -rarely — typically once, at startup, or when an operator toggles monitoring -on or off. The library ships one ready-to-use sink, `RecentWindowSink`, which -keeps a fixed-size, oldest-evicted-first buffer of the most recent samples in -memory. A host can read that buffer back at any time to see recent activity, -which is useful for a live dashboard or a diagnostic command. `RecentWindowSink` -can also wrap a second sink and forward every sample to it after recording it, -so a host can keep a live window and write to durable storage from a single -installed sink. +Reporting a sample happens in one call, `Intellectus.report(...)`. That call +takes an argument built lazily. Swift calls this feature `@autoclosure`. The +expression that builds the sample does not run right away. It runs only when +`report` decides it is needed. + +`report` checks a single on/off flag first. If monitoring is off, the +function returns right away. The sample-building expression never runs at +all. Nothing is allocated. No sink is touched. + +If monitoring is on, `report` reads the currently installed sink. It briefly +holds a lock while it does this. The lock stops it from reading a sink in +the middle of a swap. `report` then releases the lock. It builds the sample +next. Then it calls the sink's `receive(_:)` method. The sink runs outside +the lock. So a slow or reentrant sink cannot block a concurrent call. That +concurrent call might only want to install a new sink. + +A host controls the gate with `Intellectus.setEnabled(_:)`. A host installs +a receiver with `Intellectus.install(sink:)`. Both calls are meant to run +rarely. A host typically calls them once, at startup. An operator may also +call them later, when toggling monitoring on or off. + +The library ships one ready-to-use sink, `RecentWindowSink`. This sink keeps +a fixed-size buffer of the most recent samples in memory. The buffer evicts +its oldest entry first once it fills. A host can read that buffer at any +time to see recent activity. This is useful for a live dashboard or a +diagnostic command. `RecentWindowSink` can also wrap a second sink. It +forwards every sample to that second sink after recording it. This lets a +host keep a live window and also write to durable storage. Both actions come +from a single installed sink. ## How the Pieces Fit -Figure 1 shows the library's topology — its major parts and how a sample -moves through them. +Figure 1 shows the library's topology. It shows the major parts and how a +sample moves through them. ![Figure 1. Topology of IntellectusLib](topology.svg) *Figure 1. Topology of IntellectusLib. A caller's report flows through the -enabled gate; when open, the sample reaches the installed sink, which may be -the bundled `RecentWindowSink` forwarding on to a second, host-supplied sink. -The dashed region marks the parts a host, not the library, supplies.* - -`StatSample.swift` defines the data that flows through the system: the -`StatSample` enum and its two cases, `.metric` and `.event`. `StatsSink.swift` -defines the receiver contract, `StatsSink`, and the default do-nothing -receiver, `NoOpSink`, which is installed before any host calls `install(sink:)`. -`RecentWindowSink.swift` provides the one concrete sink the library ships. -`Intellectus.swift` ties the pieces together: it holds the current sink and -the on/off flag, and exposes the `Intellectus` enum as the single public entry -point that the rest of the substrate calls. +enabled gate. When the gate is open, the sample reaches the installed sink. +That sink may be the bundled `RecentWindowSink`. It may in turn forward the +sample to a second, host-supplied sink. The dashed region marks the parts a +host supplies, apart from the library itself.* + +`StatSample.swift` defines the data that flows through the system. It +defines the `StatSample` enum and its two cases, `.metric` and `.event`. +`StatsSink.swift` defines the receiver contract, `StatsSink`. It also +defines the default do-nothing receiver, `NoOpSink`. A host relies on +`NoOpSink` before it ever calls `install(sink:)`. `RecentWindowSink.swift` +provides the one concrete sink the library ships. `Intellectus.swift` ties +the pieces together. It holds the current sink and the on/off flag. It +exposes the `Intellectus` enum as the single public entry point. The rest of +the substrate calls through that one entry point. ## What Ships in the Package -The package ships four Swift source files and a matching Rust port in -`rust/`. It has no dependency on any other library in the repository — not -even the lowest substrate libraries — so that those libraries can, in turn, -depend on IntellectusLib without creating a dependency cycle. It depends only -on Foundation and on Swift's `Synchronization` module, both supplied by the -platform. The platform floor is macOS 26 and iOS 26, chosen because that is -the first OS release where `Synchronization.Atomic` is available without a -compatibility fallback. +The package ships four Swift source files. It also ships a matching Rust +port in `rust/`. The package has no dependency on any other library in the +repository. It does not even depend on the lowest substrate libraries. Those +libraries can therefore depend on IntellectusLib in turn. This avoids +creating a dependency cycle. + +The package depends only on Foundation and on Swift's `Synchronization` +module. The platform supplies both. The platform floor is macOS 26 and iOS +26. That floor was chosen for one reason. It is the first OS release that +supports `Synchronization.Atomic` without a compatibility fallback. diff --git a/packages/libs/IntellectusLib/docs/topology.svg b/packages/libs/IntellectusLib/docs/topology.svg index a935eda..f5ca99e 100644 --- a/packages/libs/IntellectusLib/docs/topology.svg +++ b/packages/libs/IntellectusLib/docs/topology.svg @@ -50,7 +50,7 @@ sources: disabled (default) - return — no sink call + return: no sink call @@ -87,5 +87,5 @@ sources: - sample.receive(_:) — outside the lock + sample.receive(_:): outside the lock diff --git a/packages/libs/SubstrateKernel/docs/AGENT_MAP.md b/packages/libs/SubstrateKernel/docs/AGENT_MAP.md index e72f061..02d0c3b 100644 --- a/packages/libs/SubstrateKernel/docs/AGENT_MAP.md +++ b/packages/libs/SubstrateKernel/docs/AGENT_MAP.md @@ -25,104 +25,104 @@ sources: blob: 143a3530a8dee15446fa5711022638e0d61219b2 --- -# AGENT_MAP — SubstrateKernel +# AGENT_MAP: SubstrateKernel -PURPOSE: layer 2 of the four-package substrate split (SubstrateTypes → SubstrateKernel → SubstrateML → SubstrateLib). Bandwidth-bound bit ops over `Fingerprint256`: Hamming distance/top-K, OR-reduce, SimHash, count-fold, popcount; plus BitField (packed-bitmap field extract/write), SHA-256 content hashing, HKDF-SHA256 key derivation, and canonical scalar float-vector ops. No pinned data artifacts — behavior is pure algorithm, not versioned reference data. +PURPOSE: layer 2 of the four-package substrate split (SubstrateTypes → SubstrateKernel → SubstrateML → SubstrateLib). Bandwidth-bound bit ops over `Fingerprint256`: Hamming distance/top-K, OR-reduce, SimHash, count-fold, popcount; plus BitField (packed-bitmap field extract/write), SHA-256 content hashing, HKDF-SHA256 key derivation, and canonical scalar float-vector ops. No pinned data artifacts: behavior is pure algorithm, not versioned reference data. -DEPS: imports SubstrateTypes (Fingerprint256, HyperplaneFamily, CountVector256, FloatSimHashPlanes, SimHash, Hamming, BlockMask), IntellectusLib (Intellectus.report telemetry — zero-dep leaf), Foundation, `simd` (Apple + aarch64 Linux), Metal (Apple only, `#if canImport(Metal)`). Imported by: SubstrateML (depends on SubstrateTypes + SubstrateKernel per dependency graph); hot-path consumers LocusKit, CorpusKit, GeniusLocusKit, EngramLib; SubstrateLib's AuditGate consumes only BitField + SHA256 (not the kernel dispatch layer). Rust port in rust/ mirrors kernel.rs/bit_field.rs/hamming_nn.rs/sha256.rs/hkdf.rs/float_vec_ops.rs function-for-function; kernel_simd.rs (nightly-gated `simd-nightly` feature) covers NEON's ground so there is no neon.rs; kernel_avx512.rs (x86_64-only, DARK path, P11-VAL-015) has no Swift counterpart; Metal has no Rust counterpart (explicit platform waiver, no Linux/Windows GPU path). +DEPS: imports SubstrateTypes (Fingerprint256, HyperplaneFamily, CountVector256, FloatSimHashPlanes, SimHash, Hamming, BlockMask), IntellectusLib (Intellectus.report telemetry: zero-dep leaf), Foundation, `simd` (Apple + aarch64 Linux), Metal (Apple only, `#if canImport(Metal)`). Imported by: SubstrateML (depends on SubstrateTypes + SubstrateKernel per dependency graph); hot-path consumers LocusKit, CorpusKit, GeniusLocusKit, EngramLib; SubstrateLib's AuditGate consumes only BitField + SHA256 (not the kernel dispatch layer). Rust port in rust/ mirrors kernel.rs/bit_field.rs/hamming_nn.rs/sha256.rs/hkdf.rs/float_vec_ops.rs function-for-function; kernel_simd.rs (nightly-gated `simd-nightly` feature) covers NEON's ground so there is no neon.rs; kernel_avx512.rs (x86_64-only, DARK path, P11-VAL-015) has no Swift counterpart; Metal has no Rust counterpart (explicit platform waiver, no Linux/Windows GPU path). ENTRY POINTS (most callers need only these): -- PortableKernel.swift:313 `PortableKernel.kernelForCurrentPlatform() -> SubstrateKernel` — auto-selects SimdKernel (arm64) or ScalarKernel (else); never selects Metal -- PortableKernel.swift:368 `PortableKernel.kernel(of: KernelKind) -> SubstrateKernel` — explicit backend selection (tests, conformance harness, opt-in GPU/NEON) -- BitField.swift:62/86 `BitField.extractField/writeField(...)` — packed-bitmap field access, the F18 centralization surface -- SHA256.swift:30 `SHA256.hash(_ bytes:) -> [UInt8]` — FIPS 180-4 content hash -- HKDF.swift:54 `GrantHKDF.deriveKey(...)` — RFC 5869 HKDF-SHA256 grant scope-key derivation +- PortableKernel.swift:313 `PortableKernel.kernelForCurrentPlatform() -> SubstrateKernel`: auto-selects SimdKernel (arm64) or ScalarKernel (else); never selects Metal +- PortableKernel.swift:368 `PortableKernel.kernel(of: KernelKind) -> SubstrateKernel`: explicit backend selection (tests, conformance harness, opt-in GPU/NEON) +- BitField.swift:62/86 `BitField.extractField/writeField(...)`: packed-bitmap field access, the F18 centralization surface +- SHA256.swift:30 `SHA256.hash(_ bytes:) -> [UInt8]`: FIPS 180-4 content hash +- HKDF.swift:54 `GrantHKDF.deriveKey(...)`: RFC 5869 HKDF-SHA256 grant scope-key derivation ## Symbol Table -### Protocol + reference kernel — PortableKernel.swift -- :49 `protocol SubstrateKernel: Sendable` — every backend conforms; all methods must be bit-identical across conformers -- :54 `kind: KernelKind` — runtime introspection; default `.scalar` (extension :137), every concrete kernel overrides -- :57 `popcount64(_:) -> Int` — 64-bit population count -- :60 `hammingDistance256(_:_:) -> Int` — bit-differences between two Fingerprint256 -- :64 `orReduce256(_:) -> Fingerprint256` — bitwise OR fold; identity = zero -- :69 `hammingTopK(probe:candidates:k:) -> [(index:Int,distance:Int)]` — K nearest by Hamming distance, ties → lower index -- :76 `simhashCompute(subhashes:families:) -> Fingerprint256` — 4-subhash → Fingerprint256 projection -- :91/:97/:102 `hammingDistanceBatch` / `simhashBlockBatch` / `orReduceBatch` — batched variants; protocol-extension defaults are naive loops (:139/:144/:157), always correct, backends may override for speed -- :112/:116 `countFold256` / `countFoldBatch` — per-bit-position set counts across a cohort; default (:166) = `CountVector256.fold`; OR-reduce is this fold's saturating-at-one degenerate case -- :127 `floatSimHashProject(vector:planes:) -> Fingerprint256` — float analogue of simhashCompute; default impl (:183) is the SCALAR ORACLE for this op — fixed left-to-right accumulation order (FP `+` non-associative, order IS the contract); precondition on `planes.dim == vector.count` -- :222 `struct ScalarKernel: SubstrateKernel` — THE ORACLE; every other backend's output must match this bit-for-bit for the same input -- :233 `hammingDistance256` — `a.zip4(b, ^).popcount()` (SubstrateTypes combinator) -- :239 `orReduce256` — `Fingerprint256.reduce4(fingerprints, |)` -- :245 `hammingTopK` — bounded max-heap O(N log k); tie-break lower-index-wins is the cross-backend contract -- :287 `enum KernelKind` — scalar|simd|neon|metal|avx512|avx2 (avx512/avx2 always fall to scalar in THIS Swift build — no Swift AVX implementation exists) -- :296 `enum PortableKernel` — dispatcher namespace -- :313 `kernelForCurrentPlatform()` — arm64→SimdKernel else ScalarKernel; COMPILE-TIME choice, no runtime CPU-feature probing (contrast Rust's Avx512 CPUID guard, which DOES probe at runtime because that backend is unsafe without it); emits `substrate.kernel.backend_selected` telemetry, autoclosure skipped (no clock read) when monitoring off -- :355 `currentArchTag` — static string ("arm64"/"x86_64"/"other"), must track the `#if arch()` predicate above it -- :368 `kernel(of:)` — explicit selector; `.metal` falls back to ScalarKernel if `MetalKernel()` init fails (no GPU); `.avx512`/`.avx2` always → ScalarKernel -- :406 `assertEqual(lhs:rhs:probe:candidates:k:)` — hammingTopK-based bit-identity check; conformance-suite building block -- :430 `ScalarKernelScored` / :440 `ScalarKernelMaxHeap` (internal) — (distance, index) max-heap shared plumbing; root = worst retained - -### SIMD backend — PortableKernel-SIMD.swift -- :39 `struct SimdKernel: SubstrateKernel` — selected by kernelForCurrentPlatform() on arm64; `import simd` → NEON codegen -- :65 `hammingDistance256` — SIMD4 XOR + per-lane nonzeroBitCount, summed -- :114 `hammingTopK` — sorted-ladder maintenance (NOT a heap); identical sorted output + tie-break to ScalarKernel -- :214 `simhashBlockBatch` — uses :233 `PackedFamily` (SoA repack of 64 hyperplanes into 16 groups of 4) built ONCE per batch, amortized across inputs via :275 `simhashBlockSIMD` -- :376 `countFold256` — bit-sliced vertical binary counter across SIMD4 lanes; O(fingerprints × ~2 ops) vs scalar's bit-by-bit walk; conformance-gated against `CountVector256.fold` - -### NEON backend — PortableKernel-NEON.swift -- :38 `struct NeonKernel: SubstrateKernel` — NOT auto-selected; reachable only via `PortableKernel.kernel(of: .neon)`; exists to be BENCHMARKED against SimdKernel, not to replace it -- :63 `toBytes(_:) -> SIMD32` (static) — reframes Fingerprint256 at byte level via single 256-bit load (not byte-by-byte copy) -- :87 `popcountBytes(_:)` (static) — 32 scalar nonzeroBitCount calls; Swift does NOT auto-vectorize this into one `cnt.16b` (measured gap, see file header) -- :104/:113/:128 `hammingDistance256` / `hammingDistanceBatch` / `hammingTopK` — byte-level XOR + popcount + horizontal sum; hammingTopK uses same ladder pattern as SimdKernel -- :170/:182/:186 `orReduce256` / `orReduceBatch` / `simhashCompute` — NOT re-framed at byte level; identical to SimdKernel's implementation (byte reframing judged not to help these ops) - -### Metal backend — PortableKernel-Metal.swift (Apple-only, `#if canImport(Metal)`) -- :67 `MetalBufferPool` (fileprivate, `@unchecked Sendable`) — persistent GPU buffers sized for `maxN`; caller-enforced per-call synchronization, harness is single-threaded -- :110 `struct MetalKernel: SubstrateKernel` — NOT auto-selected; reachable only via `PortableKernel.kernel(of: .metal)` -- :128 `defaultMaxN = 100_000` — buffer-pool sizing default (~3.6 MB), tuned to dreaming-daemon batch-index scale -- :134 `init?(maxN:)` — returns nil on ANY setup failure (no GPU, shader compile fail, pipeline fail); PortableKernel.kernel(of:) treats nil as "fall back to ScalarKernel", not an error -- :172 `popcount64` / :182 `hammingDistance256` — inherited scalar semantics; per-pair GPU dispatch not worth its ~10–30µs overhead -- :203 `hammingDistanceBatch` — :210 dispatches :221 `dispatchWithPool` (N ≤ maxN, reuse buffers) or :283 `dispatchWithFreshBuffers` (N > maxN, allocate per-call) -- :418 `scalarFallback` (private) — CPU fallback on ANY Metal failure (buffer alloc, command-buffer creation, GPU execution error) — never crashes or returns a wrong/partial result -- :364 `hammingTopK` — GPU batched distance + CPU heap-of-K selection (GPU-side top-K selection judged not worth it for small K) -- :440 `shaderSource` (private static) — embedded Metal shader source (`hamming_distance_kernel`); must be kept byte-for-byte in sync with the canonical `.metal` file or the conformance gate's CRC check fails - -### Standalone top-K reference — HammingNN.swift -- :32 `struct HammingNNHit: Hashable, Sendable` — (rowID: UUID, distance: Int) +### Protocol + reference kernel: PortableKernel.swift +- :49 `protocol SubstrateKernel: Sendable`: every backend conforms; all methods must be bit-identical across conformers +- :54 `kind: KernelKind`: runtime introspection; default `.scalar` (extension :137), every concrete kernel overrides +- :57 `popcount64(_:) -> Int`: 64-bit population count +- :60 `hammingDistance256(_:_:) -> Int`: bit-differences between two Fingerprint256 +- :64 `orReduce256(_:) -> Fingerprint256`: bitwise OR fold; identity = zero +- :69 `hammingTopK(probe:candidates:k:) -> [(index:Int,distance:Int)]`: K nearest by Hamming distance, ties → lower index +- :76 `simhashCompute(subhashes:families:) -> Fingerprint256`: 4-subhash → Fingerprint256 projection +- :91/:97/:102 `hammingDistanceBatch` / `simhashBlockBatch` / `orReduceBatch`: batched variants; protocol-extension defaults are naive loops (:139/:144/:157), always correct, backends may override for speed +- :112/:116 `countFold256` / `countFoldBatch`: per-bit-position set counts across a cohort; default (:166) = `CountVector256.fold`; OR-reduce is this fold's saturating-at-one degenerate case +- :127 `floatSimHashProject(vector:planes:) -> Fingerprint256`: float analogue of simhashCompute; default impl (:183) is the SCALAR ORACLE for this op: fixed left-to-right accumulation order (FP `+` non-associative, order IS the contract); precondition on `planes.dim == vector.count` +- :222 `struct ScalarKernel: SubstrateKernel`: THE ORACLE; every other backend's output must match this bit-for-bit for the same input +- :233 `hammingDistance256`: `a.zip4(b, ^).popcount()` (SubstrateTypes combinator) +- :239 `orReduce256`: `Fingerprint256.reduce4(fingerprints, |)` +- :245 `hammingTopK`: bounded max-heap O(N log k); tie-break lower-index-wins is the cross-backend contract +- :287 `enum KernelKind`: scalar|simd|neon|metal|avx512|avx2 (avx512/avx2 always fall to scalar in THIS Swift build: no Swift AVX implementation exists) +- :296 `enum PortableKernel`: dispatcher namespace +- :313 `kernelForCurrentPlatform()`: arm64→SimdKernel else ScalarKernel; COMPILE-TIME choice, no runtime CPU-feature probing (contrast Rust's Avx512 CPUID guard, which DOES probe at runtime because that backend is unsafe without it); emits `substrate.kernel.backend_selected` telemetry, autoclosure skipped (no clock read) when monitoring off +- :355 `currentArchTag`: static string ("arm64"/"x86_64"/"other"), must track the `#if arch()` predicate above it +- :368 `kernel(of:)`: explicit selector; `.metal` falls back to ScalarKernel if `MetalKernel()` init fails (no GPU); `.avx512`/`.avx2` always → ScalarKernel +- :406 `assertEqual(lhs:rhs:probe:candidates:k:)`: hammingTopK-based bit-identity check; conformance-suite building block +- :430 `ScalarKernelScored` / :440 `ScalarKernelMaxHeap` (internal): (distance, index) max-heap shared plumbing; root = worst retained + +### SIMD backend: PortableKernel-SIMD.swift +- :39 `struct SimdKernel: SubstrateKernel`: selected by kernelForCurrentPlatform() on arm64; `import simd` → NEON codegen +- :65 `hammingDistance256`: SIMD4 XOR + per-lane nonzeroBitCount, summed +- :114 `hammingTopK`: sorted-ladder maintenance (NOT a heap); identical sorted output + tie-break to ScalarKernel +- :214 `simhashBlockBatch`: uses :233 `PackedFamily` (SoA repack of 64 hyperplanes into 16 groups of 4) built ONCE per batch, amortized across inputs via :275 `simhashBlockSIMD` +- :376 `countFold256`: bit-sliced vertical binary counter across SIMD4 lanes; O(fingerprints × ~2 ops) vs scalar's bit-by-bit walk; conformance-gated against `CountVector256.fold` + +### NEON backend: PortableKernel-NEON.swift +- :38 `struct NeonKernel: SubstrateKernel`: NOT auto-selected; reachable only via `PortableKernel.kernel(of: .neon)`; exists to be BENCHMARKED against SimdKernel, not to replace it +- :63 `toBytes(_:) -> SIMD32` (static): reframes Fingerprint256 at byte level via single 256-bit load (not byte-by-byte copy) +- :87 `popcountBytes(_:)` (static): 32 scalar nonzeroBitCount calls; Swift does NOT auto-vectorize this into one `cnt.16b` (measured gap, see file header) +- :104/:113/:128 `hammingDistance256` / `hammingDistanceBatch` / `hammingTopK`: byte-level XOR + popcount + horizontal sum; hammingTopK uses same ladder pattern as SimdKernel +- :170/:182/:186 `orReduce256` / `orReduceBatch` / `simhashCompute`: NOT re-framed at byte level; identical to SimdKernel's implementation (byte reframing judged not to help these ops) + +### Metal backend: PortableKernel-Metal.swift (Apple-only, `#if canImport(Metal)`) +- :67 `MetalBufferPool` (fileprivate, `@unchecked Sendable`): persistent GPU buffers sized for `maxN`; caller-enforced per-call synchronization, harness is single-threaded +- :110 `struct MetalKernel: SubstrateKernel`: NOT auto-selected; reachable only via `PortableKernel.kernel(of: .metal)` +- :128 `defaultMaxN = 100_000`: buffer-pool sizing default (~3.6 MB), tuned to dreaming-daemon batch-index scale +- :134 `init?(maxN:)`: returns nil on ANY setup failure (no GPU, shader compile fail, pipeline fail); PortableKernel.kernel(of:) treats nil as "fall back to ScalarKernel", not an error +- :172 `popcount64` / :182 `hammingDistance256`: inherited scalar semantics; per-pair GPU dispatch not worth its ~10–30µs overhead +- :203 `hammingDistanceBatch`: :210 dispatches :221 `dispatchWithPool` (N ≤ maxN, reuse buffers) or :283 `dispatchWithFreshBuffers` (N > maxN, allocate per-call) +- :418 `scalarFallback` (private): CPU fallback on ANY Metal failure (buffer alloc, command-buffer creation, GPU execution error): never crashes or returns a wrong/partial result +- :364 `hammingTopK`: GPU batched distance + CPU heap-of-K selection (GPU-side top-K selection judged not worth it for small K) +- :440 `shaderSource` (private static): embedded Metal shader source (`hamming_distance_kernel`); must be kept byte-for-byte in sync with the canonical `.metal` file or the conformance gate's CRC check fails + +### Standalone top-K reference: HammingNN.swift +- :32 `struct HammingNNHit: Hashable, Sendable`: (rowID: UUID, distance: Int) - :51 `enum HammingNN` -- :53 `topK(anchor:candidates:k:blocks:) -> [HammingNNHit]` — NOT part of the SubstrateKernel/PortableKernel dispatch; operates over any `(UUID, Fingerprint256)` sequence, not a dense array; supports partial-block comparison via `blocks: BlockMask` (e.g. topic-only similarity); requires `k > 0` -- tie-break: rowID.uuidString ascending (DIFFERENT key from PortableKernel backends' index-ascending — do not conflate the two top-K contracts) -- :94/:101/:114 `heapLarger` / `heapifyUp` / `heapifyDown` (private) — bounded max-heap, same shape as ScalarKernelMaxHeap but keyed on UUID string not array index - -### Packed-bitmap fields — BitField.swift -- :45 `enum BitField` — F18 atomic-centralization surface; kits must NOT open-code bit-shift/mask logic -- :62 `extractField(_:shift:width:) -> Int64` — precondition: 0≤shift, 1≤width≤64, shift+width≤64 -- :86 `writeField(_:into:shift:width:) -> Int64` — same preconditions; value bits outside `width` are SILENTLY truncated (documented contract, not a bug) -- :135 `maskedEquals(_:mask:expected:) -> Bool` — one-step `(bitmap & mask) == expected`; caller invariant: expected must already be masked/aligned -- :153/:164 `extractFlag` / `writeFlag` — single-bit case; precondition 0≤bit<64 -- :181 `popcount(_:) -> Int` — via `UInt64(bitPattern:).nonzeroBitCount` -- :188 `hammingDistance(_:_:) -> Int` — `popcount(a ^ b)`, single-Int64 version (contrast Fingerprint256's 256-bit version elsewhere) -- :200 `xorFold(_:) -> Int64` — running XOR reduce; empty → 0 - -### Content hashing — SHA256.swift -- :27 `enum SHA256` — FIPS 180-4, dependency-free -- :30 `hash(_ bytes: [UInt8]) -> [UInt8]` — 32-byte digest; ported VERBATIM from GeniusLocusKit's prior in-kit copy so centralization preserves every existing content ID byte-for-byte -- :119 `rotr` (private) — right-rotate helper - -### Key derivation — HKDF.swift -- :37 `enum GrantHKDF` — RFC 5869 HKDF-SHA256, built entirely over this package's own SHA256 (no CryptoKit) for cross-platform byte-identical key derivation -- :54 `deriveKey(inputKeyMaterial:salt:info:outputByteCount:) -> [UInt8]` — public entry; `outputByteCount` ≤ 8160 (32×255) -- :69 `extract(salt:ikm:)` — RFC 5869 §2.2, `hmac(key: salt, data: ikm)` -- :76 `expand(prk:info:length:)` — RFC 5869 §2.3, chained HMAC with counter byte -- :101 `hmac(key:data:) -> [UInt8]` — public (not just internal) because SubstrateLib's KeyedCommitment API reuses this exact construction; RFC 2104, keys > 64 bytes pre-hashed - -### Scalar float-vector oracle — FloatVecOps.swift -- :62 `enum FloatVecOps` — canonical IEEE-754 scalar reference; every faster backend (Accelerate/BLAS/etc.) MUST match bit-for-bit -- :72 `l2Norm(_:) -> Float` — sqrt(sum(x*x)); empty → 0.0 -- :99 `l2Normalize(_:) -> [Float]` — zero-vector PASSTHROUGH (not NaN, not a panic) — the "no information" signal that projects to Engram.zero via FloatSimHash -- :121 `dot(_:_:) -> Float` — precondition a.count == b.count -- :147 `cosine(_:_:) -> Float` — DEFINED ONLY for pre-normalized unit vectors (caller must call l2Normalize first); debug-only assert checks norm ≈ 1.0 within 1e-5, compiled away in release +- :53 `topK(anchor:candidates:k:blocks:) -> [HammingNNHit]`: NOT part of the SubstrateKernel/PortableKernel dispatch; operates over any `(UUID, Fingerprint256)` sequence, not a dense array; supports partial-block comparison via `blocks: BlockMask` (e.g. topic-only similarity); requires `k > 0` +- tie-break: rowID.uuidString ascending (DIFFERENT key from PortableKernel backends' index-ascending: do not conflate the two top-K contracts) +- :94/:101/:114 `heapLarger` / `heapifyUp` / `heapifyDown` (private): bounded max-heap, same shape as ScalarKernelMaxHeap but keyed on UUID string not array index + +### Packed-bitmap fields: BitField.swift +- :45 `enum BitField`: F18 atomic-centralization surface; kits must NOT open-code bit-shift/mask logic +- :62 `extractField(_:shift:width:) -> Int64`: precondition: 0≤shift, 1≤width≤64, shift+width≤64 +- :86 `writeField(_:into:shift:width:) -> Int64`: same preconditions; value bits outside `width` are SILENTLY truncated (documented contract, not a bug) +- :135 `maskedEquals(_:mask:expected:) -> Bool`: one-step `(bitmap & mask) == expected`; caller invariant: expected must already be masked/aligned +- :153/:164 `extractFlag` / `writeFlag`: single-bit case; precondition 0≤bit<64 +- :181 `popcount(_:) -> Int`: via `UInt64(bitPattern:).nonzeroBitCount` +- :188 `hammingDistance(_:_:) -> Int`: `popcount(a ^ b)`, single-Int64 version (contrast Fingerprint256's 256-bit version elsewhere) +- :200 `xorFold(_:) -> Int64`: running XOR reduce; empty → 0 + +### Content hashing: SHA256.swift +- :27 `enum SHA256`: FIPS 180-4, dependency-free +- :30 `hash(_ bytes: [UInt8]) -> [UInt8]`: 32-byte digest; ported VERBATIM from GeniusLocusKit's prior in-kit copy so centralization preserves every existing content ID byte-for-byte +- :119 `rotr` (private): right-rotate helper + +### Key derivation: HKDF.swift +- :37 `enum GrantHKDF`: RFC 5869 HKDF-SHA256, built entirely over this package's own SHA256 (no CryptoKit) for cross-platform byte-identical key derivation +- :54 `deriveKey(inputKeyMaterial:salt:info:outputByteCount:) -> [UInt8]`: public entry; `outputByteCount` ≤ 8160 (32×255) +- :69 `extract(salt:ikm:)`: RFC 5869 §2.2, `hmac(key: salt, data: ikm)` +- :76 `expand(prk:info:length:)`: RFC 5869 §2.3, chained HMAC with counter byte +- :101 `hmac(key:data:) -> [UInt8]`: public (not just internal) because SubstrateLib's KeyedCommitment API reuses this exact construction; RFC 2104, keys > 64 bytes pre-hashed + +### Scalar float-vector oracle: FloatVecOps.swift +- :62 `enum FloatVecOps`: canonical IEEE-754 scalar reference; every faster backend (Accelerate/BLAS/etc.) MUST match bit-for-bit +- :72 `l2Norm(_:) -> Float`: sqrt(sum(x*x)); empty → 0.0 +- :99 `l2Normalize(_:) -> [Float]`: zero-vector PASSTHROUGH (not NaN, not a panic): the "no information" signal that projects to Engram.zero via FloatSimHash +- :121 `dot(_:_:) -> Float`: precondition a.count == b.count +- :147 `cosine(_:_:) -> Float`: DEFINED ONLY for pre-normalized unit vectors (caller must call l2Normalize first); debug-only assert checks norm ≈ 1.0 within 1e-5, compiled away in release ## INVARIANTS / GOTCHAS @@ -131,15 +131,15 @@ ENTRY POINTS (most callers need only these): - TWO DIFFERENT top-K contracts coexist and must not be conflated: `PortableKernel` backends' `hammingTopK` (dense array input, ties → lower ARRAY INDEX) vs. `HammingNN.topK` (candidate-sequence input, ties → lower `rowID.uuidString`, supports partial-block comparison via `BlockMask`). - `kernelForCurrentPlatform()` is a COMPILE-TIME choice (`#if arch(arm64)`), not a runtime CPU-feature probe. It never selects `.metal`; Metal requires explicit `PortableKernel.kernel(of: .metal)`. `.avx512`/`.avx2` have no Swift implementation and always resolve to `ScalarKernel` in this build. - Rust's AVX-512 backend (`kernel_avx512.rs`) is the opposite of the Swift dispatcher: it DOES check `is_x86_feature_detected!("avx512f")` && `("avx512vpopcntdq")` at runtime before every unsafe intrinsic call, because skipping that check is undefined behavior (SIGILL) on CPUs lacking those features. This guard was present from the initial commit and was hardened/test-covered in the 2026-06-28 security review (`tests/avx512_hamming_conformance.rs::cpuid_guard_prevents_intrinsics_without_feature`). `for_current_platform()` never auto-selects it regardless (dark path, P11-VAL-015 gates enable). -- `MetalKernel.init?` returning `nil` is a NORMAL, expected outcome (no GPU / headless CI / virtualization), not an error to surface — callers fall back to scalar. -- `MetalKernel.hammingDistanceBatch` falls back to `scalarFallback` on ANY Metal failure (buffer alloc, command-buffer/encoder creation, GPU execution error) — never crashes, never returns a partial/wrong result. +- `MetalKernel.init?` returning `nil` is a NORMAL, expected outcome (no GPU / headless CI / virtualization), not an error to surface: callers fall back to scalar. +- `MetalKernel.hammingDistanceBatch` falls back to `scalarFallback` on ANY Metal failure (buffer alloc, command-buffer/encoder creation, GPU execution error): never crashes, never returns a partial/wrong result. - `MetalKernel.defaultMaxN` = 100,000: batches at or under this reuse the persistent `MetalBufferPool`; batches over it pay full per-call buffer allocation via `dispatchWithFreshBuffers`. -- `MetalKernel.shaderSource` is a hand-maintained string copy of the canonical `.metal` shader file; the conformance gate's CRC check is the only thing that catches drift between them — keep them in sync manually. -- `BitField.writeField`'s value truncation (bits of `value` outside `width` are silently dropped) is an intentional contract, not a bug — matches packed-row semantics where the field's width defines its value space. +- `MetalKernel.shaderSource` is a hand-maintained string copy of the canonical `.metal` shader file; the conformance gate's CRC check is the only thing that catches drift between them: keep them in sync manually. +- `BitField.writeField`'s value truncation (bits of `value` outside `width` are silently dropped) is an intentional contract, not a bug: matches packed-row semantics where the field's width defines its value space. - `SHA256.hash` was ported VERBATIM from GeniusLocusKit specifically to preserve existing audit-log content IDs byte-for-byte; do not "clean up" the implementation without checking downstream ID stability. -- `HKDF`/`GrantHKDF` uses ONLY the in-repo `SHA256`, never CryptoKit — required for the Swift↔Rust byte-identical key-derivation guarantee (PAR-4-GL1). +- `HKDF`/`GrantHKDF` uses ONLY the in-repo `SHA256`, never CryptoKit: required for the Swift↔Rust byte-identical key-derivation guarantee (PAR-4-GL1). - `GrantHKDF.hmac` is public (not internal) because `SubstrateLib`'s KeyedCommitment API depends on reusing this exact HMAC-SHA256 construction; do not narrow its access without checking that caller. -- No pinned/versioned data artifacts ship in this package (contrast e.g. LatticeLib's Resources/ lexicon+frame+signatures) — everything here is pure algorithm; conformance is enforced by test fixtures, not artifact versioning. -- `AuditGate` (in `SubstrateLib`, NOT in this package) consumes only `BitField` + `SHA256` from here — it does not touch the kernel-dispatch layer at all. +- No pinned/versioned data artifacts ship in this package (contrast e.g. LatticeLib's Resources/ lexicon+frame+signatures): everything here is pure algorithm; conformance is enforced by test fixtures, not artifact versioning. +- `AuditGate` (in `SubstrateLib`, NOT in this package) consumes only `BitField` + `SHA256` from here: it does not touch the kernel-dispatch layer at all. - No NEON or Metal Rust files exist by design: `kernel_simd.rs` (nightly `simd-nightly` feature) covers NEON's aarch64 ground on the Rust side; Metal has no cross-platform equivalent and is an explicit, documented platform waiver. -- Telemetry (`Intellectus.report` / `substrate.kernel.backend_selected`) is off by default; when off, the reporting autoclosure is never evaluated and no clock (`Date()`) is read — selection has zero overhead in the default configuration. +- Telemetry (`Intellectus.report` / `substrate.kernel.backend_selected`) is off by default; when off, the reporting autoclosure is never evaluated and no clock (`Date()`) is read: selection has zero overhead in the default configuration. diff --git a/packages/libs/SubstrateKernel/docs/DETAILS.md b/packages/libs/SubstrateKernel/docs/DETAILS.md index 3b89114..185fb69 100644 --- a/packages/libs/SubstrateKernel/docs/DETAILS.md +++ b/packages/libs/SubstrateKernel/docs/DETAILS.md @@ -28,317 +28,326 @@ sources: # SubstrateKernel Details This document walks through every source file in the package. Read -`OVERVIEW.md` first for the big picture. Files appear here in this -order: the shared protocol and its reference implementation, the three -specialized hardware backends, the standalone top-K primitive, and -finally the three independent utilities — bit fields, hashing, and -float-vector math. +`OVERVIEW.md` first for the big picture. Files appear here in one +order. First comes the shared protocol and its reference +implementation. Next come the three specialized hardware backends. +Then comes the standalone top-K primitive. Last come three independent +utilities: bit fields, hashing, and float-vector math. ## PortableKernel.swift -This file provides the `SubstrateKernel` protocol, the `ScalarKernel` -reference implementation, and the `PortableKernel` dispatcher that -hands callers a concrete kernel. - -The protocol declares every operation a kernel must supply: -`popcount64` (count the 1-bits in a 64-bit word), `hammingDistance256` -(count differing bits between two fingerprints), `orReduce256` -(bitwise-OR a group of fingerprints into one), `hammingTopK` (find the -K candidates nearest a probe fingerprint), `simhashCompute` (fold a -set of 64-bit numbers into a fingerprint), `countFold256` (tally, per -bit position, how many fingerprints in a group have that bit set), and -`floatSimHashProject` (the same folding operation starting from a -vector of decimal numbers instead of integers). A protocol extension -supplies default, correct-but-unoptimized implementations of the -batched variants of each operation, built as simple loops over the -single-item version; a backend that has a faster batched strategy -overrides the default, but every backend is correct even before it -does. - -`ScalarKernel` implements every required method as a straightforward -loop with no special hardware instructions. It is deliberately the -simplest possible correct implementation, because it is the oracle: a -backend is defined as correct when its output for a given input -matches `ScalarKernel`'s output for that same input. Two of its -methods matter beyond their obvious job. `hammingTopK` maintains a -bounded max-heap of the K best candidates seen so far, evicting the -current worst kept candidate whenever a better one arrives, which -keeps the cost proportional to N times the log of K rather than N -times the log of N; ties between equal distances are broken toward the -candidate with the lower index, and every other backend must match -that exact tie-break. `floatSimHashProject`'s default body (defined in -the protocol extension, inherited by every backend that does not -override it) is itself the reference for that operation: for each of -256 hyperplanes it accumulates a signed sum over the input vector in a -fixed left-to-right order, because floating-point addition is not -associative and the order is therefore part of the contract, not an -implementation detail. +This file provides the `SubstrateKernel` protocol. It also provides the +`ScalarKernel` reference implementation. It provides the +`PortableKernel` dispatcher, which hands callers a concrete kernel. + +The protocol declares every operation a kernel must supply. `popcount64` +counts the one-bits in a 64-bit word. `hammingDistance256` counts +differing bits between two fingerprints. `orReduce256` combines a group +of fingerprints with a bitwise OR. `hammingTopK` finds the K candidates +nearest a probe fingerprint. `simhashCompute` folds a set of 64-bit +numbers into a fingerprint. `countFold256` tallies, per bit position, +how many fingerprints in a group have that bit set. `floatSimHashProject` +does the same folding, but starts from a vector of decimal numbers +instead of integers. A protocol extension supplies default batched +variants of each operation. These defaults are simple loops over the +single-item version. They are correct, but not optimized. A backend +with a faster batched strategy overrides the default. Every backend is +correct even before it does. + +`ScalarKernel` implements every required method as a plain loop. It +uses no special hardware instructions. It is deliberately the simplest +possible correct implementation, because it is the oracle. A backend +counts as correct only when its output matches `ScalarKernel`'s output, +for the same input. Two of its methods matter beyond their obvious +job. `hammingTopK` keeps a bounded max-heap of the K best candidates +seen so far. It evicts the current worst kept candidate whenever a +better one arrives. This keeps the cost proportional to N times the +log of K, rather than N times the log of N. Ties between equal +distances break toward the candidate with the lower index. Every other +backend must match that exact tie-break. +`floatSimHashProject`'s default body lives in the protocol extension. +Every backend that does not override it inherits this body, and that +body is itself the reference for the operation. For each of two +hundred fifty-six hyperplanes, it accumulates a signed sum over the +input vector in a fixed left-to-right order. Floating-point addition +is not associative, so the order is part of the contract. It is not +just an implementation detail. `PortableKernel.kernelForCurrentPlatform()` is the entry point most -callers use. On 64-bit ARM (`arm64`, which covers Apple Silicon and -ARM64 Linux) it returns `SimdKernel`; everywhere else it returns -`ScalarKernel`. The choice is fixed at compile time for a given -platform — there is no runtime hardware probing here, unlike the -Rust port's AVX-512 backend, which checks the processor's feature -flags at runtime because that backend is unsafe to run without them. -Every call reports which backend it chose through IntellectusLib's -telemetry system, but only when telemetry is enabled; when it is off, -the reporting closure is never evaluated and no clock is read, so the -selection has no cost in normal operation. `PortableKernel.kernel(of:)` -is the explicit-selection counterpart used by tests and by any caller -that wants a specific backend regardless of platform; requesting -`.metal` falls back to `ScalarKernel` if no GPU is available, and -requesting `.avx512` or `.avx2` always falls back to `ScalarKernel` in -this Swift build, since neither has a Swift implementation. -`PortableKernel.assertEqual` runs `hammingTopK` on two kernels with the -same inputs and reports whether their results match — the building -block the conformance test suite uses to check every backend against -the oracle. +callers use. On 64-bit ARM, known as `arm64`, it returns `SimdKernel`. +This covers both Apple Silicon and ARM64 Linux. Everywhere else it +returns `ScalarKernel`. The choice is fixed at compile time for a +given platform. There is no runtime hardware probing here. This +contrasts with the Rust port's AVX-512 backend, which checks the +processor's feature flags at runtime, because that backend is unsafe +to run without them. Every call reports which backend it chose, +through IntellectusLib's telemetry system, but only when telemetry is +enabled. When telemetry is off, the reporting closure never runs, and +no clock is read. So the selection has no cost in normal operation. +`PortableKernel.kernel(of:)` is the explicit-selection counterpart. +Tests use it. So does any caller that wants a specific backend +regardless of platform. Requesting `.metal` falls back to +`ScalarKernel` if no GPU is available. Requesting `.avx512` or `.avx2` +always falls back to `ScalarKernel` in this Swift build, since neither +has a Swift implementation. `PortableKernel.assertEqual` runs +`hammingTopK` on two kernels with the same inputs. It reports whether +their results match. This is the building block the conformance test +suite uses to check every backend against the oracle. ## PortableKernel-SIMD.swift -This file provides `SimdKernel`, the backend `kernelForCurrentPlatform` -selects on 64-bit ARM. - -`SimdKernel` uses Swift's `simd` module, which represents four 64-bit -numbers as one `SIMD4` value and lets ordinary operators like -XOR and OR act on all four at once. The Swift compiler turns those -operators into single ARM NEON instructions rather than four separate -ones. `hammingDistance256` XORs two fingerprints as `SIMD4` -values, then counts the 1-bits in each of the four resulting 64-bit -lanes and adds the four counts together. `hammingTopK` uses a -different strategy from `ScalarKernel`'s heap: a small sorted array of -the K best candidates found so far ("ladder"), shifted to make room -for a new entry, which the file's comments note is faster than a heap -for the small K values this package expects, but produces the -identical sorted output and tie-break. +This file provides `SimdKernel`. It is the backend +`kernelForCurrentPlatform` selects on 64-bit ARM. + +`SimdKernel` uses Swift's `simd` module. This module represents four +64-bit numbers as one `SIMD4` value. It lets ordinary operators +like XOR and OR act on all four at once. The Swift compiler turns +those operators into single ARM NEON instructions, rather than four +separate ones. `hammingDistance256` XORs two fingerprints as +`SIMD4` values. It then counts the one-bits in each of the four +resulting 64-bit lanes, and adds the four counts together. +`hammingTopK` uses a different strategy from `ScalarKernel`'s heap. It +keeps a small sorted array of the K best candidates found so far, +called a ladder, and shifts it to make room for a new entry. The +file's comments note that this is faster than a heap for the small K +values this package expects. It produces the identical sorted output +and tie-break. `simhashBlockBatch` computes one 64-bit SimHash block for many input -vectors against the same hyperplane family — a fixed set of 64 -reference directions used to decide, for each bit of the output, which -side of that direction the input vector falls on. Rather than compute -that decision one hyperplane at a time for every input, `PackedFamily` -rearranges the family's 64 hyperplanes once, into groups of four, so -the per-input inner loop can process four hyperplanes together with -the same `SIMD4` trick used for Hamming distance. Building this -packed layout costs a little extra time up front, which is worthwhile -because that cost is paid once and the savings apply to every input in -the batch. +vectors, against the same hyperplane family. A hyperplane family is a +fixed set of 64 reference directions. Each direction decides, for one +bit of the output, which side of that direction the input vector falls +on. Rather than compute that decision one hyperplane at a time for +every input, `PackedFamily` rearranges the family's 64 hyperplanes once, +into groups of four. This lets the per-input inner loop process four +hyperplanes together, with the same `SIMD4` trick used for +Hamming distance. Building this packed layout costs a little extra +time up front. That cost is worthwhile, because it is paid once, and +the savings apply to every input in the batch. `countFold256` computes, for a group of fingerprints, how many of them -have each of the 256 bit positions set. Rather than count each -fingerprint's contribution to each bit position one at a time, -`SimdKernel` keeps a growing binary counter for each bit position, held -across `SIMD4` lanes, and folds each new fingerprint in with -two bitwise operations: the counter's next bit is the counter XOR the -fingerprint, and the amount to carry into the following bit is the -counter AND the fingerprint. This produces the same result as adding -one for every set bit, but does the addition four fingerprint-blocks -at a time instead of bit by bit. +have each of the 256 bit positions set. It does not count each +fingerprint's contribution one position at a time. Instead, `SimdKernel` +keeps a growing binary counter for each bit position. It holds this +counter across `SIMD4` lanes, and folds each new fingerprint in +with two bitwise operations. The counter's next bit is the counter XOR the +fingerprint. The amount to carry into the following bit is the counter +AND the fingerprint. This produces the same result as adding one for +every set bit. It does the addition four fingerprint-blocks at a time, +instead of bit by bit. ## PortableKernel-NEON.swift -This file provides `NeonKernel`, an alternate ARM implementation kept -side by side with `SimdKernel` to test a different way of expressing -the same computation. +This file provides `NeonKernel`. It is an alternate ARM implementation, +kept side by side with `SimdKernel`, to test a different way of +expressing the same computation. `SimdKernel` counts bits four 64-bit words at a time. `NeonKernel` -instead treats a fingerprint as 32 individual bytes and counts bits one -byte at a time before adding all 32 counts together. The file's -comments record the reasoning: processors often have an instruction -that counts bits across a whole row of bytes in one step, and the -hope was that framing the computation at the byte level would let the -compiler use that instruction more effectively than the word-level -framing does. Whether it actually helps is a question the package -answers by measurement, not assumption — a companion benchmark -compares the two — and `NeonKernel` exists so that measurement has -something concrete to measure. It is never chosen automatically; -callers reach it only through `PortableKernel.kernel(of: .neon)`. - -Every method mirrors its `SimdKernel` counterpart at the byte level -instead of the word level: `hammingDistance256` and -`hammingDistanceBatch` XOR two byte-vectors and sum per-byte bit -counts, and `hammingTopK` uses the same ladder-based top-K selection -as `SimdKernel`. `orReduce256`, `orReduceBatch`, and `simhashCompute` -are not re-framed at the byte level, because the file's comments judge -that the byte-level trick that helps Hamming distance does not help -those operations, so they use the same implementation `SimdKernel` -uses. +instead treats a fingerprint as 32 individual bytes. It counts bits one +byte at a time, then adds all 32 counts together. The file's comments +record the reasoning. Processors often have an instruction that counts +bits across a whole row of bytes in one step. The hope was that +framing the computation at the byte level would let the compiler use +that instruction better than the word-level framing does. Whether it +actually helps is a question the package answers by measurement, not +assumption. A companion benchmark compares the two. `NeonKernel` exists +so that measurement has something concrete to measure. It is never +chosen automatically. Callers reach it only through +`PortableKernel.kernel(of: .neon)`. + +Every method mirrors its `SimdKernel` counterpart at the byte level, +instead of the word level. `hammingDistance256` and +`hammingDistanceBatch` XOR two byte-vectors, and sum per-byte bit +counts. `hammingTopK` uses the same ladder-based top-K selection as +`SimdKernel`. `orReduce256`, `orReduceBatch`, and `simhashCompute` are +not re-framed at the byte level. The file's comments judge that the +byte-level trick that helps Hamming distance does not help those +operations. So they use the same implementation `SimdKernel` uses. ## PortableKernel-Metal.swift -This file provides `MetalKernel`, which runs the batched Hamming -distance computation on the graphics processor through Apple's Metal -framework, and which is available only on Apple platforms. - -Dispatching work to the GPU has a fixed cost per call — tens of -microseconds to set up buffers and hand off the computation — that -does not shrink as the batch grows. That fixed cost makes GPU dispatch -a poor choice for comparing one pair of fingerprints, so `MetalKernel` -does not override `hammingDistance256`; it inherits the scalar version -for single pairs and only specializes the batched form, -`hammingDistanceBatch`, where the GPU's throughput advantage outweighs -its setup cost. `MetalBufferPool` removes a second fixed cost: rather -than allocate a fresh block of GPU-visible memory on every call, the -kernel allocates its buffers once, sized for up to `defaultMaxN` -(100,000) candidates, and reuses them on every call within that size; -a batch larger than that falls back to `dispatchWithFreshBuffers`, -which allocates for that call only. - -`init?(maxN:)` builds the one-time Metal state: it asks the system for -its default GPU, compiles the compute shader embedded in -`shaderSource`, and builds the buffer pool. Any step failing — -because there is no GPU (a virtualized environment, for instance) or -the shader fails to compile — makes the initializer return `nil`, -which is a normal outcome, not an error condition: `PortableKernel` -falls back to `ScalarKernel` whenever `MetalKernel`'s initializer -returns `nil`. `hammingDistanceBatch` writes the probe and candidate -fingerprints into GPU-visible memory, dispatches one GPU thread per -candidate, waits for the GPU to finish, and reads the resulting -distances back; if any Metal call along that path fails, `scalarFallback` -computes the same batch on the CPU instead, so a transient GPU problem -degrades performance rather than producing a wrong or missing answer. -`hammingTopK` computes distances through the GPU path above, then +This file provides `MetalKernel`. It runs the batched Hamming distance +computation on the graphics processor, through Apple's Metal framework. +It is available only on Apple platforms. + +Dispatching work to the GPU has a fixed cost per call. Setting up +buffers and handing off the computation takes tens of microseconds, +and that cost does not shrink as the batch grows. That fixed cost +makes GPU dispatch a poor choice for comparing one pair of +fingerprints. So `MetalKernel` does not override `hammingDistance256`. +It inherits the scalar version for single pairs. It only specializes +the batched form, `hammingDistanceBatch`, where the GPU's throughput +advantage outweighs its setup cost. `MetalBufferPool` removes a second +fixed cost. Rather than allocate a fresh block of GPU-visible memory on +every call, the kernel allocates its buffers once. It sizes them for +up to `defaultMaxN`, or one hundred thousand candidates, and reuses +them on every call within that size. A batch larger than that falls +back to `dispatchWithFreshBuffers`, which allocates fresh buffers for +that call only. + +`init?(maxN:)` builds the one-time Metal state. It asks the system for +its default GPU. It compiles the compute shader embedded in +`shaderSource`. It builds the buffer pool. Any step can fail: there may +be no GPU, as in a virtualized environment, or the shader may fail to +compile. Either failure makes the initializer return `nil`. That is a +normal outcome, not an error condition. `PortableKernel` falls back to +`ScalarKernel` whenever `MetalKernel`'s initializer returns `nil`. +`hammingDistanceBatch` writes the probe and candidate fingerprints into +GPU-visible memory. It dispatches one GPU thread per candidate. It +waits for the GPU to finish, then reads the resulting distances back. +If any Metal call along that path fails, `scalarFallback` computes the +same batch on the CPU instead. So a transient GPU problem degrades +performance, rather than producing a wrong or missing answer. +`hammingTopK` computes distances through the GPU path above. It then selects the K smallest with the same CPU-side max-heap `ScalarKernel` -uses, because GPU-side selection of a small K from a large result set -is not worth its complexity. +uses. GPU-side selection of a small K, from a large result set, is not +worth its complexity. ## HammingNN.swift -This file provides `HammingNN`, a standalone top-K search over +This file provides `HammingNN`. It is a standalone top-K search over fingerprints that does not go through the `PortableKernel` protocol at all. -Where `PortableKernel`'s backends operate on a dense array of -candidates and select the fastest hardware path, `HammingNN.topK` -operates over any sequence of `(rowID, fingerprint)` pairs and is meant -as a simple, general-purpose reference a caller can use directly +`PortableKernel`'s backends operate on a dense array of candidates, and +select the fastest hardware path. `HammingNN.topK` instead operates +over any sequence of `(rowID, fingerprint)` pairs. It is meant as a +simple, general-purpose reference. A caller can use it directly, without choosing a backend. It also supports comparing only a subset -of a fingerprint's four blocks through its `blocks` parameter — useful -when a caller wants similarity on, say, only the topic-related portion -of a fingerprint rather than the whole thing. Internally it maintains -the same kind of bounded max-heap `ScalarKernel` uses, and it breaks -ties the same way in spirit — deterministically — though by the row's -UUID string rather than by array index, since this function's +of a fingerprint's four blocks, through its `blocks` parameter. This is +useful when a caller wants similarity on just one part of a +fingerprint, such as the topic-related portion, instead of the whole +fingerprint. Internally it +keeps the same kind of bounded max-heap `ScalarKernel` uses. It breaks +ties the same way in spirit, meaning deterministically, though by the +row's UUID string rather than by array index. This function's candidates do not come with array positions of their own. `topK` -requires `k` to be positive and returns the results sorted by distance, +requires `k` to be positive. It returns the results sorted by distance, then by UUID string, so that two runs over the same data always produce the same order. ## BitField.swift -This file provides `BitField`, a set of pure functions for reading and -writing fixed-width fields packed inside a 64-bit integer. - -MOOTx01 packs several small values — flags, small counters, short -codes — into single 64-bit "bitmap" words, because doing so keeps -those words compact and fast to compare. Every package that needs to -read or write one of those packed fields is expected to call through -this file rather than write its own bit-shifting logic, so that a -change to how a field is laid out only has to be made once. -`extractField(_:shift:width:)` reads a field of `width` bits starting -at bit position `shift`; `writeField(_:into:shift:width:)` writes a new -value into that same position while leaving every other bit +This file provides `BitField`. It is a set of pure functions for +reading and writing fixed-width fields packed inside a 64-bit integer. + +MOOTx01 packs several small values, such as flags, small counters, and +short codes, into single 64-bit words, called bitmaps. Doing so keeps +those words compact and fast to compare. Every package that reads or +writes one of those packed fields is expected to call through this +file. It should not write its own bit-shifting logic. That way, a +change to a field's layout only has to be made once. +`extractField(_:shift:width:)` reads a field of `width` bits, starting +at bit position `shift`. `writeField(_:into:shift:width:)` writes a new +value into that same position, while leaving every other bit untouched. `extractFlag` and `writeFlag` are the single-bit special case of the same idea. `maskedEquals` tests whether a bitmap's masked -bits equal an expected value in one step, which is useful when a -caller only wants a yes-or-no answer and does not need the field's -actual value. `popcount`, `hammingDistance`, and `xorFold` round out -the file with the same bit-counting and bit-comparison operations the -kernel layer uses, but defined here over plain 64-bit integers instead -of 256-bit fingerprints, for callers working with a single packed -word rather than a whole fingerprint. +bits equal an expected value in one step. This is useful when a caller +only wants a yes-or-no answer, and does not need the field's actual +value. `popcount`, `hammingDistance`, and `xorFold` round out the file +with the same bit-counting and bit-comparison operations the kernel +layer uses. Here they work over plain 64-bit integers, instead of +256-bit fingerprints. This suits callers with a single packed word, +rather than a whole fingerprint. ## SHA256.swift -This file provides `SHA256`, a complete implementation of the -SHA-256 cryptographic hash algorithm with no dependency on any -system library. +This file provides `SHA256`. It is a complete implementation of the +SHA-256 cryptographic hash algorithm, with no dependency on any system +library. MOOTx01's audit log identifies each entry by a hash of its exact -contents, so that two replicas that received the same entry compute -the same identifier and can recognize it as the same entry without -comparing the whole thing. That only works if every device computes -the hash the same way, which is why this file implements the standard -itself instead of relying on a platform library that could change -behavior between operating system versions. The implementation was -carried over unchanged from an existing in-kit copy specifically so -that centralizing it here would not change any identifier a caller -had already computed and stored. `SHA256.hash(_:)` is the entire -public surface: it takes a list of bytes and returns the 32-byte -digest, following the standard's padding and compression steps -exactly. +contents. That way, two replicas that receive the same entry compute +the same identifier. Each replica can then recognize the entry as the +same, without comparing the whole thing. This only works if every +device computes the hash the same way. That is why this file +implements the standard itself. It does not rely on a platform library +that could change behavior between operating system versions. The +implementation carries +over unchanged from an existing in-kit copy. Centralizing it here does +not change any identifier a caller had already computed and stored. +`SHA256.hash(_:)` is the entire public surface. It takes a list of +bytes and returns the 32-byte digest, following the standard's padding +and compression steps exactly. ## HKDF.swift -This file provides `GrantHKDF`, an implementation of HKDF-SHA256 (a -standard method for deriving new cryptographic keys from existing key -material), built entirely on top of this package's own `SHA256`. +This file provides `GrantHKDF`. It is an implementation of +HKDF-SHA256, a standard method for deriving new cryptographic keys +from existing key material. It builds entirely on top of this +package's own `SHA256`. MOOTx01's grant system needs to derive a key specific to one sharing -grant from an estate's underlying cryptographic identity, without -exposing that identity directly. Building this derivation on the -in-repo `SHA256`, rather than on a system cryptography library, means -a key derived on one platform is guaranteed to be identical to the -same derivation run on another platform, which is essential when two -devices must independently arrive at the same derived key. +grant. It derives this key from an estate's underlying cryptographic +identity, without exposing that identity directly. This derivation +builds on the in-repo +`SHA256`, rather than on a system cryptography library. That way, a +key derived on one platform is guaranteed to match the same derivation +run on another platform. This guarantee matters when two devices must +independently arrive at the same derived key. `GrantHKDF.deriveKey(inputKeyMaterial:salt:info:outputByteCount:)` is -the public entry point; it runs the standard's two steps, extract and +the public entry point. It runs the standard's two steps, extract and expand, in sequence. `extract` and `expand` implement those two steps directly from the specification. `hmac(key:data:)` implements the -keyed-hash construction both steps depend on and is itself public, -because SubstrateLib's commitment-signing feature needs the same -HMAC-SHA256 construction over its own data and this avoids a second +keyed-hash construction both steps depend on. It is itself public, +because `SubstrateLib`'s commitment-signing feature needs the same +HMAC-SHA256 construction over its own data. This avoids a second implementation existing anywhere in the substrate. ## FloatVecOps.swift -This file provides `FloatVecOps`, the canonical scalar implementations -of four vector operations: length, normalization, dot product, and -cosine similarity, all over ordinary lists of decimal numbers. +This file provides `FloatVecOps`. It holds the canonical scalar +implementations of four vector operations: length, normalization, dot +product, and cosine similarity. All four operate over ordinary lists +of decimal numbers. Just as `ScalarKernel` is the oracle for fingerprint operations, -`FloatVecOps` is the oracle for decimal-vector operations: any faster -implementation elsewhere in the substrate — using a platform math -library, for instance — is required to match these functions bit for -bit, not merely approximately. `l2Norm(_:)` computes a vector's length -by summing the square of each element and taking the square root. +`FloatVecOps` is the oracle for decimal-vector operations. Any faster +implementation elsewhere in the substrate must match these functions +bit for bit. This includes an implementation that uses a platform math +library. The match must be exact, not merely approximate. `l2Norm(_:)` +computes a vector's length. It sums +the square of each element, then takes the square root. `l2Normalize(_:)` rescales a vector to length one, which is the form -most similarity comparisons expect; a vector of all zeros has no -meaningful direction to rescale to, so the function returns it -unchanged rather than dividing by zero, and that unchanged zero vector -is itself a meaningful signal elsewhere in the substrate for "no +most similarity comparisons expect. A vector of all zeros has no +meaningful direction to rescale to. So the function returns it +unchanged, rather than dividing by zero. That unchanged zero vector +itself carries meaning. Elsewhere in the substrate, it signals "no information available." `dot(_:_:)` sums the products of corresponding elements from two equal-length vectors. `cosine(_:_:)` measures the -angle between two vectors, but only gives a meaningful answer when -both inputs are already unit-length; rather than normalize them itself -on every call, which would be wasted work when a caller normalizes -once and compares many times, it requires the caller to normalize -first and, in debug builds only, checks that the caller did so. +angle between two vectors. It only gives a meaningful answer when both +inputs are already unit-length. Normalizing on every call would waste +work, when a caller normalizes once and compares many times. So the +function requires the caller to normalize first instead. In debug +builds only, it also checks that the caller did so. ## Rust Port and Conformance -The `rust/` directory contains the second leg of the library: `kernel.rs` +The `rust/` directory holds the second leg of the library. `kernel.rs` mirrors `PortableKernel.swift`'s protocol, `ScalarKernel`, and -dispatcher; `bit_field.rs`, `hamming_nn.rs`, `sha256.rs`, `hkdf.rs`, and -`float_vec_ops.rs` mirror their same-named Swift files function for -function. Two Swift backends have no Rust counterpart. `NeonKernel` -does not need one, because Rust's own portable-SIMD path -(`kernel_simd.rs`, gated behind the nightly-only `simd-nightly` Cargo -feature) already covers the same aarch64 targets. `MetalKernel` cannot -have one: Metal is an Apple-only framework, and the Rust port's -documentation records this explicitly as a platform waiver rather than -a gap. The Rust leg also ships one backend the Swift leg lacks -entirely: `kernel_avx512.rs`, an AVX-512 implementation for x86-64 -processors. It compiles and is reachable through explicit selection, -but `PortableKernel::for_current_platform()` never chooses it — it -stays a dark path until a future performance study proves it belongs -in the default path on real AVX-512 hardware. Because an unguarded call -into its processor-specific instructions would crash on hardware that -lacks them, every one of its entry points checks the processor's -actual feature flags at runtime before making that call, a guard -verified in a 2026-06-28 security review and covered by a dedicated -cross-platform test. Both legs share one conformance obligation: -whichever backend a caller selects, on either leg, its output must -match the scalar reference bit for bit, and the test suites for both -legs enforce this on every change. +dispatcher. Five files mirror their same-named Swift files, function +for function: `bit_field.rs`, `hamming_nn.rs`, `sha256.rs`, `hkdf.rs`, +and `float_vec_ops.rs`. Two Swift backends have no Rust counterpart. +`NeonKernel` does not need one. Rust's own portable-SIMD path, +`kernel_simd.rs`, already covers the same aarch64 targets. This path is +gated behind the nightly-only `simd-nightly` Cargo feature. +`MetalKernel` cannot have one, since Metal is an Apple-only framework. +The Rust port's documentation records this fact as a platform waiver, +not a gap. + +The Rust leg also ships one backend the Swift leg lacks entirely: +`kernel_avx512.rs`, an AVX-512 implementation for x86-64 processors. It +compiles, and is reachable through explicit selection, but +`PortableKernel::for_current_platform()` never chooses it. It stays a +dark path, until a future performance study proves it belongs in the +default path on real AVX-512 hardware. An unguarded call into its +processor-specific instructions would crash on hardware that lacks +them. So every one of its entry points checks the processor's actual +feature flags at runtime, before making that call. This guard was +verified in a 2026-06-28 security review. A dedicated cross-platform +test covers it. + +Both legs share one conformance obligation. Whichever backend a caller +selects, on either leg, its output must match the scalar reference bit +for bit. The test suites for both legs enforce this rule on every +change. diff --git a/packages/libs/SubstrateKernel/docs/OVERVIEW.md b/packages/libs/SubstrateKernel/docs/OVERVIEW.md index 8dc6a85..1cfb71b 100644 --- a/packages/libs/SubstrateKernel/docs/OVERVIEW.md +++ b/packages/libs/SubstrateKernel/docs/OVERVIEW.md @@ -29,142 +29,151 @@ sources: ## What This Library Does -SubstrateKernel computes the small set of bit-level and hash -operations that MOOTx01 runs on every memory it stores. It compares -fingerprints, extracts values from packed bitmaps, and computes -cryptographic hashes. A fingerprint is a short fixed-size code -computed from a piece of content; similar content produces similar -fingerprints, so the system compares things quickly without reading -them in full. This package works with a specific fingerprint shape, -`Fingerprint256`, a 256-bit code held as four 64-bit blocks. - -Comparing two fingerprints means counting how many bit positions -differ between them. That count is called Hamming distance: the -number of positions where two equal-length codes differ. A smaller -Hamming distance means the two fingerprints, and the content behind -them, are more similar. Finding the memories most similar to a query -means finding the fingerprints with the smallest Hamming distance to -a probe fingerprint. This is the single most frequent operation in -MOOTx01's recall path, so SubstrateKernel gives it several -implementations, each tuned for a different kind of hardware. - -SubstrateKernel is layer 2 of a four-package substrate split. Layer 1, -SubstrateTypes, defines pure data types with no logic, including -`Fingerprint256` itself. Layer 2, this package, computes over those -types. Layer 3, SubstrateML, builds learning and graph algorithms on -top of layer 2. Layer 4, SubstrateLib, orchestrates the whole -substrate for its callers. A package that composes libraries into a -larger subsystem is called a kit; kits depend on libraries like this -one, never the reverse. Four kits currently depend on SubstrateKernel -directly for their hot paths: LocusKit, CorpusKit, GeniusLocusKit, and -EngramLib. +`SubstrateKernel` computes bit-level and hash operations. MOOTx01 runs +them on every memory it stores. It compares fingerprints. It extracts +values from packed bitmaps. It computes cryptographic hashes. + +A fingerprint is a short fixed-size code computed from a piece of +content. Similar content produces similar fingerprints. This lets the +system compare things fast, without reading them in full. This package +works with one fingerprint shape, `Fingerprint256`. It is a 256-bit +code held as four 64-bit blocks. + +Comparing two fingerprints means counting how many bit positions differ +between them. This count is called Hamming distance. A smaller Hamming +distance means the two fingerprints are more alike. The content behind +each one is more alike too. + +Finding the memories most similar to a query means finding the +fingerprints with the smallest Hamming distance to a probe fingerprint. +This is the single most frequent operation in MOOTx01's recall path. So +`SubstrateKernel` gives it several implementations. Each one suits a +different kind of hardware. + +`SubstrateKernel` is layer 2 of a four-package substrate split. Layer 1 +is `SubstrateTypes`. It defines pure data types with no logic, including +`Fingerprint256` itself. Layer 2 is this package. It computes over those +types. Layer 3 is `SubstrateML`. It builds learning and graph algorithms +on top of layer 2. Layer 4 is `SubstrateLib`. It orchestrates the whole +substrate for its callers. + +A package that composes libraries into a larger subsystem is called a +kit. Kits depend on libraries like this one. The reverse never happens. +Four kits depend directly on `SubstrateKernel` for their hot paths: +`LocusKit`, `CorpusKit`, `GeniusLocusKit`, and `EngramLib`. ## The Problem It Solves -Two devices must agree on how similar two fingerprints are. MOOTx01 -estates — one user's complete memory store — can federate, meaning -separate devices share and compare memories. If one device's Hamming -distance for a pair of fingerprints disagreed with another device's -distance for the same pair, shared recall would produce different -answers on different hardware. Every operation in this package must -therefore produce the same output for the same input, on every -platform, every time. This package calls that guarantee its -conformance contract, and it treats one implementation, `ScalarKernel`, -as the oracle every other implementation must match bit for bit. - -Meeting that contract while also running fast is harder than meeting -it alone. A scan over a million fingerprints, at 32 bytes each, moves -32 megabytes through memory for a single query. That volume of -straight-line bit manipulation is bandwidth-bound: its cost is -dominated by how fast data moves through the processor, not by how -complex the arithmetic is. Different processors move bandwidth-bound -data at different speeds depending on which instructions they use, so -one fixed implementation cannot be fastest everywhere. SubstrateKernel -resolves this by defining one interface, several interchangeable -implementations behind it, and a single reference implementation that -every one of them is checked against. - -A second, smaller problem is duplication. Bit-field extraction from a -packed bitmap, and content hashing for the audit log, are needed by -many packages upstream of SubstrateKernel. Left unmanaged, each -package would write its own version, and a change to the bit layout -or the hash algorithm would require updating every copy. SubstrateKernel -centralizes both so a single implementation serves every caller. +Two devices must agree on how similar two fingerprints are. A MOOTx01 +estate is one user's complete memory store. Estates can federate. That +means separate devices share and compare memories. Suppose one device's +Hamming distance for a pair of fingerprints disagreed with another +device's distance for that same pair. Shared recall would then give +different answers on different hardware. + +Every operation in this package must therefore produce the same output +for the same input. It must do so on every platform, every time. This +package calls that guarantee its conformance contract. It treats one +implementation, `ScalarKernel`, as the oracle. Every other implementation +must match that oracle bit for bit. + +Meeting that contract while also running fast is hard. A scan over one +million fingerprints, at 32 bytes each, moves 32 megabytes through +memory for a single query. That much straight-line bit manipulation is +bandwidth-bound. Its cost comes from how fast data moves through the +processor, not from how complex the arithmetic is. Different processors +move that data at different speeds, depending on which instructions +they use. So one fixed implementation cannot be fastest everywhere. +`SubstrateKernel` solves this with one interface. Several interchangeable +implementations sit behind it. One reference implementation checks +every one of them. + +A second, smaller problem is duplication. Many packages upstream of +`SubstrateKernel` need bit-field extraction from a packed bitmap. Many +also need content hashing for the audit log. Left unmanaged, each +package would write its own version. A change to the bit layout, or to +the hash algorithm, would then require updating every copy. +`SubstrateKernel` centralizes both operations. One implementation now +serves every caller. ## How It Works The `SubstrateKernel` protocol declares the operations every backend -must supply: Hamming distance between two fingerprints, top-K nearest -neighbor search, bitwise OR-reduction across a group of fingerprints, -SimHash projection (folding a set of numbers into a fingerprint), and -a handful of batched variants of each. `ScalarKernel` implements the -protocol with a plain loop over each fingerprint's four 64-bit blocks. -It is always available, on every platform, and it is the oracle: any -other implementation that disagrees with it on any input is, by -definition, wrong. +must supply. These include Hamming distance between two fingerprints, +top-K nearest neighbor search, and bitwise OR-reduction across a group +of fingerprints. They also include SimHash projection, which folds a +set of numbers into a fingerprint, plus batched variants of each +operation. `ScalarKernel` implements the protocol with a plain loop +over each fingerprint's four 64-bit blocks. It is always available, on +every platform. It is the oracle. Any implementation that disagrees +with it, on any input, is by definition wrong. Three more implementations specialize the same protocol for particular -hardware. `SimdKernel` uses Swift's portable `simd` module, which the -compiler turns into ARM NEON instructions on Apple Silicon. `NeonKernel` -frames the same computation a different way, at the level of -individual bytes rather than 64-bit words, to test whether that shape -compiles to tighter machine code. `MetalKernel` dispatches the batched -Hamming distance computation to the GPU through Apple's Metal -framework, which pays a fixed per-call setup cost but scales well past -roughly one hundred thousand candidates. A caller selects an -implementation through `PortableKernel`, either automatically for the -current platform or explicitly by name for testing. - -Two more file pairs round out the package. `BitField` extracts and -writes fixed-width fields inside a 64-bit packed bitmap — the encoding +hardware. `SimdKernel` uses Swift's portable `simd` module. The +compiler turns this into ARM NEON instructions on Apple Silicon. +`NeonKernel` frames the same computation a different way. It works at +the byte level instead of the 64-bit-word level. This tests whether +that shape compiles to tighter machine code. `MetalKernel` +sends the batched Hamming distance computation to the GPU, through +Apple's Metal framework. This pays a fixed per-call setup cost. It then +scales well past roughly one hundred thousand candidates. A caller +picks an implementation through `PortableKernel`. It can choose one +automatically for the current platform, or ask for one by name for +testing. + +Four more files round out the package. `BitField` extracts and writes +fixed-width fields inside a 64-bit packed bitmap. This is the encoding MOOTx01 uses to pack several small values into one machine word. -`SHA256` computes a standard cryptographic hash, used both to give -each audit-log entry a unique, content-derived identifier and, through -`HKDF`, to derive keys for the estate's cryptographic grants. A last -file, `FloatVecOps`, defines the canonical floating-point vector -operations — length, normalization, dot product, cosine similarity — -that any faster backend for those operations must also match bit for -bit. A final file, `HammingNN`, offers a simpler, general-purpose top-K -search over any sequence of candidates, independent of the -`PortableKernel` backends, for callers that do not need backend -selection. +`SHA256` computes a standard cryptographic hash. It gives each +audit-log entry a unique, content-derived identifier. Through `HKDF`, +it also derives keys for the estate's cryptographic grants. +`FloatVecOps` defines the canonical floating-point vector operations: +length, normalization, dot product, and cosine similarity. Any faster +backend for those operations must match it bit for bit. `HammingNN` +offers a simpler, general-purpose top-K search over any sequence of +candidates. It works independently of the `PortableKernel` backends, +for callers who do not need backend selection. ## How the Pieces Fit -Figure 1 shows the library's topology — its major parts and how data -moves between them. +Figure 1 shows the library's topology. It shows the major parts and how +data moves between them. ![Figure 1. Topology of SubstrateKernel](topology.svg) *Figure 1. Topology of SubstrateKernel. A caller asks `PortableKernel` -for a kernel; the dispatcher hands back one of four interchangeable -implementations of the `SubstrateKernel` protocol, all conformance- -gated against the `ScalarKernel` oracle. `HammingNN`, `BitField`, -`SHA256`, `HKDF`, and `FloatVecOps` are independent primitives that do -not go through the dispatcher.* +for a kernel. The dispatcher hands back one of four interchangeable +implementations of the `SubstrateKernel` protocol. All four are +conformance-gated against the `ScalarKernel` oracle. Five primitives +skip the dispatcher entirely: `HammingNN`, `BitField`, `SHA256`, +`HKDF`, and `FloatVecOps`.* `PortableKernel.kernelForCurrentPlatform()` picks `SimdKernel` on -64-bit ARM and falls back to `ScalarKernel` everywhere else; it never -selects `MetalKernel`; a caller wanting the GPU path must ask for it -by name with `PortableKernel.kernel(of: .metal)`. `NeonKernel` is -likewise available only by explicit request, since it exists to be -measured against `SimdKernel`, not to replace it automatically. -Whichever implementation runs, its output for a given input is -required to match `ScalarKernel`'s output for that same input; a -conformance test suite enforces this for every backend, on both the -Swift and Rust legs. +64-bit ARM. It falls back to `ScalarKernel` everywhere else. It never +selects `MetalKernel`. A caller who wants the GPU path must ask for it +by name, with `PortableKernel.kernel(of: .metal)`. `NeonKernel` is also +available only by explicit request. It exists to be measured against +`SimdKernel`, not to replace it automatically. + +Whichever implementation runs, its output for a given input must match +`ScalarKernel`'s output for that same input. A conformance test suite +enforces this rule for every backend. It runs on both the Swift leg and +the Rust leg. ## What Ships in the Package -The package ships nine Swift source files and no pinned data -artifacts — unlike some sibling libraries, SubstrateKernel's behavior +The package ships nine Swift source files. It ships no pinned data +artifacts. Unlike some sibling libraries, `SubstrateKernel`'s behavior depends only on its algorithms, not on any versioned reference data. -It also ships a Rust port in `rust/`, mirroring every file except the -two that have no meaningful Rust equivalent: `NeonKernel`, because -Rust's portable-SIMD path already covers the same ground on the -relevant targets, and `MetalKernel`, because Metal is an Apple-only -framework with no Linux or Windows counterpart. The Rust leg adds one -backend the Swift leg does not have — an AVX-512 implementation for -x86-64 processors — but keeps it dark: built, tested, and reachable -only by explicit request, never chosen automatically, until a future -performance study proves it belongs in the default path. + +The package also ships a Rust port, in `rust/`. This port mirrors every +file except two. `NeonKernel` has no Rust equivalent, because Rust's +portable-SIMD path already covers the same ground on the relevant +targets. `MetalKernel` has no Rust equivalent either, because Metal is +an Apple-only framework with no Linux or Windows counterpart. + +The Rust leg adds one backend the Swift leg does not have: an AVX-512 +implementation for x86-64 processors. This backend stays dark. It is +built and tested, and reachable only by explicit request. It is never +chosen automatically, until a future performance study proves it +belongs in the default path. diff --git a/packages/libs/SubstrateKernel/docs/topology.svg b/packages/libs/SubstrateKernel/docs/topology.svg index df905ef..0de2f70 100644 --- a/packages/libs/SubstrateKernel/docs/topology.svg +++ b/packages/libs/SubstrateKernel/docs/topology.svg @@ -58,7 +58,7 @@ sources: - SubstrateKernel protocol — every backend bit-identical to Scalar + SubstrateKernel protocol: every backend bit-identical to Scalar ScalarKernel @@ -92,7 +92,7 @@ sources: - Independent primitives — no dispatcher, called directly + Independent primitives: no dispatcher, called directly HammingNN.topK diff --git a/packages/libs/SubstrateLib/docs/AGENT_MAP.md b/packages/libs/SubstrateLib/docs/AGENT_MAP.md index b9cef3c..d5ea101 100644 --- a/packages/libs/SubstrateLib/docs/AGENT_MAP.md +++ b/packages/libs/SubstrateLib/docs/AGENT_MAP.md @@ -21,122 +21,122 @@ sources: blob: b3f6b0db6b099d6932f7e57c017637193e2a4be6 --- -# AGENT_MAP — SubstrateLib +# AGENT_MAP : SubstrateLib PURPOSE: orchestration layer of the substrate stack. Single write gate (AuditGate.admit) + row-state automaton (legal transitions, I-22/S-1/S-2/S-4 forbidden combinations) + reference impl of the nine substrate verbs (Substrate struct) + Merkle content-hash pipeline + keyed-commitment pipeline for expunge provenance. Value types (Row, RowState, NounType, LatticeAnchor, AuditEvent, HLC, MatrixF/O/T, Fingerprint256) live in SubstrateTypes. DEPS: imports SubstrateTypes (value types), SubstrateKernel (BitField, SHA256, GrantHKDF.hmac), SubstrateML (transitively via Package.swift, not used directly in these 7 files), IntellectusLib (Intellectus.report telemetry sink). Imported by: LocusKit (AuditGate.admit is the sole bitmap write path), other kits per DECISION_KIT_GRAPH_REFACTOR_2026-05-19. Sibling packages in the four-way split: SubstrateTypes (data), SubstrateKernel (hot-path kernels), SubstrateML (cold-path/ML). Rust port in rust/src/ mirrors all 7 files 1:1; conformance fixtures in rust/tests/ + Tests/SubstrateLibConformanceTests/ gate byte-identity. ENTRY POINTS (most callers need only these): -- AuditGate.swift:317 `AuditGate.admit(estateUuid:rowId:nounType:verb:prior:priorLatticeAnchor:writes:afterLatticeAnchor:vocabulary:hlc:actor:) -> Result` — THE single production write gate for row bitmaps -- AuditGate.swift:222 `VocabularyValidator.freeze(union:) -> Result` — one-time vocabulary construction, required before any Vocabulary exists -- RowStateAutomaton.swift:213 `RowStateAutomaton.validate(from:on:targetingFields:) throws -> RowState` — the constitutional mutation gate (legality + I-22/S-1/S-2/S-4) -- Verbs.swift:76 `struct Substrate` — in-memory reference impl of the nine verbs; scalar oracle for production storage engines +- AuditGate.swift:317 `AuditGate.admit(estateUuid:rowId:nounType:verb:prior:priorLatticeAnchor:writes:afterLatticeAnchor:vocabulary:hlc:actor:) -> Result` : THE single production write gate for row bitmaps +- AuditGate.swift:222 `VocabularyValidator.freeze(union:) -> Result` : one-time vocabulary construction, required before any Vocabulary exists +- RowStateAutomaton.swift:213 `RowStateAutomaton.validate(from:on:targetingFields:) throws -> RowState` : the constitutional mutation gate (legality + I-22/S-1/S-2/S-4) +- Verbs.swift:76 `struct Substrate` : in-memory reference impl of the nine verbs; scalar oracle for production storage engines ## Symbol Table -### Row-state automaton — RowStateAutomaton.swift -- :50 `enum RowStateAutomaton` — namespace; RowState/RowVerb/RowStateError types live in SubstrateTypes, imported here -- :72 `canTransition(from:to:viaVerb:) -> Bool` — §10 STRING-verb vocabulary (legacy); looks up :84 `verbTable` (fileprivate `[VerbKey: RowState]`); used by Verbs.swift mutate/withdraw -- :78 `fileprivate struct VerbKey` — (RowState, String) composite key for verbTable -- :127 `static let transitions: [TransitionKey: RowState]` — CANONICAL enum-keyed table (§9 lifecycle vocabulary); absence from map = illegal -- :204 `transition(from:on:) -> RowState?` — single lookup against `transitions` -- :213 `validate(from:on:targetingFields:) throws -> RowState` — transition() + ForbiddenCombinations.check(); THE gate; bypassing is forbidden (v0.36 C1 resolution) -- :227 `struct TransitionKey` — (from: RowState, verb: RowVerb) -- :240 `struct BitmapFields` — adjective/operational/provenance UInt64 triple +### Row-state automaton : RowStateAutomaton.swift +- :50 `enum RowStateAutomaton` : namespace; RowState/RowVerb/RowStateError types live in SubstrateTypes, imported here +- :72 `canTransition(from:to:viaVerb:) -> Bool` : §10 STRING-verb vocabulary (legacy); looks up :84 `verbTable` (fileprivate `[VerbKey: RowState]`); used by Verbs.swift mutate/withdraw +- :78 `fileprivate struct VerbKey` : (RowState, String) composite key for verbTable +- :127 `static let transitions: [TransitionKey: RowState]` : CANONICAL enum-keyed table (§9 lifecycle vocabulary); absence from map = illegal +- :204 `transition(from:on:) -> RowState?` : single lookup against `transitions` +- :213 `validate(from:on:targetingFields:) throws -> RowState` : transition() + ForbiddenCombinations.check(); THE gate; bypassing is forbidden (v0.36 C1 resolution) +- :227 `struct TransitionKey` : (from: RowState, verb: RowVerb) +- :240 `struct BitmapFields` : adjective/operational/provenance UInt64 triple - :260 `enum ForbiddenCombinations` -- :270 `check(state:fields:) throws` — I-22 (secret+exportable forbidden), S-1 (accepted⇒trust≥canonical), S-2 (defensive state-encoding check), S-4 (accepted⇒sensitivity≤elevated); S-5 (tombstone completion flag) NOT YET IMPLEMENTED (queued, see comment at file end) +- :270 `check(state:fields:) throws` : I-22 (secret+exportable forbidden), S-1 (accepted⇒trust≥canonical), S-2 (defensive state-encoding check), S-4 (accepted⇒sensitivity≤elevated); S-5 (tombstone completion flag) NOT YET IMPLEMENTED (queued, see comment at file end) -### Write gate — AuditGate.swift -- :51 `struct FieldSlot: Hashable, Sendable` — one declared field-slot: column/shift/width/label/legalValues -- :52 `enum Column` — .adjective | .operational | .provenance -- :72 `capacity: Int64` — 1< Bool` — width + enumerated-legal-value check -- :106 `enum AuditState` / :113 `AuditSensitivity` / :118 `AuditExportability` / :123 `AuditTrust` — SubstrateLib-local mirrors of LocusKit's Adjectives.swift enums (raw values MUST match; can't import upward); parity enforced by GuardianPairParityTests +### Write gate : AuditGate.swift +- :51 `struct FieldSlot: Hashable, Sendable` : one declared field-slot: column/shift/width/label/legalValues +- :52 `enum Column` : .adjective | .operational | .provenance +- :72 `capacity: Int64` : 1< Bool` : width + enumerated-legal-value check +- :106 `enum AuditState` / :113 `AuditSensitivity` / :118 `AuditExportability` / :123 `AuditTrust` : SubstrateLib-local mirrors of LocusKit's Adjectives.swift enums (raw values MUST match; can't import upward); parity enforced by GuardianPairParityTests - :134 `struct Vocabulary: Sendable` -- :145 `static let basis: Set` — universal fields: state(0,w6) sensitivity(6,w6) exportability(12,w6) trust(18,w6) flags(24,w3, bits 24/25/26 = state_extension/lineage_clustering/dreaming_recalc_required); legalValues derived from allCases.map{rawValue} -- :188 `union: Set` — per-instance consumer fields, frozen -- :192 `fileprivate init(validatedUnion:)` — ONLY VocabularyValidator.freeze can construct -- :196 `slot(for:) -> FieldSlot?` — basis-first lookup, then union -- :206 `enum VocabularyError` — .overlap / .basisCollision / .malformedWidth / .valueExceedsWidth +- :145 `static let basis: Set` : universal fields: state(0,w6) sensitivity(6,w6) exportability(12,w6) trust(18,w6) flags(24,w3, bits 24/25/26 = state_extension/lineage_clustering/dreaming_recalc_required); legalValues derived from allCases.map{rawValue} +- :188 `union: Set` : per-instance consumer fields, frozen +- :192 `fileprivate init(validatedUnion:)` : ONLY VocabularyValidator.freeze can construct +- :196 `slot(for:) -> FieldSlot?` : basis-first lookup, then union +- :206 `enum VocabularyError` : .overlap / .basisCollision / .malformedWidth / .valueExceedsWidth - :222 `enum VocabularyValidator` -- :224 `freeze(union:) -> Result` — 3-step validation (width-sane → no basis collision → no union-union overlap); run ONCE before any data exists -- :258 `struct FieldWrite: Sendable` — (slot, value) one consumer-requested change -- :269 `enum GateViolation: Error, Sendable` — .undeclaredField / .illegalValue / .basisViolation(Error) / .stateInconsistentWithVerb -- :292 `description: String` — English-only at ARIA boundary; NO Swift case-name leakage; carries the "illegal state transition: ..." sentinel the describe_gate_rejection parser matches +- :224 `freeze(union:) -> Result` : 3-step validation (width-sane → no basis collision → no union-union overlap); run ONCE before any data exists +- :258 `struct FieldWrite: Sendable` : (slot, value) one consumer-requested change +- :269 `enum GateViolation: Error, Sendable` : .undeclaredField / .illegalValue / .basisViolation(Error) / .stateInconsistentWithVerb +- :292 `description: String` : English-only at ARIA boundary; NO Swift case-name leakage; carries the "illegal state transition: ..." sentinel the describe_gate_rejection parser matches - :311 `enum AuditGate` -- :317 `admit(...) -> Result` — see ENTRY POINTS; 4 steps: vocab+value gate → read-modify-write (preserves unaddressed bits) → basis gate (state must equal verb's authorized output) → contentID + AuditEvent -- :421 `static func contentID(estateUuid:rowId:hlc:verb:after:afterAnchor:) -> UUID` — SHA-256(stable byte encoding) folded to UUID; NAME-keyed (not ordinal), vocabulary-set-independent; also called directly by Verbs.swift's appendAudit so both paths produce identical event IDs +- :317 `admit(...) -> Result` : see ENTRY POINTS; 4 steps: vocab+value gate → read-modify-write (preserves unaddressed bits) → basis gate (state must equal verb's authorized output) → contentID + AuditEvent +- :421 `static func contentID(estateUuid:rowId:hlc:verb:after:afterAnchor:) -> UUID` : SHA-256(stable byte encoding) folded to UUID; NAME-keyed (not ordinal), vocabulary-set-independent; also called directly by Verbs.swift's appendAudit so both paths produce identical event IDs -### Verb orchestration — Verbs.swift -- :48 `typealias RowId = UUID` — wire-compatible with Rust's RowId(u128) -- :55 `enum SubstrateError: Error, Equatable` — invalidStateTransition / missingLatticeAnchor / invalidNounType / rowNotFound / forbiddenStateCombination / alreadyTombstoned / proposalRequired / nonProposalCannotUseProposalVerb -- :76 `struct Substrate` — see ENTRY POINTS; fields :77-84 (estateUuid, rows, auditEvents(G-Set), matrixF/O/T, hlc, rowCountActive) +### Verb orchestration : Verbs.swift +- :48 `typealias RowId = UUID` : wire-compatible with Rust's RowId(u128) +- :55 `enum SubstrateError: Error, Equatable` : invalidStateTransition / missingLatticeAnchor / invalidNounType / rowNotFound / forbiddenStateCombination / alreadyTombstoned / proposalRequired / nonProposalCannotUseProposalVerb +- :76 `struct Substrate` : see ENTRY POINTS; fields :77-84 (estateUuid, rows, auditEvents(G-Set), matrixF/O/T, hlc, rowCountActive) - :86 `init(estateUuid:hlc:)` -- :110 `capture(nounType:adjectiveBitmap:operationalBitmap:provenanceBitmap:latticeAnchor:fingerprint:lineageId:content:actor:ts:) -> Result` — ONLY verb that originates a row w/o prior state; proposal⇒pending else active; rejects null latticeAnchor -- :180 `reanchor(rowId:newLatticeAnchor:actor:ts:) -> Result<(), SubstrateError>` — lattice-link change only, bitmaps untouched -- :213 `enum MutationKind: String` — confirm/reject/contest/supersede/automated_confirm/decay/expire/lineage_advance/actuator_confirm -- :222 `mutate(rowId:mutationKind:newAdjectiveBitmap:newOperationalBitmap:newProvenanceBitmap:actor:ts:) -> Result<(), SubstrateError>` — canTransition() (§10 string verbs) + isLegalRowState() gate; general state+field change -- :304 `withdraw(rowId:actor:ts:) -> Result<(), SubstrateError>` — → .withdrawn; setStateField raw=18 -- :365 `expunge(rowId:reason:actor:ts:) -> Result<(), SubstrateError>` — → .tombstoned, content=nil; REFUSES .accepted rows (S-3, audit-grade survive intact) before any other check -- :425 `recall(matching:asOf:ts:) -> [Row]` — READ-ONLY; no mutation, no audit event; asOf reconstructs via audit-log HLC filter -- :457 `propose(...) -> Result` — capture(nounType: .proposal, ...) -- :484 `associate(rowA:rowB:signalSourcesBitset:weight:...) -> Result` — capture(nounType: .association, ...); `weight` param ACCEPTED THEN DISCARDED (vestigial, reserved for pre-2.0 gauntlet experiment) — do not silently wire it up or silently drop the parameter -- :531 `learn(...) -> Result` — capture(nounType: .learnedReference, ...) -- :555 `private appendAudit(verb:rowId:before:after:beforeAnchor:afterAnchor:actor:)` — advances hlc; calls AuditGate.contentID (same fn as the gate path) so both paths produce identical event IDs -- :596 `private isLegalRowState(state:adjective:operational:) -> SubstrateError?` — wraps ForbiddenCombinations.check; provenance passed as 0 (unused by check) +- :110 `capture(nounType:adjectiveBitmap:operationalBitmap:provenanceBitmap:latticeAnchor:fingerprint:lineageId:content:actor:ts:) -> Result` : ONLY verb that originates a row w/o prior state; proposal⇒pending else active; rejects null latticeAnchor +- :180 `reanchor(rowId:newLatticeAnchor:actor:ts:) -> Result<(), SubstrateError>` : lattice-link change only, bitmaps untouched +- :213 `enum MutationKind: String` : confirm/reject/contest/supersede/automated_confirm/decay/expire/lineage_advance/actuator_confirm +- :222 `mutate(rowId:mutationKind:newAdjectiveBitmap:newOperationalBitmap:newProvenanceBitmap:actor:ts:) -> Result<(), SubstrateError>` : canTransition() (§10 string verbs) + isLegalRowState() gate; general state+field change +- :304 `withdraw(rowId:actor:ts:) -> Result<(), SubstrateError>` : → .withdrawn; setStateField raw=18 +- :365 `expunge(rowId:reason:actor:ts:) -> Result<(), SubstrateError>` : → .tombstoned, content=nil; REFUSES .accepted rows (S-3, audit-grade survive intact) before any other check +- :425 `recall(matching:asOf:ts:) -> [Row]` : READ-ONLY; no mutation, no audit event; asOf reconstructs via audit-log HLC filter +- :457 `propose(...) -> Result` : capture(nounType: .proposal, ...) +- :484 `associate(rowA:rowB:signalSourcesBitset:weight:...) -> Result` : capture(nounType: .association, ...); `weight` param ACCEPTED THEN DISCARDED (vestigial, reserved for pre-2.0 gauntlet experiment) : do not silently wire it up or silently drop the parameter +- :531 `learn(...) -> Result` : capture(nounType: .learnedReference, ...) +- :555 `private appendAudit(verb:rowId:before:after:beforeAnchor:afterAnchor:actor:)` : advances hlc; calls AuditGate.contentID (same fn as the gate path) so both paths produce identical event IDs +- :596 `private isLegalRowState(state:adjective:operational:) -> SubstrateError?` : wraps ForbiddenCombinations.check; provenance passed as 0 (unused by check) - :615/:624/:632/:637 private bit-twiddling wrappers around SubstrateTypes.RowBitmaps (rowHasBit, extractFieldValues, extractState, setStateField) -### Merkle hash pipeline — MerkleHash.swift -- :31 `struct MerkleVectorInput: Sendable` — modelID, vectorIndex, floats (Float32 LE) +### Merkle hash pipeline : MerkleHash.swift +- :31 `struct MerkleVectorInput: Sendable` : modelID, vectorIndex, floats (Float32 LE) - :51 `enum MerkleHash` -- :76 `leaf(drawerId:content:vectors:) -> ContentHash` — v2 format: vector identity (modelID+vectorIndex) written BEFORE float payload (v1 gap = WS2-F4 vector-substitution finding, fixed 2026-06-28) -- :101 `interior(childHashes: [(UUID, ContentHash)]) -> MerkleRoot` — sorted by UUID BE bytes; empty ⇒ MerkleRoot.empty -- :130 `interior(childRoots: [(UUID, MerkleRoot)]) -> MerkleRoot` — same as above, wing/estate-level roll-up (children already MerkleRoots) -- :157 `tombstone(drawerId:) -> ContentHash` — domain tag only + drawer id; no content/vectors (already destroyed) -- :178 `static func canonicalLeafBytes(drawerId:content:vectors:domainTag:) -> [UInt8]` — SHARED encoding with KeyedCommitment.commit; domainTag is the only difference between leaf hash and commitment preimage +- :76 `leaf(drawerId:content:vectors:) -> ContentHash` : v2 format: vector identity (modelID+vectorIndex) written BEFORE float payload (v1 gap = WS2-F4 vector-substitution finding, fixed 2026-06-28) +- :101 `interior(childHashes: [(UUID, ContentHash)]) -> MerkleRoot` : sorted by UUID BE bytes; empty ⇒ MerkleRoot.empty +- :130 `interior(childRoots: [(UUID, MerkleRoot)]) -> MerkleRoot` : same as above, wing/estate-level roll-up (children already MerkleRoots) +- :157 `tombstone(drawerId:) -> ContentHash` : domain tag only + drawer id; no content/vectors (already destroyed) +- :178 `static func canonicalLeafBytes(drawerId:content:vectors:domainTag:) -> [UInt8]` : SHARED encoding with KeyedCommitment.commit; domainTag is the only difference between leaf hash and commitment preimage - :226/:230/:236 private byte-encoding helpers (uuidBytes, appendU64BE, appendU32BE) -### Keyed commitment — KeyedCommitment.swift -- :26 `struct KeyedCommitmentValue: Hashable, Sendable, Codable` — hmacBytes (32B, precondition-enforced) + keyVersion +### Keyed commitment : KeyedCommitment.swift +- :26 `struct KeyedCommitmentValue: Hashable, Sendable, Codable` : hmacBytes (32B, precondition-enforced) + keyVersion - :40 `hexString: String` - :49 `enum KeyedCommitment` -- :66 `commit(key:keyVersion:drawerId:content:vectors:) -> KeyedCommitmentValue` — MerkleHash.canonicalLeafBytes(domainTag: .commitment=0x03) → GrantHKDF.hmac (SubstrateKernel, no new HMAC impl) +- :66 `commit(key:keyVersion:drawerId:content:vectors:) -> KeyedCommitmentValue` : MerkleHash.canonicalLeafBytes(domainTag: .commitment=0x03) → GrantHKDF.hmac (SubstrateKernel, no new HMAC impl) -### Commitment audit log — KeyedCommitmentAudit.swift -- :27 `struct KeyedCommitmentAuditEntry: Hashable, Sendable` — id (content-hash), drawerId, commitment, tombstoneHLC, reason -- :40 `init(drawerId:commitment:tombstoneHLC:reason:)` — computes :62 `private static computeID(...) -> [UInt8]` (SHA-256 over identifying fields) at construction -- :92 `struct CommitmentAuditLog: Sendable` — G-Set keyed by content hash -- :103 `add(_:)` — idempotent insert -- :108 `merge(_:)` — CRDT join = set union +### Commitment audit log : KeyedCommitmentAudit.swift +- :27 `struct KeyedCommitmentAuditEntry: Hashable, Sendable` : id (content-hash), drawerId, commitment, tombstoneHLC, reason +- :40 `init(drawerId:commitment:tombstoneHLC:reason:)` : computes :62 `private static computeID(...) -> [UInt8]` (SHA-256 over identifying fields) at construction +- :92 `struct CommitmentAuditLog: Sendable` : G-Set keyed by content hash +- :103 `add(_:)` : idempotent insert +- :108 `merge(_:)` : CRDT join = set union - :114 `count: Int` -- :117 `orderedEntries: [KeyedCommitmentAuditEntry]` — sorted by tombstoneHLC +- :117 `orderedEntries: [KeyedCommitmentAuditEntry]` : sorted by tombstoneHLC - :122 `entries(forDrawer:) -> [KeyedCommitmentAuditEntry]` -### Telemetry — SubstrateLibTelemetry.swift -- :78 `enum SubstrateLibMetric` — string-constant namespace, `substratelib.*` metrics +### Telemetry : SubstrateLibTelemetry.swift +- :78 `enum SubstrateLibMetric` : string-constant namespace, `substratelib.*` metrics - :83/:86 auditGateAdmitCount / auditGateRejectCount - :91/:94/:97/:100/:103/:106 verbCaptureCount / verbMutateCount / verbWithdrawCount / verbExpungeCount / verbRecallCount / verbReanchorCount - :111/:114 writeGateAdmittedCount / writeGateRejectedCount - :132 `emitAuditGateAdmit(nounTypeRaw:ts:)`, :152 `emitAuditGateReject(violationName:ts:)` - :170 `emitWriteGateAdmitted(verb:ts:)`, :187 `emitWriteGateRejected(verb:reason:ts:)` -- :204/:219/:233/:247/:266/:280 `emitVerb{Capture,Mutate,Withdraw,Expunge,Recall,Reanchor}Count(...)` — one per Verbs.swift verb +- :204/:219/:233/:247/:266/:280 `emitVerb{Capture,Mutate,Withdraw,Expunge,Recall,Reanchor}Count(...)` : one per Verbs.swift verb - ALL emit fns `@inline(__always)`; wrap `Intellectus.report(...)`; disabled-path cost = one atomic load + branch, zero allocation ## INVARIANTS / GOTCHAS -- DETERMINISM IS THE CONTRACT (same as LatticeLib). No file in this package may read a clock. Every `ts` parameter is caller-supplied epoch seconds; monitoring on/off must never change functional output. Rust mirrors every algorithm; conformance fixtures in rust/tests/ + Tests/SubstrateLibConformanceTests/ gate byte-identity — run both suites on any change to audit_gate/merkle_hash/keyed_commitment/row_state/verbs. -- AuditGate and Substrate do NOT call each other. Two independent consumers of the same RowStateAutomaton rules, not a pipeline. Do not wire one to call the other — that would let a fast-path bug hide behind a passing reference-path test. +- DETERMINISM IS THE CONTRACT (same as LatticeLib). No file in this package may read a clock. Every `ts` parameter is caller-supplied epoch seconds; monitoring on/off must never change functional output. Rust mirrors every algorithm; conformance fixtures in rust/tests/ + Tests/SubstrateLibConformanceTests/ gate byte-identity : run both suites on any change to audit_gate/merkle_hash/keyed_commitment/row_state/verbs. +- AuditGate and Substrate do NOT call each other. Two independent consumers of the same RowStateAutomaton rules, not a pipeline. Do not wire one to call the other : that would let a fast-path bug hide behind a passing reference-path test. - AuditGate.admit and Verbs.appendAudit call the SAME contentID function (AuditGate.contentID) so both paths produce byte-identical event IDs for the same logical event. If you change the byte layout in contentID, both paths and the Rust mirror change together. - Content IDs (AuditGate.contentID, KeyedCommitmentAuditEntry.computeID) are NAME/CONTENT-keyed, never random UUIDs. This is what lets a G-Set (grow-only set) dedupe on federation merge. Do not reintroduce `UUID()` random defaults on any audit-adjacent identity. - S-3 (cookbook §9.5.3): accepted rows CANNOT transition to tombstoned. Enforced in TWO places independently: RowStateAutomaton.transitions (no TransitionKey(.accepted, .tombstone) entry) AND Verbs.expunge's explicit early-return guard. Do not remove either; they are a deliberate belt-and-suspenders pair. -- S-5 (tombstone ⇒ expunge_completed_flag) is NOT YET IMPLEMENTED in ForbiddenCombinations.check — queued pending F17 cookbook location decision (adjective bit 24 or operational reserved bits). A previous incorrect implementation (zeroing bitmaps on tombstone) was removed 2026-05-27; do not reintroduce that specific check. -- AuditState/AuditSensitivity/AuditExportability/AuditTrust (AuditGate.swift) are LOCAL COPIES of LocusKit/Adjectives.swift enums — raw values must match exactly (SubstrateLib cannot import LocusKit; dependency graph points the other way). GuardianPairParityTests enforces this; the `@guardian-pair` comment annotations mark every duplicated literal (also present in RowStateAutomaton.swift's ForbiddenCombinations.check: 48/32/3/16 = secret/public/canonical/elevated raw values). -- Vocabulary can ONLY be constructed via VocabularyValidator.freeze (fileprivate init). Never bypass with a literal struct construction — that skips overlap/collision/width validation entirely. -- FieldSlot.legalValues empty ⇒ "any value that fits width" (used for state, governed by the automaton instead of an enumerated set... actually state DOES enumerate via AuditState.allCases; empty-legalValues is used only by the flags field). Do not conflate "empty" with "no restriction beyond width" vs "enumerated" — check the specific basis slot. -- MerkleHash leaf format is v2 (vector identity before float payload). v1 had a vector-substitution security gap (WS2-F4, fixed 2026-06-28). KeyedCommitment.commit shares this exact byte encoding via canonicalLeafBytes — a leaf-format change without updating both Rust and Swift, and both the hash and commitment call sites, reopens or diverges the fix. -- MerkleHash.interior sorts children by UUID ascending (lexicographic 16-byte BE) — required for roll-up to be write-order-independent. Do not sort by insertion order or hash value. -- Verbs.associate's `weight` parameter is accepted and explicitly discarded (`_ = weight`) — vestigial, reserved for a future experiment. Do not silently start persisting it without a matching bitmap/column change, and do not remove the parameter from the signature without checking callers. -- Package.swift depends on SubstrateTypes + SubstrateKernel + SubstrateML + IntellectusLib directly (not transitively) per DECISION_LIFT_PACKAGE_SWIFT_RULE_2026-05-28 — IntellectusLib sits below SubstrateLib in the topology (depends on nothing in-repo), so this is not a layering violation. -- No pinned data artifacts ship in this package (unlike LatticeLib) — every guarantee here is computational (fixed table, fixed byte encoding, cryptographic hash), not reference-data-driven. +- S-5 (tombstone ⇒ expunge_completed_flag) is NOT YET IMPLEMENTED in ForbiddenCombinations.check : queued pending F17 cookbook location decision (adjective bit 24 or operational reserved bits). A previous incorrect implementation (zeroing bitmaps on tombstone) was removed 2026-05-27; do not reintroduce that specific check. +- AuditState/AuditSensitivity/AuditExportability/AuditTrust (AuditGate.swift) are LOCAL COPIES of LocusKit/Adjectives.swift enums : raw values must match exactly (SubstrateLib cannot import LocusKit; dependency graph points the other way). GuardianPairParityTests enforces this; the `@guardian-pair` comment annotations mark every duplicated literal (also present in RowStateAutomaton.swift's ForbiddenCombinations.check: 48/32/3/16 = secret/public/canonical/elevated raw values). +- Vocabulary can ONLY be constructed via VocabularyValidator.freeze (fileprivate init). Never bypass with a literal struct construction : that skips overlap/collision/width validation entirely. +- FieldSlot.legalValues empty ⇒ "any value that fits width" (used for state, governed by the automaton instead of an enumerated set... actually state DOES enumerate via AuditState.allCases; empty-legalValues is used only by the flags field). Do not conflate "empty" with "no restriction beyond width" vs "enumerated" : check the specific basis slot. +- MerkleHash leaf format is v2 (vector identity before float payload). v1 had a vector-substitution security gap (WS2-F4, fixed 2026-06-28). KeyedCommitment.commit shares this exact byte encoding via canonicalLeafBytes : a leaf-format change without updating both Rust and Swift, and both the hash and commitment call sites, reopens or diverges the fix. +- MerkleHash.interior sorts children by UUID ascending (lexicographic 16-byte BE) : required for roll-up to be write-order-independent. Do not sort by insertion order or hash value. +- Verbs.associate's `weight` parameter is accepted and explicitly discarded (`_ = weight`) : vestigial, reserved for a future experiment. Do not silently start persisting it without a matching bitmap/column change, and do not remove the parameter from the signature without checking callers. +- Package.swift depends on SubstrateTypes + SubstrateKernel + SubstrateML + IntellectusLib directly (not transitively) per DECISION_LIFT_PACKAGE_SWIFT_RULE_2026-05-28 : IntellectusLib sits below SubstrateLib in the topology (depends on nothing in-repo), so this is not a layering violation. +- No pinned data artifacts ship in this package (unlike LatticeLib) : every guarantee here is computational (fixed table, fixed byte encoding, cryptographic hash), not reference-data-driven. diff --git a/packages/libs/SubstrateLib/docs/DETAILS.md b/packages/libs/SubstrateLib/docs/DETAILS.md index 872f5a9..275a2f2 100644 --- a/packages/libs/SubstrateLib/docs/DETAILS.md +++ b/packages/libs/SubstrateLib/docs/DETAILS.md @@ -23,395 +23,139 @@ sources: # SubstrateLib Details -This document walks through every source file in the package. Read -`OVERVIEW.md` first for the big picture. Files appear here in dependency -order: the row-state automaton first, because both write paths consult it; -then the write gate and the reference verbs, the two paths that consult -it; then the content-integrity pipeline, which is independent of both; and -finally telemetry, which every other file calls into. - -A note on vocabulary before the walkthrough: a row is one stored memory. A -bitmap is a 64-bit integer that packs several small status fields at fixed -bit positions, so one integer can carry many independent facts cheaply. A -verb is one of the fixed set of actions a row can undergo — capture, -mutate, withdraw, and so on. An audit event is the permanent record of one -verb having happened to one row. +This document walks through every source file in the package. Read `OVERVIEW.md` first for the big picture. + +Files appear here in dependency order. The row-state automaton comes first, because both write paths consult it. The write gate and the reference verbs come next. These are the two paths that consult the automaton. The content-integrity pipeline comes after that, since it is independent of both. Telemetry comes last, because every other file calls into it. + +A note on vocabulary before the walkthrough. A row is one stored memory. A bitmap is a 64-bit integer. It packs several small status fields at fixed bit positions. One integer can carry many independent facts cheaply this way. A verb is one of a fixed set of actions a row can undergo. Capture, mutate, and withdraw are examples. An audit event is the permanent record of one verb happening to one row. ## RowStateAutomaton.swift -This file provides the row-state automaton: the transition table that -says which lifecycle-state changes are legal, and the forbidden-combination -checks that catch a legal transition landing on an illegal field -combination. - -The data types the automaton operates on — `RowState` (ten states), -`RowVerb` (twelve verbs), and `RowStateError` — are defined in the sibling -package SubstrateTypes and imported here. Only the compute stays in -SubstrateLib: the table, the validation function, and the forbidden- -combination rules. `RowState`'s ten values are deliberately scale-gapped -into three numeric clusters — active (0–3), historical (16–19), and -terminal (32–33) — so a caller can classify a stored raw value's cluster -with one shift-and-mask, without decoding the full enum. - -`RowStateAutomaton.canTransition(from:to:viaVerb:)` answers a yes-or-no -question for one older vocabulary of verb names (contest, supersede, -expunge, and others) inherited from an earlier design phase. It looks the -pair up in a private table, `verbTable`, and compares the result to the -requested target state. It exists because `Verbs.swift`'s `mutate` and -`withdraw` still speak this vocabulary; new call sites should prefer the -canonical function below instead. - -`RowStateAutomaton.transitions` is the canonical table: a dictionary -keyed by `(from state, verb)`, using the `RowVerb` enum rather than raw -strings. Any pair absent from the table is illegal by construction — there -is no default case to fall through. `RowStateAutomaton.transition(from:on:)` -looks up one pair and returns the next state, or `nil` if the pair is -absent. `RowStateAutomaton.validate(from:on:targetingFields:)` is the -single mutation gate the design calls constitutional: it requires a legal -transition, then also runs the target's bitmap fields through -`ForbiddenCombinations.check`, and only returns the next state if both -pass. An earlier design defect let one code path skip this second check -and write field combinations the table alone would not catch; every -mutation path in this package and in `AuditGate` now routes through one -of these two checks before a write is considered final. - -`ForbiddenCombinations.check(state:fields:)` enforces four safety rules -that the transition table cannot express, because they depend on more than -one field at once: - -- A row marked secret can never also be marked exportable, regardless of - its lifecycle state. Sensitivity and exportability are stored as - separate 6-bit fields in the same bitmap; nothing about the storage - format alone prevents both being set to conflicting extremes, so this - rule is checked explicitly on every write. -- An accepted row — one a person or process explicitly confirmed, treated - as audit-grade — must carry a trust level of at least "canonical." - Accepted status is a promise about reliability, and a low trust value - would make that promise false. -- A withdrawn or rejected row must actually encode the matching state - value in its bitmap, a defensive check against corrupted input rather - than a rule that a caller could realistically violate through the API. -- An accepted row's sensitivity may not exceed "elevated," because a - higher sensitivity would make an audit-grade row too restricted to - serve the purpose accepted status is meant to guarantee. - -A fifth rule the design calls for — that a tombstoned row must carry an -"expunge completed" flag — is deliberately not yet implemented here; the -comment in the source explains that an earlier attempt enforced the wrong -invariant (zeroing bitmap fields that the design says must survive -expunge) and was removed until the correct flag location is settled -elsewhere in the bitmap layout. - -`BitmapFields` and `TransitionKey` are the small carrier types the -functions above operate on: three raw 64-bit integers, and a -`(state, verb)` composite key, respectively. +This file provides the row-state automaton. The automaton is a transition table. It says which lifecycle-state changes are legal. The file also provides the forbidden-combination checks. These checks catch a legal transition that lands on an illegal field combination. + +The data types the automaton operates on live in a sibling package, SubstrateTypes. These types are `RowState` (ten states), `RowVerb` (twelve verbs), and `RowStateError`. Only the compute logic stays in SubstrateLib: the table, the validation function, and the forbidden-combination rules. + +`RowState`'s ten values fall into three numeric clusters. These clusters are active, historical, and terminal. The clusters are deliberately spaced apart in the numbering. This lets a caller classify a stored raw value's cluster with one shift-and-mask step. The caller need not decode the full enum. + +`RowStateAutomaton.canTransition(from:to:viaVerb:)` answers a yes-or-no question. It uses an older vocabulary of verb names. That vocabulary includes contest, supersede, expunge, and others. This vocabulary comes from an earlier design phase. The function looks the pair up in a private table, `verbTable`. It compares the result to the requested target state. The function exists because `Verbs.swift`'s `mutate` and `withdraw` still use this vocabulary. New call sites should prefer the canonical function described next. + +`RowStateAutomaton.transitions` is the canonical table. It is a dictionary keyed by a state-and-verb pair. It uses the `RowVerb` enum rather than raw strings. Any pair absent from the table is illegal by construction. There is no default case to fall through. + +`RowStateAutomaton.transition(from:on:)` looks up one pair. It returns the next state, or `nil` if the pair is absent. + +`RowStateAutomaton.validate(from:on:targetingFields:)` is the single mutation gate. The design calls it constitutional. It requires a legal transition. It also runs the target's bitmap fields through `ForbiddenCombinations.check`. It returns the next state only if both checks pass. + +An earlier design defect let one code path skip the second check. That path could write field combinations the table alone would not catch. Every mutation path in this package now routes through one of these two checks. `AuditGate` does the same before a write counts as final. + +`ForbiddenCombinations.check(state:fields:)` enforces four safety rules. The transition table cannot express these rules, because they depend on more than one field at once. + +- A row marked secret can never also be marked exportable. This holds regardless of the row's lifecycle state. Sensitivity and exportability are stored as separate six-bit fields in the same bitmap. Nothing about the storage format alone stops both from being set to conflicting extremes. So this rule is checked explicitly on every write. +- An accepted row must carry a trust level of at least canonical. An accepted row is one a person or process explicitly confirmed. It is treated as audit-grade. Accepted status is a promise about reliability. A low trust value would make that promise false. +- A withdrawn or rejected row must actually encode the matching state value in its bitmap. This is a defensive check against corrupted input. It is not a rule a caller could realistically violate through the API. +- An accepted row's sensitivity may not exceed elevated. A higher sensitivity would make an audit-grade row too restricted. That would defeat the purpose accepted status is meant to serve. + +A fifth rule the design calls for is not yet implemented here. That rule says a tombstoned row must carry an expunge-completed flag. The source comment explains why. An earlier attempt enforced the wrong invariant. It zeroed bitmap fields that the design says must survive expunge. The attempt was removed until the correct flag location is settled elsewhere in the bitmap layout. + +`BitmapFields` and `TransitionKey` are the small carrier types the functions above operate on. `BitmapFields` holds three raw 64-bit integers. `TransitionKey` is a state-and-verb composite key. ## AuditGate.swift -This file provides the production write gate, `AuditGate.admit`, and the -vocabulary machinery that arms it. This is the interface a storage kit — -LocusKit, in the current design — calls on every bitmap mutation. - -A write gate is only as strong as the vocabulary it enforces, so this file -first defines what a legal field looks like. `FieldSlot` describes one -declared field: which of three columns it lives in (adjective, -operational, or provenance), its bit position and width within that -column, a human label, and its set of legal values. An empty legal-value -set means "any value that fits the width," which basis fields use because -the automaton, not an enumerated list, governs their combinations. -`FieldSlot.admits(value:)` is the per-field check: the value must fit the -width, and if the field enumerates specific legal values, the value must -be one of them. - -`Vocabulary.basis` is the fixed set of fields every SubstrateLib instance -and every federation peer must agree on: state, sensitivity, exportability, -and trust, each a 6-bit field, plus a 3-bit flags field for miscellaneous -markers. Their legal-value sets are derived automatically from four local -enums — `AuditState`, `AuditSensitivity`, `AuditExportability`, -`AuditTrust` — so adding a case to one of those enums extends the gate's -vocabulary without a second, easy-to-forget update elsewhere. These four -enums duplicate raw values defined independently in a higher-level package -(LocusKit), because SubstrateLib sits below that package in the -dependency graph and cannot import its types; a separate test suite -(`GuardianPairParityTests`) checks that the two definitions never drift -apart. - -Beyond the fixed basis, each SubstrateLib instance may register its own -additional fields — the union. `VocabularyValidator.freeze(union:)` is -the one place a union is accepted: it checks that every proposed field's -width and enumerated values fit within 64 bits, that no proposed field's -bits collide with a basis field's bits, and that no two proposed fields -collide with each other. Only a union that passes all three checks becomes -a `Vocabulary`; the type's initializer is `fileprivate`, so an unvalidated -vocabulary cannot reach the gate by any other path. This validation runs -once, before any row exists, which matters because a corrupt vocabulary -discovered after data has accumulated would be far harder to recover from -than one rejected up front. - -`AuditGate.admit(estateUuid:rowId:nounType:verb:prior:priorLatticeAnchor:writes:afterLatticeAnchor:vocabulary:hlc:actor:)` -is the gate itself, and it runs in four steps. First, every field in the -caller's write list must resolve to a declared slot in the vocabulary, and -its value must be admissible; an undeclared field or an out-of-range value -is rejected immediately, before any bitmap changes. Second, the gate -performs a read-modify-write: it starts from the row's prior bitmaps (or -zero, for a new row), writes only the addressed bits, and leaves every -other bit exactly as it was — a consumer that owns one field can never -accidentally clobber a field it does not know about. Third, the gate -checks the resulting state against what the supplied verb was allowed to -produce: for a brand-new row, the verb must be `capture` and the encoded -state must be a legal starting state; for an existing row, the verb's -transition must match the state the write actually encodes, and the -forbidden-combination rules must hold. Fourth, and only if every prior -step succeeded, the gate computes a deterministic content identifier and -returns one canonical `AuditEvent`. - -`AuditGate.contentID(estateUuid:rowId:hlc:verb:after:afterAnchor:)` -computes that identifier: it serializes the identifying fields into a -fixed byte layout, hashes them with SHA-256, and folds the first sixteen -bytes of the digest into a UUID. Because the identifier is a pure function -of the event's content — not a random value — two replicas that compute -the same logical event independently produce the same identifier. That -sameness is what lets a grow-only set deduplicate the event automatically -when replicas merge, which is the property federation depends on. - -`GateViolation` is the gate's error type, covering an undeclared field, an -illegal value, a basis violation (wrapping the underlying -`RowStateAutomaton` error), and a state that does not match its verb. Its -`description` property is written to produce plain English at the -system's outer boundary — no Swift type or case name ever appears in -user-visible text — including a specific sentinel phrase, -"illegal state transition: ...", that a downstream parser matches on. +This file provides the production write gate, `AuditGate.admit`. It also provides the vocabulary machinery that arms the gate. Storage kits call this interface on every bitmap mutation. LocusKit is the current example. + +A write gate is only as strong as the vocabulary it enforces. So this file first defines what a legal field looks like. `FieldSlot` describes one declared field. It records which of three columns the field lives in: adjective, operational, or provenance. It also records the field's bit position and width within that column. It records a human label too. It records the field's set of legal values as well. An empty legal-value set means any value that fits the width. Basis fields use this feature. The automaton governs their combinations instead of an enumerated list. `FieldSlot.admits(value:)` is the per-field check. The value must fit the width. If the field enumerates specific legal values, the value must be one of them. + +`Vocabulary.basis` is the fixed set of fields every SubstrateLib instance must agree on. Every federation peer must agree on it too. The fields are state, sensitivity, exportability, and trust. Each one is six bits wide. A three-bit flags field covers miscellaneous markers. Their legal-value sets come from four local enums: `AuditState`, `AuditSensitivity`, `AuditExportability`, and `AuditTrust`. Adding a case to one of those enums extends the gate's vocabulary automatically. This avoids a second, easy-to-forget update elsewhere. These four enums duplicate raw values defined independently in a higher-level package, LocusKit. SubstrateLib sits below that package in the dependency graph, so it cannot import LocusKit's types. A separate test suite, `GuardianPairParityTests`, checks that the two definitions never drift apart. + +Beyond the fixed basis, each SubstrateLib instance may register its own additional fields. These form the union. `VocabularyValidator.freeze(union:)` is the one place a union gets accepted. It checks three things. Every proposed field's width and enumerated values must fit within 64 bits. No proposed field's bits may collide with a basis field's bits. No two proposed fields may collide with each other. Only a union that passes all three checks becomes a `Vocabulary`. The type's initializer is `fileprivate`. So an unvalidated vocabulary cannot reach the gate by any other path. This validation runs once, before any row exists. That timing matters, because a corrupt vocabulary found after data has accumulated would be far harder to recover from. + +`AuditGate.admit(...)` is the gate itself. It runs in four steps. + +First, every field in the caller's write list must resolve to a declared slot in the vocabulary. Its value must be admissible. An undeclared field or an out-of-range value is rejected immediately, before any bitmap changes happen. + +Second, the gate performs a read-modify-write. It starts from the row's prior bitmaps, or zero for a new row. It writes only the addressed bits. It leaves every other bit exactly as it was. A consumer that owns one field can never accidentally clobber a field it does not know about. + +Third, the gate checks the resulting state against what the supplied verb was allowed to produce. For a brand-new row, the verb must be `capture`. The encoded state must also be a legal starting state. For an existing row, the verb's transition must match the state the write actually encodes. The forbidden-combination rules must also hold. + +Fourth, and only if every prior step succeeded, the gate computes a deterministic content identifier. It returns one canonical `AuditEvent`. + +`AuditGate.contentID(...)` computes that identifier. It serializes the identifying fields into a fixed byte layout. It hashes them with SHA-256. It folds the first sixteen bytes of the digest into a UUID. The identifier is a pure function of the event's content, not a random value. So two replicas that independently compute the same logical event produce the same identifier. That sameness is what lets a grow-only set deduplicate the event automatically when replicas merge. Federation depends on this property. + +`GateViolation` is the gate's error type. It covers four cases: an undeclared field, an illegal value, a basis violation, and a mismatched state. The basis violation wraps the underlying `RowStateAutomaton` error. Its `description` property produces plain English at the system's outer boundary. No Swift type or case name ever appears in user-visible text. This includes a specific sentinel phrase: "illegal state transition: ...". A downstream parser matches on that phrase. ## Verbs.swift -This file provides the nine substrate verbs and `Substrate`, the in-memory -reference implementation of all of them. This is the scalar oracle: -production storage engines implement the same nine verbs against real -persistence, and their behavior is checked against this reference. - -`RowId` is a typealias for `UUID`, kept as a distinct name so call sites -read in the substrate's own vocabulary while staying a plain `UUID` under -the hood; the Rust port spells the same identifier as a `u128` newtype, -and the two encodings are byte-identical. - -`Substrate` bundles everything one estate's in-memory reference state -needs: a dictionary of rows, an append-only list of audit events (treated -as a grow-only set), three summary matrices (`matrixF`, `matrixO`, -`matrixT`, defined in SubstrateTypes) that track aggregate statistics -across all rows without re-scanning them, a hybrid logical clock (`hlc`) -that orders events across replicas without relying on wall time, and a -running count of non-tombstoned rows. - -Each verb method follows the same shape: check preconditions against the -row-state automaton, apply the bitmap change, update the two matrices with -a symmetric delta (subtract the row's old contribution, add its new one), -append one audit event, and emit one telemetry count. Timestamps arrive as -a caller-supplied `ts` parameter defaulting to `0.0`, never read from a -clock inside the method, which keeps the verb's functional output -independent of when it happens to run. - -`capture(nounType:adjectiveBitmap:operationalBitmap:provenanceBitmap:latticeAnchor:fingerprint:lineageId:content:actor:ts:)` -creates a new row. It matters because it is the only verb that can -originate a row without a prior state to transition from: a proposal noun -type starts `pending`, awaiting confirmation, and every other noun type -starts `active`. A missing lattice anchor is rejected outright, because -every row must be locatable in the classification lattice from the moment -it exists. - -`reanchor(rowId:newLatticeAnchor:actor:ts:)` changes only a row's link -into the classification lattice, leaving its bitmaps untouched. It exists -separately from `mutate` because reclassifying a memory is a different -kind of change from changing its lifecycle state or its status fields. - -`mutate(rowId:mutationKind:newAdjectiveBitmap:newOperationalBitmap:newProvenanceBitmap:actor:ts:)` -is the general-purpose state-and-field change, covering confirm, reject, -contest, supersede, decay, expire, and the actuator/automated confirm -variants. It matters because it is where the automaton's transition check -and the forbidden-combination check both gate a single call: an illegal -transition or an illegal resulting combination is rejected before any row -or matrix is touched. - -`withdraw(rowId:actor:ts:)` retracts a row by moving it to the -`withdrawn` state. It matters as a distinct verb, rather than a case of -`mutate`, because withdrawal is available from more starting states than -most other transitions and is a common enough action to warrant its own -simple call shape. - -`expunge(rowId:reason:actor:ts:)` is the legal-compliance hard delete: it -moves a row to `tombstoned` and sets its content to `nil`, destroying the -verbatim payload. It matters that `expunge` explicitly refuses an -`accepted` row before doing anything else — an audit-grade row must -survive intact, by design, and no combination of inputs can route around -that refusal. - -`recall(matching:asOf:ts:)` is the only read-only verb: it filters rows -by a caller-supplied predicate, optionally reconstructing the row set as -of a past point in the hybrid logical clock by intersecting with the -audit log. It matters that recall never mutates and never appends an audit -event — a memory system that logged every read as if it were a write -would make querying create liability. - -`propose`, `associate`, and `learn` are thin verbs that each delegate to -`capture` with a fixed `nounType` — proposal, association, and learned -reference, respectively. They matter as distinct entry points because each -represents a semantically different reason a row came to exist, even -though the mechanics underneath are identical. `associate` additionally -accepts a computed `weight` parameter that it deliberately discards after -accepting it: the parameter is reserved for a future experiment, and the -source documents that the value is accepted and thrown away today rather -than silently invented if it were missing. - -The four private helpers at the bottom of the file — -`appendAudit`, `isLegalRowState`, `rowHasBit` / -`extractFieldValues` / `extractState` / `setStateField` — are internal -plumbing: `appendAudit` advances the clock and builds one `AuditEvent` -using the same content-identifier function `AuditGate` uses, so an event -produced by either path is identical in shape; `isLegalRowState` is the -verb layer's call into `ForbiddenCombinations.check`; the rest are small -bit-manipulation wrappers around the shared `RowBitmaps` type in -SubstrateTypes. +This file provides the nine substrate verbs. It also provides `Substrate`, the in-memory reference implementation of all nine. This is the scalar oracle. Production storage engines implement the same nine verbs against real persistence. Their behavior gets checked against this reference. + +`RowId` is a type alias for `UUID`. It keeps a distinct name so call sites read in the substrate's own vocabulary. Under the hood it stays a plain `UUID`. The Rust port spells the same identifier as a `u128` newtype. The two encodings are byte-identical. + +`Substrate` bundles everything one estate's in-memory reference state needs. It holds a dictionary of rows. It holds an append-only list of audit events, treated as a grow-only set. It holds three summary matrices, `matrixF`, `matrixO`, and `matrixT`, all defined in SubstrateTypes. These matrices track aggregate statistics across all rows without re-scanning them. `Substrate` also holds a hybrid logical clock, `hlc`. This clock orders events across replicas without relying on wall time. Finally, it holds a running count of non-tombstoned rows. + +Each verb method follows the same shape. It checks preconditions against the row-state automaton. It applies the bitmap change. It updates the two matrices with a symmetric delta. It subtracts the row's old contribution and adds its new one. It appends one audit event. It emits one telemetry count. + +Timestamps arrive as a caller-supplied `ts` parameter, defaulting to `0.0`. The method never reads a clock internally. This keeps the verb's functional output independent of when it happens to run. + +`capture(...)` creates a new row. It matters because it is the only verb that can originate a row without a prior state to transition from. A proposal noun type starts as `pending`, awaiting confirmation. Every other noun type starts as `active`. A missing lattice anchor is rejected outright. Every row must be locatable in the classification lattice from the moment it exists. + +`reanchor(...)` changes only a row's link into the classification lattice. It leaves the row's bitmaps untouched. It exists separately from `mutate`, because reclassifying a memory differs from changing its lifecycle state or its status fields. + +`mutate(...)` is the general-purpose state-and-field change. It covers six mutation kinds: confirm, reject, contest, supersede, decay, and expire. It also covers the actuator and automated confirm variants. It matters because it is where the automaton's transition check and the forbidden-combination check both gate a single call. An illegal transition or an illegal resulting combination gets rejected before any row or matrix is touched. + +`withdraw(...)` retracts a row by moving it to the `withdrawn` state. It matters as a distinct verb, rather than a case of `mutate`, because withdrawal is available from more starting states than most other transitions. It is also common enough to warrant its own simple call shape. + +`expunge(...)` is the legal-compliance hard delete. It moves a row to `tombstoned` and sets its content to `nil`, destroying the verbatim payload. It matters that `expunge` explicitly refuses an `accepted` row before doing anything else. An audit-grade row must survive intact by design. No combination of inputs can route around that refusal. + +`recall(...)` is the only read-only verb. It filters rows by a caller-supplied predicate. It can also reconstruct the row set as of a past point in the hybrid logical clock, by intersecting with the audit log. It matters that recall never mutates and never appends an audit event. A memory system that logged every read as if it were a write would make querying create liability. + +`propose`, `associate`, and `learn` are thin verbs. Each delegates to `capture` with a fixed `nounType`: proposal, association, and learned reference. They matter as distinct entry points, because each represents a different reason a row came to exist, even though the mechanics underneath are identical. `associate` additionally accepts a computed `weight` parameter. It deliberately discards that parameter after accepting it. The parameter is reserved for a future experiment. The source documents that the value is accepted and thrown away today. It is not silently invented if it were missing instead. + +The four private helpers at the bottom of the file are internal plumbing. `appendAudit` advances the clock. It builds one `AuditEvent` using the same content-identifier function `AuditGate` uses. So an event produced by either path is identical in shape. `isLegalRowState` is the verb layer's call into `ForbiddenCombinations.check`. The rest are small bit-manipulation wrappers around the shared `RowBitmaps` type in SubstrateTypes: `rowHasBit`, `extractFieldValues`, `extractState`, and `setStateField`. ## MerkleHash.swift -This file provides the public hash pipeline for the Merkle-style -content-integrity tree: three functions that turn a row's content, a set -of child hashes, or a tombstoned row into one deterministic SHA-256 value. - -A Merkle tree proves that content has not changed by comparing hashes -instead of comparing the content itself, and by letting a parent's hash -summarize all of its children's hashes. `MerkleDomain` tags (defined in -SubstrateTypes) — a leaf tag, an interior tag, and a tombstone tag — are -prepended to each function's input before hashing, so a hash computed one -way can never be mistaken for, or substituted as, a hash computed another -way. - -`MerkleHash.leaf(drawerId:content:vectors:)` hashes one memory's content -and its vector embeddings into a `ContentHash`. It matters because its -byte encoding is versioned: an earlier version wrote each vector's -identifying information (which embedding model produced it, and which -slot in a multi-vector row) only as a sort key, not into the hashed bytes -themselves, which meant an attacker could substitute one model's vector -for another's without changing the hash — a security defect the source -calls out by name (finding WS2-F4). The current version writes vector -identity into the hash input before the float payload, closing that gap. - -`MerkleHash.interior(childHashes:)` and `interior(childRoots:)` roll up a -node's children into one parent hash, sorting children by UUID first so -the result never depends on the order children happened to be written or -iterated in. Both return `MerkleRoot.empty` for a childless node rather -than hashing nothing, which keeps the empty case an explicit, named value -instead of an arbitrary hash of zero bytes. - -`MerkleHash.tombstone(drawerId:)` hashes an expunged row's identity alone, -with no content and no vectors, because expunge has already destroyed -both. It matters that the tombstone hash still exists at all: it lets the -Merkle tree continue to prove that a specific row occupied a specific -position, even after everything about that row except its identity is -gone. - -`canonicalLeafBytes(drawerId:content:vectors:domainTag:)` is the shared -byte-encoding function both this file and `KeyedCommitment.swift` call. -It matters that it is shared rather than duplicated: the content hash and -the keyed commitment are deliberately computed from the exact same -preimage, differing only in the domain tag and the algorithm applied to -it (a plain hash versus a keyed HMAC), which is what lets the two values -be compared meaningfully as two views of the same underlying commitment -to content. +This file provides the public hash pipeline for the Merkle-style content-integrity tree. It has three functions. They turn a row's content, a set of child hashes, or a tombstoned row into one deterministic SHA-256 value. + +A Merkle tree proves that content has not changed. It does this by comparing hashes instead of comparing the content itself. It also lets a parent's hash summarize all of its children's hashes. `MerkleDomain` tags, defined in SubstrateTypes, mark each function's input before hashing. There is a leaf tag, an interior tag, and a tombstone tag. This tagging stops a hash computed one way from ever being mistaken for a hash computed another way. + +`MerkleHash.leaf(...)` hashes one memory's content and its vector embeddings into a `ContentHash`. It matters because its byte encoding is versioned. An earlier version wrote each vector's identifying information only as a sort key. That information was which embedding model produced the vector and which slot in a multi-vector row it occupies. The earlier version did not write that information into the hashed bytes themselves. This meant an attacker could substitute one model's vector for another's without changing the hash. The source calls out this security defect by name: finding WS2-F4. The current version writes vector identity into the hash input before the float payload. This closes that gap. + +`MerkleHash.interior(childHashes:)` and `interior(childRoots:)` roll up a node's children into one parent hash. They sort children by UUID first. This way, the result never depends on the order children happened to be written or iterated in. Both functions return `MerkleRoot.empty` for a childless node, rather than hashing nothing. This keeps the empty case an explicit, named value instead of an arbitrary hash of zero bytes. + +`MerkleHash.tombstone(drawerId:)` hashes an expunged row's identity alone. There is no content and no vectors, because expunge has already destroyed both. It matters that the tombstone hash still exists at all. It lets the Merkle tree keep proving that a specific row occupied a specific position. This holds even after everything about that row except its identity is gone. + +`canonicalLeafBytes(...)` is the shared byte-encoding function. Both this file and `KeyedCommitment.swift` call it. It matters that this function is shared rather than duplicated. The content hash and the keyed commitment are deliberately computed from the exact same preimage. They differ only in the domain tag and the algorithm applied to it: a plain hash versus a keyed HMAC. This is what lets the two values be compared meaningfully, as two views of the same underlying commitment to content. ## KeyedCommitment.swift -This file provides the public keyed-commitment API used at expunge time: -a way to prove a payload existed without keeping anything a reader could -reconstruct the payload from. - -`KeyedCommitmentValue` carries the output: 32 raw HMAC-SHA256 bytes plus -the key version that produced them. Recording the key version matters -because estate keys rotate over time, and a commitment made under an -older key must remain verifiable against that older key even after -rotation; without the version, a rotated key would silently invalidate -every commitment made before it. - -`KeyedCommitment.commit(key:keyVersion:drawerId:content:vectors:)` builds -the commitment. It reuses `MerkleHash.canonicalLeafBytes`, the exact same -byte encoding `MerkleHash.leaf` uses, but with the commitment domain tag -instead of the leaf tag, and then runs that byte sequence through an -existing HMAC-SHA256 implementation from the sibling package -SubstrateKernel rather than a new one written for this purpose. The domain -separation matters for a specific reason: it guarantees a commitment and a -plain content hash of the same underlying payload are always different -values, so one can never be mistaken for, or substituted as, the other. +This file provides the public keyed-commitment API used at expunge time. It offers a way to prove a payload existed, without keeping anything a reader could reconstruct the payload from. + +`KeyedCommitmentValue` carries the output. This is thirty-two raw HMAC-SHA256 bytes, plus the key version that produced them. Recording the key version matters, because estate keys rotate over time. A commitment made under an older key must remain verifiable against that older key even after rotation. Without the version, a rotated key would silently invalidate every commitment made before it. + +`KeyedCommitment.commit(...)` builds the commitment. It reuses `MerkleHash.canonicalLeafBytes`, the exact same byte encoding `MerkleHash.leaf` uses. But it applies the commitment domain tag instead of the leaf tag. It then runs that byte sequence through an existing HMAC-SHA256 implementation. That implementation comes from the sibling package SubstrateKernel, rather than a new one written for this purpose. The domain separation matters for a specific reason. It guarantees a commitment and a plain content hash of the same underlying payload are always different values. So one can never be mistaken for, or substituted as, the other. ## KeyedCommitmentAudit.swift -This file provides the audit-log entry type and append-only container for -keyed commitments made at expunge time — a record distinct from the -ordinary state-transition audit event, because it records that a -commitment was made, not that a row's state changed. - -`KeyedCommitmentAuditEntry` bundles the drawer identity, the commitment -value, the hybrid logical clock time the tombstone was applied, and a -human-readable reason. Its identifier, computed by -`computeID(drawerId:commitment:tombstoneHLC:reason:)`, is a SHA-256 hash -over the same identifying fields, which matters for the same reason -`AuditGate.contentID` matters elsewhere in this package: two replicas -that independently record the same logical commitment compute the same -identifier, so a grow-only set deduplicates them without extra -coordination. - -`CommitmentAuditLog` is that grow-only set: entries are keyed by their -content hash in a dictionary, `add(_:)` inserts one entry idempotently, -and `merge(_:)` unions two logs together — the CRDT join that lets two -replicas' commitment logs converge regardless of the order in which -entries arrive. `orderedEntries` and `entries(forDrawer:)` give callers a -stable, HLC-ordered view for display or audit review. +This file provides the audit-log entry type and an append-only container for keyed commitments made at expunge time. This record is distinct from the ordinary state-transition audit event. It records that a commitment was made. It does not record that a row's state changed. + +`KeyedCommitmentAuditEntry` bundles four things: the drawer identity, the commitment value, the tombstone time, and a human-readable reason. The tombstone time comes from the hybrid logical clock. Its identifier comes from `computeID(...)`. That function computes a SHA-256 hash over the same identifying fields. This matters for the same reason `AuditGate.contentID` matters elsewhere in this package. Two replicas that independently record the same logical commitment compute the same identifier. So a grow-only set deduplicates them without extra coordination. + +`CommitmentAuditLog` is that grow-only set. Entries are keyed by their content hash in a dictionary. `add(_:)` inserts one entry idempotently. `merge(_:)` unions two logs together. This is the CRDT join that lets two replicas' commitment logs converge, regardless of the order in which entries arrive. `orderedEntries` and `entries(forDrawer:)` give callers a stable, HLC-ordered view for display or audit review. ## SubstrateLibTelemetry.swift -This file provides the metric name catalogue and the emit functions every -other file in this package calls into when reporting activity. - -SubstrateLib's own comment calls it the "determinism floor" of the -substrate stack: nothing in this package may read a clock or change its -functional output depending on whether monitoring is enabled. This file -is where that promise is implemented, not merely stated. Every emit -function wraps its call in `Intellectus.report(...)` (from the sibling -package IntellectusLib), which — when monitoring is disabled, the -default — costs exactly one lock-free flag check and a branch: no metric -object is built, no lock is taken, nothing is allocated. Every timestamp -the emit functions accept is a parameter supplied by the caller, never -read from a clock inside this file, so the substrate's behavior is -byte-identical whether monitoring is on or off. - -`SubstrateLibMetric` is a namespace of string constants — one per metric -name — so the complete list of everything this package can report is -visible in one place, and a typo in a metric name is caught at compile -time rather than showing up as a silently missing metric in production. - -The eight `emit...` functions each correspond to one point in the gate or -verb flow: `emitAuditGateAdmit` and `emitAuditGateReject` fire from -`AuditGate.admit`'s two outcomes; `emitWriteGateAdmitted` and -`emitWriteGateRejected` fire from the same gate but tagged by verb and -rejection reason, for operators who want to break down gate traffic -differently than by outcome alone; and one function per verb — -`emitVerbCaptureCount`, `emitVerbMutateCount`, `emitVerbWithdrawCount`, -`emitVerbExpungeCount`, `emitVerbRecallCount`, `emitVerbReanchorCount` — -fires from the matching method in `Verbs.swift`. Each is marked -`@inline(__always)` so the disabled-path cost stays a single flag check -with no function-call overhead added on top. +This file provides the metric name catalogue. It also provides the emit functions every other file in this package calls into when reporting activity. + +SubstrateLib's own comment calls this file the determinism floor of the substrate stack. Nothing in this package may read a clock or change its functional output depending on whether monitoring is enabled. This file is where that promise gets implemented, not merely stated. + +Every emit function wraps its call in `Intellectus.report(...)`, from the sibling package IntellectusLib. When monitoring is disabled, which is the default, this costs exactly one lock-free flag check and a branch. No metric object gets built. No lock gets taken. Nothing gets allocated. Every timestamp the emit functions accept is a parameter supplied by the caller. It is never read from a clock inside this file. So the substrate's behavior stays identical whether monitoring is on or off. + +`SubstrateLibMetric` is a namespace of string constants, one per metric name. This makes the complete list of everything this package can report visible in one place. A typo in a metric name gets caught at compile time. It does not show up as a silently missing metric in production. + +The eight `emit...` functions each correspond to one point in the gate or verb flow. `emitAuditGateAdmit` and `emitAuditGateReject` fire from `AuditGate.admit`'s two outcomes. `emitWriteGateAdmitted` and `emitWriteGateRejected` fire from the same gate, tagged by verb and rejection reason. This lets operators break down gate traffic differently than by outcome alone. One function fires per verb: `emitVerbCaptureCount`, `emitVerbMutateCount`, `emitVerbWithdrawCount`, `emitVerbExpungeCount`, `emitVerbRecallCount`, and `emitVerbReanchorCount`. Each is marked `@inline(__always)`. So the disabled-path cost stays a single flag check with no function-call overhead added on top. ## Rust Port and Conformance -The `rust/` directory contains the second leg of the library: seven -source files mirroring the Swift implementations one for one — -`audit_gate.rs`, `keyed_commitment.rs`, `merkle_hash.rs`, `row_state.rs`, -`verbs.rs`, and `substrate_lib_telemetry.rs`, plus `lib.rs` as the crate -root. Rust has no `CaseIterable`, so the adjective-vocabulary values that -the Swift leg derives from enum cases are instead typed constant slices in -`audit_gate.rs`, kept in step with the Swift enums by convention and -checked by the same cross-layer parity tests. The two legs share -conformance fixtures in `rust/tests/` and -`Tests/SubstrateLibConformanceTests/` — recorded input-output pairs for -the audit-log fold, bitmap field constants, the SimHash-derived float -comparisons, and the wire format — that both implementations must -reproduce byte for byte. When you change either leg, run both test suites; -the fixtures are the contract. +The `rust/` directory contains the second leg of the library. Seven source files mirror the Swift implementations one for one: `audit_gate.rs`, `keyed_commitment.rs`, `merkle_hash.rs`, `row_state.rs`, `verbs.rs`, and `substrate_lib_telemetry.rs`. The crate root is `lib.rs`. + +Rust has no `CaseIterable`. So the adjective-vocabulary values that the Swift leg derives from enum cases are instead typed constant slices in `audit_gate.rs`. These stay in step with the Swift enums by convention. The same cross-layer parity tests check this. + +The two legs share conformance fixtures. These live in `rust/tests/` and `Tests/SubstrateLibConformanceTests/`. The fixtures are recorded input-output pairs. They cover the audit-log fold, bitmap field constants, the SimHash-derived float comparisons, and the wire format. Both implementations must reproduce these byte for byte. When you change either leg, run both test suites. The fixtures are the contract. diff --git a/packages/libs/SubstrateLib/docs/OVERVIEW.md b/packages/libs/SubstrateLib/docs/OVERVIEW.md index 28636c4..a8b6e74 100644 --- a/packages/libs/SubstrateLib/docs/OVERVIEW.md +++ b/packages/libs/SubstrateLib/docs/OVERVIEW.md @@ -25,157 +25,94 @@ sources: ## What This Library Does -SubstrateLib is the orchestration layer of MOOTx01's storage substrate — the -part that decides what a legal change to a memory looks like, and refuses -every other kind. MOOTx01 is an on-device AI memory system. It stores what -an AI observes over time and helps the AI recall it later. Each stored -memory is one row: an entry with a lifecycle state, a link into the -classification lattice built by LatticeLib, and three 64-bit integers -called bitmaps that pack many small status fields — sensitivity, trust, -exportability, and more — into a compact wire form. - -SubstrateLib supplies three things a row needs to stay trustworthy. First, -a single write gate, `AuditGate.admit`, that every bitmap mutation must -pass through. Second, a row-state automaton that says which of ten -lifecycle states a row may hold and which state changes are legal. Third, -a reference implementation of the nine verbs — the fixed vocabulary of -actions a row can undergo, such as capture, mutate, and withdraw — against -which production storage kits can check their own faster implementations. - -Two supporting pipelines round out the package. A Merkle hash pipeline -computes deterministic content hashes so the system can prove a memory's -content is unchanged without re-reading the whole memory store. A -keyed-commitment pipeline proves that a piece of content once existed, -after the content itself has been destroyed to satisfy a deletion request. - -The value types SubstrateLib operates on — `Row`, `RowState`, `NounType`, -`LatticeAnchor`, `AuditEvent` — live in the sibling package SubstrateTypes. -SubstrateLib imports them and adds the logic: the rules, the gate, and the -verbs. +SubstrateLib is the orchestration layer of MOOTx01's storage substrate. It decides what counts as a legal change to a memory. It refuses every other kind of change. + +MOOTx01 is an on-device AI memory system. It stores what an AI observes over time. Later, the AI can recall that content. + +Each stored memory is one row. A row is an entry with a lifecycle state. A row also links into a classification lattice. LatticeLib builds that lattice. + +Each row carries three 64-bit integers called bitmaps. A bitmap packs many small status fields into one compact form. These fields include sensitivity, trust, and exportability. + +SubstrateLib supplies three things a row needs to stay trustworthy. + +First is a single write gate, `AuditGate.admit`. Every bitmap change must pass through this gate. + +Second is a row-state automaton. It lists which of ten lifecycle states a row may hold. It also lists which state changes are legal. + +Third is a reference set of the nine verbs. A verb is one fixed action a row can undergo. Capture, mutate, and withdraw are three examples. Production storage kits check their own faster code against this reference. + +Two supporting pipelines round out the package. + +A Merkle hash pipeline computes deterministic content hashes. These hashes let the system prove a memory's content has not changed. The system proves this without reading the whole memory store again. + +A keyed-commitment pipeline proves that a piece of content once existed. It does this after the content has been destroyed. Destruction happens to satisfy a deletion request. + +The value types SubstrateLib works with live in a sibling package. That package is called SubstrateTypes. Its types include `Row`, `RowState`, `NounType`, `LatticeAnchor`, and `AuditEvent`. SubstrateLib imports these types. It adds the logic on top: the rules, the gate, and the verbs. ## The Problem It Solves -MOOTx01 estates can federate, which means separate devices exchange and -merge memories. Merging only works cleanly if two devices that record the -same logical event compute the same event identifier. SubstrateLib gives -every audit event a deterministic content identifier: a SHA-256 hash of -the event's stable fields, folded into a UUID, rather than a random one. -Two replicas that reach the same write independently produce the same -identifier, so a grow-only set — a data structure that can gain entries -but never lose them, often called a G-Set — simply keeps one copy. A -random identifier would defeat this: the same logical event would look -like two different events on two devices, and federation would double-count -it. - -A second problem is structural corruption. A row's status lives in three -64-bit integers, with several unrelated fields packed into each one at -fixed bit positions. Nothing at the type level stops a careless write from -touching the wrong bits, writing a value wider than its field, or moving a -row into a combination the design forbids — for example, marking a secret -memory as exportable. SubstrateLib closes this gap with one gate that every -write must pass. A write either lands as a legal value in a declared field, -producing a legal state and a legal combination of fields, or the gate -refuses it. Corruption of this kind becomes structurally unrepresentable, -not merely discouraged by convention. - -A third problem is privacy against accountability. When a memory is -expunged, MOOTx01 destroys its content for good — a legal-compliance -requirement. But an audit trail that simply drops the event looks the same -whether nothing happened or something was hidden; from the outside, an -absence proves nothing. SubstrateLib's keyed-commitment pipeline resolves -this: it commits, cryptographically, to the fact that a payload existed, -using a secret key the estate holds, without retaining anything the -payload could be reconstructed from. - -The mathematics in this package is conformance-gated. A Rust port in -`rust/` mirrors every algorithm, and shared test fixtures require both -legs to produce byte-identical output, so a device running the Swift leg -and a device running the Rust leg agree on every hash, every content -identifier, and every gate decision. +MOOTx01 estates can federate. Federation means separate devices exchange and merge memories. Merging works cleanly only when two devices agree on one thing. They must compute the same event identifier for the same logical event. + +SubstrateLib solves this with a deterministic content identifier. Each audit event gets a SHA-256 hash of its stable fields. That hash is folded into a UUID, instead of a random one. Two replicas that reach the same write independently produce the same identifier. + +This sameness matters for a grow-only set. A grow-only set, often called a G-Set, can gain entries but never lose them. Because the identifier is deterministic, a G-Set simply keeps one copy. A random identifier would defeat this design. The same logical event would look like two different events on two devices. Federation would then double-count it. + +A second problem is structural corruption. A row's status lives in three 64-bit integers. Each integer packs several unrelated fields at fixed bit positions. Nothing at the type level stops a careless write from causing harm. A careless write could touch the wrong bits. It could write a value wider than its field. It could move a row into a combination the design forbids. One example is marking a secret memory as exportable. + +SubstrateLib closes this gap with one gate that every write must pass. A write must land as a legal value in a declared field. It must produce a legal state and a legal combination of fields. Otherwise, the gate refuses it. Corruption of this kind becomes structurally impossible this way. Convention alone would only discourage it, never prevent it. + +A third problem is privacy against accountability. When a memory is expunged, MOOTx01 destroys its content for good. This satisfies a legal-compliance requirement. An audit trail that simply drops the event creates a puzzle, though. Nothing happening and something being hidden look the same from the outside. An absence proves nothing on its own. + +SubstrateLib's keyed-commitment pipeline resolves this problem. It commits, cryptographically, to the fact that a payload once existed. It uses a secret key the estate holds. It retains nothing the payload could be rebuilt from. + +The mathematics in this package is conformance-gated. A Rust port in `rust/` mirrors every algorithm. Shared test fixtures require both legs to produce identical output. A device running the Swift leg and a device running the Rust leg must agree. They must agree on every hash, every content identifier, and every gate decision. ## How It Works -The write gate and the verb reference sit side by side, both built on the -same row-state automaton, but they serve different callers. - -`AuditGate.admit` is the gate production kits call directly. A caller -supplies the row's identity, the prior snapshot of its bitmaps, and only -the field values it owns. The gate looks up each written field in the -instance's admitted vocabulary — the substrate's universal fields (state, -sensitivity, exportability, trust) plus whatever fields the calling kit -registered for itself — and rejects any field that is undeclared or any -value that is out of range or not in the field's legal set. It then reads -the prior bitmaps, writes only the addressed bits, and leaves everything -else untouched: a read-modify-write, never a blind overwrite. Before -returning, it re-derives the encoded state and checks that state against -what the supplied verb was allowed to produce, and checks the whole result -against the automaton's forbidden-combination rules. Only then does it -compute the deterministic content identifier and emit one canonical audit -event. - -The row-state automaton itself is a fixed transition table: for each -(state, verb) pair, either a next state is defined, or the transition is -illegal and the automaton says so. Layered on top of the table is a second -check, the forbidden-combination rules, because a transition can be legal -in isolation yet still produce a field combination the design never -allows — a secret memory that is also exportable, for instance. Both -checks together are what "legal state" means in this package. - -`Substrate`, the reference verb implementation, is a second, independent -route to the same guarantees. It is an in-memory struct that implements -all nine verbs end to end: it validates a mutation against the automaton -directly, applies the bitmap change, updates two summary structures called -matrices that track how many rows carry each field value, and appends an -audit event, all inside one call. Production storage engines — a SQLite -tail, a memory-mapped bit-slice tensor — reimplement these same nine verbs -against real persistence; `Substrate` is the scalar oracle they are tested -against, not itself the production path. - -The two supporting pipelines are simpler and pure. The Merkle hash -pipeline builds a fixed byte encoding of a memory's content and vector -embeddings, tags it with a domain byte so a leaf hash can never be -mistaken for an interior or tombstone hash, and runs it through SHA-256. -The keyed-commitment pipeline reuses that same byte encoding with a -different domain tag, then runs it through a keyed HMAC instead of a -plain hash, binding the commitment to a secret only the estate holds. - -Every one of these code paths — the gate, the verbs, both hash pipelines — -can report activity to an injected telemetry sink, but none of them ever -reads a clock or changes its output because monitoring is on. Timestamps -arrive as a caller-supplied parameter, and when the sink is disabled, the -entire telemetry call collapses to one lock-free flag check. +The write gate and the verb reference sit side by side. Both build on the same row-state automaton. But they serve different callers. + +`AuditGate.admit` is the gate production kits call directly. A caller supplies the row's identity. A caller also supplies the prior snapshot of its bitmaps. A caller supplies only the field values it owns. + +The gate looks up each written field in the instance's admitted vocabulary. That vocabulary holds the substrate's universal fields: state, sensitivity, exportability, and trust. It also holds whatever fields the calling kit registered for itself. The gate rejects any field that is undeclared. It rejects any value that is out of range or not in the field's legal set. + +The gate then reads the prior bitmaps. It writes only the addressed bits. It leaves everything else untouched. This is a read-modify-write, never a blind overwrite. + +Before returning, the gate re-derives the encoded state. It checks that state against what the supplied verb was allowed to produce. It also checks the whole result against the automaton's forbidden-combination rules. Only then does it compute the deterministic content identifier. Only then does it emit one canonical audit event. + +The row-state automaton itself is a fixed transition table. For each pair of a state and a verb, either a next state is defined, or the transition is illegal. The automaton reports illegal transitions as such. + +A second check sits on top of the table: the forbidden-combination rules. A transition can be legal on its own. Yet it can still produce a field combination the design never allows. One example is a secret memory that is also exportable. Both checks together define what "legal state" means in this package. + +`Substrate` is the reference verb implementation. It offers a second, independent route to the same guarantees. It is an in-memory struct. It implements all nine verbs end to end. + +`Substrate` validates a mutation against the automaton directly. It applies the bitmap change. It updates two summary structures called matrices. These matrices track how many rows carry each field value. It appends an audit event. All of this happens inside one call. + +Production storage engines reimplement these same nine verbs against real storage. Examples include a SQLite tail and a memory-mapped bit-slice tensor. `Substrate` is the scalar oracle these engines are tested against. It is not itself the production path. + +The two supporting pipelines are simpler and pure. + +The Merkle hash pipeline builds a fixed byte encoding of a memory's content. It also encodes the memory's vector embeddings. It tags this encoding with a domain byte. That tag stops a leaf hash from being mistaken for an interior or tombstone hash. The pipeline runs the tagged bytes through SHA-256. + +The keyed-commitment pipeline reuses that same byte encoding. It applies a different domain tag. It runs the bytes through a keyed HMAC instead of a plain hash. This binds the commitment to a secret only the estate holds. + +Every one of these code paths can report activity to an injected telemetry sink. This includes the gate, the verbs, and both hash pipelines. But none of them ever reads a clock. None of them changes its output because monitoring is on. Timestamps arrive as a caller-supplied parameter. When the sink is disabled, the entire telemetry call collapses to one lock-free flag check. ## How the Pieces Fit -Figure 1 shows the library's topology — its major parts and how a write -moves through them. +Figure 1 shows the library's topology. It shows the major parts and how a write moves through them. ![Figure 1. Topology of SubstrateLib](topology.svg) -*Figure 1. Topology of SubstrateLib. Two independent write paths — the -production `AuditGate` and the reference `Substrate` verbs — both consult -the shared row-state automaton. A separate, unrelated pipeline computes -content hashes and keyed commitments for the integrity and privacy -guarantees. Both paths report to the telemetry sink, shown as a dashed -external region because it lives in the sibling package IntellectusLib.* - -`AuditGate` and `Substrate` do not call one another. They are two -consumers of the same rules, not a pipeline where one feeds the other. A -kit such as LocusKit calls `AuditGate.admit` directly on every bitmap -write; `Substrate` exists so that alternative or future storage engines -have a known-correct in-memory reference to test against. Keeping them -independent, rather than layering one on the other, is deliberate: it -guarantees that a bug in the fast path cannot hide behind a passing -reference-path test, because the two paths share only the rules, not the -code. +*Figure 1. Topology of SubstrateLib. Two independent write paths exist: the production `AuditGate` and the reference `Substrate` verbs. Both consult the shared row-state automaton. A separate, unrelated pipeline computes content hashes and keyed commitments. These give the integrity and privacy guarantees. Both paths report to the telemetry sink. The sink is shown as a dashed external region because it lives in the sibling package IntellectusLib.* + +`AuditGate` and `Substrate` do not call one another. They are two consumers of the same rules. Neither feeds the other in a pipeline. A kit such as LocusKit calls `AuditGate.admit` directly on every bitmap write. `Substrate` exists for a different reason. Alternative or future storage engines need a known-correct in-memory reference to test against. + +Keeping the two paths independent, rather than layering one on the other, is deliberate. It guarantees that a bug in the fast path cannot hide behind a passing reference-path test. The two paths share only the rules. They do not share code. ## What Ships in the Package -The package ships the seven Swift sources listed above and the Rust port -in `rust/`, which mirrors each of them file for file. It ships no pinned -data artifacts of its own — no bundled JSON, no trained models — because -SubstrateLib's guarantees are entirely computational: a fixed transition -table, a fixed byte encoding, and a cryptographic hash, all deterministic -by construction rather than by shipped reference data. Conformance -fixtures live in `Tests/SubstrateLibConformanceTests/` and -`rust/tests/`, and both suites must pass before either leg changes ship. +The package ships seven Swift source files, listed above. It also ships a Rust port in `rust/`. The Rust port mirrors each Swift file one for one. + +The package ships no pinned data artifacts of its own. There is no bundled JSON and no trained model. SubstrateLib's guarantees are entirely computational. They rest on a fixed transition table, a fixed byte encoding, and a cryptographic hash. All of these are deterministic by construction. None depend on shipped reference data. + +Conformance fixtures live in `Tests/SubstrateLibConformanceTests/` and in `rust/tests/`. Both suites must pass before either leg changes ship. diff --git a/packages/libs/SubstrateLib/docs/topology.svg b/packages/libs/SubstrateLib/docs/topology.svg index 8ba0278..955ffab 100644 --- a/packages/libs/SubstrateLib/docs/topology.svg +++ b/packages/libs/SubstrateLib/docs/topology.svg @@ -45,11 +45,11 @@ sources: AuditGate - admit() — production write gate + admit() : production write gate Substrate - nine verbs — reference impl + nine verbs : reference impl @@ -85,7 +85,7 @@ sources: - Telemetry sink (external — IntellectusLib) + Telemetry sink (external : IntellectusLib) Intellectus.report diff --git a/packages/libs/SubstrateML/docs/AGENT_MAP.md b/packages/libs/SubstrateML/docs/AGENT_MAP.md index 41753bb..5856d6f 100644 --- a/packages/libs/SubstrateML/docs/AGENT_MAP.md +++ b/packages/libs/SubstrateML/docs/AGENT_MAP.md @@ -83,97 +83,97 @@ sources: blob: 4f90110f2a19cb92f3d45baf3bc4d82ac9c3dc65 --- -# AGENT_MAP — SubstrateML +# AGENT_MAP: SubstrateML -PURPOSE: layer-3 cold-path/dreaming algorithm library of the MOOTx01 substrate — learning (decay, Bradley-Terry, calibration, NMF), graph analytics (Louvain, centrality, walks, SVD, FFT, anomaly), pattern mining (association rules, Apriori, FCA, D-G implications, temporal causality), distillation (5-stage cluster→factoid), fingerprint/distance math (FloatSimHash, composite/lattice/partial distances, moment/temporal summaries), and federation privacy math (pairing, tier contribution, tier query, DP OR-reduce). Pure functions + value types only; no storage, no clocks, no hidden state. +PURPOSE: layer-3 cold-path/dreaming algorithm library of the MOOTx01 substrate: learning (decay, Bradley-Terry, calibration, NMF), graph analytics (Louvain, centrality, walks, SVD, FFT, anomaly), pattern mining (association rules, Apriori, FCA, D-G implications, temporal causality), distillation (5-stage cluster→factoid), fingerprint/distance math (FloatSimHash, composite/lattice/partial distances, moment/temporal summaries), and federation privacy math (pairing, tier contribution, tier query, DP OR-reduce). Pure functions + value types only; no storage, no clocks, no hidden state. DEPS: imports SubstrateTypes (Fingerprint256, HLC, RowId, MatrixO/F/C, SplitMix64, RecallResult…), SubstrateKernel (PortableKernel dispatch: float SimHash projection, orReduce256), IntellectusLib (telemetry; off-path = one atomic-bool load), Foundation. Imported by (per Package.swift/README): LocusKit, CognitionKit, GeniusLocusKit, NeuronKit, dreaming-daemon paths; LatticeLib (moot-semantics) uses EigenvalueCentrality for LexRank; CorpusKit LsaProvider uses JacobiSVD. Within this repo only tests + SubstrateLib's temporary `@_exported` re-export reference it (four-package split mid-migration). Rust port `rust/` = crate `substrate-ml` v1.0.0-skeleton, 38 modules 1:1 with Swift files; conformance tests rust/tests/{distillation_conformance,float_simhash_kernel_equivalence,viz_graph_signals_tests}.rs + shared JSON vectors in the engineering test harness. ENTRY POINTS (no single facade; per-family): -- DistillationPipeline.swift:277 `DistillationPipeline.run(input:extractFeatures:intraItem:) -> DistillationOutput` — full 5-stage distillation -- CommunityDetection.swift:152 `CommunityDetection.detectFull(adjacency:maxLevels:maxPasses:resolution:estate:ts:) -> [Int]` — multi-level Louvain -- EigenvalueCentrality.swift:87 `EigenvalueCentrality.compute(adjacency:maxIterations:tolerance:estate:ts:) -> [Double]` — keystone scores -- NMFAlternatingLeastSquares.swift:81 `factorize(V:rank:...) -> NMFFactorization` — latent themes -- FloatSimHash.swift:42 `FloatSimHash.project(vector:seed:) -> Fingerprint256` — embedding → fingerprint -- CompositeDistance.swift:50 `CompositeDistance.distance(latticeDistance:fingerprintHammingDistance:...)` — primary recall metric -- AuditLogFold.swift:101 `AuditLogFold.projectAll(events:asOf:nounTypeFor:)` — CRDT state reconstruction -- AprioriMining.swift:167 `AprioriMining.mine(rows:thresholds:) -> [AprioriRule]` — multi-antecedent rules +- DistillationPipeline.swift:277 `DistillationPipeline.run(input:extractFeatures:intraItem:) -> DistillationOutput`: full 5-stage distillation +- CommunityDetection.swift:152 `CommunityDetection.detectFull(adjacency:maxLevels:maxPasses:resolution:estate:ts:) -> [Int]`: multi-level Louvain +- EigenvalueCentrality.swift:87 `EigenvalueCentrality.compute(adjacency:maxIterations:tolerance:estate:ts:) -> [Double]`: keystone scores +- NMFAlternatingLeastSquares.swift:81 `factorize(V:rank:...) -> NMFFactorization`: latent themes +- FloatSimHash.swift:42 `FloatSimHash.project(vector:seed:) -> Fingerprint256`: embedding → fingerprint +- CompositeDistance.swift:50 `CompositeDistance.distance(latticeDistance:fingerprintHammingDistance:...)`: primary recall metric +- AuditLogFold.swift:101 `AuditLogFold.projectAll(events:asOf:nounTypeFor:)`: CRDT state reconstruction +- AprioriMining.swift:167 `AprioriMining.mine(rows:thresholds:) -> [AprioriRule]`: multi-antecedent rules ## Symbol Table -### Telemetry names — VizGraphSignals.swift -- :65 `enum VizGraphSignals` — canonical metric-name constants; one authoritative file prevents orphaned metrics +### Telemetry names: VizGraphSignals.swift +- :65 `enum VizGraphSignals`: canonical metric-name constants; one authoritative file prevents orphaned metrics - :69 `communityAssignment = "community.assignment"` · :73 `centralityScore = "centrality.score"` · :77 `nmfFactor = "nmf.factor"` · :82 `anomalyFlag = "anomaly.flag"` · :86 `edgeDecayedWeight = "edge.decayed_weight"` ### Fingerprint + distance math -- FloatSimHash.swift:30 `enum FloatSimHash`; :42 `project(vector:seed:) -> Fingerprint256` — 256-hyperplane signed projection; empty vector → zero fp; :76 `planes(seed:dim:) -> FloatSimHashPlanes` — deterministic ±1 plane set, cacheable per (seed,dim); :61 private cached kernel (`PortableKernel.kernelForCurrentPlatform()`, selected once) -- CompositeDistance.swift:37 `defaultAlphaLattice = 0.5` / :38 `defaultAlphaFingerprint = 0.5` / :40 `fingerprintTotalBits = 256`; :50 `distance(latticeDistance:fingerprintHammingDistance:alphaLattice:alphaFingerprint:compatibleSeedScope:) -> Double` — preconditions (not clamps); scope-incompatible drops fp term WITHOUT renormalizing (intended) -- LatticeDistance.swift:45 `struct LatticeAnchorStr` (udc + qid; 0 = null); :59 `enum UDCTreeDistance` (:63 `longestCommonPrefixLength`, :83 `distance` clamped [0,1]); :104 `protocol WikidataAdjacencyProvider`; :110 `enum WikidataGraphDistance` (:112 `maxDepth = 4`, :113 `normalizationScale = 3.0`, :118 `shortestPathLength` bounded BFS, :144 `distance` = 1-exp(-len/3), unreachable/null → 1.0); :161 `enum LatticeDistance` (:163/:164 default alphas 0.5, :185 combined `distance(_:_:provider:alphaUDC:alphaQID:)`, :172/:180 legacy hashed-anchor overloads — 0/1 only, compatibility) -- PartialStateRecall.swift:58 `enum PartialStateRecall`; :75 `score(rowFingerprint:anchor:matchBlocks:differBlocks:) -> Double` — matchScore × differScore, blocks ∈ {0,1,2,3} precondition-enforced, empty set → 0; :104 `topK(anchor:rows:matchBlocks:differBlocks:k:)` — linear scan reference; :128 `hammingBlocks(_:_:blocks:)` — block-restricted Hamming -- ShingleSimilarity.swift:53 `enum ShingleSimilarity`; :58 `windowSize = 3`; :69 `shingles(_:) -> Set` — lowercase 3-char window; <3 chars → whole-string shingle; empty → empty set; :96 `similarity(_:_:) -> Float32` — Jaccard; both-empty → 0.0 -- MomentSummary.swift:54 `struct RowLite` (fingerprint + captureHLC, 40 bytes); :70 `summarize(rows:[Row]...)` / :89 `summarize(rows:[RowLite]...)` — OR-reduce of rows matching caller predicate; byte-identical, RowLite ~4-5× faster (conformance CRC 0x6762440b); :100 `capturedDuring(_:_:)` convenience predicate; :106 `orReduce(_:)` raw reduction -- TemporalCompression.swift:30 `enum WindowLevel` hour→year (:43 `nextCoarser`, :48 `approxSeconds` — deliberately approximate bucket divisors); :60 `struct TemporalWindow` (:80 `empty(level:)` identity); :93 `compress(rows:startHLC:endHLC:level:)`; :110 `rollup(windows:to:)` — OR fp union, `&+` counts, min/max bounds; :135 `cascadeRollup(hourWindows:upTo:)` — epoch-seconds integer-division bucketing per level +- FloatSimHash.swift:30 `enum FloatSimHash`; :42 `project(vector:seed:) -> Fingerprint256`: 256-hyperplane signed projection; empty vector → zero fp; :76 `planes(seed:dim:) -> FloatSimHashPlanes`: deterministic ±1 plane set, cacheable per (seed,dim); :61 private cached kernel (`PortableKernel.kernelForCurrentPlatform()`, selected once) +- CompositeDistance.swift:37 `defaultAlphaLattice = 0.5` / :38 `defaultAlphaFingerprint = 0.5` / :40 `fingerprintTotalBits = 256`; :50 `distance(latticeDistance:fingerprintHammingDistance:alphaLattice:alphaFingerprint:compatibleSeedScope:) -> Double`: preconditions (not clamps); scope-incompatible drops fp term WITHOUT renormalizing (intended) +- LatticeDistance.swift:45 `struct LatticeAnchorStr` (udc + qid; 0 = null); :59 `enum UDCTreeDistance` (:63 `longestCommonPrefixLength`, :83 `distance` clamped [0,1]); :104 `protocol WikidataAdjacencyProvider`; :110 `enum WikidataGraphDistance` (:112 `maxDepth = 4`, :113 `normalizationScale = 3.0`, :118 `shortestPathLength` bounded BFS, :144 `distance` = 1-exp(-len/3), unreachable/null → 1.0); :161 `enum LatticeDistance` (:163/:164 default alphas 0.5, :185 combined `distance(_:_:provider:alphaUDC:alphaQID:)`, :172/:180 legacy hashed-anchor overloads: 0/1 only, compatibility) +- PartialStateRecall.swift:58 `enum PartialStateRecall`; :75 `score(rowFingerprint:anchor:matchBlocks:differBlocks:) -> Double`: matchScore × differScore, blocks ∈ {0,1,2,3} precondition-enforced, empty set → 0; :104 `topK(anchor:rows:matchBlocks:differBlocks:k:)`: linear scan reference; :128 `hammingBlocks(_:_:blocks:)`: block-restricted Hamming +- ShingleSimilarity.swift:53 `enum ShingleSimilarity`; :58 `windowSize = 3`; :69 `shingles(_:) -> Set`: lowercase 3-char window; <3 chars → whole-string shingle; empty → empty set; :96 `similarity(_:_:) -> Float32`: Jaccard; both-empty → 0.0 +- MomentSummary.swift:54 `struct RowLite` (fingerprint + captureHLC, 40 bytes); :70 `summarize(rows:[Row]...)` / :89 `summarize(rows:[RowLite]...)`: OR-reduce of rows matching caller predicate; byte-identical, RowLite ~4-5× faster (conformance CRC 0x6762440b); :100 `capturedDuring(_:_:)` convenience predicate; :106 `orReduce(_:)` raw reduction +- TemporalCompression.swift:30 `enum WindowLevel` hour→year (:43 `nextCoarser`, :48 `approxSeconds`: deliberately approximate bucket divisors); :60 `struct TemporalWindow` (:80 `empty(level:)` identity); :93 `compress(rows:startHLC:endHLC:level:)`; :110 `rollup(windows:to:)`: OR fp union, `&+` counts, min/max bounds; :135 `cascadeRollup(hourWindows:upTo:)`: epoch-seconds integer-division bucketing per level ### Ingestion shaping - FeatureExtractors.swift:41 `enum StreamSourceFlag: UInt8`; :50 `struct AmbientSampleRow`; five sample+extractor pairs, each `extract(_:hlc:rowId:) -> AmbientSampleRow`: :82/:102/:109 HealthKit, :136/:158/:165 CoreLocation (+ :190 `quantizeGeohash` internal), :200/:223/:230 EventKit (attendees sorted before hashing), :257/:279/:286 ScreenTime, :308/:333/:340 SystemTelemetry; :363 internal `encodePayload` key-sorted deterministic binary. Pinned UDC anchors: 613.71 / 914 / 65.012.4 / 004.5 / 004.2. Only HealthKit rows carry payload -- AuditLogFold.swift:44 `struct ProjectedRowState` (no fingerprint — recompute from bitmaps+anchor); :79 `projectCurrentState(rowId:nounType:events:)`; :90 `projectStateAt(...asOf:)`; :101 `projectAll(events:asOf:nounTypeFor:)`; :131 private `foldOrdered` — HLC-sorted sequential fold; tombstone stateRaw == 33 (adjectiveBitmap & 0x3F) is STICKY (I-22); fold is permutation-invariant over the event set (I-21 sync convergence) -- RowAttributeView.swift:51 `enum RowAuditValue` (.bitmap/.integer/.null); :70 `struct RowAuditEntry`; :111 `struct RowAttributeView` (:130 init sorts attributes, :140/:151 manual Eq/Hash); :189 `from(auditEntries:) -> [RowAttributeView]` — vocab sorted+capped 64 (6-bit field), latest-write-wins per fieldPath by HLC, bitmap → one attr per set bit, integer → low byte, null → nothing, empty rows dropped; vocab is EPHEMERAL per call (pre-merge batches for stable indices) +- AuditLogFold.swift:44 `struct ProjectedRowState` (no fingerprint: recompute from bitmaps+anchor); :79 `projectCurrentState(rowId:nounType:events:)`; :90 `projectStateAt(...asOf:)`; :101 `projectAll(events:asOf:nounTypeFor:)`; :131 private `foldOrdered`: HLC-sorted sequential fold; tombstone stateRaw == 33 (adjectiveBitmap & 0x3F) is STICKY (I-22); fold is permutation-invariant over the event set (I-21 sync convergence) +- RowAttributeView.swift:51 `enum RowAuditValue` (.bitmap/.integer/.null); :70 `struct RowAuditEntry`; :111 `struct RowAttributeView` (:130 init sorts attributes, :140/:151 manual Eq/Hash); :189 `from(auditEntries:) -> [RowAttributeView]`: vocab sorted+capped 64 (6-bit field), latest-write-wins per fieldPath by HLC, bitmap → one attr per set bit, integer → low byte, null → nothing, empty rows dropped; vocab is EPHEMERAL per call (pre-merge batches for stable indices) ### Learning + decay -- MatrixDecay.swift:39 `struct DecayingMatrix` (values + halfLifeSeconds + lastDecayTimeSeconds; :60 subscript); :84 `MatrixDecay.apply(to:nowSeconds:estate:ts:)` — factor = exp(-dt·ln2/τ), dt ≤ 0 no-op but still emits factor 1.0; saturating Int64 cast for telemetry tag; :148 `decayFactor(elapsedSeconds:halfLifeSeconds:)` pure; :157 `decayAndAdd(...)` decay-then-add; :177/:183/:189 `applyExponentialDecay(to: MatrixF/O/C ...)` — intentional NO-OP reference adapters (real logic in production kit); :199 `Double.ln2 = 0.6931471805599453` (local, matches Rust f64::LN_2); :208 `enum DecayHalfLives` — F 90d, C 180d, O 60d, T 30d, ActionOutcomes 365d, calibration 730d, W_ranking 90d (illustrative; manifest is authoritative) -- ActionOutcomeMatrix.swift:37 `struct ActionOutcomeKey` — 6-bit preconditions (o07/o08), :51 `packed: UInt16`, :55 Comparable; :60 `struct ActionOutcomeCell` (:75 `successRate`, :83 `wilsonLowerBound` 95%, z=1.96); :95 `struct ActionOutcomeMatrix` (:101 `observe(action:outcome:success:at:)` `&+=`, :113 `successRate`, :119 `observationCount`, :139 `topActions(forOutcome:k:minObservations:)` — ranked by Wilson LB, ties count desc then action asc, returns rate+LB+count together, :158 `populatedCellCount`) -- BradleyTerry.swift:31 `struct PreferenceObservation` (winnerID/losers/weight); :53 `struct BradleyTerryEstimator` (`theta: [UUID: Double]` private(set)); :67 `init(learningRate: 0.05, l2: 0.001, theta: [:])`; :78 `observe(_:)` — SGD in log-space, multi-loser SEQUENTIAL against running theta (pinned parity contract); :105 `observeBatch(_:)`; :113 `strength(of:)` = exp(theta); :120 `probability(_:beats:)` = sigmoid(Δtheta). No RNG; warm-restartable; ranking signal only, never truth -- LLMCalibrationCurve.swift:37 `struct LLMCalibrationCurve`; :38 `binCount = 20` / :39 `binWidth = 0.05`; :50 `observe(claimedConfidence:actualOutcome:)` — clamp [0, 0.999999], wrapping counters; :62 `actualRate(in:)`; :72 `midpoint(of:)`; :78 `expectedCalibrationError()`; :92 `brierScore()`; :107 `decay(factor:)` fractional (730d half-life policy, daemon-driven) -- Sampling.swift:57 `enum Sampling`; :75 `sampleNormal(rng: inout SplitMix64)` — Box-Muller COSINE branch only, exactly 2 draws/call, u1 floored at leastNormalMagnitude; :102 `sampleGamma(shape:rng:)` — Marsaglia-Tsang (squeeze 0.0331), shape<1 via Ahrens-Dieter with extra draw BEFORE recursion (pinned order); :157 `sampleBeta(alpha:beta:rng:)` — g1/(g1+g2), alpha-Gamma drawn first, 0.5 degenerate fallback. Conformance vector sampling.json; relies on libm bit-identity +- MatrixDecay.swift:39 `struct DecayingMatrix` (values + halfLifeSeconds + lastDecayTimeSeconds; :60 subscript); :84 `MatrixDecay.apply(to:nowSeconds:estate:ts:)`: factor = exp(-dt·ln2/τ), dt ≤ 0 no-op but still emits factor 1.0; saturating Int64 cast for telemetry tag; :148 `decayFactor(elapsedSeconds:halfLifeSeconds:)` pure; :157 `decayAndAdd(...)` decay-then-add; :177/:183/:189 `applyExponentialDecay(to: MatrixF/O/C ...)`: intentional NO-OP reference adapters (real logic in production kit); :199 `Double.ln2 = 0.6931471805599453` (local, matches Rust f64::LN_2); :208 `enum DecayHalfLives`: F 90d, C 180d, O 60d, T 30d, ActionOutcomes 365d, calibration 730d, W_ranking 90d (illustrative; manifest is authoritative) +- ActionOutcomeMatrix.swift:37 `struct ActionOutcomeKey`: 6-bit preconditions (o07/o08), :51 `packed: UInt16`, :55 Comparable; :60 `struct ActionOutcomeCell` (:75 `successRate`, :83 `wilsonLowerBound` 95%, z=1.96); :95 `struct ActionOutcomeMatrix` (:101 `observe(action:outcome:success:at:)` `&+=`, :113 `successRate`, :119 `observationCount`, :139 `topActions(forOutcome:k:minObservations:)`: ranked by Wilson LB, ties count desc then action asc, returns rate+LB+count together, :158 `populatedCellCount`) +- BradleyTerry.swift:31 `struct PreferenceObservation` (winnerID/losers/weight); :53 `struct BradleyTerryEstimator` (`theta: [UUID: Double]` private(set)); :67 `init(learningRate: 0.05, l2: 0.001, theta: [:])`; :78 `observe(_:)`: SGD in log-space, multi-loser SEQUENTIAL against running theta (pinned parity contract); :105 `observeBatch(_:)`; :113 `strength(of:)` = exp(theta); :120 `probability(_:beats:)` = sigmoid(Δtheta). No RNG; warm-restartable; ranking signal only, never truth +- LLMCalibrationCurve.swift:37 `struct LLMCalibrationCurve`; :38 `binCount = 20` / :39 `binWidth = 0.05`; :50 `observe(claimedConfidence:actualOutcome:)`: clamp [0, 0.999999], wrapping counters; :62 `actualRate(in:)`; :72 `midpoint(of:)`; :78 `expectedCalibrationError()`; :92 `brierScore()`; :107 `decay(factor:)` fractional (730d half-life policy, daemon-driven) +- Sampling.swift:57 `enum Sampling`; :75 `sampleNormal(rng: inout SplitMix64)`: Box-Muller COSINE branch only, exactly 2 draws/call, u1 floored at leastNormalMagnitude; :102 `sampleGamma(shape:rng:)`: Marsaglia-Tsang (squeeze 0.0331), shape<1 via Ahrens-Dieter with extra draw BEFORE recursion (pinned order); :157 `sampleBeta(alpha:beta:rng:)`: g1/(g1+g2), alpha-Gamma drawn first, 0.5 degenerate fallback. Conformance vector sampling.json; relies on libm bit-identity ### Graph analytics -- CommunityDetection.swift:80 `typealias Adjacency = [[(neighbor: Int, weight: Double)]]`; :100 `detect(adjacency:maxPasses:estate:ts:)` — Phase 1 only; :152 `detectFull(adjacency:maxLevels:maxPasses:resolution:estate:ts:)` — full Louvain; resolution gamma default 1.0 = classical (IEEE 1.0*x == x, legacy-safe), NeuronKit callers pass 0.05 (pair-lock escape); :368 `canonicalize(_:)` — first-appearance renumbering; private: :202 totalEdgeWeight (zero threshold 1.0e-30 → identity labels, NO emit), :214 detectCore (gain ties → lowest label asc), :317 condense, :345 emit. Defaults maxPasses 10, maxLevels 10; exactly ONE `community.assignment` emit per call -- EigenvalueCentrality.swift:60 `typealias Adjacency`; :62 `defaultMaxIterations = 100` / :63 `defaultTolerance = 1e-6`; :87 `compute(...) -> [Double]` — AUTHORITY via Aᵀx (`xNext[j] += w*x[i]` for edge i→j; design-council 2026-06-04, conformance-locked; symmetrize for undirected), Perron shift 1.0 per iteration (breaks bipartite ±λ oscillation), zero-norm (<1e-30) → uniform 1/√n fallback, all 3 return paths emit `centrality.score` (value 1.0) -- RandomWalks.swift:36 `typealias Adjacency`; :38 `defaultRestartProb = 0.15`; :43 `walk(adjacency:start:length:restartProb:seed:) -> [Int]` — hard preconditions on kernel validity (indices, finite non-negative weights); walk[0] == start; dead end = implicit restart; :99 `sampleWeighted(_:rng:)` — roulette wheel; total ≤ 0 → uniform; rounding shortfall → last neighbor; :127 `uniform01(_:)` — top 53 bits, Rust bit-match; :141 `walkWithRestart(seed:steps:restartProbability:rngSeed:adjacency:) -> [RowId: Int]` — RowId-space uniform-choice visit counts (recall_exploratory); :172 `struct SplitMix64` (public; canonical constants 0x9E3779B97F4A7C15 / 0xBF58476D1CE4E5B9 / 0x94D049BB133111EB) -- NMFAlternatingLeastSquares.swift:45 `struct NMFFactorization` (W m×k, H k×n, rank, iterations, finalError RMS); :81 `factorize(V:rank:maxIterations:100:tolerance:1e-4:seed:0xDEADBEEFCAFEBABE:estate:ts:)` — Lee-Seung multiplicative updates, eps 1e-9, init = SplitMix64 16-bit draws /0xFFFF, preconditions rectangular/finite/non-negative; flat-buffer unsafe-pointer hot loops (loop nest + reduction order preserved EXACTLY; conformance CRC 0x300bf633; ~100× perf vs bounds-checked); emits `nmf.factor` = finalError; :220 `reconstructionError(V:W:H:)` public RMS; internal flat/nested matMul helpers :247/:288/:309/:329/:353/:370/:388 -- NMFDoubleFrobeniusSquared.swift:54 `struct NMFDoubleFrobeniusSquaredFactorization` (:93 `loadings(forRow:)`); :113 `enum NMFDoubleFrobeniusSquared` — :116 `defaultMaxIterations = 100`, :122 `defaultTolerance = 1e-6`, eps 1e-9; :153 `factorize(o:rows:cols:rank:seed:0xC0FFEE_BABE_BEEF:...)` — f64 scalar, RAW Frobenius² convergence (not RMS), init floored at 1e-3; :302 `struct NMFDoubleFrobeniusSquaredRNG` (SplitMix64 core, 53-bit output, floor 1e-3). PRODUCTION GATE: no consumer may wire to this pending substrate_math_performance benchmark (ruling 2026-06-13); still conformance-gated vs Rust (nmf_double_frobenius_squared.json) -- AnomalyDetection.swift:26 `enum AnomalyDetection`; :31 `zScore(value:mean:stddev:)` (stddev ≤ 0 → 0); :51 `rollingZScore(window:current:estate:ts:)` — emits `anomaly.flag`; :90 `modifiedZScore(value:median:mad:)` — 0.6745 consistency constant, mad ≤ 0 → 0; :110 `rollingModifiedZScore(...)` — IN-PLACE sort (5× perf, ~400KB saved at 100k; CRC 0x6c6fda4d; must match Rust sort_by partial_cmp for non-NaN); :152 `isAnomalous(zScore:threshold: 3.0)` -- JacobiSVD.swift:74 `struct SVDResult` (U, singularValues non-increasing, Vt, rank); :129 `decompose(A:rank:sweeps:) -> SVDResult` — one-sided cyclic Jacobi; sweeps default 30 PINNED (fixed count, not convergence-tested; MUST match across ports); eps 1e-9 skip/zero cutoff; scalar Float32 only, NO SIMD/FMA/Accelerate; sign convention: largest-|entry| of each left vector forced positive; preconditions m ≥ n, rectangular, non-empty; unsafe-buffer sweep loops (perf, indices loop-derived); :368 `jacobiCS(alpha:beta:gamma:)` — pinned expression tree (ζ,t,c,s), DO NOT refactor -- FFT.swift:36 `struct Complex` (:51 `magnitude`, :58 `magnitudeSquared`, :63/:68/:73 +,-,*); :83 `enum FFT` — :99 `forward(real:) -> [Complex]` Cooley-Tukey radix-2 DIT, length MUST be power of two (constitutional; callers zero-pad), :162 `magnitudeSpectrum(real:)`; :169/:180 `bitReverse`/`log2Floor` (@usableFromInline); :201 `struct RhythmResult` (dominantPeriodSeconds nil = "no rhythm", honest); :218 `enum RhythmAnalysis` — :229 `analyze(series:bucketDurationSeconds:)` scans bins 1...N/2, energy excludes DC; :281 `analyze(fingerprints:block:bitPosition:bucketDurationSeconds:)` — bit-series extraction, block 0..3 / bit 0..63 preconditions. Production vDSP path must be BIT-IDENTICAL to this scalar reference -- InformationTheory.swift:38 `entropy(_:)`; :48 `mutualInformation(joint:)`; :75 `klDivergence(_:_:)` — length precondition; skips p>0,q=0 terms ⇒ result is a LOWER BOUND; :88 `crossEntropy(_:_:)`; :99 `jensenShannon(_:_:)` bounded [0,1]; :112 `normalizedMutualInformation(joint:)` — ragged matrix → 0 sentinel. All log2 (bits); 0·log0 = 0 by term-skipping; distribution validity is caller's contract +- CommunityDetection.swift:80 `typealias Adjacency = [[(neighbor: Int, weight: Double)]]`; :100 `detect(adjacency:maxPasses:estate:ts:)`: Phase 1 only; :152 `detectFull(adjacency:maxLevels:maxPasses:resolution:estate:ts:)`: full Louvain; resolution gamma default 1.0 = classical (IEEE 1.0*x == x, legacy-safe), NeuronKit callers pass 0.05 (pair-lock escape); :368 `canonicalize(_:)`: first-appearance renumbering; private: :202 totalEdgeWeight (zero threshold 1.0e-30 → identity labels, NO emit), :214 detectCore (gain ties → lowest label asc), :317 condense, :345 emit. Defaults maxPasses 10, maxLevels 10; exactly ONE `community.assignment` emit per call +- EigenvalueCentrality.swift:60 `typealias Adjacency`; :62 `defaultMaxIterations = 100` / :63 `defaultTolerance = 1e-6`; :87 `compute(...) -> [Double]`: AUTHORITY via Aᵀx (`xNext[j] += w*x[i]` for edge i→j; design-council 2026-06-04, conformance-locked; symmetrize for undirected), Perron shift 1.0 per iteration (breaks bipartite ±λ oscillation), zero-norm (<1e-30) → uniform 1/√n fallback, all 3 return paths emit `centrality.score` (value 1.0) +- RandomWalks.swift:36 `typealias Adjacency`; :38 `defaultRestartProb = 0.15`; :43 `walk(adjacency:start:length:restartProb:seed:) -> [Int]`: hard preconditions on kernel validity (indices, finite non-negative weights); walk[0] == start; dead end = implicit restart; :99 `sampleWeighted(_:rng:)`: roulette wheel; total ≤ 0 → uniform; rounding shortfall → last neighbor; :127 `uniform01(_:)`: top 53 bits, Rust bit-match; :141 `walkWithRestart(seed:steps:restartProbability:rngSeed:adjacency:) -> [RowId: Int]`: RowId-space uniform-choice visit counts (recall_exploratory); :172 `struct SplitMix64` (public; canonical constants 0x9E3779B97F4A7C15 / 0xBF58476D1CE4E5B9 / 0x94D049BB133111EB) +- NMFAlternatingLeastSquares.swift:45 `struct NMFFactorization` (W m×k, H k×n, rank, iterations, finalError RMS); :81 `factorize(V:rank:maxIterations:100:tolerance:1e-4:seed:0xDEADBEEFCAFEBABE:estate:ts:)`: Lee-Seung multiplicative updates, eps 1e-9, init = SplitMix64 16-bit draws /0xFFFF, preconditions rectangular/finite/non-negative; flat-buffer unsafe-pointer hot loops (loop nest + reduction order preserved EXACTLY; conformance CRC 0x300bf633; ~100× perf vs bounds-checked); emits `nmf.factor` = finalError; :220 `reconstructionError(V:W:H:)` public RMS; internal flat/nested matMul helpers :247/:288/:309/:329/:353/:370/:388 +- NMFDoubleFrobeniusSquared.swift:54 `struct NMFDoubleFrobeniusSquaredFactorization` (:93 `loadings(forRow:)`); :113 `enum NMFDoubleFrobeniusSquared`: :116 `defaultMaxIterations = 100`, :122 `defaultTolerance = 1e-6`, eps 1e-9; :153 `factorize(o:rows:cols:rank:seed:0xC0FFEE_BABE_BEEF:...)`: f64 scalar, RAW Frobenius² convergence (not RMS), init floored at 1e-3; :302 `struct NMFDoubleFrobeniusSquaredRNG` (SplitMix64 core, 53-bit output, floor 1e-3). PRODUCTION GATE: no consumer may wire to this pending substrate_math_performance benchmark (ruling 2026-06-13); still conformance-gated vs Rust (nmf_double_frobenius_squared.json) +- AnomalyDetection.swift:26 `enum AnomalyDetection`; :31 `zScore(value:mean:stddev:)` (stddev ≤ 0 → 0); :51 `rollingZScore(window:current:estate:ts:)`: emits `anomaly.flag`; :90 `modifiedZScore(value:median:mad:)`: 0.6745 consistency constant, mad ≤ 0 → 0; :110 `rollingModifiedZScore(...)`: IN-PLACE sort (5× perf, ~400KB saved at 100k; CRC 0x6c6fda4d; must match Rust sort_by partial_cmp for non-NaN); :152 `isAnomalous(zScore:threshold: 3.0)` +- JacobiSVD.swift:74 `struct SVDResult` (U, singularValues non-increasing, Vt, rank); :129 `decompose(A:rank:sweeps:) -> SVDResult`: one-sided cyclic Jacobi; sweeps default 30 PINNED (fixed count, not convergence-tested; MUST match across ports); eps 1e-9 skip/zero cutoff; scalar Float32 only, NO SIMD/FMA/Accelerate; sign convention: largest-|entry| of each left vector forced positive; preconditions m ≥ n, rectangular, non-empty; unsafe-buffer sweep loops (perf, indices loop-derived); :368 `jacobiCS(alpha:beta:gamma:)`: pinned expression tree (ζ,t,c,s), DO NOT refactor +- FFT.swift:36 `struct Complex` (:51 `magnitude`, :58 `magnitudeSquared`, :63/:68/:73 +,-,*); :83 `enum FFT`: :99 `forward(real:) -> [Complex]` Cooley-Tukey radix-2 DIT, length MUST be power of two (constitutional; callers zero-pad), :162 `magnitudeSpectrum(real:)`; :169/:180 `bitReverse`/`log2Floor` (@usableFromInline); :201 `struct RhythmResult` (dominantPeriodSeconds nil = "no rhythm", honest); :218 `enum RhythmAnalysis`: :229 `analyze(series:bucketDurationSeconds:)` scans bins 1...N/2, energy excludes DC; :281 `analyze(fingerprints:block:bitPosition:bucketDurationSeconds:)`: bit-series extraction, block 0..3 / bit 0..63 preconditions. Production vDSP path must be BIT-IDENTICAL to this scalar reference +- InformationTheory.swift:38 `entropy(_:)`; :48 `mutualInformation(joint:)`; :75 `klDivergence(_:_:)`: length precondition; skips p>0,q=0 terms ⇒ result is a LOWER BOUND; :88 `crossEntropy(_:_:)`; :99 `jensenShannon(_:_:)` bounded [0,1]; :112 `normalizedMutualInformation(joint:)`: ragged matrix → 0 sentinel. All log2 (bits); 0·log0 = 0 by term-skipping; distribution validity is caller's contract ### Pattern mining -- AssociationRuleMining.swift:52 `struct Item` (field/value UInt8; :64 `packed: UInt16` = field<<8|value; :68 Comparable); :76 `struct AssociationRule` (single-Item antecedent + 5 metrics); :107 `struct MiningThresholds` (minSupport/minConfidence); :136 `mineAssociationRules(matrix: MatrixO, activeRowCount:thresholds:)` — N injected (matrix has no row count); activeRowCount ≤ 0 → []; :154/:164 internal `AssociationRuleEngine.mine` — two passes: diagonal = single supports, off-diagonal in canonical packed-key order (NO final sort needed — conformance-relied ordering); self-rules skipped; conviction = +inf at confidence 1; k>2 → AprioriMining -- AprioriMining.swift:55 `struct AprioriThresholds` (minSupport/minConfidence/minLift ≥ 1.0 default/maxK default 3, floored ≥ 2); :89 custom `init(from:)` routes through public init so serialized maxK < 2 cannot bypass the clamp; :115 `struct AprioriRule` (antecedent `[Item]` sorted by packed, + evidenceCount); :167 `AprioriMining.mine(rows:thresholds:)` — join on (k-2)-prefix, subset-count, prune; rule extraction iterates itemsets in lex order (dict order undefined); 4-key sort lift↓ confidence↓ evidence↓ lex↑; :310 `mineAprioriRules(rows:thresholds:)` free-function wrapper; private `HashableItemset`/`itemsetLess`/`aprioriJoin` (:325/:340/:366) -- FormalConceptAnalysis.swift:41 `struct FormalAttribute: Comparable` — lex (namespace,key,value) order = determinism backbone; :70 `enum SeedMode` .single/.multi; :90 `struct CoverDelta` / :117 `struct ConceptCoverDeltas` (:140 `covering(concepts:)` — cover edges ≠ implications; O(n²m) bounded); :210 `struct FormalConcept` (extent/intent/support/stability?); :247 `struct FormalContext` — :251 nested `RowID = UInt32` (avoids LocusKit RowID collision), :285 `from(rowAttributeViews:)` (namespace "row"), :298 `init(rows:)`, :334 `extent(of:)`, :343 `intent(of:)`, :354 `closure(of:)`; internal FCABitSet plumbing :366/:380/:386/:665; :415 `struct BoundedConceptMiner` (minSupport clamped ≥1, maxIntentSize, maxConcepts, seedMode, maxSeeds, stabilityBudget 0 default → stability nil, stabilitySeed 0xCAFEBABEDEADBEEF) — :475 `mine(context:)`: one closure per seed, dedup by intent, truncate; sort support↓ intent-size↑ lex; :595 `enum StabilityEstimator` (:616 `estimate(concept:context:budget:seed:)` — Bernoulli-half subsets, per-concept SplitMix64 seed = caller seed XOR FNV(canonical key)) -- ConceptImplications.swift:62 `struct Implication` (premise/conclusion; caller ensures disjoint); :96 `struct ConceptImplications` (implications + isTruncated); :148 `conceptImplications(over:context:maxImplications:maxPremiseSize:)` — Next-Closure pseudo-intent enumeration in increasing size order; maxImplications hard cap sets isTruncated (with look-ahead :358 for honesty); maxPremiseSize skips silently (soundness unaffected); output sort premise-size → lex premise → lex conclusion; empty context → empty untruncated -- TemporalCausalityFold.swift:61 `struct TemporalFieldCoord` (fieldPath + valueRepr stable string); :98 `struct TemporalAuditEntry` (hlc + fieldCoords); :121 `struct FoldResult` (deltas in first-insertion order + newWatermark); :139 `struct TemporalCausalityKey` (source/target/lagBucket); :174 `lagBuckets = [1,2,4,8,16,32,64,128]` — must mirror MatrixTier.lagBuckets exactly; :178 `defaultWindowMinutes = 256`; :191 `maxWindowOccupancy = 512` — bulk-import quadratic guard (2026-07-02 decision), keeps most-recent in-window sources; :202 `lagBucket(forMinutes:)` — smallest boundary ≥ delta, clamp 128; zero-ms → minute 1; :241 `fold(entries:windowMinutes:startWatermark:)` — entries MUST be pre-sorted by HLC (not re-sorted; silent corruption otherwise); hourly batch cadence supersedes weekly +- AssociationRuleMining.swift:52 `struct Item` (field/value UInt8; :64 `packed: UInt16` = field<<8|value; :68 Comparable); :76 `struct AssociationRule` (single-Item antecedent + 5 metrics); :107 `struct MiningThresholds` (minSupport/minConfidence); :136 `mineAssociationRules(matrix: MatrixO, activeRowCount:thresholds:)`: N injected (matrix has no row count); activeRowCount ≤ 0 → []; :154/:164 internal `AssociationRuleEngine.mine`: two passes: diagonal = single supports, off-diagonal in canonical packed-key order (NO final sort needed: conformance-relied ordering); self-rules skipped; conviction = +inf at confidence 1; k>2 → AprioriMining +- AprioriMining.swift:55 `struct AprioriThresholds` (minSupport/minConfidence/minLift ≥ 1.0 default/maxK default 3, floored ≥ 2); :89 custom `init(from:)` routes through public init so serialized maxK < 2 cannot bypass the clamp; :115 `struct AprioriRule` (antecedent `[Item]` sorted by packed, + evidenceCount); :167 `AprioriMining.mine(rows:thresholds:)`: join on (k-2)-prefix, subset-count, prune; rule extraction iterates itemsets in lex order (dict order undefined); 4-key sort lift↓ confidence↓ evidence↓ lex↑; :310 `mineAprioriRules(rows:thresholds:)` free-function wrapper; private `HashableItemset`/`itemsetLess`/`aprioriJoin` (:325/:340/:366) +- FormalConceptAnalysis.swift:41 `struct FormalAttribute: Comparable`: lex (namespace,key,value) order = determinism backbone; :70 `enum SeedMode` .single/.multi; :90 `struct CoverDelta` / :117 `struct ConceptCoverDeltas` (:140 `covering(concepts:)`: cover edges ≠ implications; O(n²m) bounded); :210 `struct FormalConcept` (extent/intent/support/stability?); :247 `struct FormalContext`: :251 nested `RowID = UInt32` (avoids LocusKit RowID collision), :285 `from(rowAttributeViews:)` (namespace "row"), :298 `init(rows:)`, :334 `extent(of:)`, :343 `intent(of:)`, :354 `closure(of:)`; internal FCABitSet plumbing :366/:380/:386/:665; :415 `struct BoundedConceptMiner` (minSupport clamped ≥1, maxIntentSize, maxConcepts, seedMode, maxSeeds, stabilityBudget 0 default → stability nil, stabilitySeed 0xCAFEBABEDEADBEEF): :475 `mine(context:)`: one closure per seed, dedup by intent, truncate; sort support↓ intent-size↑ lex; :595 `enum StabilityEstimator` (:616 `estimate(concept:context:budget:seed:)`: Bernoulli-half subsets, per-concept SplitMix64 seed = caller seed XOR FNV(canonical key)) +- ConceptImplications.swift:62 `struct Implication` (premise/conclusion; caller ensures disjoint); :96 `struct ConceptImplications` (implications + isTruncated); :148 `conceptImplications(over:context:maxImplications:maxPremiseSize:)`: Next-Closure pseudo-intent enumeration in increasing size order; maxImplications hard cap sets isTruncated (with look-ahead :358 for honesty); maxPremiseSize skips silently (soundness unaffected); output sort premise-size → lex premise → lex conclusion; empty context → empty untruncated +- TemporalCausalityFold.swift:61 `struct TemporalFieldCoord` (fieldPath + valueRepr stable string); :98 `struct TemporalAuditEntry` (hlc + fieldCoords); :121 `struct FoldResult` (deltas in first-insertion order + newWatermark); :139 `struct TemporalCausalityKey` (source/target/lagBucket); :174 `lagBuckets = [1,2,4,8,16,32,64,128]`: must mirror MatrixTier.lagBuckets exactly; :178 `defaultWindowMinutes = 256`; :191 `maxWindowOccupancy = 512`: bulk-import quadratic guard (2026-07-02 decision), keeps most-recent in-window sources; :202 `lagBucket(forMinutes:)`: smallest boundary ≥ delta, clamp 128; zero-ms → minute 1; :241 `fold(entries:windowMinutes:startWatermark:)`: entries MUST be pre-sorted by HLC (not re-sorted; silent corruption otherwise); hourly batch cadence supersedes weekly ### Distillation -- TypedDecayWeighting.swift:32 `enum DistillationFeatureType: String` — ENT/REL/TMP/NUM; :40 `decayLambda` = 0.1/0.2/0.5/0.8 (pinned, diffusion-schedule analogy); :52 `enum TypedDecayWeighting`; :65 `weight(featureType:ageInUnits:)` = exp(-λ·max(0,age)); :86 `weightedDocFrequency(featureType:presenceTimestamps:allMemoryTimestamps:referenceDate:timeUnit: 86_400)` — decayed presence weight / total weight; empty or zero-total → 0 -- DeltaFeatureExtractor.swift:29 `enum DeltaType: String` STATIC/CONVERGENT/MONOTONE/OSCILLATING/DIVERGENT; :41 `struct DeltaAnalysis` (deltaType/terminalValue/convergenceScore/slope?/confidence); :86 `analyzeCategorical(sequence:decayLambda: 0.5)` — oscillation = period-2 over LAST 4 obs only (≥4 required); convergence = trailing-run fraction vs λ; :148 `analyzeNumerical(sequence:decayLambda: 0.8)` — diffs all-zero STATIC / one-sign MONOTONE (confidence = λ fixed) / strict alternation OSCILLATING / else DIVERGENT. Pure; zero RNG -- DistillationScorer.swift:16 `struct ExtractedFeature` (:41 init, display defaults to value); :58 `struct DistillationSNR` (readyToDistill iff snr ≥ 2.0); :71 `struct FeatureGraph` (components sorted by weighted df ↓ — dominant first); :94 internal `binaryEntropy(_:)`; :114 `structuralThreshold(M:)` = 2/M (M=1 → 2.0, correctly blocks); :120 `applyStructuralThreshold(features:M:)`; :145 `computeSNR(features:M:)` — episodic floor 1e-6; :174 `computeStructuralScores(features: inout)` σ = df·(1−H(df)); :194 `buildPMIGraph(thresholdFeatures:incidenceMatrix:M:)` — PMI log2 ONLY (Foundation.log2 ⇔ f32::log2 bit-match), positive-PMI edges, DFS components O(n²) ok ≤ ~150; :257 `selectDominantComponent(graph:)`; :277 `computeConfidence(selected:allThreshold:)` = meanDf × coherence ratio -- DistillationPipeline.swift:27 `struct DistillationInput` (memoryContents/timestamps?/clusterID/sourceIDs; computed M); :53 `struct DistillationOutput` (drawerContent/confidence/uncertain/snr/deltaType/succeeded/failureReason/featureFingerprint); :81 `struct DistilledHeader` (:102 `parse(_:)` — nil unless "[DIST|" prefix); :172 `typealias FeatureExtractor = @Sendable (String, DistillationFeatureType) -> [ExtractedFeature]`; :184 `featureSimHashSeed = 0x44495354494C4C41` ("DISTILLA") — conformance-critical, changing invalidates ALL stored fingerprints; :193 `featureHash(_:) -> Fingerprint256`; :221 `queryFingerprint(query:extractFeatures:)` — probe fp, no inference; :240 `defaultExtractor` (capitalization heuristic; tests/default); :277 `run(input:extractFeatures:intraItem: false)` — 5 stages; intraItem skips SNR gate + keeps ALL passing features (no PMI pruning of a single doc); ubiquity re-add at df ≥ (M-1)/M (zero-PMI spine rescue); confidence < 0.4 fail, [0.4,0.7) uncertain; `src=` = sourceIDs.count NOT M; deliberate monolithic body +- TypedDecayWeighting.swift:32 `enum DistillationFeatureType: String`: ENT/REL/TMP/NUM; :40 `decayLambda` = 0.1/0.2/0.5/0.8 (pinned, diffusion-schedule analogy); :52 `enum TypedDecayWeighting`; :65 `weight(featureType:ageInUnits:)` = exp(-λ·max(0,age)); :86 `weightedDocFrequency(featureType:presenceTimestamps:allMemoryTimestamps:referenceDate:timeUnit: 86_400)`: decayed presence weight / total weight; empty or zero-total → 0 +- DeltaFeatureExtractor.swift:29 `enum DeltaType: String` STATIC/CONVERGENT/MONOTONE/OSCILLATING/DIVERGENT; :41 `struct DeltaAnalysis` (deltaType/terminalValue/convergenceScore/slope?/confidence); :86 `analyzeCategorical(sequence:decayLambda: 0.5)`: oscillation = period-2 over LAST 4 obs only (≥4 required); convergence = trailing-run fraction vs λ; :148 `analyzeNumerical(sequence:decayLambda: 0.8)`: diffs all-zero STATIC / one-sign MONOTONE (confidence = λ fixed) / strict alternation OSCILLATING / else DIVERGENT. Pure; zero RNG +- DistillationScorer.swift:16 `struct ExtractedFeature` (:41 init, display defaults to value); :58 `struct DistillationSNR` (readyToDistill iff snr ≥ 2.0); :71 `struct FeatureGraph` (components sorted by weighted df ↓: dominant first); :94 internal `binaryEntropy(_:)`; :114 `structuralThreshold(M:)` = 2/M (M=1 → 2.0, correctly blocks); :120 `applyStructuralThreshold(features:M:)`; :145 `computeSNR(features:M:)`: episodic floor 1e-6; :174 `computeStructuralScores(features: inout)` σ = df·(1−H(df)); :194 `buildPMIGraph(thresholdFeatures:incidenceMatrix:M:)`: PMI log2 ONLY (Foundation.log2 ⇔ f32::log2 bit-match), positive-PMI edges, DFS components O(n²) ok ≤ ~150; :257 `selectDominantComponent(graph:)`; :277 `computeConfidence(selected:allThreshold:)` = meanDf × coherence ratio +- DistillationPipeline.swift:27 `struct DistillationInput` (memoryContents/timestamps?/clusterID/sourceIDs; computed M); :53 `struct DistillationOutput` (drawerContent/confidence/uncertain/snr/deltaType/succeeded/failureReason/featureFingerprint); :81 `struct DistilledHeader` (:102 `parse(_:)`: nil unless "[DIST|" prefix); :172 `typealias FeatureExtractor = @Sendable (String, DistillationFeatureType) -> [ExtractedFeature]`; :184 `featureSimHashSeed = 0x44495354494C4C41` ("DISTILLA"): conformance-critical, changing invalidates ALL stored fingerprints; :193 `featureHash(_:) -> Fingerprint256`; :221 `queryFingerprint(query:extractFeatures:)`: probe fp, no inference; :240 `defaultExtractor` (capitalization heuristic; tests/default); :277 `run(input:extractFeatures:intraItem: false)`: 5 stages; intraItem skips SNR gate + keeps ALL passing features (no PMI pruning of a single doc); ubiquity re-add at df ≥ (M-1)/M (zero-PMI spine rescue); confidence < 0.4 fail, [0.4,0.7) uncertain; `src=` = sourceIDs.count NOT M; deliberate monolithic body ### Federation + privacy -- PairingHandshake.swift:37 `struct PairingNonce` — exactly 32 bytes (precondition); :48 `seedWith(estateA:estateB:)` — UUIDs ordered by RAW BYTES, never uuidString (ASCII hex order ≠ byte order; Rust compares [u8;16] — string order would derive incompatible families); :79 `struct PairingRecord` (:87 `isActive`); :114 `struct PairingAuditPayload`; :133 `enum PairingHandshake` — :145 `generateSharedFamily(nonce:estateA:estateB:density:)` (both sides derive identical family, no round trip), :156 `sharedFamilyKey(case:peerEstate:)` = "H_shared__", :168 `buildPairEvent(...)`, :179 `buildUnpairEvent(...)` (unpair RETAINS family for asOf queries); :192 private `combinedFamilyHash`; :209 fileprivate `lexLessOrEqual` raw-byte compare. FNV-1a 0xCBF29CE484222325 / 0x100000001B3 mixers -- TierContributionFingerprint.swift:45 `enum FederationCase: UInt32` household 1 / fleet 2 / industry 3; :51 `struct TierContribution`; :81 `build(estateUUID:case:shareableFingerprints:hlc:)` — kernel-dispatched orReduce256; :95 `encode(_:) -> Data` — 64-byte canonical wire: UUID(16) | case u32 BE | rowCount u32 BE | 4× fp block u64 BE | hlc.packed u64 BE — BE-UNIFORM (earlier LE-fp draft diverged from Rust by 32 bytes; conformance caught it); :117 `decode(_:)` — nil unless exactly 64 bytes + known case; :148/:154/:160/:166 endian helpers. No signing/checksum at this layer (egress signs; DP at aggregator) -- TierAscendingQuery.swift:35 `enum TargetTier` .peer/.fleetAggregate/.industryAggregate; :41 `struct TierAscendingQuery` (originatingEstate/primitiveName/primitiveInput/targetTier/privacyBudget/queryHLC); :61 `struct PeerResponse`; :68 `enum TierAscendingQueryProtocol` — :72 `computeLocal(query:dispatch:)` (injected closure; no CognitionKit dep), :80 `applyDPToContribution(_:budget:rngSeed:)` — Laplace scale 1/ε per score, CI = ±1.96·scale; NOTE Swift forwards `breakdown` unchanged while Rust zeroes it (side-channel; parity nuance), :100 `combine(local:peers:)` — per-RowId score sum, sort score↓ RowId↑, WIDEST CI wins (conservative); :133 `struct PrivacyLedger` (:141 `remaining`, :147 `canConsume`, :152 `consume`, :159 `dailyReset` — manual, no auto-expiry) -- DPORReduction.swift:31 `struct DPParameters` (:36 init — ε 1.0 > 0, δ 1e-9 ∈ [0,1) VALIDATED BUT UNUSED by Laplace (documented), k 3 ≥ 1); :48 `enum DPORReduction`; :54 `reduce(fingerprints:params:rngSeed:) -> Fingerprint256` — per-bit popcount + Laplace(1/ε) noise + k-anonymity threshold; empty → .zero; fresh SplitMix64(rngSeed) inside; :85 internal `laplaceNoise(scale:rng:)` — inverse CDF from top 53 bits (raw >> 11) +- PairingHandshake.swift:37 `struct PairingNonce`: exactly 32 bytes (precondition); :48 `seedWith(estateA:estateB:)`: UUIDs ordered by RAW BYTES, never uuidString (ASCII hex order ≠ byte order; Rust compares [u8;16]: string order would derive incompatible families); :79 `struct PairingRecord` (:87 `isActive`); :114 `struct PairingAuditPayload`; :133 `enum PairingHandshake`: :145 `generateSharedFamily(nonce:estateA:estateB:density:)` (both sides derive identical family, no round trip), :156 `sharedFamilyKey(case:peerEstate:)` = "H_shared__", :168 `buildPairEvent(...)`, :179 `buildUnpairEvent(...)` (unpair RETAINS family for asOf queries); :192 private `combinedFamilyHash`; :209 fileprivate `lexLessOrEqual` raw-byte compare. FNV-1a 0xCBF29CE484222325 / 0x100000001B3 mixers +- TierContributionFingerprint.swift:45 `enum FederationCase: UInt32` household 1 / fleet 2 / industry 3; :51 `struct TierContribution`; :81 `build(estateUUID:case:shareableFingerprints:hlc:)`: kernel-dispatched orReduce256; :95 `encode(_:) -> Data`: 64-byte canonical wire: UUID(16) | case u32 BE | rowCount u32 BE | 4× fp block u64 BE | hlc.packed u64 BE: BE-UNIFORM (earlier LE-fp draft diverged from Rust by 32 bytes; conformance caught it); :117 `decode(_:)`: nil unless exactly 64 bytes + known case; :148/:154/:160/:166 endian helpers. No signing/checksum at this layer (egress signs; DP at aggregator) +- TierAscendingQuery.swift:35 `enum TargetTier` .peer/.fleetAggregate/.industryAggregate; :41 `struct TierAscendingQuery` (originatingEstate/primitiveName/primitiveInput/targetTier/privacyBudget/queryHLC); :61 `struct PeerResponse`; :68 `enum TierAscendingQueryProtocol`: :72 `computeLocal(query:dispatch:)` (injected closure; no CognitionKit dep), :80 `applyDPToContribution(_:budget:rngSeed:)`: Laplace scale 1/ε per score, CI = ±1.96·scale; NOTE Swift forwards `breakdown` unchanged while Rust zeroes it (side-channel; parity nuance), :100 `combine(local:peers:)`: per-RowId score sum, sort score↓ RowId↑, WIDEST CI wins (conservative); :133 `struct PrivacyLedger` (:141 `remaining`, :147 `canConsume`, :152 `consume`, :159 `dailyReset`: manual, no auto-expiry) +- DPORReduction.swift:31 `struct DPParameters` (:36 init: ε 1.0 > 0, δ 1e-9 ∈ [0,1) VALIDATED BUT UNUSED by Laplace (documented), k 3 ≥ 1); :48 `enum DPORReduction`; :54 `reduce(fingerprints:params:rngSeed:) -> Fingerprint256`: per-bit popcount + Laplace(1/ε) noise + k-anonymity threshold; empty → .zero; fresh SplitMix64(rngSeed) inside; :85 internal `laplaceNoise(scale:rng:)`: inverse CDF from top 53 bits (raw >> 11) ## INVARIANTS / GOTCHAS - DETERMINISM IS THE CONTRACT. Swift and Rust legs must agree bit-for-bit on shared conformance vectors. Any change to seeds, draw orders, tie-breaks, loop-nest/reduction order, or pinned constants must be mirrored in `rust/src/` and pass both test suites. SplitMix64 is the only PRNG; all uniform doubles come from the top 53 bits. -- NO CLOCKS, NO STATE, NO I/O anywhere in the package. `ts`, `nowSeconds`, `referenceDate`, watermarks — all caller-supplied. Everything is Sendable value types or pure static functions. -- Pinned seeds: canonical conformance seed 0xCAFEBABEDEADBEEF (FCA stability, temporal-fold vectors); NMF default 0xDEADBEEFCAFEBABE; parked-NMF 0xC0FFEE_BABE_BEEF (historical, deliberate); distillation featureSimHashSeed 0x44495354494C4C41 — changing it orphans every stored distillation fingerprint. -- Pinned constants — do not change without conformance regen: JacobiSVD sweeps 30; FFT power-of-two input; Louvain zero-weight 1e-30 + gamma 1.0 legacy identity; centrality Perron shift 1.0 + tolerance 1e-6 + cap 100; restartProb 0.15; Marsaglia-Tsang 0.0331; anomaly threshold 3.0 + MAD 0.6745 (CRC 0x6c6fda4d); shingle window 3; lag buckets {1..128} + window 256 min + occupancy 512; SNR gate 2.0 + τ 2/M + confidence 0.4/0.7 + ubiquity (M-1)/M; decay λ ENT/REL/TMP/NUM 0.1/0.2/0.5/0.8; half-life table (90/180/60/30/365/730/90 d); calibration 20 × 0.05 bins; Wilson z 1.96; DP defaults ε 1.0, δ 1e-9, k 3; CompositeDistance 256 bits, alphas 0.5/0.5; UDC clamp, BFS depth 4, scale 3.0; NMF eps 1e-9 + tolerance 1e-4 (CRC 0x300bf633); MomentSummary CRC 0x6762440b. +- NO CLOCKS, NO STATE, NO I/O anywhere in the package. `ts`, `nowSeconds`, `referenceDate`, watermarks: all caller-supplied. Everything is Sendable value types or pure static functions. +- Pinned seeds: canonical conformance seed 0xCAFEBABEDEADBEEF (FCA stability, temporal-fold vectors); NMF default 0xDEADBEEFCAFEBABE; parked-NMF 0xC0FFEE_BABE_BEEF (historical, deliberate); distillation featureSimHashSeed 0x44495354494C4C41: changing it orphans every stored distillation fingerprint. +- Pinned constants: do not change without conformance regen: JacobiSVD sweeps 30; FFT power-of-two input; Louvain zero-weight 1e-30 + gamma 1.0 legacy identity; centrality Perron shift 1.0 + tolerance 1e-6 + cap 100; restartProb 0.15; Marsaglia-Tsang 0.0331; anomaly threshold 3.0 + MAD 0.6745 (CRC 0x6c6fda4d); shingle window 3; lag buckets {1..128} + window 256 min + occupancy 512; SNR gate 2.0 + τ 2/M + confidence 0.4/0.7 + ubiquity (M-1)/M; decay λ ENT/REL/TMP/NUM 0.1/0.2/0.5/0.8; half-life table (90/180/60/30/365/730/90 d); calibration 20 × 0.05 bins; Wilson z 1.96; DP defaults ε 1.0, δ 1e-9, k 3; CompositeDistance 256 bits, alphas 0.5/0.5; UDC clamp, BFS depth 4, scale 3.0; NMF eps 1e-9 + tolerance 1e-4 (CRC 0x300bf633); MomentSummary CRC 0x6762440b. - NMFDoubleFrobeniusSquared is PRODUCTION-GATED: no consumer may wire to it until the substrate_math_performance benchmark passes. Do not "clean it up" into use; do not delete it either (it is the benchmark baseline). - EigenvalueCentrality is AUTHORITY (Aᵀx), locked by conformance vector. Do not flip to hub semantics. Symmetrize adjacency for undirected use. - MatrixDecay is constitutional: decay only shrinks; the three typed applyExponentialDecay overloads are intentional no-ops at reference level. dt ≤ 0 is a no-op that still emits factor 1.0. -- AuditLogFold tombstone (stateRaw 33) is sticky forever; fold is event-set permutation-invariant — that IS the sync-convergence proof. Fingerprint deliberately absent from projections. -- TemporalCausalityFold requires pre-sorted input by HLC; violation silently corrupts (no crash). RowAttributeView vocab (≤64 fields) is per-call ephemeral — merge batches first for stable indices. -- PairingHandshake seed ordering compares raw UUID bytes, never uuidString — ASCII hex order can invert byte order and desync from Rust. Nonce exactly 32 bytes. +- AuditLogFold tombstone (stateRaw 33) is sticky forever; fold is event-set permutation-invariant: that IS the sync-convergence proof. Fingerprint deliberately absent from projections. +- TemporalCausalityFold requires pre-sorted input by HLC; violation silently corrupts (no crash). RowAttributeView vocab (≤64 fields) is per-call ephemeral: merge batches first for stable indices. +- PairingHandshake seed ordering compares raw UUID bytes, never uuidString: ASCII hex order can invert byte order and desync from Rust. Nonce exactly 32 bytes. - TierContribution wire format is BE-uniform 64 bytes (fingerprint blocks BE, not the LE storage form). Unsigned at this layer by design. - TierAscendingQuery parity nuance: Rust zeroes DistanceBreakdown in DP contributions; Swift forwards it unchanged. Flag before relying on either behavior. -- CompositeDistance with incompatible seed scope DROPS the fingerprint term without renormalizing — smaller distance is intended, not a bug. Preconditions, not clamps, on inputs. +- CompositeDistance with incompatible seed scope DROPS the fingerprint term without renormalizing: smaller distance is intended, not a bug. Preconditions, not clamps, on inputs. - InformationTheory.klDivergence is a LOWER BOUND when q has zeros where p has mass. NMI returns 0 sentinel on ragged input. All log2. - Apriori maxK floor 2 is enforced in init AND in custom Decodable (synthesized decode would bypass it). Association mining needs caller-injected N; matrix diagonal doubles as single-item support. - Hard preconditions (crash, not error) on malformed inputs throughout: RandomWalks kernel validity, NMF domain (rectangular/finite/non-negative), SVD shape (m ≥ n), FFT length, block IDs {0..3}, 6-bit action/outcome keys, 32-byte nonce. -- Unsafe-buffer hot loops (NMF, SVD, moment/anomaly paths) exist for auto-vectorization; all indices are loop-derived, never data-derived. Preserve loop nest and reduction order exactly — Float32 lowest-bit rounding is conformance-visible. +- Unsafe-buffer hot loops (NMF, SVD, moment/anomaly paths) exist for auto-vectorization; all indices are loop-derived, never data-derived. Preserve loop nest and reduction order exactly: Float32 lowest-bit rounding is conformance-visible. - Telemetry: exactly five signals (VizGraphSignals); one emit per invocation; off-path = single atomic-bool load. Degenerate zero-weight Louvain emits nothing. Only AnomalyDetection/CommunityDetection/EigenvalueCentrality/NMF/MatrixDecay emit. - Distillation `src=` field counts sourceIDs, not cluster members (differs in intra-item mode). intraItem skips the SNR gate and PMI pruning by design. diff --git a/packages/libs/SubstrateML/docs/DETAILS.md b/packages/libs/SubstrateML/docs/DETAILS.md index 822712f..c59db28 100644 --- a/packages/libs/SubstrateML/docs/DETAILS.md +++ b/packages/libs/SubstrateML/docs/DETAILS.md @@ -85,1038 +85,1154 @@ sources: # SubstrateML Details -This document walks through every source file in the package. Read -`OVERVIEW.md` first for the big picture. Files appear in working-group -order: the telemetry names, the fingerprint and distance math, the -ingestion shapers, the learning and decay machinery, the graph analytics, -the pattern miners, the distillation stages, and finally the federation -and privacy layer. +This document walks through each source file in the package. Read +`OVERVIEW.md` first for the big picture. The files appear in +working-group order. First come the telemetry names. Then come the +fingerprint and distance math. Then the ingestion shapers. Then the +learning and decay machinery. Then the graph analytics. Then the +pattern miners. Then the distillation stages. Last come the +federation and privacy layer. ## VizGraphSignals.swift -This file provides the canonical metric names for the package's telemetry -layer: five string constants used as signal names when the five -graph-analytic algorithms report completion. - -Telemetry is optional observation data for the Topology visualization — a -live picture of the memory graph. Each algorithm emits exactly one metric -per invocation through `IntellectusLib`, and only when monitoring is -enabled. When monitoring is off (the default), the emit costs a single -atomic boolean load and a branch: no clock read, no allocation. Keeping -the names in one authoritative file prevents a typo from producing an -orphaned metric that no dashboard ever finds. - -The five constants are `communityAssignment` ("community.assignment", -value is the community count), `centralityScore` ("centrality.score", -value 1.0 as a completion marker), `nmfFactor` ("nmf.factor", value is the -final reconstruction error), `anomalyFlag` ("anomaly.flag", value is the -absolute z-score), and `edgeDecayedWeight` ("edge.decayed_weight", value -is the applied decay factor). The file also records the caller contract: -callers supply the estate tag and the timestamp, because SubstrateML never -reads a clock. +This file gives the canonical metric names for the package's +telemetry layer. It defines five string constants. The five +graph-analytic algorithms use them as signal names, one per call, +when each reports completion. + +Telemetry is optional. It feeds the Topology visualization, a live +picture of the memory graph. Each algorithm emits exactly one metric +per invocation through `IntellectusLib`. It emits only when +monitoring is enabled. When monitoring is off, the default, the emit +costs one atomic boolean load and a branch. No clock read happens. No +allocation happens. Keeping the names in one file prevents a typo. A +typo could otherwise produce an orphaned metric that no dashboard +ever finds. + +The five constants are `communityAssignment`, `centralityScore`, +`nmfFactor`, `anomalyFlag`, and `edgeDecayedWeight`. +`communityAssignment` is `"community.assignment"`. Its value is the +community count. `centralityScore` is `"centrality.score"`. Its +value is always 1.0, a completion marker. `nmfFactor` is +`"nmf.factor"`. Its value is the final reconstruction error. +`anomalyFlag` is `"anomaly.flag"`. Its value is the absolute z-score. +`edgeDecayedWeight` is `"edge.decayed_weight"`. Its value is the +applied decay factor. The file also records the caller contract. +Callers supply the estate tag and the timestamp, because SubstrateML +never reads a clock. ## FloatSimHash.swift -This file provides `FloatSimHash`, which projects a dense float embedding -vector — for example a 384-dimension MiniLM or 768-dimension BERT vector -from an external model — into the substrate's canonical 256-bit -fingerprint type, `Fingerprint256`. - -The substrate compares memories by Hamming distance, the number of -differing bit positions between two equal-length codes. External -embeddings are floats, not bits, so they cannot join that comparison -directly. `FloatSimHash` bridges the gap with signed random-hyperplane -projection, the classic SimHash construction: generate 256 pseudo-random -"+1 or -1" direction vectors (hyperplanes) matching the input dimension, -take the sign of each hyperplane's dot product with the input, and pack -the 256 sign bits into four 64-bit blocks. Vectors that point the same way -land on the same side of most hyperplanes, so cosine similarity in float -space is approximately preserved as Hamming closeness in bit space. - -`project(vector:seed:)` is the entry point. The hyperplanes come from a -SplitMix64 generator seeded by the caller, drawn in a pinned order (256 -planes outer, coordinates inner) with a pinned sign convention (low bit 1 -means +1). The same seed always yields the same planes, which matters -because stored fingerprints must stay comparable to freshly computed ones. -Different embedding providers use different seeds, so each provider's -fingerprints live in their own namespace. The dot-product work itself is -dispatched to a `SubstrateKernel` backend, selected once per process, so a -future SIMD kernel can drop in without touching this file. -`planes(seed:dim:)` materializes the hyperplane set for callers, such as -the kernel-equivalence conformance test, that need it directly. +This file gives `FloatSimHash`. It projects a dense float +embedding vector into the substrate's canonical fingerprint type, +`Fingerprint256`. A typical vector comes from an external model. +Examples include a 384-dimension MiniLM vector or a 768-dimension +BERT vector. + +The substrate compares memories by Hamming distance. Hamming +distance counts the differing bit positions between two +equal-length codes. External embeddings are floats, not bits. They +cannot join that comparison directly. `FloatSimHash` bridges the gap +with signed random-hyperplane projection, the classic SimHash +construction. The method generates 256 pseudo-random direction +vectors, each plus one or minus one, matching the input dimension. +It takes the sign of each hyperplane's dot product with the input. +It packs the 256 sign bits into four 64-bit blocks. Vectors that +point the same way land on the same side of most hyperplanes. Cosine +similarity in float space is thus approximately preserved as +Hamming closeness in bit space. + +`project(vector:seed:)` is the entry point. The hyperplanes come +from a SplitMix64 generator seeded by the caller. They are drawn in +a pinned order: 256 planes outer, coordinates inner. They follow a +pinned sign convention, where a low bit of one means plus one. The +same seed always yields the same planes. This matters because stored +fingerprints must stay comparable to freshly computed ones. Different +embedding providers use different seeds. Each provider's fingerprints +thus live in their own namespace. The dot-product work itself is +dispatched to a `SubstrateKernel` backend, selected once per process. +A future SIMD kernel can drop in without touching this file. +`planes(seed:dim:)` materializes the hyperplane set directly. Some +callers, such as the kernel-equivalence conformance test, need it in +that raw form. ## CompositeDistance.swift -This file provides `CompositeDistance.distance`, the substrate's primary -retrieval metric: one number in the range zero to one that says how far -apart two memory rows are. - -The score is a weighted blend of two normalized parts: -`alphaLattice * latticeDistance + alphaFingerprint * (hamming / 256)`. -The lattice part measures topical distance in the classification -hierarchy. The fingerprint part is Hamming distance divided by the fixed -256-bit width. Both weights default to 0.5, but the defaults are only the -starting point; the weights are learned per user over time through -Bradley-Terry feedback (see `BradleyTerry.swift`). - -One edge case is engineered, not accidental. Two fingerprints are only -comparable when they were computed under the same hyperplane family — the -`compatibleSeedScope` argument. That always holds inside one estate, but -across estates it requires a completed pairing handshake. When the scopes -are not compatible, the function keeps only the lattice term and does not -rescale it. The resulting smaller distance is the documented intended -behavior. Inputs are checked with preconditions rather than clamped, so an -out-of-range value fails loudly at the source. +This file gives `CompositeDistance.distance`, the substrate's +primary retrieval metric. It returns one number between zero and +one. That number says how far apart two memory rows sit. + +The score blends two normalized parts: +`alphaLattice * latticeDistance + alphaFingerprint * (hamming / +256)`. The lattice part shows topical distance in the +classification hierarchy. The fingerprint part is Hamming distance +divided by the fixed 256-bit width. Both weights default to 0.5. +Those defaults are only a starting point. The weights are learned +per user over time, through Bradley-Terry feedback (see +`BradleyTerry.swift`). + +One edge case is engineered on purpose. Two fingerprints are only +comparable when they come from the same hyperplane family. That +family match is the `compatibleSeedScope` argument. It always holds +inside one estate. Across estates it requires a completed pairing +handshake. When the scopes are not compatible, the function keeps +only the lattice term. It does not rescale that term. The resulting +smaller distance is the documented, intended behavior. Inputs are +checked with preconditions rather than clamped. An out-of-range value +thus fails loudly at the source. ## LatticeDistance.swift -This file provides the two conceptual distance axes that feed the -composite metric: tree distance in the UDC classification hierarchy and -graph distance in the Wikidata concept graph. - -`LatticeAnchorStr` pairs a UDC code string with an optional Wikidata Q-ID -(zero means none). `UDCTreeDistance.distance(_:_:)` compares two code -strings by their longest common prefix: the more leading characters two -codes share, the closer their subjects sit in the hierarchy. The raw -formula can reach two, so the result is clamped to the zero-to-one range -the composite metric requires. `WikidataGraphDistance` runs a -breadth-first search over a caller-injected `WikidataAdjacencyProvider` -(the "is-a" and "part-of" edges), bounded at depth 4, and normalizes the -path length with `1 - exp(-length / 3)`. Unreachable concepts, and the -null Q-ID zero, score the maximum 1.0. The graph is directed as given, so -symmetry holds only if the provider supplies reverse edges. - -`LatticeDistance.distance(_:_:provider:alphaUDC:alphaQID:)` blends the two -axes with weights that default to 0.5 each. A legacy overload also accepts -the older integer-hashed anchor type; hashing destroys prefix structure, -so that overload can only answer zero (identical) or one (different) and -is kept purely for compatibility with earlier call sites. +This file gives the two conceptual distance axes that feed the +composite metric. One axis is tree distance in the UDC +classification hierarchy. The other is graph distance in the +Wikidata concept graph. + +`LatticeAnchorStr` pairs a UDC code string with an optional Wikidata +Q-ID, where zero means none. `UDCTreeDistance.distance(_:_:)` +compares two code strings by their longest common prefix. The more +leading characters two codes share, the closer their subjects sit in +the hierarchy. The raw formula can reach two, so the result is +clamped to the zero-to-one range the composite metric needs. +`WikidataGraphDistance` runs a breadth-first search over a +caller-injected `WikidataAdjacencyProvider`. That provider supplies +the "is-a" and "part-of" edges. The search is bounded at depth four. +It normalizes the path length with `1 - exp(-length / 3)`. +Unreachable concepts score the maximum, 1.0. The null Q-ID, which is +zero, also scores 1.0. The graph is directed as given. Symmetry holds +only if the provider supplies reverse edges. + +`LatticeDistance.distance(_:_:provider:alphaUDC:alphaQID:)` blends +the two axes. Each weight defaults to 0.5. A legacy overload also +accepts the older integer-hashed anchor type. Hashing destroys prefix +structure. That overload can only answer zero for a match or one +for a difference. It survives purely for compatibility with earlier +call sites. ## PartialStateRecall.swift -This file provides the "same in one way, different in another" recall -primitive. A row fingerprint has four 64-bit blocks, and different blocks -encode different aspects of a memory. `PartialStateRecall` scores a -candidate row by how well it matches an anchor on one chosen block subset -while differing from it on another — for example, same topic but different -behavioral state. - -`score(rowFingerprint:anchor:matchBlocks:differBlocks:)` computes a -restricted Hamming distance over each block subset. The match score is one -minus the normalized distance on the match blocks; the differ score is the -normalized distance on the differ blocks. The final score is their -product, so a candidate must satisfy both constraints at once: a row -identical to the anchor everywhere scores zero (it fails to differ), and a -row unrelated on the match blocks also scores zero. Block identifiers must -come from the set {0, 1, 2, 3}; a precondition enforces this because an -out-of-range block would silently corrupt the denominator and produce a -subtly wrong score instead of a crash. `topK(anchor:rows:...)` scores every -row in a linear scan and returns the best k — the reference form of the -`recall_partial_match` query. `hammingBlocks(_:_:blocks:)` exposes the -block-restricted Hamming count itself. +This file gives a recall primitive for rows that match in one way +and differ in another. A row fingerprint has four 64-bit blocks. +Different blocks encode different aspects of a memory. +`PartialStateRecall` scores a candidate row against an anchor. It +checks one chosen block subset for a match. It checks another subset +for a difference. A real example pairs the same topic with a +different behavioral state. + +`score(rowFingerprint:anchor:matchBlocks:differBlocks:)` finds a +restricted Hamming distance over each block subset. The match score +is one minus the normalized distance on the match blocks. The differ +score is the normalized distance on the differ blocks. The final +score is their product. A candidate must satisfy both constraints at +once. A row that matches the anchor everywhere scores zero, since it +fails to differ. A row unrelated on the match blocks also scores +zero. Block identifiers must come from the set `{0, 1, 2, 3}`. A +precondition enforces this rule. An out-of-range block would +silently corrupt the denominator and produce a subtly wrong score +instead of a crash. `topK(anchor:rows:...)` scores each row in a +linear scan. It returns the best k rows. This is the reference form +of the `recall_partial_match` query. `hammingBlocks(_:_:blocks:)` +exposes the block-restricted Hamming count on its own. ## ShingleSimilarity.swift -This file provides the substrate's one character-shingle Jaccard -similarity: how alike two strings are, as the overlap of their -three-character substring sets. - -Two consuming kits needed exactly this function for recall de-duplication, -and neither may depend on the other. Following the "one implementation per -substrate atomic" rule, the shared math lives here, below both. -`shingles(_:)` lowercases the input and slides a three-character window -across it, collecting each substring into a set; strings shorter than the -window collapse to a single whole-string shingle, and the empty string -yields the empty set. `similarity(_:_:)` returns intersection size divided -by union size, with "both empty" defined as 0.0 rather than undefined. No -stemming, tokenization, or locale-sensitive transform is applied, which is -what keeps Swift and Rust byte-identical on the content the recall path -actually compares. The window size 3 is pinned: long enough to -discriminate near-duplicates, short enough to survive paraphrase. +This file gives the substrate's one character-shingle Jaccard +similarity. It shows how alike two strings are. The measure is +the overlap of their three-character substring sets. + +Two consuming kits each needed this exact function for recall +de-duplication. Neither kit may depend on the other. The shared math +thus lives here, in a layer below both, following the "one +implementation per substrate atomic" rule. `shingles(_:)` lowercases +the input. It slides a three-character window across the string. It +collects each substring into a set. Strings shorter than the window +collapse to one whole-string shingle. The empty string yields the +empty set. `similarity(_:_:)` returns the intersection size divided +by the union size. The case of both strings empty is defined as +0.0, a plain rule rather than an undefined case. No stemming +happens. No tokenization happens. No locale-sensitive transform +happens either. This restraint keeps Swift and Rust byte-identical +on the content the recall path actually compares. The window size, +three, is pinned. It is long enough to discriminate near-duplicates. +It is short enough to survive paraphrase. ## MomentSummary.swift -This file provides the moment summary: one 256-bit fingerprint that -stands for everything active during a time window, built by OR-reducing -the fingerprints of the matching rows. - -OR-reduction sets a bit in the output when any input has it set. The -choice of OR gives the summary useful algebra for free: it is -order-independent, idempotent, and monotonic — more rows can only add -bits, never clear them. An empty match yields the all-zero fingerprint. -Recall can then find similar past moments by Hamming search over cached -window summaries. - -What counts as "active during" the window is deliberately not hardcoded. -The cookbook defines three legitimate meanings, so both `summarize` -overloads take the predicate as a caller-supplied closure. One overload -works on full `Row` values; the other works on `RowLite`, a 40-byte -fingerprint-plus-timestamp pair, and runs four to five times faster on -large windows purely because the loop touches less memory. Both paths are -verified byte-identical against a conformance vector (CRC `0x6762440b`). -`capturedDuring(_:_:)` is the common convenience predicate, and -`orReduce(_:)` exposes the raw reduction for callers that already hold a -fingerprint list. +This file gives the moment summary. It is one 256-bit fingerprint +that stands for everything active during a time window. The system +builds it by OR-reducing the fingerprints of the matching rows. + +OR-reduction sets a bit in the output whenever any input has that +bit set. This choice gives the summary useful algebra for free. The +result is order-independent. It is idempotent. It is monotonic: more +rows can only add bits, never clear them. An empty match yields the +all-zero fingerprint. Recall can then find similar past moments by +Hamming search over cached window summaries. + +What counts as "active during" a window is left open on purpose. The +cookbook defines three legitimate meanings. Both `summarize` +overloads thus take the predicate as a caller-supplied closure. +One overload works on full `Row` values. The other works on +`RowLite`, a 40-byte fingerprint-plus-timestamp pair. The `RowLite` +path runs four to five times faster on large windows, purely +because its loop touches less memory. Both paths are verified +byte-identical against a conformance vector, CRC `0x6762440b`. +`capturedDuring(_:_:)` is the common convenience predicate. +`orReduce(_:)` exposes the raw reduction for callers that already +hold a fingerprint list. ## TemporalCompression.swift -This file provides hierarchical time roll-up: hour summaries roll into -days, days into weeks, and onward through months and quarters to years. -Each level is a `TemporalWindow` — a summary fingerprint, a row count, and -start and end timestamps. - -Because the summary operator is OR-reduction, rolling up is associative -and commutative: the result does not depend on the order windows arrive. -`compress(rows:startHLC:endHLC:level:)` builds a base window from raw row -fingerprints. `rollup(windows:to:)` merges finer windows into one coarser -window, summing row counts with overflow-safe addition and taking the -outermost time bounds. `cascadeRollup(hourWindows:upTo:)` automates the -whole ladder: it buckets the current level's windows by which coarser slot -they fall into (integer division of epoch seconds by the coarser level's -approximate duration) and rolls each bucket, one level at a time. - -The `WindowLevel` durations are deliberately approximate — a month is -thirty days, a year 365 — because they serve only as bucket divisors, not -calendar math. Storage and scheduling belong to the caller (the dreaming -daemon); this file is pure computation and never reads a clock. +This file gives hierarchical time roll-up. Hour summaries roll +into days. Days roll into weeks. Weeks roll onward through months and +quarters into years. Each level is a `TemporalWindow`: a summary +fingerprint, a row count, and start and end timestamps. + +The summary operator is OR-reduction, so rolling up is associative +and commutative. The result does not depend on the order windows +arrive in. `compress(rows:startHLC:endHLC:level:)` builds a base +window from raw row fingerprints. `rollup(windows:to:)` merges finer +windows into one coarser window. It sums row counts with +overflow-safe addition. It takes the outermost time bounds. +`cascadeRollup(hourWindows:upTo:)` automates the whole ladder. It +buckets the current level's windows by which coarser slot they fall +into. That bucketing is integer division of epoch seconds by the +coarser level's approximate duration. It then rolls each bucket, one +level at a time. + +The `WindowLevel` durations are approximate, on purpose. A month +counts as thirty days. A year counts as 365. These numbers serve only +as bucket divisors, never as calendar math. Storage and scheduling +belong to the caller, the dreaming daemon. This file is pure +computation. It never reads a clock. ## FeatureExtractors.swift -This file provides the ingestion boundary for ambient signals: five -encoder types that turn raw OS sensor samples into `AmbientSampleRow` -values — fingerprinted, classification-anchored rows ready for storage -and recall. - -Each extractor covers one stream: `HealthKitExtractor` (biometrics), -`CoreLocationExtractor` (position), `EventKitExtractor` (calendar), -`ScreenTimeExtractor` (app usage), and `SystemTelemetryExtractor` (system -load). The OS-specific capture code lives in the host apps; only the pure -encoding lives here. Each `extract` call computes four 64-bit subhashes -from the sample's salient fields, then feeds them with a hyperplane family -into the substrate's SimHash to produce the 256-bit fingerprint. Numeric -fields are quantized to integers before hashing (for example, coordinates -times one million) so floating-point representation can never change a -fingerprint. Attendee lists are sorted before hashing so the same meeting -fingerprints identically regardless of listing order. Each stream carries -a fixed UDC lattice code (for example "613.71" for health, "914" for -location) as its coarse topical anchor. - -Only HealthKit rows carry a `payload` — a small, key-sorted, deterministic -binary encoding for human-readable recall detail. The other streams emit -no payload by design: the fingerprint already carries all recall-relevant -signal, and less stored raw data means less to protect. The FNV-1a hash -constants serve as the file's general 64-bit mixers. +This file gives the ingestion boundary for ambient signals. Five +encoder types turn raw OS sensor samples into `AmbientSampleRow` +values. Each value is a fingerprinted, classification-anchored row +ready for storage and recall. + +Each extractor covers one stream. `HealthKitExtractor` covers +biometrics. `CoreLocationExtractor` covers position. +`EventKitExtractor` covers calendar events. `ScreenTimeExtractor` +covers app usage. `SystemTelemetryExtractor` covers system load. The +OS-specific capture code lives in the host apps. Only the pure +encoding lives here. Each `extract` call finds four 64-bit +subhashes from the sample's salient fields. It feeds them, with a +hyperplane family, into the substrate's SimHash. That step produces +the 256-bit fingerprint. Numeric fields are quantized to integers +before hashing. Coordinates, for example, are multiplied by one +million first. This step means floating-point representation can +never change a fingerprint. Attendee lists are sorted before hashing. +The same meeting thus fingerprints identically, regardless of +listing order. Each stream carries a fixed UDC lattice code as its +coarse topical anchor. Health uses `"613.71"`. Location uses +`"914"`. + +Only HealthKit rows carry a `payload`. It is a small, key-sorted, +deterministic binary encoding for human-readable recall detail. The +other streams emit no payload, by design. The fingerprint already +carries all recall-relevant signal. Storing less raw data also means +less to protect. The FNV-1a hash constants serve as the file's +general 64-bit mixers. ## AuditLogFold.swift -This file provides `AuditLogFold`, which reconstructs what a memory row -looked like at any point in time by replaying its audit log. +This file gives `AuditLogFold`. It reconstructs what a memory row +looked like at any point in time, by replaying its audit log. -The audit log is a grow-only set (a G-Set, a conflict-free replicated data -type): events are only ever added, never edited. Current state is defined -as the fold — the sequential replay — of a row's events sorted by their -hybrid logical clock (HLC) timestamps, and "state as of time T" is the -same fold truncated at T. Because the fold depends only on the set of -events, not their arrival order, two devices that have exchanged the same -events converge to identical state. That property is what makes sync +The audit log is a grow-only set, called a G-Set in CRDT terms. Its +events are only ever added, never edited. Current state is the fold, +the sequential replay, of a row's events sorted by their HLC +timestamps. State as of time T is that same fold, truncated at T. +The fold depends only on the set of events, never on their arrival +order. Two devices that have exchanged the same events thus +converge to the same state. That property is what makes sync correct without coordination. -`projectCurrentState(rowId:nounType:events:)` folds a full history; -`projectStateAt(...)` folds up to a cutoff; `projectAll(events:asOf:...)` -groups a whole log by row and folds each. The result type, -`ProjectedRowState`, carries the row's bitmaps, lattice anchor, tombstone -flag, and last-event time. One rule is deliberately sticky: once the -tombstone marker (state raw value 33) appears, the row stays tombstoned -even if a later, illegal event tries to revive it. The fingerprint is -intentionally absent from the projection; callers recompute it from the -bitmaps and anchor when needed. +`projectCurrentState(rowId:nounType:events:)` folds a full history. +`projectStateAt(...)` folds up to a cutoff. +`projectAll(events:asOf:...)` groups a whole log by row and folds +each one. The result type, `ProjectedRowState`, carries the row's +bitmaps, the lattice anchor, the tombstone flag, and the last-event +time. One rule is sticky, on purpose. Once the tombstone marker, +state raw value 33, appears, the row stays tombstoned. It stays +tombstoned even if a later, illegal event tries to revive it. The +fingerprint is intentionally absent from the projection. Callers +recompute it from the bitmaps and anchor when needed. ## RowAttributeView.swift -This file provides the bridge from the audit log to the pattern miners: -`RowAttributeView`, one row's field writes flattened into sorted -`(field, value)` byte pairs — the exact shape Apriori and Formal Concept -Analysis consume. - -`from(auditEntries:)` runs a four-step pipeline. It first builds a -vocabulary of all distinct field paths in the batch, sorted -alphabetically and capped at 64 entries, because the miners' item type -stores the field index in six bits. It then groups entries by row, keeps -only the latest write per field by HLC order (the same last-write-wins -rule the audit fold uses, so the two subsystems never disagree), and -expands each surviving value: a bitmap value becomes one attribute per set -bit, an integer contributes its low byte, and a null contributes nothing. -Rows left with no attributes are dropped. Attributes are sorted within a -view and views are sorted by row identity, so the output is deterministic. - -One subtlety is documented rather than hidden: the vocabulary is ephemeral -to a single call. Two separate calls can assign different field indexes, -so callers who need a stable vocabulary must merge their entry lists -before calling. The input types `RowAuditEntry` and `RowAuditValue` are -defined here, in a lower layer than the kit that owns the real audit -types, precisely so the dependency arrow keeps pointing downward. +This file gives the bridge from the audit log to the pattern +miners. That bridge is `RowAttributeView`: one row's field writes, +flattened into sorted field-and-value byte pairs. This is the exact +shape Apriori and Formal Concept Analysis consume. + +`from(auditEntries:)` runs a four-step pipeline. First, it builds a +vocabulary of all distinct field paths in the batch. That vocabulary +is sorted alphabetically and capped at 64 entries, because the +miners' item type stores the field index in six bits. Second, it +groups entries by row. It keeps only the latest write per field, by +HLC order. That rule is the same last-write-wins rule the audit fold +uses, so the two subsystems never disagree. Third, it expands each +surviving value. A bitmap value becomes one attribute per set bit. An +integer contributes its low byte. A null contributes nothing. Rows +left with no attributes are dropped. Attributes are sorted within a +view. Views are sorted by row identity. The output is thus +deterministic. + +One subtlety is documented in the open. The vocabulary is ephemeral +to a single call. Two separate calls can assign different field +indexes. Callers who need a stable vocabulary must merge their entry +lists before calling. The input types `RowAuditEntry` and +`RowAuditValue` are defined here, in a lower layer than the kit that +owns the real audit types. That placement keeps the dependency arrow +pointing downward. ## MatrixDecay.swift -This file provides the substrate's forgetting mechanism: exponential -half-life decay for every matrix in the matrix tier — the accumulated -statistics tables such as field presence, correlation, co-activation, and -action outcomes. - -The rule is constitutional: decay may only shrink values. New evidence -enters through separate write operations, never through this file. -`DecayingMatrix` is a plain row-major matrix that carries its own -half-life and its last-decay timestamp. `apply(to:nowSeconds:estate:ts:)` -computes the elapsed time, derives one shared factor -`exp(-elapsed * ln2 / halfLife)`, and multiplies every cell by it. A -non-positive elapsed time is a documented no-op — decay never runs -backward — but it still emits its telemetry metric with factor 1.0, so a -watcher can tell "daemon ran, nothing to do" from "daemon never ran." -Using one scalar factor per pass (rather than per-cell clocks) is what -makes decay commute with addition, a property the file states formally. - -`decayFactor(elapsedSeconds:halfLifeSeconds:)` is the pure projection -helper, and `decayAndAdd(...)` is the decay-then-add step the online write -path uses. `DecayHalfLives` records the per-matrix schedule from the -cookbook: 90 days for field presence, 180 for correlation, 60 for -co-activation, 30 for temporal causality, 365 for action outcomes, and -730 for calibration. Two details protect determinism and safety: `ln2` is -a local constant chosen to match Rust's value bit-for-bit, and the -elapsed-seconds cast for the telemetry tag saturates instead of trapping -on absurdly large values. Three `applyExponentialDecay` overloads for the -typed matrices are deliberate no-op adapters at this reference level; the -production per-matrix routine lives in the consuming kit. +This file gives the substrate's forgetting mechanism. It applies +exponential half-life decay to each matrix in the matrix tier. +Those matrices are the accumulated statistics tables: field +presence, correlation, co-activation, and action outcomes. + +The rule is constitutional. Decay may only shrink values. New +evidence enters through separate write operations. It never enters +through this file. `DecayingMatrix` is a plain row-major matrix. It +carries its own half-life and its last-decay timestamp. +`apply(to:nowSeconds:estate:ts:)` finds the elapsed time. It +derives one shared factor, `exp(-elapsed * ln2 / halfLife)`. It +multiplies each cell by that factor. A non-positive elapsed time is +a documented no-op, since decay never runs backward. The function +still emits its telemetry metric with factor 1.0 in that case. A +watcher can then use the emitted factor to tell two states apart. +Either the daemon ran and found nothing to decay. Or the daemon +never ran at all. Using one scalar factor per pass, rather than +per-cell clocks, is what makes decay commute with addition. The file +states +that property formally. + +`decayFactor(elapsedSeconds:halfLifeSeconds:)` is the pure +projection helper. `decayAndAdd(...)` is the decay-then-add step the +online write path uses. `DecayHalfLives` records the per-matrix +schedule from the cookbook. Field presence gets 90 days. Correlation +gets 180. Co-activation gets 60. Temporal causality gets 30. Action +outcomes get 365. Calibration gets 730. Two details protect +determinism and safety. `ln2` is a local constant, chosen to match +Rust's value bit-for-bit. The elapsed-seconds cast for the telemetry +tag saturates instead of trapping on absurdly large values. Three +`applyExponentialDecay` overloads for the typed matrices are +deliberate no-op adapters at this reference level. The production +per-matrix routine lives in the consuming kit. ## ActionOutcomeMatrix.swift -This file provides the action-outcome matrix: the substrate's -reinforcement record of which actions succeed. Each cell, keyed by a -six-bit action kind and a six-bit outcome category, accumulates a success -count, a total count, and the last-update HLC. - -`ActionOutcomeKey` enforces the six-bit ranges with preconditions and -packs both bytes into one `UInt16` ordering key. `observe(...)` increments -a cell with overflow-safe arithmetic. `successRate(...)` and -`observationCount(...)` read cells back. The interesting method is -`topActions(forOutcome:k:minObservations:)`: it ranks actions not by raw -success rate but by the Wilson lower bound — the bottom of a 95 percent -confidence interval around the rate. A cell with two observations and a -perfect record should not outrank a cell with two hundred observations and -a 95 percent record; the Wilson bound encodes exactly that caution. The -method returns rate, bound, and count together so callers see the value -the ranking actually used. Ties break by total count, then by ascending -action, keeping output deterministic. Decay for this matrix (365-day -half-life) is applied lazily by the dreaming daemon through -`MatrixDecay`, not here. +This file gives the action-outcome matrix. It is the substrate's +reinforcement record of which actions succeed. Each cell is keyed by +a six-bit action kind and a six-bit outcome category. Each cell +accumulates a success count, a total count, and the last-update HLC. + +`ActionOutcomeKey` enforces the six-bit ranges with preconditions. +It packs both bytes into one `UInt16` ordering key. `observe(...)` +increments a cell with overflow-safe arithmetic. +`successRate(...)` and `observationCount(...)` read cells back. The +most interesting method is `topActions(forOutcome:k:minObservations:)`. +It ranks actions by the Wilson lower bound, rather than by the raw +success rate. The Wilson bound is the bottom of a 95 percent +confidence interval around the rate. A cell with two observations +and a perfect record should score below a cell with two hundred +observations and a 95 percent record. The Wilson bound encodes +exactly that caution. The +method returns the rate, the bound, and the count together. Callers +can then see the value the ranking actually used. Ties break by +total count first, then by ascending action. Output stays +deterministic. Decay for this matrix runs on a 365-day half-life. +The dreaming daemon applies it lazily through `MatrixDecay`, not +here. ## BradleyTerry.swift -This file provides an online Bradley-Terry estimator: it learns a -per-row "strength" score from pairwise preference feedback, and that -strength reorders future recall candidates. - -Every time recall surfaces a candidate set and the user or agent picks -one, that pick is an observation: the winner beat the losers. The -Bradley-Terry model says the probability that row i beats row j is -`w_i / (w_i + w_j)`. The implementation works in log-space (`theta = log -w`), so the win probability becomes a sigmoid of the theta difference and -gradient updates never have to guard positivity. `observe(_:)` performs -one stochastic gradient step per observation with L2 regularization -pulling theta toward zero, which keeps early sparse data from overfitting. -For multi-loser observations the winner's theta is updated sequentially -against each loser, recomputing against the running value — a pinned -ordering that the Rust port mirrors exactly. +This file gives an online Bradley-Terry estimator. It learns a +per-row strength score from pairwise preference feedback. That +strength then reorders future recall candidates. + +Each time recall surfaces a candidate set, the user or agent picks +one. That pick becomes an observation: the winner beat the losers. +The Bradley-Terry model says the probability that row i +beats row j equals `w_i / (w_i + w_j)`. The implementation works in +log-space, where `theta = log w`. The win probability then becomes a +sigmoid of the theta difference. Gradient updates never need to +guard positivity. `observe(_:)` performs one stochastic gradient step +per observation. It applies L2 regularization, pulling theta toward +zero. This pull keeps early sparse data from overfitting. For +multi-loser observations, the winner's theta updates sequentially +against each loser. Each update recomputes against the running +value. This is a pinned order, and the Rust port mirrors it +exactly. `observeBatch(_:)` applies a day's worth of observations in order. -`strength(of:)` and `probability(_:beats:)` read the model. There is no -randomness anywhere: the same observation sequence always produces -bit-identical theta, and a serialized theta dictionary warm-restarts the -estimator exactly. The defaults (learning rate 0.05, L2 0.001) are tuned -for the ten-to-one-hundred observations per day a personal estate -produces. Strength is a ranking signal only; it never changes what is -stored or filters what exists. +`strength(of:)` and `probability(_:beats:)` read the model. There is +no randomness anywhere. The same observation sequence always +produces bit-identical theta. A serialized theta dictionary +warm-restarts the estimator exactly. The defaults are a learning +rate of 0.05 and an L2 of 0.001. These values are tuned for the +ten-to-one-hundred observations per day a personal estate produces. +Strength is a +ranking signal only. It never changes what is stored. It never +filters what exists. ## LLMCalibrationCurve.swift -This file provides a twenty-bucket calibration histogram that tracks how -well an LLM's stated confidence matches reality. +This file gives a twenty-bucket calibration histogram. It tracks +how well a model's stated confidence matches reality. -When the cognition tier resolves a probabilistic claim — the model said -"80 percent sure" and the claim turned out true or false — -`observe(claimedConfidence:actualOutcome:)` clamps the confidence into -the range zero to just under one, maps it to one of twenty 0.05-wide -buckets, and bumps that bucket's predicted and hit counters. Counters use -wrapping arithmetic on purpose: this is a long-lived accumulator, and -overflow should wrap rather than crash. +The cognition tier sometimes resolves a probabilistic claim. The +model might say it is 80 percent sure, and the claim later turns out +true or false. `observe(claimedConfidence:actualOutcome:)` clamps the +confidence into the range zero to just under one. It maps that value +to one of twenty buckets, each 0.05 wide. It bumps that bucket's +predicted and hit counters. The counters use wrapping arithmetic on +purpose. This is a long-lived accumulator, so overflow should wrap +rather than crash. `actualRate(in:)` reads one bucket's observed hit rate. -`expectedCalibrationError()` measures the weighted average gap between -each bucket's observed rate and its midpoint — the "perfectly calibrated" -diagonal. `brierScore()` is the same with squared gaps. `decay(factor:)` -scales every counter by a fraction so stale calibration evidence fades -smoothly; the dreaming daemon drives it on the 730-day half-life schedule. -The result lets recall annotate model-claimed confidence with how much -that model's confidence has historically been worth. +`expectedCalibrationError()` shows the weighted average gap +between each bucket's observed rate and its midpoint, the perfectly +calibrated diagonal. `brierScore()` does the same with squared gaps. +`decay(factor:)` scales each counter by a fraction, so stale +calibration evidence fades smoothly. The dreaming daemon drives this +decay on a 730-day half-life schedule. The result lets recall +annotate a model's claimed confidence. It shows how much that +confidence has historically been worth. ## Sampling.swift -This file provides the three deterministic distribution samplers — -Normal, Gamma, and Beta — that sit under Thompson sampling. Thompson -sampling is a decision strategy that picks among options by drawing from a -belief distribution over each option's payoff; the dreaming daemon uses it -to choose maintenance strategies. The policy lives in the consuming kit; -the sampling math is centralized here so every consumer shares one -implementation. - -All three samplers take the caller's SplitMix64 generator by `inout` -reference, so the random stream advances visibly and reproducibly. -`sampleNormal(rng:)` uses the Box-Muller transform and deliberately keeps -only the cosine branch, consuming exactly two uniform draws per call — -caching the sine branch would desynchronize the stream between ports. -`sampleGamma(shape:rng:)` implements Marsaglia-Tsang rejection sampling -for shape at least one, with the standard squeeze constant 0.0331, and -reduces smaller shapes through the Ahrens-Dieter identity; the reduction's -extra uniform draw happens before the recursive call, an order pinned for -cross-port stream identity. `sampleBeta(alpha:beta:rng:)` derives Beta -from the ratio of two Gamma draws, alpha first. Given the same seed and -inputs, both legs consume the same words in the same order and return the -same values, gated by a shared conformance vector. +This file gives three deterministic distribution samplers: +Normal, Gamma, and Beta. They sit under Thompson sampling. Thompson +sampling is a decision strategy. It picks among options by drawing +from a belief distribution over each option's payoff. The dreaming +daemon uses it to choose maintenance strategies. The policy lives in +the consuming kit. The sampling math is centralized here, so each +consumer shares one implementation. + +All three samplers take the caller's SplitMix64 generator by +`inout` reference. The random stream thus advances visibly and +reproducibly. `sampleNormal(rng:)` uses the Box-Muller transform. It +keeps only the cosine branch, on purpose, consuming exactly two +uniform draws per call. Caching the sine branch would desynchronize +the stream between ports. `sampleGamma(shape:rng:)` implements +Marsaglia-Tsang rejection sampling for a shape of at least one, with +the standard squeeze constant 0.0331. It reduces smaller shapes +through the Ahrens-Dieter identity. That reduction's extra uniform +draw happens before the recursive call. This order is pinned for +cross-port stream identity. `sampleBeta(alpha:beta:rng:)` derives +Beta from the ratio of two Gamma draws, with alpha drawn first. +Given the same seed and inputs, both legs consume the same words in +the same order. Both return the same values, gated by a shared +conformance vector. ## CommunityDetection.swift -This file provides Louvain community detection over the estate graph: it -labels every node with a community so related memories cluster together. -Auto-rooming, keystone recall, and the daily dreaming refresh all consume -the partition. - -Louvain works in two phases. Phase 1 repeatedly tries moving each node -into a neighboring community and keeps the move that most improves -modularity — a score that rewards putting well-connected nodes together -relative to chance. Phase 2 condenses each community into a supernode and -repeats on the smaller graph. `detect(...)` runs Phase 1 only; -`detectFull(...)` runs the full multi-level algorithm and is the normal -entry point. - -Two design choices deserve explanation. First, determinism: candidate -communities are evaluated in ascending label order, so score ties always -resolve to the lowest label instead of depending on hash-map iteration -order; a `canonicalize(_:)` step then renumbers labels by first -appearance, so Swift and Rust emit identical partitions. Second, the -resolution parameter: plain Louvain can get "pair-locked" on graphs with -strongly bonded pairs plus weak star edges, unable to escape a local -optimum. The Reichardt-Bornholdt gamma scales only the degree-penalty -term; small gamma (consumers use 0.05) forces pairs to merge into hub -communities, while the default 1.0 reproduces classical Louvain exactly — -IEEE arithmetic guarantees `1.0 * x == x`, so the refactor cannot perturb -legacy results. The implementation is intentionally the compact -O(N·E)-per-pass reference rather than the fastest possible variant. One -`community.assignment` telemetry signal fires per call when monitoring is -on; a degenerate all-zero-weight graph returns identity labels and emits -nothing. +This file gives Louvain community detection over the estate +graph. It labels each node with a community, so related memories +cluster together. Auto-rooming, keystone recall, and the daily +dreaming refresh all consume the resulting partition. + +Louvain works in two phases. Phase one repeatedly tries moving each +node into a neighboring community. It keeps whichever move most +improves modularity, a score that rewards putting well-connected +nodes together relative to chance. Phase two condenses each +community into a supernode. It then repeats phase one on the smaller +graph. `detect(...)` runs phase one only. `detectFull(...)` runs the +full multi-level algorithm and is the normal entry point. + +Two design choices deserve explanation. The first is determinism. +Candidate communities are evaluated in ascending label order, so +score ties always resolve to the lowest label. This avoids depending +on hash-map iteration order. A `canonicalize(_:)` step then renumbers +labels by first appearance, so Swift and Rust emit the same +partitions. The second choice is the resolution parameter. Plain +Louvain can get pair-locked on graphs that have strongly bonded pairs +plus weak star edges. It becomes unable to escape a local optimum. +The Reichardt-Bornholdt gamma scales only the degree-penalty term. A +small gamma, 0.05, forces pairs to merge into hub communities. The +default is 1.0. It reproduces classical Louvain exactly, since IEEE +arithmetic guarantees `1.0 * x == x`. The refactor thus cannot +perturb legacy results. The implementation is intentionally the +compact O(N·E)-per-pass reference, rather than the fastest possible +variant. One `community.assignment` telemetry signal fires per call, +when monitoring is on. A degenerate all-zero-weight graph returns +identity labels and emits nothing. ## EigenvalueCentrality.swift -This file provides eigenvalue centrality: a per-row authority score -computed by power iteration over the estate graph, cached as each row's -keystone score for `recall_keystone`. +This file gives eigenvalue centrality. It is a per-row authority +score, computed by power iteration over the estate graph. The score +is cached as each row's keystone score for `recall_keystone`. Power iteration repeatedly multiplies a score vector by the graph's -adjacency and renormalizes; the vector converges to the principal -eigenvector, where a node scores highly when high-scoring nodes connect to -it. The direction matters and is a locked design decision: `compute(...)` -multiplies by the transpose, accumulating at node j the influence of nodes -that point TO j. That is authority ("many rows reference this one"), not -hub ("this row references many"), and it is the right meaning for -cognitive keystones. A conformance vector pins the directed behavior so it -cannot drift back. For undirected centrality, callers symmetrize the -adjacency first. - -Each iteration adds a Perron shift (`xNext += 1.0 * x`), which shifts -every eigenvalue without changing eigenvectors and thereby breaks the -plus-minus oscillation that bipartite graphs such as stars exhibit under -raw power iteration. Iteration stops at convergence (L2 difference below -1e-6) or at the 100-iteration safety cap. A zero-norm vector falls back to -the uniform distribution — an edgeless graph makes every row equally -non-central. All three return paths emit the single `centrality.score` -telemetry signal, tagged with the node count and iterations used. +adjacency, then renormalizes it. The vector converges to the +principal eigenvector. A node scores highly there when high-scoring +nodes connect to it. Direction matters, and it is a locked design +decision. `compute(...)` multiplies by the transpose. It +accumulates, at node j, the influence of nodes that point to node j. +That is authority: many rows reference this one. It is a different +meaning from a hub, where this row references many others. +Authority is the right meaning for cognitive keystones. A +conformance vector pins the directed behavior, so it cannot drift +back. For undirected centrality, callers symmetrize the adjacency +first. + +Each iteration adds a Perron shift, `xNext += 1.0 * x`. This shift +moves each eigenvalue without changing the eigenvectors. It breaks +the plus-minus oscillation that bipartite graphs, such as stars, +exhibit under raw power iteration. Iteration stops at convergence, +when the L2 difference falls below 1e-6, or at a 100-iteration +safety cap. A zero-norm vector falls back to the uniform +distribution: an edgeless graph makes each row equally non-central. +All three return paths emit a single `centrality.score` telemetry +signal, tagged with the node count and the iterations used. ## RandomWalks.swift -This file provides random-walk-with-restart over the memory graph — the -engine behind exploratory recall, which wanders outward from a starting -memory and reports where it keeps landing. - -`walk(adjacency:start:length:restartProb:seed:)` runs one walk over a -densely indexed weighted adjacency list. Each step either restarts at the -start node (probability 0.15 by default) or moves to a neighbor chosen by -roulette-wheel weighted sampling. A dead end — a node with no out-edges — -is an implicit restart rather than an error. The walk is seeded -explicitly: the same graph, start, length, and seed reproduce the same -walk on both legs, because the file carries its own public `SplitMix64` -and a `uniform01(_:)` that derives a double from the top 53 bits exactly -as Rust does. Malformed graphs (out-of-range neighbor, negative or -non-finite weight) fail hard preconditions immediately, because a bad -adjacency cannot represent probabilities and silence would corrupt -results downstream. - -`walkWithRestart(seed:steps:restartProbability:rngSeed:adjacency:)` is the -row-identifier variant: it walks a dictionary adjacency with uniform -neighbor choice and returns visit counts per row, the shape the -exploratory recall recipe consumes. `sampleWeighted(_:rng:)` exposes the -weighted sampler, including its pinned fallbacks (uniform choice when -total weight is zero; last neighbor on cumulative rounding shortfall). +This file gives random-walk-with-restart over the memory graph. +It is the engine behind exploratory recall. That recall wanders +outward from a starting memory and reports where it keeps landing. + +`walk(adjacency:start:length:restartProb:seed:)` runs one walk over +a densely indexed weighted adjacency list. Each step does one of two +things. It restarts at the start node, with probability 0.15 by +default. Or it moves to a neighbor, chosen by roulette-wheel weighted +sampling. A dead end, a node with no out-edges, counts as an implicit +restart rather than an +error. The walk is seeded explicitly. Four things pin it: the graph, +the start, the length, and the seed. The same four always reproduce +the same walk on both legs. The file carries its +own public `SplitMix64`. Its `uniform01(_:)` derives a double from +the top 53 bits, exactly as Rust does. Malformed graphs fail hard +preconditions immediately. An out-of-range neighbor fails this way. +A negative or non-finite weight fails this way too. A bad adjacency +cannot represent probabilities, and silence would corrupt results +downstream. + +`walkWithRestart(seed:steps:restartProbability:rngSeed:adjacency:)` +is the row-identifier variant. It walks a dictionary adjacency with +uniform neighbor choice. It returns visit counts per row, the shape +the exploratory recall recipe consumes. `sampleWeighted(_:rng:)` +exposes the weighted sampler. It includes pinned fallbacks: a +uniform choice when the total weight is zero, and the last neighbor +on a cumulative rounding shortfall. ## NMFAlternatingLeastSquares.swift -This file provides the canonical non-negative matrix factorization (NMF) -engine. NMF approximates a non-negative matrix V as the product of two -smaller non-negative matrices, V ≈ W × H. Rows of H are latent "themes"; -rows of W say how much each original row loads on each theme. The -substrate runs NMF over its field-presence and co-occurrence matrices to -surface themes for recall and for the Topology theme overlay; the -dreaming daemon reruns it monthly. +This file gives the canonical non-negative matrix factorization +engine, known as NMF. NMF approximates a non-negative matrix V as +the product of two smaller non-negative matrices, `V ≈ W × H`. Rows +of H are latent themes. Rows of W say how much each original row +loads on each theme. The substrate runs NMF over its field-presence +and co-occurrence matrices. This surfaces themes for recall and for +the Topology theme overlay. The dreaming daemon reruns it monthly. `factorize(V:rank:maxIterations:tolerance:seed:estate:ts:)` uses the -Lee-Seung multiplicative update rules, which keep every entry non-negative -automatically: each iteration rescales H by the ratio of `WᵀV` to `WᵀWH`, -then W by the ratio of `VHᵀ` to `WHHᵀ`, with epsilon 1e-9 in denominators -against division by zero. W and H start as uniform random values from a -SplitMix64 seeded by the caller (default `0xDEADBEEFCAFEBABE`), so -initialization is bit-identical everywhere. Iteration stops when the -root-mean-square reconstruction error stops changing by more than the -tolerance, or at the cap. Entry preconditions enforce a rectangular, -finite, non-negative V, because the Lee-Seung theorem is undefined -otherwise. - -The hot loops run on flat row-major arrays through unsafe buffer pointers. -This is a documented performance necessity: bounds-checked nested-array -subscripts block auto-vectorization and left Swift roughly one hundred -times behind the Rust port on large reindex drains. The loop nest and -reduction order were preserved exactly through that rewrite, so the NMF -conformance vector (CRC `0x300bf633`) remains byte-identical. -`reconstructionError(V:W:H:)` exposes the RMS error on nested arrays, and -the nested-array matrix helpers remain for external callers. On -completion the engine emits the `nmf.factor` telemetry signal carrying -the final error. +Lee-Seung multiplicative update rules. These rules keep each entry +non-negative automatically. Each iteration first rescales H by the +ratio of `WᵀV` to `WᵀWH`. It then rescales W by the ratio of `VHᵀ` +to `WHHᵀ`. An epsilon of 1e-9 guards the denominators against +division by zero. W and H start as uniform random values from a +SplitMix64 generator, seeded by the caller with a default of +`0xDEADBEEFCAFEBABE`. Initialization is thus bit-identical +everywhere. Iteration stops when the root-mean-square reconstruction +error stops changing by more than the tolerance, or at a fixed cap. +Entry preconditions enforce a rectangular, finite, non-negative V. +The Lee-Seung theorem is undefined otherwise. + +The hot loops run on flat row-major arrays, through unsafe buffer +pointers. This is a documented performance necessity. +Bounds-checked nested-array subscripts block auto-vectorization. +They once left Swift roughly one hundred times behind the Rust port +on large reindex drains. The loop nest and reduction order were +preserved exactly through that rewrite. The NMF conformance vector, +CRC `0x300bf633`, thus remains byte-identical. +`reconstructionError(V:W:H:)` exposes the RMS error on nested +arrays. The nested-array matrix helpers remain for external callers. +On completion, the engine emits the `nmf.factor` telemetry signal, +carrying the final error. ## NMFDoubleFrobeniusSquared.swift -This file provides a parked, explicitly non-production NMF variant: the -double-precision engine with a raw Frobenius-squared convergence test -that one consumer used before migrating to the canonical engine above. - -It exists for one reason: honesty in benchmarking. The migration decision -requires a future benchmark comparing the old f64/Frobenius-squared -approach against the canonical f32/RMS approach on iteration count, time, -memory, and recall quality. Deleting the old algorithm would make that -comparison impossible, so it is preserved verbatim behind a production -gate: no consumer may wire to this type until the benchmark passes, and -even SIMD acceleration of the f64 path is separately gated. - -The algorithm is the same Lee-Seung scheme in scalar f64, with three -deliberate differences: convergence checks the raw (unnormalized) -Frobenius-squared error rather than RMS; initialization floors every -starting cell at 1e-3, because a cell at exact zero can never recover -under multiplicative updates; and the default seed is its own historical -value (`0xC0FFEE_BABE_BEEF`), distinct from the package's canonical seeds. -`factorize(...)` returns an `NMFDoubleFrobeniusSquaredFactorization` whose -`loadings(forRow:)` yields a row's latent factors. Despite being parked, -the file is still conformance-gated: Swift and Rust must stay -bit-identical on its named vector, so the eventual benchmark measures the -algorithm, not implementation drift. +This file gives a parked, non-production NMF variant. It is a +double-precision engine with a raw Frobenius-squared convergence +test. One consumer used it before migrating to the canonical engine +above. + +It exists for one reason: honesty in benchmarking. The migration +decision needs a future benchmark. That benchmark must compare the +old f64 Frobenius-squared approach against the canonical f32 RMS +approach. It must compare iteration count, time, memory, and recall +quality. Deleting the old algorithm would make that comparison +impossible. The file is thus preserved verbatim, behind a +production gate. No consumer may wire to this type until the +benchmark passes. Even SIMD acceleration of the f64 path stays +separately gated. + +The algorithm is the same Lee-Seung scheme, in scalar f64, with +three deliberate differences. Convergence checks the raw, +unnormalized Frobenius-squared error, rather than RMS. +Initialization floors each starting cell at 1e-3, because a cell at +exact zero can never recover under multiplicative updates. The +default seed is its own historical value, `0xC0FFEE_BABE_BEEF`, +distinct from the package's canonical seeds. `factorize(...)` +returns an `NMFDoubleFrobeniusSquaredFactorization`, whose +`loadings(forRow:)` yields a row's latent factors. The file is +parked, but it stays conformance-gated. Swift and Rust must stay +bit-identical on its named vector, so the eventual benchmark +shows the algorithm, not implementation drift. ## AnomalyDetection.swift -This file provides statistical anomaly scoring for telemetry streams and -matrix cells: the classic z-score and a robust variant, plus the -threshold check that turns a score into a flag. - -A z-score says how many standard deviations a value sits from the mean of -its baseline window. `zScore(value:mean:stddev:)` is the pure formula; -`rollingZScore(window:current:...)` computes mean and deviation from the -window first. The robust variant, `modifiedZScore(value:median:mad:)`, -replaces mean with median and deviation with the median absolute -deviation, then scales by the standard constant 0.6745 so the same -threshold applies to both variants; outliers already inside the window -cannot drag the baseline the way they drag a mean. -`isAnomalous(zScore:threshold:)` compares the absolute score against the -default threshold 3.0, roughly a one-in-370 false-positive rate on normal -data. - -`rollingModifiedZScore(...)` sorts a mutable copy of the window in place -rather than allocating a sorted copy; at large windows this measured five -times faster, and the behavior is pinned by a CRC marker (`0x6c6fda4d`) -plus a note matching Rust's sort semantics for all non-NaN input. Both -rolling functions emit the `anomaly.flag` telemetry signal with the -absolute score as the value; the timestamp is always caller-supplied. +This file gives statistical anomaly scoring for telemetry +streams and matrix cells. It offers the classic z-score, a robust +variant, and a threshold check that turns a score into a flag. + +A z-score says how many standard deviations a value sits from the +mean of a baseline window. `zScore(value:mean:stddev:)` is the pure +formula. `rollingZScore(window:current:...)` finds the mean and +deviation from the window first. The robust variant, +`modifiedZScore(value:median:mad:)`, replaces the mean with the +median. It replaces the deviation with the median absolute +deviation. It scales by the standard constant 0.6745, so the same +threshold applies to both variants. Outliers already inside the +window cannot drag the baseline the way they drag a mean. +`isAnomalous(zScore:threshold:)` compares the absolute score against +the default threshold, 3.0. That threshold gives roughly a +one-in-370 false-positive rate on normal data. + +`rollingModifiedZScore(...)` sorts a mutable copy of the window in +place, rather than allocating a sorted copy. At large windows this +measured five times faster. The behavior is pinned by a CRC marker, +`0x6c6fda4d`. It must also match Rust's sort semantics for all +non-NaN input. Both rolling functions emit the +`anomaly.flag` telemetry signal, with the absolute score as the +value. The timestamp is always caller-supplied. ## JacobiSVD.swift -This file provides a deterministic truncated singular value -decomposition (SVD) — the linear algebra that turns a term-document -matrix into low-dimensional semantic embeddings for latent semantic -analysis in CorpusKit. - -SVD factors a matrix into rotation, scaling, and rotation: `A = U S Vᵀ`. -The implementation is one-sided cyclic Jacobi: it repeatedly applies -two-by-two planar rotations to every pair of columns in a fixed cyclic -order, accumulating the rotations into V; after the sweeps, column norms -become the singular values and normalized columns become U. A sign -convention (largest-magnitude entry of each left vector forced positive) -removes the inherent plus-minus ambiguity. - -Every convergence-affecting choice is pinned for cross-port bit-identity, -which is why this exists instead of a call to Accelerate or LAPACK: the -sweep count is fixed (default 30, not convergence-tested), the rotation -order is fixed, all arithmetic is scalar Float32 with no SIMD or fused -multiply-add, and the rotation formula in `jacobiCS(alpha:beta:gamma:)` +This file gives a deterministic truncated singular value +decomposition, or SVD. It is the linear algebra that turns a +term-document matrix into low-dimensional semantic embeddings, for +latent semantic analysis in CorpusKit. + +SVD factors a matrix into rotation, scaling, and rotation: +`A = U S Vᵀ`. The implementation is one-sided cyclic Jacobi. It +repeatedly applies two-by-two planar rotations to each pair of +columns, in a fixed cyclic order. It accumulates the rotations into +V. After the sweeps, the column norms become the singular values. +The normalized columns become U. A sign convention forces the +largest-magnitude entry of each left vector positive. This removes +the inherent plus-minus ambiguity. + +Each convergence-affecting choice is pinned, for cross-port +bit-identity. That is why this exists instead of a call to +Accelerate or LAPACK. The sweep count is fixed, at a default of 30, +and is not convergence-tested. The rotation order is fixed. All +arithmetic is scalar Float32, with no SIMD and no fused +multiply-add. The rotation formula in `jacobiCS(alpha:beta:gamma:)` keeps a pinned expression tree that must not be refactored. -`decompose(A:rank:sweeps:)` requires at least as many rows as columns and -returns an `SVDResult` (U, non-increasing singular values, Vt, rank). The -sweep loops use unsafe buffer pointers for the same documented -vectorization reason as the NMF engine; every index is derived from loop -bounds, never from data. +`decompose(A:rank:sweeps:)` requires at least as many rows as +columns. It returns an `SVDResult`: U, non-increasing singular +values, Vt, and rank. The sweep loops use unsafe buffer pointers, +for the same documented vectorization reason as the NMF engine. +Each index is derived from loop bounds, never from data. ## FFT.swift -This file provides the discrete Fourier transform and the rhythm analysis -built on it. The substrate uses it to detect periodicity: read one -fingerprint bit across the most recent N time buckets, and the dominant -frequency of that 0/1 series reveals the rhythm of whatever that bit -encodes — circadian patterns in heart-rate bits, weekly cycles in -calendar bits. - -`Complex` is a plain double-precision complex number with the arithmetic -the transform needs. `FFT.forward(real:)` is the Cooley-Tukey radix-2 -algorithm: a bit-reversal permutation followed by log2(N) butterfly -stages. Input length must be a power of two — a constitutional constraint; -callers zero-pad shorter windows. The production Apple-silicon path routes -to the Accelerate framework but must produce bit-identical output to this -scalar reference on the conformance vectors; conformance is output -equality, not speed parity. `magnitudeSpectrum(real:)` maps the spectrum -to magnitudes. +This file gives the discrete Fourier transform, and the rhythm +analysis built on it. The substrate uses this to detect periodicity. +It reads one fingerprint bit across the most recent N time buckets. +The dominant frequency of that zero-or-one series reveals the +rhythm of whatever that bit encodes. Circadian patterns show up in +heart-rate bits. Weekly cycles show up in calendar bits. + +`Complex` is a plain double-precision complex number, with the +arithmetic the transform needs. `FFT.forward(real:)` is the +Cooley-Tukey radix-2 algorithm: a bit-reversal permutation, followed +by log2(N) butterfly stages. The input length must be a power of +two. This is a constitutional constraint, and callers zero-pad +shorter windows. The production Apple-silicon path routes to the +Accelerate framework. It must still produce bit-identical output to +this scalar reference, on the conformance vectors. Conformance means +output equality, and never speed parity. `magnitudeSpectrum(real:)` +maps the spectrum to magnitudes. `RhythmAnalysis.analyze(series:bucketDurationSeconds:)` runs the -transform, finds the strongest bin among the unique positive frequencies -(bin 1 through N/2), and converts it to a dominant period in seconds, -alongside total spectral energy excluding the DC bin. A constant series -has no dominant period and reports `nil` — the cookbook treats "no rhythm" -as an honest answer, not a default. The fingerprint-based overload -extracts the bit series from a list of fingerprints first, with the block -and bit position precondition-checked. +transform. It finds the strongest bin among the unique positive +frequencies, bin one through N divided by two. It converts that bin +to a dominant period in seconds. It also reports total spectral +energy, excluding the DC bin. A constant series has no dominant +period, so it reports `nil`. The cookbook treats "no rhythm" as an +honest answer, and a deliberate one. The fingerprint-based overload +extracts the bit series from a list of fingerprints first. The block +and bit position are precondition-checked. ## InformationTheory.swift -This file provides the information-theoretic primitives — entropy, mutual -information, KL divergence, cross-entropy, Jensen-Shannon divergence, and -normalized mutual information — used to quantify uncertainty in bitmap -distributions and to detect drift between a recent window and a long-term -baseline. - -All quantities use log base 2, so results are in bits. Entropy measures -how unpredictable a distribution is. Mutual information measures how much -knowing one variable tells you about another. KL divergence measures how -badly a model distribution q explains an observed distribution p. -Everywhere, terms with probability exactly zero are skipped, implementing -the standard `0 · log 0 = 0` limit without ever producing NaN. - -Two honesty notes are documented in the code. When p has mass where q has -none, true KL divergence is infinite; `klDivergence(_:_:)` skips such -terms and its result is therefore a lower bound, which callers must know. -And `normalizedMutualInformation(joint:)` returns the zero sentinel for a -ragged joint matrix rather than silently computing a plausible-looking -wrong value from corrupted marginals. Length mismatches between p and q -are hard preconditions; distribution validity itself (non-negative, -summing to one) is the caller's responsibility by contract. +This file gives the information-theoretic primitives: entropy, +mutual information, KL divergence, cross-entropy, Jensen-Shannon +divergence, and normalized mutual information. These primitives +quantify uncertainty in bitmap distributions. They also detect drift +between a recent window and a long-term baseline. + +All quantities use log base two, so results come out in bits. +Entropy shows how unpredictable a distribution is. Mutual +information shows how much knowing one variable tells you about +another. KL divergence shows how badly a model distribution q +explains an observed distribution p. Terms with probability exactly +zero are skipped everywhere. This implements the standard rule that +zero times log of zero equals zero, without ever producing NaN. + +Two honesty notes are documented in the code. When p has mass where +q has none, the true KL divergence is infinite. +`klDivergence(_:_:)` skips such terms, so its result is a lower +bound. Callers must know this. `normalizedMutualInformation(joint:)` +returns a zero sentinel for a ragged joint matrix. This avoids +silently computing a plausible-looking wrong value from corrupted +marginals. Length mismatches between p and q are hard preconditions. +Distribution validity itself, meaning non-negative values that sum +to one, is the caller's responsibility by contract. ## AssociationRuleMining.swift -This file provides pairwise association-rule mining over `MatrixO`, the -co-occurrence matrix the substrate updates on every capture: it surfaces -rules of the form "when field A has value x, field B tends to have value -y," with the standard support, confidence, lift, conviction, and leverage -metrics. - -The design leans on how `MatrixO` is built. Its diagonal cells already -hold single-item counts, and its off-diagonal cells hold pair counts, so -`mineAssociationRules(matrix:activeRowCount:thresholds:)` needs only two -passes over the stored entries: one to collect single supports from the -diagonal, one to walk the pairs, compute the five metrics, and keep rules -that clear the `MiningThresholds` gates. Self-rules (A implies A) are -skipped as noise, conviction is defined as positive infinity at -confidence one, and the total row count N is injected by the caller -because the matrix itself does not know it. Because the entries iterate -in canonical packed-key order, the output needs no final sort — an -ordering both legs rely on for byte-identical results. - -`Item` is the shared atom: a `(field, value)` byte pair with a packed -16-bit ordering key, reused by the Apriori engine. The internal -`AssociationRuleEngine` holds the actual math so conformance tests can -drive it directly; multi-item antecedents are out of scope here and -belong to `AprioriMining.swift`. +This file gives pairwise association-rule mining, over +`MatrixO`, the co-occurrence matrix the substrate updates on each +capture. It surfaces rules of the form: when field A has value x, +field B tends to have value y. Each rule carries the standard +support, confidence, lift, conviction, and leverage metrics. + +The design leans on how `MatrixO` is built. Its diagonal cells +already hold single-item counts. Its off-diagonal cells hold pair +counts. `mineAssociationRules(matrix:activeRowCount:thresholds:)` +thus needs only two passes over the stored entries. One pass +collects single supports from the diagonal. The other walks the +pairs, finds the five metrics, and keeps rules that clear the +`MiningThresholds` gates. Self-rules, where A implies A, are skipped +as noise. Conviction is defined as positive infinity at confidence +one. The total row count, N, is injected by the caller. The matrix +itself does not know it. The entries iterate in canonical +packed-key order, so the output needs no final sort. Both legs rely +on that order for byte-identical results. + +`Item` is the shared atom: a field-and-value byte pair, with a +packed 16-bit ordering key. The Apriori engine reuses it. The +internal `AssociationRuleEngine` holds the actual math, so +conformance tests can drive it directly. Multi-item antecedents fall +outside this file's scope. They belong to `AprioriMining.swift`. ## AprioriMining.swift -This file provides the multi-antecedent generalization: the classic -Apriori algorithm over `RowAttributeView` rows, producing rules such as -"{A, B} implies C." At its minimum itemset size it reproduces the pairwise -engine's answers; beyond that it finds combinations the pairwise engine -cannot see. - -`AprioriMining.mine(rows:thresholds:)` follows the textbook outline. Each -row becomes an item set. Frequent single items seed the search; each -level joins frequent (k-1)-item sets that share a lexicographic prefix, -counts candidate support with subset tests, and prunes below minimum -support, until the size cap `maxK` (default 3). Rules are then extracted -from every frequent itemset of size two or more, with each member tried -as the consequent, and filtered by the support, confidence, and lift -gates in `AprioriThresholds`. Requiring lift at least 1.0 suppresses -anti-correlated coincidences. - -Determinism gets specific care because Swift dictionaries iterate in -undefined order: itemsets are processed in canonical lexicographic order, -and the final sort uses a four-key total order (lift, confidence, -evidence count, then lexicographic tie-break), so output is identical -across runs and across languages. One defensive detail: the type's custom -`Decodable` initializer routes through the public initializer so a -serialized `maxK` of zero or one cannot bypass the floor of two and -corrupt the level loop. `mineAprioriRules(rows:thresholds:)` is a free -function wrapper matching the pairwise engine's call style; `AprioriRule` -carries the five metrics plus a raw `evidenceCount` for interpretability. +This file gives the multi-antecedent generalization: the classic +Apriori algorithm, over `RowAttributeView` rows. It produces rules +such as: if A and B are set, then C is set too. At the minimum +itemset size, it reproduces the pairwise engine's answers. Beyond +that size, it finds combinations the pairwise engine misses. + +`AprioriMining.mine(rows:thresholds:)` follows the textbook +outline. Each row becomes an item set. Frequent single items seed +the search. Each level joins frequent item sets of size k minus one +that share a lexicographic prefix. It counts candidate support with +subset tests. It prunes anything below minimum support, up to the +size cap `maxK`, which defaults to three. Rules are extracted from +each frequent itemset of size two or more. Each member is tried as +the consequent. Rules are then filtered by the support, confidence, +and lift gates in `AprioriThresholds`. Requiring a lift of at least +1.0 suppresses anti-correlated coincidences. + +Determinism gets specific care here, because Swift dictionaries +iterate in undefined order. Itemsets are processed in canonical +lexicographic order. The final sort uses a four-key total order: +lift, confidence, evidence count, and then a lexicographic +tie-break. Output stays the same across runs and across languages. +One defensive detail matters. The type's custom `Decodable` +initializer routes through the public initializer. A serialized +`maxK` of zero or one thus cannot bypass the floor of two and +corrupt the level loop. `mineAprioriRules(rows:thresholds:)` is a +free-function wrapper, matching the pairwise engine's call style. +`AprioriRule` carries the five metrics, plus a raw `evidenceCount` +for interpretability. ## FormalConceptAnalysis.swift -This file provides bounded Formal Concept Analysis (FCA): it finds the -exact groupings hidden in a table of rows and attributes. A formal -concept is a maximal pair — a set of rows that share exactly a set of -attributes — and the concepts of an estate are the data-driven "kinds of -memory" that emerged from what was actually recorded, as opposed to the -soft themes NMF finds or the graph clusters Louvain finds. - -`FormalAttribute` is one typed attribute (namespace, key, value), ordered -lexicographically — that ordering is the determinism backbone of the whole -file. `FormalContext` stores the row-attribute relation in both directions -as bitsets, so the derivation operators `extent(of:)` (rows carrying all -given attributes), `intent(of:)` (attributes common to all given rows), -and `closure(of:)` reduce to word-wise bit operations. Contexts build from -raw per-row attribute lists or from `RowAttributeView` batches. - -Full concept-lattice enumeration is exponential, so `BoundedConceptMiner` -never attempts it. It seeds from attributes meeting `minSupport` (and -optionally frequent attribute pairs in `.multi` seed mode, capped by -`maxSeeds`), takes exactly one closure per seed, deduplicates by intent, -and truncates at `maxConcepts` — polynomial cost by construction, with a +This file gives bounded Formal Concept Analysis, or FCA. It +finds the exact groupings hidden in a table of rows and attributes. +A formal concept is a maximal pair: a set of rows that share exactly +a set of attributes. The concepts of an estate are data-driven +"kinds of memory." They emerge from what was actually recorded. This +differs from the soft themes NMF finds, and from the graph clusters +Louvain finds. + +`FormalAttribute` is one typed attribute: a namespace, a key, and a +value. It is ordered lexicographically, and that order is the +determinism backbone of the whole file. `FormalContext` stores the +row-attribute relation in both directions, as bitsets. The +derivation operators reduce to word-wise bit operations because of +that choice. `extent(of:)` finds rows carrying all given attributes. +`intent(of:)` finds attributes common to all given rows. +`closure(of:)` combines both. Contexts build from raw per-row +attribute lists, or from `RowAttributeView` batches. + +Full concept-lattice enumeration is exponential, so +`BoundedConceptMiner` never attempts it. It seeds from attributes +meeting `minSupport`. It can optionally seed from frequent attribute +pairs, in `.multi` seed mode, capped by `maxSeeds`. It takes exactly +one closure per seed. It deduplicates by intent. It truncates at +`maxConcepts`. This gives polynomial cost by construction, with a fully specified output sort. `ConceptCoverDeltas.covering(concepts:)` -computes the direct-neighbor edges of the concept order, the "what -attributes were added" structural lens; the file is explicit that a cover -edge is not a logical implication. `StabilityEstimator.estimate(...)` -approximates how robust a concept is to noise by re-closing random -half-subsets of its extent under a per-concept SplitMix64 stream (caller -seed mixed with a hash of the concept; default seed -`0xCAFEBABEDEADBEEF`), trading exactness for a bounded budget. A zero -budget, the default, leaves stability `nil`. +finds the direct-neighbor edges of the concept order, a +structural lens showing which attributes were added. The file +explains that a cover edge shows structure only. It is not a +logical implication. `StabilityEstimator.estimate(...)` +approximates how robust a concept is to noise. It re-closes random +half-subsets of the extent, under a per-concept SplitMix64 stream. +That stream mixes the caller seed with a hash of the concept, using +a default seed of `0xCAFEBABEDEADBEEF`. This trades exactness for a +bounded budget. A zero budget, the default, leaves stability as +`nil`. ## ConceptImplications.swift -This file provides the Duquenne-Guigues canonical basis: the minimal set -of implications — "every row with attributes X also has attribute Y" — -that hold without exception across a formal context. Where FCA finds the -groupings, this finds the rules the groupings obey, feeding consolidation +This file gives the Duquenne-Guigues canonical basis. That basis +is the minimal set of implications that hold without exception +across a formal context. Each implication reads: each row with +attributes X also has attribute Y. Where FCA finds groupings, this +file finds the rules those groupings obey. That feeds consolidation and synthesis. `conceptImplications(over:context:maxImplications:maxPremiseSize:)` -enumerates candidate premises in strictly increasing size order and tests -each for pseudo-intent status: its closure must add something, and every -smaller pseudo-intent's conclusion must already be inside it. Processing -in size order makes that incremental test equivalent to full minimality -checking. Two independent caps keep an NP-hard enumeration bounded: -`maxImplications` is a hard stop that sets `isTruncated` (with a -look-ahead so the flag is honest when the cap lands mid-run), and -`maxPremiseSize` silently skips larger premises without affecting the -soundness of what is emitted. Every ordering — the attribute universe, -the combination enumeration, and the final sort by premise size then -lexicographic content — is pinned, so both legs emit bit-identical bases. -`Implication` carries the premise and conclusion sets; an empty context -short-circuits to an empty, untruncated result. +enumerates candidate premises, in strictly increasing size order. It +tests each one for pseudo-intent status. The closure must add +something new. Each smaller pseudo-intent's conclusion must +already sit inside it. Processing in size order makes that +incremental test equivalent to full minimality checking. Two +independent caps keep this NP-hard enumeration bounded. +`maxImplications` is a hard stop, and it sets `isTruncated`, with a +look-ahead so the flag stays honest when the cap lands mid-run. +`maxPremiseSize` silently skips larger premises, without affecting +the soundness of what gets emitted. Each order is pinned: the +attribute universe, the combination enumeration, and the final sort +by premise size then lexicographic content. Both legs thus emit +bit-identical bases. `Implication` carries the premise and +conclusion sets. An empty context short-circuits to an empty, +untruncated result. ## TemporalCausalityFold.swift -This file provides the fold that feeds the temporal-causality matrix: it -scans a time-sorted stream of audit entries and counts "field X changed, -then field Y changed N minutes later" pairs, bucketed by lag. - -`fold(entries:windowMinutes:startWatermark:)` processes each entry newer -than the caller's watermark. It evicts buffered entries older than the -window (256 minutes by default), then pairs the new entry against every -remaining buffered entry, mapping each minute delta to one of eight -log-spaced lag buckets — 1, 2, 4, up to 128 minutes, via -`lagBucket(forMinutes:)` — and emitting one delta increment per source -coordinate, target coordinate, and bucket. The result carries the deltas -plus a new watermark, so the hourly batch run resumes where it stopped. - -Two contracts matter. Entries must arrive pre-sorted by HLC; the fold -does not re-sort and would silently compute wrong deltas otherwise. And -the rolling buffer is capped at 512 entries (`maxWindowOccupancy`), -because a bulk import can drop tens of thousands of events into one -window and unbounded pairing is quadratic; keeping only the most recent -in-window entries preserves the near-lag signal the causality matrix -actually cares about. Output deltas are aggregated by key but returned in -first-insertion order, so repeated runs are bit-identical. The lag bucket -table and the occupancy cap are pinned constants mirrored exactly in the -downstream matrix type and the Rust port. The input shapes -(`TemporalFieldCoord`, `TemporalAuditEntry`) are defined locally so the -kit that owns the real audit types stays upstream. +This file gives the fold that feeds the temporal-causality +matrix. It scans a time-sorted stream of audit entries. It counts +pairs of the form: field X changed, then field Y changed some +minutes later. Each pair is bucketed by lag. + +`fold(entries:windowMinutes:startWatermark:)` processes each entry +newer than the caller's watermark. It evicts buffered entries older +than the window, 256 minutes by default. It pairs the new entry +against each remaining buffered entry. Each minute delta maps to +one of eight log-spaced lag buckets, from one up to 128 minutes, +through `lagBucket(forMinutes:)`. The fold emits one delta increment +per source coordinate, target coordinate, and bucket. The result +carries the deltas plus a new watermark, so the hourly batch run +resumes where it stopped. + +Two contracts matter here. Entries must arrive pre-sorted by HLC. +The fold does not re-sort them, and would silently compute wrong +deltas otherwise. The rolling buffer is capped at 512 entries, +`maxWindowOccupancy`. A bulk import can drop tens of thousands of +events into one window, and unbounded pairing is quadratic. Keeping +only the most recent in-window entries preserves the near-lag +signal the causality matrix actually cares about. Output deltas are +aggregated by key, but returned in first-insertion order, so +repeated runs stay bit-identical. The lag bucket table and the +occupancy cap are pinned constants, mirrored exactly in the +downstream matrix type and the Rust port. The input shapes are +`TemporalFieldCoord` and `TemporalAuditEntry`. They are defined +locally, so the kit that owns the real audit types stays upstream. ## TypedDecayWeighting.swift -This file provides type-aware staleness weighting for distillation -features. Different kinds of extracted facts age at different speeds: a -version number (numerical) goes stale in days, while a domain concept -(entity) stays true across major changes. - -`DistillationFeatureType` names the four categories with their wire tags — -ENT, REL, TMP, NUM — and pins each one's decay rate lambda: 0.1, 0.2, -0.5, and 0.8 respectively, by analogy with frequency bands in diffusion -noise schedules. `weight(featureType:ageInUnits:)` computes the -exponential weight `exp(-lambda * age)`, clamping negative ages to zero -so future-dated observations count as present. One age unit is one day by -convention (86,400 seconds), though callers choose the unit. - -`weightedDocFrequency(featureType:presenceTimestamps:allMemoryTimestamps: -referenceDate:timeUnit:)` computes the decayed document frequency: the -summed weight of the memories where a feature is present, divided by the -summed weight of all memories in the cluster. Recent evidence therefore -counts more than old evidence, at a rate set by the feature's type. This -is the frequency the distillation pipeline uses whenever timestamps are -available. +This file gives type-aware staleness weighting for distillation +features. Different kinds of extracted facts age at different +speeds. A version number, a numerical fact, goes stale within days. +A domain concept, an entity fact, stays true across major changes. + +`DistillationFeatureType` names four categories, with wire tags ENT, +REL, TMP, and NUM. It pins each category's decay rate, lambda: 0.1, +0.2, 0.5, and 0.8. This scheme is an analogy with frequency bands in +diffusion noise schedules. `weight(featureType:ageInUnits:)` +finds the exponential weight `exp(-lambda * age)`. It clamps +negative ages to zero, so future-dated observations count as +present. One age unit is one day by convention. That equals `86,400` +seconds, though callers may choose a different unit. + +`weightedDocFrequency(featureType:presenceTimestamps:allMemoryTimestamps:referenceDate:timeUnit:)` +finds the decayed document frequency. That frequency is a ratio of +two weights. The first is the summed weight of memories where a +feature is present. The second is the summed weight of all memories +in the cluster. Recent evidence +thus counts more than old evidence, at a rate set by the +feature's type. This is the frequency the distillation pipeline +uses, whenever timestamps are available. ## DeltaFeatureExtractor.swift -This file provides trajectory classification for a single feature's value -sequence across a cluster: has the value stabilized, is it trending, is -it flip-flopping, or is it noise? Its job is rescue — a feature that -fails the recurrence threshold on raw counts may still carry a real -pattern over time, and this classifier finds it. - -`DeltaType` names the five verdicts: STATIC, CONVERGENT, MONOTONE, -OSCILLATING, DIVERGENT. `analyzeCategorical(sequence:decayLambda:)` -handles string-valued features: all-identical values are STATIC; an -A-B-A-B pattern over the last four observations (four required, to avoid -false positives on short sequences) is OSCILLATING; otherwise the length -of the trailing run of the final value, as a fraction of the sequence, -decides CONVERGENT (at least the threshold, default 0.5) or DIVERGENT. -`analyzeNumerical(sequence:decayLambda:)` handles numbers via consecutive -differences: all zero is STATIC, all one sign is MONOTONE (with slope and -a fixed confidence of the type's decay rate, default 0.8), strict sign -alternation is OSCILLATING, anything else DIVERGENT. - -Both paths return a full `DeltaAnalysis` — verdict, terminal value, -convergence score, optional slope, confidence — so the caller always has -a well-formed result. The file is a pure namespace: no state, no clock, -no randomness at all. The oscillation check looks only at the recent -tail on purpose; the question is the current trajectory, not history. +This file classifies a feature's value sequence across a cluster, +into four trajectory states. A value can stabilize. A value can +trend. A value can flip-flop. A value can turn to noise. Its job is +rescue. A feature may fail the recurrence threshold on raw counts. +It may still carry a real pattern over time. This classifier finds +that pattern. + +`DeltaType` names five verdicts: STATIC, CONVERGENT, MONOTONE, +OSCILLATING, and DIVERGENT. +`analyzeCategorical(sequence:decayLambda:)` handles string-valued +features. All-identical values score STATIC. An A-B-A-B pattern over +the last four observations scores OSCILLATING. Four observations are +required, to avoid false positives on short sequences. Otherwise, the +trailing run of the final value decides the verdict. Its length +counts as a fraction of the sequence. That fraction scores CONVERGENT +at or above a default threshold of 0.5, and DIVERGENT below it. +`analyzeNumerical(sequence:decayLambda:)` handles numbers through +consecutive differences. All-zero differences score STATIC. All one +sign scores MONOTONE, with a slope and a fixed confidence equal to +the type's decay rate, default 0.8. Strict sign alternation scores +OSCILLATING. Anything else scores DIVERGENT. + +Both paths return a full `DeltaAnalysis`: verdict, terminal value, +convergence score, optional slope, and confidence. The caller +thus always has a well-formed result. The file is a pure +namespace. It holds no state. It reads no clock. It uses no +randomness at all. The oscillation check looks only at the recent +tail, on purpose. The question this file answers is the current +trajectory, and never the full history. ## DistillationScorer.swift -This file provides the scoring stages of distillation: the math that -decides whether a cluster of memories is coherent enough to compress and, -if so, which features form the core factoid. - -The pipeline of pure functions runs in order. A structural threshold, -`structuralThreshold(M:)` = 2/M, splits features into structural -(recurring in at least two of the M units) and episodic (one-off); -`applyStructuralThreshold(features:M:)` performs the split. The file's -long comment explains why this is intra-cluster recurrence rather than a -majority vote: majority voting was found to discard real distillable -cores when a single document is split into sentences. -`computeSNR(features:M:)` then gates readiness: the summed frequency of -structural features must be at least twice the episodic sum (the -signal-to-noise ratio, floored at a tiny epsilon against division by -zero). `computeStructuralScores(features:)` assigns each feature a score -rewarding frequency and consistency via binary entropy. +This file gives the scoring stages of distillation. It decides +whether a cluster of memories is coherent enough to compress. If so, +it decides which features form the core factoid. + +A pipeline of pure functions runs in order. A structural threshold, +`structuralThreshold(M:) = 2/M`, splits features into two groups. +Structural features recur in at least two of the M units. Episodic +features appear only once. `applyStructuralThreshold(features:M:)` +performs the split. The file's long comment explains a design +choice: it uses intra-cluster recurrence, rather than a majority +vote. A majority vote was found to discard real distillable cores, +when a single document was split into sentences. +`computeSNR(features:M:)` then gates readiness. The summed frequency +of structural features must be at least twice the episodic sum. That +ratio is the signal-to-noise ratio, floored at a tiny epsilon +against division by zero. `computeStructuralScores(features:)` +assigns each feature a score. The score rewards frequency and +consistency, through binary entropy. `buildPMIGraph(thresholdFeatures:incidenceMatrix:M:)` builds the -coherence graph: pointwise mutual information (PMI), in log base 2, -measures whether two features co-occur more than chance; positive-PMI -pairs become edges, and connected components are found by depth-first -search (quadratic, acceptable at the documented cap of about 150 -features) and sorted by descending weighted frequency. -`selectDominantComponent(graph:)` returns the top component — the factoid -core — and `computeConfidence(selected:allThreshold:)` scores it as mean -frequency times a fragmentation penalty (the fraction of threshold -features the core kept). Everything is deterministic and log2-based; the -Rust port's comments stress that log2, not natural log, must match -bit-for-bit. +coherence graph. Pointwise mutual information, in log base two, +shows whether two features co-occur more than chance would +predict. Positive-PMI pairs become edges. Connected components are +found by depth-first search, quadratic but acceptable at the +documented cap of about 150 features. Components are sorted by +descending weighted frequency. `selectDominantComponent(graph:)` +returns the top component, the factoid core. +`computeConfidence(selected:allThreshold:)` scores it as the mean +frequency, times a fragmentation penalty: the fraction of threshold +features the core kept. Everything here is deterministic and +log2-based. The Rust port's comments stress that log2, and never +natural log, must match bit-for-bit. ## DistillationPipeline.swift -This file provides the complete five-stage distillation algorithm: a -cluster of raw memory strings in, one condensed factoid out. The output -`DistillationOutput` carries the formatted drawer content -(`"[DIST|conf=…|src=…|snr=…|delta=…] prose"`), the confidence, the SNR, -an optional delta verdict, a success flag, and a structural fingerprint -for Hamming-space lookup. - -`run(input:extractFeatures:intraItem:)` executes the stages. Stage 1 -extracts features per memory per feature type through an injected -`FeatureExtractor` closure — production injects a real entity tagger, -tests use the bundled capitalization-heuristic `defaultExtractor` — and -builds the vocabulary and incidence matrix. Document frequencies come -from `TypedDecayWeighting` when timestamps exist. Stage 2 applies the SNR -gate and the recurrence threshold. Stage 2.5 is the delta pre-pass: -features that failed recurrence are grouped by predicate key and their -value sequences run through `DeltaFeatureExtractor`; converging or -trending features are rescued with their terminal value. Stage 3 builds -the PMI graph and keeps the dominant component, with two deliberate -exceptions: in intra-item mode (one document split into sentences) all -passing features are kept, because PMI pruning would wrongly split a -single coherent document; and a ubiquity fix-up re-adds any feature -present in nearly every member, because a ubiquitous feature has zero PMI -with everything and would otherwise vanish despite being the semantic -spine. Stages 4 and 5 score, format the header, and OR-reduce per-feature -hashes into the fingerprint. - -`featureHash(_:)` maps a feature string to a `Fingerprint256` via -SplitMix64 under the pinned seed `featureSimHashSeed` -(`0x44495354494C4C41`, ASCII "DISTILLA"); changing that seed invalidates -every stored distillation fingerprint, so it is conformance-locked. -`queryFingerprint(query:extractFeatures:)` builds the matching probe -fingerprint at query time with no model inference. `DistilledHeader.parse` -reads the `[DIST|...]` header back out of stored content. Confidence -below 0.4 is failure; between 0.4 and 0.7 the output is marked uncertain. -One formatting subtlety is pinned: the `src=` field counts source drawer -IDs, not cluster members — the two differ in intra-item mode. The `run` -body is deliberately one large function; splitting it would thread the -incidence matrix and vocabulary through every helper signature. +This file gives the complete five-stage distillation algorithm. +A cluster of raw memory strings goes in. One condensed factoid comes +out. The output type is `DistillationOutput`. It carries the +formatted drawer content, the confidence, the SNR, and an optional +delta verdict. It also carries a success flag and a structural +fingerprint for Hamming-space lookup. + +`run(input:extractFeatures:intraItem:)` executes the stages. Stage +one extracts features, per memory and per feature type, through an +injected `FeatureExtractor` closure. Production injects a real +entity tagger. Tests use the bundled capitalization-heuristic +`defaultExtractor`. This stage builds the vocabulary and the +incidence matrix. Document frequencies come from +`TypedDecayWeighting` when timestamps exist. Stage two applies the +SNR gate and the recurrence threshold. Stage two-and-a-half is the +delta pre-pass. Features that failed recurrence are grouped by +predicate key. Their value sequences run through +`DeltaFeatureExtractor`. Converging or trending features get rescued +with their terminal value. Stage three builds the PMI graph and +keeps the dominant component, with two deliberate exceptions. In +intra-item mode, where one document splits into sentences, all +passing features are kept. PMI pruning would wrongly split a single +coherent document otherwise. A ubiquity fix-up also re-adds any +feature present in nearly each member. A ubiquitous feature has zero +PMI with everything, and would otherwise vanish despite being the +semantic spine. Stages four and five score the result, format the +header, and OR-reduce the per-feature hashes into the fingerprint. + +`featureHash(_:)` maps a feature string to a `Fingerprint256`, +through SplitMix64, under the pinned seed `featureSimHashSeed`. That +seed, `0x44495354494C4C41`, spells "DISTILLA" in ASCII. Changing the +seed invalidates each stored distillation fingerprint, so it stays +conformance-locked. `queryFingerprint(query:extractFeatures:)` +builds the matching probe fingerprint at query time, with no model +inference. `DistilledHeader.parse` reads the stored header back out +of stored content. Confidence below 0.4 counts as failure. +Confidence between 0.4 and 0.7 marks the output uncertain. One +formatting subtlety is pinned. The `src=` field counts source +drawer IDs, not cluster members. The two differ in intra-item mode. +The `run` body stays one large function, on purpose. Splitting it +would thread the incidence matrix and vocabulary through each +helper signature. ## PairingHandshake.swift -This file provides the pairing handshake: the protocol steps that let two -estates establish a shared fingerprint basis so their memories become -comparable for federation. - -Fingerprints are only comparable under the same hyperplane family, and -each estate normally has its own. The handshake fixes that without a -negotiation round trip: after the two estates exchange a 32-byte nonce -out of band, `generateSharedFamily(nonce:estateA:estateB:density:)` lets -each side independently derive the identical shared family, because the -SplitMix64 seed is a pure function of the nonce and the two estate UUIDs. -`sharedFamilyKey(case:peerEstate:)` computes the canonical manifest key -(`H_shared__`) both sides store it under, and -`buildPairEvent(...)` and `buildUnpairEvent(...)` produce the audit -payloads that record the relationship. Dissolving a pairing keeps the -shared family — historical "as of" queries must still work — but stops -further sync. - -The subtlest line in the file is `PairingNonce.seedWith(estateA:estateB:)`: -it orders the two UUIDs by raw byte comparison, never by their hexadecimal -string form. ASCII comparison of hex strings can rank two UUIDs in the -opposite order from their raw bytes, and the Rust leg always compares raw -bytes — a string-ordered Swift would derive a different seed and the two -devices would build incompatible families. The nonce length (exactly 32 -bytes) is precondition-enforced, and the FNV-1a constants do the seed and -family-hash mixing. +This file gives the pairing handshake. These are the protocol +steps that let two estates establish a shared fingerprint basis, so +their memories become comparable for federation. + +Fingerprints are only comparable under the same hyperplane family, +and each estate normally has its own. The handshake fixes that +without a negotiation round trip. The two estates first exchange a +32-byte nonce, out of band. +`generateSharedFamily(nonce:estateA:estateB:density:)` then lets +each side independently derives the same shared family. This +works because the SplitMix64 seed is a pure function of the nonce +and the two estate UUIDs. `sharedFamilyKey(case:peerEstate:)` +finds the canonical manifest key both sides store the family +under. `buildPairEvent(...)` and `buildUnpairEvent(...)` produce the +audit payloads that record the relationship. Dissolving a pairing +keeps the shared family, since historical "as of" queries must +still work. It does stop further sync, however. + +The subtlest line in the file is +`PairingNonce.seedWith(estateA:estateB:)`. It orders the two UUIDs by +raw byte comparison, never by their hexadecimal string form. ASCII +comparison of hex strings can rank two UUIDs in the opposite order +from their raw bytes. The Rust leg always compares raw bytes. A +string-ordered Swift would derive a different seed, and the two +devices would build incompatible families. The nonce length, exactly +32 bytes, is precondition-enforced. The FNV-1a constants handle the +seed and family-hash mixing. ## TierContributionFingerprint.swift -This file provides the tier contribution: the fixed 64-byte payload an -estate sends up the federation hierarchy, summarizing its shareable rows -as one OR-reduced fingerprint. - -`FederationCase` names the three tiers (household, fleet, industry). -`build(estateUUID:case:shareableFingerprints:hlc:)` OR-reduces the -shareable fingerprints — routed through the platform kernel so federation -work uses the best available SIMD backend while preserving the -commutative, associative, idempotent math — and wraps the result with the -estate UUID, the case, the row count, and the HLC into a +This file gives the tier contribution. It is the fixed 64-byte +payload an estate sends up the federation hierarchy. It summarizes +an estate's shareable rows as one OR-reduced fingerprint. + +`FederationCase` names the three tiers: household, fleet, and +industry. `build(estateUUID:case:shareableFingerprints:hlc:)` +OR-reduces the shareable fingerprints. This work is routed through +the platform kernel. Federation work thus uses the best +available SIMD backend. The math still stays commutative, +associative, and idempotent. The function wraps the result with the +estate UUID, the case, the row count, and the HLC, into a `TierContribution`. -`encode(_:)` and `decode(_:)` implement the canonical wire format: 16 -UUID bytes, then case and row count as big-endian 32-bit integers, then -the four fingerprint blocks as big-endian 64-bit integers, then the -packed HLC — 64 bytes exactly, big-endian throughout. The comment records -a real caught bug: an earlier draft emitted the fingerprint in its -little-endian storage form and diverged from Rust by 32 bytes until the -cross-language conformance gate flagged it. This layer neither signs nor -checksums; authenticity is a federation-egress concern applied at the -share point, and privacy noise is applied at the aggregator, not by the -contributor. +`encode(_:)` and `decode(_:)` implement the canonical wire format. +Sixteen UUID bytes come first. Then come the case and row count, as +big-endian 32-bit integers. Then come the four fingerprint blocks, +as big-endian 64-bit integers. Then comes the packed HLC. The total +runs exactly 64 bytes, big-endian throughout. A comment records a +real caught bug. An earlier draft emitted the fingerprint in its +little-endian storage form. That draft diverged from Rust by 32 +bytes, until the cross-language conformance gate flagged it. This +layer neither signs nor checksums its output. Authenticity is a +federation-egress concern, applied at the share point. Privacy +noise is applied at the aggregator, and never by the contributor. ## TierAscendingQuery.swift -This file provides the local-scope steps of the tier-ascending query -protocol: how a recall query travels to peer estates and how the answers -come back combined, without exposing any single estate's data. - -`TierAscendingQuery` models the query itself — the originating estate, -the named recall primitive to run, the target tier, the privacy budget, -and the query HLC. `computeLocal(query:dispatch:)` runs the local -primitive through an injected closure, keeping this package independent -of the cognition layer. On the peer side, -`applyDPToContribution(_:budget:rngSeed:)` adds Laplace noise (scale one -over epsilon, drawn from a seeded SplitMix64) to every score in the -peer's result and attaches a fixed 95 percent confidence interval of plus -or minus 1.96 times the scale, so the requester knows how blurry the -contribution is. `combine(local:peers:)` merges the exact local result -with the noised peer results: scores sum per row, ordering is descending -score with row identity as the deterministic tie-break, and the widest -confidence interval seen wins — a deliberately conservative choice, since -the least certain contribution bounds the certainty of the union. - -`PrivacyLedger` tracks per-peer epsilon and delta consumption against a -daily budget: `canConsume(peer:query:)` gates a query, `consume(...)` -records it, and `dailyReset()` starts a new day. Networking, signing, and -scheduling are explicitly the caller's job. One parity nuance is worth -knowing: the Rust leg zeroes the per-lane distance breakdown in a noised -contribution to close a side channel, while the Swift reference forwards -the breakdown unchanged. +This file gives the local-scope steps of the tier-ascending +query protocol. It defines how a recall query travels to peer +estates. It also defines how the answers come back combined. Neither +step exposes any single estate's data. + +`TierAscendingQuery` models the query itself. It names the +originating estate, the named recall primitive, the target tier, the +privacy budget, and the query HLC. `computeLocal(query:dispatch:)` +runs the local primitive through an injected closure. This keeps the +package independent of the cognition layer. On the peer side, +`applyDPToContribution(_:budget:rngSeed:)` adds Laplace noise to +each score in the peer's result. The noise scale is one over +epsilon, drawn from a seeded SplitMix64. The function attaches a +fixed 95 percent confidence interval, plus or minus 1.96 times the +scale. This lets the requester know how blurry the contribution is. +`combine(local:peers:)` merges the exact local result with the +noised peer results. Scores sum per row. The order runs by +descending score, with row identity as the deterministic tie-break. +The widest confidence interval seen always wins. This choice is +conservative on purpose, since the least certain contribution bounds +the certainty of the union. + +`PrivacyLedger` tracks per-peer epsilon and delta consumption +against a daily budget. `canConsume(peer:query:)` gates a query. +`consume(...)` records it. `dailyReset()` starts a new day. +Networking, signing, and scheduling are explicitly the caller's job. +One parity nuance is worth knowing. The Rust leg zeroes the +per-lane distance breakdown in a noised contribution, to close a +side channel. The Swift reference forwards that breakdown unchanged. ## DPORReduction.swift -This file provides the differentially private OR-reduction used at the -aggregation point of federated queries: many estates' fingerprints in, -one aggregate fingerprint out, with mathematical privacy protection -layered in. - -`reduce(fingerprints:params:rngSeed:)` works per bit position. For each -of the 256 positions it counts how many contributors set that bit, adds -Laplace noise with scale one over epsilon (drawn from a SplitMix64 seeded -by the caller, so the noise is reproducible for testing), and sets the -output bit only if the noised count reaches the k-anonymity threshold. -The two mechanisms stack: differential privacy means no observer can tell -whether any one estate contributed, and the k-anonymity floor (default -k=3) means no bit lights up from too few contributors even without -adversarial analysis. Empty input returns the zero fingerprint. - -`DPParameters` pins the defaults from the cookbook: epsilon 1.0, delta -1e-9, k 3, with validity preconditions. One documented asymmetry: delta -is validated and stored for API completeness but is not consumed by the -Laplace mechanism itself. The `laplaceNoise(scale:rng:)` helper derives a -uniform value from the top 53 bits of the raw draw — the same convention -the package's other samplers use — and inverts the Laplace distribution's -cumulative curve. +This file gives the differentially private OR-reduction used at +the aggregation point of federated queries. Many estates' +fingerprints go in. One aggregate fingerprint comes out, with +mathematical privacy protection layered in. + +`reduce(fingerprints:params:rngSeed:)` works per bit position. For +each of the 256 positions, it counts how many contributors set that +bit. It adds Laplace noise, at a scale of one over epsilon. The +noise comes from a SplitMix64 generator, seeded by the caller. This +keeps the noise reproducible for testing. It sets the output bit +only if the noised count reaches the k-anonymity threshold. Two +mechanisms stack here. Differential privacy means no observer can +tell whether any one estate contributed. The k-anonymity floor +defaults to three. It means no bit lights up from too few +contributors, even without adversarial analysis. Empty input returns +the zero fingerprint. + +`DPParameters` pins the defaults from the cookbook: epsilon 1.0, +delta 1e-9, and k 3, with validity preconditions. One documented +asymmetry stands out. Delta is validated and stored for API +completeness, but the Laplace mechanism itself does not consume it. +The `laplaceNoise(scale:rng:)` helper derives a uniform value from +the top 53 bits of the raw draw. This is the same convention the +package's other samplers use. The helper then inverts the Laplace +distribution's cumulative curve. ## Rust Port and Conformance -The `rust/` directory contains the second leg of the library: crate -`substrate-ml`, with one module per Swift file — thirty-eight modules from -`anomaly.rs` through `viz_graph_signals.rs`, declared in `rust/src/lib.rs` -with the same layering notes and the same production gate on the parked -NMF variant. The crate depends on `substrate-types`, `substrate-kernel`, -and `intellectus-lib`, mirroring the Swift package manifest exactly. - -Most modules open with an explicit "mirror of" header naming their Swift -source, and the determinism-bearing files repeat the pinned constants, -seeds, draw orders, and tie-break rules line for line. Shared gates prove -the agreement property: `rust/tests/distillation_conformance.rs` locks the -delta extractor, the typed decay weights, `featureHash`, and full -pipeline runs against fixed vectors mirrored in Swift's -`DistillationConformanceTests` ("fix the algorithm, not the vector"); +The `rust/` directory contains the second leg of the library: the +crate `substrate-ml`. It has one module per Swift file, thirty-eight +modules from `anomaly.rs` through `viz_graph_signals.rs`. These +modules are declared in `rust/src/lib.rs`, with the same layering +notes and the same production gate on the parked NMF variant. The +crate depends on `substrate-types`, `substrate-kernel`, and +`intellectus-lib`. This mirrors the Swift package manifest exactly. + +Most modules open with an explicit "mirror of" header, naming their +Swift source. The determinism-bearing files repeat the pinned +constants, seeds, draw orders, and tie-break rules, line for line. +Shared gates prove the agreement property. +`rust/tests/distillation_conformance.rs` locks four things: the +delta extractor, the typed decay weights, `featureHash`, and the +full pipeline. All four are checked against fixed vectors mirrored +in Swift's `DistillationConformanceTests`. The rule there is simple: +fix the algorithm, never the vector. `rust/tests/float_simhash_kernel_equivalence.rs` proves the kernel -projection reproduces the oracle bit for bit across seeds and dimensions; -and `rust/tests/viz_graph_signals_tests.rs` exercises the telemetry emit -of all five graph algorithms under a process-wide lock. Several -algorithms carry additional shared JSON vectors under the engineering -test harness (sampling, temporal causality fold, NMF, community -detection). When you change either leg, change both and run both suites; -the fixtures are the contract. +projection reproduces the oracle bit for bit, across seeds and +dimensions. `rust/tests/viz_graph_signals_tests.rs` exercises the +telemetry emit of all five graph algorithms, under a process-wide +lock. Several algorithms carry additional shared JSON vectors under +the engineering test harness. These vectors cover sampling, temporal +causality fold, NMF, and community detection. When you change either +leg, change both, and run both suites. The fixtures are the contract. diff --git a/packages/libs/SubstrateML/docs/OVERVIEW.md b/packages/libs/SubstrateML/docs/OVERVIEW.md index 6955ac7..7541513 100644 --- a/packages/libs/SubstrateML/docs/OVERVIEW.md +++ b/packages/libs/SubstrateML/docs/OVERVIEW.md @@ -87,136 +87,165 @@ sources: ## What This Library Does -SubstrateML is the learning layer of the MOOTx01 substrate. MOOTx01 is an -on-device AI memory system. It stores what an AI observes over time and -helps the AI recall it later. The substrate is the mathematical foundation -under that store: the types, the bit operations, and the algorithms that -every higher layer builds on. - -The substrate is split into layered packages. `SubstrateTypes` holds pure -data types, such as the 256-bit row fingerprint. `SubstrateKernel` holds -hot-path bit operations that run on every capture. SubstrateML — this -package, layer 3 — holds the cold-path algorithms: the math that learns -structure from stored memories rather than storing them. A memory in this -system is a row: one record with a fingerprint, a classification anchor, -and a set of bitmap fields. SubstrateML never touches storage. Every -function here takes plain values in and returns plain values out. - -Most of these algorithms run during dreaming. Dreaming is the system's -idle-time maintenance cycle: a background daemon that clusters, decays, -summarizes, and re-scores the memory estate while the device is otherwise -quiet. An estate is one user's complete memory store. The algorithms here -find the estate's themes, its communities, its habits, its rules, and its -condensable clusters — and they forget on schedule, so old evidence fades. +SubstrateML is the learning layer of the MOOTx01 substrate. MOOTx01 is +an on-device AI memory system. It stores what an AI observes over +time. It helps the AI recall that later. The substrate is the math +base under that store. It holds the types, the bit operations, and the +algorithms that every higher layer needs. + +The substrate splits into layered packages. `SubstrateTypes` holds +pure data types. One example is the 256-bit row fingerprint. +`SubstrateKernel` holds hot-path bit operations. These run on every +capture. SubstrateML is layer three. It holds the cold-path +algorithms: math that learns structure from stored memories. It does +not store memories itself. + +A memory in this system is a row. Each row has one fingerprint, one +classification anchor, and a set of bitmap fields. SubstrateML never +touches storage. Every function here takes plain values in and +returns plain values out. + +Most of these algorithms run during dreaming. Dreaming is the +system's idle-time maintenance cycle. A background daemon runs it +while the device sits quiet. The daemon clusters memories. It decays +old evidence. It summarizes clusters and re-scores the memory estate. + +An estate is one user's complete memory store. The algorithms here +find the estate's themes, its communities, and its habits. They find +its rules and its condensable clusters. They also forget on a +schedule, so old evidence fades over time. ## The Problem It Solves An on-device memory system must learn without a cloud. Cloud machine -learning changes without notice, needs a network, and sees private data. -SubstrateML instead ships small, exact, well-bounded reference algorithms -that run entirely on the device. - -It must also learn identically everywhere. MOOTx01 estates can federate, -which means separate devices share and compare results. Two devices that -factor the same matrix or mine the same rules must get the same answer, or -shared recall falls apart. SubstrateML therefore holds one agreement -property across two implementations: a Swift leg for Apple platforms and a -Rust leg (in `rust/`) for everything else. Both legs use the same canonical -pseudo-random number generator (SplitMix64), the same pinned seeds, the -same tie-breaking rules, and the same arithmetic order, so results are -bit-identical. Conformance fixtures — recorded input and output pairs both -legs must reproduce exactly — gate every change. - -Two library-wide rules protect that promise. First, SubstrateML never reads -a clock; every timestamp is passed in by the caller. Second, no algorithm -here holds hidden state; everything is a pure function or a plain value -type, safe to run from any thread. +learning changes without notice. It needs a network. It sees private +data. SubstrateML avoids all three problems. It ships small, exact, +well-bounded reference algorithms. Every one of them runs entirely on +the device. + +The substrate must also learn the same way everywhere. MOOTx01 +estates can federate: separate devices share and compare results. Two +devices that factor the same matrix must get the same answer. Two +devices that mine the same rules must also agree. Otherwise shared +recall falls apart. + +SubstrateML holds one agreement property across two implementations. +A Swift leg serves Apple platforms. A Rust leg, in `rust/`, serves +everything else. Both legs share one canonical pseudo-random number +generator, SplitMix64. Both use the same pinned seeds. Both use the +same tie-breaking rules and the same arithmetic order. The result is +bit-identical output on both legs. Conformance fixtures gate every +change. A fixture is a recorded input and output pair that both legs +must reproduce exactly. + +Two library-wide rules protect that promise. First, SubstrateML never +reads a clock. Every timestamp comes from the caller instead. Second, +no algorithm here holds hidden state. Each one is a pure function or +a plain value type. Each one is safe to run from any thread. ## How It Works -The thirty-eight source files form seven working groups. - -**Fingerprint and distance math.** A fingerprint is a short fixed-size code -computed from content; similar content gives similar fingerprints. -`FloatSimHash` projects float embedding vectors from external models into -the substrate's 256-bit fingerprint form. `CompositeDistance` blends -classification distance and fingerprint distance into the one score recall -ranks by. `LatticeDistance`, `PartialStateRecall`, and `ShingleSimilarity` -supply specialized distances. `MomentSummary` and `TemporalCompression` -OR-reduce many row fingerprints into one signature per time window, so the -system can ask "what was going on during this hour" as a single comparison. - -**Ingestion shaping.** `FeatureExtractors` turns raw ambient sensor samples -(health, location, calendar, screen time, telemetry) into fingerprinted -rows. `AuditLogFold` replays a row's append-only change log to reconstruct -its state at any point in time. `RowAttributeView` reshapes that same log -into flat attribute lists that the pattern miners consume. - -**Learning and decay.** `MatrixDecay` applies exponential half-life decay -to every statistics matrix — the system's forgetting mechanism. -`ActionOutcomeMatrix` tracks which actions succeed. `BradleyTerry` learns -per-row ranking strength from recall feedback. `LLMCalibrationCurve` tracks -how honest an LLM's confidence claims are. `Sampling` provides the -deterministic Normal, Gamma, and Beta samplers under Thompson-sampling -decisions. +The package holds thirty-eight source files. They form seven working +groups. + +**Fingerprint and distance math.** A fingerprint is a short fixed-size +code computed from content. Similar content gives similar +fingerprints. `FloatSimHash` projects float embedding vectors from +external models into the substrate's 256-bit fingerprint form. +`CompositeDistance` blends classification distance and fingerprint +distance into one score. Recall ranks candidates by that score. +`LatticeDistance`, `PartialStateRecall`, and `ShingleSimilarity` supply +specialized distances for narrower needs. `MomentSummary` and +`TemporalCompression` reduce many row fingerprints into one signature +per time window. That lets the system compare "what was going on +during this hour" as a single lookup. + +**Ingestion shaping.** `FeatureExtractors` turns raw ambient sensor +samples into fingerprinted rows. The samples come from health, +location, calendar, screen time, and telemetry sources. +`AuditLogFold` replays a row's append-only change log. That replay +reconstructs the row's state at any point in time. `RowAttributeView` +reshapes that same log into flat attribute lists. The pattern miners +consume those lists directly. + +**Learning and decay.** `MatrixDecay` applies exponential half-life +decay to every statistics matrix. This is the system's forgetting +mechanism. `ActionOutcomeMatrix` tracks which actions succeed. +`BradleyTerry` learns a per-row ranking strength from recall feedback. +`LLMCalibrationCurve` tracks how honest a model's confidence claims +are. `Sampling` provides deterministic Normal, Gamma, and Beta +samplers. Thompson-sampling decisions draw on these samplers. **Graph analytics.** The estate graph connects rows by association. -`CommunityDetection` (Louvain) finds its clusters. `EigenvalueCentrality` -scores each row's authority for keystone recall. `RandomWalks` wanders the -graph for exploratory recall. `NMFAlternatingLeastSquares` factors the -matrices into latent themes. `AnomalyDetection` flags unusual values. -`JacobiSVD` and `FFT` supply the deterministic linear algebra and rhythm -analysis beneath semantic embeddings and periodicity detection. These five -graph algorithms emit telemetry signals, named in `VizGraphSignals`, when -monitoring is enabled. +`CommunityDetection` runs Louvain clustering to find its communities. +`EigenvalueCentrality` scores each row's authority for keystone +recall. `RandomWalks` wanders the graph for exploratory recall. +`NMFAlternatingLeastSquares` factors the matrices into latent themes. +`AnomalyDetection` flags unusual values. `JacobiSVD` and `FFT` supply +deterministic linear algebra and rhythm analysis. Both sit beneath +semantic embeddings and periodicity detection. These five graph +algorithms emit telemetry signals when monitoring is enabled. The +signal names live in `VizGraphSignals`. **Pattern mining.** `AssociationRuleMining` and `AprioriMining` find -"when A is set, B tends to be set" rules. `FormalConceptAnalysis` and -`ConceptImplications` find exact groupings and always-true implications. -`TemporalCausalityFold` mines "X changed, then Y changed N minutes later" -statistics. `InformationTheory` supplies the entropy and divergence math. - -**Distillation.** Distillation compresses a cluster of related memories -into one condensed factoid. `DistillationScorer` decides whether a cluster -is coherent enough. `DeltaFeatureExtractor` and `TypedDecayWeighting` -handle trends and staleness. `DistillationPipeline` runs the whole -five-stage algorithm and emits the factoid plus its fingerprint. - -**Federation and privacy.** `PairingHandshake` lets two estates derive a -shared fingerprint basis with no extra network round trip. -`TierContributionFingerprint` packs an estate's shareable summary into a -fixed 64-byte wire format. `TierAscendingQuery` and `DPORReduction` add -differential-privacy noise and k-anonymity so aggregate answers never -expose any single estate's contribution. +rules of the form "when A is set, B tends to be set." +`FormalConceptAnalysis` and `ConceptImplications` find exact groupings +and implications that always hold. `TemporalCausalityFold` mines +statistics of the form "X changed, then Y changed some minutes later." +`InformationTheory` supplies the entropy and divergence math behind +these miners. + +**Distillation.** Distillation compresses a cluster of related +memories into one condensed factoid. `DistillationScorer` decides +whether a cluster is coherent enough to compress. +`DeltaFeatureExtractor` and `TypedDecayWeighting` handle trends and +staleness in the underlying features. `DistillationPipeline` runs the +whole five-stage algorithm. It emits the factoid plus its +fingerprint. + +**Federation and privacy.** `PairingHandshake` lets two estates derive +a shared fingerprint basis. It needs no extra network round trip. +`TierContributionFingerprint` packs an estate's shareable summary into +a fixed sixty-four-byte wire format. `TierAscendingQuery` and +`DPORReduction` add differential-privacy noise and k-anonymity. Both +protections keep any single estate's contribution hidden from +aggregate answers. ## How the Pieces Fit -Figure 1 shows the library's topology — its major parts and how data moves -between them. +Figure 1 shows the library's topology. It shows the major parts and +how data moves between them. ![Figure 1. Topology of SubstrateML](topology.svg) -*Figure 1. Topology of SubstrateML. Rows, audit events, and sensor samples -enter on the left. The shaping layer feeds the four algorithm families. -Dashed regions mark the substrate packages below and the telemetry seam; -all outputs return to the calling kits, never to storage.* - -SubstrateML has no single facade. Each algorithm family is its own entry -point, and the consuming kits — LocusKit, CognitionKit, GeniusLocusKit, -NeuronKit, and the dreaming daemon — call the piece they need. A kit is a -larger package that composes libraries into a subsystem; kits depend on -libs, never the reverse. The package depends downward only: on -`SubstrateTypes` for shared types, on `SubstrateKernel` for dispatched bit -kernels, and on `IntellectusLib`, the zero-dependency telemetry leaf. When -monitoring is off (the default), every telemetry emit costs one atomic -boolean load and nothing more. +*Figure 1. Topology of SubstrateML. Rows, audit events, and sensor +samples enter on the left. The shaping layer feeds the four algorithm +families. Dashed regions mark the substrate packages below and the +telemetry seam. All outputs return to the calling kits, never to +storage.* + +SubstrateML has no single facade. Each algorithm family is its own +entry point. The consuming kits call the piece each one needs. These +kits are LocusKit, CognitionKit, GeniusLocusKit, NeuronKit, and the +dreaming daemon. A kit is a larger package that composes libraries +into a subsystem. Kits depend on libs. Libs never depend back on +kits. + +The package depends downward only. It depends on `SubstrateTypes` for +shared types. It depends on `SubstrateKernel` for dispatched bit +kernels. It depends on `IntellectusLib`, the zero-dependency telemetry +leaf. Monitoring is off by default. When it is off, every telemetry +emit costs one atomic boolean load and nothing more. ## What Ships in the Package -The package ships the thirty-eight Swift sources, a matching test suite, -and the Rust port in `rust/` (crate `substrate-ml`, one module per Swift -file plus shared conformance tests). There are no bundled data artifacts; -every input is caller-supplied. Determinism is carried instead by pinned -constants — seeds, thresholds, half-lives, and bucket tables recorded in -the sources and locked by conformance vectors — so the same input produces -the same result on every platform, every time. +The package ships the thirty-eight Swift sources. It ships a matching +test suite. It also ships the Rust port in `rust/`, crate +`substrate-ml`, with one module per Swift file plus shared conformance +tests. There are no bundled data artifacts. Every input comes from the +caller. + +Determinism comes from pinned constants instead: seeds, thresholds, +half-lives, and bucket tables. These live in the sources. Conformance +vectors lock them in place. The same input therefore produces the +same result on every platform, every time. diff --git a/packages/libs/SubstrateTypes/docs/AGENT_MAP.md b/packages/libs/SubstrateTypes/docs/AGENT_MAP.md index ac1d88b..6af0b5f 100644 --- a/packages/libs/SubstrateTypes/docs/AGENT_MAP.md +++ b/packages/libs/SubstrateTypes/docs/AGENT_MAP.md @@ -67,7 +67,7 @@ sources: blob: e1b7fbed94b6505b596ac5701cfd7c693e7d2425 --- -# AGENT_MAP — SubstrateTypes +# AGENT_MAP: SubstrateTypes PURPOSE: layer 1 of the four-package substrate split. Pure data shapes (Row, Fingerprint256, LatticeAnchor, HLC, RowState/RowBitmaps, AuditEvent) plus @@ -76,161 +76,161 @@ canonical reference compute logically inseparable from those shapes transcendentals, zero I/O, zero pinned data artifacts. Consumers needing only substrate-shape (e.g. ConvergenceKit) depend on this package alone. -DEPS: imports Foundation only (no intra-SDK imports — this is the dependency +DEPS: imports Foundation only (no intra-SDK imports: this is the dependency floor). Imported by: SubstrateKernel, SubstrateML, SubstrateLib, EngramLib, IntellectusLib, and the root app package (Package.swift). SubstrateLib RETAINS the row-state automaton, the verb mechanics, and the full -multi-row substrate object — those are compute, not shape, and stay one +multi-row substrate object: those are compute, not shape, and stay one layer up. Rust port in `rust/src/` mirrors every module 1:1 (see DEPS note -in Rust Port and Conformance, DETAILS.md); no fixture directory — agreement +in Rust Port and Conformance, DETAILS.md); no fixture directory: agreement is verified by matched unit-test vectors in each module, not a shared fixtures/ dir. ENTRY POINTS (most callers need only these): -- Row.swift:23 `struct Row` — the substrate's central data shape -- Fingerprint256.swift:43 `struct Fingerprint256` — 256-bit structural coordinate -- LatticeAnchor.swift:12 `struct LatticeAnchor` — topic coordinate (hash of a lattice code + Q-ID) -- HLC.swift:37 `struct HLC` — ordering coordinate across replicas -- Hamming.swift:37 `Hamming.distance(_:_:blocks:)` — structural-similarity primitive -- GSetAuditLog.swift:134 `struct GSetAuditLog` — CRDT audit trail +- Row.swift:23 `struct Row`: the substrate's central data shape +- Fingerprint256.swift:43 `struct Fingerprint256`: 256-bit structural coordinate +- LatticeAnchor.swift:12 `struct LatticeAnchor`: topic coordinate (hash of a lattice code + Q-ID) +- HLC.swift:37 `struct HLC`: ordering coordinate across replicas +- Hamming.swift:37 `Hamming.distance(_:_:blocks:)`: structural-similarity primitive +- GSetAuditLog.swift:134 `struct GSetAuditLog`: CRDT audit trail ## Symbol Table ### Row shape and lifecycle -- Row.swift:19 `typealias RowId = UUID` — wire-identical to Rust `RowId(u128)` -- Row.swift:23 `struct Row: Sendable` — id, nounType, state, 3 bitmap columns, fingerprint, latticeAnchor, lineageId?, content? -- NounType.swift:12 `enum NounType: UInt8` — drawer=0/tunnel=1/kgFact=2/diaryEntry=3/proposal=4/association=5/learnedReference=6/ambientSample=7; wire-stable, never renumber -- RowState.swift:35 `enum RowState: UInt8` — 10 scale-gapped states (active=0..accepted=3, superseded=16..expired=19, rejected=32/tombstoned=33) -- RowState.swift:63 `enum RowStateCluster: UInt8` — a=active/becoming, b=superseded/historical, c=terminal -- RowState.swift:98 `RowState.cluster: RowStateCluster` — `(rawValue >> 4) & 0x3`; canonical active/retired partition, never re-derive by hand -- RowState.swift:90 `RowState.activeClusterUpperBoundRaw = 16` — the one boundary constant for storage-layer predicates that can't call `cluster` -- RowState.swift:115 `RowState.cluster(ofRawState:)` — classify from a raw byte, nil if undefined -- RowState.swift:122 `enum RowVerb: String` — 12 verbs the (SubstrateLib) automaton accepts -- RowState.swift:137 `enum RowStateError` — `.illegalTransition(RowState, RowVerb)` / `.violatesInvariant(String)` -- RowBitmaps.swift:37 `struct RowBitmaps: Sendable, Hashable, Codable` — adjective/operational/provenance Int64 columns -- RowBitmaps.swift:65 `RowBitmaps.field(_:) -> UInt8` — 36-field/6-bit uniform-grid view over the 3 named Int64 columns; UInt64 cast before shift (avoids Int64 arithmetic-shift sign bleed) -- RowBitmaps.swift:110 `RowBitmaps.fieldValues() -> [(field,value)]` — feeds MatrixO.applyRow / MatrixT.applyPair -- RowBitmaps.swift:151 `struct BitVector216: Sendable, Hashable` — dense 216-bit view; feeds MatrixF -- RowBitmaps.swift:167 `BitVector216.init(rowBitmaps:)` — real-row path, bits 60-71 always zero -- RowBitmaps.swift:191 `BitVector216.init(presenceBytes:)` — raw 27-byte harness/wire path, can set bits 60-71 +- Row.swift:19 `typealias RowId = UUID`: wire-identical to Rust `RowId(u128)` +- Row.swift:23 `struct Row: Sendable`: id, nounType, state, 3 bitmap columns, fingerprint, latticeAnchor, lineageId?, content? +- NounType.swift:12 `enum NounType: UInt8`: drawer=0/tunnel=1/kgFact=2/diaryEntry=3/proposal=4/association=5/learnedReference=6/ambientSample=7; wire-stable, never renumber +- RowState.swift:35 `enum RowState: UInt8`: 10 scale-gapped states (active=0..accepted=3, superseded=16..expired=19, rejected=32/tombstoned=33) +- RowState.swift:63 `enum RowStateCluster: UInt8`: a=active/becoming, b=superseded/historical, c=terminal +- RowState.swift:98 `RowState.cluster: RowStateCluster`: `(rawValue >> 4) & 0x3`; canonical active/retired partition, never re-derive by hand +- RowState.swift:90 `RowState.activeClusterUpperBoundRaw = 16`: the one boundary constant for storage-layer predicates that can't call `cluster` +- RowState.swift:115 `RowState.cluster(ofRawState:)`: classify from a raw byte, nil if undefined +- RowState.swift:122 `enum RowVerb: String`: 12 verbs the (SubstrateLib) automaton accepts +- RowState.swift:137 `enum RowStateError`: `.illegalTransition(RowState, RowVerb)` / `.violatesInvariant(String)` +- RowBitmaps.swift:37 `struct RowBitmaps: Sendable, Hashable, Codable`: adjective/operational/provenance Int64 columns +- RowBitmaps.swift:65 `RowBitmaps.field(_:) -> UInt8`: 36-field/6-bit uniform-grid view over the 3 named Int64 columns; UInt64 cast before shift (avoids Int64 arithmetic-shift sign bleed) +- RowBitmaps.swift:110 `RowBitmaps.fieldValues() -> [(field,value)]`: feeds MatrixO.applyRow / MatrixT.applyPair +- RowBitmaps.swift:151 `struct BitVector216: Sendable, Hashable`: dense 216-bit view; feeds MatrixF +- RowBitmaps.swift:167 `BitVector216.init(rowBitmaps:)`: real-row path, bits 60-71 always zero +- RowBitmaps.swift:191 `BitVector216.init(presenceBytes:)`: raw 27-byte harness/wire path, can set bits 60-71 ### Lattice anchor (topic coordinate) -- LatticeAnchor.swift:12 `struct LatticeAnchor: Hashable, Sendable` — udcCode: UInt64, qidPointer: UInt64 (0 = null) -- LatticeAnchor.swift:21 `isNull: Bool` — both halves zero -- LatticeAnchor.swift:41 `LatticeAnchor.udc(_:)` — FNV-1a64 hash of a UDC string, null Q-ID -- LatticeAnchor.swift:50 `LatticeAnchor.udcQid(_:qid:)` — hashes both; empty qidString ⇒ same as `udc(_:)` +- LatticeAnchor.swift:12 `struct LatticeAnchor: Hashable, Sendable`: udcCode: UInt64, qidPointer: UInt64 (0 = null) +- LatticeAnchor.swift:21 `isNull: Bool`: both halves zero +- LatticeAnchor.swift:41 `LatticeAnchor.udc(_:)`: FNV-1a64 hash of a UDC string, null Q-ID +- LatticeAnchor.swift:50 `LatticeAnchor.udcQid(_:qid:)`: hashes both; empty qidString ⇒ same as `udc(_:)` ### Time coordinates -- HLC.swift:37 `struct HLC: Hashable, Sendable, Codable, Comparable` — (physicalTime, logicalCount, nodeID), lexicographic order -- HLC.swift:58 `HLC.zero` — sentinel earliest value -- HLC.swift:77 `HLC.advanced()` — bump logicalCount, hold physical/node -- HLC.swift:99 `HLC.packed: UInt64` / :108 `init(packed:)` — lossy 8-byte federation form (40-bit ms ≈ 34yr range) -- HLC.swift:206 `HLC.wireBytes` / :224 `init(wireBytes:)` — lossless 16-byte form -- HLC.swift:125 `struct HLCGenerator: Sendable` — per-replica HLC state machine (Kulkarni et al. 2014) -- HLC.swift:139 `HLCGenerator.send(now:)` — local-event timestamp; strictly monotonic per replica -- HLC.swift:156 `HLCGenerator.receive(remote:now:)` — merge rule on message receipt; underlies GSetAuditLog convergence -- TimeRange.swift:11 `struct TimeRange: Sendable, Equatable` — closed [start,end] HLC interval; init preconditions end >= start -- TimeRange.swift:24 `contains(_:) -> Bool` — inclusive both ends -- AsOfCoordinate.swift:18 `enum AsOfCoordinate: Hashable, Sendable, Codable` — `.present` | `.asOf(HLC)`; enum discriminant prevents zero-HLC ⇔ "now" ambiguity -- SnapshotId.swift:15 `struct SnapshotId: Hashable, Sendable, Codable` — UUID wrapper, type-distinct from RowId/drawer/estate ids +- HLC.swift:37 `struct HLC: Hashable, Sendable, Codable, Comparable`: (physicalTime, logicalCount, nodeID), lexicographic order +- HLC.swift:58 `HLC.zero`: sentinel earliest value +- HLC.swift:77 `HLC.advanced()`: bump logicalCount, hold physical/node +- HLC.swift:99 `HLC.packed: UInt64` / :108 `init(packed:)`: lossy 8-byte federation form (40-bit ms ≈ 34yr range) +- HLC.swift:206 `HLC.wireBytes` / :224 `init(wireBytes:)`: lossless 16-byte form +- HLC.swift:125 `struct HLCGenerator: Sendable`: per-replica HLC state machine (Kulkarni et al. 2014) +- HLC.swift:139 `HLCGenerator.send(now:)`: local-event timestamp; strictly monotonic per replica +- HLC.swift:156 `HLCGenerator.receive(remote:now:)`: merge rule on message receipt; underlies GSetAuditLog convergence +- TimeRange.swift:11 `struct TimeRange: Sendable, Equatable`: closed [start,end] HLC interval; init preconditions end >= start +- TimeRange.swift:24 `contains(_:) -> Bool`: inclusive both ends +- AsOfCoordinate.swift:18 `enum AsOfCoordinate: Hashable, Sendable, Codable`: `.present` | `.asOf(HLC)`; enum discriminant prevents zero-HLC ⇔ "now" ambiguity +- SnapshotId.swift:15 `struct SnapshotId: Hashable, Sendable, Codable`: UUID wrapper, type-distinct from RowId/drawer/estate ids ### Fingerprint construction -- Fingerprint256.swift:43 `struct Fingerprint256: Hashable, Sendable, Codable` — block0..block3: UInt64 (256 bits); block0=Bitmap-LSH, block1=Lattice-LSH, block2=Lineage+Temporal, block3=Channel+Source -- Fingerprint256.swift:59 `Fingerprint256.zero` — OR-identity / null-block value -- Fingerprint256.swift:88 `with(bit:set:) -> Fingerprint256` — value-semantic bit set/clear (replaces old mutating setBit) -- Fingerprint256.swift:120 `union(_:) -> Fingerprint256` — bitwise OR -- Fingerprint256.swift:153 `Fingerprint256.fromBits(_:)` — from [Bool] of 256 -- Fingerprint256.swift:166/178 `wireBytes` / `init(wireBytes:)` — 32-byte LE canonical form; `toBytes()`/`fromBytes(_:)` are throws-free aliases -- Fingerprint256.swift:252 `zip4(_:_:)` — pairwise per-block binary op; the ONE unroll every block-wise op in this package (and Hamming/ORReduce/BitwiseArithmetic) must route through -- Fingerprint256.swift:270 `Fingerprint256.reduce4(_:_:)` — block-wise fold over a fingerprint sequence -- Fingerprint256.swift:281 `map4(_:)` — per-block unary op -- Fingerprint256.swift:296 `popcount() -> Int` — sum of 4 block popcounts -- Fingerprint256.swift:313/331 `zip4Batch` / `map4Batch` — vectorize across row arrays, not across one fingerprint's blocks -- HyperplaneFamily.swift:30 `struct Hyperplane: Sendable, Codable, Equatable` — ±1 plane as (positiveMask, negativeMask) bit-pair; 0/0 = sparse zero entry -- HyperplaneFamily.swift:53 `sign(over:) -> Bool` — popcount(v&pos) > popcount(v&neg); tie ⇒ false -- HyperplaneFamily.swift:68 `struct HyperplaneFamily: Sendable, Codable, Equatable` — exactly 64 planes, fixed blockIndex 0..3 + inputBitLength (192 for block0, 64 for 1-3) -- HyperplaneFamily.swift:95 `HyperplaneFamily.generate(seed:blockIndex:inputBitLength:density:)` — deterministic from 32-byte seed; CONSTITUTIONAL: seeds fixed at estate creation, never rotate -- HyperplaneFamily.swift:147 `canonicalHash() -> UInt64` — FNV-1a over canonical wire form, for pairing-audit labels -- HyperplaneFamily.swift:247 `HyperplaneFamily.blockFamilies(baseSeed:density:)` — canonical 4-family generator; fixes prior bug of reusing 1 seed across blocks + assuming uniform 64-bit width -- HyperplaneFamily.swift:261 `diversifiedSeed(base:blockIndex:)` / :273 `expandSeed64(_:)` — per-block seed derivation (SplitMix64-style) -- FloatSimHashPlanes.swift:30 `struct FloatSimHashPlanes: Sendable, Equatable` — materialized ±1 planes for the FLOAT-input SimHash path; lives here so SubstrateKernel (applies) and SubstrateML (generates) both reach it without a layering inversion -- SimHash.swift:37 `SimHash.block(over:family:) -> UInt64` — one 64-bit block: bit k = sign() -- SimHash.swift:52 `SimHash.fingerprint(bitmapInput:latticeInput:lineageTemporalInput:channelSourceInput:families:)` — assembles all 4 blocks; families order MUST be H_0..H_3 -- SimHash.swift:74 `SimHash.fingerprintBatch(...)` — reference loop; hardware backends must match bit-for-bit -- SimHash.swift:102 `SimHash.fingerprint(fromSubhashes:hyperplanes:)` — 4-subhash convenience path -- SimHashInput (SimHash.swift:123) — :128 `bitmap(...)`, :140 `lattice(...)`, :156 `lineageTemporal(...)`, :178 `channelSource(...)` — fixed bit-offset input-vector assemblers per cookbook §3.2-3.5; offset mismatch ⇒ silently incompatible fingerprints +- Fingerprint256.swift:43 `struct Fingerprint256: Hashable, Sendable, Codable`: block0..block3: UInt64 (256 bits); block0=Bitmap-LSH, block1=Lattice-LSH, block2=Lineage+Temporal, block3=Channel+Source +- Fingerprint256.swift:59 `Fingerprint256.zero`: OR-identity / null-block value +- Fingerprint256.swift:88 `with(bit:set:) -> Fingerprint256`: value-semantic bit set/clear (replaces old mutating setBit) +- Fingerprint256.swift:120 `union(_:) -> Fingerprint256`: bitwise OR +- Fingerprint256.swift:153 `Fingerprint256.fromBits(_:)`: from [Bool] of 256 +- Fingerprint256.swift:166/178 `wireBytes` / `init(wireBytes:)`: 32-byte LE canonical form; `toBytes()`/`fromBytes(_:)` are throws-free aliases +- Fingerprint256.swift:252 `zip4(_:_:)`: pairwise per-block binary op; the ONE unroll every block-wise op in this package (and Hamming/ORReduce/BitwiseArithmetic) must route through +- Fingerprint256.swift:270 `Fingerprint256.reduce4(_:_:)`: block-wise fold over a fingerprint sequence +- Fingerprint256.swift:281 `map4(_:)`: per-block unary op +- Fingerprint256.swift:296 `popcount() -> Int`: sum of 4 block popcounts +- Fingerprint256.swift:313/331 `zip4Batch` / `map4Batch`: vectorize across row arrays, not across one fingerprint's blocks +- HyperplaneFamily.swift:30 `struct Hyperplane: Sendable, Codable, Equatable`: ±1 plane as (positiveMask, negativeMask) bit-pair; 0/0 = sparse zero entry +- HyperplaneFamily.swift:53 `sign(over:) -> Bool`: popcount(v&pos) > popcount(v&neg); tie ⇒ false +- HyperplaneFamily.swift:68 `struct HyperplaneFamily: Sendable, Codable, Equatable`: exactly 64 planes, fixed blockIndex 0..3 + inputBitLength (192 for block0, 64 for 1-3) +- HyperplaneFamily.swift:95 `HyperplaneFamily.generate(seed:blockIndex:inputBitLength:density:)`: deterministic from 32-byte seed; CONSTITUTIONAL: seeds fixed at estate creation, never rotate +- HyperplaneFamily.swift:147 `canonicalHash() -> UInt64`: FNV-1a over canonical wire form, for pairing-audit labels +- HyperplaneFamily.swift:247 `HyperplaneFamily.blockFamilies(baseSeed:density:)`: canonical 4-family generator; fixes prior bug of reusing 1 seed across blocks + assuming uniform 64-bit width +- HyperplaneFamily.swift:261 `diversifiedSeed(base:blockIndex:)` / :273 `expandSeed64(_:)`: per-block seed derivation (SplitMix64-style) +- FloatSimHashPlanes.swift:30 `struct FloatSimHashPlanes: Sendable, Equatable`: materialized ±1 planes for the FLOAT-input SimHash path; lives here so SubstrateKernel (applies) and SubstrateML (generates) both reach it without a layering inversion +- SimHash.swift:37 `SimHash.block(over:family:) -> UInt64`: one 64-bit block: bit k = sign() +- SimHash.swift:52 `SimHash.fingerprint(bitmapInput:latticeInput:lineageTemporalInput:channelSourceInput:families:)`: assembles all 4 blocks; families order MUST be H_0..H_3 +- SimHash.swift:74 `SimHash.fingerprintBatch(...)`: reference loop; hardware backends must match bit-for-bit +- SimHash.swift:102 `SimHash.fingerprint(fromSubhashes:hyperplanes:)`: 4-subhash convenience path +- SimHashInput (SimHash.swift:123): :128 `bitmap(...)`, :140 `lattice(...)`, :156 `lineageTemporal(...)`, :178 `channelSource(...)`: fixed bit-offset input-vector assemblers per cookbook §3.2-3.5; offset mismatch ⇒ silently incompatible fingerprints ### Fingerprint algebra - Hamming.swift:21 `typealias HammingDistance = Hamming` -- Hamming.swift:37 `Hamming.distance(_:_:blocks:) -> Int` — XOR+popcount; `.all` fast path is `zip4(^).popcount()` -- Hamming.swift:54 `Hamming.similarity(_:_:blocks:) -> Double` — 1 - distance/max, max = 64*blockCount -- BlockMask.swift:26 `struct BlockMask: OptionSet` — .block0/.block1/.block2/.block3, .all, .none; replaced a Set-per-call allocation hot-path -- BlockMask.swift:42 `blockCount: Int` — popcount of rawValue -- BitwiseArithmetic.swift:31 `intersect(_:_:)` — AND (zip4 &) -- BitwiseArithmetic.swift:45 `difference(_:_:)` — XOR (zip4 ^) -- BitwiseArithmetic.swift:63 `prototype(_:) -> Fingerprint256` — cohort majority vote via CountVector256.fold(...).majorityVote() -- BitwiseArithmetic.swift:77 `indirect enum FingerprintBuilder` — literal/intersect/difference/prototypeOf; `.evaluate()` walks the tree -- ORReduce.swift:31 `ORReduce.reduce(_:) -> Fingerprint256` — OR-fold, zero for empty (identity); delegates to `Fingerprint256.reduce4(_:|:)` -- ORReduce.swift:46 `ORReduce.reduce(_:blocks:defaults:)` — block-subset OR, unselected blocks pass through `defaults` -- CountVector256.swift:41 `struct CountVector256: Sendable, Equatable, Codable` — counts:[UInt32] (256), n:UInt32; wrapping arithmetic (`&+=`) -- CountVector256.swift:52 `CountVector256.zero` — fold/merge identity -- CountVector256.swift:75 `accumulate(_:)` — fold one fingerprint (leaf step) -- CountVector256.swift:94/103 `merge(_:)` / `+` — commutative/associative tree-fold step; order-independent -- CountVector256.swift:120 `majorityVote() -> Fingerprint256` — bit set iff `2*count > n` (strict; exact tie does NOT set); n==0 ⇒ zero -- CountVector256.swift:134 `profile() -> [Float]` — per-bit Bernoulli parameter, not stored, recomputed on demand -- CountVector256.swift:148 `CountVector256.fold(_:)` — reference whole-array accumulate; kernel-layer vectorized backends gate against this -- FNV.swift:25 `FNV.hash64(_:) -> UInt64` — offset 0xCBF29CE484222325, prime 0x100000001B3 -- FNV.swift:40 `FNV.hash32(_:) -> UInt32` — INDEPENDENT hash family, not a truncation of hash64 -- FNV.swift:57 `FNV.hash16(_:) -> UInt16` — low-16 fold of hash64 (NOT a from-scratch FNV variant) +- Hamming.swift:37 `Hamming.distance(_:_:blocks:) -> Int`: XOR+popcount; `.all` fast path is `zip4(^).popcount()` +- Hamming.swift:54 `Hamming.similarity(_:_:blocks:) -> Double`: 1 - distance/max, max = 64*blockCount +- BlockMask.swift:26 `struct BlockMask: OptionSet`: .block0/.block1/.block2/.block3, .all, .none; replaced a Set-per-call allocation hot-path +- BlockMask.swift:42 `blockCount: Int`: popcount of rawValue +- BitwiseArithmetic.swift:31 `intersect(_:_:)`: AND (zip4 &) +- BitwiseArithmetic.swift:45 `difference(_:_:)`: XOR (zip4 ^) +- BitwiseArithmetic.swift:63 `prototype(_:) -> Fingerprint256`: cohort majority vote via CountVector256.fold(...).majorityVote() +- BitwiseArithmetic.swift:77 `indirect enum FingerprintBuilder`: literal/intersect/difference/prototypeOf; `.evaluate()` walks the tree +- ORReduce.swift:31 `ORReduce.reduce(_:) -> Fingerprint256`: OR-fold, zero for empty (identity); delegates to `Fingerprint256.reduce4(_:|:)` +- ORReduce.swift:46 `ORReduce.reduce(_:blocks:defaults:)`: block-subset OR, unselected blocks pass through `defaults` +- CountVector256.swift:41 `struct CountVector256: Sendable, Equatable, Codable`: counts:[UInt32] (256), n:UInt32; wrapping arithmetic (`&+=`) +- CountVector256.swift:52 `CountVector256.zero`: fold/merge identity +- CountVector256.swift:75 `accumulate(_:)`: fold one fingerprint (leaf step) +- CountVector256.swift:94/103 `merge(_:)` / `+`: commutative/associative tree-fold step; order-independent +- CountVector256.swift:120 `majorityVote() -> Fingerprint256`: bit set iff `2*count > n` (strict; exact tie does NOT set); n==0 ⇒ zero +- CountVector256.swift:134 `profile() -> [Float]`: per-bit Bernoulli parameter, not stored, recomputed on demand +- CountVector256.swift:148 `CountVector256.fold(_:)`: reference whole-array accumulate; kernel-layer vectorized backends gate against this +- FNV.swift:25 `FNV.hash64(_:) -> UInt64`: offset 0xCBF29CE484222325, prime 0x100000001B3 +- FNV.swift:40 `FNV.hash32(_:) -> UInt32`: INDEPENDENT hash family, not a truncation of hash64 +- FNV.swift:57 `FNV.hash16(_:) -> UInt16`: low-16 fold of hash64 (NOT a from-scratch FNV variant) ### Integrity hashing -- MerkleDomain.swift:20 `enum MerkleDomain` — leaf=0x00, interior=0x01, tombstone=0x02, commitment=0x03; conformance-frozen forever -- ContentHash.swift:20 `struct ContentHash: Hashable, Sendable, Codable` — 32-byte SHA-256 leaf-payload digest; private storage, `bytes`/`hexString` accessors -- ContentHash.swift:45 `ContentHash.tombstone` — literal bytes (SHA256 of MerkleDomain.tombstone); literal because layer-1 can't import SubstrateKernel's SHA256 -- MerkleRoot.swift:16 `struct MerkleRoot: Hashable, Sendable, Codable` — 32-byte subtree-summary hash; NOT interchangeable with ContentHash (type-distinct on purpose) -- MerkleRoot.swift:41 `MerkleRoot.empty` — literal bytes (SHA256 of MerkleDomain.interior), same layering reason +- MerkleDomain.swift:20 `enum MerkleDomain`: leaf=0x00, interior=0x01, tombstone=0x02, commitment=0x03; conformance-frozen forever +- ContentHash.swift:20 `struct ContentHash: Hashable, Sendable, Codable`: 32-byte SHA-256 leaf-payload digest; private storage, `bytes`/`hexString` accessors +- ContentHash.swift:45 `ContentHash.tombstone`: literal bytes (SHA256 of MerkleDomain.tombstone); literal because layer-1 can't import SubstrateKernel's SHA256 +- MerkleRoot.swift:16 `struct MerkleRoot: Hashable, Sendable, Codable`: 32-byte subtree-summary hash; NOT interchangeable with ContentHash (type-distinct on purpose) +- MerkleRoot.swift:41 `MerkleRoot.empty`: literal bytes (SHA256 of MerkleDomain.interior), same layering reason ### Audit trail (CRDT) -- AuditEvent.swift:15 `struct AuditEvent: Sendable` — eventID+hlc = idempotence key; before/afterBitmaps, before/afterLatticeAnchor, actor, reason? -- AuditEvent.swift:63 `withReason(_:) -> AuditEvent` — attach a reason post hoc (AuditGate.admit itself is reason-less) -- GSetAuditLog.swift:35 `struct AuditEntry: Hashable, Sendable, Codable` — id = 32-byte content hash (SHA-256 over wire fields); dedupe key -- GSetAuditLog.swift:62 `enum AuditVerb: String` — 9 cookbook verbs + migrate/dreamCompact -- GSetAuditLog.swift:95 `enum AuditValue: Hashable, Sendable, Codable` — bitmap/string/fingerprint/integer; HAND-WRITTEN Codable ⇒ externally-tagged `{"bitmap":42}` shape matching Rust serde, NOT Swift's default `{"bitmap":{"_0":42}}` -- GSetAuditLog.swift:134 `struct GSetAuditLog: Sendable, Codable` — internal store keyed by id (O(1) dedupe); WIRE FORM is a sorted array (`{"entries":[...]}`), not the hash map -- GSetAuditLog.swift:180 `add(_:)` — idempotent insert -- GSetAuditLog.swift:187 `merge(_:)` — CRDT join = set union; commutative/associative/idempotent -- GSetAuditLog.swift:198 `orderedEntries` — full HLC-order replay (drives projection) -- GSetAuditLog.swift:204 `entries(forRow:)` — HLC-order, one row (drives row-state automaton in SubstrateLib) -- GSetAuditLog.swift:212 `entries(since:)` — HLC-order, exclusive cutoff (sync delta) +- AuditEvent.swift:15 `struct AuditEvent: Sendable`: eventID+hlc = idempotence key; before/afterBitmaps, before/afterLatticeAnchor, actor, reason? +- AuditEvent.swift:63 `withReason(_:) -> AuditEvent`: attach a reason post hoc (AuditGate.admit itself is reason-less) +- GSetAuditLog.swift:35 `struct AuditEntry: Hashable, Sendable, Codable`: id = 32-byte content hash (SHA-256 over wire fields); dedupe key +- GSetAuditLog.swift:62 `enum AuditVerb: String`: 9 cookbook verbs + migrate/dreamCompact +- GSetAuditLog.swift:95 `enum AuditValue: Hashable, Sendable, Codable`: bitmap/string/fingerprint/integer; HAND-WRITTEN Codable ⇒ externally-tagged `{"bitmap":42}` shape matching Rust serde, NOT Swift's default `{"bitmap":{"_0":42}}` +- GSetAuditLog.swift:134 `struct GSetAuditLog: Sendable, Codable`: internal store keyed by id (O(1) dedupe); WIRE FORM is a sorted array (`{"entries":[...]}`), not the hash map +- GSetAuditLog.swift:180 `add(_:)`: idempotent insert +- GSetAuditLog.swift:187 `merge(_:)`: CRDT join = set union; commutative/associative/idempotent +- GSetAuditLog.swift:198 `orderedEntries`: full HLC-order replay (drives projection) +- GSetAuditLog.swift:204 `entries(forRow:)`: HLC-order, one row (drives row-state automaton in SubstrateLib) +- GSetAuditLog.swift:212 `entries(since:)`: HLC-order, exclusive cutoff (sync delta) ### Population-statistics matrices -- MatrixF.swift:24 `struct MatrixF: Sendable, Equatable` — 216-cell Int64 field-presence counts; NO decay -- MatrixF.swift:31 `fieldCount`/`bitsPerField`/`cellCount` — ALIASES onto RowBitmaps' constants (must match by definition) -- MatrixF.swift:76 `applyRow(delta:bitVector:)` — +1 capture / -1 expunge / two calls for mutate -- MatrixC.swift:26 `struct MatrixC: Sendable, Equatable` — 216-cell Float32 marginal probabilities; derived only, NO decay -- MatrixC.swift:67 `MatrixC.derive(from:nRows:)` — Int64→Double→Float32 cast order is the cross-language bit-identity contract; nRows==0 ⇒ all zero -- MatrixO.swift:38 `struct CooccurrenceKey: Hashable, Comparable, Sendable` — packed UInt32 (fieldI|valueI|fieldJ|valueJ, 8 bits each) -- MatrixO.swift:70 `struct MatrixO: Sendable, Equatable` — sorted sparse cell list (NOT a Dictionary — deterministic iteration/serialization); decay half-life 365 days (applied elsewhere, by MatrixDecay) -- MatrixO.swift:165 `applyRow(delta:fieldValues:)` — ALL ordered pairs incl. i==j; NOT required symmetric in storage (unordered pair ⇒ 2 cells) -- MatrixT.swift:43 `struct CausalityKey: Hashable, Comparable, Sendable` — packed UInt64, adds lagBucket (0..7) to CooccurrenceKey's shape -- MatrixT.swift:84 `struct MatrixT: Sendable, Equatable` — ASYMMETRIC by design (source→target ≠ target→source); decay half-life 90 days -- MatrixT.swift:89 `bucketEdgesMinutes = [1,2,4,8,16,32,64,128]` — log-spaced lag buckets -- MatrixT.swift:99 `MatrixT.lagBucket(forMinutes:)` — nil outside [1,256); applyPair is then a no-op -- ThreeDBitTensor.swift:36 `struct ThreeDBitTensor: Sendable` — 6 bit-slices (one per bit position), row-major within each slice; hot-path layout, ~27MiB at 1M rows -- ThreeDBitTensor.swift:71 `setValue(row:field:value:)` — precondition value < 64 (6-bit width, I-6) -- ThreeDBitTensor.swift:107 `scanFieldEquals(field:value:) -> [UInt8]` — O(rowCount/8) per bit-slice, NOT yet word-vectorized -- ThreeDBitTensor.swift:140 `reserveCapacity(_:)` — grow-only, no-op if not larger, preserves existing data +- MatrixF.swift:24 `struct MatrixF: Sendable, Equatable`: 216-cell Int64 field-presence counts; NO decay +- MatrixF.swift:31 `fieldCount`/`bitsPerField`/`cellCount`: ALIASES onto RowBitmaps' constants (must match by definition) +- MatrixF.swift:76 `applyRow(delta:bitVector:)`: +1 capture / -1 expunge / two calls for mutate +- MatrixC.swift:26 `struct MatrixC: Sendable, Equatable`: 216-cell Float32 marginal probabilities; derived only, NO decay +- MatrixC.swift:67 `MatrixC.derive(from:nRows:)`: Int64→Double→Float32 cast order is the cross-language bit-identity contract; nRows==0 ⇒ all zero +- MatrixO.swift:38 `struct CooccurrenceKey: Hashable, Comparable, Sendable`: packed UInt32 (fieldI|valueI|fieldJ|valueJ, 8 bits each) +- MatrixO.swift:70 `struct MatrixO: Sendable, Equatable`: sorted sparse cell list (NOT a Dictionary: deterministic iteration/serialization); decay half-life 365 days (applied elsewhere, by MatrixDecay) +- MatrixO.swift:165 `applyRow(delta:fieldValues:)`: ALL ordered pairs incl. i==j; NOT required symmetric in storage (unordered pair ⇒ 2 cells) +- MatrixT.swift:43 `struct CausalityKey: Hashable, Comparable, Sendable`: packed UInt64, adds lagBucket (0..7) to CooccurrenceKey's shape +- MatrixT.swift:84 `struct MatrixT: Sendable, Equatable`: ASYMMETRIC by design (source→target ≠ target→source); decay half-life 90 days +- MatrixT.swift:89 `bucketEdgesMinutes = [1,2,4,8,16,32,64,128]`: log-spaced lag buckets +- MatrixT.swift:99 `MatrixT.lagBucket(forMinutes:)`: nil outside [1,256); applyPair is then a no-op +- ThreeDBitTensor.swift:36 `struct ThreeDBitTensor: Sendable`: 6 bit-slices (one per bit position), row-major within each slice; hot-path layout, ~27MiB at 1M rows +- ThreeDBitTensor.swift:71 `setValue(row:field:value:)`: precondition value < 64 (6-bit width, I-6) +- ThreeDBitTensor.swift:107 `scanFieldEquals(field:value:) -> [UInt8]`: O(rowCount/8) per bit-slice, NOT yet word-vectorized +- ThreeDBitTensor.swift:140 `reserveCapacity(_:)`: grow-only, no-op if not larger, preserves existing data ### Recall wire vocabulary -- RecallTypes.swift:55 `struct RecallScore: Equatable, Sendable` — (RowId, Float32); score meaning is PER-PRIMITIVE, normalize before combining -- RecallTypes.swift:69 `struct DistanceBreakdown: Equatable, Sendable` — lattice/fingerprint/temporal/bitmap contributions, each [0,1] -- RecallTypes.swift:89 `struct RecallResult: Sendable` — rows + breakdown + confidenceInterval? + primitiveName (drives RRF/MMR composition) -- RecallTypes.swift:112 `struct RowProjection: Sendable` — rowId/captureHLC/fingerprint/lattice/bitmaps/rowState; DELIBERATELY omits verbatim content + rung-2 metadata +- RecallTypes.swift:55 `struct RecallScore: Equatable, Sendable`: (RowId, Float32); score meaning is PER-PRIMITIVE, normalize before combining +- RecallTypes.swift:69 `struct DistanceBreakdown: Equatable, Sendable`: lattice/fingerprint/temporal/bitmap contributions, each [0,1] +- RecallTypes.swift:89 `struct RecallResult: Sendable`: rows + breakdown + confidenceInterval? + primitiveName (drives RRF/MMR composition) +- RecallTypes.swift:112 `struct RowProjection: Sendable`: rowId/captureHLC/fingerprint/lattice/bitmaps/rowState; DELIBERATELY omits verbatim content + rung-2 metadata ## INVARIANTS / GOTCHAS - LAYERING FLOOR: this package imports Foundation only. Never add an - intra-SDK import here — that is what would invert the four-package split + intra-SDK import here: that is what would invert the four-package split (SubstrateTypes → SubstrateKernel → SubstrateML, with SubstrateLib orchestrating all three). `ContentHash.tombstone` and `MerkleRoot.empty` are byte literals, NOT computed SHA-256 calls, for exactly this reason; @@ -240,7 +240,7 @@ ENTRY POINTS (most callers need only these): `BitwiseArithmetic.intersect/difference` all delegate to it. Do not reintroduce a hand-unrolled block0/block1/block2/block3 loop anywhere else. - `RowBitmaps.field(_:)` casts to `UInt64` before shifting. Do not shift the - raw `Int64` — arithmetic (sign-extending) right shift on a negative Int64 + raw `Int64`: arithmetic (sign-extending) right shift on a negative Int64 bleeds the sign bit into the field value. Also guards `shift >= 64` explicitly; Swift masks shift amounts by 63 and would otherwise compute the wrong result for fields 11 (shift=66) silently. @@ -253,10 +253,10 @@ ENTRY POINTS (most callers need only these): never rotated. Rotating changes every fingerprint the estate has ever produced and breaks CRDT convergence across replicas. `blockFamilies` is the only sanctioned way to derive all four block families from one - base seed — do not reuse one un-diversified seed across blocks. + base seed: do not reuse one un-diversified seed across blocks. - `CountVector256.majorityVote()` uses STRICT `2*count > n`; an exact tie does not set the bit. This convention is identical across every kernel - backend and both language ports — do not change the inequality direction. + backend and both language ports: do not change the inequality direction. - `GSetAuditLog`'s wire format is a sorted array (`{"entries":[...]}`), not its internal `[id: AuditEntry]` dictionary. `AuditValue`'s Codable is hand-written to match Rust serde's externally-tagged shape. Do not let @@ -278,7 +278,7 @@ ENTRY POINTS (most callers need only these): `SnapshotId` into another typealias. - `ContentHash` and `MerkleRoot` are both 32-byte hashes but are distinct, non-interchangeable types (payload digest vs. subtree summary). Same - pattern for `RecallScore.score` — meaning is per-primitive, never compare + pattern for `RecallScore.score`: meaning is per-primitive, never compare raw scores across primitives without normalizing first. - No pinned data artifacts ship in this package (contrast with LatticeLib). Every value is either caller-supplied (a seed, a row's fields) or a pure diff --git a/packages/libs/SubstrateTypes/docs/DETAILS.md b/packages/libs/SubstrateTypes/docs/DETAILS.md index e77a9fc..1fb2cb9 100644 --- a/packages/libs/SubstrateTypes/docs/DETAILS.md +++ b/packages/libs/SubstrateTypes/docs/DETAILS.md @@ -70,412 +70,439 @@ sources: # SubstrateTypes Details This document walks through every source file in the package. Read -`OVERVIEW.md` first for the big picture. Files appear here in conceptual -order: the row shape and its lifecycle, the two coordinate systems a row -carries (lattice anchor and time), the fingerprint construction pipeline and -its algebra, integrity hashing, the audit trail, the population-statistics -matrices, and finally the recall wire vocabulary. +`OVERVIEW.md` first for the big picture. Files appear here in one order. +First comes the row shape and its lifecycle. Next come the two coordinate +systems a row carries: the lattice anchor and time. Then comes the +fingerprint construction pipeline and its algebra. After that comes +integrity hashing, the audit trail, and the population-statistics +matrices. The recall wire vocabulary closes the document. ## NounType.swift -This file provides `NounType`, an eight-case enum naming the kind of thing a -row holds: a drawer, a tunnel, a knowledge-graph fact, a diary entry, a -proposal, an association, a learned reference, or an ambient sample. - -The enum is backed by `UInt8` raw values (0 through 7) rather than a string, -because the noun type is stored in a compact wire format alongside the rest -of a row. The file's opening comment states the values are wire-stable and -must never be renumbered: a stored row's noun type is read back by its raw -number, so changing a number would silently reclassify every row already on +This file provides `NounType`. It is an eight-case enum. It names the kind +of thing a row holds. A row can be a drawer, a tunnel, or a +knowledge-graph fact. It can be a diary entry, a proposal, or an +association. It can also be a learned reference or an ambient sample. + +The enum uses `UInt8` raw values, zero through seven, instead of a string. +A row's noun type is stored in a compact wire format alongside the rest of +the row. That format needs a small fixed-size number, not a string. The +file's opening comment states a rule. These values are wire-stable. They +must never be renumbered. A stored row's noun type is read back by its raw +number. Changing a number would silently reclassify every row already on disk. ## RowState.swift -This file provides the three pure-data types that describe a row's -lifecycle: `RowState` (the ten states a row can be in), `RowVerb` (the twelve -mutations the row-state automaton accepts), and `RowStateError` (the typed -failure a rejected transition returns). The automaton that enforces which -transitions are legal is compute, not data, so it remains one layer up, in -SubstrateLib; this file only names the states and verbs the automaton -operates on. - -The ten states use deliberately spaced raw values — 0 to 3, then a gap, then -16 to 19, another gap, then 32 and 33 — rather than a plain 0-to-9 count. -This spacing lets any consumer recover a state's lifecycle cluster with one -shift-and-mask operation instead of a lookup table. `RowState.cluster` reads -`(rawValue >> 4) & 0x3` to classify a state as cluster A (active or -becoming: active, pending, contested, accepted), cluster B (superseded or -historical: superseded, decayed, withdrawn, expired — retired but -revivable), or cluster C (terminal: rejected, tombstoned — retired and -final). `RowState.isActiveCluster` is the convenience most callers want: a -single true-or-false answer to "is this row currently believed." - -`RowState.cluster(ofRawState:)` performs the same classification starting -from a raw `UInt8` rather than a typed `RowState`, for code that has only -read a bare byte out of a persisted bitmap and has not decoded it yet. -`RowState.activeClusterUpperBoundRaw` names the one boundary value (16) a -storage-layer predicate can compare against directly — for example, in a SQL -`WHERE` clause — when it cannot call a Swift function; the file is explicit -that any code holding a decoded `RowState` should prefer `cluster` over a -hand-rolled numeric comparison, because a future state added inside one of -the gaps would silently misclassify under a magic-number boundary but not -under the shift-and-mask. - -`RowState.description` and `RowStateError.description` supply the exact -lowercase English strings ("active", "pending", and so on, and the compact -"active --reject-->" form) that downstream parsers depend on for parity with -the Rust port's `Display` implementation. Both are written as explicit -`switch` statements rather than the default `String(describing:)`, because -calling that default from inside a type's own `description` recurses forever. +This file provides three pure-data types. They describe a row's +lifecycle. `RowState` names the ten states a row can be in. `RowVerb` +names the twelve mutations the row-state automaton accepts. +`RowStateError` is the typed failure a rejected transition returns. The +automaton that enforces which transitions are legal is compute, not data. +It stays one layer up, in SubstrateLib. This file only names the states +and verbs the automaton works with. + +The ten states use deliberately spaced raw values. They run zero to +three, then a gap, then sixteen to nineteen, another gap, then thirty-two +and thirty-three. This is not a plain zero-to-nine count. The spacing lets +any consumer recover a state's lifecycle cluster with one shift-and-mask +operation. No lookup table is needed. `RowState.cluster` reads +`(rawValue >> 4) & 0x3`. This classifies a state into one of three +clusters. Cluster A is active or becoming: active, pending, contested, +accepted. Cluster B is superseded or historical: superseded, decayed, +withdrawn, expired. These are retired but revivable. Cluster C is +terminal: rejected, tombstoned. These are retired and final. +`RowState.isActiveCluster` is the convenience most callers want. It gives +a single true-or-false answer to "is this row currently believed." + +`RowState.cluster(ofRawState:)` performs the same classification from a +raw `UInt8`. It works even when code has only read a bare byte out of a +persisted bitmap and has not decoded it yet. +`RowState.activeClusterUpperBoundRaw` names one boundary value: sixteen. +A storage-layer predicate can compare against this directly. One example +is a SQL `WHERE` clause, when it cannot call a Swift function. The file is +explicit about one thing. Code holding a decoded `RowState` should prefer +`cluster` over a hand-rolled numeric comparison. A future state added +inside one of the gaps would silently misclassify under a magic-number +boundary. It would not misclassify under the shift-and-mask. + +`RowState.description` and `RowStateError.description` supply exact +lowercase English strings. Examples are "active", "pending", and the +compact "active --reject-->" form. Downstream parsers depend on these +strings for parity with the Rust port's `Display` implementation. Both are +written as explicit `switch` statements. They do not use the default +`String(describing:)`. Calling that default from inside a type's own +`description` recurses forever. ## RowBitmaps.swift -This file provides `RowBitmaps`, the named layout over a row's three bitmap -columns, and `BitVector216`, a dense 216-bit view over that same data for -consumers that want to treat it as a flat grid. - -A row carries three `Int64` columns — adjective, operational, and -provenance — and each column packs several independently named fields at -specific bit ranges. For some purposes (updating the population-statistics -matrices), it is easier to think of these three columns as one uniform grid -of thirty-six fields of six bits each, even though the real fields are not -laid out that uniformly. `RowBitmaps.field(_:)` performs this translation: -given a field index from 0 to 35, it locates the right column and bit -offset and returns that field's six-bit value. The function carries two -defensive casts the comment explains in detail: a shift-amount guard, -because Swift silently wraps an out-of-range shift instead of returning -zero, and a cast to `UInt64` before shifting, because right-shifting a -negative `Int64` sign-extends and would leak the sign bit into a field's -value. +This file provides `RowBitmaps`. It is the named layout over a row's +three bitmap columns. The file also provides `BitVector216`, a dense +216-bit view over that same data, for consumers that want to treat it as +a flat grid. + +A row carries three `Int64` columns: adjective, operational, and +provenance. Each column packs several independently named fields at +specific bit ranges. For some purposes, such as updating the +population-statistics matrices, it helps to think of these three columns +as one uniform grid. That grid holds thirty-six fields of six bits each. +The real fields are not laid out quite that uniformly, though. +`RowBitmaps.field(_:)` performs this translation. Given a field index +from zero to thirty-five, it locates the right column and bit offset. It +returns that field's six-bit value. The function carries two defensive +casts, and the source comment explains both in detail. One is a +shift-amount guard: Swift silently wraps an out-of-range shift instead of +returning zero. The other is a cast to `UInt64` before shifting: +right-shifting a negative `Int64` sign-extends. That would leak the sign +bit into a field's value. `RowBitmaps.bit(field:bit:)` reads one bit out of a field's six-bit value. `RowBitmaps.fieldValues()` returns all thirty-six (field, value) pairs in -order, which is what `MatrixO.applyRow` and `MatrixT.applyPair` consume to +order. `MatrixO.applyRow` and `MatrixT.applyPair` consume this list to update the co-occurrence and causality matrices. `RowBitmaps.bitVector()` builds a `BitVector216` from a live row. `BitVector216` has two constructors for two different sources of the same -216-bit shape. `init(rowBitmaps:)` builds it from a real row through -`RowBitmaps.field(_:)`; because real rows never set bits above position 71 -of the abstract grid, this path is safe for updating `MatrixF` from row -capture and expunge events. `init(presenceBytes:)` builds it directly from a -raw 27-byte pattern that may set any of the 216 bits, including ones no real -`RowBitmaps` value can represent — the shape a conformance-test harness -vector uses. `bit(at:)` and `bit(field:bit:)` read one bit by absolute index +216-bit shape. `init(rowBitmaps:)` builds it from a real row, through +`RowBitmaps.field(_:)`. Real rows never set bits above position +seventy-one of the abstract grid. This path is therefore safe for +updating `MatrixF` from row capture and expunge events. +`init(presenceBytes:)` builds it directly from a raw 27-byte pattern. That +pattern may set any of the 216 bits, including ones no real `RowBitmaps` +value can represent. A conformance-test harness vector uses this second +shape. `bit(at:)` and `bit(field:bit:)` read one bit, by absolute index or by (field, bit) coordinate. ## Row.swift -This file provides `Row`, the substrate's central data shape, and `RowId`, -a type alias for `UUID` used wherever the substrate vocabulary calls for a -row identifier. - -`RowId` is a plain alias rather than a wrapper type because a `UUID`'s -16-byte big-endian wire form is already byte-identical to the Rust port's -`RowId(u128)` newtype; no translation step is needed at the boundary, and -callers can use `UUID`'s existing API directly. - -`Row` itself is a value type holding one row's complete current state: its -identifier, noun type, lifecycle state, three bitmap columns, fingerprint, -lattice anchor, and two optional fields — a lineage identifier, linking a -row to the row it was derived from, and the verbatim content. The file is -explicit that `Row` is pure data: no logic and no I/O. The full substrate -object that bundles many rows together with the audit log and the -statistics matrices, and the verb functions that mutate a row's state, -remain in SubstrateLib. `Row` only says what a row looks like at rest. +This file provides `Row`, the substrate's central data shape. It also +provides `RowId`, a type alias for `UUID`. Every place the substrate +vocabulary calls for a row identifier uses `RowId`. + +`RowId` is a plain alias, not a wrapper type. A `UUID`'s 16-byte +big-endian wire form is already byte-identical to the Rust port's +`RowId(u128)` newtype. No translation step is needed at the boundary. +Callers can use `UUID`'s existing API directly. + +`Row` itself is a value type. It holds one row's complete current state: +its identifier, noun type, lifecycle state, three bitmap columns, +fingerprint, and lattice anchor. It also holds two optional fields. One is +a lineage identifier, linking a row to the row it was derived from. The +other is the verbatim content. The file is explicit that `Row` is pure +data. It has no logic and no input or output. The full substrate object +that bundles many rows together, with the audit log and the statistics +matrices, remains in SubstrateLib. So do the verb functions that mutate a +row's state. `Row` only says what a row looks like at rest. ## LatticeAnchor.swift -This file provides `LatticeAnchor`, the sixteen-byte reference a row carries -to its position in the classification lattice: an eight-byte hash of a -lattice code plus an eight-byte hash of a concept identity (a Q-ID), or zero -for either half when absent. - -A lattice anchor does not store the lattice code text itself — it stores a -deterministic hash of it. `LatticeAnchor.fnv1a64(_:)` is the private FNV-1a -64-bit hash used for this purpose; because it is the exact FNV-1a algorithm -(matching `FNV.hash64` elsewhere in the package), two callers hashing the -same code string always agree, and the result matches the Rust port bit for -bit. - -`LatticeAnchor.udc(_:)` builds an anchor from a lattice code string alone, -with a null Q-ID pointer. `LatticeAnchor.udcQid(_:qid:)` builds an anchor -that carries both the lattice code and a specific concept identity; the file -explains why this second form matters: without it, every row about a broad -subject collapses onto one anchor value, losing the distinction between -different things classified under the same broad code. An empty Q-ID string -yields the same null pointer as `udc(_:)`, so the two constructors agree -when no concept identity is available. `LatticeAnchor.isNull` reports -whether both halves are zero — the anchor equivalent of "unclassified." +This file provides `LatticeAnchor`. It is the sixteen-byte reference a +row carries to its position in the classification lattice. It packs an +eight-byte hash of a lattice code plus an eight-byte hash of a concept +identity, called a Q-ID. Either half can be zero when absent. + +A lattice anchor does not store the lattice code text itself. It stores +a deterministic hash of it. `LatticeAnchor.fnv1a64(_:)` is the private +FNV-1a 64-bit hash used for this purpose. It is the exact FNV-1a +algorithm, matching `FNV.hash64` elsewhere in the package. Two callers +hashing the same code string always agree. The result also matches the +Rust port bit for bit. + +`LatticeAnchor.udc(_:)` builds an anchor from a lattice code string +alone, with a null Q-ID pointer. `LatticeAnchor.udcQid(_:qid:)` builds an +anchor that carries both the lattice code and a specific concept +identity. The file explains why this second form matters. Without it, +every row about a broad subject collapses onto one anchor value. That +loses the distinction between different things classified under the same +broad code. An empty Q-ID string yields the same null pointer as +`udc(_:)`. The two constructors therefore agree when no concept identity +is available. `LatticeAnchor.isNull` reports whether both halves are +zero. This is the anchor's way of saying "unclassified." ## HLC.swift -This file provides `HLC`, the Hybrid Logical Clock timestamp that orders -events across replicas of an estate without requiring synchronized wall -clocks, and `HLCGenerator`, the per-replica state machine that produces -HLC values. - -An `HLC` is three fields compared in order: `physicalTime` (milliseconds -since the Unix epoch), `logicalCount` (a counter that advances when -physical time does not), and `nodeID` (a per-replica tiebreaker). `HLC.<` -compares these three fields lexicographically, which gives any two HLC -values an unambiguous order even when two replicas' clocks briefly agree or -run backward relative to each other. `HLC.zero` is the earliest possible -value, used as a sentinel. `HLC.advanced()` returns a copy with the logical -counter incremented, holding physical time and node fixed. +This file provides `HLC`, the Hybrid Logical Clock timestamp. It orders +events across replicas of an estate without needing synchronized wall +clocks. The file also provides `HLCGenerator`, the per-replica state +machine that produces HLC values. + +An `HLC` has three fields, compared in order. `physicalTime` is +milliseconds since the Unix epoch. `logicalCount` is a counter that +advances when physical time does not. `nodeID` is a per-replica +tiebreaker. `HLC.<` compares these three fields in that order. This gives +any two HLC values an unambiguous order. That holds even when two +replicas' clocks briefly agree, or run backward relative to each other. +`HLC.zero` is the earliest possible value, used as a sentinel. +`HLC.advanced()` returns a copy with the logical counter incremented. It +holds physical time and node fixed. `HLCGenerator.send(now:)` produces the timestamp for a locally originated -event: if the wall clock has moved forward since the last call, it adopts -the new time and resets the logical counter to zero; otherwise it holds the -physical time and bumps the logical counter, guaranteeing every timestamp -this replica emits is strictly greater than the last. `HLCGenerator.receive(remote:now:)` -implements the HLC paper's merge rule for an event arriving from another -replica: it takes the maximum of the local clock, the remote clock, and the -current wall time, and sets the logical counter according to which of the -three inputs was tied for that maximum. This is the operation that lets two -replicas exchange audit events and converge on one shared sense of "what -happened before what," which the convergence proof in `GSetAuditLog.swift` -depends on. - -`HLC.packed` and `init(packed:)` convert to and from a lossy 8-byte form -used by federation tier-contribution messages, trading precision (40 bits of -milliseconds, roughly 34 years of range) for a smaller wire size. `HLC.wireBytes` -and `init(wireBytes:)` are the lossless 16-byte round trip used elsewhere. -The file also carries a `nodeId`-spelled initializer overload, kept only so -older call sites written against that casing keep compiling unchanged. +event. If the wall clock has moved forward since the last call, it adopts +the new time and resets the logical counter to zero. Otherwise it holds +the physical time and bumps the logical counter. Either way, every +timestamp this replica emits is strictly greater than the last. +`HLCGenerator.receive(remote:now:)` implements the HLC paper's merge rule +for an event arriving from another replica. It takes the maximum of the +local clock, the remote clock, and the current wall time. It sets the +logical counter according to which input was tied for that maximum. This +is the operation that lets two replicas exchange audit events. They +converge on one shared sense of "what happened before what." The +convergence proof in `GSetAuditLog.swift` depends on this operation. + +`HLC.packed` and `init(packed:)` convert to and from a lossy 8-byte form. +Federation tier-contribution messages use this form. It trades precision, +forty bits of milliseconds, for a smaller wire size. That is roughly +thirty-four years of range. `HLC.wireBytes` and `init(wireBytes:)` are +the lossless 16-byte round trip used elsewhere. The file also carries a +`nodeId`-spelled initializer overload. It exists only so older call sites +written against that casing keep compiling unchanged. ## TimeRange.swift -This file provides `TimeRange`, a closed interval `[start, end]` of two HLC -values. +This file provides `TimeRange`, a closed interval `[start, end]` of two +HLC values. `TimeRange.init(start:end:)` rejects an interval whose end precedes its -start with a precondition, because an inverted range has no sensible -meaning for any caller. `TimeRange.contains(_:)` reports whether a given HLC -falls within the closed interval, inclusive of both endpoints — the -predicate a caller uses to test "did this event happen during this window." -The file notes that the higher-level primitive consuming these windows, -`MomentSummary`, stays in SubstrateLib; this type is only the interval -shape. +start, with a precondition. An inverted range has no sensible meaning for +any caller. `TimeRange.contains(_:)` reports whether a given HLC falls +within the closed interval. Both endpoints are included. This is the +predicate a caller uses to test "did this event happen during this +window." The file notes that the higher-level primitive consuming these +windows, `MomentSummary`, stays in SubstrateLib. This type is only the +interval shape. ## AsOfCoordinate.swift -This file provides `AsOfCoordinate`, a two-case enum that says whether a -read should return the live state (`.present`) or the state as of a -specific HLC (`.asOf(hlc)`). +This file provides `AsOfCoordinate`, a two-case enum. It says whether a +read should return the live state or the state as of a specific HLC. The +first case is `.present`. The second case is `.asOf(hlc)`. -The file explains the bug this type prevents: if "present" were represented -by a sentinel HLC value such as zero, a caller could not distinguish "give -me now" from "give me the state at the dawn of time." Making the choice an -explicit enum discriminant removes that ambiguity by construction. The -custom `Codable` conformance encodes the case as an explicit `"kind"` field -(`"present"` or `"asOf"`) plus, for the second case, the HLC itself, rather -than relying on Swift's default enum encoding, so the wire form stays stable -and self-describing across the Swift and Rust legs. +The file explains the bug this type prevents. Suppose "present" were +represented by a sentinel HLC value such as zero. A caller could not then +distinguish "give me now" from "give me the state at the dawn of time." +Making the choice an explicit enum discriminant removes that ambiguity by +construction. The custom `Codable` conformance encodes the case as an +explicit `"kind"` field, either `"present"` or `"asOf"`. For the second +case, it also encodes the HLC itself. This is not Swift's default enum +encoding. The wire form stays stable and self-describing across the Swift +and Rust legs. ## SnapshotId.swift -This file provides `SnapshotId`, a `UUID` wrapper identifying one +This file provides `SnapshotId`, a `UUID` wrapper. It identifies one point-in-time snapshot of an estate. Unlike `RowId`, which is a bare type alias, `SnapshotId` is a distinct -struct. The file states the reason: a snapshot identifier must not be -substitutable for a drawer, node, or estate identifier at the type level, -even though all of these are UUIDs underneath. `init(uuidString:)` and the -`uuidString` accessor round-trip through the standard UUID string form; -`Codable` conformance encodes and decodes that same string form, throwing -`SnapshotIdError.invalidUUID` on a malformed string rather than crashing. +struct. The file states the reason. A snapshot identifier must not be +substitutable for a drawer, node, or estate identifier at the type level. +This holds even though all of these are UUIDs underneath. +`init(uuidString:)` and the `uuidString` accessor round-trip through the +standard UUID string form. `Codable` conformance encodes and decodes that +same string form. It throws `SnapshotIdError.invalidUUID` on a malformed +string, rather than crashing. ## BlockMask.swift -This file provides `BlockMask`, an `OptionSet` selecting any subset of the -fingerprint's four 64-bit blocks. +This file provides `BlockMask`, an `OptionSet`. It selects any subset of +the fingerprint's four 64-bit blocks. -Before this type existed, callers passed a `Set` to select blocks, -which allocated memory on every call — a measured hot-path cost in nearest- -neighbor search over large row collections. `BlockMask` replaces that with -four fixed bit flags (`.block0` through `.block3`), an `.all` convenience -covering every block, and an empty `.none`. Because it is a plain integer -under the hood, membership tests are branchless and allocation-free. -`BlockMask.blockCount` counts how many blocks are selected (0 through 4), -which `Hamming.similarity` uses to compute the correct maximum-distance -denominator for a partial-block comparison. +Before this type existed, callers passed a `Set` to select blocks. +That allocated memory on every call, a measured hot-path cost in +nearest-neighbor search over large row collections. `BlockMask` replaces +that with four fixed bit flags, `.block0` through `.block3`. It adds an +`.all` convenience covering every block, and an empty `.none`. It is a +plain integer under the hood. So membership tests are branchless and need +no allocation. `BlockMask.blockCount` counts how many blocks are +selected, zero through four. `Hamming.similarity` uses this count to +compute the correct maximum-distance denominator for a partial-block +comparison. ## Fingerprint256.swift This file provides `Fingerprint256`, the substrate's 256-bit structural -fingerprint, along with the pure combinators that other files in this -package build on. - -A fingerprint is four independent 64-bit blocks — `block0` through -`block3` — each a SimHash over a different aspect of a row: its bitmaps, -its lattice position, its lineage and timing, and its channel and source. -The file is careful to distinguish a fingerprint (the coordinate for -structural similarity) from a lattice anchor (the coordinate for topic -similarity); the two answer different questions and neither substitutes for -the other. - -`Fingerprint256.bit(at:)` and `Fingerprint256.with(bit:set:)` give bit-level -read and copy-on-write access across the full 256-bit span, translating a -flat index into the right block automatically. `with(bit:set:)` returns a -new value rather than mutating in place, matching the package's general -preference for value semantics over in-place mutation. `Fingerprint256.union(_:)` -computes the bitwise OR of two fingerprints — set-union over the bits each -one has set. `Fingerprint256.fromBits(_:)` builds a fingerprint from a -256-element array of booleans, mainly useful for tests and for converting +fingerprint. It also provides the pure combinators that other files in +this package build on. + +A fingerprint is four independent 64-bit blocks: `block0` through +`block3`. Each is a SimHash over a different aspect of a row. One covers +its bitmaps. One covers its lattice position. One covers its lineage and +timing. One covers its channel and source. The file is careful to +distinguish a fingerprint from a lattice anchor. A fingerprint is the +coordinate for structural similarity. A lattice anchor is the coordinate +for topic similarity. The two answer different questions. Neither +substitutes for the other. + +`Fingerprint256.bit(at:)` and `Fingerprint256.with(bit:set:)` give +bit-level read and copy-on-write access across the full 256-bit span. +Both translate a flat index into the right block automatically. +`with(bit:set:)` returns a new value rather than mutating in place. This +matches the package's general preference for value semantics over +in-place mutation. `Fingerprint256.union(_:)` computes the bitwise OR of +two fingerprints: set-union over the bits each one has set. +`Fingerprint256.fromBits(_:)` builds a fingerprint from a 256-element +array of booleans. This is mainly useful for tests, and for converting from an external bit representation. `wireBytes` and `init(wireBytes:)` -convert to and from the canonical 32-byte little-endian wire form; -`toBytes()` and `fromBytes(_:)` are thin aliases kept for parity with call -sites that expect a throws-free, optional-returning variant. - -The file's closing section defines a small combinator layer: -`zip4(_:_:)` applies a binary operation to each of the four block pairs -between two fingerprints in one call, `map4(_:)` applies a unary operation -to all four blocks of one fingerprint, `reduce4(_:_:)` folds a whole -sequence of fingerprints block-wise with a binary operation, and -`popcount()` counts set bits across all four blocks. These four functions -exist because the same "unroll across four blocks" pattern previously -appeared, hand-written, in seven or more places across the codebase — this -package's `ORReduce`, `BitwiseArithmetic`, and `Hamming` among them. -Expressing the unroll once here, and asking every other block-wise operation -to call it, means a future change to how blocks are combined only has one -call site to change, with no risk that one of the seven repeated copies -falls out of sync with the rest. `zip4Batch(_:_:_:)` and `map4Batch(_:_:)` -extend the same idea across an array of fingerprints, for callers vectorizing -across many rows rather than across one fingerprint's four blocks. +convert to and from the canonical 32-byte little-endian wire form. +`toBytes()` and `fromBytes(_:)` are thin aliases, kept for parity with +call sites that expect a throws-free, optional-returning variant. + +The file's closing section defines a small combinator layer. `zip4(_:_:)` +applies a binary operation to each of the four block pairs between two +fingerprints, in one call. `map4(_:)` applies a unary operation to all +four blocks of one fingerprint. `reduce4(_:_:)` folds a whole sequence of +fingerprints block-wise, with a binary operation. `popcount()` counts set +bits across all four blocks. These four functions exist for a reason. +The same "unroll across four blocks" pattern used to appear, hand-written, +in seven or more places across the codebase. This package's `ORReduce`, +`BitwiseArithmetic`, and `Hamming` were among them. Expressing the unroll +once here means a future change has only one call site to update. There is +no risk that one of the seven repeated copies falls out of sync with the +rest. `zip4Batch(_:_:_:)` and `map4Batch(_:_:)` extend the same idea across +an array of fingerprints. These serve callers vectorizing across many +rows, rather than across one fingerprint's four blocks. ## HyperplaneFamily.swift -This file provides `Hyperplane`, one ±1-valued dividing plane used by -SimHash, and `HyperplaneFamily`, the fixed set of sixty-four such planes -that produce one fingerprint block. - -A hyperplane of length *n* would normally be an array of *n* values, each -+1, −1, or 0. Because binary inputs make this arithmetic reduce to counting -set bits, `Hyperplane` instead stores two bitmasks — `positiveMask` and -`negativeMask` — with a 1 wherever the plane is +1 or −1 respectively, and 0 -in both where the plane's value is 0 at that position (a "sparse -hyperplane"). `Hyperplane.sign(over:)` computes whether the plane's dot -product with a binary input vector is strictly positive, by comparing the -popcount of the input ANDed with each mask; a tied dot product resolves to -false, which the file notes is vanishingly rare in practice because the -per-block input has enough bits that an exact tie is unlikely. - -`HyperplaneFamily.generate(seed:blockIndex:inputBitLength:density:)` builds -a family of sixty-four hyperplanes deterministically from a 32-byte seed -using a private, non-cryptographic pseudo-random generator -(`HyperplanePRNG`). Determinism here is load-bearing: the file states that -hyperplane seeds are fixed once, at estate creation, and never rotate, -because rotating them would change every fingerprint an estate has ever -computed and break cross-replica sync. The `density` parameter controls -what fraction of a plane's positions are active (nonzero) rather than zero; -at density 1.0 every position is active, and the code takes a direct path -around a floating-point computation that would otherwise round up past -`UInt64.max` and trap. - -`HyperplaneFamily.canonicalHash()` computes a stable 64-bit fingerprint of -a whole family's content by folding every mask word through FNV-1a, used to -label a pairing arrangement in an audit trail without storing the full -family. `HyperplaneFamily.blockFamilies(baseSeed:density:)` is the single -routine that builds all four block families — for block 0 (192 bits) and -blocks 1 through 3 (64 bits each) — from one base seed, fixing two mistakes -an earlier, separately-derived generation path had: reusing one seed across -all four blocks (which collapsed them into one projection instead of four -independent ones) and assuming a uniform 64-bit width for every block -(block 0 is 192 bits). `diversifiedSeed(base:blockIndex:)` mixes a block -index into the base seed so each block gets an independent 32-byte seed, and -`expandSeed64(_:)` stretches a 64-bit seed back out to 32 bytes through four -rounds of a SplitMix64-style mixing function. +This file provides `Hyperplane`, one plus-or-minus-one-valued dividing +plane used by SimHash. It also provides `HyperplaneFamily`, the fixed set +of sixty-four such planes that produce one fingerprint block. + +A hyperplane of length *n* would normally be an array of *n* values. Each +value would be plus one, minus one, or zero. Binary inputs make this +arithmetic reduce to counting set bits. So `Hyperplane` instead stores two +bitmasks, `positiveMask` and `negativeMask`. Each has a one wherever the +plane is plus one or minus one, and a zero in both where the plane's +value is zero at that position. This is called a sparse hyperplane. +`Hyperplane.sign(over:)` computes whether the plane's dot product with a +binary input vector is strictly positive. It does this by comparing the +popcount of the input, ANDed with each mask. A tied dot product resolves +to false. The file notes this is vanishingly rare in practice, because +the per-block input has enough bits that an exact tie is unlikely. + +`HyperplaneFamily.generate(seed:blockIndex:inputBitLength:density:)` +builds a family of sixty-four hyperplanes deterministically. It starts +from a 32-byte seed, using a private, non-cryptographic pseudo-random +generator called `HyperplanePRNG`. Determinism here is load-bearing. The +file states that hyperplane seeds are fixed once, at estate creation, and +never rotate. Rotating them would change every fingerprint an estate has +ever computed. It would break cross-replica sync. The `density` parameter +controls what fraction of a plane's positions are active, rather than +zero. At density 1.0, every position is active. The code then takes a +direct path around a floating-point computation. That computation would +otherwise round up past `UInt64.max` and trap. + +`HyperplaneFamily.canonicalHash()` computes a stable 64-bit fingerprint +of a whole family's content. It folds every mask word through FNV-1a. +This labels a pairing arrangement in an audit trail without storing the +full family. `HyperplaneFamily.blockFamilies(baseSeed:density:)` is the +single routine that builds all four block families from one base seed. +Block 0 needs 192 bits; blocks 1 through 3 need 64 bits each. This +routine fixes two mistakes an earlier, separately derived generation path +had made. One mistake was reusing one seed across all four blocks, which +collapsed them into one projection instead of four independent ones. The +other was assuming a uniform 64-bit width for every block, when block 0 +is 192 bits. `diversifiedSeed(base:blockIndex:)` mixes a block index into +the base seed. Each block then gets an independent 32-byte seed. +`expandSeed64(_:)` stretches a 64-bit seed back out to 32 bytes. It does +this through four rounds of a SplitMix64-style mixing function. ## FloatSimHashPlanes.swift -This file provides `FloatSimHashPlanes`, the materialized hyperplane set for -the floating-point-input variant of SimHash — the float analogue of -`HyperplaneFamily`, which serves the binary-input variant. +This file provides `FloatSimHashPlanes`, the materialized hyperplane set +for the floating-point-input variant of SimHash. It is the float +analogue of `HyperplaneFamily`, which serves the binary-input variant. -The file explains why this type lives in SubstrateTypes rather than in -either of the two packages that use it: `SubstrateKernel` applies these -planes (pure signed-sum-and-sign arithmetic, no randomness) and -`SubstrateML` generates them from a seed (using the SplitMix64 random -generator). Placing the shared data shape at the lowest layer lets both -higher packages depend on it without either depending on the other, which +The file explains why this type lives in SubstrateTypes, rather than in +either of the two packages that use it. `SubstrateKernel` applies these +planes, through pure signed-sum-and-sign arithmetic with no randomness. +`SubstrateML` generates them from a seed, using the SplitMix64 random +generator. Placing the shared data shape at the lowest layer lets both +higher packages depend on it without either depending on the other. That would otherwise invert the SDK's layering. `FloatSimHashPlanes` packs -`256 * dim` sign bits — one per (hyperplane, coordinate) pair — into a flat -array of 64-bit words, row-major by hyperplane, matching the draw order the -generator produces them in. Because the planes are immutable once generated -for a given seed and dimensionality, a caller materializes one instance per -(seed, dimension) pair and reuses it across every projection under that seed. +`256 * dim` sign bits, one per hyperplane-coordinate pair, into a flat +array of 64-bit words. The words are ordered row-major by hyperplane, +matching the draw order the generator produces them in. The planes are +immutable once generated for a given seed and dimensionality. So a caller +materializes one instance per seed-dimension pair. That instance is +reused across every projection under that seed. ## SimHash.swift -This file provides `SimHash`, the construction that turns an input vector -and a hyperplane family into one 64-bit fingerprint block, plus -`SimHashInput`, the set of helpers that assemble the four blocks' input -vectors from a row's actual fields. - -`SimHash.block(over:family:)` is the core operation: for each of the -family's sixty-four hyperplanes, it sets one output bit if the hyperplane's -dot product with the input vector is positive. `SimHash.fingerprint(bitmapInput:latticeInput:lineageTemporalInput:channelSourceInput:families:)` -calls this once per block and assembles the four results into a complete -`Fingerprint256`. `SimHash.fingerprintBatch(...)` is the same computation -repeated across many rows, returning results in input order; the file notes -that this reference loop is what a hardware-accelerated backend elsewhere in -the SDK must reproduce bit for bit, even though that backend vectorizes the -popcount work across many rows at once for speed. -`SimHash.fingerprint(fromSubhashes:hyperplanes:)` is a convenience path for -callers that have already reduced each block's input down to one 64-bit -subhash rather than a full input vector. +This file provides `SimHash`, the construction that turns an input +vector and a hyperplane family into one 64-bit fingerprint block. It also +provides `SimHashInput`, the set of helpers that assemble the four +blocks' input vectors from a row's actual fields. + +`SimHash.block(over:family:)` is the core operation. For each of the +family's sixty-four hyperplanes, it sets one output bit. It sets that bit +if the hyperplane's dot product with the input vector is positive. +`SimHash.fingerprint(bitmapInput:latticeInput:lineageTemporalInput:channelSourceInput:families:)` +calls this once per block. It assembles the four results into a complete +`Fingerprint256`. `SimHash.fingerprintBatch(...)` is the same computation, +repeated across many rows, returning results in input order. The file +notes that this reference loop is what a hardware-accelerated backend +elsewhere in the SDK must reproduce bit for bit. That backend vectorizes +the popcount work across many rows at once, for speed, but the result +must match exactly. +`SimHash.fingerprint(fromSubhashes:hyperplanes:)` is a convenience path. +It serves callers that have already reduced each block's input down to +one 64-bit subhash, rather than a full input vector. `SimHashInput.bitmap(adjective:operational:provenance:)` concatenates a row's three bitmap columns into block 0's 192-bit input. `SimHashInput.lattice(udcPrefixHash:qidDirectHash:qidClosureHash:)` packs -three hash components into block 1's 64-bit input at fixed bit offsets. +three hash components into block 1's 64-bit input, at fixed bit offsets. `SimHashInput.lineageTemporal(...)` and `SimHashInput.channelSource(...)` -do the same for blocks 2 and 3. Each function's bit-offset layout is fixed -by the cookbook specification the file cites; a mismatched offset here would -silently produce fingerprints that no longer compare correctly against ones -built the standard way, so these helpers exist precisely to keep every -caller assembling the same layout. +do the same for blocks 2 and 3. Each function's bit-offset layout is +fixed by the cookbook specification the file cites. A mismatched offset +here would silently produce fingerprints that no longer compare +correctly against ones built the standard way. These helpers exist +precisely so every caller assembles the same layout. ## Hamming.swift -This file provides `Hamming`, the substrate's primary structural-similarity -measurement: how many bit positions differ between two fingerprints. +This file provides `Hamming`, the substrate's primary +structural-similarity measurement. It counts how many bit positions +differ between two fingerprints. `Hamming.distance(_:_:blocks:)` computes this by XORing the two -fingerprints and counting the set bits in the result — `a.zip4(b, ^).popcount()` -on the common case of all four blocks, or a per-block loop when the caller -restricts the comparison to a `BlockMask` subset. Restricting to a subset -answers a more specific question, such as "how similar are these two rows -in lattice position alone, ignoring timing and provenance." `Hamming.similarity(_:_:blocks:)` -converts a distance into a `[0, 1]` score, where 1.0 means identical and 0.0 -means maximally distant over the selected blocks. `HammingDistance` is a -type alias re-exporting `Hamming` under the name some call sites use. +fingerprints and counting the set bits in the result. That is +`a.zip4(b, ^).popcount()` on the common case of all four blocks. A +per-block loop handles the case where the caller restricts the comparison +to a `BlockMask` subset. Restricting to a subset answers a more specific +question, such as "how similar are these two rows in lattice position +alone, ignoring timing and provenance." `Hamming.similarity(_:_:blocks:)` +converts a distance into a `[0, 1]` score. A score of 1.0 means +identical. A score of 0.0 means maximally distant over the selected +blocks. `HammingDistance` is a type alias, re-exporting `Hamming` under +the name some call sites use. ## BitwiseArithmetic.swift -This file provides three fingerprint operations beyond OR-reduction — -intersection, symmetric difference, and a cohort prototype — plus -`FingerprintBuilder`, a small expression type for composing them -declaratively. +This file provides three fingerprint operations beyond OR-reduction. +They are intersection, symmetric difference, and a cohort prototype. It +also provides `FingerprintBuilder`, a small expression type for composing +them declaratively. `BitwiseArithmetic.intersect(_:_:)` computes the bitwise AND of two fingerprints: the bits both rows share. `BitwiseArithmetic.difference(_:_:)` -computes the bitwise XOR: the bits where the two disagree. Both delegate to -`Fingerprint256.zip4(_:_:)`. `BitwiseArithmetic.prototype(_:)` computes the -bit-for-bit majority vote across a whole cohort of fingerprints — the -fingerprint of a "typical" member — by folding the cohort into a -`CountVector256` and reading off its majority-vote view; the file notes this -replaced an earlier hand-written per-bit counting loop with an equivalent -call into the one canonical cohort-fold primitive. - -`FingerprintBuilder` is an indirect enum with four cases — a literal -fingerprint, an intersection of two sub-expressions, a difference of two -sub-expressions, or the prototype of a fingerprint list — and one method, -`evaluate()`, that walks the tree and computes the result. This lets a -caller build up a compound fingerprint query as data (for example, -"the intersection of Bob's typical pattern and Amelia's typical pattern") -and evaluate it once, rather than nesting direct function calls. +computes the bitwise XOR: the bits where the two disagree. Both delegate +to `Fingerprint256.zip4(_:_:)`. `BitwiseArithmetic.prototype(_:)` computes +the bit-for-bit majority vote across a whole cohort of fingerprints. This +is the fingerprint of a "typical" member. It works by folding the cohort +into a `CountVector256` and reading off its majority-vote view. The file +notes this replaced an earlier hand-written per-bit counting loop. That +loop is now an equivalent call into the one canonical cohort-fold +primitive. + +`FingerprintBuilder` is an indirect enum with four cases. One is a +literal fingerprint. One is an intersection of two sub-expressions. One +is a difference of two sub-expressions. One is the prototype of a +fingerprint list. It has one method, `evaluate()`, that walks the tree +and computes the result. This lets a caller build up a compound +fingerprint query as data. One example is "the intersection of Bob's +typical pattern and Amelia's typical pattern." The caller can evaluate it +once, rather than nesting direct function calls. ## ORReduce.swift @@ -483,339 +510,363 @@ This file provides `ORReduce`, the substrate's universal aggregation primitive over collections of fingerprints. `ORReduce.reduce(_:)` computes the bitwise OR across a whole sequence of -fingerprints, returning `Fingerprint256.zero` for an empty sequence (OR's -identity element); it delegates to `Fingerprint256.reduce4(_:|:)`. +fingerprints. It returns `Fingerprint256.zero` for an empty sequence, +OR's identity element. It delegates to `Fingerprint256.reduce4(_:|:)`. `ORReduce.reduce(_:blocks:defaults:)` restricts the reduction to specific -blocks, filling any block outside the selection from a supplied default — -useful when a caller wants only the topic-block aggregate across a cohort, -say, without touching the other three blocks. The file explains why OR -reduction matters beyond simple aggregation: because it is commutative, -associative, and idempotent, it works as the join operator for a CRDT of -fingerprints — replicas can merge any number of contributions, in any -order, with any duplication, and converge on the same result. It is also -a privacy mechanism: once several fingerprints are OR-reduced together, a -reader can see which structural patterns appear somewhere in the group but -cannot recover which specific contributor set which bit. +blocks. It fills any block outside the selection from a supplied +default. This helps when a caller wants only the topic-block aggregate +across a cohort, without touching the other three blocks. The file +explains why OR reduction matters beyond simple aggregation. It is +commutative, associative, and idempotent. So it works as the join +operator for a CRDT of fingerprints. Replicas can merge any number of +contributions. They can merge in any order, with any duplication. They +still converge on the same result. It is also a privacy mechanism. Once several fingerprints +are OR-reduced together, a reader can see which structural patterns +appear somewhere in the group. The reader cannot recover which specific +contributor set which bit. ## CountVector256.swift -This file provides `CountVector256`, the per-bit counting accumulator that -underlies cohort-level fingerprint statistics, including the majority-vote -prototype `BitwiseArithmetic.prototype(_:)` reads from it. - -The file's opening comment makes the key design argument: a majority-vote -summary does not compose. If node A's majority is computed from its -children, and node B's majority from its children, the majority of "A's -majority and B's majority" is not generally the same as the true majority -over every leaf under both A and B combined. A count vector does compose: -counts add, member totals add, and the sum is exact regardless of the order -members are folded in. So the count vector, not the majority-vote -fingerprint, is the object worth storing at every level of a tree; the -majority-vote fingerprint is a read-time view computed from it whenever a -caller actually needs a Hamming-comparable engram. - -`CountVector256.accumulate(_:)` folds one fingerprint into the vector: for -every set bit, its counter increases by one, and the member count increases -by one. `CountVector256.merge(_:)` and its operator form `+` combine two -count vectors by adding element-wise — the tree-fold step, safe to apply to -a node's children in any order because addition is commutative and -associative. `CountVector256.majorityVote()` reads off the fingerprint whose -bit `j` is set exactly when a strict majority of accumulated members had bit -`j` set (`2 * count > n`; an exact tie at half does not set the bit, a -convention the file calls out as identical across every kernel backend and -both language ports). `CountVector256.profile()` returns the 256 -per-bit probabilities instead of a threshold, for callers that want the -Bernoulli parameter of each bit rather than a hard yes-or-no answer. The -static `CountVector256.fold(_:)` is the reference implementation of folding -a whole array of fingerprints at once, which a vectorized kernel-layer -backend is required to match exactly. +This file provides `CountVector256`, the per-bit counting accumulator. +It underlies cohort-level fingerprint statistics, including the +majority-vote prototype that `BitwiseArithmetic.prototype(_:)` reads from +it. + +The file's opening comment makes the key design argument. A +majority-vote summary does not compose. Suppose node A's majority is +computed from its children, and node B's majority from its children. The +majority of "A's majority and B's majority" is not generally the same as +the true majority over every leaf under both A and B combined. A count +vector does compose. Counts add. Member totals add. The sum is exact +regardless of the order members are folded in. So the count vector, not +the majority-vote fingerprint, is the object worth storing at every level +of a tree. The majority-vote fingerprint is a read-time view, computed +from it whenever a caller actually needs a Hamming-comparable engram. + +`CountVector256.accumulate(_:)` folds one fingerprint into the vector. +For every set bit, its counter increases by one, and the member count +increases by one. `CountVector256.merge(_:)` and its operator form `+` +combine two count vectors by adding element-wise. This is the tree-fold +step. It is safe to apply to a node's children in any order, because +addition is commutative and associative. `CountVector256.majorityVote()` +reads off the fingerprint whose bit `j` is set exactly when a strict +majority of accumulated members had bit `j` set. The rule is +`2 * count > n`. An exact tie at half does not set the bit. The file +calls this convention identical across every kernel backend and both +language ports. `CountVector256.profile()` returns the 256 per-bit +probabilities instead of a threshold. This suits callers that want the +Bernoulli parameter of each bit, rather than a hard yes-or-no answer. The +static `CountVector256.fold(_:)` is the reference implementation of +folding a whole array of fingerprints at once. A vectorized kernel-layer +backend is required to match it exactly. ## FNV.swift -This file provides `FNV`, the Fowler–Noll–Vo 1a string hash family the -substrate uses everywhere a deterministic string hash is required: drawer -fingerprints, manifest-derived identifiers, and deterministic tokenization. +This file provides `FNV`, the Fowler-Noll-Vo 1a string hash family. The +substrate uses it everywhere a deterministic string hash is required: +drawer fingerprints, manifest-derived identifiers, and deterministic +tokenization. -`FNV.hash64(_:)` and `FNV.hash32(_:)` are two independent FNV-1a hashes with -different offset bases and primes — the file is explicit that the 32-bit -result is not a truncation of the 64-bit one, but a wholly separate -computation over the same input bytes. `FNV.hash16(_:)` is different in -kind: it is the low 16 bits of `hash64(_:)`, not a from-scratch FNV-1a -variant (FNV-1a has no official 16-bit definition), used where the substrate -needs a compact prefix hash such as a lattice sub-hash. +`FNV.hash64(_:)` and `FNV.hash32(_:)` are two independent FNV-1a hashes. +They use different offset bases and primes. The file is explicit that +the 32-bit result is not a truncation of the 64-bit one. It is a wholly +separate computation over the same input bytes. `FNV.hash16(_:)` is +different in kind. It is the low sixteen bits of `hash64(_:)`, not a +from-scratch FNV-1a variant, because FNV-1a has no official 16-bit +definition. The substrate uses it where it needs a compact prefix hash, +such as a lattice sub-hash. ## MerkleDomain.swift -This file provides `MerkleDomain`, the four one-byte tags prepended before -hashing to keep different kinds of nodes in the Merkle content-integrity -tree from ever colliding: `leaf` (a single drawer's content and vectors), -`interior` (a parent summarizing its children), `tombstone` (an expunged -payload), and `commitment` (a keyed HMAC-SHA256 commitment). Domain -separation means a leaf hash and an interior hash can never be mistaken for -each other even if their underlying bytes happened to coincide, because -each was computed with a different one-byte prefix. The file states these -four values are frozen and must match exactly between the Swift and Rust -legs, permanently. +This file provides `MerkleDomain`, four one-byte tags. They are +prepended before hashing, to keep different kinds of nodes in the Merkle +content-integrity tree from ever colliding. `leaf` tags a single +drawer's content and vectors. `interior` tags a parent summarizing its +children. `tombstone` tags an expunged payload. `commitment` tags a keyed +HMAC-SHA256 commitment. Domain separation means a leaf hash and an +interior hash can never be mistaken for each other. This holds even if +their underlying bytes happened to coincide, because each was computed +with a different one-byte prefix. The file states these four values are +frozen. They must match exactly between the Swift and Rust legs, +permanently. ## ContentHash.swift -This file provides `ContentHash`, the 32-byte SHA-256 digest over one leaf -payload — a drawer's content plus its vectors. +This file provides `ContentHash`, the 32-byte SHA-256 digest over one +leaf payload: a drawer's content plus its vectors. The type stores its 32 bytes privately, exposed only through the `bytes` -accessor, so the fixed size is enforced by construction rather than by -convention. `ContentHash.tombstone` is a named constant: the SHA-256 hash of -the bare tombstone domain tag byte, used as the sentinel content hash for an -expunged (hard-deleted) drawer. The file explains why this constant is a -literal byte array rather than a value computed at runtime: SubstrateTypes, -as the lowest layer, cannot import SubstrateKernel (which owns the SHA-256 -implementation) without inverting the SDK's dependency direction, so the -precomputed literal stands in, checked against the real computation by a -bridge test in SubstrateKernel. `hexString` and `description` render the -bytes as lowercase hex; the custom `Codable` conformance encodes and decodes -that same hex string, validating its length and character set on decode -rather than trusting external input. The file is explicit that `ContentHash` -is not interchangeable with `MerkleRoot`, even though both are 32-byte -hashes: one is a payload digest, the other a subtree summary, and the type -system keeps them from being swapped by mistake. +accessor. The fixed size is enforced by construction, not by convention. +`ContentHash.tombstone` is a named constant. It is the SHA-256 hash of +the bare tombstone domain tag byte, used as the sentinel content hash for +an expunged, hard-deleted drawer. The file explains why this constant is +a literal byte array, rather than a value computed at runtime. +SubstrateTypes is the lowest layer. It cannot import SubstrateKernel, +which owns the SHA-256 implementation. Importing it would invert the +SDK's dependency direction. So the precomputed literal stands in instead. A bridge test in +SubstrateKernel checks it against the real computation. `hexString` and +`description` render the bytes as lowercase hex. The custom `Codable` +conformance encodes and decodes that same hex string. It validates the +string's length and character set on decode, rather than trusting +external input. The file is explicit that `ContentHash` is not +interchangeable with `MerkleRoot`, even though both are 32-byte hashes. +One is a payload digest. The other is a subtree summary. The type system +keeps them from being swapped by mistake. ## MerkleRoot.swift -This file provides `MerkleRoot`, the 32-byte hash summarizing an interior -node's children in the Merkle content-integrity tree — the counterpart to -`ContentHash`'s single-payload digest. - -Its shape mirrors `ContentHash` closely: private byte storage, a `bytes` -accessor, hex rendering, and the same style of custom `Codable` -implementation. `MerkleRoot.empty` is the named constant for the hash of a -node with no live children — the SHA-256 of the bare interior domain tag — -again stored as a literal for the same layering reason `ContentHash.tombstone` -is. Keeping `MerkleRoot` and `ContentHash` as two distinct types, rather than -one hash type used for both purposes, means a function that expects a -subtree summary cannot accidentally be handed a single payload's digest; the -compiler catches the mistake instead of a runtime bug surfacing later. +This file provides `MerkleRoot`, the 32-byte hash summarizing an +interior node's children in the Merkle content-integrity tree. It is the +counterpart to `ContentHash`'s single-payload digest. + +Its shape mirrors `ContentHash` closely. It uses private byte storage. It +has a `bytes` accessor, hex rendering, and the same `Codable` style. +`MerkleRoot.empty` is the named constant for the hash of +a node with no live children. It is the SHA-256 of the bare interior +domain tag, again stored as a literal, for the same layering reason +`ContentHash.tombstone` is. `MerkleRoot` and `ContentHash` stay two +distinct types, rather than one hash type used for both purposes. That +way, a function that expects a subtree summary cannot accidentally be +handed a single payload's digest. The compiler catches the mistake, +instead of a runtime bug surfacing later. ## AuditEvent.swift -This file provides `AuditEvent`, a single recorded mutation of a row: the -before-and-after bitmap and lattice-anchor state, which verb performed it, -who performed it, and when. - -`AuditEvent.eventID`, paired with `hlc`, gives every event a compound key -that makes replaying the same event twice — for example, after a sync -retry — a safe no-op rather than a duplicate. `beforeBitmaps` is optional -because a row's very first event, its capture, has no prior state to record. -`reason` is an optional human-readable explanation threaded from the call -site that performed the mutation; the file notes it is populated for -explicit actions such as an expunge but left `nil` for the great majority of +This file provides `AuditEvent`, a single recorded mutation of a row. It +holds the before-and-after bitmap and lattice-anchor state. It records +which verb performed the mutation. It records who performed it, and +when. + +`AuditEvent.eventID`, paired with `hlc`, gives every event a compound +key. That key makes replaying the same event twice, after a sync retry +for example, a safe no-op rather than a duplicate. `beforeBitmaps` is optional. A row's very first event is its capture. +That first event has no prior state to record. `reason` is an optional +human-readable explanation. It is threaded from the call site that +performed the mutation. The file notes it is populated for explicit +actions, such as an expunge, but left `nil` for the great majority of routine mutations. `AuditEvent.withReason(_:)` returns a copy of an event -with a caller-supplied reason attached, used because the structural -validator that first produces an event (`AuditGate.admit`, in SubstrateLib) -is a pure checker with no concept of "why," so the reason is layered on -afterward by the verb that called it. +with a caller-supplied reason attached. This exists because the +structural validator that first produces an event is a pure checker. +That validator, `AuditGate.admit` in SubstrateLib, has no idea why a +mutation happened. The reason is layered on afterward, by the verb that +called it. ## GSetAuditLog.swift -This file provides `GSetAuditLog`, the substrate's append-only audit log, -and its two supporting types, `AuditEntry` (one immutable log row) and -`AuditValue` (a typed field value inside an entry). - -The log is the substrate's source of truth: a row's visible current state is -a projection computed by replaying its log entries in order, never stored -independently of the log. `AuditEntry.id` is a 32-byte SHA-256 content hash -computed over the entry's other fields, which gives the log a natural way to -deduplicate: two replicas that independently record the same logical -mutation compute the same id and, once merged, keep only one copy. -`AuditValue` is an enum covering the four kinds of value a field change can -carry — a bitmap, a string, a fingerprint, or an integer — with a hand-written -`Codable` conformance that encodes each case as a single-key JSON object -(`{"bitmap": 42}`) rather than Swift's default synthesized shape -(`{"bitmap": {"_0": 42}}`), so the wire format matches what the Rust port's -serde derive produces natively and both legs agree on one JSON shape. - -`GSetAuditLog` itself stores its entries keyed by content hash internally, -for O(1) deduplication, but its `Codable` conformance serializes them as a -plain array sorted by id, because the conceptually correct wire -representation of a grow-only set is a set, not a hash map — the file -explains this distinction explicitly to justify writing custom `Codable` -rather than relying on the default. `GSetAuditLog.add(_:)` inserts one -entry, a no-op if the same id is already present. `GSetAuditLog.merge(_:)` -is the CRDT join: the union of two logs' entries, correct regardless of -which replica calls it or in what order two logs are merged. -`GSetAuditLog.orderedEntries` replays every entry in HLC order, the -sequence a projection applies to compute visible state. -`GSetAuditLog.entries(forRow:)` and `GSetAuditLog.entries(since:)` scope -that same ordered replay to one row or to everything after a cutoff, -serving the row-state automaton and the sync protocol respectively. The -file's closing comment sketches why this design converges: G-Set merge is -set union, set union is commutative, associative, and idempotent, and HLC -gives any two entries an unambiguous order to replay them in — so any two -replicas that have exchanged all of each other's entries compute identical -visible state. +This file provides `GSetAuditLog`, the substrate's append-only audit +log. It also provides two supporting types. `AuditEntry` is one immutable +log row. `AuditValue` is a typed field value inside an entry. + +The log is the substrate's source of truth. A row's visible current +state is a projection, computed by replaying its log entries in order. +It is never stored independently of the log. `AuditEntry.id` is a +32-byte SHA-256 content hash, computed over the entry's other fields. +This gives the log a natural way to deduplicate. Two replicas that +independently record the same logical mutation compute the same id. +Once merged, only one copy remains. `AuditValue` is an enum covering four +kinds of value a field change can carry: a bitmap, a string, a +fingerprint, or an integer. It has a hand-written `Codable` conformance. +That conformance encodes each case as a single-key JSON object, such as +`{"bitmap": 42}`. It does not use Swift's default synthesized shape, +`{"bitmap": {"_0": 42}}`. This way the wire format matches what the Rust +port's serde derive produces natively. Both legs agree on one JSON shape. + +`GSetAuditLog` itself stores its entries keyed by content hash +internally, for constant-time deduplication. Its `Codable` conformance +serializes them as a plain array sorted by id instead. The conceptually +correct wire representation of a grow-only set is a set, not a hash map. +The file explains this distinction explicitly, to justify writing custom +`Codable` rather than relying on the default. `GSetAuditLog.add(_:)` +inserts one entry. It is a no-op if the same id is already present. +`GSetAuditLog.merge(_:)` is the CRDT join: the union of two logs' +entries. This is correct regardless of which replica calls it, or in +what order two logs are merged. `GSetAuditLog.orderedEntries` replays +every entry in HLC order. That is the sequence a projection applies to +compute visible state. `GSetAuditLog.entries(forRow:)` and +`GSetAuditLog.entries(since:)` scope that same ordered replay. One scopes +to one row; the other scopes to everything after a cutoff. They serve +the row-state automaton and the sync protocol respectively. The file's +closing comment sketches why this design converges. G-Set merge is set +union. Set union is commutative, associative, and idempotent. HLC gives +any two entries an unambiguous order to replay them in. So any two +replicas that have exchanged all of each other's entries compute +identical visible state. ## MatrixF.swift -This file provides `MatrixF`, the field-presence matrix: for every -(field, bit) pair, how many rows in the estate currently have that bit set. - -`MatrixF` stores 216 `Int64` counts in one flat array, indexed by -`field * bitsPerField + bit`; `MatrixF.cellIndex(field:bit:)` computes that -index with bounds checking. Its layout constants (`fieldCount`, -`bitsPerField`, `cellCount`) are aliases onto the identical constants in -`RowBitmaps`, kept as aliases rather than a second independent definition so -the two types cannot silently drift out of agreement about the shape of the -216-cell grid they both describe. `MatrixF.applyRow(delta:bitVector:)` -updates every cell whose bit is set in a `BitVector216`, adding `delta` — -positive one on a row's capture, negative one on its expunge, or the pair of -calls needed to represent a mutation as a removal followed by an addition. -`MatrixF.totalCount` sums every cell as a sanity check. -`MatrixF.writeWire(into:)` and `MatrixF.readWire(_:)` serialize the matrix -to and from its canonical 1,728-byte little-endian wire form (216 cells of 8 -bytes each). +This file provides `MatrixF`, the field-presence matrix. For every +(field, bit) pair, it counts how many rows in the estate currently have +that bit set. + +`MatrixF` stores 216 `Int64` counts in one flat array. It indexes them +by `field * bitsPerField + bit`. `MatrixF.cellIndex(field:bit:)` computes +that index, with bounds checking. Its layout constants are `fieldCount`, +`bitsPerField`, and `cellCount`. These are aliases onto the identical +constants in `RowBitmaps`. They are kept as aliases, rather than a second +independent definition, so the two types cannot silently drift out of +agreement about the shape of the 216-cell grid they both describe. +`MatrixF.applyRow(delta:bitVector:)` updates every cell whose bit is set +in a `BitVector216`, adding `delta`. That delta is positive one on a +row's capture, negative one on its expunge, or the pair of calls needed +to represent a mutation as a removal followed by an addition. +`MatrixF.totalCount` sums every cell, as a sanity check. +`MatrixF.writeWire(into:)` and `MatrixF.readWire(_:)` serialize the +matrix to and from its canonical 1,728-byte little-endian wire form: +216 cells of eight bytes each. ## MatrixC.swift -This file provides `MatrixC`, the correlation matrix: the marginal -probability of each (field, bit) pair, derived from `MatrixF` rather than -updated independently. +This file provides `MatrixC`, the correlation matrix. It holds the +marginal probability of each (field, bit) pair, derived from `MatrixF` +rather than updated independently. -`MatrixC` shares `MatrixF`'s 216-cell shape but stores `Float` values in +`MatrixC` shares `MatrixF`'s 216-cell shape. It stores `Float` values in `[0, 1]` instead of raw counts. `MatrixC.derive(from:nRows:)` is its only -way to change: it divides each of `MatrixF`'s counts by the total row count, -routing the division through `Double` before narrowing back to `Float32`, -which the file notes is necessary for cross-language agreement — both the -Swift and Rust legs perform the identical cast-divide-cast sequence, and -IEEE-754 guarantees the same bit pattern results on both. When there are no -rows at all, every cell is zero rather than an undefined division result. -`MatrixC.writeWire(into:)` and `MatrixC.readWire(_:)` serialize to and from -an 864-byte wire form (216 cells of 4 bytes each, `Float32` bit patterns). +way to change. It divides each of `MatrixF`'s counts by the total row +count. The division routes through `Double` before narrowing back to +`Float32`. The file notes this is necessary for cross-language +agreement. Both the Swift and Rust legs perform the identical +cast-divide-cast sequence. IEEE-754 guarantees the same bit pattern +results on both. When there are no rows at all, every cell is zero, +rather than an undefined division result. `MatrixC.writeWire(into:)` and +`MatrixC.readWire(_:)` serialize to and from an 864-byte wire form: +216 cells of four bytes each, as `Float32` bit patterns. ## MatrixO.swift -This file provides `MatrixO`, the co-occurrence matrix: how often each pair -of (field, value) settings appears together across rows, and -`CooccurrenceKey`, the packed key identifying one such pair. - -Because most of the roughly 46,000 possible (field, value, field, value) -combinations never occur in a typical estate, `MatrixO` stores only nonzero -cells, as a list sorted by `CooccurrenceKey.packed` rather than as a dense -array or an unordered dictionary — sorted order makes both iteration and -serialization deterministic across languages, which an unordered dictionary -could not guarantee. `CooccurrenceKey.packed` folds all four components -into one `UInt32` for compact storage and for the ordering comparison. -`MatrixO.count(_:)` and `MatrixO.increment(_:by:)` use binary search over -the sorted entries to read or update one cell in logarithmic time, and -`increment` removes a cell entirely if its count returns to zero, keeping -the canonical form free of dead entries. `MatrixO.applyRow(delta:fieldValues:)` -updates the matrix for one row: it iterates every ordered pair of the row's -(field, value) settings, including a field paired with itself, and -increments each pair's cell by `delta`. The file notes explicitly that the -matrix is not required to be symmetric in storage even though co-occurrence -is conceptually symmetric — each unordered pair contributes to two distinct -cells, (i, j) and (j, i) — and that an implementation which stores only one -direction and infers the other is a valid optimization, but the reference -here stores both for clarity. `MatrixO.writeWire(into:)` and -`MatrixO.readWire(_:)` define the canonical wire form: an entry count -followed by each entry's packed key and count. +This file provides `MatrixO`, the co-occurrence matrix. It counts how +often each pair of (field, value) settings appears together across rows. +It also provides `CooccurrenceKey`, the packed key identifying one such +pair. + +Each combination pairs one field-value setting against another. Most of +the roughly forty-six thousand possible combinations never occur in a +typical estate. So `MatrixO` stores +only nonzero cells, as a list sorted by `CooccurrenceKey.packed`, rather +than as a dense array or an unordered dictionary. Sorted order makes both +iteration and serialization deterministic across languages. An unordered +dictionary could not guarantee that. `CooccurrenceKey.packed` folds all +four components into one `UInt32`, for compact storage and for the +ordering comparison. `MatrixO.count(_:)` and `MatrixO.increment(_:by:)` +use binary search over the sorted entries. This reads or updates one +cell in logarithmic time. `increment` removes a cell entirely if its +count returns to zero, keeping the canonical form free of dead entries. +`MatrixO.applyRow(delta:fieldValues:)` updates the matrix for one row. It +iterates every ordered pair of the row's field-value settings. This +includes a field paired with itself. It increments each pair's cell by +`delta`. The file notes the matrix is not required to be symmetric in +storage, even though co-occurrence is conceptually symmetric. Each +unordered pair contributes to two distinct cells, (i, j) and (j, i). An +implementation that stores only one direction and infers the other is a +valid optimization. The reference here stores both, for clarity. +`MatrixO.writeWire(into:)` and `MatrixO.readWire(_:)` define the +canonical wire form: an entry count, followed by each entry's packed key +and count. ## MatrixT.swift -This file provides `MatrixT`, the temporal causality matrix: how often a row -with one (field, value) setting is followed, within a bounded time window, -by a row with another (field, value) setting — the substrate's tool for -distinguishing "these things happen together" from "one of these things +This file provides `MatrixT`, the temporal causality matrix. It counts +how often a row with one field-value setting is followed by a row with +another field-value setting. This must happen within a bounded time +window. It is the substrate's tool for telling apart two ideas. One idea +is "these things happen together." The other is "one of these things tends to precede the other." `CausalityKey` extends `CooccurrenceKey`'s idea with a fifth component, a -lag bucket, and packs all five into a `UInt64`. `MatrixT.lagBucket(forMinutes:)` -converts a raw time difference in minutes into one of eight log-spaced -buckets — 1, 2, 4, 8, 16, 32, 64, and 128 minutes as lower bounds — returning -`nil` for a difference outside the supported 1-to-256-minute window, in -which case no update to the matrix occurs at all. The file states this -matrix is explicitly asymmetric: the cell for "(field i, value i) preceding +lag bucket. It packs all five into a `UInt64`. +`MatrixT.lagBucket(forMinutes:)` converts a raw time difference in +minutes into one of eight log-spaced buckets. The lower bounds are one, +two, four, eight, sixteen, thirty-two, sixty-four, and one hundred +twenty-eight minutes. It returns `nil` for a difference outside the +supported one-to-two-hundred-fifty-six-minute window. In that case, no +update to the matrix occurs at all. The file states this matrix is +explicitly asymmetric. The cell for "(field i, value i) preceding (field j, value j)" is tracked completely separately from the reverse -direction, unlike `MatrixO`'s conceptually symmetric co-occurrence. Storage, -lookup (`count(_:)`), update (`increment(_:by:)`), and wire serialization -follow the identical sorted-array, binary-search pattern `MatrixO` uses. -`MatrixT.applyPair(delta:rowAFieldValues:rowBFieldValues:lagMinutes:)` is the -row-pair version of `MatrixO.applyRow`: given that row A precedes row B by a -known number of minutes, it increments the cell for every combination of A's -field values against B's field values, once a valid lag bucket exists for -that separation. +direction. This is unlike `MatrixO`'s conceptually symmetric +co-occurrence. Storage works the same way `MatrixO`'s storage does. +Lookup through `count(_:)` and update through `increment(_:by:)` follow +the same sorted-array, binary-search pattern too. So does wire +serialization. +`MatrixT.applyPair(delta:rowAFieldValues:rowBFieldValues:lagMinutes:)` is +the row-pair version of `MatrixO.applyRow`. Given that row A precedes row +B by a known number of minutes, it increments the cell for every +combination of A's field values against B's field values. This happens +once a valid lag bucket exists for that separation. ## ThreeDBitTensor.swift -This file provides `ThreeDBitTensor`, the dense bit-sliced storage layout -that answers "which rows have field X set to value Y" quickly across up to -a million rows. - -Rather than storing one 216-bit block per row (a row-major layout), the -tensor keeps six separate bit-slices, one per bit position 0 through 5 of a -field's six-bit value; each slice packs one bit per (row, field) pair. This -"bit-sliced" arrangement is what makes `scanFieldEquals(field:value:)` fast: -testing whether a whole batch of rows matches a target value reduces to -scanning six flat byte buffers rather than reading and unpacking a -six-bit value out of every row individually. `valueAt(row:field:)` and -`setValue(row:field:value:)` provide ordinary cell-level read and write, -each internally looping over the six bit-slices; `setValue` enforces the -six-bit width invariant with a precondition. `bitSet(row:field:bit:)` and -`setBit(row:field:bit:on:)` are the underlying single-bit primitives both -higher-level functions call. `scanFieldEquals(field:value:)` returns a byte -mask marking every row matching the target value; `enumerateMatches(_:)` -turns that mask into a plain array of matching row indices. -`reserveCapacity(_:)` grows the tensor to hold more rows, extending each -slice with zero bytes and leaving all existing data untouched — a no-op if -the requested size is not larger than the current one. The file notes the -current scan implementation loops row by row within each bit-slice pass -rather than operating on whole 64-bit words at once, which is a known -optimization opportunity, not yet taken, for a future vectorized version. +This file provides `ThreeDBitTensor`, the dense bit-sliced storage +layout. It answers "which rows have field X set to value Y" quickly, +across up to a million rows. + +Rather than storing one 216-bit block per row, a row-major layout, the +tensor keeps six separate bit-slices. There is one slice per bit position +zero through five of a field's six-bit value. Each slice packs one bit +per (row, field) pair. This bit-sliced arrangement is what makes +`scanFieldEquals(field:value:)` fast. Testing whether a whole batch of +rows matches a target value reduces to scanning six flat byte buffers. +That beats reading and unpacking a six-bit value out of every row +individually. `valueAt(row:field:)` and `setValue(row:field:value:)` +provide ordinary cell-level read and write. Each internally loops over +the six bit-slices. `setValue` enforces the six-bit width invariant with +a precondition. `bitSet(row:field:bit:)` and `setBit(row:field:bit:on:)` +are the underlying single-bit primitives both higher-level functions +call. `scanFieldEquals(field:value:)` returns a byte mask, marking every +row matching the target value. `enumerateMatches(_:)` turns that mask +into a plain array of matching row indices. `reserveCapacity(_:)` grows +the tensor to hold more rows. It extends each slice with zero bytes and +leaves all existing data untouched. This is a no-op if the requested +size is not larger than the current one. The file notes the current scan +implementation loops row by row within each bit-slice pass, rather than +operating on whole 64-bit words at once. This is a known optimization +opportunity, not yet taken, for a future vectorized version. ## RecallTypes.swift -This file provides the shared wire vocabulary that every recall primitive -and every federation query returns: `RecallScore`, `DistanceBreakdown`, -`RecallResult`, and `RowProjection`. - -These four types live in SubstrateTypes, rather than in whichever library -happens to implement a particular recall strategy, because federation needs -them too — a federated query's response is shaped exactly like a local -recall result — and because defining them once here means no two libraries -can quietly redefine the same shape and drift apart. `RecallScore` pairs one -`RowId` with one `Float32` score; the file is explicit that the score's -meaning is specific to whichever recall primitive produced it (cosine -similarity for vector recall, inverted Hamming distance for fingerprint -recall, BM25 for text recall), so any code that combines scores from -different primitives must normalize first. `DistanceBreakdown` reports how -much each of four components — lattice, fingerprint, temporal, bitmap — -contributed to a match, each normalized to `[0, 1]`, both to explain "why -this matched" to a caller and to weight Reciprocal Rank Fusion when -combining several recall strategies' results. `RecallResult` bundles a -ranked list of scores with an optional breakdown, an optional confidence -interval, and the name of the primitive that produced it, so a composition -step downstream knows which combination rule to apply. `RowProjection` is -the minimal slice of a row that a recall primitive actually needs to rank -candidates: its identifier, capture time, fingerprint, lattice anchor, -bitmaps, and lifecycle state, deliberately omitting the row's verbatim -content and any metadata beyond the bitmaps. The file explains this -omission is intentional — ranking operates on structure, and the heavier -verbatim content is fetched separately, only for the rows that survive -ranking. +This file provides the shared wire vocabulary that every recall +primitive and every federation query returns: `RecallScore`, +`DistanceBreakdown`, `RecallResult`, and `RowProjection`. + +These four types live in SubstrateTypes, rather than in whichever +library happens to implement a particular recall strategy. Federation +needs them too. A federated query's response is shaped exactly like a +local recall result. Defining them once here also means no two libraries +can quietly redefine the same shape and drift apart. `RecallScore` pairs +one `RowId` with one `Float32` score. The file is explicit that the +score's meaning is specific to whichever recall primitive produced it. +Vector recall uses cosine similarity. Fingerprint recall uses inverted +Hamming distance. Text recall uses BM25. Any code that combines scores +from different primitives must normalize first. `DistanceBreakdown` +reports how much each of four components contributed to a match: lattice, +fingerprint, temporal, and bitmap. Each is normalized to `[0, 1]`. This +serves two purposes: it explains "why this matched" to a caller, and it +weights Reciprocal Rank Fusion when combining several recall strategies' +results. `RecallResult` bundles a ranked list of scores with an optional +breakdown, an optional confidence interval, and the name of the primitive +that produced it. A composition step downstream then knows which +combination rule to apply. `RowProjection` is the minimal slice of a row +that a recall primitive actually needs to rank candidates. It carries the +row's identifier, capture time, fingerprint, lattice anchor, bitmaps, and +lifecycle state. It deliberately omits the row's verbatim content and any +metadata beyond the bitmaps. The file explains this omission is +intentional. Ranking operates on structure. The heavier verbatim content +is fetched separately, only for the rows that survive ranking. ## Rust Port and Conformance -The `rust/` directory mirrors this package one module per Swift file: -`rust/src/row.rs` for `Row.swift`, `rust/src/fingerprint256.rs` for -`Fingerprint256.swift`, and so on through all thirty files, with -`rust/src/lib.rs` re-exporting the primary types at the crate root the same -way this package's types are used directly by Swift callers. There is no -separate conformance-fixture directory analogous to LatticeLib's -`rust/tests/fixtures/`; because SubstrateTypes ships no pinned data -artifacts, agreement between the two legs is instead verified by shared -test vectors embedded directly in each module's own test block (for -example, the illustrative Hamming and Fingerprint256 test vectors quoted in -their Swift source comments), checked against the equivalent Rust unit -tests. When a function in this package changes — most sensitively the -fingerprint combinators, the Hamming and SimHash arithmetic, and the FNV -hash — the corresponding Rust module must change identically, because every -one of these functions is a cross-platform agreement contract, not an -implementation detail local to one language. +The `rust/` directory mirrors this package, one module per Swift file. +`rust/src/row.rs` matches `Row.swift`. `rust/src/fingerprint256.rs` +matches `Fingerprint256.swift`. The pattern continues through all thirty +files. `rust/src/lib.rs` re-exports the primary types at the crate root, +the same way this package's types are used directly by Swift callers. +There is no separate conformance-fixture directory, unlike LatticeLib's +`rust/tests/fixtures/`. SubstrateTypes ships no pinned data artifacts. So +agreement between the two legs is instead verified by shared test +vectors, embedded directly in each module's own test block. The +illustrative Hamming and Fingerprint256 test vectors, quoted in their +Swift source comments, are one example. These are checked against the +equivalent Rust unit tests. When a function in this package changes, the +corresponding Rust module must change identically. This matters most for +the fingerprint combinators, the Hamming and SimHash arithmetic, and the +FNV hash. Every one of these functions is a cross-platform agreement +contract, not an implementation detail local to one language. diff --git a/packages/libs/SubstrateTypes/docs/OVERVIEW.md b/packages/libs/SubstrateTypes/docs/OVERVIEW.md index 2710de4..385d5e1 100644 --- a/packages/libs/SubstrateTypes/docs/OVERVIEW.md +++ b/packages/libs/SubstrateTypes/docs/OVERVIEW.md @@ -71,144 +71,155 @@ sources: ## What This Library Does -SubstrateTypes defines the smallest shared unit of information that MOOTx01 -stores: a row. A row is one recorded observation — a diary entry, a fact -pulled from a knowledge graph, a proposal, a sample of ambient context — held -in one common shape so every other library can read it and write it the same -way. SubstrateTypes is where that shape, and the handful of coordinate -systems attached to it, are defined once for the whole SDK. - -Two coordinate systems ride on every row. A fingerprint is a 256-bit code -that captures structural similarity: rows that look alike in shape produce -fingerprints that share many bits. A lattice anchor is a compact reference to -a lattice code, the classification code that LatticeLib's FDC engine assigns; -rows about the same subject carry related anchors. A third value, the Hybrid -Logical Clock (HLC), timestamps every change to a row so that two devices -holding copies of the same estate can agree on the order of events without -trusting either device's wall clock. - -Around these three coordinate systems sit a small set of pure, deterministic -functions: Hamming distance and bitwise arithmetic over fingerprints, -SimHash construction from hyperplane families, FNV string hashing, and -OR-reduction for combining many fingerprints into one. These functions are -"canonical reference compute" — not heavy algorithms, but the exact -arithmetic that every consumer of the shapes must reproduce identically, on -every platform, forever. +SubstrateTypes defines the smallest shared unit that MOOTx01 stores. That +unit is a row. A row is one recorded observation. A row can be a diary +entry. It can be a fact pulled from a knowledge graph. It can be a +proposal. It can be a sample of ambient context. Every row holds the same +shape. That way, every other library can read it and write it the same +way. SubstrateTypes is where that shape lives. It also defines a few +coordinate systems that ride on every row. + +Two coordinate systems sit on every row. The first is a fingerprint. A +fingerprint is a 256-bit code that captures structural similarity. Rows that +look alike in shape produce fingerprints that share many bits. The second is +a lattice anchor. This is a compact reference to a lattice code. LatticeLib's +FDC engine assigns that code. Rows about the same subject carry related +anchors. A third value, the Hybrid Logical Clock (HLC), timestamps every +change to a row. This lets two devices holding copies of the same estate +agree on the order of events. Neither device needs to trust the other's wall +clock. + +Around these three coordinate systems sit a small set of pure functions. +These include Hamming distance and bitwise arithmetic over fingerprints. +They also include SimHash construction from hyperplane families. Two more +are FNV string hashing and OR-reduction, which folds many fingerprints into +one. Each function is deterministic: the same input always gives the same +output. Call this set "canonical reference compute." These are not heavy +algorithms. They are the exact arithmetic that every consumer of the shapes +must reproduce the same way, on every platform, forever. ## The Problem It Solves -MOOTx01's higher libraries — the ones that rank memories, detect drift, or -sync an estate across a phone and a laptop — all need to talk about the same -row shape. Without one shared definition, each library would invent its own -version of a fingerprint or a timestamp, and the definitions would drift -apart. A comparison between two rows would then depend on which library -produced them, which breaks every guarantee the system makes about -comparing memories. +MOOTx01 has higher libraries. They rank memories. They detect drift. They +sync an estate across a phone and a laptop. All of them need to talk about +the same row shape. Without one shared definition, each library would +invent its own version of a fingerprint or a timestamp. Those definitions +would drift apart. A comparison between two rows would then depend on +which library produced them. That breaks every guarantee the system makes +about comparing memories. SubstrateTypes solves this by being the single, dependency-free home for these shapes. It ships as the lowest of four packages that together make up -the substrate: SubstrateTypes (this package, pure shape and zero compute -beyond canonical reference arithmetic), SubstrateKernel (dispatch to -hardware-accelerated backends), SubstrateML (learned models), and -SubstrateLib (the orchestration layer, which still owns the row-state -automaton and the verb mechanics that operate the shapes). SubstrateTypes +the substrate. SubstrateTypes, this package, holds pure shape and zero +compute beyond canonical reference arithmetic. SubstrateKernel dispatches to +hardware-accelerated backends. SubstrateML holds learned models. +SubstrateLib is the orchestration layer. It still owns the row-state +automaton and the verb mechanics that operate the shapes. SubstrateTypes depends on none of the other three. A library that only needs to describe a -row's shape — for example, ConvergenceKit, which serializes rows for -CloudKit — depends on SubstrateTypes alone and pulls in no compute it does -not need. +row's shape depends on SubstrateTypes alone. ConvergenceKit is one example. +It serializes rows for CloudKit. It pulls in no compute it does not need. A second problem the package solves is cross-platform agreement. MOOTx01 -estates can federate, so a Swift-based Apple client and a Rust-based -service must compute the identical Hamming distance, the identical -fingerprint, and the identical hash for the same input. The package ships a -Rust port in `rust/` that mirrors every type and function here, so both legs -of the SDK share one arithmetic contract. +estates can federate. A Swift-based Apple client and a Rust-based service +often work on the same estate. Both must compute the identical Hamming +distance for one input. Both must compute the identical fingerprint. Both +must compute the identical hash. The package ships a Rust port in `rust/` +that mirrors every type and function here. Both legs of the SDK share one +arithmetic contract. ## How It Works -Every row (`Row`) carries a fixed set of fields: an identifier, a noun type -(what kind of thing it is — a diary entry, a proposal, and so on), a -lifecycle state, three bitmap columns of adjective, operational, and -provenance flags, a fingerprint, a lattice anchor, and optional lineage and -content references. `RowState` and `RowBitmaps` describe the lifecycle and -the bitmap layout in more detail; both are pure data, so the automaton that -enforces which state transitions are legal lives one layer up, in -SubstrateLib. - -The fingerprint is built by SimHash, a locality-sensitive hash: unlike a -cryptographic hash, where a one-bit change in the input scrambles the whole -output, SimHash produces outputs where similar inputs share many bits. Each -of the fingerprint's four 64-bit blocks is computed by comparing an input -vector against sixty-four random hyperplanes — flat dividing surfaces fixed -at estate creation — recorded in a `HyperplaneFamily`. Once built, -fingerprints are compared and combined by a small algebra: `Hamming.distance` -counts differing bits, `BitwiseArithmetic` computes intersection and -symmetric difference, `ORReduce` folds many fingerprints into one that keeps -their shared structure while losing which fingerprint contributed which bit, -and `CountVector256` accumulates per-bit counts across a whole cohort so a -majority-vote fingerprint, or any other statistic, can be read off later. +Every row (`Row`) carries a fixed set of fields. These are an identifier, a +noun type, and a lifecycle state. The noun type says what kind of thing the +row is: a diary entry, a proposal, and so on. A row also carries three +bitmap columns of adjective, operational, and provenance flags. It carries a +fingerprint, a lattice anchor, and optional lineage and content references. +`RowState` and `RowBitmaps` describe the lifecycle and the bitmap layout in +more detail. Both are pure data. The automaton that enforces which state +transitions are legal lives one layer up, in SubstrateLib. + +The fingerprint is built by SimHash, a locality-sensitive hash. A +cryptographic hash scrambles its whole output when one bit of input changes. +SimHash works the opposite way: similar inputs share many output bits. Each +of the fingerprint's four 64-bit blocks compares an input vector against +sixty-four random hyperplanes. A hyperplane is a flat dividing surface, +fixed once at estate creation. The full set of them is recorded in a +`HyperplaneFamily`. + +Once built, fingerprints are compared and combined by a small algebra. +`Hamming.distance` counts differing bits. `BitwiseArithmetic` computes +intersection and symmetric difference. `ORReduce` folds many fingerprints +into one. That merged fingerprint keeps their shared structure. It loses +which original fingerprint set which bit. `CountVector256` accumulates +per-bit counts across a whole cohort. A caller can read off a majority-vote +fingerprint from those counts later. A caller can also read off any other +statistic the same way. Every change to a row is recorded, never overwritten. `AuditEvent` is one -recorded mutation; `GSetAuditLog` is a grow-only set of such events, a -conflict-free replicated data type (CRDT). Two replicas of an estate can -exchange their audit logs in any order, merge by taking the union of -entries, and land on the same result, because set union does not care about -order. The `HLC` timestamp on every event gives the log a total order to -replay, and `TimeRange` and `AsOfCoordinate` let a caller ask what a row -looked like at a past point in that order. - -A separate family of types tracks population-level statistics across an -entire estate rather than one row at a time. `MatrixF` counts how often each -bitmap bit is set across all rows; `MatrixC` derives the marginal probability -of each bit from `MatrixF`; `MatrixO` counts how often pairs of bitmap values -co-occur; `MatrixT` counts how often one bitmap value precedes another within -a time window, which is the substrate's tool for telling correlation apart -from likely causation. `ThreeDBitTensor` is the dense, bit-sliced storage -layout that answers "which rows have field X set to value Y" quickly across -a million rows. +recorded mutation. `GSetAuditLog` is a grow-only set of such events: a +conflict-free replicated data type, or CRDT. Two replicas of an estate can +exchange their audit logs in any order. They merge by taking the union of +entries. They land on the same result. Set union does not care about order. +The `HLC` timestamp on every event gives the log a total order to replay. +`TimeRange` and `AsOfCoordinate` let a caller ask what a row looked like at +a past point in that order. + +A separate family of types tracks statistics across a whole estate, rather +than one row at a time. `MatrixF` counts how often each bitmap bit is set +across all rows. `MatrixC` derives the marginal probability of each bit from +`MatrixF`. `MatrixO` counts how often pairs of bitmap values co-occur. +`MatrixT` counts how often one bitmap value precedes another within a time +window. This is the substrate's tool for telling correlation apart from +likely causation. `ThreeDBitTensor` is a dense, bit-sliced storage layout. +It answers "which rows have field X set to value Y" quickly, across a +million rows. A last family of types protects data integrity and supports recall. -`ContentHash` and `MerkleRoot` are two distinct fixed-size hashes — -one per leaf payload, one per subtree of children — kept as separate types so -the compiler rejects any code that confuses them; `MerkleDomain` supplies the -one-byte tags that keep leaf, interior, and tombstone hashes from ever -colliding. `RecallTypes` defines the wire vocabulary — `RecallScore`, -`RecallResult`, `DistanceBreakdown`, `RowProjection` — that every recall -primitive and every federation query returns, so composing results from -different recall strategies never requires an ad hoc translation step. +`ContentHash` and `MerkleRoot` are two distinct fixed-size hashes: one per +leaf payload, one per subtree of children. They are kept as separate types +so the compiler rejects any code that confuses them. `MerkleDomain` supplies +the one-byte tags that keep leaf, interior, and tombstone hashes from ever +colliding. `RecallTypes` defines the wire vocabulary that every recall +primitive returns: `RecallScore`, `RecallResult`, `DistanceBreakdown`, and +`RowProjection`. Every federation query returns this same vocabulary. +Composing results from different recall strategies never needs an ad hoc +translation step. ## How the Pieces Fit -Figure 1 shows how the major types connect: a row's own fields, the -fingerprint construction pipeline, the fingerprint algebra, the -population-statistics matrices, the audit trail, and the integrity and -recall vocabularies that read from a row without holding a reference to it. +Figure 1 shows how the major types connect. It shows a row's own fields, +the fingerprint construction pipeline, and the fingerprint algebra. It also +shows the population-statistics matrices, the audit trail, and the +integrity and recall vocabularies. These last two read from a row without +holding a reference to it. ![Figure 1. Topology of SubstrateTypes](topology.svg) *Figure 1. Topology of SubstrateTypes. A row carries a fingerprint and a -lattice anchor. Hyperplane families and SimHash build the fingerprint; the -fingerprint algebra (Hamming distance, bitwise arithmetic, OR-reduction, -count-vectors) operates on it afterward. The row's bitmaps feed the -population-statistics matrices. Every mutation is recorded as an audit event -under Hybrid-Logical-Clock ordering in the grow-only audit log. Integrity -hashes and the recall wire vocabulary both read from row data without -belonging to the row shape itself.* - -Nothing in this package reaches upward. The row-state automaton, the verb -implementations that call it, and the full substrate object that bundles -rows with an audit log and the statistics matrices all remain in -SubstrateLib. SubstrateTypes supplies the vocabulary; SubstrateLib supplies -the behavior. +lattice anchor. Hyperplane families and SimHash build the fingerprint. The +fingerprint algebra, made of Hamming distance, bitwise arithmetic, +OR-reduction, and count-vectors, operates on it afterward. The row's +bitmaps feed the population-statistics matrices. Every mutation is recorded +as an audit event. This happens under Hybrid Logical Clock ordering, in the +grow-only audit log. Integrity hashes and the recall wire vocabulary both +read from row data. Neither belongs to the row shape itself.* + +Nothing in this package reaches upward. The row-state automaton stays in +SubstrateLib. So do the verb implementations that call it. So does the +full substrate object that bundles rows with an audit log and the +statistics matrices. SubstrateTypes supplies the vocabulary. SubstrateLib +supplies the behavior. ## What Ships in the Package -The package ships thirty Swift source files under `Sources/SubstrateTypes/` -and a mirrored Rust crate under `rust/src/`, one Rust module per Swift file -with matching names. There are no pinned data artifacts in this package — -unlike a library such as LatticeLib, SubstrateTypes carries no bundled -reference tables, because every value here is either supplied by a caller -(a hyperplane seed, a row's own fields) or computed from caller-supplied -input by a pure function (a Hamming distance, a SimHash block, an FNV hash). -Reproducibility rests on the arithmetic itself being pinned, verified by -conformance tests that check the Swift and Rust legs agree bit for bit. +The package ships thirty Swift source files under `Sources/SubstrateTypes/`. +It also ships a mirrored Rust crate under `rust/src/`. One Rust module +matches each Swift file, by name. There are no pinned data artifacts in +this package. A library such as LatticeLib carries bundled reference tables. +SubstrateTypes carries none, because every value here comes from one of two +places. A caller supplies it directly, such as a hyperplane seed or a row's +own fields. A pure function computes it from caller-supplied input instead. +Examples are a Hamming distance, a SimHash block, and an FNV hash. The +arithmetic itself is pinned, and that is what makes results reproducible. +Conformance tests verify this: they check that the Swift and Rust legs +agree bit for bit.