diff --git a/packages/libs/EngramLib/docs/AGENT_MAP.md b/packages/libs/EngramLib/docs/AGENT_MAP.md new file mode 100644 index 0000000..0e539e6 --- /dev/null +++ b/packages/libs/EngramLib/docs/AGENT_MAP.md @@ -0,0 +1,62 @@ +--- +doc: AGENT_MAP +package: EngramLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/EngramLib/EngramLib.swift + blob: b7f236bfaa193967daf5c5d6061ef4591207ed22 + - path: Sources/EngramLib/Match.swift + blob: 5c61fd65888a795c703ab9f52022dffeae9c7595 +--- + +# AGENT_MAP: EngramLib + +PURPOSE: product-facing similarity/retrieval/aggregation API over 256-bit engrams (`Fingerprint256`). Thin wrapper that hides substrate kernel selection: distance, batch-distance, top-K nearest, radius filter, OR-union. Two files total; no internal multi-stage pipeline. + +DEPS: imports SubstrateTypes (`Fingerprint256`, aliased `Engram`), SubstrateKernel (`SubstrateKernel` protocol, `PortableKernel.kernelForCurrentPlatform()`). Imported by (within SDK): VectorKit, CorpusKit (kits, moot-memory repo: kits compose libs, never the reverse). Rust port in `rust/` (`engram_lib` crate) mirrors the full API 1:1 against `substrate-types`/`substrate-kernel` Rust crates; conformance is behavioral parity checked by mirrored test suites (`rust/tests/engram_lib_tests.rs` vs `Tests/EngramLibTests/`), not a shared fixture file. + +ENTRY POINTS (most callers need only these): +- EngramLib.swift:77 `EngramLib.distance(_ a: Engram, _ b: Engram) -> Int` | Hamming distance, 0...256 +- EngramLib.swift:99 `EngramLib.findNearest(probe:in:k:) -> [Match]` | top-K nearest, distance-asc / index-asc tiebreak +- EngramLib.swift:216 `EngramLib.session() -> Session` | reusable kernel handle for hot loops + +## Symbol Table + +### Engram type: EngramLib.swift +- :43 `public typealias Engram = Fingerprint256` | substrate type, product-facing name; do not add a wrapping struct +- :45-53 `extension Engram { init(blocks b0:_ b1:_ b2:_ b3:) }` | convenience 4-block constructor +- :33 `internal let _engramLibCachedKernel: any SubstrateKernel` | resolved ONCE at module load via `PortableKernel.kernelForCurrentPlatform()`; every call path reads this, never re-resolves + +### Static API: enum EngramLib (EngramLib.swift:71) +- :77 `distance(_:_:) -> Int` | single-pair Hamming distance +- :84 `distances(probe:candidates:) -> [Int]` | batched, same order/count as `candidates`; `[]` on empty input +- :99 `findNearest(probe:in:k:) -> [Match]` | `[]` if `k <= 0` or candidates empty; clamps to `min(k, candidates.count)` +- :112 `findNearest(probe:in:) -> Match?` | k=1 convenience; `nil` on empty candidates +- :124 `findWithin(probe:in:maxDistance:) -> [Match]` | `[]` if candidates empty OR `maxDistance < 0`; inclusive upper bound +- :149 `union(_ engrams: [Engram]) -> Engram` | bitwise OR-reduce; zero engram is the empty-input identity +- :154 `union(_ a:_ b:) -> Engram` | pairwise OR convenience, delegates to `Fingerprint256.union(_:)` +- :222 `private static func kernel() -> any SubstrateKernel` | single choke point back to the cached kernel + +### Session: struct EngramLib.Session (EngramLib.swift:166), Sendable +- :169 `init()` | captures `_engramLibCachedKernel` +- :173/:177/:184/:193/:210 `distance` / `distances` / `findNearest` / `findWithin` / `union` | instance mirrors of the static functions above, byte-identical results, no new behavior +- :216 `EngramLib.session() -> Session` | factory, equivalent to `Session()` + +### Result type: Match.swift +- :14 `public struct Match: Hashable, Sendable, Codable` | `index: Int` (position in caller's candidate array), `distance: Int` (Hamming distance from probe) +- :18 `init(index:distance:)` | public memberwise init +- :24-28 `extension Match: Comparable` | `<` compares `distance` first, `index` second; this IS the sort/tiebreak contract used by `findNearest`/`findWithin` + +## INVARIANTS / GOTCHAS + +- Kernel is resolved exactly once per process (`_engramLibCachedKernel`, EngramLib.swift:33), at module load, not per call. Do not reintroduce per-call kernel resolution: it was removed as a measured, needless dispatch cost (see file header comment, Phase 4.3 / decision 2026-05-28 §6.4.3). +- DO NOT REIMPLEMENT SUBSTRATE MATH (EngramLib.swift:14-24 banner comment). Every primitive this library exposes, Hamming distance, batch distance, top-K, OR-reduce, already exists in SubstrateTypes/SubstrateKernel/SubstrateML; EngramLib only forwards to it. +- Tie-break rule is fixed and load-bearing: distance ascending, then candidate index ascending (`Match.<`, Match.swift:25-27). `findNearest` and `findWithin` both depend on this for deterministic, reproducible output: same probe + same candidate list + same order = same result, always. +- All empty/degenerate-input guards return empty collections or `nil`, never throw and never crash: empty candidates → `[]`/`nil`; `k <= 0` → `[]`; `maxDistance < 0` → `[]`; empty `union` input → `Engram.zero` (the OR identity). +- `Session` and the static functions are result-identical by construction: both bottom out in the same cached kernel. `Session` exists only to save the small per-call cost of re-touching the static cache in a hot loop; it is not a different code path. +- `Engram` is a type alias, not a wrapper type. Treat the underlying representation (`Fingerprint256`, four `UInt64` blocks) as substrate-owned; construct via `Engram(blocks:_:_:_:)` or substrate constructors, not by touching blocks directly from product code. +- Whole library is stateless and thread-safe (file header, EngramLib.swift:7-10): every static method creates no persistent state; `Session` is `Sendable` and safe to share across tasks. +- Rust and Swift legs must stay behaviorally identical: same guard conditions, same tie-break order, same identity values. There is no shared byte-fixture gate here (contrast LatticeLib's conformance JSON): parity is enforced by keeping `rust/tests/engram_lib_tests.rs` and `Tests/EngramLibTests/` mirrored. Changing a guard or ordering rule in one leg without the other is a silent cross-platform divergence. +- Package has no pinned artifacts, no build-time-only code, and no privacy seams: unlike larger sibling libs (e.g. LatticeLib), there is nothing here gated by an environment variable or a resource bundle. diff --git a/packages/libs/EngramLib/docs/DETAILS.md b/packages/libs/EngramLib/docs/DETAILS.md new file mode 100644 index 0000000..97272b1 --- /dev/null +++ b/packages/libs/EngramLib/docs/DETAILS.md @@ -0,0 +1,180 @@ +--- +doc: DETAILS +package: EngramLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/EngramLib/EngramLib.swift + blob: b7f236bfaa193967daf5c5d6061ef4591207ed22 + - path: Sources/EngramLib/Match.swift + blob: 5c61fd65888a795c703ab9f52022dffeae9c7595 +--- + +# EngramLib Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. The package has two files: the +public API surface, and the result type it returns. + +## EngramLib.swift + +This file holds the entire public surface of the library. It defines the +`Engram` type alias, the `EngramLib` enum with its static comparison and +retrieval functions, and the `Session` type. Callers that run many +comparisons in a loop use `Session`. + +### How It Works + +The file opens with a type alias: `public typealias Engram = +Fingerprint256`. `Fingerprint256` is a 256-bit code defined in the +substrate's `SubstrateTypes` package. It packs four 64-bit blocks into one +fixed-size fingerprint. EngramLib does not define its own bit shape. It +reuses the substrate's shape, under a product-facing name. This choice +matters for one reason. If a future substrate version widens the shape, +product code that only ever wrote `Engram` never has to change. An +extension adds `Engram.init(blocks:_:_:_:)`. This convenience initializer +takes the four blocks as separate arguments. Callers can write +`Engram(blocks: 0xDEAD, 0xBEEF, 0xCAFE, 0xBABE)` instead of naming the +underlying block parameters. + +Every comparison in the file calls a kernel in the end. A kernel is an +object, supplied by the substrate's `SubstrateKernel` package, that knows +how to run bit operations well on the current device. The file resolves +this kernel exactly once, at module load time. It stores that kernel in +the private constant `_engramLibCachedKernel`. Every static function then +reads that same cached value, instead of asking the substrate to choose +again. The file's own comment explains why. The kernel choice does not +change while the program runs. Resolving it on every call was pure +overhead, paid on every single check. + +### Function Coverage + +`EngramLib.distance(_:_:)` returns the Hamming distance between two +engrams. Hamming distance counts the bit positions where two equal-length +codes differ. Identical engrams return zero. Engrams that are exact bit +inverses of each other return 256, the full width of the fingerprint. +This function matters because every other comparison in the file builds +from it. + +`EngramLib.distances(probe:candidates:)` computes the distance from one +probe engram to every candidate in a list. It returns an array in the +same order as the input. This matters because computing distances one at +a time, in a loop, would call into the kernel once per candidate. This +function hands the whole batch to the kernel in one call. The kernel then +processes it more efficiently. It returns an empty array right away when +the candidate list is empty, so no work goes to waste. + +`EngramLib.findNearest(probe:in:k:)` returns the `k` candidates closest +to the probe. It sorts them by distance, from nearest to farthest. Ties +break by the candidate's position in the input list. This is the +library's core retrieval function. It answers which memories look most +like this one, bounded to a fixed count. It returns an empty array when +`k` is zero or negative, or when the candidate list is empty. Neither +case counts as an error. When `k` exceeds the candidate count, the +function returns every candidate rather than failing. + +`EngramLib.findNearest(probe:in:)` is the two-argument overload. It is a +convenience for the common case of wanting only the single closest match. +It calls the three-argument version with `k` fixed to one. It returns the +first result, or `nil` when the candidate list is empty. This overload +matters because finding the closest one is common enough to earn its own +name. Callers should not have to unwrap a one-element array by hand. + +`EngramLib.findWithin(probe:in:maxDistance:)` returns every candidate +whose distance from the probe sits at or under `maxDistance`. It sorts +results the same way as `findNearest`. This matters because some callers +do not know in advance how many results they want. They want everything +within a similarity radius, however many candidates that turns out to +include. The function builds the full distance array first, then filters +it. It sorts only the matches that passed the filter. So the sort never +touches a candidate it is about to drop. The function also guards against +a negative `maxDistance`. A negative value cannot match any real +distance. Without the guard, it would silently match nothing, which +would confuse a caller. + +`EngramLib.union(_:)` runs a bitwise OR across a list of engrams. The +result carries a one-bit at every position where at least one input +engram had one. This matters because a memory system often wants one +summary engram for a whole group, such as a cohort of related memories, +or the union of a topic's structural traits. An empty input returns the +zero engram. That is the correct identity value for OR, since combining +it with anything leaves that thing unchanged. + +`EngramLib.union(_:_:)` is the two-argument overload. It ORs exactly two +engrams together. It exists as a light convenience for the common +pairwise case. A caller merging two engrams does not have to build a +two-element array just to call the list version. + +`EngramLib.Session` is a struct that holds one kernel reference. It +exposes `distance`, `distances`, `findNearest`, `findWithin`, and `union` +as instance methods. Each mirrors the behavior of its static counterpart. +It matters for one reason: reuse. The static functions already share one +cached kernel inside, so a `Session` unlocks no new capability. It exists +so that call sites processing thousands of comparisons in a tight loop +can hold one reference. That reference skips the small, repeated cost of +reaching back into the static cache on every call. A session is +`Sendable`. So the same session can be shared safely across concurrent +tasks. `EngramLib.session()` is a static factory that returns a new +`Session`. It is equivalent to calling `Session()` directly. It exists so +callers do not need to know the type's name to build one. + +The private function `kernel()` is the single choke point every static +function calls through to reach the cached kernel. Routing every call +through one small function keeps that indirection in one place. This +matters should the caching strategy ever need to change. + +## Match.swift + +This file provides `Match`, the value every retrieval function in +`EngramLib.swift` returns. + +### How It Works + +A `Match` is a pair of two integers. `index` is the position of the +matched candidate in the caller's original input array. `distance` is the +Hamming distance from the probe to that candidate. The type carries no +reference to the engram itself, only its position and its distance, +because the caller already holds the original candidate array. The caller +can look the engram up by index if needed. Keeping the type this small +makes it cheap to build in bulk during a retrieval scan. + +`Match` conforms to `Hashable`, `Sendable`, and `Codable`. It can be +stored in sets. It can be passed safely across concurrent code. It can be +serialized for storage or logging, without any extra work. + +### Function Coverage + +`Match.init(index:distance:)` is the memberwise initializer. It is public +so that code outside the package can build a `Match` directly. This +covers test suites, or a caller assembling results from its own logic, +rather than only receiving a `Match` from an `EngramLib` function. + +The `Comparable` conformance sits in a small extension. It defines the +order that every retrieval function in `EngramLib.swift` relies on. +Distance ascends first. For two matches at equal distance, index ascends +second. + +This order is what makes `findNearest` and `findWithin` deterministic. +Given the same probe and the same candidate list, the result always +sorts the same way. The tie-break rule never depends on anything but the +candidate's fixed position in the input. + +## Rust Port and Conformance + +The `rust/` directory holds a second implementation of the same API, in +two files. `lib.rs` defines the `EngramLib` struct with its associated +functions (`distance`, `find_nearest`, `find_within`, `union`, and the +rest), plus the `Session` type. `matchx.rs` defines the `Match` struct +with its `Ord` implementation. The function names differ only in the +ways Swift and Rust naming conventions differ. `findNearest` in Swift is +`find_nearest` in Rust. `findNearest(probe:in:)` becomes +`find_nearest_one`, since Rust has no argument-label overloading. The +behavior matches the Swift version function for function, including +every empty-input guard and every tie-break rule. Both legs depend on +their own language's version of `SubstrateTypes` and `SubstrateKernel`. +Both route their bit math through the same kernel abstraction. So a +Swift caller and a Rust caller, comparing the same two engrams, compute +the same distance. `rust/tests/engram_lib_tests.rs` mirrors the Swift +test suite in `Tests/EngramLibTests/`. A change to one leg's behavior +requires the same change to the other leg, and to its tests. diff --git a/packages/libs/EngramLib/docs/INTERFACE_DOCTRINE.md b/packages/libs/EngramLib/docs/INTERFACE_DOCTRINE.md deleted file mode 100644 index d666f58..0000000 --- a/packages/libs/EngramLib/docs/INTERFACE_DOCTRINE.md +++ /dev/null @@ -1,79 +0,0 @@ -# EngramLib Interface Doctrine - -For coding agents using EngramLib in product code or downstream kits. - -## 1. Always go through the public API - -Call `EngramLib.distance`, `EngramLib.findNearest`, `EngramLib.findWithin`, `EngramLib.union`, or `EngramLib.session()`. Never import SubstrateLib directly from product code just to call a kernel method. The dispatcher and kernel selection are internal; consumers do not see them. - -```swift -// CORRECT -let matches = EngramLib.findNearest(probe: query, in: estate, k: 10) - -// WRONG -import SubstrateLib -let raw = PortableKernel.kernelForCurrentPlatform().hammingTopK(...) -``` - -If you find yourself reaching for SubstrateLib directly from a non-substrate kit, file a decision record proposing a new EngramLib surface for that need. - -## 2. Treat Engram as opaque - -The `Engram` typealias maps to `SubstrateLib.Fingerprint256` today. Future substrate versions may use a different representation (wider fingerprint, different block layout). Product code that constructs engrams via `Engram(blocks: b0, b1, b2, b3)` and treats them as opaque survives the change; code that reaches into `.block0`, `.block1` directly does not. - -```swift -// CORRECT -let e = Engram(blocks: 0xDEAD, 0xBEEF, 0xCAFE, 0xBABE) -let d = EngramLib.distance(e1, e2) - -// FRAGILE: depends on the underlying representation -let e: Engram = something -let firstBlock = e.block0 -``` - -If you genuinely need the block layout (federation handshakes, on-wire encoding), do that work inside a substrate-aware kit, not in product code. - -## 3. Use Session for hot loops - -The stateless static methods (`EngramLib.distance`, `EngramLib.findNearest`, etc.) construct a kernel per call. For hundreds or thousands of operations in a hot path, prefer `EngramLib.session()`: the session holds one kernel for reuse. - -```swift -let session = EngramLib.session() -for q in queries { - let matches = session.findNearest(probe: q, in: estate, k: 10) - process(matches) -} -``` - -`EngramLib.Session` is `Sendable` and safe to share across tasks. - -## 4. Match ordering is stable - -`findNearest` and `findWithin` return matches ordered by distance ascending, with ties broken by candidate index ascending. This ordering is deterministic across kernel implementations (NEON, BNNS, Metal, SIMD, scalar). Tests can assume it; product code can rely on it for UI rendering. - -## 5. Distance semantics are Hamming - -All distance methods return Hamming distance, 0...256. Zero means identical engrams; 256 means bit-inverse. Cosine, Euclidean, Jaccard, and Hyperplane-family-aware distances live in VectorKit (mission 6) and federation-aware code (SubstrateLib + ConvergenceKit-Federation). EngramLib's stable contract is Hamming. - -## 6. Union semantics are bitwise OR - -`EngramLib.union(_ engrams:)` returns the bitwise OR of all inputs. The result has a 1-bit at every position where at least one input had a 1-bit. Empty input returns the zero engram. This is the substrate's standard cohort/union operator (paper section 5.3); the same operation backs federation OR-reductions. - -## 7. Empty inputs return empty outputs - -`findNearest(probe:in:[], k:)` returns `[]`. `findWithin(probe:in:[], maxDistance:)` returns `[]`. `distances(probe:candidates: [])` returns `[]`. `union([])` returns `Engram.zero`. No `nil`, no throw, no error. Product code can call these with empty collections safely. - -## 8. When you need something EngramLib does not expose - -File a decision record. Examples of valid additions: - -- A different distance metric (cosine over engrams, hyperplane-family-aware) -- Approximate nearest neighbor (LSH index, HNSW over engrams) -- Streaming top-K over a sequence rather than an array -- Bit-pattern queries (find engrams with all bits in a mask set) - -Examples of invalid additions: - -- Exposing kernel selection (`EngramLib.useMetalKernel`): defeats the dispatcher -- Returning raw substrate types (`EngramLib.rawKernel()`): leaks abstraction -- Mutating-shape methods (engrams are values; product code never mutates them) diff --git a/packages/libs/EngramLib/docs/OVERVIEW.md b/packages/libs/EngramLib/docs/OVERVIEW.md new file mode 100644 index 0000000..380afd4 --- /dev/null +++ b/packages/libs/EngramLib/docs/OVERVIEW.md @@ -0,0 +1,123 @@ +--- +doc: OVERVIEW +package: EngramLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/EngramLib/EngramLib.swift + blob: b7f236bfaa193967daf5c5d6061ef4591207ed22 + - path: Sources/EngramLib/Match.swift + blob: 5c61fd65888a795c703ab9f52022dffeae9c7595 +--- + +# EngramLib Overview + +## What This Library Does + +EngramLib compares engrams. An engram is a fingerprint. It is a short, +fixed-size code computed from a piece of content. Similar content produces +similar fingerprints. So comparing two engrams is fast, even when the +content behind them is long. EngramLib answers three questions about a set +of engrams. How far apart are two of them? Which ones sit nearest to a +given engram? How do many of them combine into one summary engram? + +MOOTx01 is an on-device AI memory system. It stores what an AI observes +over time. It helps the AI recall that memory later. Every stored memory +can carry an engram that sums up its structure. When the system looks for +memories that resemble a new one, it does not compare the memories +directly. It compares their engrams instead. It uses the distance and +nearest-neighbor functions this library provides. + +## The Problem It Solves + +Comparing bit patterns sounds simple. Doing it well and fast, across a +whole memory store, is not simple. The building blocks already exist in a +lower-level package called the substrate. The substrate holds routines +that measure the distance between two 256-bit codes. It holds routines +that scan many codes to find the closest ones. It holds routines that +combine codes with a bitwise OR. The substrate also picks the fastest way +to run these routines on the current device. It chooses among a plain +reference version and hardware-tuned versions for different chip +families. + +Product code should not need to know any of that. Picture a consumer that +wants the ten memories most like a probe engram. That consumer should not +have to pick a chip-specific routine. It should not have to manage a +kernel object. It should not have to reason about which hardware path is +active. EngramLib is a thin, stable layer over the substrate that hides +those choices. It exposes a small set of static functions with plain +names: `distance`, `findNearest`, `findWithin`, `union`. It always picks +the best version for the device it runs on. + +## How It Works + +Every public function in EngramLib passes its work to a kernel. A kernel +is an object that knows how to run bit operations on the current device. +EngramLib picks a kernel once, when the module first loads. It reuses +that kernel for every call after that. Consumers never see the kernel. +They never pick one themselves. + +`EngramLib.distance(_:_:)` reports the Hamming distance between two +engrams. Hamming distance counts the bit positions where two equal-length +codes differ. A smaller number means the two engrams are more alike. The +content they sum up likely is more alike too. `EngramLib.distances` +computes that same distance from one probe engram to many candidates in a +single call. This is faster than calling `distance` in a loop, because the +kernel can batch the work. + +`EngramLib.findNearest` and `EngramLib.findWithin` build on that distance +measure to answer retrieval questions. `findNearest` returns the closest +matches to a probe, ranked by distance. Use it when a caller wants a fixed +number of results, such as the ten memories most like this one. `findWithin` +instead returns every candidate inside a distance radius. Use it when a +caller wants everything similar enough, however many results that turns +out to be. Both functions break ties the same way. When two candidates sit +at equal distance, the one that appeared earlier in the input list comes +first. This tie-break rule matters, because it makes the result +reproducible. The same probe against the same candidate list always +produces the same ranking, on every run. + +`EngramLib.union` combines many engrams into one, bit by bit. It keeps a +bit set in the result wherever any input had that bit set. This helps +build one summary engram for a group of memories, such as a cohort, a +conversation, or a topic cluster. That summary engram carries the +structural traits of every member. + +Most calls create a brief link to the shared kernel and finish. Some code +runs the same kind of comparison many thousands of times in a loop. For +that code, EngramLib also offers a `Session`. A session is a small object +that holds the kernel once. It exposes the same functions as instance +methods. A session returns the same results as the static functions. It +exists only to skip the small cost of finding the kernel again on every +call. + +## How the Pieces Fit + +The package stays small on purpose. One file defines the public API. One +file defines its result type. `EngramLib.swift` holds the `EngramLib` +enum with its static functions, the `Session` type for reuse in hot +loops, and the `Engram` type alias itself. `Match.swift` holds `Match`. +`Match` is the small value every retrieval function returns. It carries +the position of a candidate in its input list and its distance from the +probe. + +`Engram` is a public alias for `Fingerprint256`, a type owned by the +substrate's `SubstrateTypes` package. EngramLib does not define its own +fingerprint shape. It borrows the substrate's shape and gives it a +product-facing name. A future change to that shape then does not force +every consumer to rename their variables. The kernel that does the actual +bit math comes from `SubstrateKernel`, the substrate's other half. The +whole library is two short files that wrap a single external kernel call. +So there is no internal layout worth a diagram. Every function tells the +same story: check the input, then call the kernel. + +## What Ships in the Package + +The package ships the two Swift source files and their test suites. It +also ships a Rust port in `rust/` that mirrors the Swift API function for +function: `EngramLib::distance`, `EngramLib::find_nearest`, +`EngramLib::union`, and the rest. Both legs depend on their own +language's version of the substrate packages. Both forward to the same +kernel logic. So a Swift caller and a Rust caller, comparing the same two +engrams, get the same distance. diff --git a/packages/libs/IntellectusLib/docs/AGENT_MAP.md b/packages/libs/IntellectusLib/docs/AGENT_MAP.md new file mode 100644 index 0000000..e9ea444 --- /dev/null +++ b/packages/libs/IntellectusLib/docs/AGENT_MAP.md @@ -0,0 +1,85 @@ +--- +doc: AGENT_MAP +package: IntellectusLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/IntellectusLib/Intellectus.swift + blob: 47d1dbe60d0ed880eaf657986ef9371f66423fda + - path: Sources/IntellectusLib/RecentWindowSink.swift + blob: 275f059bb1c0b5ebea68c7d32d2e6bb5ad5ed034 + - path: Sources/IntellectusLib/StatSample.swift + blob: 5a63d181680c35a1b79279f83651f8cb62dcd9b7 + - path: Sources/IntellectusLib/StatsSink.swift + blob: 60dc64a00370e491273c2328409a9d56fc46463b +--- + +# AGENT_MAP: IntellectusLib + +PURPOSE: substrate self-report telemetry faculty. Zero-dependency LEAF/floor library. Caller → `Intellectus.report(sample)` → lock-free enabled gate → (if on) installed `StatsSink.receive(_:)`. Off-path = one atomic load + branch, payload never built. + +DEPS: imports Foundation (NSLock; StatSample/RecentWindowSink/Intellectus), Synchronization (Atomic; Intellectus.swift only). Zero repo dependencies: this IS the floor of the dependency tree (SubstrateTypes/SubstrateKernel/SubstrateLib/SubstrateML are meant to depend on this in a later mission per Package.swift header; not yet wired as of this commit). Imported by: not enumerated in Package.swift/product graph as of this commit; source-comment call sites (forward references, not verified imports): `AriaResident.installManagerTelemetry` / `AriaResident.runResidentDaemon` (StatsSink.swift, Intellectus.swift), `LocusKit.EstateVerbs`, `NeuronKit.EstateDreamingSink`, `ObserverSink.PersistenceStatsSink` (StatSample.swift). Rust port in `rust/` mirrors every file 1:1 (sample.rs/sink.rs/window.rs/holder.rs/global.rs/lib.rs); NO shared conformance fixtures: StatSample is opaque caller-supplied data, not a computed/derived value, so there is nothing for the two legs to agree on byte-for-byte. + +ENTRY POINTS (most callers need only these): +- Intellectus.swift:238 `Intellectus.report(_ make: @autoclosure () -> StatSample)`: the only call substrate hot-path code makes; off when disabled, no payload build +- Intellectus.swift:185 `Intellectus.install(sink: any StatsSink)`: host wires a real sink, once, at startup +- Intellectus.swift:205 `Intellectus.setEnabled(_ enabled: Bool)`: host/daemon flips the gate + +## Symbol Table + +### Sample type: StatSample.swift +- :32 `enum StatSample: Sendable`: the telemetry datum; value type, two cases +- :43 `case metric(name:value:tags:ts:)`: named Double measurement + String:String tags +- :70 `case event(kind:nounType:rowID:estate:ts:)`: substrate lifecycle transition; nounType is Int (caller casts NounType→Int to keep this crate dependency-free) +- :86 `enum EventKind: String, Sendable, Hashable, CaseIterable`: `.capture` (:87) | `.think` (:88); rawValue is the wire string +- :96 `StatSample.ts: Double` (extension): timestamp regardless of case + +### Receiver contract: StatsSink.swift +- :36 `protocol StatsSink: Sendable`: one requirement +- :47 `receive(_ sample: StatSample)`: must be non-blocking/inexpensive; not self-enforced, caller (Intellectus) owns the enabled gate, sink does not re-check it +- :60 `struct NoOpSink: StatsSink`: default installed sink; discards, O(1), no allocation +- :63 `NoOpSink.shared`: shared singleton instance +- :69 `NoOpSink.receive(_:)`: intentional no-op + +### Bundled sink: RecentWindowSink.swift +- :63 `final class RecentWindowSink: StatsSink, @unchecked Sendable`: bounded ring buffer + optional forward decorator +- :69 `capacity: Int`: fixed at init, clamped to min 1 +- :72 `forward: (any StatsSink)?` (private): decorator target; nil = window-only +- :76 `lock = NSLock()` (private): guards ring/head/filled/_totalReceived only +- :80-86 `ring`/`head`/`filled`/`_totalReceived` (private): backing storage; _totalReceived counts every receive() ever, ignoring eviction +- :97 `init(capacity:forward:)`: capacity clamped via `max(1, capacity)` +- :114 `receive(_ sample:)`: O(1) ring write under lock (oldest slot overwritten when full); forward call happens AFTER lock release +- :133 `snapshot() -> [StatSample]`: point-in-time copy, oldest-first; safe to iterate lock-free +- :148 `count: Int`: current retained count, 0...capacity +- :157 `totalReceived: Int`: monotonic lifetime receive count; 0 = "nothing has ever crossed the gate," distinct from count==0 + +### Global facade + holder: Intellectus.swift +- :44 `final class _IntellectusHolder: @unchecked Sendable` (internal, not public): all mutable state lives here; `Intellectus` enum is a thin wrapper over the singleton +- :56 `_enabled: Atomic` (private): lock-free gate, default false +- :62 `_sinkLock = NSLock()` (private): guards `_sink` only; acquired only on the on-path (enabled) and during install() +- :67 `_sink: any StatsSink` (private, var): starts as `NoOpSink.shared` +- :77 `install(sink:)`: replace sink under lock; rare/write-path call +- :91 `setEnabled(_:)`: `.releasing` store +- :100 `isEnabled: Bool`: `.acquiring` load +- :118 `report(_ make: @autoclosure () -> StatSample)`: OFF-path: single atomic load(.acquiring) + branch, autoclosure never forced. ON-path: snapshot sink under lock, release lock, THEN evaluate autoclosure, THEN `sink.receive(...)`: payload build and sink call both happen outside the lock +- :141 `let _intellectus = _IntellectusHolder()`: module-private global singleton; the only instance the public API touches +- :172 `enum Intellectus`: public, caseless namespace; all members static +- :185 `Intellectus.install(sink:)`: forwards to `_intellectus.install` +- :205 `Intellectus.setEnabled(_:)`: forwards to `_intellectus.setEnabled`; driven in production by `AriaResident.runResidentDaemon`'s poll loop reading the stats-store monitoring flag (per doc comment; that call site is outside this package) +- :210 `Intellectus.isEnabled: Bool`: forwards to `_intellectus.isEnabled` +- :238 `Intellectus.report(_:)`: public entry; re-declares `@autoclosure` so short-circuit holds end-to-end from call site to atomic check + +## INVARIANTS / GOTCHAS + +- OFF-PATH CONTRACT: when `Intellectus.isEnabled` is false, the `@autoclosure` argument to `report(_:)` is NEVER evaluated: no allocation, no lock, no sink call. This is enforced by ordering the `guard _enabled.load(...)` check BEFORE any reference to `make`. Do not restructure `report(_:)` in a way that forces the closure before the gate check. +- Atomic ordering pairing is load-bearing, not decorative: `setEnabled` stores with `.releasing`, `isEnabled`/the gate load with `.acquiring`. This is what guarantees a `setEnabled(true)` on one thread makes state written before it (e.g., a just-completed `install(sink:)`) visible to `report(_:)` on another thread. Do not change either ordering independently: they are a matched pair, mirrored by Rust's `Ordering::Release`/`Ordering::Acquire` in holder.rs. (Rust's own inline comment on `_enabled` load calls it `Relaxed`-sufficient in one place but the code actually uses `Ordering::Acquire`/`Ordering::Release`: treat the Swift acquire/release pairing as the authoritative contract.) +- Sink lock discipline: `_sinkLock` / `RecentWindowSink.lock` are held ONLY for the O(1) state mutation (snapshot/copy the sink reference, or write one ring slot). The actual `sink.receive(...)` / forward call ALWAYS runs after the lock is released. This is SPEC I-7 in the source comments. Breaking this (calling receive under the lock) risks deadlock if a sink's own `receive` re-enters this library or takes a lock a caller also holds. +- `RecentWindowSink` bound is contractual (SPEC I-8 in source comments): memory is O(capacity) regardless of how many samples are ever received. Do not add unbounded accumulation (e.g., a growing log) to this type. +- `RecentWindowSink.totalReceived` counts EVERY direct `receive(_:)` call, including calls made outside the `Intellectus` gate (e.g., in tests that call `window.receive(...)` directly). It is not scoped to gated traffic only. +- `NoOpSink` is the sink installed before any `install(sink:)` call. Because the default `isEnabled` is false, `NoOpSink.receive` is not expected to run in default configuration: but it must remain safe to call, because a host could call `setEnabled(true)` before `install(sink:)`. +- Timestamps are ALWAYS caller-supplied (`Double` epoch seconds). No file in this package reads a clock. Do not add a `Date()`/clock read inside `IntellectusLib`: determinism for replay/testing depends on this. +- Zero repo dependencies is a structural constraint, not a preference: Package.swift's header comment explicitly forbids importing SubstrateTypes/SubstrateKernel or any other repo target from this package, because this package is meant to become the dependency floor those targets build on. Adding a repo import here would create the exact layering cycle the package exists to avoid. +- `EventKind` is a two-case, `CaseIterable`, `String`-backed enum (`.capture`/`.think`). The test suite pins `allCases.count == 2`: adding a case is a breaking, deliberate surface change, not a silent addition. +- Platform floor is macOS 26 / iOS 26 specifically because `Synchronization.Atomic` needs it per project policy; do not lower the floor without re-deriving the atomic gate's performance argument. +- Rust parity is structural (file-for-file, function-for-function) but NOT byte-fixture-gated like LatticeLib: there is no `rust/tests/fixtures/*.json` here, because `StatSample` values are opaque caller data, not values this library derives from an input. Conformance is judged by matching test suite sections (see rust/tests/intellectus_lib_tests.rs vs. Tests/IntellectusLibTests/IntellectusLibTests.swift §1–§8), not by fixture replay. diff --git a/packages/libs/IntellectusLib/docs/DETAILS.md b/packages/libs/IntellectusLib/docs/DETAILS.md new file mode 100644 index 0000000..c99cc26 --- /dev/null +++ b/packages/libs/IntellectusLib/docs/DETAILS.md @@ -0,0 +1,255 @@ +--- +doc: DETAILS +package: IntellectusLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/IntellectusLib/Intellectus.swift + blob: 47d1dbe60d0ed880eaf657986ef9371f66423fda + - path: Sources/IntellectusLib/RecentWindowSink.swift + blob: 275f059bb1c0b5ebea68c7d32d2e6bb5ad5ed034 + - path: Sources/IntellectusLib/StatSample.swift + blob: 5a63d181680c35a1b79279f83651f8cb62dcd9b7 + - path: Sources/IntellectusLib/StatsSink.swift + blob: 60dc64a00370e491273c2328409a9d56fc46463b +--- + +# IntellectusLib Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in the order data +flows through the library: the sample type, the receiver contract, and the +one bundled receiver. The global facade comes last. It ties everything +together for callers. + +## StatSample.swift + +This file provides `StatSample`, the telemetry datum that flows through the +whole library. It also provides `EventKind`, a small enum used by one of +`StatSample`'s two cases. + +`StatSample` is a value type. It is an enum with associated values, not a +class. It conforms to `Sendable`. That means the Swift compiler checks that +it is safe to pass between threads or concurrent tasks without extra +synchronization. Value types copy instead of sharing memory. That copying is +what makes the safety check possible, since no two threads can then race on +the same storage. + +The enum has two cases. `.metric` carries a dot-separated name, such as +`"locus.capture.latency_ms"`. It also carries a floating-point value, an +optional dictionary of string tags for context, and a timestamp. An example +tag dictionary looks like `["kit": "LocusKit"]`. `.event` carries a +lifecycle event instead. It carries an `EventKind`, either `.capture` or +`.think`. It also carries the integer form of a substrate row's noun type, +the row's identifier string, the estate identifier string, and a timestamp. +`EventKind` distinguishes two broad classes of activity. `.capture` marks a +caller-driven write. `.think` marks a substrate-driven autonomous +transition. Both classes can be triggered by the substrate's Brain layer. + +Every timestamp in the library is a caller-supplied number of seconds since +the epoch. No timestamp is ever read from the system clock inside this +package. This matters for determinism. A caller that replays the same +sequence of events gets the same timestamps back. Tests can construct +samples with fixed, predictable times instead of racing against the real +clock. + +`ts` is a computed property added in an extension. It returns the timestamp +regardless of which case the sample is. This spares every caller that only +cares about ordering samples in time. Such a caller need not write a +`switch` over both cases. + +## StatsSink.swift + +This file provides `StatsSink`, the protocol every receiver of telemetry +samples must conform to. It also provides `NoOpSink`, the receiver installed +before any host supplies its own. + +A protocol in Swift is a contract. It lists the methods and properties a +conforming type must supply, without dictating how. `StatsSink` requires +exactly one method, `receive(_:)`. It also requires conformers to be +`Sendable`, because the one sink installed at any moment lives in global +state. That sink is called from whatever thread happens to be reporting a +sample at the time. + +`receive(_:)` delivers one `StatSample` to the sink. The file documents an +expectation rather than an enforced rule. Implementations should return +quickly. They should not block. Anything slow, such as writing to a database +or sending data over a network, belongs in the sink's own background work. +That background work should be scheduled from inside `receive(_:)`, not run +synchronously inside it. A slow sink would otherwise add latency to whatever +substrate code path triggered the sample. + +`NoOpSink` is the simplest possible conformer. Its `receive(_:)` does +nothing and returns immediately. It exists so the global holder in +`Intellectus.swift` always has a valid sink installed. This holds even +before a host calls `install(sink:)`. Because monitoring is off by default, +`NoOpSink.receive(_:)` is not expected to run in ordinary use. The enabled +gate in `Intellectus.swift` stops a sample before it ever reaches a sink. +Still, the no-op is the correct, safe answer if a host enables monitoring +before installing a real sink. + +## RecentWindowSink.swift + +This file provides `RecentWindowSink`, the one concrete `StatsSink` +implementation the library ships. It is a fixed-size, in-memory buffer of +the most recently received samples. + +The buffer is what programmers call a ring buffer, or FIFO ring. FIFO means +first in, first out. It is a fixed-length array. New entries overwrite the +oldest slot once the array is full. So the memory used never grows past the +buffer's capacity, no matter how many samples arrive over the buffer's +lifetime. This bound is a deliberate contract, documented in the file as +SPEC I-8. A component that helps observe the system must not itself become a +source of unbounded memory growth. + +`RecentWindowSink` optionally wraps a second sink, called the forward sink. +Every sample the window records is, after recording, also handed to the +forward sink if one was supplied at construction. This is a decorator +arrangement, one sink wrapping another. It lets a host keep a live, readable +window of recent activity. At the same time, the host can persist every +sample to durable storage. Both effects come from a single sink installed +with `Intellectus.install(sink:)`. + +`init(capacity:forward:)` builds the sink. The requested capacity is clamped +to a minimum of one. A zero or negative capacity would make the window +useless. So the initializer silently corrects it rather than crashing or +throwing. It treats the input as a caller mistake, not a fatal condition. + +`receive(_:)` is the `StatsSink` method. It takes an internal lock. It writes +the sample into the ring buffer's next slot. It advances the write position. +It counts the sample toward a running total. Then it releases the lock +before calling the forward sink. Running the forward sink outside the lock +matters for the same reason `Intellectus.report(_:)` calls its sink outside +its own lock. A forward sink might itself need to take a lock, its own or +even this same one if reinstalled recursively. That sink cannot deadlock +against this method, because this method holds nothing by the time the +forward sink runs. + +`snapshot()` returns a copy of the buffer's current contents, oldest sample +first. It is a copy taken while holding the lock only long enough to read +the buffer. So a caller can freely inspect or iterate the returned array. No +risk exists of it changing underneath them. No call blocks further calls to +`receive(_:)` for any longer than the copy itself takes. + +`count` reports how many samples the buffer currently holds. This is at most +its capacity. `totalReceived` reports how many samples have ever been handed +to `receive(_:)`, including ones already evicted from the buffer. The +distinction matters for a host reading the window. A `count` of zero could +mean monitoring just started. Or it could mean monitoring has been off the +whole time. `totalReceived` tells the two apart. A nonzero total with an +empty buffer would be a contradiction that never actually occurs. A zero +total is the library's explicit signal that nothing has crossed the gate +yet. + +## Intellectus.swift + +This file provides the library's single entry point, the `Intellectus` +public enum. It also provides its private backing implementation, +`_IntellectusHolder`. + +`Intellectus` is what the file calls a stateless namespace. It is an enum +with no cases, used purely to group related static functions under one +name, because Swift has no notion of free-standing module-level functions. +Every piece of mutable state lives instead in a single global instance of +`_IntellectusHolder`, a class not exposed outside the package. That state +includes the installed sink and the on/off flag. This split keeps the +public surface simple, at four static members. It also keeps the actual +synchronization logic in one well-tested place. + +The on/off flag is stored as a `Synchronization.Atomic`. This type is +supplied by the platform. It lets a single boolean be read and written +directly by the processor, without an operating-system lock. The file +explains the reasoning in a comment. An uncontended `NSLock` still costs +roughly six nanoseconds to acquire and release on Apple Silicon. A single +atomic read costs roughly one nanosecond. It can sometimes be optimized away +entirely inside a tight loop. Since the flag is checked on every single call +to `report(_:)`, the library's hottest path by far, this difference compounds +across a busy substrate. + +Reading and writing the flag use matched atomic orderings. Reads use +`.acquiring`. Writes use `.releasing`. These orderings are a promise to the +compiler and processor. Once a read observes the flag as `true`, it is also +guaranteed to observe any other data a writer set up before flipping the +flag to `true`. Without that promise, a thread could theoretically see the +flag turn on before it saw the sink installed alongside it. This could +happen even though program order says the sink was installed first. + +The installed sink, by contrast, is protected by an ordinary `NSLock`. An +"any `StatsSink`" value is a type-erased container that lets any conforming +type be stored. It cannot be swapped as a single atomic operation the way a +boolean can. The lock is held only long enough to copy the sink reference +out. The sink's `receive(_:)` method always runs after the lock has already +been released. This matches the same outside-the-lock discipline +`RecentWindowSink.receive(_:)` uses for its own forward sink. + +`_IntellectusHolder.install(sink:)` replaces the currently installed sink. +It is expected to run rarely, typically once, at process startup. So it can +afford the ordinary lock. A report already in flight on another thread +finishes with whichever sink it captured. The very next report sees the new +one. + +`_IntellectusHolder.setEnabled(_:)` flips the atomic flag. `isEnabled` reads +it back. Both are simple one-line wrappers. Their simplicity is the point, +though: this is the only place the ordering guarantees above need to be +stated and reasoned about. + +`_IntellectusHolder.report(_:)` is the function every other function in this +file exists to support quickly. It takes its sample-building argument as an +`@autoclosure`. This Swift feature wraps an expression in a closure +automatically at the call site. So the expression is not evaluated as soon +as it is written, only when and if the closure is actually called. +`report(_:)` reads the atomic flag first. If it is `false`, the function +returns immediately. The argument expression is never evaluated at all. This +is exactly the off-path cost the whole file is designed around: one atomic +load and a branch, nothing more. Only when the flag is `true` does the +function take the sink lock. It copies the sink reference, releases the +lock, then finally evaluates the sample-building expression and hands the +result to the sink. + +`Intellectus.install(sink:)`, `Intellectus.setEnabled(_:)`, and +`Intellectus.isEnabled` are thin public wrappers. Each forwards to the +matching `_IntellectusHolder` member, on the single global instance, +`_intellectus`. `Intellectus.report(_:)` does the same for the reporting +path. It repeats the `@autoclosure` on its own parameter. This lets the +short-circuit behavior hold all the way from a substrate call site down to +the atomic check. No intermediate function ever forces early evaluation of +the sample expression. + +## Rust Port and Conformance + +The `rust/` directory contains the second leg of the library. It is +structured to mirror the Swift files one for one. `sample.rs` matches +`StatSample.swift`. `sink.rs` matches `StatsSink.swift`. `window.rs` matches +`RecentWindowSink.swift`. `holder.rs`, together with `global.rs`, matches +`Intellectus.swift`. `lib.rs` assembles the crate's public surface. It adds +one addition the Swift side does not need: a `report!` macro. This macro +gives Rust callers the same short-circuit evaluation that Swift gets for +free from `@autoclosure`. The macro expands to an `if` check around the +sample expression. So the expression compiles into a branch that only runs +when monitoring is enabled. + +The two legs use different low-level tools for the same guarantees. Swift +uses `Synchronization.Atomic` with `.acquiring` and `.releasing` +orderings. Rust uses `std::sync::atomic::AtomicBool` with `Ordering::Acquire` +and `Ordering::Release`, the same memory-ordering concept under the standard +library's own name. Swift protects the installed sink with `NSLock`. Rust +protects it with `std::sync::Mutex`, wrapping the sink in an `Arc`. That is +Rust's shared, reference-counted pointer. It lets the sink be cloned out +from under the lock, the same way Swift copies the "any `StatsSink`" +existential out from under its own lock. `RecentWindowSink`'s ring buffer is +a fixed-size Swift array on the Swift side, with a manually tracked head +position. On the Rust side, it is a `VecDeque`, a double-ended queue +supplied by Rust's standard library. Both sides enforce the same bound and +the same oldest-first eviction rule. + +The package's `Tests/IntellectusLibTests/IntellectusLibTests.swift` suite +and the Rust suite in `rust/tests/intellectus_lib_tests.rs` are organized +around matching sections. Those sections cover gating behavior, the no-op +sink, thread safety, the sample type, `EventKind`, a performance smoke +check, and the recent-window sink. So a change in one leg has a direct +counterpart to check in the other. This differs from LatticeLib in one way, +though: this package has no shared byte-for-byte conformance fixtures. +Nothing here produces a value that two platforms need to agree on, since a +`StatSample` is opaque data the caller constructs. It is not a value this +library computes from an input. diff --git a/packages/libs/IntellectusLib/docs/OVERVIEW.md b/packages/libs/IntellectusLib/docs/OVERVIEW.md new file mode 100644 index 0000000..e18a078 --- /dev/null +++ b/packages/libs/IntellectusLib/docs/OVERVIEW.md @@ -0,0 +1,137 @@ +--- +doc: OVERVIEW +package: IntellectusLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/IntellectusLib/Intellectus.swift + blob: 47d1dbe60d0ed880eaf657986ef9371f66423fda + - path: Sources/IntellectusLib/RecentWindowSink.swift + blob: 275f059bb1c0b5ebea68c7d32d2e6bb5ad5ed034 + - path: Sources/IntellectusLib/StatSample.swift + blob: 5a63d181680c35a1b79279f83651f8cb62dcd9b7 + - path: Sources/IntellectusLib/StatsSink.swift + blob: 60dc64a00370e491273c2328409a9d56fc46463b +--- + +# IntellectusLib Overview + +## What This Library Does + +IntellectusLib lets any part of the MOOTx01 substrate report a small fact +about itself. The fact can be a number, such as a latency in milliseconds. +Or the fact can be a lifecycle event, such as a capture. The library calls +this small fact a telemetry sample. + +A telemetry sample takes one of two forms. The first form is a named +measurement. An example is "capture latency was 42.5 milliseconds." The +second form is a record of a lifecycle transition. An example is "row X was +captured in estate Y." + +The library does not decide what to do with a sample. It hands each sample +to a receiver called a sink. The host program installs that sink. A host +that wants to persist samples to disk installs a sink that writes to a +database. A host that only wants a live look at recent activity installs a +different kind of sink. That sink keeps the last few hundred samples in +memory. + +IntellectusLib supplies one such sink, `RecentWindowSink`. It also supplies +a protocol, `StatsSink`, so any host can build its own sink. + +## The Problem It Solves + +Substrate code runs on a hot path. The same functions execute many times per +second. This happens while a user captures memories or the system dreams +through them. Measuring that code is valuable. An always-on measurement +system is not free, though. Building a telemetry payload costs time and +memory. That cost applies even if nothing reads the result afterward. + +Most of the time, no one is watching. A production device runs with +monitoring off by default. IntellectusLib is built around one guarantee. +When monitoring is off, reporting a sample costs almost nothing. It costs a +single memory read and one comparison, and nothing more. The library never +builds a payload when monitoring is off. It never takes a lock in that case +either. It never calls a sink unless a host has explicitly turned monitoring +on. This guarantee lets substrate code call `Intellectus.report(...)` freely +and everywhere. A caller never has to worry about the cost of that call. + +The library also must work correctly under concurrency. Many threads can +report samples at the same time, since the substrate itself runs +concurrently. Turning monitoring on or off must never corrupt a sample in +flight. Swapping in a new sink must never corrupt one either. + +IntellectusLib keeps this promise through two independent implementations. A +Swift leg serves Apple platforms. A Rust leg, found in `rust/`, mirrors the +Swift leg section for section. It mirrors the Swift leg function for +function too. Hosts that need a Rust build of the substrate use this leg. +Both legs share one design. Each uses a lock-free flag for the on/off gate. +Each holds a lock only briefly, and only around the installed sink. + +## How It Works + +Reporting a sample happens in one call, `Intellectus.report(...)`. That call +takes an argument built lazily. Swift calls this feature `@autoclosure`. The +expression that builds the sample does not run right away. It runs only when +`report` decides it is needed. + +`report` checks a single on/off flag first. If monitoring is off, the +function returns right away. The sample-building expression never runs at +all. Nothing is allocated. No sink is touched. + +If monitoring is on, `report` reads the currently installed sink. It briefly +holds a lock while it does this. The lock stops it from reading a sink in +the middle of a swap. `report` then releases the lock. It builds the sample +next. Then it calls the sink's `receive(_:)` method. The sink runs outside +the lock. So a slow or reentrant sink cannot block a concurrent call. That +concurrent call might only want to install a new sink. + +A host controls the gate with `Intellectus.setEnabled(_:)`. A host installs +a receiver with `Intellectus.install(sink:)`. Both calls are meant to run +rarely. A host typically calls them once, at startup. An operator may also +call them later, when toggling monitoring on or off. + +The library ships one ready-to-use sink, `RecentWindowSink`. This sink keeps +a fixed-size buffer of the most recent samples in memory. The buffer evicts +its oldest entry first once it fills. A host can read that buffer at any +time to see recent activity. This is useful for a live dashboard or a +diagnostic command. `RecentWindowSink` can also wrap a second sink. It +forwards every sample to that second sink after recording it. This lets a +host keep a live window and also write to durable storage. Both actions come +from a single installed sink. + +## How the Pieces Fit + +Figure 1 shows the library's topology. It shows the major parts and how a +sample moves through them. + +![Figure 1. Topology of IntellectusLib](topology.svg) + +*Figure 1. Topology of IntellectusLib. A caller's report flows through the +enabled gate. When the gate is open, the sample reaches the installed sink. +That sink may be the bundled `RecentWindowSink`. It may in turn forward the +sample to a second, host-supplied sink. The dashed region marks the parts a +host supplies, apart from the library itself.* + +`StatSample.swift` defines the data that flows through the system. It +defines the `StatSample` enum and its two cases, `.metric` and `.event`. +`StatsSink.swift` defines the receiver contract, `StatsSink`. It also +defines the default do-nothing receiver, `NoOpSink`. A host relies on +`NoOpSink` before it ever calls `install(sink:)`. `RecentWindowSink.swift` +provides the one concrete sink the library ships. `Intellectus.swift` ties +the pieces together. It holds the current sink and the on/off flag. It +exposes the `Intellectus` enum as the single public entry point. The rest of +the substrate calls through that one entry point. + +## What Ships in the Package + +The package ships four Swift source files. It also ships a matching Rust +port in `rust/`. The package has no dependency on any other library in the +repository. It does not even depend on the lowest substrate libraries. Those +libraries can therefore depend on IntellectusLib in turn. This avoids +creating a dependency cycle. + +The package depends only on Foundation and on Swift's `Synchronization` +module. The platform supplies both. The platform floor is macOS 26 and iOS +26. That floor was chosen for one reason. It is the first OS release that +supports `Synchronization.Atomic` without a compatibility fallback. diff --git a/packages/libs/IntellectusLib/docs/topology.svg b/packages/libs/IntellectusLib/docs/topology.svg new file mode 100644 index 0000000..f5ca99e --- /dev/null +++ b/packages/libs/IntellectusLib/docs/topology.svg @@ -0,0 +1,91 @@ + + + + + + + + + + + + + IntellectusLib: sample in, gated delivery out + + + + Caller + Intellectus.report(...) + + + + Enabled gate + Atomic<Bool> load + off = autoclosure never runs + + + + disabled (default) + return: no sink call + + + + _IntellectusHolder + NSLock snapshot of + installed sink, then unlock + + + + host-installed StatsSink (protocol boundary) + + + NoOpSink + default, discards + + + Custom sink + host-supplied + + + RecentWindowSink + bundled: fixed-capacity ring buffer, + oldest evicted; snapshot() / count / totalReceived + + + forward sink + optional, host durable store + + + + + + + + + + sample.receive(_:): outside the lock + diff --git a/packages/libs/SubstrateKernel/docs/AGENT_MAP.md b/packages/libs/SubstrateKernel/docs/AGENT_MAP.md new file mode 100644 index 0000000..02d0c3b --- /dev/null +++ b/packages/libs/SubstrateKernel/docs/AGENT_MAP.md @@ -0,0 +1,145 @@ +--- +doc: AGENT_MAP +package: SubstrateKernel +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateKernel/BitField.swift + blob: 755ce095e0db4d4d4f5f01cc3568cb8403e5952f + - path: Sources/SubstrateKernel/FloatVecOps.swift + blob: b1c4593a99fe078b19ad0058a2e76e9949abd524 + - path: Sources/SubstrateKernel/HammingNN.swift + blob: c810cb5ada7b0c1cdc7789791552478b19bdad07 + - path: Sources/SubstrateKernel/HKDF.swift + blob: 53cd1dbef1ea0b9a8c932cdcfc303b50eb49adba + - path: Sources/SubstrateKernel/PortableKernel-Metal.swift + blob: 4c9837448547cb44fde52a08b8e639876d23f150 + - path: Sources/SubstrateKernel/PortableKernel-NEON.swift + blob: 996063a101bcd76d2ea965b58072eac2141adf98 + - path: Sources/SubstrateKernel/PortableKernel-SIMD.swift + blob: 88caebd2a66c6541f9dff3aff7555dabc557ee5f + - path: Sources/SubstrateKernel/PortableKernel.swift + blob: 5279dadeb909d4d8980f2fc3a3100caf7d9066d3 + - path: Sources/SubstrateKernel/SHA256.swift + blob: 143a3530a8dee15446fa5711022638e0d61219b2 +--- + +# AGENT_MAP: SubstrateKernel + +PURPOSE: layer 2 of the four-package substrate split (SubstrateTypes → SubstrateKernel → SubstrateML → SubstrateLib). Bandwidth-bound bit ops over `Fingerprint256`: Hamming distance/top-K, OR-reduce, SimHash, count-fold, popcount; plus BitField (packed-bitmap field extract/write), SHA-256 content hashing, HKDF-SHA256 key derivation, and canonical scalar float-vector ops. No pinned data artifacts: behavior is pure algorithm, not versioned reference data. + +DEPS: imports SubstrateTypes (Fingerprint256, HyperplaneFamily, CountVector256, FloatSimHashPlanes, SimHash, Hamming, BlockMask), IntellectusLib (Intellectus.report telemetry: zero-dep leaf), Foundation, `simd` (Apple + aarch64 Linux), Metal (Apple only, `#if canImport(Metal)`). Imported by: SubstrateML (depends on SubstrateTypes + SubstrateKernel per dependency graph); hot-path consumers LocusKit, CorpusKit, GeniusLocusKit, EngramLib; SubstrateLib's AuditGate consumes only BitField + SHA256 (not the kernel dispatch layer). Rust port in rust/ mirrors kernel.rs/bit_field.rs/hamming_nn.rs/sha256.rs/hkdf.rs/float_vec_ops.rs function-for-function; kernel_simd.rs (nightly-gated `simd-nightly` feature) covers NEON's ground so there is no neon.rs; kernel_avx512.rs (x86_64-only, DARK path, P11-VAL-015) has no Swift counterpart; Metal has no Rust counterpart (explicit platform waiver, no Linux/Windows GPU path). + +ENTRY POINTS (most callers need only these): +- PortableKernel.swift:313 `PortableKernel.kernelForCurrentPlatform() -> SubstrateKernel`: auto-selects SimdKernel (arm64) or ScalarKernel (else); never selects Metal +- PortableKernel.swift:368 `PortableKernel.kernel(of: KernelKind) -> SubstrateKernel`: explicit backend selection (tests, conformance harness, opt-in GPU/NEON) +- BitField.swift:62/86 `BitField.extractField/writeField(...)`: packed-bitmap field access, the F18 centralization surface +- SHA256.swift:30 `SHA256.hash(_ bytes:) -> [UInt8]`: FIPS 180-4 content hash +- HKDF.swift:54 `GrantHKDF.deriveKey(...)`: RFC 5869 HKDF-SHA256 grant scope-key derivation + +## Symbol Table + +### Protocol + reference kernel: PortableKernel.swift +- :49 `protocol SubstrateKernel: Sendable`: every backend conforms; all methods must be bit-identical across conformers +- :54 `kind: KernelKind`: runtime introspection; default `.scalar` (extension :137), every concrete kernel overrides +- :57 `popcount64(_:) -> Int`: 64-bit population count +- :60 `hammingDistance256(_:_:) -> Int`: bit-differences between two Fingerprint256 +- :64 `orReduce256(_:) -> Fingerprint256`: bitwise OR fold; identity = zero +- :69 `hammingTopK(probe:candidates:k:) -> [(index:Int,distance:Int)]`: K nearest by Hamming distance, ties → lower index +- :76 `simhashCompute(subhashes:families:) -> Fingerprint256`: 4-subhash → Fingerprint256 projection +- :91/:97/:102 `hammingDistanceBatch` / `simhashBlockBatch` / `orReduceBatch`: batched variants; protocol-extension defaults are naive loops (:139/:144/:157), always correct, backends may override for speed +- :112/:116 `countFold256` / `countFoldBatch`: per-bit-position set counts across a cohort; default (:166) = `CountVector256.fold`; OR-reduce is this fold's saturating-at-one degenerate case +- :127 `floatSimHashProject(vector:planes:) -> Fingerprint256`: float analogue of simhashCompute; default impl (:183) is the SCALAR ORACLE for this op: fixed left-to-right accumulation order (FP `+` non-associative, order IS the contract); precondition on `planes.dim == vector.count` +- :222 `struct ScalarKernel: SubstrateKernel`: THE ORACLE; every other backend's output must match this bit-for-bit for the same input +- :233 `hammingDistance256`: `a.zip4(b, ^).popcount()` (SubstrateTypes combinator) +- :239 `orReduce256`: `Fingerprint256.reduce4(fingerprints, |)` +- :245 `hammingTopK`: bounded max-heap O(N log k); tie-break lower-index-wins is the cross-backend contract +- :287 `enum KernelKind`: scalar|simd|neon|metal|avx512|avx2 (avx512/avx2 always fall to scalar in THIS Swift build: no Swift AVX implementation exists) +- :296 `enum PortableKernel`: dispatcher namespace +- :313 `kernelForCurrentPlatform()`: arm64→SimdKernel else ScalarKernel; COMPILE-TIME choice, no runtime CPU-feature probing (contrast Rust's Avx512 CPUID guard, which DOES probe at runtime because that backend is unsafe without it); emits `substrate.kernel.backend_selected` telemetry, autoclosure skipped (no clock read) when monitoring off +- :355 `currentArchTag`: static string ("arm64"/"x86_64"/"other"), must track the `#if arch()` predicate above it +- :368 `kernel(of:)`: explicit selector; `.metal` falls back to ScalarKernel if `MetalKernel()` init fails (no GPU); `.avx512`/`.avx2` always → ScalarKernel +- :406 `assertEqual(lhs:rhs:probe:candidates:k:)`: hammingTopK-based bit-identity check; conformance-suite building block +- :430 `ScalarKernelScored` / :440 `ScalarKernelMaxHeap` (internal): (distance, index) max-heap shared plumbing; root = worst retained + +### SIMD backend: PortableKernel-SIMD.swift +- :39 `struct SimdKernel: SubstrateKernel`: selected by kernelForCurrentPlatform() on arm64; `import simd` → NEON codegen +- :65 `hammingDistance256`: SIMD4 XOR + per-lane nonzeroBitCount, summed +- :114 `hammingTopK`: sorted-ladder maintenance (NOT a heap); identical sorted output + tie-break to ScalarKernel +- :214 `simhashBlockBatch`: uses :233 `PackedFamily` (SoA repack of 64 hyperplanes into 16 groups of 4) built ONCE per batch, amortized across inputs via :275 `simhashBlockSIMD` +- :376 `countFold256`: bit-sliced vertical binary counter across SIMD4 lanes; O(fingerprints × ~2 ops) vs scalar's bit-by-bit walk; conformance-gated against `CountVector256.fold` + +### NEON backend: PortableKernel-NEON.swift +- :38 `struct NeonKernel: SubstrateKernel`: NOT auto-selected; reachable only via `PortableKernel.kernel(of: .neon)`; exists to be BENCHMARKED against SimdKernel, not to replace it +- :63 `toBytes(_:) -> SIMD32` (static): reframes Fingerprint256 at byte level via single 256-bit load (not byte-by-byte copy) +- :87 `popcountBytes(_:)` (static): 32 scalar nonzeroBitCount calls; Swift does NOT auto-vectorize this into one `cnt.16b` (measured gap, see file header) +- :104/:113/:128 `hammingDistance256` / `hammingDistanceBatch` / `hammingTopK`: byte-level XOR + popcount + horizontal sum; hammingTopK uses same ladder pattern as SimdKernel +- :170/:182/:186 `orReduce256` / `orReduceBatch` / `simhashCompute`: NOT re-framed at byte level; identical to SimdKernel's implementation (byte reframing judged not to help these ops) + +### Metal backend: PortableKernel-Metal.swift (Apple-only, `#if canImport(Metal)`) +- :67 `MetalBufferPool` (fileprivate, `@unchecked Sendable`): persistent GPU buffers sized for `maxN`; caller-enforced per-call synchronization, harness is single-threaded +- :110 `struct MetalKernel: SubstrateKernel`: NOT auto-selected; reachable only via `PortableKernel.kernel(of: .metal)` +- :128 `defaultMaxN = 100_000`: buffer-pool sizing default (~3.6 MB), tuned to dreaming-daemon batch-index scale +- :134 `init?(maxN:)`: returns nil on ANY setup failure (no GPU, shader compile fail, pipeline fail); PortableKernel.kernel(of:) treats nil as "fall back to ScalarKernel", not an error +- :172 `popcount64` / :182 `hammingDistance256`: inherited scalar semantics; per-pair GPU dispatch not worth its ~10–30µs overhead +- :203 `hammingDistanceBatch`: :210 dispatches :221 `dispatchWithPool` (N ≤ maxN, reuse buffers) or :283 `dispatchWithFreshBuffers` (N > maxN, allocate per-call) +- :418 `scalarFallback` (private): CPU fallback on ANY Metal failure (buffer alloc, command-buffer creation, GPU execution error): never crashes or returns a wrong/partial result +- :364 `hammingTopK`: GPU batched distance + CPU heap-of-K selection (GPU-side top-K selection judged not worth it for small K) +- :440 `shaderSource` (private static): embedded Metal shader source (`hamming_distance_kernel`); must be kept byte-for-byte in sync with the canonical `.metal` file or the conformance gate's CRC check fails + +### Standalone top-K reference: HammingNN.swift +- :32 `struct HammingNNHit: Hashable, Sendable`: (rowID: UUID, distance: Int) +- :51 `enum HammingNN` +- :53 `topK(anchor:candidates:k:blocks:) -> [HammingNNHit]`: NOT part of the SubstrateKernel/PortableKernel dispatch; operates over any `(UUID, Fingerprint256)` sequence, not a dense array; supports partial-block comparison via `blocks: BlockMask` (e.g. topic-only similarity); requires `k > 0` +- tie-break: rowID.uuidString ascending (DIFFERENT key from PortableKernel backends' index-ascending: do not conflate the two top-K contracts) +- :94/:101/:114 `heapLarger` / `heapifyUp` / `heapifyDown` (private): bounded max-heap, same shape as ScalarKernelMaxHeap but keyed on UUID string not array index + +### Packed-bitmap fields: BitField.swift +- :45 `enum BitField`: F18 atomic-centralization surface; kits must NOT open-code bit-shift/mask logic +- :62 `extractField(_:shift:width:) -> Int64`: precondition: 0≤shift, 1≤width≤64, shift+width≤64 +- :86 `writeField(_:into:shift:width:) -> Int64`: same preconditions; value bits outside `width` are SILENTLY truncated (documented contract, not a bug) +- :135 `maskedEquals(_:mask:expected:) -> Bool`: one-step `(bitmap & mask) == expected`; caller invariant: expected must already be masked/aligned +- :153/:164 `extractFlag` / `writeFlag`: single-bit case; precondition 0≤bit<64 +- :181 `popcount(_:) -> Int`: via `UInt64(bitPattern:).nonzeroBitCount` +- :188 `hammingDistance(_:_:) -> Int`: `popcount(a ^ b)`, single-Int64 version (contrast Fingerprint256's 256-bit version elsewhere) +- :200 `xorFold(_:) -> Int64`: running XOR reduce; empty → 0 + +### Content hashing: SHA256.swift +- :27 `enum SHA256`: FIPS 180-4, dependency-free +- :30 `hash(_ bytes: [UInt8]) -> [UInt8]`: 32-byte digest; ported VERBATIM from GeniusLocusKit's prior in-kit copy so centralization preserves every existing content ID byte-for-byte +- :119 `rotr` (private): right-rotate helper + +### Key derivation: HKDF.swift +- :37 `enum GrantHKDF`: RFC 5869 HKDF-SHA256, built entirely over this package's own SHA256 (no CryptoKit) for cross-platform byte-identical key derivation +- :54 `deriveKey(inputKeyMaterial:salt:info:outputByteCount:) -> [UInt8]`: public entry; `outputByteCount` ≤ 8160 (32×255) +- :69 `extract(salt:ikm:)`: RFC 5869 §2.2, `hmac(key: salt, data: ikm)` +- :76 `expand(prk:info:length:)`: RFC 5869 §2.3, chained HMAC with counter byte +- :101 `hmac(key:data:) -> [UInt8]`: public (not just internal) because SubstrateLib's KeyedCommitment API reuses this exact construction; RFC 2104, keys > 64 bytes pre-hashed + +### Scalar float-vector oracle: FloatVecOps.swift +- :62 `enum FloatVecOps`: canonical IEEE-754 scalar reference; every faster backend (Accelerate/BLAS/etc.) MUST match bit-for-bit +- :72 `l2Norm(_:) -> Float`: sqrt(sum(x*x)); empty → 0.0 +- :99 `l2Normalize(_:) -> [Float]`: zero-vector PASSTHROUGH (not NaN, not a panic): the "no information" signal that projects to Engram.zero via FloatSimHash +- :121 `dot(_:_:) -> Float`: precondition a.count == b.count +- :147 `cosine(_:_:) -> Float`: DEFINED ONLY for pre-normalized unit vectors (caller must call l2Normalize first); debug-only assert checks norm ≈ 1.0 within 1e-5, compiled away in release + +## INVARIANTS / GOTCHAS + +- CONFORMANCE IS THE CONTRACT. Every `SubstrateKernel` backend (Scalar/Simd/Neon/Metal, both legs) must produce bit-identical output to `ScalarKernel` for the same input. A divergence is a bug in the backend, never grounds to change the scalar oracle. +- `FloatVecOps` is a second, independent oracle for decimal-vector math (l2Norm/l2Normalize/dot/cosine); same bit-identical-to-scalar rule applies to any faster override elsewhere in the substrate. +- TWO DIFFERENT top-K contracts coexist and must not be conflated: `PortableKernel` backends' `hammingTopK` (dense array input, ties → lower ARRAY INDEX) vs. `HammingNN.topK` (candidate-sequence input, ties → lower `rowID.uuidString`, supports partial-block comparison via `BlockMask`). +- `kernelForCurrentPlatform()` is a COMPILE-TIME choice (`#if arch(arm64)`), not a runtime CPU-feature probe. It never selects `.metal`; Metal requires explicit `PortableKernel.kernel(of: .metal)`. `.avx512`/`.avx2` have no Swift implementation and always resolve to `ScalarKernel` in this build. +- Rust's AVX-512 backend (`kernel_avx512.rs`) is the opposite of the Swift dispatcher: it DOES check `is_x86_feature_detected!("avx512f")` && `("avx512vpopcntdq")` at runtime before every unsafe intrinsic call, because skipping that check is undefined behavior (SIGILL) on CPUs lacking those features. This guard was present from the initial commit and was hardened/test-covered in the 2026-06-28 security review (`tests/avx512_hamming_conformance.rs::cpuid_guard_prevents_intrinsics_without_feature`). `for_current_platform()` never auto-selects it regardless (dark path, P11-VAL-015 gates enable). +- `MetalKernel.init?` returning `nil` is a NORMAL, expected outcome (no GPU / headless CI / virtualization), not an error to surface: callers fall back to scalar. +- `MetalKernel.hammingDistanceBatch` falls back to `scalarFallback` on ANY Metal failure (buffer alloc, command-buffer/encoder creation, GPU execution error): never crashes, never returns a partial/wrong result. +- `MetalKernel.defaultMaxN` = 100,000: batches at or under this reuse the persistent `MetalBufferPool`; batches over it pay full per-call buffer allocation via `dispatchWithFreshBuffers`. +- `MetalKernel.shaderSource` is a hand-maintained string copy of the canonical `.metal` shader file; the conformance gate's CRC check is the only thing that catches drift between them: keep them in sync manually. +- `BitField.writeField`'s value truncation (bits of `value` outside `width` are silently dropped) is an intentional contract, not a bug: matches packed-row semantics where the field's width defines its value space. +- `SHA256.hash` was ported VERBATIM from GeniusLocusKit specifically to preserve existing audit-log content IDs byte-for-byte; do not "clean up" the implementation without checking downstream ID stability. +- `HKDF`/`GrantHKDF` uses ONLY the in-repo `SHA256`, never CryptoKit: required for the Swift↔Rust byte-identical key-derivation guarantee (PAR-4-GL1). +- `GrantHKDF.hmac` is public (not internal) because `SubstrateLib`'s KeyedCommitment API depends on reusing this exact HMAC-SHA256 construction; do not narrow its access without checking that caller. +- No pinned/versioned data artifacts ship in this package (contrast e.g. LatticeLib's Resources/ lexicon+frame+signatures): everything here is pure algorithm; conformance is enforced by test fixtures, not artifact versioning. +- `AuditGate` (in `SubstrateLib`, NOT in this package) consumes only `BitField` + `SHA256` from here: it does not touch the kernel-dispatch layer at all. +- No NEON or Metal Rust files exist by design: `kernel_simd.rs` (nightly `simd-nightly` feature) covers NEON's aarch64 ground on the Rust side; Metal has no cross-platform equivalent and is an explicit, documented platform waiver. +- Telemetry (`Intellectus.report` / `substrate.kernel.backend_selected`) is off by default; when off, the reporting autoclosure is never evaluated and no clock (`Date()`) is read: selection has zero overhead in the default configuration. diff --git a/packages/libs/SubstrateKernel/docs/DETAILS.md b/packages/libs/SubstrateKernel/docs/DETAILS.md new file mode 100644 index 0000000..185fb69 --- /dev/null +++ b/packages/libs/SubstrateKernel/docs/DETAILS.md @@ -0,0 +1,353 @@ +--- +doc: DETAILS +package: SubstrateKernel +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateKernel/BitField.swift + blob: 755ce095e0db4d4d4f5f01cc3568cb8403e5952f + - path: Sources/SubstrateKernel/FloatVecOps.swift + blob: b1c4593a99fe078b19ad0058a2e76e9949abd524 + - path: Sources/SubstrateKernel/HammingNN.swift + blob: c810cb5ada7b0c1cdc7789791552478b19bdad07 + - path: Sources/SubstrateKernel/HKDF.swift + blob: 53cd1dbef1ea0b9a8c932cdcfc303b50eb49adba + - path: Sources/SubstrateKernel/PortableKernel-Metal.swift + blob: 4c9837448547cb44fde52a08b8e639876d23f150 + - path: Sources/SubstrateKernel/PortableKernel-NEON.swift + blob: 996063a101bcd76d2ea965b58072eac2141adf98 + - path: Sources/SubstrateKernel/PortableKernel-SIMD.swift + blob: 88caebd2a66c6541f9dff3aff7555dabc557ee5f + - path: Sources/SubstrateKernel/PortableKernel.swift + blob: 5279dadeb909d4d8980f2fc3a3100caf7d9066d3 + - path: Sources/SubstrateKernel/SHA256.swift + blob: 143a3530a8dee15446fa5711022638e0d61219b2 +--- + +# SubstrateKernel Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in one +order. First comes the shared protocol and its reference +implementation. Next come the three specialized hardware backends. +Then comes the standalone top-K primitive. Last come three independent +utilities: bit fields, hashing, and float-vector math. + +## PortableKernel.swift + +This file provides the `SubstrateKernel` protocol. It also provides the +`ScalarKernel` reference implementation. It provides the +`PortableKernel` dispatcher, which hands callers a concrete kernel. + +The protocol declares every operation a kernel must supply. `popcount64` +counts the one-bits in a 64-bit word. `hammingDistance256` counts +differing bits between two fingerprints. `orReduce256` combines a group +of fingerprints with a bitwise OR. `hammingTopK` finds the K candidates +nearest a probe fingerprint. `simhashCompute` folds a set of 64-bit +numbers into a fingerprint. `countFold256` tallies, per bit position, +how many fingerprints in a group have that bit set. `floatSimHashProject` +does the same folding, but starts from a vector of decimal numbers +instead of integers. A protocol extension supplies default batched +variants of each operation. These defaults are simple loops over the +single-item version. They are correct, but not optimized. A backend +with a faster batched strategy overrides the default. Every backend is +correct even before it does. + +`ScalarKernel` implements every required method as a plain loop. It +uses no special hardware instructions. It is deliberately the simplest +possible correct implementation, because it is the oracle. A backend +counts as correct only when its output matches `ScalarKernel`'s output, +for the same input. Two of its methods matter beyond their obvious +job. `hammingTopK` keeps a bounded max-heap of the K best candidates +seen so far. It evicts the current worst kept candidate whenever a +better one arrives. This keeps the cost proportional to N times the +log of K, rather than N times the log of N. Ties between equal +distances break toward the candidate with the lower index. Every other +backend must match that exact tie-break. +`floatSimHashProject`'s default body lives in the protocol extension. +Every backend that does not override it inherits this body, and that +body is itself the reference for the operation. For each of two +hundred fifty-six hyperplanes, it accumulates a signed sum over the +input vector in a fixed left-to-right order. Floating-point addition +is not associative, so the order is part of the contract. It is not +just an implementation detail. + +`PortableKernel.kernelForCurrentPlatform()` is the entry point most +callers use. On 64-bit ARM, known as `arm64`, it returns `SimdKernel`. +This covers both Apple Silicon and ARM64 Linux. Everywhere else it +returns `ScalarKernel`. The choice is fixed at compile time for a +given platform. There is no runtime hardware probing here. This +contrasts with the Rust port's AVX-512 backend, which checks the +processor's feature flags at runtime, because that backend is unsafe +to run without them. Every call reports which backend it chose, +through IntellectusLib's telemetry system, but only when telemetry is +enabled. When telemetry is off, the reporting closure never runs, and +no clock is read. So the selection has no cost in normal operation. +`PortableKernel.kernel(of:)` is the explicit-selection counterpart. +Tests use it. So does any caller that wants a specific backend +regardless of platform. Requesting `.metal` falls back to +`ScalarKernel` if no GPU is available. Requesting `.avx512` or `.avx2` +always falls back to `ScalarKernel` in this Swift build, since neither +has a Swift implementation. `PortableKernel.assertEqual` runs +`hammingTopK` on two kernels with the same inputs. It reports whether +their results match. This is the building block the conformance test +suite uses to check every backend against the oracle. + +## PortableKernel-SIMD.swift + +This file provides `SimdKernel`. It is the backend +`kernelForCurrentPlatform` selects on 64-bit ARM. + +`SimdKernel` uses Swift's `simd` module. This module represents four +64-bit numbers as one `SIMD4` value. It lets ordinary operators +like XOR and OR act on all four at once. The Swift compiler turns +those operators into single ARM NEON instructions, rather than four +separate ones. `hammingDistance256` XORs two fingerprints as +`SIMD4` values. It then counts the one-bits in each of the four +resulting 64-bit lanes, and adds the four counts together. +`hammingTopK` uses a different strategy from `ScalarKernel`'s heap. It +keeps a small sorted array of the K best candidates found so far, +called a ladder, and shifts it to make room for a new entry. The +file's comments note that this is faster than a heap for the small K +values this package expects. It produces the identical sorted output +and tie-break. + +`simhashBlockBatch` computes one 64-bit SimHash block for many input +vectors, against the same hyperplane family. A hyperplane family is a +fixed set of 64 reference directions. Each direction decides, for one +bit of the output, which side of that direction the input vector falls +on. Rather than compute that decision one hyperplane at a time for +every input, `PackedFamily` rearranges the family's 64 hyperplanes once, +into groups of four. This lets the per-input inner loop process four +hyperplanes together, with the same `SIMD4` trick used for +Hamming distance. Building this packed layout costs a little extra +time up front. That cost is worthwhile, because it is paid once, and +the savings apply to every input in the batch. + +`countFold256` computes, for a group of fingerprints, how many of them +have each of the 256 bit positions set. It does not count each +fingerprint's contribution one position at a time. Instead, `SimdKernel` +keeps a growing binary counter for each bit position. It holds this +counter across `SIMD4` lanes, and folds each new fingerprint in +with two bitwise operations. The counter's next bit is the counter XOR the +fingerprint. The amount to carry into the following bit is the counter +AND the fingerprint. This produces the same result as adding one for +every set bit. It does the addition four fingerprint-blocks at a time, +instead of bit by bit. + +## PortableKernel-NEON.swift + +This file provides `NeonKernel`. It is an alternate ARM implementation, +kept side by side with `SimdKernel`, to test a different way of +expressing the same computation. + +`SimdKernel` counts bits four 64-bit words at a time. `NeonKernel` +instead treats a fingerprint as 32 individual bytes. It counts bits one +byte at a time, then adds all 32 counts together. The file's comments +record the reasoning. Processors often have an instruction that counts +bits across a whole row of bytes in one step. The hope was that +framing the computation at the byte level would let the compiler use +that instruction better than the word-level framing does. Whether it +actually helps is a question the package answers by measurement, not +assumption. A companion benchmark compares the two. `NeonKernel` exists +so that measurement has something concrete to measure. It is never +chosen automatically. Callers reach it only through +`PortableKernel.kernel(of: .neon)`. + +Every method mirrors its `SimdKernel` counterpart at the byte level, +instead of the word level. `hammingDistance256` and +`hammingDistanceBatch` XOR two byte-vectors, and sum per-byte bit +counts. `hammingTopK` uses the same ladder-based top-K selection as +`SimdKernel`. `orReduce256`, `orReduceBatch`, and `simhashCompute` are +not re-framed at the byte level. The file's comments judge that the +byte-level trick that helps Hamming distance does not help those +operations. So they use the same implementation `SimdKernel` uses. + +## PortableKernel-Metal.swift + +This file provides `MetalKernel`. It runs the batched Hamming distance +computation on the graphics processor, through Apple's Metal framework. +It is available only on Apple platforms. + +Dispatching work to the GPU has a fixed cost per call. Setting up +buffers and handing off the computation takes tens of microseconds, +and that cost does not shrink as the batch grows. That fixed cost +makes GPU dispatch a poor choice for comparing one pair of +fingerprints. So `MetalKernel` does not override `hammingDistance256`. +It inherits the scalar version for single pairs. It only specializes +the batched form, `hammingDistanceBatch`, where the GPU's throughput +advantage outweighs its setup cost. `MetalBufferPool` removes a second +fixed cost. Rather than allocate a fresh block of GPU-visible memory on +every call, the kernel allocates its buffers once. It sizes them for +up to `defaultMaxN`, or one hundred thousand candidates, and reuses +them on every call within that size. A batch larger than that falls +back to `dispatchWithFreshBuffers`, which allocates fresh buffers for +that call only. + +`init?(maxN:)` builds the one-time Metal state. It asks the system for +its default GPU. It compiles the compute shader embedded in +`shaderSource`. It builds the buffer pool. Any step can fail: there may +be no GPU, as in a virtualized environment, or the shader may fail to +compile. Either failure makes the initializer return `nil`. That is a +normal outcome, not an error condition. `PortableKernel` falls back to +`ScalarKernel` whenever `MetalKernel`'s initializer returns `nil`. +`hammingDistanceBatch` writes the probe and candidate fingerprints into +GPU-visible memory. It dispatches one GPU thread per candidate. It +waits for the GPU to finish, then reads the resulting distances back. +If any Metal call along that path fails, `scalarFallback` computes the +same batch on the CPU instead. So a transient GPU problem degrades +performance, rather than producing a wrong or missing answer. +`hammingTopK` computes distances through the GPU path above. It then +selects the K smallest with the same CPU-side max-heap `ScalarKernel` +uses. GPU-side selection of a small K, from a large result set, is not +worth its complexity. + +## HammingNN.swift + +This file provides `HammingNN`. It is a standalone top-K search over +fingerprints that does not go through the `PortableKernel` protocol at +all. + +`PortableKernel`'s backends operate on a dense array of candidates, and +select the fastest hardware path. `HammingNN.topK` instead operates +over any sequence of `(rowID, fingerprint)` pairs. It is meant as a +simple, general-purpose reference. A caller can use it directly, +without choosing a backend. It also supports comparing only a subset +of a fingerprint's four blocks, through its `blocks` parameter. This is +useful when a caller wants similarity on just one part of a +fingerprint, such as the topic-related portion, instead of the whole +fingerprint. Internally it +keeps the same kind of bounded max-heap `ScalarKernel` uses. It breaks +ties the same way in spirit, meaning deterministically, though by the +row's UUID string rather than by array index. This function's +candidates do not come with array positions of their own. `topK` +requires `k` to be positive. It returns the results sorted by distance, +then by UUID string, so that two runs over the same data always +produce the same order. + +## BitField.swift + +This file provides `BitField`. It is a set of pure functions for +reading and writing fixed-width fields packed inside a 64-bit integer. + +MOOTx01 packs several small values, such as flags, small counters, and +short codes, into single 64-bit words, called bitmaps. Doing so keeps +those words compact and fast to compare. Every package that reads or +writes one of those packed fields is expected to call through this +file. It should not write its own bit-shifting logic. That way, a +change to a field's layout only has to be made once. +`extractField(_:shift:width:)` reads a field of `width` bits, starting +at bit position `shift`. `writeField(_:into:shift:width:)` writes a new +value into that same position, while leaving every other bit +untouched. `extractFlag` and `writeFlag` are the single-bit special +case of the same idea. `maskedEquals` tests whether a bitmap's masked +bits equal an expected value in one step. This is useful when a caller +only wants a yes-or-no answer, and does not need the field's actual +value. `popcount`, `hammingDistance`, and `xorFold` round out the file +with the same bit-counting and bit-comparison operations the kernel +layer uses. Here they work over plain 64-bit integers, instead of +256-bit fingerprints. This suits callers with a single packed word, +rather than a whole fingerprint. + +## SHA256.swift + +This file provides `SHA256`. It is a complete implementation of the +SHA-256 cryptographic hash algorithm, with no dependency on any system +library. + +MOOTx01's audit log identifies each entry by a hash of its exact +contents. That way, two replicas that receive the same entry compute +the same identifier. Each replica can then recognize the entry as the +same, without comparing the whole thing. This only works if every +device computes the hash the same way. That is why this file +implements the standard itself. It does not rely on a platform library +that could change behavior between operating system versions. The +implementation carries +over unchanged from an existing in-kit copy. Centralizing it here does +not change any identifier a caller had already computed and stored. +`SHA256.hash(_:)` is the entire public surface. It takes a list of +bytes and returns the 32-byte digest, following the standard's padding +and compression steps exactly. + +## HKDF.swift + +This file provides `GrantHKDF`. It is an implementation of +HKDF-SHA256, a standard method for deriving new cryptographic keys +from existing key material. It builds entirely on top of this +package's own `SHA256`. + +MOOTx01's grant system needs to derive a key specific to one sharing +grant. It derives this key from an estate's underlying cryptographic +identity, without exposing that identity directly. This derivation +builds on the in-repo +`SHA256`, rather than on a system cryptography library. That way, a +key derived on one platform is guaranteed to match the same derivation +run on another platform. This guarantee matters when two devices must +independently arrive at the same derived key. +`GrantHKDF.deriveKey(inputKeyMaterial:salt:info:outputByteCount:)` is +the public entry point. It runs the standard's two steps, extract and +expand, in sequence. `extract` and `expand` implement those two steps +directly from the specification. `hmac(key:data:)` implements the +keyed-hash construction both steps depend on. It is itself public, +because `SubstrateLib`'s commitment-signing feature needs the same +HMAC-SHA256 construction over its own data. This avoids a second +implementation existing anywhere in the substrate. + +## FloatVecOps.swift + +This file provides `FloatVecOps`. It holds the canonical scalar +implementations of four vector operations: length, normalization, dot +product, and cosine similarity. All four operate over ordinary lists +of decimal numbers. + +Just as `ScalarKernel` is the oracle for fingerprint operations, +`FloatVecOps` is the oracle for decimal-vector operations. Any faster +implementation elsewhere in the substrate must match these functions +bit for bit. This includes an implementation that uses a platform math +library. The match must be exact, not merely approximate. `l2Norm(_:)` +computes a vector's length. It sums +the square of each element, then takes the square root. +`l2Normalize(_:)` rescales a vector to length one, which is the form +most similarity comparisons expect. A vector of all zeros has no +meaningful direction to rescale to. So the function returns it +unchanged, rather than dividing by zero. That unchanged zero vector +itself carries meaning. Elsewhere in the substrate, it signals "no +information available." `dot(_:_:)` sums the products of corresponding +elements from two equal-length vectors. `cosine(_:_:)` measures the +angle between two vectors. It only gives a meaningful answer when both +inputs are already unit-length. Normalizing on every call would waste +work, when a caller normalizes once and compares many times. So the +function requires the caller to normalize first instead. In debug +builds only, it also checks that the caller did so. + +## Rust Port and Conformance + +The `rust/` directory holds the second leg of the library. `kernel.rs` +mirrors `PortableKernel.swift`'s protocol, `ScalarKernel`, and +dispatcher. Five files mirror their same-named Swift files, function +for function: `bit_field.rs`, `hamming_nn.rs`, `sha256.rs`, `hkdf.rs`, +and `float_vec_ops.rs`. Two Swift backends have no Rust counterpart. +`NeonKernel` does not need one. Rust's own portable-SIMD path, +`kernel_simd.rs`, already covers the same aarch64 targets. This path is +gated behind the nightly-only `simd-nightly` Cargo feature. +`MetalKernel` cannot have one, since Metal is an Apple-only framework. +The Rust port's documentation records this fact as a platform waiver, +not a gap. + +The Rust leg also ships one backend the Swift leg lacks entirely: +`kernel_avx512.rs`, an AVX-512 implementation for x86-64 processors. It +compiles, and is reachable through explicit selection, but +`PortableKernel::for_current_platform()` never chooses it. It stays a +dark path, until a future performance study proves it belongs in the +default path on real AVX-512 hardware. An unguarded call into its +processor-specific instructions would crash on hardware that lacks +them. So every one of its entry points checks the processor's actual +feature flags at runtime, before making that call. This guard was +verified in a 2026-06-28 security review. A dedicated cross-platform +test covers it. + +Both legs share one conformance obligation. Whichever backend a caller +selects, on either leg, its output must match the scalar reference bit +for bit. The test suites for both legs enforce this rule on every +change. diff --git a/packages/libs/SubstrateKernel/docs/OVERVIEW.md b/packages/libs/SubstrateKernel/docs/OVERVIEW.md new file mode 100644 index 0000000..1cfb71b --- /dev/null +++ b/packages/libs/SubstrateKernel/docs/OVERVIEW.md @@ -0,0 +1,179 @@ +--- +doc: OVERVIEW +package: SubstrateKernel +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateKernel/BitField.swift + blob: 755ce095e0db4d4d4f5f01cc3568cb8403e5952f + - path: Sources/SubstrateKernel/FloatVecOps.swift + blob: b1c4593a99fe078b19ad0058a2e76e9949abd524 + - path: Sources/SubstrateKernel/HammingNN.swift + blob: c810cb5ada7b0c1cdc7789791552478b19bdad07 + - path: Sources/SubstrateKernel/HKDF.swift + blob: 53cd1dbef1ea0b9a8c932cdcfc303b50eb49adba + - path: Sources/SubstrateKernel/PortableKernel-Metal.swift + blob: 4c9837448547cb44fde52a08b8e639876d23f150 + - path: Sources/SubstrateKernel/PortableKernel-NEON.swift + blob: 996063a101bcd76d2ea965b58072eac2141adf98 + - path: Sources/SubstrateKernel/PortableKernel-SIMD.swift + blob: 88caebd2a66c6541f9dff3aff7555dabc557ee5f + - path: Sources/SubstrateKernel/PortableKernel.swift + blob: 5279dadeb909d4d8980f2fc3a3100caf7d9066d3 + - path: Sources/SubstrateKernel/SHA256.swift + blob: 143a3530a8dee15446fa5711022638e0d61219b2 +--- + +# SubstrateKernel Overview + +## What This Library Does + +`SubstrateKernel` computes bit-level and hash operations. MOOTx01 runs +them on every memory it stores. It compares fingerprints. It extracts +values from packed bitmaps. It computes cryptographic hashes. + +A fingerprint is a short fixed-size code computed from a piece of +content. Similar content produces similar fingerprints. This lets the +system compare things fast, without reading them in full. This package +works with one fingerprint shape, `Fingerprint256`. It is a 256-bit +code held as four 64-bit blocks. + +Comparing two fingerprints means counting how many bit positions differ +between them. This count is called Hamming distance. A smaller Hamming +distance means the two fingerprints are more alike. The content behind +each one is more alike too. + +Finding the memories most similar to a query means finding the +fingerprints with the smallest Hamming distance to a probe fingerprint. +This is the single most frequent operation in MOOTx01's recall path. So +`SubstrateKernel` gives it several implementations. Each one suits a +different kind of hardware. + +`SubstrateKernel` is layer 2 of a four-package substrate split. Layer 1 +is `SubstrateTypes`. It defines pure data types with no logic, including +`Fingerprint256` itself. Layer 2 is this package. It computes over those +types. Layer 3 is `SubstrateML`. It builds learning and graph algorithms +on top of layer 2. Layer 4 is `SubstrateLib`. It orchestrates the whole +substrate for its callers. + +A package that composes libraries into a larger subsystem is called a +kit. Kits depend on libraries like this one. The reverse never happens. +Four kits depend directly on `SubstrateKernel` for their hot paths: +`LocusKit`, `CorpusKit`, `GeniusLocusKit`, and `EngramLib`. + +## The Problem It Solves + +Two devices must agree on how similar two fingerprints are. A MOOTx01 +estate is one user's complete memory store. Estates can federate. That +means separate devices share and compare memories. Suppose one device's +Hamming distance for a pair of fingerprints disagreed with another +device's distance for that same pair. Shared recall would then give +different answers on different hardware. + +Every operation in this package must therefore produce the same output +for the same input. It must do so on every platform, every time. This +package calls that guarantee its conformance contract. It treats one +implementation, `ScalarKernel`, as the oracle. Every other implementation +must match that oracle bit for bit. + +Meeting that contract while also running fast is hard. A scan over one +million fingerprints, at 32 bytes each, moves 32 megabytes through +memory for a single query. That much straight-line bit manipulation is +bandwidth-bound. Its cost comes from how fast data moves through the +processor, not from how complex the arithmetic is. Different processors +move that data at different speeds, depending on which instructions +they use. So one fixed implementation cannot be fastest everywhere. +`SubstrateKernel` solves this with one interface. Several interchangeable +implementations sit behind it. One reference implementation checks +every one of them. + +A second, smaller problem is duplication. Many packages upstream of +`SubstrateKernel` need bit-field extraction from a packed bitmap. Many +also need content hashing for the audit log. Left unmanaged, each +package would write its own version. A change to the bit layout, or to +the hash algorithm, would then require updating every copy. +`SubstrateKernel` centralizes both operations. One implementation now +serves every caller. + +## How It Works + +The `SubstrateKernel` protocol declares the operations every backend +must supply. These include Hamming distance between two fingerprints, +top-K nearest neighbor search, and bitwise OR-reduction across a group +of fingerprints. They also include SimHash projection, which folds a +set of numbers into a fingerprint, plus batched variants of each +operation. `ScalarKernel` implements the protocol with a plain loop +over each fingerprint's four 64-bit blocks. It is always available, on +every platform. It is the oracle. Any implementation that disagrees +with it, on any input, is by definition wrong. + +Three more implementations specialize the same protocol for particular +hardware. `SimdKernel` uses Swift's portable `simd` module. The +compiler turns this into ARM NEON instructions on Apple Silicon. +`NeonKernel` frames the same computation a different way. It works at +the byte level instead of the 64-bit-word level. This tests whether +that shape compiles to tighter machine code. `MetalKernel` +sends the batched Hamming distance computation to the GPU, through +Apple's Metal framework. This pays a fixed per-call setup cost. It then +scales well past roughly one hundred thousand candidates. A caller +picks an implementation through `PortableKernel`. It can choose one +automatically for the current platform, or ask for one by name for +testing. + +Four more files round out the package. `BitField` extracts and writes +fixed-width fields inside a 64-bit packed bitmap. This is the encoding +MOOTx01 uses to pack several small values into one machine word. +`SHA256` computes a standard cryptographic hash. It gives each +audit-log entry a unique, content-derived identifier. Through `HKDF`, +it also derives keys for the estate's cryptographic grants. +`FloatVecOps` defines the canonical floating-point vector operations: +length, normalization, dot product, and cosine similarity. Any faster +backend for those operations must match it bit for bit. `HammingNN` +offers a simpler, general-purpose top-K search over any sequence of +candidates. It works independently of the `PortableKernel` backends, +for callers who do not need backend selection. + +## How the Pieces Fit + +Figure 1 shows the library's topology. It shows the major parts and how +data moves between them. + +![Figure 1. Topology of SubstrateKernel](topology.svg) + +*Figure 1. Topology of SubstrateKernel. A caller asks `PortableKernel` +for a kernel. The dispatcher hands back one of four interchangeable +implementations of the `SubstrateKernel` protocol. All four are +conformance-gated against the `ScalarKernel` oracle. Five primitives +skip the dispatcher entirely: `HammingNN`, `BitField`, `SHA256`, +`HKDF`, and `FloatVecOps`.* + +`PortableKernel.kernelForCurrentPlatform()` picks `SimdKernel` on +64-bit ARM. It falls back to `ScalarKernel` everywhere else. It never +selects `MetalKernel`. A caller who wants the GPU path must ask for it +by name, with `PortableKernel.kernel(of: .metal)`. `NeonKernel` is also +available only by explicit request. It exists to be measured against +`SimdKernel`, not to replace it automatically. + +Whichever implementation runs, its output for a given input must match +`ScalarKernel`'s output for that same input. A conformance test suite +enforces this rule for every backend. It runs on both the Swift leg and +the Rust leg. + +## What Ships in the Package + +The package ships nine Swift source files. It ships no pinned data +artifacts. Unlike some sibling libraries, `SubstrateKernel`'s behavior +depends only on its algorithms, not on any versioned reference data. + +The package also ships a Rust port, in `rust/`. This port mirrors every +file except two. `NeonKernel` has no Rust equivalent, because Rust's +portable-SIMD path already covers the same ground on the relevant +targets. `MetalKernel` has no Rust equivalent either, because Metal is +an Apple-only framework with no Linux or Windows counterpart. + +The Rust leg adds one backend the Swift leg does not have: an AVX-512 +implementation for x86-64 processors. This backend stays dark. It is +built and tested, and reachable only by explicit request. It is never +chosen automatically, until a future performance study proves it +belongs in the default path. diff --git a/packages/libs/SubstrateKernel/docs/topology.svg b/packages/libs/SubstrateKernel/docs/topology.svg new file mode 100644 index 0000000..0de2f70 --- /dev/null +++ b/packages/libs/SubstrateKernel/docs/topology.svg @@ -0,0 +1,127 @@ + + + + + + + + + + + + + SubstrateKernel: kernel dispatch plus independent primitives + + + + Caller + e.g. LocusKit + + + + PortableKernel + kernelForCurrentPlatform / kernel(of:) + + + + SubstrateKernel protocol: every backend bit-identical to Scalar + + + ScalarKernel + the oracle + + + SimdKernel + arm64 default + + + NeonKernel + opt-in, benchmark + + + MetalKernel + opt-in, GPU + + + Hamming / OR-reduce / + SimHash / count-fold + result: identical across backends + + + + + + + + + + + + + Independent primitives: no dispatcher, called directly + + + HammingNN.topK + candidate-iterator top-K, + rowID-string tie-break + + + BitField + packed-bitmap + field extract/write + + + FloatVecOps + scalar oracle: + norm / dot / cosine + + + SHA256 + FIPS 180-4 + content hash + + + GrantHKDF + RFC 5869 + key derivation + + + + Every primitive above is called directly by its consumer; none passes through PortableKernel. + Audit-log content IDs (SubstrateLib's AuditGate) consume only BitField + SHA256 from this package. + Metal is Apple-only; the Rust port has no Metal leg (platform waiver) and adds a dark AVX-512 backend Swift lacks. + diff --git a/packages/libs/SubstrateLib/docs/AGENT_MAP.md b/packages/libs/SubstrateLib/docs/AGENT_MAP.md new file mode 100644 index 0000000..d5ea101 --- /dev/null +++ b/packages/libs/SubstrateLib/docs/AGENT_MAP.md @@ -0,0 +1,142 @@ +--- +doc: AGENT_MAP +package: SubstrateLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateLib/AuditGate.swift + blob: 1f5ea748a82d21756518ed2dc4dda9e0a2ceed4a + - path: Sources/SubstrateLib/KeyedCommitment.swift + blob: 2b93f766914a2261cc4d63067469a0609157665f + - path: Sources/SubstrateLib/KeyedCommitmentAudit.swift + blob: afdac27147578c4b0cd2324e3db861812f89097a + - path: Sources/SubstrateLib/MerkleHash.swift + blob: 2a7e1b957d0f38633ee9048108fb9b59f8ee5648 + - path: Sources/SubstrateLib/RowStateAutomaton.swift + blob: b3111584e1495a41d66e51de620e27e55f31d4cd + - path: Sources/SubstrateLib/SubstrateLibTelemetry.swift + blob: b76dd8280311aeb67d8f5499108a0c0b080dc787 + - path: Sources/SubstrateLib/Verbs.swift + blob: b3f6b0db6b099d6932f7e57c017637193e2a4be6 +--- + +# AGENT_MAP : SubstrateLib + +PURPOSE: orchestration layer of the substrate stack. Single write gate (AuditGate.admit) + row-state automaton (legal transitions, I-22/S-1/S-2/S-4 forbidden combinations) + reference impl of the nine substrate verbs (Substrate struct) + Merkle content-hash pipeline + keyed-commitment pipeline for expunge provenance. Value types (Row, RowState, NounType, LatticeAnchor, AuditEvent, HLC, MatrixF/O/T, Fingerprint256) live in SubstrateTypes. + +DEPS: imports SubstrateTypes (value types), SubstrateKernel (BitField, SHA256, GrantHKDF.hmac), SubstrateML (transitively via Package.swift, not used directly in these 7 files), IntellectusLib (Intellectus.report telemetry sink). Imported by: LocusKit (AuditGate.admit is the sole bitmap write path), other kits per DECISION_KIT_GRAPH_REFACTOR_2026-05-19. Sibling packages in the four-way split: SubstrateTypes (data), SubstrateKernel (hot-path kernels), SubstrateML (cold-path/ML). Rust port in rust/src/ mirrors all 7 files 1:1; conformance fixtures in rust/tests/ + Tests/SubstrateLibConformanceTests/ gate byte-identity. + +ENTRY POINTS (most callers need only these): +- AuditGate.swift:317 `AuditGate.admit(estateUuid:rowId:nounType:verb:prior:priorLatticeAnchor:writes:afterLatticeAnchor:vocabulary:hlc:actor:) -> Result` : THE single production write gate for row bitmaps +- AuditGate.swift:222 `VocabularyValidator.freeze(union:) -> Result` : one-time vocabulary construction, required before any Vocabulary exists +- RowStateAutomaton.swift:213 `RowStateAutomaton.validate(from:on:targetingFields:) throws -> RowState` : the constitutional mutation gate (legality + I-22/S-1/S-2/S-4) +- Verbs.swift:76 `struct Substrate` : in-memory reference impl of the nine verbs; scalar oracle for production storage engines + +## Symbol Table + +### Row-state automaton : RowStateAutomaton.swift +- :50 `enum RowStateAutomaton` : namespace; RowState/RowVerb/RowStateError types live in SubstrateTypes, imported here +- :72 `canTransition(from:to:viaVerb:) -> Bool` : §10 STRING-verb vocabulary (legacy); looks up :84 `verbTable` (fileprivate `[VerbKey: RowState]`); used by Verbs.swift mutate/withdraw +- :78 `fileprivate struct VerbKey` : (RowState, String) composite key for verbTable +- :127 `static let transitions: [TransitionKey: RowState]` : CANONICAL enum-keyed table (§9 lifecycle vocabulary); absence from map = illegal +- :204 `transition(from:on:) -> RowState?` : single lookup against `transitions` +- :213 `validate(from:on:targetingFields:) throws -> RowState` : transition() + ForbiddenCombinations.check(); THE gate; bypassing is forbidden (v0.36 C1 resolution) +- :227 `struct TransitionKey` : (from: RowState, verb: RowVerb) +- :240 `struct BitmapFields` : adjective/operational/provenance UInt64 triple +- :260 `enum ForbiddenCombinations` +- :270 `check(state:fields:) throws` : I-22 (secret+exportable forbidden), S-1 (accepted⇒trust≥canonical), S-2 (defensive state-encoding check), S-4 (accepted⇒sensitivity≤elevated); S-5 (tombstone completion flag) NOT YET IMPLEMENTED (queued, see comment at file end) + +### Write gate : AuditGate.swift +- :51 `struct FieldSlot: Hashable, Sendable` : one declared field-slot: column/shift/width/label/legalValues +- :52 `enum Column` : .adjective | .operational | .provenance +- :72 `capacity: Int64` : 1< Bool` : width + enumerated-legal-value check +- :106 `enum AuditState` / :113 `AuditSensitivity` / :118 `AuditExportability` / :123 `AuditTrust` : SubstrateLib-local mirrors of LocusKit's Adjectives.swift enums (raw values MUST match; can't import upward); parity enforced by GuardianPairParityTests +- :134 `struct Vocabulary: Sendable` +- :145 `static let basis: Set` : universal fields: state(0,w6) sensitivity(6,w6) exportability(12,w6) trust(18,w6) flags(24,w3, bits 24/25/26 = state_extension/lineage_clustering/dreaming_recalc_required); legalValues derived from allCases.map{rawValue} +- :188 `union: Set` : per-instance consumer fields, frozen +- :192 `fileprivate init(validatedUnion:)` : ONLY VocabularyValidator.freeze can construct +- :196 `slot(for:) -> FieldSlot?` : basis-first lookup, then union +- :206 `enum VocabularyError` : .overlap / .basisCollision / .malformedWidth / .valueExceedsWidth +- :222 `enum VocabularyValidator` +- :224 `freeze(union:) -> Result` : 3-step validation (width-sane → no basis collision → no union-union overlap); run ONCE before any data exists +- :258 `struct FieldWrite: Sendable` : (slot, value) one consumer-requested change +- :269 `enum GateViolation: Error, Sendable` : .undeclaredField / .illegalValue / .basisViolation(Error) / .stateInconsistentWithVerb +- :292 `description: String` : English-only at ARIA boundary; NO Swift case-name leakage; carries the "illegal state transition: ..." sentinel the describe_gate_rejection parser matches +- :311 `enum AuditGate` +- :317 `admit(...) -> Result` : see ENTRY POINTS; 4 steps: vocab+value gate → read-modify-write (preserves unaddressed bits) → basis gate (state must equal verb's authorized output) → contentID + AuditEvent +- :421 `static func contentID(estateUuid:rowId:hlc:verb:after:afterAnchor:) -> UUID` : SHA-256(stable byte encoding) folded to UUID; NAME-keyed (not ordinal), vocabulary-set-independent; also called directly by Verbs.swift's appendAudit so both paths produce identical event IDs + +### Verb orchestration : Verbs.swift +- :48 `typealias RowId = UUID` : wire-compatible with Rust's RowId(u128) +- :55 `enum SubstrateError: Error, Equatable` : invalidStateTransition / missingLatticeAnchor / invalidNounType / rowNotFound / forbiddenStateCombination / alreadyTombstoned / proposalRequired / nonProposalCannotUseProposalVerb +- :76 `struct Substrate` : see ENTRY POINTS; fields :77-84 (estateUuid, rows, auditEvents(G-Set), matrixF/O/T, hlc, rowCountActive) +- :86 `init(estateUuid:hlc:)` +- :110 `capture(nounType:adjectiveBitmap:operationalBitmap:provenanceBitmap:latticeAnchor:fingerprint:lineageId:content:actor:ts:) -> Result` : ONLY verb that originates a row w/o prior state; proposal⇒pending else active; rejects null latticeAnchor +- :180 `reanchor(rowId:newLatticeAnchor:actor:ts:) -> Result<(), SubstrateError>` : lattice-link change only, bitmaps untouched +- :213 `enum MutationKind: String` : confirm/reject/contest/supersede/automated_confirm/decay/expire/lineage_advance/actuator_confirm +- :222 `mutate(rowId:mutationKind:newAdjectiveBitmap:newOperationalBitmap:newProvenanceBitmap:actor:ts:) -> Result<(), SubstrateError>` : canTransition() (§10 string verbs) + isLegalRowState() gate; general state+field change +- :304 `withdraw(rowId:actor:ts:) -> Result<(), SubstrateError>` : → .withdrawn; setStateField raw=18 +- :365 `expunge(rowId:reason:actor:ts:) -> Result<(), SubstrateError>` : → .tombstoned, content=nil; REFUSES .accepted rows (S-3, audit-grade survive intact) before any other check +- :425 `recall(matching:asOf:ts:) -> [Row]` : READ-ONLY; no mutation, no audit event; asOf reconstructs via audit-log HLC filter +- :457 `propose(...) -> Result` : capture(nounType: .proposal, ...) +- :484 `associate(rowA:rowB:signalSourcesBitset:weight:...) -> Result` : capture(nounType: .association, ...); `weight` param ACCEPTED THEN DISCARDED (vestigial, reserved for pre-2.0 gauntlet experiment) : do not silently wire it up or silently drop the parameter +- :531 `learn(...) -> Result` : capture(nounType: .learnedReference, ...) +- :555 `private appendAudit(verb:rowId:before:after:beforeAnchor:afterAnchor:actor:)` : advances hlc; calls AuditGate.contentID (same fn as the gate path) so both paths produce identical event IDs +- :596 `private isLegalRowState(state:adjective:operational:) -> SubstrateError?` : wraps ForbiddenCombinations.check; provenance passed as 0 (unused by check) +- :615/:624/:632/:637 private bit-twiddling wrappers around SubstrateTypes.RowBitmaps (rowHasBit, extractFieldValues, extractState, setStateField) + +### Merkle hash pipeline : MerkleHash.swift +- :31 `struct MerkleVectorInput: Sendable` : modelID, vectorIndex, floats (Float32 LE) +- :51 `enum MerkleHash` +- :76 `leaf(drawerId:content:vectors:) -> ContentHash` : v2 format: vector identity (modelID+vectorIndex) written BEFORE float payload (v1 gap = WS2-F4 vector-substitution finding, fixed 2026-06-28) +- :101 `interior(childHashes: [(UUID, ContentHash)]) -> MerkleRoot` : sorted by UUID BE bytes; empty ⇒ MerkleRoot.empty +- :130 `interior(childRoots: [(UUID, MerkleRoot)]) -> MerkleRoot` : same as above, wing/estate-level roll-up (children already MerkleRoots) +- :157 `tombstone(drawerId:) -> ContentHash` : domain tag only + drawer id; no content/vectors (already destroyed) +- :178 `static func canonicalLeafBytes(drawerId:content:vectors:domainTag:) -> [UInt8]` : SHARED encoding with KeyedCommitment.commit; domainTag is the only difference between leaf hash and commitment preimage +- :226/:230/:236 private byte-encoding helpers (uuidBytes, appendU64BE, appendU32BE) + +### Keyed commitment : KeyedCommitment.swift +- :26 `struct KeyedCommitmentValue: Hashable, Sendable, Codable` : hmacBytes (32B, precondition-enforced) + keyVersion +- :40 `hexString: String` +- :49 `enum KeyedCommitment` +- :66 `commit(key:keyVersion:drawerId:content:vectors:) -> KeyedCommitmentValue` : MerkleHash.canonicalLeafBytes(domainTag: .commitment=0x03) → GrantHKDF.hmac (SubstrateKernel, no new HMAC impl) + +### Commitment audit log : KeyedCommitmentAudit.swift +- :27 `struct KeyedCommitmentAuditEntry: Hashable, Sendable` : id (content-hash), drawerId, commitment, tombstoneHLC, reason +- :40 `init(drawerId:commitment:tombstoneHLC:reason:)` : computes :62 `private static computeID(...) -> [UInt8]` (SHA-256 over identifying fields) at construction +- :92 `struct CommitmentAuditLog: Sendable` : G-Set keyed by content hash +- :103 `add(_:)` : idempotent insert +- :108 `merge(_:)` : CRDT join = set union +- :114 `count: Int` +- :117 `orderedEntries: [KeyedCommitmentAuditEntry]` : sorted by tombstoneHLC +- :122 `entries(forDrawer:) -> [KeyedCommitmentAuditEntry]` + +### Telemetry : SubstrateLibTelemetry.swift +- :78 `enum SubstrateLibMetric` : string-constant namespace, `substratelib.*` metrics +- :83/:86 auditGateAdmitCount / auditGateRejectCount +- :91/:94/:97/:100/:103/:106 verbCaptureCount / verbMutateCount / verbWithdrawCount / verbExpungeCount / verbRecallCount / verbReanchorCount +- :111/:114 writeGateAdmittedCount / writeGateRejectedCount +- :132 `emitAuditGateAdmit(nounTypeRaw:ts:)`, :152 `emitAuditGateReject(violationName:ts:)` +- :170 `emitWriteGateAdmitted(verb:ts:)`, :187 `emitWriteGateRejected(verb:reason:ts:)` +- :204/:219/:233/:247/:266/:280 `emitVerb{Capture,Mutate,Withdraw,Expunge,Recall,Reanchor}Count(...)` : one per Verbs.swift verb +- ALL emit fns `@inline(__always)`; wrap `Intellectus.report(...)`; disabled-path cost = one atomic load + branch, zero allocation + +## INVARIANTS / GOTCHAS + +- DETERMINISM IS THE CONTRACT (same as LatticeLib). No file in this package may read a clock. Every `ts` parameter is caller-supplied epoch seconds; monitoring on/off must never change functional output. Rust mirrors every algorithm; conformance fixtures in rust/tests/ + Tests/SubstrateLibConformanceTests/ gate byte-identity : run both suites on any change to audit_gate/merkle_hash/keyed_commitment/row_state/verbs. +- AuditGate and Substrate do NOT call each other. Two independent consumers of the same RowStateAutomaton rules, not a pipeline. Do not wire one to call the other : that would let a fast-path bug hide behind a passing reference-path test. +- AuditGate.admit and Verbs.appendAudit call the SAME contentID function (AuditGate.contentID) so both paths produce byte-identical event IDs for the same logical event. If you change the byte layout in contentID, both paths and the Rust mirror change together. +- Content IDs (AuditGate.contentID, KeyedCommitmentAuditEntry.computeID) are NAME/CONTENT-keyed, never random UUIDs. This is what lets a G-Set (grow-only set) dedupe on federation merge. Do not reintroduce `UUID()` random defaults on any audit-adjacent identity. +- S-3 (cookbook §9.5.3): accepted rows CANNOT transition to tombstoned. Enforced in TWO places independently: RowStateAutomaton.transitions (no TransitionKey(.accepted, .tombstone) entry) AND Verbs.expunge's explicit early-return guard. Do not remove either; they are a deliberate belt-and-suspenders pair. +- S-5 (tombstone ⇒ expunge_completed_flag) is NOT YET IMPLEMENTED in ForbiddenCombinations.check : queued pending F17 cookbook location decision (adjective bit 24 or operational reserved bits). A previous incorrect implementation (zeroing bitmaps on tombstone) was removed 2026-05-27; do not reintroduce that specific check. +- AuditState/AuditSensitivity/AuditExportability/AuditTrust (AuditGate.swift) are LOCAL COPIES of LocusKit/Adjectives.swift enums : raw values must match exactly (SubstrateLib cannot import LocusKit; dependency graph points the other way). GuardianPairParityTests enforces this; the `@guardian-pair` comment annotations mark every duplicated literal (also present in RowStateAutomaton.swift's ForbiddenCombinations.check: 48/32/3/16 = secret/public/canonical/elevated raw values). +- Vocabulary can ONLY be constructed via VocabularyValidator.freeze (fileprivate init). Never bypass with a literal struct construction : that skips overlap/collision/width validation entirely. +- FieldSlot.legalValues empty ⇒ "any value that fits width" (used for state, governed by the automaton instead of an enumerated set... actually state DOES enumerate via AuditState.allCases; empty-legalValues is used only by the flags field). Do not conflate "empty" with "no restriction beyond width" vs "enumerated" : check the specific basis slot. +- MerkleHash leaf format is v2 (vector identity before float payload). v1 had a vector-substitution security gap (WS2-F4, fixed 2026-06-28). KeyedCommitment.commit shares this exact byte encoding via canonicalLeafBytes : a leaf-format change without updating both Rust and Swift, and both the hash and commitment call sites, reopens or diverges the fix. +- MerkleHash.interior sorts children by UUID ascending (lexicographic 16-byte BE) : required for roll-up to be write-order-independent. Do not sort by insertion order or hash value. +- Verbs.associate's `weight` parameter is accepted and explicitly discarded (`_ = weight`) : vestigial, reserved for a future experiment. Do not silently start persisting it without a matching bitmap/column change, and do not remove the parameter from the signature without checking callers. +- Package.swift depends on SubstrateTypes + SubstrateKernel + SubstrateML + IntellectusLib directly (not transitively) per DECISION_LIFT_PACKAGE_SWIFT_RULE_2026-05-28 : IntellectusLib sits below SubstrateLib in the topology (depends on nothing in-repo), so this is not a layering violation. +- No pinned data artifacts ship in this package (unlike LatticeLib) : every guarantee here is computational (fixed table, fixed byte encoding, cryptographic hash), not reference-data-driven. diff --git a/packages/libs/SubstrateLib/docs/DETAILS.md b/packages/libs/SubstrateLib/docs/DETAILS.md new file mode 100644 index 0000000..275a2f2 --- /dev/null +++ b/packages/libs/SubstrateLib/docs/DETAILS.md @@ -0,0 +1,161 @@ +--- +doc: DETAILS +package: SubstrateLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateLib/AuditGate.swift + blob: 1f5ea748a82d21756518ed2dc4dda9e0a2ceed4a + - path: Sources/SubstrateLib/KeyedCommitment.swift + blob: 2b93f766914a2261cc4d63067469a0609157665f + - path: Sources/SubstrateLib/KeyedCommitmentAudit.swift + blob: afdac27147578c4b0cd2324e3db861812f89097a + - path: Sources/SubstrateLib/MerkleHash.swift + blob: 2a7e1b957d0f38633ee9048108fb9b59f8ee5648 + - path: Sources/SubstrateLib/RowStateAutomaton.swift + blob: b3111584e1495a41d66e51de620e27e55f31d4cd + - path: Sources/SubstrateLib/SubstrateLibTelemetry.swift + blob: b76dd8280311aeb67d8f5499108a0c0b080dc787 + - path: Sources/SubstrateLib/Verbs.swift + blob: b3f6b0db6b099d6932f7e57c017637193e2a4be6 +--- + +# SubstrateLib Details + +This document walks through every source file in the package. Read `OVERVIEW.md` first for the big picture. + +Files appear here in dependency order. The row-state automaton comes first, because both write paths consult it. The write gate and the reference verbs come next. These are the two paths that consult the automaton. The content-integrity pipeline comes after that, since it is independent of both. Telemetry comes last, because every other file calls into it. + +A note on vocabulary before the walkthrough. A row is one stored memory. A bitmap is a 64-bit integer. It packs several small status fields at fixed bit positions. One integer can carry many independent facts cheaply this way. A verb is one of a fixed set of actions a row can undergo. Capture, mutate, and withdraw are examples. An audit event is the permanent record of one verb happening to one row. + +## RowStateAutomaton.swift + +This file provides the row-state automaton. The automaton is a transition table. It says which lifecycle-state changes are legal. The file also provides the forbidden-combination checks. These checks catch a legal transition that lands on an illegal field combination. + +The data types the automaton operates on live in a sibling package, SubstrateTypes. These types are `RowState` (ten states), `RowVerb` (twelve verbs), and `RowStateError`. Only the compute logic stays in SubstrateLib: the table, the validation function, and the forbidden-combination rules. + +`RowState`'s ten values fall into three numeric clusters. These clusters are active, historical, and terminal. The clusters are deliberately spaced apart in the numbering. This lets a caller classify a stored raw value's cluster with one shift-and-mask step. The caller need not decode the full enum. + +`RowStateAutomaton.canTransition(from:to:viaVerb:)` answers a yes-or-no question. It uses an older vocabulary of verb names. That vocabulary includes contest, supersede, expunge, and others. This vocabulary comes from an earlier design phase. The function looks the pair up in a private table, `verbTable`. It compares the result to the requested target state. The function exists because `Verbs.swift`'s `mutate` and `withdraw` still use this vocabulary. New call sites should prefer the canonical function described next. + +`RowStateAutomaton.transitions` is the canonical table. It is a dictionary keyed by a state-and-verb pair. It uses the `RowVerb` enum rather than raw strings. Any pair absent from the table is illegal by construction. There is no default case to fall through. + +`RowStateAutomaton.transition(from:on:)` looks up one pair. It returns the next state, or `nil` if the pair is absent. + +`RowStateAutomaton.validate(from:on:targetingFields:)` is the single mutation gate. The design calls it constitutional. It requires a legal transition. It also runs the target's bitmap fields through `ForbiddenCombinations.check`. It returns the next state only if both checks pass. + +An earlier design defect let one code path skip the second check. That path could write field combinations the table alone would not catch. Every mutation path in this package now routes through one of these two checks. `AuditGate` does the same before a write counts as final. + +`ForbiddenCombinations.check(state:fields:)` enforces four safety rules. The transition table cannot express these rules, because they depend on more than one field at once. + +- A row marked secret can never also be marked exportable. This holds regardless of the row's lifecycle state. Sensitivity and exportability are stored as separate six-bit fields in the same bitmap. Nothing about the storage format alone stops both from being set to conflicting extremes. So this rule is checked explicitly on every write. +- An accepted row must carry a trust level of at least canonical. An accepted row is one a person or process explicitly confirmed. It is treated as audit-grade. Accepted status is a promise about reliability. A low trust value would make that promise false. +- A withdrawn or rejected row must actually encode the matching state value in its bitmap. This is a defensive check against corrupted input. It is not a rule a caller could realistically violate through the API. +- An accepted row's sensitivity may not exceed elevated. A higher sensitivity would make an audit-grade row too restricted. That would defeat the purpose accepted status is meant to serve. + +A fifth rule the design calls for is not yet implemented here. That rule says a tombstoned row must carry an expunge-completed flag. The source comment explains why. An earlier attempt enforced the wrong invariant. It zeroed bitmap fields that the design says must survive expunge. The attempt was removed until the correct flag location is settled elsewhere in the bitmap layout. + +`BitmapFields` and `TransitionKey` are the small carrier types the functions above operate on. `BitmapFields` holds three raw 64-bit integers. `TransitionKey` is a state-and-verb composite key. + +## AuditGate.swift + +This file provides the production write gate, `AuditGate.admit`. It also provides the vocabulary machinery that arms the gate. Storage kits call this interface on every bitmap mutation. LocusKit is the current example. + +A write gate is only as strong as the vocabulary it enforces. So this file first defines what a legal field looks like. `FieldSlot` describes one declared field. It records which of three columns the field lives in: adjective, operational, or provenance. It also records the field's bit position and width within that column. It records a human label too. It records the field's set of legal values as well. An empty legal-value set means any value that fits the width. Basis fields use this feature. The automaton governs their combinations instead of an enumerated list. `FieldSlot.admits(value:)` is the per-field check. The value must fit the width. If the field enumerates specific legal values, the value must be one of them. + +`Vocabulary.basis` is the fixed set of fields every SubstrateLib instance must agree on. Every federation peer must agree on it too. The fields are state, sensitivity, exportability, and trust. Each one is six bits wide. A three-bit flags field covers miscellaneous markers. Their legal-value sets come from four local enums: `AuditState`, `AuditSensitivity`, `AuditExportability`, and `AuditTrust`. Adding a case to one of those enums extends the gate's vocabulary automatically. This avoids a second, easy-to-forget update elsewhere. These four enums duplicate raw values defined independently in a higher-level package, LocusKit. SubstrateLib sits below that package in the dependency graph, so it cannot import LocusKit's types. A separate test suite, `GuardianPairParityTests`, checks that the two definitions never drift apart. + +Beyond the fixed basis, each SubstrateLib instance may register its own additional fields. These form the union. `VocabularyValidator.freeze(union:)` is the one place a union gets accepted. It checks three things. Every proposed field's width and enumerated values must fit within 64 bits. No proposed field's bits may collide with a basis field's bits. No two proposed fields may collide with each other. Only a union that passes all three checks becomes a `Vocabulary`. The type's initializer is `fileprivate`. So an unvalidated vocabulary cannot reach the gate by any other path. This validation runs once, before any row exists. That timing matters, because a corrupt vocabulary found after data has accumulated would be far harder to recover from. + +`AuditGate.admit(...)` is the gate itself. It runs in four steps. + +First, every field in the caller's write list must resolve to a declared slot in the vocabulary. Its value must be admissible. An undeclared field or an out-of-range value is rejected immediately, before any bitmap changes happen. + +Second, the gate performs a read-modify-write. It starts from the row's prior bitmaps, or zero for a new row. It writes only the addressed bits. It leaves every other bit exactly as it was. A consumer that owns one field can never accidentally clobber a field it does not know about. + +Third, the gate checks the resulting state against what the supplied verb was allowed to produce. For a brand-new row, the verb must be `capture`. The encoded state must also be a legal starting state. For an existing row, the verb's transition must match the state the write actually encodes. The forbidden-combination rules must also hold. + +Fourth, and only if every prior step succeeded, the gate computes a deterministic content identifier. It returns one canonical `AuditEvent`. + +`AuditGate.contentID(...)` computes that identifier. It serializes the identifying fields into a fixed byte layout. It hashes them with SHA-256. It folds the first sixteen bytes of the digest into a UUID. The identifier is a pure function of the event's content, not a random value. So two replicas that independently compute the same logical event produce the same identifier. That sameness is what lets a grow-only set deduplicate the event automatically when replicas merge. Federation depends on this property. + +`GateViolation` is the gate's error type. It covers four cases: an undeclared field, an illegal value, a basis violation, and a mismatched state. The basis violation wraps the underlying `RowStateAutomaton` error. Its `description` property produces plain English at the system's outer boundary. No Swift type or case name ever appears in user-visible text. This includes a specific sentinel phrase: "illegal state transition: ...". A downstream parser matches on that phrase. + +## Verbs.swift + +This file provides the nine substrate verbs. It also provides `Substrate`, the in-memory reference implementation of all nine. This is the scalar oracle. Production storage engines implement the same nine verbs against real persistence. Their behavior gets checked against this reference. + +`RowId` is a type alias for `UUID`. It keeps a distinct name so call sites read in the substrate's own vocabulary. Under the hood it stays a plain `UUID`. The Rust port spells the same identifier as a `u128` newtype. The two encodings are byte-identical. + +`Substrate` bundles everything one estate's in-memory reference state needs. It holds a dictionary of rows. It holds an append-only list of audit events, treated as a grow-only set. It holds three summary matrices, `matrixF`, `matrixO`, and `matrixT`, all defined in SubstrateTypes. These matrices track aggregate statistics across all rows without re-scanning them. `Substrate` also holds a hybrid logical clock, `hlc`. This clock orders events across replicas without relying on wall time. Finally, it holds a running count of non-tombstoned rows. + +Each verb method follows the same shape. It checks preconditions against the row-state automaton. It applies the bitmap change. It updates the two matrices with a symmetric delta. It subtracts the row's old contribution and adds its new one. It appends one audit event. It emits one telemetry count. + +Timestamps arrive as a caller-supplied `ts` parameter, defaulting to `0.0`. The method never reads a clock internally. This keeps the verb's functional output independent of when it happens to run. + +`capture(...)` creates a new row. It matters because it is the only verb that can originate a row without a prior state to transition from. A proposal noun type starts as `pending`, awaiting confirmation. Every other noun type starts as `active`. A missing lattice anchor is rejected outright. Every row must be locatable in the classification lattice from the moment it exists. + +`reanchor(...)` changes only a row's link into the classification lattice. It leaves the row's bitmaps untouched. It exists separately from `mutate`, because reclassifying a memory differs from changing its lifecycle state or its status fields. + +`mutate(...)` is the general-purpose state-and-field change. It covers six mutation kinds: confirm, reject, contest, supersede, decay, and expire. It also covers the actuator and automated confirm variants. It matters because it is where the automaton's transition check and the forbidden-combination check both gate a single call. An illegal transition or an illegal resulting combination gets rejected before any row or matrix is touched. + +`withdraw(...)` retracts a row by moving it to the `withdrawn` state. It matters as a distinct verb, rather than a case of `mutate`, because withdrawal is available from more starting states than most other transitions. It is also common enough to warrant its own simple call shape. + +`expunge(...)` is the legal-compliance hard delete. It moves a row to `tombstoned` and sets its content to `nil`, destroying the verbatim payload. It matters that `expunge` explicitly refuses an `accepted` row before doing anything else. An audit-grade row must survive intact by design. No combination of inputs can route around that refusal. + +`recall(...)` is the only read-only verb. It filters rows by a caller-supplied predicate. It can also reconstruct the row set as of a past point in the hybrid logical clock, by intersecting with the audit log. It matters that recall never mutates and never appends an audit event. A memory system that logged every read as if it were a write would make querying create liability. + +`propose`, `associate`, and `learn` are thin verbs. Each delegates to `capture` with a fixed `nounType`: proposal, association, and learned reference. They matter as distinct entry points, because each represents a different reason a row came to exist, even though the mechanics underneath are identical. `associate` additionally accepts a computed `weight` parameter. It deliberately discards that parameter after accepting it. The parameter is reserved for a future experiment. The source documents that the value is accepted and thrown away today. It is not silently invented if it were missing instead. + +The four private helpers at the bottom of the file are internal plumbing. `appendAudit` advances the clock. It builds one `AuditEvent` using the same content-identifier function `AuditGate` uses. So an event produced by either path is identical in shape. `isLegalRowState` is the verb layer's call into `ForbiddenCombinations.check`. The rest are small bit-manipulation wrappers around the shared `RowBitmaps` type in SubstrateTypes: `rowHasBit`, `extractFieldValues`, `extractState`, and `setStateField`. + +## MerkleHash.swift + +This file provides the public hash pipeline for the Merkle-style content-integrity tree. It has three functions. They turn a row's content, a set of child hashes, or a tombstoned row into one deterministic SHA-256 value. + +A Merkle tree proves that content has not changed. It does this by comparing hashes instead of comparing the content itself. It also lets a parent's hash summarize all of its children's hashes. `MerkleDomain` tags, defined in SubstrateTypes, mark each function's input before hashing. There is a leaf tag, an interior tag, and a tombstone tag. This tagging stops a hash computed one way from ever being mistaken for a hash computed another way. + +`MerkleHash.leaf(...)` hashes one memory's content and its vector embeddings into a `ContentHash`. It matters because its byte encoding is versioned. An earlier version wrote each vector's identifying information only as a sort key. That information was which embedding model produced the vector and which slot in a multi-vector row it occupies. The earlier version did not write that information into the hashed bytes themselves. This meant an attacker could substitute one model's vector for another's without changing the hash. The source calls out this security defect by name: finding WS2-F4. The current version writes vector identity into the hash input before the float payload. This closes that gap. + +`MerkleHash.interior(childHashes:)` and `interior(childRoots:)` roll up a node's children into one parent hash. They sort children by UUID first. This way, the result never depends on the order children happened to be written or iterated in. Both functions return `MerkleRoot.empty` for a childless node, rather than hashing nothing. This keeps the empty case an explicit, named value instead of an arbitrary hash of zero bytes. + +`MerkleHash.tombstone(drawerId:)` hashes an expunged row's identity alone. There is no content and no vectors, because expunge has already destroyed both. It matters that the tombstone hash still exists at all. It lets the Merkle tree keep proving that a specific row occupied a specific position. This holds even after everything about that row except its identity is gone. + +`canonicalLeafBytes(...)` is the shared byte-encoding function. Both this file and `KeyedCommitment.swift` call it. It matters that this function is shared rather than duplicated. The content hash and the keyed commitment are deliberately computed from the exact same preimage. They differ only in the domain tag and the algorithm applied to it: a plain hash versus a keyed HMAC. This is what lets the two values be compared meaningfully, as two views of the same underlying commitment to content. + +## KeyedCommitment.swift + +This file provides the public keyed-commitment API used at expunge time. It offers a way to prove a payload existed, without keeping anything a reader could reconstruct the payload from. + +`KeyedCommitmentValue` carries the output. This is thirty-two raw HMAC-SHA256 bytes, plus the key version that produced them. Recording the key version matters, because estate keys rotate over time. A commitment made under an older key must remain verifiable against that older key even after rotation. Without the version, a rotated key would silently invalidate every commitment made before it. + +`KeyedCommitment.commit(...)` builds the commitment. It reuses `MerkleHash.canonicalLeafBytes`, the exact same byte encoding `MerkleHash.leaf` uses. But it applies the commitment domain tag instead of the leaf tag. It then runs that byte sequence through an existing HMAC-SHA256 implementation. That implementation comes from the sibling package SubstrateKernel, rather than a new one written for this purpose. The domain separation matters for a specific reason. It guarantees a commitment and a plain content hash of the same underlying payload are always different values. So one can never be mistaken for, or substituted as, the other. + +## KeyedCommitmentAudit.swift + +This file provides the audit-log entry type and an append-only container for keyed commitments made at expunge time. This record is distinct from the ordinary state-transition audit event. It records that a commitment was made. It does not record that a row's state changed. + +`KeyedCommitmentAuditEntry` bundles four things: the drawer identity, the commitment value, the tombstone time, and a human-readable reason. The tombstone time comes from the hybrid logical clock. Its identifier comes from `computeID(...)`. That function computes a SHA-256 hash over the same identifying fields. This matters for the same reason `AuditGate.contentID` matters elsewhere in this package. Two replicas that independently record the same logical commitment compute the same identifier. So a grow-only set deduplicates them without extra coordination. + +`CommitmentAuditLog` is that grow-only set. Entries are keyed by their content hash in a dictionary. `add(_:)` inserts one entry idempotently. `merge(_:)` unions two logs together. This is the CRDT join that lets two replicas' commitment logs converge, regardless of the order in which entries arrive. `orderedEntries` and `entries(forDrawer:)` give callers a stable, HLC-ordered view for display or audit review. + +## SubstrateLibTelemetry.swift + +This file provides the metric name catalogue. It also provides the emit functions every other file in this package calls into when reporting activity. + +SubstrateLib's own comment calls this file the determinism floor of the substrate stack. Nothing in this package may read a clock or change its functional output depending on whether monitoring is enabled. This file is where that promise gets implemented, not merely stated. + +Every emit function wraps its call in `Intellectus.report(...)`, from the sibling package IntellectusLib. When monitoring is disabled, which is the default, this costs exactly one lock-free flag check and a branch. No metric object gets built. No lock gets taken. Nothing gets allocated. Every timestamp the emit functions accept is a parameter supplied by the caller. It is never read from a clock inside this file. So the substrate's behavior stays identical whether monitoring is on or off. + +`SubstrateLibMetric` is a namespace of string constants, one per metric name. This makes the complete list of everything this package can report visible in one place. A typo in a metric name gets caught at compile time. It does not show up as a silently missing metric in production. + +The eight `emit...` functions each correspond to one point in the gate or verb flow. `emitAuditGateAdmit` and `emitAuditGateReject` fire from `AuditGate.admit`'s two outcomes. `emitWriteGateAdmitted` and `emitWriteGateRejected` fire from the same gate, tagged by verb and rejection reason. This lets operators break down gate traffic differently than by outcome alone. One function fires per verb: `emitVerbCaptureCount`, `emitVerbMutateCount`, `emitVerbWithdrawCount`, `emitVerbExpungeCount`, `emitVerbRecallCount`, and `emitVerbReanchorCount`. Each is marked `@inline(__always)`. So the disabled-path cost stays a single flag check with no function-call overhead added on top. + +## Rust Port and Conformance + +The `rust/` directory contains the second leg of the library. Seven source files mirror the Swift implementations one for one: `audit_gate.rs`, `keyed_commitment.rs`, `merkle_hash.rs`, `row_state.rs`, `verbs.rs`, and `substrate_lib_telemetry.rs`. The crate root is `lib.rs`. + +Rust has no `CaseIterable`. So the adjective-vocabulary values that the Swift leg derives from enum cases are instead typed constant slices in `audit_gate.rs`. These stay in step with the Swift enums by convention. The same cross-layer parity tests check this. + +The two legs share conformance fixtures. These live in `rust/tests/` and `Tests/SubstrateLibConformanceTests/`. The fixtures are recorded input-output pairs. They cover the audit-log fold, bitmap field constants, the SimHash-derived float comparisons, and the wire format. Both implementations must reproduce these byte for byte. When you change either leg, run both test suites. The fixtures are the contract. diff --git a/packages/libs/SubstrateLib/docs/OVERVIEW.md b/packages/libs/SubstrateLib/docs/OVERVIEW.md new file mode 100644 index 0000000..a8b6e74 --- /dev/null +++ b/packages/libs/SubstrateLib/docs/OVERVIEW.md @@ -0,0 +1,118 @@ +--- +doc: OVERVIEW +package: SubstrateLib +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateLib/AuditGate.swift + blob: 1f5ea748a82d21756518ed2dc4dda9e0a2ceed4a + - path: Sources/SubstrateLib/KeyedCommitment.swift + blob: 2b93f766914a2261cc4d63067469a0609157665f + - path: Sources/SubstrateLib/KeyedCommitmentAudit.swift + blob: afdac27147578c4b0cd2324e3db861812f89097a + - path: Sources/SubstrateLib/MerkleHash.swift + blob: 2a7e1b957d0f38633ee9048108fb9b59f8ee5648 + - path: Sources/SubstrateLib/RowStateAutomaton.swift + blob: b3111584e1495a41d66e51de620e27e55f31d4cd + - path: Sources/SubstrateLib/SubstrateLibTelemetry.swift + blob: b76dd8280311aeb67d8f5499108a0c0b080dc787 + - path: Sources/SubstrateLib/Verbs.swift + blob: b3f6b0db6b099d6932f7e57c017637193e2a4be6 +--- + +# SubstrateLib Overview + +## What This Library Does + +SubstrateLib is the orchestration layer of MOOTx01's storage substrate. It decides what counts as a legal change to a memory. It refuses every other kind of change. + +MOOTx01 is an on-device AI memory system. It stores what an AI observes over time. Later, the AI can recall that content. + +Each stored memory is one row. A row is an entry with a lifecycle state. A row also links into a classification lattice. LatticeLib builds that lattice. + +Each row carries three 64-bit integers called bitmaps. A bitmap packs many small status fields into one compact form. These fields include sensitivity, trust, and exportability. + +SubstrateLib supplies three things a row needs to stay trustworthy. + +First is a single write gate, `AuditGate.admit`. Every bitmap change must pass through this gate. + +Second is a row-state automaton. It lists which of ten lifecycle states a row may hold. It also lists which state changes are legal. + +Third is a reference set of the nine verbs. A verb is one fixed action a row can undergo. Capture, mutate, and withdraw are three examples. Production storage kits check their own faster code against this reference. + +Two supporting pipelines round out the package. + +A Merkle hash pipeline computes deterministic content hashes. These hashes let the system prove a memory's content has not changed. The system proves this without reading the whole memory store again. + +A keyed-commitment pipeline proves that a piece of content once existed. It does this after the content has been destroyed. Destruction happens to satisfy a deletion request. + +The value types SubstrateLib works with live in a sibling package. That package is called SubstrateTypes. Its types include `Row`, `RowState`, `NounType`, `LatticeAnchor`, and `AuditEvent`. SubstrateLib imports these types. It adds the logic on top: the rules, the gate, and the verbs. + +## The Problem It Solves + +MOOTx01 estates can federate. Federation means separate devices exchange and merge memories. Merging works cleanly only when two devices agree on one thing. They must compute the same event identifier for the same logical event. + +SubstrateLib solves this with a deterministic content identifier. Each audit event gets a SHA-256 hash of its stable fields. That hash is folded into a UUID, instead of a random one. Two replicas that reach the same write independently produce the same identifier. + +This sameness matters for a grow-only set. A grow-only set, often called a G-Set, can gain entries but never lose them. Because the identifier is deterministic, a G-Set simply keeps one copy. A random identifier would defeat this design. The same logical event would look like two different events on two devices. Federation would then double-count it. + +A second problem is structural corruption. A row's status lives in three 64-bit integers. Each integer packs several unrelated fields at fixed bit positions. Nothing at the type level stops a careless write from causing harm. A careless write could touch the wrong bits. It could write a value wider than its field. It could move a row into a combination the design forbids. One example is marking a secret memory as exportable. + +SubstrateLib closes this gap with one gate that every write must pass. A write must land as a legal value in a declared field. It must produce a legal state and a legal combination of fields. Otherwise, the gate refuses it. Corruption of this kind becomes structurally impossible this way. Convention alone would only discourage it, never prevent it. + +A third problem is privacy against accountability. When a memory is expunged, MOOTx01 destroys its content for good. This satisfies a legal-compliance requirement. An audit trail that simply drops the event creates a puzzle, though. Nothing happening and something being hidden look the same from the outside. An absence proves nothing on its own. + +SubstrateLib's keyed-commitment pipeline resolves this problem. It commits, cryptographically, to the fact that a payload once existed. It uses a secret key the estate holds. It retains nothing the payload could be rebuilt from. + +The mathematics in this package is conformance-gated. A Rust port in `rust/` mirrors every algorithm. Shared test fixtures require both legs to produce identical output. A device running the Swift leg and a device running the Rust leg must agree. They must agree on every hash, every content identifier, and every gate decision. + +## How It Works + +The write gate and the verb reference sit side by side. Both build on the same row-state automaton. But they serve different callers. + +`AuditGate.admit` is the gate production kits call directly. A caller supplies the row's identity. A caller also supplies the prior snapshot of its bitmaps. A caller supplies only the field values it owns. + +The gate looks up each written field in the instance's admitted vocabulary. That vocabulary holds the substrate's universal fields: state, sensitivity, exportability, and trust. It also holds whatever fields the calling kit registered for itself. The gate rejects any field that is undeclared. It rejects any value that is out of range or not in the field's legal set. + +The gate then reads the prior bitmaps. It writes only the addressed bits. It leaves everything else untouched. This is a read-modify-write, never a blind overwrite. + +Before returning, the gate re-derives the encoded state. It checks that state against what the supplied verb was allowed to produce. It also checks the whole result against the automaton's forbidden-combination rules. Only then does it compute the deterministic content identifier. Only then does it emit one canonical audit event. + +The row-state automaton itself is a fixed transition table. For each pair of a state and a verb, either a next state is defined, or the transition is illegal. The automaton reports illegal transitions as such. + +A second check sits on top of the table: the forbidden-combination rules. A transition can be legal on its own. Yet it can still produce a field combination the design never allows. One example is a secret memory that is also exportable. Both checks together define what "legal state" means in this package. + +`Substrate` is the reference verb implementation. It offers a second, independent route to the same guarantees. It is an in-memory struct. It implements all nine verbs end to end. + +`Substrate` validates a mutation against the automaton directly. It applies the bitmap change. It updates two summary structures called matrices. These matrices track how many rows carry each field value. It appends an audit event. All of this happens inside one call. + +Production storage engines reimplement these same nine verbs against real storage. Examples include a SQLite tail and a memory-mapped bit-slice tensor. `Substrate` is the scalar oracle these engines are tested against. It is not itself the production path. + +The two supporting pipelines are simpler and pure. + +The Merkle hash pipeline builds a fixed byte encoding of a memory's content. It also encodes the memory's vector embeddings. It tags this encoding with a domain byte. That tag stops a leaf hash from being mistaken for an interior or tombstone hash. The pipeline runs the tagged bytes through SHA-256. + +The keyed-commitment pipeline reuses that same byte encoding. It applies a different domain tag. It runs the bytes through a keyed HMAC instead of a plain hash. This binds the commitment to a secret only the estate holds. + +Every one of these code paths can report activity to an injected telemetry sink. This includes the gate, the verbs, and both hash pipelines. But none of them ever reads a clock. None of them changes its output because monitoring is on. Timestamps arrive as a caller-supplied parameter. When the sink is disabled, the entire telemetry call collapses to one lock-free flag check. + +## How the Pieces Fit + +Figure 1 shows the library's topology. It shows the major parts and how a write moves through them. + +![Figure 1. Topology of SubstrateLib](topology.svg) + +*Figure 1. Topology of SubstrateLib. Two independent write paths exist: the production `AuditGate` and the reference `Substrate` verbs. Both consult the shared row-state automaton. A separate, unrelated pipeline computes content hashes and keyed commitments. These give the integrity and privacy guarantees. Both paths report to the telemetry sink. The sink is shown as a dashed external region because it lives in the sibling package IntellectusLib.* + +`AuditGate` and `Substrate` do not call one another. They are two consumers of the same rules. Neither feeds the other in a pipeline. A kit such as LocusKit calls `AuditGate.admit` directly on every bitmap write. `Substrate` exists for a different reason. Alternative or future storage engines need a known-correct in-memory reference to test against. + +Keeping the two paths independent, rather than layering one on the other, is deliberate. It guarantees that a bug in the fast path cannot hide behind a passing reference-path test. The two paths share only the rules. They do not share code. + +## What Ships in the Package + +The package ships seven Swift source files, listed above. It also ships a Rust port in `rust/`. The Rust port mirrors each Swift file one for one. + +The package ships no pinned data artifacts of its own. There is no bundled JSON and no trained model. SubstrateLib's guarantees are entirely computational. They rest on a fixed transition table, a fixed byte encoding, and a cryptographic hash. All of these are deterministic by construction. None depend on shipped reference data. + +Conformance fixtures live in `Tests/SubstrateLibConformanceTests/` and in `rust/tests/`. Both suites must pass before either leg changes ship. diff --git a/packages/libs/SubstrateLib/docs/topology.svg b/packages/libs/SubstrateLib/docs/topology.svg new file mode 100644 index 0000000..955ffab --- /dev/null +++ b/packages/libs/SubstrateLib/docs/topology.svg @@ -0,0 +1,99 @@ + + + + + + + + + + + + + SubstrateLib: two write paths, one automaton + + + + AuditGate + admit() : production write gate + + + Substrate + nine verbs : reference impl + + + + RowStateAutomaton + transitions + ForbiddenCombinations + + + + + consults + consults + + + + Content-integrity pipeline (independent of AuditGate and Substrate) + + + MerkleHash + leaf · interior · tombstone + + + KeyedCommitment + HMAC over shared leaf bytes + + + CommitmentAuditLog + G-Set, expunge provenance + + + shared encoding + + records + + + + Telemetry sink (external : IntellectusLib) + + + Intellectus.report + off-path: one atomic load, no alloc + + + + + + diff --git a/packages/libs/SubstrateML/docs/AGENT_MAP.md b/packages/libs/SubstrateML/docs/AGENT_MAP.md new file mode 100644 index 0000000..5856d6f --- /dev/null +++ b/packages/libs/SubstrateML/docs/AGENT_MAP.md @@ -0,0 +1,179 @@ +--- +doc: AGENT_MAP +package: SubstrateML +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateML/ActionOutcomeMatrix.swift + blob: 612ee840126c72dd0d07505147ba2991ac3bb0b9 + - path: Sources/SubstrateML/AnomalyDetection.swift + blob: ff7d227353ebfc69cf01b5c468b2612d307b8632 + - path: Sources/SubstrateML/AprioriMining.swift + blob: 189a4408f3c6e600a538a6cfc560d24688cbd2af + - path: Sources/SubstrateML/AssociationRuleMining.swift + blob: 9f20faa808883c065412face0063c7682eabbfd6 + - path: Sources/SubstrateML/AuditLogFold.swift + blob: 7423db7764c628cf5fd2aab0e9bb2d98d8c687bd + - path: Sources/SubstrateML/BradleyTerry.swift + blob: 63d335ee50ccee803f0508eb6f3135a29b6545db + - path: Sources/SubstrateML/CommunityDetection.swift + blob: bfba8e0255baeb66d4c79ccfaf34cd11e21a5ebd + - path: Sources/SubstrateML/CompositeDistance.swift + blob: a5f94c631d89c92d45b60e9da43be7fe6bdd19a2 + - path: Sources/SubstrateML/ConceptImplications.swift + blob: 1ac6ccbb70665532f135d2813f260cbbdad21716 + - path: Sources/SubstrateML/DeltaFeatureExtractor.swift + blob: 1ca396968e5e565096dbebb059b45b8fc3e5719f + - path: Sources/SubstrateML/DistillationPipeline.swift + blob: 82ce8bfaaa2e7a92077510ef239a6997a585c888 + - path: Sources/SubstrateML/DistillationScorer.swift + blob: 2cdc6dfe121080d432859d17487f2c8d15fd2910 + - path: Sources/SubstrateML/DPORReduction.swift + blob: 3d3c79e8ce7694b8308bb1c970e86a24de0446b0 + - path: Sources/SubstrateML/EigenvalueCentrality.swift + blob: f427430c34add289231520ba19ae4781316cbd13 + - path: Sources/SubstrateML/FeatureExtractors.swift + blob: 7e6d388a588ff7bec04be631948387a086a253ed + - path: Sources/SubstrateML/FFT.swift + blob: fdfaceeb902428149ead96107ed69c308bca0fff + - path: Sources/SubstrateML/FloatSimHash.swift + blob: 39cd2c4c36492c7cbaf55d6d22a2f32e273cfbae + - path: Sources/SubstrateML/FormalConceptAnalysis.swift + blob: 9cdf9613b30bf2e3a9a85088489c0e2b683a8a77 + - path: Sources/SubstrateML/InformationTheory.swift + blob: f7ba2a27904ceb19eef0fb5b06d1c5aad8b3894b + - path: Sources/SubstrateML/JacobiSVD.swift + blob: fcd924ff0a8409f224b56e7229f59659a6b5f51e + - path: Sources/SubstrateML/LatticeDistance.swift + blob: a7b96a376dfba0c815e4dc91962da1ccaeb2ac1d + - path: Sources/SubstrateML/LLMCalibrationCurve.swift + blob: 4e01882a65570ccce72e2789c23e95c46b6399a7 + - path: Sources/SubstrateML/MatrixDecay.swift + blob: b9595064a54b79f8ddee343385af4b8ebdc7f3e5 + - path: Sources/SubstrateML/MomentSummary.swift + blob: e8c07841c8c3440deb0375500ebcd35a6f116bb4 + - path: Sources/SubstrateML/NMFAlternatingLeastSquares.swift + blob: d64aadc8db165cc3b9f51b4fb1c644a6d6a4ca3c + - path: Sources/SubstrateML/NMFDoubleFrobeniusSquared.swift + blob: 0a725d72b70120828cb5911de594b0278ca450e5 + - path: Sources/SubstrateML/PairingHandshake.swift + blob: cb608882478ec57a6e6ba1f7728646ff3ea610f1 + - path: Sources/SubstrateML/PartialStateRecall.swift + blob: ad7b42bc25f9c4f283d75593d2c42cd546ccf074 + - path: Sources/SubstrateML/RandomWalks.swift + blob: 99f40b1685a876d81f84ca38b4857e5c6ba4e1dd + - path: Sources/SubstrateML/RowAttributeView.swift + blob: f63cf544a87adde435bede527531d1a9414dad03 + - path: Sources/SubstrateML/Sampling.swift + blob: 84fcdcade559229003688a7fc151f5c0a6dac6e6 + - path: Sources/SubstrateML/ShingleSimilarity.swift + blob: bf9dbaa5b0d5a5ee4ec5dfe42a5d65c85bdd0326 + - path: Sources/SubstrateML/TemporalCausalityFold.swift + blob: d99d7358893eeae0a442e0bd4e52f8a9b1cd7f5e + - path: Sources/SubstrateML/TemporalCompression.swift + blob: f34dde022faadd40058e35b62da12c8985763e59 + - path: Sources/SubstrateML/TierAscendingQuery.swift + blob: 8681cbdae9af7facfe7792392dcee15c8f8fe0aa + - path: Sources/SubstrateML/TierContributionFingerprint.swift + blob: bb56b7a82532f8c324168bd81edff6bf68fb33e9 + - path: Sources/SubstrateML/TypedDecayWeighting.swift + blob: 82412f392848c378b5b28221bbddade9f03bf1bf + - path: Sources/SubstrateML/VizGraphSignals.swift + blob: 4f90110f2a19cb92f3d45baf3bc4d82ac9c3dc65 +--- + +# AGENT_MAP: SubstrateML + +PURPOSE: layer-3 cold-path/dreaming algorithm library of the MOOTx01 substrate: learning (decay, Bradley-Terry, calibration, NMF), graph analytics (Louvain, centrality, walks, SVD, FFT, anomaly), pattern mining (association rules, Apriori, FCA, D-G implications, temporal causality), distillation (5-stage cluster→factoid), fingerprint/distance math (FloatSimHash, composite/lattice/partial distances, moment/temporal summaries), and federation privacy math (pairing, tier contribution, tier query, DP OR-reduce). Pure functions + value types only; no storage, no clocks, no hidden state. + +DEPS: imports SubstrateTypes (Fingerprint256, HLC, RowId, MatrixO/F/C, SplitMix64, RecallResult…), SubstrateKernel (PortableKernel dispatch: float SimHash projection, orReduce256), IntellectusLib (telemetry; off-path = one atomic-bool load), Foundation. Imported by (per Package.swift/README): LocusKit, CognitionKit, GeniusLocusKit, NeuronKit, dreaming-daemon paths; LatticeLib (moot-semantics) uses EigenvalueCentrality for LexRank; CorpusKit LsaProvider uses JacobiSVD. Within this repo only tests + SubstrateLib's temporary `@_exported` re-export reference it (four-package split mid-migration). Rust port `rust/` = crate `substrate-ml` v1.0.0-skeleton, 38 modules 1:1 with Swift files; conformance tests rust/tests/{distillation_conformance,float_simhash_kernel_equivalence,viz_graph_signals_tests}.rs + shared JSON vectors in the engineering test harness. + +ENTRY POINTS (no single facade; per-family): +- DistillationPipeline.swift:277 `DistillationPipeline.run(input:extractFeatures:intraItem:) -> DistillationOutput`: full 5-stage distillation +- CommunityDetection.swift:152 `CommunityDetection.detectFull(adjacency:maxLevels:maxPasses:resolution:estate:ts:) -> [Int]`: multi-level Louvain +- EigenvalueCentrality.swift:87 `EigenvalueCentrality.compute(adjacency:maxIterations:tolerance:estate:ts:) -> [Double]`: keystone scores +- NMFAlternatingLeastSquares.swift:81 `factorize(V:rank:...) -> NMFFactorization`: latent themes +- FloatSimHash.swift:42 `FloatSimHash.project(vector:seed:) -> Fingerprint256`: embedding → fingerprint +- CompositeDistance.swift:50 `CompositeDistance.distance(latticeDistance:fingerprintHammingDistance:...)`: primary recall metric +- AuditLogFold.swift:101 `AuditLogFold.projectAll(events:asOf:nounTypeFor:)`: CRDT state reconstruction +- AprioriMining.swift:167 `AprioriMining.mine(rows:thresholds:) -> [AprioriRule]`: multi-antecedent rules + +## Symbol Table + +### Telemetry names: VizGraphSignals.swift +- :65 `enum VizGraphSignals`: canonical metric-name constants; one authoritative file prevents orphaned metrics +- :69 `communityAssignment = "community.assignment"` · :73 `centralityScore = "centrality.score"` · :77 `nmfFactor = "nmf.factor"` · :82 `anomalyFlag = "anomaly.flag"` · :86 `edgeDecayedWeight = "edge.decayed_weight"` + +### Fingerprint + distance math +- FloatSimHash.swift:30 `enum FloatSimHash`; :42 `project(vector:seed:) -> Fingerprint256`: 256-hyperplane signed projection; empty vector → zero fp; :76 `planes(seed:dim:) -> FloatSimHashPlanes`: deterministic ±1 plane set, cacheable per (seed,dim); :61 private cached kernel (`PortableKernel.kernelForCurrentPlatform()`, selected once) +- CompositeDistance.swift:37 `defaultAlphaLattice = 0.5` / :38 `defaultAlphaFingerprint = 0.5` / :40 `fingerprintTotalBits = 256`; :50 `distance(latticeDistance:fingerprintHammingDistance:alphaLattice:alphaFingerprint:compatibleSeedScope:) -> Double`: preconditions (not clamps); scope-incompatible drops fp term WITHOUT renormalizing (intended) +- LatticeDistance.swift:45 `struct LatticeAnchorStr` (udc + qid; 0 = null); :59 `enum UDCTreeDistance` (:63 `longestCommonPrefixLength`, :83 `distance` clamped [0,1]); :104 `protocol WikidataAdjacencyProvider`; :110 `enum WikidataGraphDistance` (:112 `maxDepth = 4`, :113 `normalizationScale = 3.0`, :118 `shortestPathLength` bounded BFS, :144 `distance` = 1-exp(-len/3), unreachable/null → 1.0); :161 `enum LatticeDistance` (:163/:164 default alphas 0.5, :185 combined `distance(_:_:provider:alphaUDC:alphaQID:)`, :172/:180 legacy hashed-anchor overloads: 0/1 only, compatibility) +- PartialStateRecall.swift:58 `enum PartialStateRecall`; :75 `score(rowFingerprint:anchor:matchBlocks:differBlocks:) -> Double`: matchScore × differScore, blocks ∈ {0,1,2,3} precondition-enforced, empty set → 0; :104 `topK(anchor:rows:matchBlocks:differBlocks:k:)`: linear scan reference; :128 `hammingBlocks(_:_:blocks:)`: block-restricted Hamming +- ShingleSimilarity.swift:53 `enum ShingleSimilarity`; :58 `windowSize = 3`; :69 `shingles(_:) -> Set`: lowercase 3-char window; <3 chars → whole-string shingle; empty → empty set; :96 `similarity(_:_:) -> Float32`: Jaccard; both-empty → 0.0 +- MomentSummary.swift:54 `struct RowLite` (fingerprint + captureHLC, 40 bytes); :70 `summarize(rows:[Row]...)` / :89 `summarize(rows:[RowLite]...)`: OR-reduce of rows matching caller predicate; byte-identical, RowLite ~4-5× faster (conformance CRC 0x6762440b); :100 `capturedDuring(_:_:)` convenience predicate; :106 `orReduce(_:)` raw reduction +- TemporalCompression.swift:30 `enum WindowLevel` hour→year (:43 `nextCoarser`, :48 `approxSeconds`: deliberately approximate bucket divisors); :60 `struct TemporalWindow` (:80 `empty(level:)` identity); :93 `compress(rows:startHLC:endHLC:level:)`; :110 `rollup(windows:to:)`: OR fp union, `&+` counts, min/max bounds; :135 `cascadeRollup(hourWindows:upTo:)`: epoch-seconds integer-division bucketing per level + +### Ingestion shaping +- FeatureExtractors.swift:41 `enum StreamSourceFlag: UInt8`; :50 `struct AmbientSampleRow`; five sample+extractor pairs, each `extract(_:hlc:rowId:) -> AmbientSampleRow`: :82/:102/:109 HealthKit, :136/:158/:165 CoreLocation (+ :190 `quantizeGeohash` internal), :200/:223/:230 EventKit (attendees sorted before hashing), :257/:279/:286 ScreenTime, :308/:333/:340 SystemTelemetry; :363 internal `encodePayload` key-sorted deterministic binary. Pinned UDC anchors: 613.71 / 914 / 65.012.4 / 004.5 / 004.2. Only HealthKit rows carry payload +- AuditLogFold.swift:44 `struct ProjectedRowState` (no fingerprint: recompute from bitmaps+anchor); :79 `projectCurrentState(rowId:nounType:events:)`; :90 `projectStateAt(...asOf:)`; :101 `projectAll(events:asOf:nounTypeFor:)`; :131 private `foldOrdered`: HLC-sorted sequential fold; tombstone stateRaw == 33 (adjectiveBitmap & 0x3F) is STICKY (I-22); fold is permutation-invariant over the event set (I-21 sync convergence) +- RowAttributeView.swift:51 `enum RowAuditValue` (.bitmap/.integer/.null); :70 `struct RowAuditEntry`; :111 `struct RowAttributeView` (:130 init sorts attributes, :140/:151 manual Eq/Hash); :189 `from(auditEntries:) -> [RowAttributeView]`: vocab sorted+capped 64 (6-bit field), latest-write-wins per fieldPath by HLC, bitmap → one attr per set bit, integer → low byte, null → nothing, empty rows dropped; vocab is EPHEMERAL per call (pre-merge batches for stable indices) + +### Learning + decay +- MatrixDecay.swift:39 `struct DecayingMatrix` (values + halfLifeSeconds + lastDecayTimeSeconds; :60 subscript); :84 `MatrixDecay.apply(to:nowSeconds:estate:ts:)`: factor = exp(-dt·ln2/τ), dt ≤ 0 no-op but still emits factor 1.0; saturating Int64 cast for telemetry tag; :148 `decayFactor(elapsedSeconds:halfLifeSeconds:)` pure; :157 `decayAndAdd(...)` decay-then-add; :177/:183/:189 `applyExponentialDecay(to: MatrixF/O/C ...)`: intentional NO-OP reference adapters (real logic in production kit); :199 `Double.ln2 = 0.6931471805599453` (local, matches Rust f64::LN_2); :208 `enum DecayHalfLives`: F 90d, C 180d, O 60d, T 30d, ActionOutcomes 365d, calibration 730d, W_ranking 90d (illustrative; manifest is authoritative) +- ActionOutcomeMatrix.swift:37 `struct ActionOutcomeKey`: 6-bit preconditions (o07/o08), :51 `packed: UInt16`, :55 Comparable; :60 `struct ActionOutcomeCell` (:75 `successRate`, :83 `wilsonLowerBound` 95%, z=1.96); :95 `struct ActionOutcomeMatrix` (:101 `observe(action:outcome:success:at:)` `&+=`, :113 `successRate`, :119 `observationCount`, :139 `topActions(forOutcome:k:minObservations:)`: ranked by Wilson LB, ties count desc then action asc, returns rate+LB+count together, :158 `populatedCellCount`) +- BradleyTerry.swift:31 `struct PreferenceObservation` (winnerID/losers/weight); :53 `struct BradleyTerryEstimator` (`theta: [UUID: Double]` private(set)); :67 `init(learningRate: 0.05, l2: 0.001, theta: [:])`; :78 `observe(_:)`: SGD in log-space, multi-loser SEQUENTIAL against running theta (pinned parity contract); :105 `observeBatch(_:)`; :113 `strength(of:)` = exp(theta); :120 `probability(_:beats:)` = sigmoid(Δtheta). No RNG; warm-restartable; ranking signal only, never truth +- LLMCalibrationCurve.swift:37 `struct LLMCalibrationCurve`; :38 `binCount = 20` / :39 `binWidth = 0.05`; :50 `observe(claimedConfidence:actualOutcome:)`: clamp [0, 0.999999], wrapping counters; :62 `actualRate(in:)`; :72 `midpoint(of:)`; :78 `expectedCalibrationError()`; :92 `brierScore()`; :107 `decay(factor:)` fractional (730d half-life policy, daemon-driven) +- Sampling.swift:57 `enum Sampling`; :75 `sampleNormal(rng: inout SplitMix64)`: Box-Muller COSINE branch only, exactly 2 draws/call, u1 floored at leastNormalMagnitude; :102 `sampleGamma(shape:rng:)`: Marsaglia-Tsang (squeeze 0.0331), shape<1 via Ahrens-Dieter with extra draw BEFORE recursion (pinned order); :157 `sampleBeta(alpha:beta:rng:)`: g1/(g1+g2), alpha-Gamma drawn first, 0.5 degenerate fallback. Conformance vector sampling.json; relies on libm bit-identity + +### Graph analytics +- CommunityDetection.swift:80 `typealias Adjacency = [[(neighbor: Int, weight: Double)]]`; :100 `detect(adjacency:maxPasses:estate:ts:)`: Phase 1 only; :152 `detectFull(adjacency:maxLevels:maxPasses:resolution:estate:ts:)`: full Louvain; resolution gamma default 1.0 = classical (IEEE 1.0*x == x, legacy-safe), NeuronKit callers pass 0.05 (pair-lock escape); :368 `canonicalize(_:)`: first-appearance renumbering; private: :202 totalEdgeWeight (zero threshold 1.0e-30 → identity labels, NO emit), :214 detectCore (gain ties → lowest label asc), :317 condense, :345 emit. Defaults maxPasses 10, maxLevels 10; exactly ONE `community.assignment` emit per call +- EigenvalueCentrality.swift:60 `typealias Adjacency`; :62 `defaultMaxIterations = 100` / :63 `defaultTolerance = 1e-6`; :87 `compute(...) -> [Double]`: AUTHORITY via Aᵀx (`xNext[j] += w*x[i]` for edge i→j; design-council 2026-06-04, conformance-locked; symmetrize for undirected), Perron shift 1.0 per iteration (breaks bipartite ±λ oscillation), zero-norm (<1e-30) → uniform 1/√n fallback, all 3 return paths emit `centrality.score` (value 1.0) +- RandomWalks.swift:36 `typealias Adjacency`; :38 `defaultRestartProb = 0.15`; :43 `walk(adjacency:start:length:restartProb:seed:) -> [Int]`: hard preconditions on kernel validity (indices, finite non-negative weights); walk[0] == start; dead end = implicit restart; :99 `sampleWeighted(_:rng:)`: roulette wheel; total ≤ 0 → uniform; rounding shortfall → last neighbor; :127 `uniform01(_:)`: top 53 bits, Rust bit-match; :141 `walkWithRestart(seed:steps:restartProbability:rngSeed:adjacency:) -> [RowId: Int]`: RowId-space uniform-choice visit counts (recall_exploratory); :172 `struct SplitMix64` (public; canonical constants 0x9E3779B97F4A7C15 / 0xBF58476D1CE4E5B9 / 0x94D049BB133111EB) +- NMFAlternatingLeastSquares.swift:45 `struct NMFFactorization` (W m×k, H k×n, rank, iterations, finalError RMS); :81 `factorize(V:rank:maxIterations:100:tolerance:1e-4:seed:0xDEADBEEFCAFEBABE:estate:ts:)`: Lee-Seung multiplicative updates, eps 1e-9, init = SplitMix64 16-bit draws /0xFFFF, preconditions rectangular/finite/non-negative; flat-buffer unsafe-pointer hot loops (loop nest + reduction order preserved EXACTLY; conformance CRC 0x300bf633; ~100× perf vs bounds-checked); emits `nmf.factor` = finalError; :220 `reconstructionError(V:W:H:)` public RMS; internal flat/nested matMul helpers :247/:288/:309/:329/:353/:370/:388 +- NMFDoubleFrobeniusSquared.swift:54 `struct NMFDoubleFrobeniusSquaredFactorization` (:93 `loadings(forRow:)`); :113 `enum NMFDoubleFrobeniusSquared`: :116 `defaultMaxIterations = 100`, :122 `defaultTolerance = 1e-6`, eps 1e-9; :153 `factorize(o:rows:cols:rank:seed:0xC0FFEE_BABE_BEEF:...)`: f64 scalar, RAW Frobenius² convergence (not RMS), init floored at 1e-3; :302 `struct NMFDoubleFrobeniusSquaredRNG` (SplitMix64 core, 53-bit output, floor 1e-3). PRODUCTION GATE: no consumer may wire to this pending substrate_math_performance benchmark (ruling 2026-06-13); still conformance-gated vs Rust (nmf_double_frobenius_squared.json) +- AnomalyDetection.swift:26 `enum AnomalyDetection`; :31 `zScore(value:mean:stddev:)` (stddev ≤ 0 → 0); :51 `rollingZScore(window:current:estate:ts:)`: emits `anomaly.flag`; :90 `modifiedZScore(value:median:mad:)`: 0.6745 consistency constant, mad ≤ 0 → 0; :110 `rollingModifiedZScore(...)`: IN-PLACE sort (5× perf, ~400KB saved at 100k; CRC 0x6c6fda4d; must match Rust sort_by partial_cmp for non-NaN); :152 `isAnomalous(zScore:threshold: 3.0)` +- JacobiSVD.swift:74 `struct SVDResult` (U, singularValues non-increasing, Vt, rank); :129 `decompose(A:rank:sweeps:) -> SVDResult`: one-sided cyclic Jacobi; sweeps default 30 PINNED (fixed count, not convergence-tested; MUST match across ports); eps 1e-9 skip/zero cutoff; scalar Float32 only, NO SIMD/FMA/Accelerate; sign convention: largest-|entry| of each left vector forced positive; preconditions m ≥ n, rectangular, non-empty; unsafe-buffer sweep loops (perf, indices loop-derived); :368 `jacobiCS(alpha:beta:gamma:)`: pinned expression tree (ζ,t,c,s), DO NOT refactor +- FFT.swift:36 `struct Complex` (:51 `magnitude`, :58 `magnitudeSquared`, :63/:68/:73 +,-,*); :83 `enum FFT`: :99 `forward(real:) -> [Complex]` Cooley-Tukey radix-2 DIT, length MUST be power of two (constitutional; callers zero-pad), :162 `magnitudeSpectrum(real:)`; :169/:180 `bitReverse`/`log2Floor` (@usableFromInline); :201 `struct RhythmResult` (dominantPeriodSeconds nil = "no rhythm", honest); :218 `enum RhythmAnalysis`: :229 `analyze(series:bucketDurationSeconds:)` scans bins 1...N/2, energy excludes DC; :281 `analyze(fingerprints:block:bitPosition:bucketDurationSeconds:)`: bit-series extraction, block 0..3 / bit 0..63 preconditions. Production vDSP path must be BIT-IDENTICAL to this scalar reference +- InformationTheory.swift:38 `entropy(_:)`; :48 `mutualInformation(joint:)`; :75 `klDivergence(_:_:)`: length precondition; skips p>0,q=0 terms ⇒ result is a LOWER BOUND; :88 `crossEntropy(_:_:)`; :99 `jensenShannon(_:_:)` bounded [0,1]; :112 `normalizedMutualInformation(joint:)`: ragged matrix → 0 sentinel. All log2 (bits); 0·log0 = 0 by term-skipping; distribution validity is caller's contract + +### Pattern mining +- AssociationRuleMining.swift:52 `struct Item` (field/value UInt8; :64 `packed: UInt16` = field<<8|value; :68 Comparable); :76 `struct AssociationRule` (single-Item antecedent + 5 metrics); :107 `struct MiningThresholds` (minSupport/minConfidence); :136 `mineAssociationRules(matrix: MatrixO, activeRowCount:thresholds:)`: N injected (matrix has no row count); activeRowCount ≤ 0 → []; :154/:164 internal `AssociationRuleEngine.mine`: two passes: diagonal = single supports, off-diagonal in canonical packed-key order (NO final sort needed: conformance-relied ordering); self-rules skipped; conviction = +inf at confidence 1; k>2 → AprioriMining +- AprioriMining.swift:55 `struct AprioriThresholds` (minSupport/minConfidence/minLift ≥ 1.0 default/maxK default 3, floored ≥ 2); :89 custom `init(from:)` routes through public init so serialized maxK < 2 cannot bypass the clamp; :115 `struct AprioriRule` (antecedent `[Item]` sorted by packed, + evidenceCount); :167 `AprioriMining.mine(rows:thresholds:)`: join on (k-2)-prefix, subset-count, prune; rule extraction iterates itemsets in lex order (dict order undefined); 4-key sort lift↓ confidence↓ evidence↓ lex↑; :310 `mineAprioriRules(rows:thresholds:)` free-function wrapper; private `HashableItemset`/`itemsetLess`/`aprioriJoin` (:325/:340/:366) +- FormalConceptAnalysis.swift:41 `struct FormalAttribute: Comparable`: lex (namespace,key,value) order = determinism backbone; :70 `enum SeedMode` .single/.multi; :90 `struct CoverDelta` / :117 `struct ConceptCoverDeltas` (:140 `covering(concepts:)`: cover edges ≠ implications; O(n²m) bounded); :210 `struct FormalConcept` (extent/intent/support/stability?); :247 `struct FormalContext`: :251 nested `RowID = UInt32` (avoids LocusKit RowID collision), :285 `from(rowAttributeViews:)` (namespace "row"), :298 `init(rows:)`, :334 `extent(of:)`, :343 `intent(of:)`, :354 `closure(of:)`; internal FCABitSet plumbing :366/:380/:386/:665; :415 `struct BoundedConceptMiner` (minSupport clamped ≥1, maxIntentSize, maxConcepts, seedMode, maxSeeds, stabilityBudget 0 default → stability nil, stabilitySeed 0xCAFEBABEDEADBEEF): :475 `mine(context:)`: one closure per seed, dedup by intent, truncate; sort support↓ intent-size↑ lex; :595 `enum StabilityEstimator` (:616 `estimate(concept:context:budget:seed:)`: Bernoulli-half subsets, per-concept SplitMix64 seed = caller seed XOR FNV(canonical key)) +- ConceptImplications.swift:62 `struct Implication` (premise/conclusion; caller ensures disjoint); :96 `struct ConceptImplications` (implications + isTruncated); :148 `conceptImplications(over:context:maxImplications:maxPremiseSize:)`: Next-Closure pseudo-intent enumeration in increasing size order; maxImplications hard cap sets isTruncated (with look-ahead :358 for honesty); maxPremiseSize skips silently (soundness unaffected); output sort premise-size → lex premise → lex conclusion; empty context → empty untruncated +- TemporalCausalityFold.swift:61 `struct TemporalFieldCoord` (fieldPath + valueRepr stable string); :98 `struct TemporalAuditEntry` (hlc + fieldCoords); :121 `struct FoldResult` (deltas in first-insertion order + newWatermark); :139 `struct TemporalCausalityKey` (source/target/lagBucket); :174 `lagBuckets = [1,2,4,8,16,32,64,128]`: must mirror MatrixTier.lagBuckets exactly; :178 `defaultWindowMinutes = 256`; :191 `maxWindowOccupancy = 512`: bulk-import quadratic guard (2026-07-02 decision), keeps most-recent in-window sources; :202 `lagBucket(forMinutes:)`: smallest boundary ≥ delta, clamp 128; zero-ms → minute 1; :241 `fold(entries:windowMinutes:startWatermark:)`: entries MUST be pre-sorted by HLC (not re-sorted; silent corruption otherwise); hourly batch cadence supersedes weekly + +### Distillation +- TypedDecayWeighting.swift:32 `enum DistillationFeatureType: String`: ENT/REL/TMP/NUM; :40 `decayLambda` = 0.1/0.2/0.5/0.8 (pinned, diffusion-schedule analogy); :52 `enum TypedDecayWeighting`; :65 `weight(featureType:ageInUnits:)` = exp(-λ·max(0,age)); :86 `weightedDocFrequency(featureType:presenceTimestamps:allMemoryTimestamps:referenceDate:timeUnit: 86_400)`: decayed presence weight / total weight; empty or zero-total → 0 +- DeltaFeatureExtractor.swift:29 `enum DeltaType: String` STATIC/CONVERGENT/MONOTONE/OSCILLATING/DIVERGENT; :41 `struct DeltaAnalysis` (deltaType/terminalValue/convergenceScore/slope?/confidence); :86 `analyzeCategorical(sequence:decayLambda: 0.5)`: oscillation = period-2 over LAST 4 obs only (≥4 required); convergence = trailing-run fraction vs λ; :148 `analyzeNumerical(sequence:decayLambda: 0.8)`: diffs all-zero STATIC / one-sign MONOTONE (confidence = λ fixed) / strict alternation OSCILLATING / else DIVERGENT. Pure; zero RNG +- DistillationScorer.swift:16 `struct ExtractedFeature` (:41 init, display defaults to value); :58 `struct DistillationSNR` (readyToDistill iff snr ≥ 2.0); :71 `struct FeatureGraph` (components sorted by weighted df ↓: dominant first); :94 internal `binaryEntropy(_:)`; :114 `structuralThreshold(M:)` = 2/M (M=1 → 2.0, correctly blocks); :120 `applyStructuralThreshold(features:M:)`; :145 `computeSNR(features:M:)`: episodic floor 1e-6; :174 `computeStructuralScores(features: inout)` σ = df·(1−H(df)); :194 `buildPMIGraph(thresholdFeatures:incidenceMatrix:M:)`: PMI log2 ONLY (Foundation.log2 ⇔ f32::log2 bit-match), positive-PMI edges, DFS components O(n²) ok ≤ ~150; :257 `selectDominantComponent(graph:)`; :277 `computeConfidence(selected:allThreshold:)` = meanDf × coherence ratio +- DistillationPipeline.swift:27 `struct DistillationInput` (memoryContents/timestamps?/clusterID/sourceIDs; computed M); :53 `struct DistillationOutput` (drawerContent/confidence/uncertain/snr/deltaType/succeeded/failureReason/featureFingerprint); :81 `struct DistilledHeader` (:102 `parse(_:)`: nil unless "[DIST|" prefix); :172 `typealias FeatureExtractor = @Sendable (String, DistillationFeatureType) -> [ExtractedFeature]`; :184 `featureSimHashSeed = 0x44495354494C4C41` ("DISTILLA"): conformance-critical, changing invalidates ALL stored fingerprints; :193 `featureHash(_:) -> Fingerprint256`; :221 `queryFingerprint(query:extractFeatures:)`: probe fp, no inference; :240 `defaultExtractor` (capitalization heuristic; tests/default); :277 `run(input:extractFeatures:intraItem: false)`: 5 stages; intraItem skips SNR gate + keeps ALL passing features (no PMI pruning of a single doc); ubiquity re-add at df ≥ (M-1)/M (zero-PMI spine rescue); confidence < 0.4 fail, [0.4,0.7) uncertain; `src=` = sourceIDs.count NOT M; deliberate monolithic body + +### Federation + privacy +- PairingHandshake.swift:37 `struct PairingNonce`: exactly 32 bytes (precondition); :48 `seedWith(estateA:estateB:)`: UUIDs ordered by RAW BYTES, never uuidString (ASCII hex order ≠ byte order; Rust compares [u8;16]: string order would derive incompatible families); :79 `struct PairingRecord` (:87 `isActive`); :114 `struct PairingAuditPayload`; :133 `enum PairingHandshake`: :145 `generateSharedFamily(nonce:estateA:estateB:density:)` (both sides derive identical family, no round trip), :156 `sharedFamilyKey(case:peerEstate:)` = "H_shared__", :168 `buildPairEvent(...)`, :179 `buildUnpairEvent(...)` (unpair RETAINS family for asOf queries); :192 private `combinedFamilyHash`; :209 fileprivate `lexLessOrEqual` raw-byte compare. FNV-1a 0xCBF29CE484222325 / 0x100000001B3 mixers +- TierContributionFingerprint.swift:45 `enum FederationCase: UInt32` household 1 / fleet 2 / industry 3; :51 `struct TierContribution`; :81 `build(estateUUID:case:shareableFingerprints:hlc:)`: kernel-dispatched orReduce256; :95 `encode(_:) -> Data`: 64-byte canonical wire: UUID(16) | case u32 BE | rowCount u32 BE | 4× fp block u64 BE | hlc.packed u64 BE: BE-UNIFORM (earlier LE-fp draft diverged from Rust by 32 bytes; conformance caught it); :117 `decode(_:)`: nil unless exactly 64 bytes + known case; :148/:154/:160/:166 endian helpers. No signing/checksum at this layer (egress signs; DP at aggregator) +- TierAscendingQuery.swift:35 `enum TargetTier` .peer/.fleetAggregate/.industryAggregate; :41 `struct TierAscendingQuery` (originatingEstate/primitiveName/primitiveInput/targetTier/privacyBudget/queryHLC); :61 `struct PeerResponse`; :68 `enum TierAscendingQueryProtocol`: :72 `computeLocal(query:dispatch:)` (injected closure; no CognitionKit dep), :80 `applyDPToContribution(_:budget:rngSeed:)`: Laplace scale 1/ε per score, CI = ±1.96·scale; NOTE Swift forwards `breakdown` unchanged while Rust zeroes it (side-channel; parity nuance), :100 `combine(local:peers:)`: per-RowId score sum, sort score↓ RowId↑, WIDEST CI wins (conservative); :133 `struct PrivacyLedger` (:141 `remaining`, :147 `canConsume`, :152 `consume`, :159 `dailyReset`: manual, no auto-expiry) +- DPORReduction.swift:31 `struct DPParameters` (:36 init: ε 1.0 > 0, δ 1e-9 ∈ [0,1) VALIDATED BUT UNUSED by Laplace (documented), k 3 ≥ 1); :48 `enum DPORReduction`; :54 `reduce(fingerprints:params:rngSeed:) -> Fingerprint256`: per-bit popcount + Laplace(1/ε) noise + k-anonymity threshold; empty → .zero; fresh SplitMix64(rngSeed) inside; :85 internal `laplaceNoise(scale:rng:)`: inverse CDF from top 53 bits (raw >> 11) + +## INVARIANTS / GOTCHAS + +- DETERMINISM IS THE CONTRACT. Swift and Rust legs must agree bit-for-bit on shared conformance vectors. Any change to seeds, draw orders, tie-breaks, loop-nest/reduction order, or pinned constants must be mirrored in `rust/src/` and pass both test suites. SplitMix64 is the only PRNG; all uniform doubles come from the top 53 bits. +- NO CLOCKS, NO STATE, NO I/O anywhere in the package. `ts`, `nowSeconds`, `referenceDate`, watermarks: all caller-supplied. Everything is Sendable value types or pure static functions. +- Pinned seeds: canonical conformance seed 0xCAFEBABEDEADBEEF (FCA stability, temporal-fold vectors); NMF default 0xDEADBEEFCAFEBABE; parked-NMF 0xC0FFEE_BABE_BEEF (historical, deliberate); distillation featureSimHashSeed 0x44495354494C4C41: changing it orphans every stored distillation fingerprint. +- Pinned constants: do not change without conformance regen: JacobiSVD sweeps 30; FFT power-of-two input; Louvain zero-weight 1e-30 + gamma 1.0 legacy identity; centrality Perron shift 1.0 + tolerance 1e-6 + cap 100; restartProb 0.15; Marsaglia-Tsang 0.0331; anomaly threshold 3.0 + MAD 0.6745 (CRC 0x6c6fda4d); shingle window 3; lag buckets {1..128} + window 256 min + occupancy 512; SNR gate 2.0 + τ 2/M + confidence 0.4/0.7 + ubiquity (M-1)/M; decay λ ENT/REL/TMP/NUM 0.1/0.2/0.5/0.8; half-life table (90/180/60/30/365/730/90 d); calibration 20 × 0.05 bins; Wilson z 1.96; DP defaults ε 1.0, δ 1e-9, k 3; CompositeDistance 256 bits, alphas 0.5/0.5; UDC clamp, BFS depth 4, scale 3.0; NMF eps 1e-9 + tolerance 1e-4 (CRC 0x300bf633); MomentSummary CRC 0x6762440b. +- NMFDoubleFrobeniusSquared is PRODUCTION-GATED: no consumer may wire to it until the substrate_math_performance benchmark passes. Do not "clean it up" into use; do not delete it either (it is the benchmark baseline). +- EigenvalueCentrality is AUTHORITY (Aᵀx), locked by conformance vector. Do not flip to hub semantics. Symmetrize adjacency for undirected use. +- MatrixDecay is constitutional: decay only shrinks; the three typed applyExponentialDecay overloads are intentional no-ops at reference level. dt ≤ 0 is a no-op that still emits factor 1.0. +- AuditLogFold tombstone (stateRaw 33) is sticky forever; fold is event-set permutation-invariant: that IS the sync-convergence proof. Fingerprint deliberately absent from projections. +- TemporalCausalityFold requires pre-sorted input by HLC; violation silently corrupts (no crash). RowAttributeView vocab (≤64 fields) is per-call ephemeral: merge batches first for stable indices. +- PairingHandshake seed ordering compares raw UUID bytes, never uuidString: ASCII hex order can invert byte order and desync from Rust. Nonce exactly 32 bytes. +- TierContribution wire format is BE-uniform 64 bytes (fingerprint blocks BE, not the LE storage form). Unsigned at this layer by design. +- TierAscendingQuery parity nuance: Rust zeroes DistanceBreakdown in DP contributions; Swift forwards it unchanged. Flag before relying on either behavior. +- CompositeDistance with incompatible seed scope DROPS the fingerprint term without renormalizing: smaller distance is intended, not a bug. Preconditions, not clamps, on inputs. +- InformationTheory.klDivergence is a LOWER BOUND when q has zeros where p has mass. NMI returns 0 sentinel on ragged input. All log2. +- Apriori maxK floor 2 is enforced in init AND in custom Decodable (synthesized decode would bypass it). Association mining needs caller-injected N; matrix diagonal doubles as single-item support. +- Hard preconditions (crash, not error) on malformed inputs throughout: RandomWalks kernel validity, NMF domain (rectangular/finite/non-negative), SVD shape (m ≥ n), FFT length, block IDs {0..3}, 6-bit action/outcome keys, 32-byte nonce. +- Unsafe-buffer hot loops (NMF, SVD, moment/anomaly paths) exist for auto-vectorization; all indices are loop-derived, never data-derived. Preserve loop nest and reduction order exactly: Float32 lowest-bit rounding is conformance-visible. +- Telemetry: exactly five signals (VizGraphSignals); one emit per invocation; off-path = single atomic-bool load. Degenerate zero-weight Louvain emits nothing. Only AnomalyDetection/CommunityDetection/EigenvalueCentrality/NMF/MatrixDecay emit. +- Distillation `src=` field counts sourceIDs, not cluster members (differs in intra-item mode). intraItem skips the SNR gate and PMI pruning by design. diff --git a/packages/libs/SubstrateML/docs/DETAILS.md b/packages/libs/SubstrateML/docs/DETAILS.md new file mode 100644 index 0000000..c59db28 --- /dev/null +++ b/packages/libs/SubstrateML/docs/DETAILS.md @@ -0,0 +1,1238 @@ +--- +doc: DETAILS +package: SubstrateML +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateML/ActionOutcomeMatrix.swift + blob: 612ee840126c72dd0d07505147ba2991ac3bb0b9 + - path: Sources/SubstrateML/AnomalyDetection.swift + blob: ff7d227353ebfc69cf01b5c468b2612d307b8632 + - path: Sources/SubstrateML/AprioriMining.swift + blob: 189a4408f3c6e600a538a6cfc560d24688cbd2af + - path: Sources/SubstrateML/AssociationRuleMining.swift + blob: 9f20faa808883c065412face0063c7682eabbfd6 + - path: Sources/SubstrateML/AuditLogFold.swift + blob: 7423db7764c628cf5fd2aab0e9bb2d98d8c687bd + - path: Sources/SubstrateML/BradleyTerry.swift + blob: 63d335ee50ccee803f0508eb6f3135a29b6545db + - path: Sources/SubstrateML/CommunityDetection.swift + blob: bfba8e0255baeb66d4c79ccfaf34cd11e21a5ebd + - path: Sources/SubstrateML/CompositeDistance.swift + blob: a5f94c631d89c92d45b60e9da43be7fe6bdd19a2 + - path: Sources/SubstrateML/ConceptImplications.swift + blob: 1ac6ccbb70665532f135d2813f260cbbdad21716 + - path: Sources/SubstrateML/DeltaFeatureExtractor.swift + blob: 1ca396968e5e565096dbebb059b45b8fc3e5719f + - path: Sources/SubstrateML/DistillationPipeline.swift + blob: 82ce8bfaaa2e7a92077510ef239a6997a585c888 + - path: Sources/SubstrateML/DistillationScorer.swift + blob: 2cdc6dfe121080d432859d17487f2c8d15fd2910 + - path: Sources/SubstrateML/DPORReduction.swift + blob: 3d3c79e8ce7694b8308bb1c970e86a24de0446b0 + - path: Sources/SubstrateML/EigenvalueCentrality.swift + blob: f427430c34add289231520ba19ae4781316cbd13 + - path: Sources/SubstrateML/FeatureExtractors.swift + blob: 7e6d388a588ff7bec04be631948387a086a253ed + - path: Sources/SubstrateML/FFT.swift + blob: fdfaceeb902428149ead96107ed69c308bca0fff + - path: Sources/SubstrateML/FloatSimHash.swift + blob: 39cd2c4c36492c7cbaf55d6d22a2f32e273cfbae + - path: Sources/SubstrateML/FormalConceptAnalysis.swift + blob: 9cdf9613b30bf2e3a9a85088489c0e2b683a8a77 + - path: Sources/SubstrateML/InformationTheory.swift + blob: f7ba2a27904ceb19eef0fb5b06d1c5aad8b3894b + - path: Sources/SubstrateML/JacobiSVD.swift + blob: fcd924ff0a8409f224b56e7229f59659a6b5f51e + - path: Sources/SubstrateML/LatticeDistance.swift + blob: a7b96a376dfba0c815e4dc91962da1ccaeb2ac1d + - path: Sources/SubstrateML/LLMCalibrationCurve.swift + blob: 4e01882a65570ccce72e2789c23e95c46b6399a7 + - path: Sources/SubstrateML/MatrixDecay.swift + blob: b9595064a54b79f8ddee343385af4b8ebdc7f3e5 + - path: Sources/SubstrateML/MomentSummary.swift + blob: e8c07841c8c3440deb0375500ebcd35a6f116bb4 + - path: Sources/SubstrateML/NMFAlternatingLeastSquares.swift + blob: d64aadc8db165cc3b9f51b4fb1c644a6d6a4ca3c + - path: Sources/SubstrateML/NMFDoubleFrobeniusSquared.swift + blob: 0a725d72b70120828cb5911de594b0278ca450e5 + - path: Sources/SubstrateML/PairingHandshake.swift + blob: cb608882478ec57a6e6ba1f7728646ff3ea610f1 + - path: Sources/SubstrateML/PartialStateRecall.swift + blob: ad7b42bc25f9c4f283d75593d2c42cd546ccf074 + - path: Sources/SubstrateML/RandomWalks.swift + blob: 99f40b1685a876d81f84ca38b4857e5c6ba4e1dd + - path: Sources/SubstrateML/RowAttributeView.swift + blob: f63cf544a87adde435bede527531d1a9414dad03 + - path: Sources/SubstrateML/Sampling.swift + blob: 84fcdcade559229003688a7fc151f5c0a6dac6e6 + - path: Sources/SubstrateML/ShingleSimilarity.swift + blob: bf9dbaa5b0d5a5ee4ec5dfe42a5d65c85bdd0326 + - path: Sources/SubstrateML/TemporalCausalityFold.swift + blob: d99d7358893eeae0a442e0bd4e52f8a9b1cd7f5e + - path: Sources/SubstrateML/TemporalCompression.swift + blob: f34dde022faadd40058e35b62da12c8985763e59 + - path: Sources/SubstrateML/TierAscendingQuery.swift + blob: 8681cbdae9af7facfe7792392dcee15c8f8fe0aa + - path: Sources/SubstrateML/TierContributionFingerprint.swift + blob: bb56b7a82532f8c324168bd81edff6bf68fb33e9 + - path: Sources/SubstrateML/TypedDecayWeighting.swift + blob: 82412f392848c378b5b28221bbddade9f03bf1bf + - path: Sources/SubstrateML/VizGraphSignals.swift + blob: 4f90110f2a19cb92f3d45baf3bc4d82ac9c3dc65 +--- + +# SubstrateML Details + +This document walks through each source file in the package. Read +`OVERVIEW.md` first for the big picture. The files appear in +working-group order. First come the telemetry names. Then come the +fingerprint and distance math. Then the ingestion shapers. Then the +learning and decay machinery. Then the graph analytics. Then the +pattern miners. Then the distillation stages. Last come the +federation and privacy layer. + +## VizGraphSignals.swift + +This file gives the canonical metric names for the package's +telemetry layer. It defines five string constants. The five +graph-analytic algorithms use them as signal names, one per call, +when each reports completion. + +Telemetry is optional. It feeds the Topology visualization, a live +picture of the memory graph. Each algorithm emits exactly one metric +per invocation through `IntellectusLib`. It emits only when +monitoring is enabled. When monitoring is off, the default, the emit +costs one atomic boolean load and a branch. No clock read happens. No +allocation happens. Keeping the names in one file prevents a typo. A +typo could otherwise produce an orphaned metric that no dashboard +ever finds. + +The five constants are `communityAssignment`, `centralityScore`, +`nmfFactor`, `anomalyFlag`, and `edgeDecayedWeight`. +`communityAssignment` is `"community.assignment"`. Its value is the +community count. `centralityScore` is `"centrality.score"`. Its +value is always 1.0, a completion marker. `nmfFactor` is +`"nmf.factor"`. Its value is the final reconstruction error. +`anomalyFlag` is `"anomaly.flag"`. Its value is the absolute z-score. +`edgeDecayedWeight` is `"edge.decayed_weight"`. Its value is the +applied decay factor. The file also records the caller contract. +Callers supply the estate tag and the timestamp, because SubstrateML +never reads a clock. + +## FloatSimHash.swift + +This file gives `FloatSimHash`. It projects a dense float +embedding vector into the substrate's canonical fingerprint type, +`Fingerprint256`. A typical vector comes from an external model. +Examples include a 384-dimension MiniLM vector or a 768-dimension +BERT vector. + +The substrate compares memories by Hamming distance. Hamming +distance counts the differing bit positions between two +equal-length codes. External embeddings are floats, not bits. They +cannot join that comparison directly. `FloatSimHash` bridges the gap +with signed random-hyperplane projection, the classic SimHash +construction. The method generates 256 pseudo-random direction +vectors, each plus one or minus one, matching the input dimension. +It takes the sign of each hyperplane's dot product with the input. +It packs the 256 sign bits into four 64-bit blocks. Vectors that +point the same way land on the same side of most hyperplanes. Cosine +similarity in float space is thus approximately preserved as +Hamming closeness in bit space. + +`project(vector:seed:)` is the entry point. The hyperplanes come +from a SplitMix64 generator seeded by the caller. They are drawn in +a pinned order: 256 planes outer, coordinates inner. They follow a +pinned sign convention, where a low bit of one means plus one. The +same seed always yields the same planes. This matters because stored +fingerprints must stay comparable to freshly computed ones. Different +embedding providers use different seeds. Each provider's fingerprints +thus live in their own namespace. The dot-product work itself is +dispatched to a `SubstrateKernel` backend, selected once per process. +A future SIMD kernel can drop in without touching this file. +`planes(seed:dim:)` materializes the hyperplane set directly. Some +callers, such as the kernel-equivalence conformance test, need it in +that raw form. + +## CompositeDistance.swift + +This file gives `CompositeDistance.distance`, the substrate's +primary retrieval metric. It returns one number between zero and +one. That number says how far apart two memory rows sit. + +The score blends two normalized parts: +`alphaLattice * latticeDistance + alphaFingerprint * (hamming / +256)`. The lattice part shows topical distance in the +classification hierarchy. The fingerprint part is Hamming distance +divided by the fixed 256-bit width. Both weights default to 0.5. +Those defaults are only a starting point. The weights are learned +per user over time, through Bradley-Terry feedback (see +`BradleyTerry.swift`). + +One edge case is engineered on purpose. Two fingerprints are only +comparable when they come from the same hyperplane family. That +family match is the `compatibleSeedScope` argument. It always holds +inside one estate. Across estates it requires a completed pairing +handshake. When the scopes are not compatible, the function keeps +only the lattice term. It does not rescale that term. The resulting +smaller distance is the documented, intended behavior. Inputs are +checked with preconditions rather than clamped. An out-of-range value +thus fails loudly at the source. + +## LatticeDistance.swift + +This file gives the two conceptual distance axes that feed the +composite metric. One axis is tree distance in the UDC +classification hierarchy. The other is graph distance in the +Wikidata concept graph. + +`LatticeAnchorStr` pairs a UDC code string with an optional Wikidata +Q-ID, where zero means none. `UDCTreeDistance.distance(_:_:)` +compares two code strings by their longest common prefix. The more +leading characters two codes share, the closer their subjects sit in +the hierarchy. The raw formula can reach two, so the result is +clamped to the zero-to-one range the composite metric needs. +`WikidataGraphDistance` runs a breadth-first search over a +caller-injected `WikidataAdjacencyProvider`. That provider supplies +the "is-a" and "part-of" edges. The search is bounded at depth four. +It normalizes the path length with `1 - exp(-length / 3)`. +Unreachable concepts score the maximum, 1.0. The null Q-ID, which is +zero, also scores 1.0. The graph is directed as given. Symmetry holds +only if the provider supplies reverse edges. + +`LatticeDistance.distance(_:_:provider:alphaUDC:alphaQID:)` blends +the two axes. Each weight defaults to 0.5. A legacy overload also +accepts the older integer-hashed anchor type. Hashing destroys prefix +structure. That overload can only answer zero for a match or one +for a difference. It survives purely for compatibility with earlier +call sites. + +## PartialStateRecall.swift + +This file gives a recall primitive for rows that match in one way +and differ in another. A row fingerprint has four 64-bit blocks. +Different blocks encode different aspects of a memory. +`PartialStateRecall` scores a candidate row against an anchor. It +checks one chosen block subset for a match. It checks another subset +for a difference. A real example pairs the same topic with a +different behavioral state. + +`score(rowFingerprint:anchor:matchBlocks:differBlocks:)` finds a +restricted Hamming distance over each block subset. The match score +is one minus the normalized distance on the match blocks. The differ +score is the normalized distance on the differ blocks. The final +score is their product. A candidate must satisfy both constraints at +once. A row that matches the anchor everywhere scores zero, since it +fails to differ. A row unrelated on the match blocks also scores +zero. Block identifiers must come from the set `{0, 1, 2, 3}`. A +precondition enforces this rule. An out-of-range block would +silently corrupt the denominator and produce a subtly wrong score +instead of a crash. `topK(anchor:rows:...)` scores each row in a +linear scan. It returns the best k rows. This is the reference form +of the `recall_partial_match` query. `hammingBlocks(_:_:blocks:)` +exposes the block-restricted Hamming count on its own. + +## ShingleSimilarity.swift + +This file gives the substrate's one character-shingle Jaccard +similarity. It shows how alike two strings are. The measure is +the overlap of their three-character substring sets. + +Two consuming kits each needed this exact function for recall +de-duplication. Neither kit may depend on the other. The shared math +thus lives here, in a layer below both, following the "one +implementation per substrate atomic" rule. `shingles(_:)` lowercases +the input. It slides a three-character window across the string. It +collects each substring into a set. Strings shorter than the window +collapse to one whole-string shingle. The empty string yields the +empty set. `similarity(_:_:)` returns the intersection size divided +by the union size. The case of both strings empty is defined as +0.0, a plain rule rather than an undefined case. No stemming +happens. No tokenization happens. No locale-sensitive transform +happens either. This restraint keeps Swift and Rust byte-identical +on the content the recall path actually compares. The window size, +three, is pinned. It is long enough to discriminate near-duplicates. +It is short enough to survive paraphrase. + +## MomentSummary.swift + +This file gives the moment summary. It is one 256-bit fingerprint +that stands for everything active during a time window. The system +builds it by OR-reducing the fingerprints of the matching rows. + +OR-reduction sets a bit in the output whenever any input has that +bit set. This choice gives the summary useful algebra for free. The +result is order-independent. It is idempotent. It is monotonic: more +rows can only add bits, never clear them. An empty match yields the +all-zero fingerprint. Recall can then find similar past moments by +Hamming search over cached window summaries. + +What counts as "active during" a window is left open on purpose. The +cookbook defines three legitimate meanings. Both `summarize` +overloads thus take the predicate as a caller-supplied closure. +One overload works on full `Row` values. The other works on +`RowLite`, a 40-byte fingerprint-plus-timestamp pair. The `RowLite` +path runs four to five times faster on large windows, purely +because its loop touches less memory. Both paths are verified +byte-identical against a conformance vector, CRC `0x6762440b`. +`capturedDuring(_:_:)` is the common convenience predicate. +`orReduce(_:)` exposes the raw reduction for callers that already +hold a fingerprint list. + +## TemporalCompression.swift + +This file gives hierarchical time roll-up. Hour summaries roll +into days. Days roll into weeks. Weeks roll onward through months and +quarters into years. Each level is a `TemporalWindow`: a summary +fingerprint, a row count, and start and end timestamps. + +The summary operator is OR-reduction, so rolling up is associative +and commutative. The result does not depend on the order windows +arrive in. `compress(rows:startHLC:endHLC:level:)` builds a base +window from raw row fingerprints. `rollup(windows:to:)` merges finer +windows into one coarser window. It sums row counts with +overflow-safe addition. It takes the outermost time bounds. +`cascadeRollup(hourWindows:upTo:)` automates the whole ladder. It +buckets the current level's windows by which coarser slot they fall +into. That bucketing is integer division of epoch seconds by the +coarser level's approximate duration. It then rolls each bucket, one +level at a time. + +The `WindowLevel` durations are approximate, on purpose. A month +counts as thirty days. A year counts as 365. These numbers serve only +as bucket divisors, never as calendar math. Storage and scheduling +belong to the caller, the dreaming daemon. This file is pure +computation. It never reads a clock. + +## FeatureExtractors.swift + +This file gives the ingestion boundary for ambient signals. Five +encoder types turn raw OS sensor samples into `AmbientSampleRow` +values. Each value is a fingerprinted, classification-anchored row +ready for storage and recall. + +Each extractor covers one stream. `HealthKitExtractor` covers +biometrics. `CoreLocationExtractor` covers position. +`EventKitExtractor` covers calendar events. `ScreenTimeExtractor` +covers app usage. `SystemTelemetryExtractor` covers system load. The +OS-specific capture code lives in the host apps. Only the pure +encoding lives here. Each `extract` call finds four 64-bit +subhashes from the sample's salient fields. It feeds them, with a +hyperplane family, into the substrate's SimHash. That step produces +the 256-bit fingerprint. Numeric fields are quantized to integers +before hashing. Coordinates, for example, are multiplied by one +million first. This step means floating-point representation can +never change a fingerprint. Attendee lists are sorted before hashing. +The same meeting thus fingerprints identically, regardless of +listing order. Each stream carries a fixed UDC lattice code as its +coarse topical anchor. Health uses `"613.71"`. Location uses +`"914"`. + +Only HealthKit rows carry a `payload`. It is a small, key-sorted, +deterministic binary encoding for human-readable recall detail. The +other streams emit no payload, by design. The fingerprint already +carries all recall-relevant signal. Storing less raw data also means +less to protect. The FNV-1a hash constants serve as the file's +general 64-bit mixers. + +## AuditLogFold.swift + +This file gives `AuditLogFold`. It reconstructs what a memory row +looked like at any point in time, by replaying its audit log. + +The audit log is a grow-only set, called a G-Set in CRDT terms. Its +events are only ever added, never edited. Current state is the fold, +the sequential replay, of a row's events sorted by their HLC +timestamps. State as of time T is that same fold, truncated at T. +The fold depends only on the set of events, never on their arrival +order. Two devices that have exchanged the same events thus +converge to the same state. That property is what makes sync +correct without coordination. + +`projectCurrentState(rowId:nounType:events:)` folds a full history. +`projectStateAt(...)` folds up to a cutoff. +`projectAll(events:asOf:...)` groups a whole log by row and folds +each one. The result type, `ProjectedRowState`, carries the row's +bitmaps, the lattice anchor, the tombstone flag, and the last-event +time. One rule is sticky, on purpose. Once the tombstone marker, +state raw value 33, appears, the row stays tombstoned. It stays +tombstoned even if a later, illegal event tries to revive it. The +fingerprint is intentionally absent from the projection. Callers +recompute it from the bitmaps and anchor when needed. + +## RowAttributeView.swift + +This file gives the bridge from the audit log to the pattern +miners. That bridge is `RowAttributeView`: one row's field writes, +flattened into sorted field-and-value byte pairs. This is the exact +shape Apriori and Formal Concept Analysis consume. + +`from(auditEntries:)` runs a four-step pipeline. First, it builds a +vocabulary of all distinct field paths in the batch. That vocabulary +is sorted alphabetically and capped at 64 entries, because the +miners' item type stores the field index in six bits. Second, it +groups entries by row. It keeps only the latest write per field, by +HLC order. That rule is the same last-write-wins rule the audit fold +uses, so the two subsystems never disagree. Third, it expands each +surviving value. A bitmap value becomes one attribute per set bit. An +integer contributes its low byte. A null contributes nothing. Rows +left with no attributes are dropped. Attributes are sorted within a +view. Views are sorted by row identity. The output is thus +deterministic. + +One subtlety is documented in the open. The vocabulary is ephemeral +to a single call. Two separate calls can assign different field +indexes. Callers who need a stable vocabulary must merge their entry +lists before calling. The input types `RowAuditEntry` and +`RowAuditValue` are defined here, in a lower layer than the kit that +owns the real audit types. That placement keeps the dependency arrow +pointing downward. + +## MatrixDecay.swift + +This file gives the substrate's forgetting mechanism. It applies +exponential half-life decay to each matrix in the matrix tier. +Those matrices are the accumulated statistics tables: field +presence, correlation, co-activation, and action outcomes. + +The rule is constitutional. Decay may only shrink values. New +evidence enters through separate write operations. It never enters +through this file. `DecayingMatrix` is a plain row-major matrix. It +carries its own half-life and its last-decay timestamp. +`apply(to:nowSeconds:estate:ts:)` finds the elapsed time. It +derives one shared factor, `exp(-elapsed * ln2 / halfLife)`. It +multiplies each cell by that factor. A non-positive elapsed time is +a documented no-op, since decay never runs backward. The function +still emits its telemetry metric with factor 1.0 in that case. A +watcher can then use the emitted factor to tell two states apart. +Either the daemon ran and found nothing to decay. Or the daemon +never ran at all. Using one scalar factor per pass, rather than +per-cell clocks, is what makes decay commute with addition. The file +states +that property formally. + +`decayFactor(elapsedSeconds:halfLifeSeconds:)` is the pure +projection helper. `decayAndAdd(...)` is the decay-then-add step the +online write path uses. `DecayHalfLives` records the per-matrix +schedule from the cookbook. Field presence gets 90 days. Correlation +gets 180. Co-activation gets 60. Temporal causality gets 30. Action +outcomes get 365. Calibration gets 730. Two details protect +determinism and safety. `ln2` is a local constant, chosen to match +Rust's value bit-for-bit. The elapsed-seconds cast for the telemetry +tag saturates instead of trapping on absurdly large values. Three +`applyExponentialDecay` overloads for the typed matrices are +deliberate no-op adapters at this reference level. The production +per-matrix routine lives in the consuming kit. + +## ActionOutcomeMatrix.swift + +This file gives the action-outcome matrix. It is the substrate's +reinforcement record of which actions succeed. Each cell is keyed by +a six-bit action kind and a six-bit outcome category. Each cell +accumulates a success count, a total count, and the last-update HLC. + +`ActionOutcomeKey` enforces the six-bit ranges with preconditions. +It packs both bytes into one `UInt16` ordering key. `observe(...)` +increments a cell with overflow-safe arithmetic. +`successRate(...)` and `observationCount(...)` read cells back. The +most interesting method is `topActions(forOutcome:k:minObservations:)`. +It ranks actions by the Wilson lower bound, rather than by the raw +success rate. The Wilson bound is the bottom of a 95 percent +confidence interval around the rate. A cell with two observations +and a perfect record should score below a cell with two hundred +observations and a 95 percent record. The Wilson bound encodes +exactly that caution. The +method returns the rate, the bound, and the count together. Callers +can then see the value the ranking actually used. Ties break by +total count first, then by ascending action. Output stays +deterministic. Decay for this matrix runs on a 365-day half-life. +The dreaming daemon applies it lazily through `MatrixDecay`, not +here. + +## BradleyTerry.swift + +This file gives an online Bradley-Terry estimator. It learns a +per-row strength score from pairwise preference feedback. That +strength then reorders future recall candidates. + +Each time recall surfaces a candidate set, the user or agent picks +one. That pick becomes an observation: the winner beat the losers. +The Bradley-Terry model says the probability that row i +beats row j equals `w_i / (w_i + w_j)`. The implementation works in +log-space, where `theta = log w`. The win probability then becomes a +sigmoid of the theta difference. Gradient updates never need to +guard positivity. `observe(_:)` performs one stochastic gradient step +per observation. It applies L2 regularization, pulling theta toward +zero. This pull keeps early sparse data from overfitting. For +multi-loser observations, the winner's theta updates sequentially +against each loser. Each update recomputes against the running +value. This is a pinned order, and the Rust port mirrors it +exactly. + +`observeBatch(_:)` applies a day's worth of observations in order. +`strength(of:)` and `probability(_:beats:)` read the model. There is +no randomness anywhere. The same observation sequence always +produces bit-identical theta. A serialized theta dictionary +warm-restarts the estimator exactly. The defaults are a learning +rate of 0.05 and an L2 of 0.001. These values are tuned for the +ten-to-one-hundred observations per day a personal estate produces. +Strength is a +ranking signal only. It never changes what is stored. It never +filters what exists. + +## LLMCalibrationCurve.swift + +This file gives a twenty-bucket calibration histogram. It tracks +how well a model's stated confidence matches reality. + +The cognition tier sometimes resolves a probabilistic claim. The +model might say it is 80 percent sure, and the claim later turns out +true or false. `observe(claimedConfidence:actualOutcome:)` clamps the +confidence into the range zero to just under one. It maps that value +to one of twenty buckets, each 0.05 wide. It bumps that bucket's +predicted and hit counters. The counters use wrapping arithmetic on +purpose. This is a long-lived accumulator, so overflow should wrap +rather than crash. + +`actualRate(in:)` reads one bucket's observed hit rate. +`expectedCalibrationError()` shows the weighted average gap +between each bucket's observed rate and its midpoint, the perfectly +calibrated diagonal. `brierScore()` does the same with squared gaps. +`decay(factor:)` scales each counter by a fraction, so stale +calibration evidence fades smoothly. The dreaming daemon drives this +decay on a 730-day half-life schedule. The result lets recall +annotate a model's claimed confidence. It shows how much that +confidence has historically been worth. + +## Sampling.swift + +This file gives three deterministic distribution samplers: +Normal, Gamma, and Beta. They sit under Thompson sampling. Thompson +sampling is a decision strategy. It picks among options by drawing +from a belief distribution over each option's payoff. The dreaming +daemon uses it to choose maintenance strategies. The policy lives in +the consuming kit. The sampling math is centralized here, so each +consumer shares one implementation. + +All three samplers take the caller's SplitMix64 generator by +`inout` reference. The random stream thus advances visibly and +reproducibly. `sampleNormal(rng:)` uses the Box-Muller transform. It +keeps only the cosine branch, on purpose, consuming exactly two +uniform draws per call. Caching the sine branch would desynchronize +the stream between ports. `sampleGamma(shape:rng:)` implements +Marsaglia-Tsang rejection sampling for a shape of at least one, with +the standard squeeze constant 0.0331. It reduces smaller shapes +through the Ahrens-Dieter identity. That reduction's extra uniform +draw happens before the recursive call. This order is pinned for +cross-port stream identity. `sampleBeta(alpha:beta:rng:)` derives +Beta from the ratio of two Gamma draws, with alpha drawn first. +Given the same seed and inputs, both legs consume the same words in +the same order. Both return the same values, gated by a shared +conformance vector. + +## CommunityDetection.swift + +This file gives Louvain community detection over the estate +graph. It labels each node with a community, so related memories +cluster together. Auto-rooming, keystone recall, and the daily +dreaming refresh all consume the resulting partition. + +Louvain works in two phases. Phase one repeatedly tries moving each +node into a neighboring community. It keeps whichever move most +improves modularity, a score that rewards putting well-connected +nodes together relative to chance. Phase two condenses each +community into a supernode. It then repeats phase one on the smaller +graph. `detect(...)` runs phase one only. `detectFull(...)` runs the +full multi-level algorithm and is the normal entry point. + +Two design choices deserve explanation. The first is determinism. +Candidate communities are evaluated in ascending label order, so +score ties always resolve to the lowest label. This avoids depending +on hash-map iteration order. A `canonicalize(_:)` step then renumbers +labels by first appearance, so Swift and Rust emit the same +partitions. The second choice is the resolution parameter. Plain +Louvain can get pair-locked on graphs that have strongly bonded pairs +plus weak star edges. It becomes unable to escape a local optimum. +The Reichardt-Bornholdt gamma scales only the degree-penalty term. A +small gamma, 0.05, forces pairs to merge into hub communities. The +default is 1.0. It reproduces classical Louvain exactly, since IEEE +arithmetic guarantees `1.0 * x == x`. The refactor thus cannot +perturb legacy results. The implementation is intentionally the +compact O(N·E)-per-pass reference, rather than the fastest possible +variant. One `community.assignment` telemetry signal fires per call, +when monitoring is on. A degenerate all-zero-weight graph returns +identity labels and emits nothing. + +## EigenvalueCentrality.swift + +This file gives eigenvalue centrality. It is a per-row authority +score, computed by power iteration over the estate graph. The score +is cached as each row's keystone score for `recall_keystone`. + +Power iteration repeatedly multiplies a score vector by the graph's +adjacency, then renormalizes it. The vector converges to the +principal eigenvector. A node scores highly there when high-scoring +nodes connect to it. Direction matters, and it is a locked design +decision. `compute(...)` multiplies by the transpose. It +accumulates, at node j, the influence of nodes that point to node j. +That is authority: many rows reference this one. It is a different +meaning from a hub, where this row references many others. +Authority is the right meaning for cognitive keystones. A +conformance vector pins the directed behavior, so it cannot drift +back. For undirected centrality, callers symmetrize the adjacency +first. + +Each iteration adds a Perron shift, `xNext += 1.0 * x`. This shift +moves each eigenvalue without changing the eigenvectors. It breaks +the plus-minus oscillation that bipartite graphs, such as stars, +exhibit under raw power iteration. Iteration stops at convergence, +when the L2 difference falls below 1e-6, or at a 100-iteration +safety cap. A zero-norm vector falls back to the uniform +distribution: an edgeless graph makes each row equally non-central. +All three return paths emit a single `centrality.score` telemetry +signal, tagged with the node count and the iterations used. + +## RandomWalks.swift + +This file gives random-walk-with-restart over the memory graph. +It is the engine behind exploratory recall. That recall wanders +outward from a starting memory and reports where it keeps landing. + +`walk(adjacency:start:length:restartProb:seed:)` runs one walk over +a densely indexed weighted adjacency list. Each step does one of two +things. It restarts at the start node, with probability 0.15 by +default. Or it moves to a neighbor, chosen by roulette-wheel weighted +sampling. A dead end, a node with no out-edges, counts as an implicit +restart rather than an +error. The walk is seeded explicitly. Four things pin it: the graph, +the start, the length, and the seed. The same four always reproduce +the same walk on both legs. The file carries its +own public `SplitMix64`. Its `uniform01(_:)` derives a double from +the top 53 bits, exactly as Rust does. Malformed graphs fail hard +preconditions immediately. An out-of-range neighbor fails this way. +A negative or non-finite weight fails this way too. A bad adjacency +cannot represent probabilities, and silence would corrupt results +downstream. + +`walkWithRestart(seed:steps:restartProbability:rngSeed:adjacency:)` +is the row-identifier variant. It walks a dictionary adjacency with +uniform neighbor choice. It returns visit counts per row, the shape +the exploratory recall recipe consumes. `sampleWeighted(_:rng:)` +exposes the weighted sampler. It includes pinned fallbacks: a +uniform choice when the total weight is zero, and the last neighbor +on a cumulative rounding shortfall. + +## NMFAlternatingLeastSquares.swift + +This file gives the canonical non-negative matrix factorization +engine, known as NMF. NMF approximates a non-negative matrix V as +the product of two smaller non-negative matrices, `V ≈ W × H`. Rows +of H are latent themes. Rows of W say how much each original row +loads on each theme. The substrate runs NMF over its field-presence +and co-occurrence matrices. This surfaces themes for recall and for +the Topology theme overlay. The dreaming daemon reruns it monthly. + +`factorize(V:rank:maxIterations:tolerance:seed:estate:ts:)` uses the +Lee-Seung multiplicative update rules. These rules keep each entry +non-negative automatically. Each iteration first rescales H by the +ratio of `WᵀV` to `WᵀWH`. It then rescales W by the ratio of `VHᵀ` +to `WHHᵀ`. An epsilon of 1e-9 guards the denominators against +division by zero. W and H start as uniform random values from a +SplitMix64 generator, seeded by the caller with a default of +`0xDEADBEEFCAFEBABE`. Initialization is thus bit-identical +everywhere. Iteration stops when the root-mean-square reconstruction +error stops changing by more than the tolerance, or at a fixed cap. +Entry preconditions enforce a rectangular, finite, non-negative V. +The Lee-Seung theorem is undefined otherwise. + +The hot loops run on flat row-major arrays, through unsafe buffer +pointers. This is a documented performance necessity. +Bounds-checked nested-array subscripts block auto-vectorization. +They once left Swift roughly one hundred times behind the Rust port +on large reindex drains. The loop nest and reduction order were +preserved exactly through that rewrite. The NMF conformance vector, +CRC `0x300bf633`, thus remains byte-identical. +`reconstructionError(V:W:H:)` exposes the RMS error on nested +arrays. The nested-array matrix helpers remain for external callers. +On completion, the engine emits the `nmf.factor` telemetry signal, +carrying the final error. + +## NMFDoubleFrobeniusSquared.swift + +This file gives a parked, non-production NMF variant. It is a +double-precision engine with a raw Frobenius-squared convergence +test. One consumer used it before migrating to the canonical engine +above. + +It exists for one reason: honesty in benchmarking. The migration +decision needs a future benchmark. That benchmark must compare the +old f64 Frobenius-squared approach against the canonical f32 RMS +approach. It must compare iteration count, time, memory, and recall +quality. Deleting the old algorithm would make that comparison +impossible. The file is thus preserved verbatim, behind a +production gate. No consumer may wire to this type until the +benchmark passes. Even SIMD acceleration of the f64 path stays +separately gated. + +The algorithm is the same Lee-Seung scheme, in scalar f64, with +three deliberate differences. Convergence checks the raw, +unnormalized Frobenius-squared error, rather than RMS. +Initialization floors each starting cell at 1e-3, because a cell at +exact zero can never recover under multiplicative updates. The +default seed is its own historical value, `0xC0FFEE_BABE_BEEF`, +distinct from the package's canonical seeds. `factorize(...)` +returns an `NMFDoubleFrobeniusSquaredFactorization`, whose +`loadings(forRow:)` yields a row's latent factors. The file is +parked, but it stays conformance-gated. Swift and Rust must stay +bit-identical on its named vector, so the eventual benchmark +shows the algorithm, not implementation drift. + +## AnomalyDetection.swift + +This file gives statistical anomaly scoring for telemetry +streams and matrix cells. It offers the classic z-score, a robust +variant, and a threshold check that turns a score into a flag. + +A z-score says how many standard deviations a value sits from the +mean of a baseline window. `zScore(value:mean:stddev:)` is the pure +formula. `rollingZScore(window:current:...)` finds the mean and +deviation from the window first. The robust variant, +`modifiedZScore(value:median:mad:)`, replaces the mean with the +median. It replaces the deviation with the median absolute +deviation. It scales by the standard constant 0.6745, so the same +threshold applies to both variants. Outliers already inside the +window cannot drag the baseline the way they drag a mean. +`isAnomalous(zScore:threshold:)` compares the absolute score against +the default threshold, 3.0. That threshold gives roughly a +one-in-370 false-positive rate on normal data. + +`rollingModifiedZScore(...)` sorts a mutable copy of the window in +place, rather than allocating a sorted copy. At large windows this +measured five times faster. The behavior is pinned by a CRC marker, +`0x6c6fda4d`. It must also match Rust's sort semantics for all +non-NaN input. Both rolling functions emit the +`anomaly.flag` telemetry signal, with the absolute score as the +value. The timestamp is always caller-supplied. + +## JacobiSVD.swift + +This file gives a deterministic truncated singular value +decomposition, or SVD. It is the linear algebra that turns a +term-document matrix into low-dimensional semantic embeddings, for +latent semantic analysis in CorpusKit. + +SVD factors a matrix into rotation, scaling, and rotation: +`A = U S Vᵀ`. The implementation is one-sided cyclic Jacobi. It +repeatedly applies two-by-two planar rotations to each pair of +columns, in a fixed cyclic order. It accumulates the rotations into +V. After the sweeps, the column norms become the singular values. +The normalized columns become U. A sign convention forces the +largest-magnitude entry of each left vector positive. This removes +the inherent plus-minus ambiguity. + +Each convergence-affecting choice is pinned, for cross-port +bit-identity. That is why this exists instead of a call to +Accelerate or LAPACK. The sweep count is fixed, at a default of 30, +and is not convergence-tested. The rotation order is fixed. All +arithmetic is scalar Float32, with no SIMD and no fused +multiply-add. The rotation formula in `jacobiCS(alpha:beta:gamma:)` +keeps a pinned expression tree that must not be refactored. +`decompose(A:rank:sweeps:)` requires at least as many rows as +columns. It returns an `SVDResult`: U, non-increasing singular +values, Vt, and rank. The sweep loops use unsafe buffer pointers, +for the same documented vectorization reason as the NMF engine. +Each index is derived from loop bounds, never from data. + +## FFT.swift + +This file gives the discrete Fourier transform, and the rhythm +analysis built on it. The substrate uses this to detect periodicity. +It reads one fingerprint bit across the most recent N time buckets. +The dominant frequency of that zero-or-one series reveals the +rhythm of whatever that bit encodes. Circadian patterns show up in +heart-rate bits. Weekly cycles show up in calendar bits. + +`Complex` is a plain double-precision complex number, with the +arithmetic the transform needs. `FFT.forward(real:)` is the +Cooley-Tukey radix-2 algorithm: a bit-reversal permutation, followed +by log2(N) butterfly stages. The input length must be a power of +two. This is a constitutional constraint, and callers zero-pad +shorter windows. The production Apple-silicon path routes to the +Accelerate framework. It must still produce bit-identical output to +this scalar reference, on the conformance vectors. Conformance means +output equality, and never speed parity. `magnitudeSpectrum(real:)` +maps the spectrum to magnitudes. + +`RhythmAnalysis.analyze(series:bucketDurationSeconds:)` runs the +transform. It finds the strongest bin among the unique positive +frequencies, bin one through N divided by two. It converts that bin +to a dominant period in seconds. It also reports total spectral +energy, excluding the DC bin. A constant series has no dominant +period, so it reports `nil`. The cookbook treats "no rhythm" as an +honest answer, and a deliberate one. The fingerprint-based overload +extracts the bit series from a list of fingerprints first. The block +and bit position are precondition-checked. + +## InformationTheory.swift + +This file gives the information-theoretic primitives: entropy, +mutual information, KL divergence, cross-entropy, Jensen-Shannon +divergence, and normalized mutual information. These primitives +quantify uncertainty in bitmap distributions. They also detect drift +between a recent window and a long-term baseline. + +All quantities use log base two, so results come out in bits. +Entropy shows how unpredictable a distribution is. Mutual +information shows how much knowing one variable tells you about +another. KL divergence shows how badly a model distribution q +explains an observed distribution p. Terms with probability exactly +zero are skipped everywhere. This implements the standard rule that +zero times log of zero equals zero, without ever producing NaN. + +Two honesty notes are documented in the code. When p has mass where +q has none, the true KL divergence is infinite. +`klDivergence(_:_:)` skips such terms, so its result is a lower +bound. Callers must know this. `normalizedMutualInformation(joint:)` +returns a zero sentinel for a ragged joint matrix. This avoids +silently computing a plausible-looking wrong value from corrupted +marginals. Length mismatches between p and q are hard preconditions. +Distribution validity itself, meaning non-negative values that sum +to one, is the caller's responsibility by contract. + +## AssociationRuleMining.swift + +This file gives pairwise association-rule mining, over +`MatrixO`, the co-occurrence matrix the substrate updates on each +capture. It surfaces rules of the form: when field A has value x, +field B tends to have value y. Each rule carries the standard +support, confidence, lift, conviction, and leverage metrics. + +The design leans on how `MatrixO` is built. Its diagonal cells +already hold single-item counts. Its off-diagonal cells hold pair +counts. `mineAssociationRules(matrix:activeRowCount:thresholds:)` +thus needs only two passes over the stored entries. One pass +collects single supports from the diagonal. The other walks the +pairs, finds the five metrics, and keeps rules that clear the +`MiningThresholds` gates. Self-rules, where A implies A, are skipped +as noise. Conviction is defined as positive infinity at confidence +one. The total row count, N, is injected by the caller. The matrix +itself does not know it. The entries iterate in canonical +packed-key order, so the output needs no final sort. Both legs rely +on that order for byte-identical results. + +`Item` is the shared atom: a field-and-value byte pair, with a +packed 16-bit ordering key. The Apriori engine reuses it. The +internal `AssociationRuleEngine` holds the actual math, so +conformance tests can drive it directly. Multi-item antecedents fall +outside this file's scope. They belong to `AprioriMining.swift`. + +## AprioriMining.swift + +This file gives the multi-antecedent generalization: the classic +Apriori algorithm, over `RowAttributeView` rows. It produces rules +such as: if A and B are set, then C is set too. At the minimum +itemset size, it reproduces the pairwise engine's answers. Beyond +that size, it finds combinations the pairwise engine misses. + +`AprioriMining.mine(rows:thresholds:)` follows the textbook +outline. Each row becomes an item set. Frequent single items seed +the search. Each level joins frequent item sets of size k minus one +that share a lexicographic prefix. It counts candidate support with +subset tests. It prunes anything below minimum support, up to the +size cap `maxK`, which defaults to three. Rules are extracted from +each frequent itemset of size two or more. Each member is tried as +the consequent. Rules are then filtered by the support, confidence, +and lift gates in `AprioriThresholds`. Requiring a lift of at least +1.0 suppresses anti-correlated coincidences. + +Determinism gets specific care here, because Swift dictionaries +iterate in undefined order. Itemsets are processed in canonical +lexicographic order. The final sort uses a four-key total order: +lift, confidence, evidence count, and then a lexicographic +tie-break. Output stays the same across runs and across languages. +One defensive detail matters. The type's custom `Decodable` +initializer routes through the public initializer. A serialized +`maxK` of zero or one thus cannot bypass the floor of two and +corrupt the level loop. `mineAprioriRules(rows:thresholds:)` is a +free-function wrapper, matching the pairwise engine's call style. +`AprioriRule` carries the five metrics, plus a raw `evidenceCount` +for interpretability. + +## FormalConceptAnalysis.swift + +This file gives bounded Formal Concept Analysis, or FCA. It +finds the exact groupings hidden in a table of rows and attributes. +A formal concept is a maximal pair: a set of rows that share exactly +a set of attributes. The concepts of an estate are data-driven +"kinds of memory." They emerge from what was actually recorded. This +differs from the soft themes NMF finds, and from the graph clusters +Louvain finds. + +`FormalAttribute` is one typed attribute: a namespace, a key, and a +value. It is ordered lexicographically, and that order is the +determinism backbone of the whole file. `FormalContext` stores the +row-attribute relation in both directions, as bitsets. The +derivation operators reduce to word-wise bit operations because of +that choice. `extent(of:)` finds rows carrying all given attributes. +`intent(of:)` finds attributes common to all given rows. +`closure(of:)` combines both. Contexts build from raw per-row +attribute lists, or from `RowAttributeView` batches. + +Full concept-lattice enumeration is exponential, so +`BoundedConceptMiner` never attempts it. It seeds from attributes +meeting `minSupport`. It can optionally seed from frequent attribute +pairs, in `.multi` seed mode, capped by `maxSeeds`. It takes exactly +one closure per seed. It deduplicates by intent. It truncates at +`maxConcepts`. This gives polynomial cost by construction, with a +fully specified output sort. `ConceptCoverDeltas.covering(concepts:)` +finds the direct-neighbor edges of the concept order, a +structural lens showing which attributes were added. The file +explains that a cover edge shows structure only. It is not a +logical implication. `StabilityEstimator.estimate(...)` +approximates how robust a concept is to noise. It re-closes random +half-subsets of the extent, under a per-concept SplitMix64 stream. +That stream mixes the caller seed with a hash of the concept, using +a default seed of `0xCAFEBABEDEADBEEF`. This trades exactness for a +bounded budget. A zero budget, the default, leaves stability as +`nil`. + +## ConceptImplications.swift + +This file gives the Duquenne-Guigues canonical basis. That basis +is the minimal set of implications that hold without exception +across a formal context. Each implication reads: each row with +attributes X also has attribute Y. Where FCA finds groupings, this +file finds the rules those groupings obey. That feeds consolidation +and synthesis. + +`conceptImplications(over:context:maxImplications:maxPremiseSize:)` +enumerates candidate premises, in strictly increasing size order. It +tests each one for pseudo-intent status. The closure must add +something new. Each smaller pseudo-intent's conclusion must +already sit inside it. Processing in size order makes that +incremental test equivalent to full minimality checking. Two +independent caps keep this NP-hard enumeration bounded. +`maxImplications` is a hard stop, and it sets `isTruncated`, with a +look-ahead so the flag stays honest when the cap lands mid-run. +`maxPremiseSize` silently skips larger premises, without affecting +the soundness of what gets emitted. Each order is pinned: the +attribute universe, the combination enumeration, and the final sort +by premise size then lexicographic content. Both legs thus emit +bit-identical bases. `Implication` carries the premise and +conclusion sets. An empty context short-circuits to an empty, +untruncated result. + +## TemporalCausalityFold.swift + +This file gives the fold that feeds the temporal-causality +matrix. It scans a time-sorted stream of audit entries. It counts +pairs of the form: field X changed, then field Y changed some +minutes later. Each pair is bucketed by lag. + +`fold(entries:windowMinutes:startWatermark:)` processes each entry +newer than the caller's watermark. It evicts buffered entries older +than the window, 256 minutes by default. It pairs the new entry +against each remaining buffered entry. Each minute delta maps to +one of eight log-spaced lag buckets, from one up to 128 minutes, +through `lagBucket(forMinutes:)`. The fold emits one delta increment +per source coordinate, target coordinate, and bucket. The result +carries the deltas plus a new watermark, so the hourly batch run +resumes where it stopped. + +Two contracts matter here. Entries must arrive pre-sorted by HLC. +The fold does not re-sort them, and would silently compute wrong +deltas otherwise. The rolling buffer is capped at 512 entries, +`maxWindowOccupancy`. A bulk import can drop tens of thousands of +events into one window, and unbounded pairing is quadratic. Keeping +only the most recent in-window entries preserves the near-lag +signal the causality matrix actually cares about. Output deltas are +aggregated by key, but returned in first-insertion order, so +repeated runs stay bit-identical. The lag bucket table and the +occupancy cap are pinned constants, mirrored exactly in the +downstream matrix type and the Rust port. The input shapes are +`TemporalFieldCoord` and `TemporalAuditEntry`. They are defined +locally, so the kit that owns the real audit types stays upstream. + +## TypedDecayWeighting.swift + +This file gives type-aware staleness weighting for distillation +features. Different kinds of extracted facts age at different +speeds. A version number, a numerical fact, goes stale within days. +A domain concept, an entity fact, stays true across major changes. + +`DistillationFeatureType` names four categories, with wire tags ENT, +REL, TMP, and NUM. It pins each category's decay rate, lambda: 0.1, +0.2, 0.5, and 0.8. This scheme is an analogy with frequency bands in +diffusion noise schedules. `weight(featureType:ageInUnits:)` +finds the exponential weight `exp(-lambda * age)`. It clamps +negative ages to zero, so future-dated observations count as +present. One age unit is one day by convention. That equals `86,400` +seconds, though callers may choose a different unit. + +`weightedDocFrequency(featureType:presenceTimestamps:allMemoryTimestamps:referenceDate:timeUnit:)` +finds the decayed document frequency. That frequency is a ratio of +two weights. The first is the summed weight of memories where a +feature is present. The second is the summed weight of all memories +in the cluster. Recent evidence +thus counts more than old evidence, at a rate set by the +feature's type. This is the frequency the distillation pipeline +uses, whenever timestamps are available. + +## DeltaFeatureExtractor.swift + +This file classifies a feature's value sequence across a cluster, +into four trajectory states. A value can stabilize. A value can +trend. A value can flip-flop. A value can turn to noise. Its job is +rescue. A feature may fail the recurrence threshold on raw counts. +It may still carry a real pattern over time. This classifier finds +that pattern. + +`DeltaType` names five verdicts: STATIC, CONVERGENT, MONOTONE, +OSCILLATING, and DIVERGENT. +`analyzeCategorical(sequence:decayLambda:)` handles string-valued +features. All-identical values score STATIC. An A-B-A-B pattern over +the last four observations scores OSCILLATING. Four observations are +required, to avoid false positives on short sequences. Otherwise, the +trailing run of the final value decides the verdict. Its length +counts as a fraction of the sequence. That fraction scores CONVERGENT +at or above a default threshold of 0.5, and DIVERGENT below it. +`analyzeNumerical(sequence:decayLambda:)` handles numbers through +consecutive differences. All-zero differences score STATIC. All one +sign scores MONOTONE, with a slope and a fixed confidence equal to +the type's decay rate, default 0.8. Strict sign alternation scores +OSCILLATING. Anything else scores DIVERGENT. + +Both paths return a full `DeltaAnalysis`: verdict, terminal value, +convergence score, optional slope, and confidence. The caller +thus always has a well-formed result. The file is a pure +namespace. It holds no state. It reads no clock. It uses no +randomness at all. The oscillation check looks only at the recent +tail, on purpose. The question this file answers is the current +trajectory, and never the full history. + +## DistillationScorer.swift + +This file gives the scoring stages of distillation. It decides +whether a cluster of memories is coherent enough to compress. If so, +it decides which features form the core factoid. + +A pipeline of pure functions runs in order. A structural threshold, +`structuralThreshold(M:) = 2/M`, splits features into two groups. +Structural features recur in at least two of the M units. Episodic +features appear only once. `applyStructuralThreshold(features:M:)` +performs the split. The file's long comment explains a design +choice: it uses intra-cluster recurrence, rather than a majority +vote. A majority vote was found to discard real distillable cores, +when a single document was split into sentences. +`computeSNR(features:M:)` then gates readiness. The summed frequency +of structural features must be at least twice the episodic sum. That +ratio is the signal-to-noise ratio, floored at a tiny epsilon +against division by zero. `computeStructuralScores(features:)` +assigns each feature a score. The score rewards frequency and +consistency, through binary entropy. + +`buildPMIGraph(thresholdFeatures:incidenceMatrix:M:)` builds the +coherence graph. Pointwise mutual information, in log base two, +shows whether two features co-occur more than chance would +predict. Positive-PMI pairs become edges. Connected components are +found by depth-first search, quadratic but acceptable at the +documented cap of about 150 features. Components are sorted by +descending weighted frequency. `selectDominantComponent(graph:)` +returns the top component, the factoid core. +`computeConfidence(selected:allThreshold:)` scores it as the mean +frequency, times a fragmentation penalty: the fraction of threshold +features the core kept. Everything here is deterministic and +log2-based. The Rust port's comments stress that log2, and never +natural log, must match bit-for-bit. + +## DistillationPipeline.swift + +This file gives the complete five-stage distillation algorithm. +A cluster of raw memory strings goes in. One condensed factoid comes +out. The output type is `DistillationOutput`. It carries the +formatted drawer content, the confidence, the SNR, and an optional +delta verdict. It also carries a success flag and a structural +fingerprint for Hamming-space lookup. + +`run(input:extractFeatures:intraItem:)` executes the stages. Stage +one extracts features, per memory and per feature type, through an +injected `FeatureExtractor` closure. Production injects a real +entity tagger. Tests use the bundled capitalization-heuristic +`defaultExtractor`. This stage builds the vocabulary and the +incidence matrix. Document frequencies come from +`TypedDecayWeighting` when timestamps exist. Stage two applies the +SNR gate and the recurrence threshold. Stage two-and-a-half is the +delta pre-pass. Features that failed recurrence are grouped by +predicate key. Their value sequences run through +`DeltaFeatureExtractor`. Converging or trending features get rescued +with their terminal value. Stage three builds the PMI graph and +keeps the dominant component, with two deliberate exceptions. In +intra-item mode, where one document splits into sentences, all +passing features are kept. PMI pruning would wrongly split a single +coherent document otherwise. A ubiquity fix-up also re-adds any +feature present in nearly each member. A ubiquitous feature has zero +PMI with everything, and would otherwise vanish despite being the +semantic spine. Stages four and five score the result, format the +header, and OR-reduce the per-feature hashes into the fingerprint. + +`featureHash(_:)` maps a feature string to a `Fingerprint256`, +through SplitMix64, under the pinned seed `featureSimHashSeed`. That +seed, `0x44495354494C4C41`, spells "DISTILLA" in ASCII. Changing the +seed invalidates each stored distillation fingerprint, so it stays +conformance-locked. `queryFingerprint(query:extractFeatures:)` +builds the matching probe fingerprint at query time, with no model +inference. `DistilledHeader.parse` reads the stored header back out +of stored content. Confidence below 0.4 counts as failure. +Confidence between 0.4 and 0.7 marks the output uncertain. One +formatting subtlety is pinned. The `src=` field counts source +drawer IDs, not cluster members. The two differ in intra-item mode. +The `run` body stays one large function, on purpose. Splitting it +would thread the incidence matrix and vocabulary through each +helper signature. + +## PairingHandshake.swift + +This file gives the pairing handshake. These are the protocol +steps that let two estates establish a shared fingerprint basis, so +their memories become comparable for federation. + +Fingerprints are only comparable under the same hyperplane family, +and each estate normally has its own. The handshake fixes that +without a negotiation round trip. The two estates first exchange a +32-byte nonce, out of band. +`generateSharedFamily(nonce:estateA:estateB:density:)` then lets +each side independently derives the same shared family. This +works because the SplitMix64 seed is a pure function of the nonce +and the two estate UUIDs. `sharedFamilyKey(case:peerEstate:)` +finds the canonical manifest key both sides store the family +under. `buildPairEvent(...)` and `buildUnpairEvent(...)` produce the +audit payloads that record the relationship. Dissolving a pairing +keeps the shared family, since historical "as of" queries must +still work. It does stop further sync, however. + +The subtlest line in the file is +`PairingNonce.seedWith(estateA:estateB:)`. It orders the two UUIDs by +raw byte comparison, never by their hexadecimal string form. ASCII +comparison of hex strings can rank two UUIDs in the opposite order +from their raw bytes. The Rust leg always compares raw bytes. A +string-ordered Swift would derive a different seed, and the two +devices would build incompatible families. The nonce length, exactly +32 bytes, is precondition-enforced. The FNV-1a constants handle the +seed and family-hash mixing. + +## TierContributionFingerprint.swift + +This file gives the tier contribution. It is the fixed 64-byte +payload an estate sends up the federation hierarchy. It summarizes +an estate's shareable rows as one OR-reduced fingerprint. + +`FederationCase` names the three tiers: household, fleet, and +industry. `build(estateUUID:case:shareableFingerprints:hlc:)` +OR-reduces the shareable fingerprints. This work is routed through +the platform kernel. Federation work thus uses the best +available SIMD backend. The math still stays commutative, +associative, and idempotent. The function wraps the result with the +estate UUID, the case, the row count, and the HLC, into a +`TierContribution`. + +`encode(_:)` and `decode(_:)` implement the canonical wire format. +Sixteen UUID bytes come first. Then come the case and row count, as +big-endian 32-bit integers. Then come the four fingerprint blocks, +as big-endian 64-bit integers. Then comes the packed HLC. The total +runs exactly 64 bytes, big-endian throughout. A comment records a +real caught bug. An earlier draft emitted the fingerprint in its +little-endian storage form. That draft diverged from Rust by 32 +bytes, until the cross-language conformance gate flagged it. This +layer neither signs nor checksums its output. Authenticity is a +federation-egress concern, applied at the share point. Privacy +noise is applied at the aggregator, and never by the contributor. + +## TierAscendingQuery.swift + +This file gives the local-scope steps of the tier-ascending +query protocol. It defines how a recall query travels to peer +estates. It also defines how the answers come back combined. Neither +step exposes any single estate's data. + +`TierAscendingQuery` models the query itself. It names the +originating estate, the named recall primitive, the target tier, the +privacy budget, and the query HLC. `computeLocal(query:dispatch:)` +runs the local primitive through an injected closure. This keeps the +package independent of the cognition layer. On the peer side, +`applyDPToContribution(_:budget:rngSeed:)` adds Laplace noise to +each score in the peer's result. The noise scale is one over +epsilon, drawn from a seeded SplitMix64. The function attaches a +fixed 95 percent confidence interval, plus or minus 1.96 times the +scale. This lets the requester know how blurry the contribution is. +`combine(local:peers:)` merges the exact local result with the +noised peer results. Scores sum per row. The order runs by +descending score, with row identity as the deterministic tie-break. +The widest confidence interval seen always wins. This choice is +conservative on purpose, since the least certain contribution bounds +the certainty of the union. + +`PrivacyLedger` tracks per-peer epsilon and delta consumption +against a daily budget. `canConsume(peer:query:)` gates a query. +`consume(...)` records it. `dailyReset()` starts a new day. +Networking, signing, and scheduling are explicitly the caller's job. +One parity nuance is worth knowing. The Rust leg zeroes the +per-lane distance breakdown in a noised contribution, to close a +side channel. The Swift reference forwards that breakdown unchanged. + +## DPORReduction.swift + +This file gives the differentially private OR-reduction used at +the aggregation point of federated queries. Many estates' +fingerprints go in. One aggregate fingerprint comes out, with +mathematical privacy protection layered in. + +`reduce(fingerprints:params:rngSeed:)` works per bit position. For +each of the 256 positions, it counts how many contributors set that +bit. It adds Laplace noise, at a scale of one over epsilon. The +noise comes from a SplitMix64 generator, seeded by the caller. This +keeps the noise reproducible for testing. It sets the output bit +only if the noised count reaches the k-anonymity threshold. Two +mechanisms stack here. Differential privacy means no observer can +tell whether any one estate contributed. The k-anonymity floor +defaults to three. It means no bit lights up from too few +contributors, even without adversarial analysis. Empty input returns +the zero fingerprint. + +`DPParameters` pins the defaults from the cookbook: epsilon 1.0, +delta 1e-9, and k 3, with validity preconditions. One documented +asymmetry stands out. Delta is validated and stored for API +completeness, but the Laplace mechanism itself does not consume it. +The `laplaceNoise(scale:rng:)` helper derives a uniform value from +the top 53 bits of the raw draw. This is the same convention the +package's other samplers use. The helper then inverts the Laplace +distribution's cumulative curve. + +## Rust Port and Conformance + +The `rust/` directory contains the second leg of the library: the +crate `substrate-ml`. It has one module per Swift file, thirty-eight +modules from `anomaly.rs` through `viz_graph_signals.rs`. These +modules are declared in `rust/src/lib.rs`, with the same layering +notes and the same production gate on the parked NMF variant. The +crate depends on `substrate-types`, `substrate-kernel`, and +`intellectus-lib`. This mirrors the Swift package manifest exactly. + +Most modules open with an explicit "mirror of" header, naming their +Swift source. The determinism-bearing files repeat the pinned +constants, seeds, draw orders, and tie-break rules, line for line. +Shared gates prove the agreement property. +`rust/tests/distillation_conformance.rs` locks four things: the +delta extractor, the typed decay weights, `featureHash`, and the +full pipeline. All four are checked against fixed vectors mirrored +in Swift's `DistillationConformanceTests`. The rule there is simple: +fix the algorithm, never the vector. +`rust/tests/float_simhash_kernel_equivalence.rs` proves the kernel +projection reproduces the oracle bit for bit, across seeds and +dimensions. `rust/tests/viz_graph_signals_tests.rs` exercises the +telemetry emit of all five graph algorithms, under a process-wide +lock. Several algorithms carry additional shared JSON vectors under +the engineering test harness. These vectors cover sampling, temporal +causality fold, NMF, and community detection. When you change either +leg, change both, and run both suites. The fixtures are the contract. diff --git a/packages/libs/SubstrateML/docs/OVERVIEW.md b/packages/libs/SubstrateML/docs/OVERVIEW.md new file mode 100644 index 0000000..7541513 --- /dev/null +++ b/packages/libs/SubstrateML/docs/OVERVIEW.md @@ -0,0 +1,251 @@ +--- +doc: OVERVIEW +package: SubstrateML +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateML/ActionOutcomeMatrix.swift + blob: 612ee840126c72dd0d07505147ba2991ac3bb0b9 + - path: Sources/SubstrateML/AnomalyDetection.swift + blob: ff7d227353ebfc69cf01b5c468b2612d307b8632 + - path: Sources/SubstrateML/AprioriMining.swift + blob: 189a4408f3c6e600a538a6cfc560d24688cbd2af + - path: Sources/SubstrateML/AssociationRuleMining.swift + blob: 9f20faa808883c065412face0063c7682eabbfd6 + - path: Sources/SubstrateML/AuditLogFold.swift + blob: 7423db7764c628cf5fd2aab0e9bb2d98d8c687bd + - path: Sources/SubstrateML/BradleyTerry.swift + blob: 63d335ee50ccee803f0508eb6f3135a29b6545db + - path: Sources/SubstrateML/CommunityDetection.swift + blob: bfba8e0255baeb66d4c79ccfaf34cd11e21a5ebd + - path: Sources/SubstrateML/CompositeDistance.swift + blob: a5f94c631d89c92d45b60e9da43be7fe6bdd19a2 + - path: Sources/SubstrateML/ConceptImplications.swift + blob: 1ac6ccbb70665532f135d2813f260cbbdad21716 + - path: Sources/SubstrateML/DeltaFeatureExtractor.swift + blob: 1ca396968e5e565096dbebb059b45b8fc3e5719f + - path: Sources/SubstrateML/DistillationPipeline.swift + blob: 82ce8bfaaa2e7a92077510ef239a6997a585c888 + - path: Sources/SubstrateML/DistillationScorer.swift + blob: 2cdc6dfe121080d432859d17487f2c8d15fd2910 + - path: Sources/SubstrateML/DPORReduction.swift + blob: 3d3c79e8ce7694b8308bb1c970e86a24de0446b0 + - path: Sources/SubstrateML/EigenvalueCentrality.swift + blob: f427430c34add289231520ba19ae4781316cbd13 + - path: Sources/SubstrateML/FeatureExtractors.swift + blob: 7e6d388a588ff7bec04be631948387a086a253ed + - path: Sources/SubstrateML/FFT.swift + blob: fdfaceeb902428149ead96107ed69c308bca0fff + - path: Sources/SubstrateML/FloatSimHash.swift + blob: 39cd2c4c36492c7cbaf55d6d22a2f32e273cfbae + - path: Sources/SubstrateML/FormalConceptAnalysis.swift + blob: 9cdf9613b30bf2e3a9a85088489c0e2b683a8a77 + - path: Sources/SubstrateML/InformationTheory.swift + blob: f7ba2a27904ceb19eef0fb5b06d1c5aad8b3894b + - path: Sources/SubstrateML/JacobiSVD.swift + blob: fcd924ff0a8409f224b56e7229f59659a6b5f51e + - path: Sources/SubstrateML/LatticeDistance.swift + blob: a7b96a376dfba0c815e4dc91962da1ccaeb2ac1d + - path: Sources/SubstrateML/LLMCalibrationCurve.swift + blob: 4e01882a65570ccce72e2789c23e95c46b6399a7 + - path: Sources/SubstrateML/MatrixDecay.swift + blob: b9595064a54b79f8ddee343385af4b8ebdc7f3e5 + - path: Sources/SubstrateML/MomentSummary.swift + blob: e8c07841c8c3440deb0375500ebcd35a6f116bb4 + - path: Sources/SubstrateML/NMFAlternatingLeastSquares.swift + blob: d64aadc8db165cc3b9f51b4fb1c644a6d6a4ca3c + - path: Sources/SubstrateML/NMFDoubleFrobeniusSquared.swift + blob: 0a725d72b70120828cb5911de594b0278ca450e5 + - path: Sources/SubstrateML/PairingHandshake.swift + blob: cb608882478ec57a6e6ba1f7728646ff3ea610f1 + - path: Sources/SubstrateML/PartialStateRecall.swift + blob: ad7b42bc25f9c4f283d75593d2c42cd546ccf074 + - path: Sources/SubstrateML/RandomWalks.swift + blob: 99f40b1685a876d81f84ca38b4857e5c6ba4e1dd + - path: Sources/SubstrateML/RowAttributeView.swift + blob: f63cf544a87adde435bede527531d1a9414dad03 + - path: Sources/SubstrateML/Sampling.swift + blob: 84fcdcade559229003688a7fc151f5c0a6dac6e6 + - path: Sources/SubstrateML/ShingleSimilarity.swift + blob: bf9dbaa5b0d5a5ee4ec5dfe42a5d65c85bdd0326 + - path: Sources/SubstrateML/TemporalCausalityFold.swift + blob: d99d7358893eeae0a442e0bd4e52f8a9b1cd7f5e + - path: Sources/SubstrateML/TemporalCompression.swift + blob: f34dde022faadd40058e35b62da12c8985763e59 + - path: Sources/SubstrateML/TierAscendingQuery.swift + blob: 8681cbdae9af7facfe7792392dcee15c8f8fe0aa + - path: Sources/SubstrateML/TierContributionFingerprint.swift + blob: bb56b7a82532f8c324168bd81edff6bf68fb33e9 + - path: Sources/SubstrateML/TypedDecayWeighting.swift + blob: 82412f392848c378b5b28221bbddade9f03bf1bf + - path: Sources/SubstrateML/VizGraphSignals.swift + blob: 4f90110f2a19cb92f3d45baf3bc4d82ac9c3dc65 +--- + +# SubstrateML Overview + +## What This Library Does + +SubstrateML is the learning layer of the MOOTx01 substrate. MOOTx01 is +an on-device AI memory system. It stores what an AI observes over +time. It helps the AI recall that later. The substrate is the math +base under that store. It holds the types, the bit operations, and the +algorithms that every higher layer needs. + +The substrate splits into layered packages. `SubstrateTypes` holds +pure data types. One example is the 256-bit row fingerprint. +`SubstrateKernel` holds hot-path bit operations. These run on every +capture. SubstrateML is layer three. It holds the cold-path +algorithms: math that learns structure from stored memories. It does +not store memories itself. + +A memory in this system is a row. Each row has one fingerprint, one +classification anchor, and a set of bitmap fields. SubstrateML never +touches storage. Every function here takes plain values in and +returns plain values out. + +Most of these algorithms run during dreaming. Dreaming is the +system's idle-time maintenance cycle. A background daemon runs it +while the device sits quiet. The daemon clusters memories. It decays +old evidence. It summarizes clusters and re-scores the memory estate. + +An estate is one user's complete memory store. The algorithms here +find the estate's themes, its communities, and its habits. They find +its rules and its condensable clusters. They also forget on a +schedule, so old evidence fades over time. + +## The Problem It Solves + +An on-device memory system must learn without a cloud. Cloud machine +learning changes without notice. It needs a network. It sees private +data. SubstrateML avoids all three problems. It ships small, exact, +well-bounded reference algorithms. Every one of them runs entirely on +the device. + +The substrate must also learn the same way everywhere. MOOTx01 +estates can federate: separate devices share and compare results. Two +devices that factor the same matrix must get the same answer. Two +devices that mine the same rules must also agree. Otherwise shared +recall falls apart. + +SubstrateML holds one agreement property across two implementations. +A Swift leg serves Apple platforms. A Rust leg, in `rust/`, serves +everything else. Both legs share one canonical pseudo-random number +generator, SplitMix64. Both use the same pinned seeds. Both use the +same tie-breaking rules and the same arithmetic order. The result is +bit-identical output on both legs. Conformance fixtures gate every +change. A fixture is a recorded input and output pair that both legs +must reproduce exactly. + +Two library-wide rules protect that promise. First, SubstrateML never +reads a clock. Every timestamp comes from the caller instead. Second, +no algorithm here holds hidden state. Each one is a pure function or +a plain value type. Each one is safe to run from any thread. + +## How It Works + +The package holds thirty-eight source files. They form seven working +groups. + +**Fingerprint and distance math.** A fingerprint is a short fixed-size +code computed from content. Similar content gives similar +fingerprints. `FloatSimHash` projects float embedding vectors from +external models into the substrate's 256-bit fingerprint form. +`CompositeDistance` blends classification distance and fingerprint +distance into one score. Recall ranks candidates by that score. +`LatticeDistance`, `PartialStateRecall`, and `ShingleSimilarity` supply +specialized distances for narrower needs. `MomentSummary` and +`TemporalCompression` reduce many row fingerprints into one signature +per time window. That lets the system compare "what was going on +during this hour" as a single lookup. + +**Ingestion shaping.** `FeatureExtractors` turns raw ambient sensor +samples into fingerprinted rows. The samples come from health, +location, calendar, screen time, and telemetry sources. +`AuditLogFold` replays a row's append-only change log. That replay +reconstructs the row's state at any point in time. `RowAttributeView` +reshapes that same log into flat attribute lists. The pattern miners +consume those lists directly. + +**Learning and decay.** `MatrixDecay` applies exponential half-life +decay to every statistics matrix. This is the system's forgetting +mechanism. `ActionOutcomeMatrix` tracks which actions succeed. +`BradleyTerry` learns a per-row ranking strength from recall feedback. +`LLMCalibrationCurve` tracks how honest a model's confidence claims +are. `Sampling` provides deterministic Normal, Gamma, and Beta +samplers. Thompson-sampling decisions draw on these samplers. + +**Graph analytics.** The estate graph connects rows by association. +`CommunityDetection` runs Louvain clustering to find its communities. +`EigenvalueCentrality` scores each row's authority for keystone +recall. `RandomWalks` wanders the graph for exploratory recall. +`NMFAlternatingLeastSquares` factors the matrices into latent themes. +`AnomalyDetection` flags unusual values. `JacobiSVD` and `FFT` supply +deterministic linear algebra and rhythm analysis. Both sit beneath +semantic embeddings and periodicity detection. These five graph +algorithms emit telemetry signals when monitoring is enabled. The +signal names live in `VizGraphSignals`. + +**Pattern mining.** `AssociationRuleMining` and `AprioriMining` find +rules of the form "when A is set, B tends to be set." +`FormalConceptAnalysis` and `ConceptImplications` find exact groupings +and implications that always hold. `TemporalCausalityFold` mines +statistics of the form "X changed, then Y changed some minutes later." +`InformationTheory` supplies the entropy and divergence math behind +these miners. + +**Distillation.** Distillation compresses a cluster of related +memories into one condensed factoid. `DistillationScorer` decides +whether a cluster is coherent enough to compress. +`DeltaFeatureExtractor` and `TypedDecayWeighting` handle trends and +staleness in the underlying features. `DistillationPipeline` runs the +whole five-stage algorithm. It emits the factoid plus its +fingerprint. + +**Federation and privacy.** `PairingHandshake` lets two estates derive +a shared fingerprint basis. It needs no extra network round trip. +`TierContributionFingerprint` packs an estate's shareable summary into +a fixed sixty-four-byte wire format. `TierAscendingQuery` and +`DPORReduction` add differential-privacy noise and k-anonymity. Both +protections keep any single estate's contribution hidden from +aggregate answers. + +## How the Pieces Fit + +Figure 1 shows the library's topology. It shows the major parts and +how data moves between them. + +![Figure 1. Topology of SubstrateML](topology.svg) + +*Figure 1. Topology of SubstrateML. Rows, audit events, and sensor +samples enter on the left. The shaping layer feeds the four algorithm +families. Dashed regions mark the substrate packages below and the +telemetry seam. All outputs return to the calling kits, never to +storage.* + +SubstrateML has no single facade. Each algorithm family is its own +entry point. The consuming kits call the piece each one needs. These +kits are LocusKit, CognitionKit, GeniusLocusKit, NeuronKit, and the +dreaming daemon. A kit is a larger package that composes libraries +into a subsystem. Kits depend on libs. Libs never depend back on +kits. + +The package depends downward only. It depends on `SubstrateTypes` for +shared types. It depends on `SubstrateKernel` for dispatched bit +kernels. It depends on `IntellectusLib`, the zero-dependency telemetry +leaf. Monitoring is off by default. When it is off, every telemetry +emit costs one atomic boolean load and nothing more. + +## What Ships in the Package + +The package ships the thirty-eight Swift sources. It ships a matching +test suite. It also ships the Rust port in `rust/`, crate +`substrate-ml`, with one module per Swift file plus shared conformance +tests. There are no bundled data artifacts. Every input comes from the +caller. + +Determinism comes from pinned constants instead: seeds, thresholds, +half-lives, and bucket tables. These live in the sources. Conformance +vectors lock them in place. The same input therefore produces the +same result on every platform, every time. diff --git a/packages/libs/SubstrateML/docs/topology.svg b/packages/libs/SubstrateML/docs/topology.svg new file mode 100644 index 0000000..d90fde6 --- /dev/null +++ b/packages/libs/SubstrateML/docs/topology.svg @@ -0,0 +1,162 @@ + + + + + + + + + + + + + + + + SubstrateML: cold-path learning over the memory estate + + + + Caller inputs + rows · audit events + sensor samples · matrices + + + + Ingestion shaping + FeatureExtractors · AuditLogFold + RowAttributeView + + + + Fingerprint + distance + FloatSimHash · Composite/Lattice + PartialState · Moment/Temporal + + + + Graph analytics + Louvain · Centrality · Walks + NMF · SVD · FFT · Anomaly + + + + Pattern mining + Association · Apriori · FCA + Implications · TemporalFold + + + + Distillation + Scorer · DeltaExtractor + TypedDecay · Pipeline + + + + Learning + decay + MatrixDecay · BradleyTerry + ActionOutcome · Calibration + + + + Federation privacy + Pairing · TierContribution + TierQuery · DP OR-reduce + + + + Consuming kits + LocusKit · CognitionKit · NeuronKit + GeniusLocusKit · dreaming daemon + + + + + + + + + + + + + + + + + + Substrate layers below (types + dispatched kernels) + + SubstrateTypes + Fingerprint256 · HLC · SplitMix64 + + SubstrateKernel + SimHash project · orReduce256 + + + + IntellectusLib telemetry + + VizGraphSignals + community.assignment + centrality.score · nmf.factor + anomaly.flag · edge.decayed_weight + + + + + + + + + diff --git a/packages/libs/SubstrateTypes/docs/AGENT_MAP.md b/packages/libs/SubstrateTypes/docs/AGENT_MAP.md new file mode 100644 index 0000000..6af0b5f --- /dev/null +++ b/packages/libs/SubstrateTypes/docs/AGENT_MAP.md @@ -0,0 +1,292 @@ +--- +doc: AGENT_MAP +package: SubstrateTypes +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateTypes/AsOfCoordinate.swift + blob: b4433864e36b5803533266c97b153a836ee04adb + - path: Sources/SubstrateTypes/AuditEvent.swift + blob: fbfabeb4eaba4bf43b6e45d8c2b772d994a12b7b + - path: Sources/SubstrateTypes/BitwiseArithmetic.swift + blob: a6270b0f7395c17915c54fdc119e854ca5a44077 + - path: Sources/SubstrateTypes/BlockMask.swift + blob: 904c2179be96d5c20472f0f356947e82e9d8cf37 + - path: Sources/SubstrateTypes/ContentHash.swift + blob: 835744792dc191e824e676d8715c96111b4945ff + - path: Sources/SubstrateTypes/CountVector256.swift + blob: edabb34cb02005135d745ffa6da5541835159927 + - path: Sources/SubstrateTypes/Fingerprint256.swift + blob: 465607fe4164bb0dd94009dbbb20d6e766b973a8 + - path: Sources/SubstrateTypes/FloatSimHashPlanes.swift + blob: b5506e7d12fd153b721f4ca7a52a65a6667ea2f6 + - path: Sources/SubstrateTypes/FNV.swift + blob: f3d8874705ad66f513e49ccbf0c714980a1172e8 + - path: Sources/SubstrateTypes/GSetAuditLog.swift + blob: 1a4452e4c8a147d4fac299a3319b0cb4e70cb795 + - path: Sources/SubstrateTypes/Hamming.swift + blob: dc899742f77434910d51bb3cd830cc9603f56f0a + - path: Sources/SubstrateTypes/HLC.swift + blob: a74442251bd542e8f403029ee475c2ac9d7b91b0 + - path: Sources/SubstrateTypes/HyperplaneFamily.swift + blob: f30c4640629116b1952df48790ed6f5ae67e9025 + - path: Sources/SubstrateTypes/LatticeAnchor.swift + blob: 5f3ba24efa5a85109d20f3b61e6acb17e21cb0a5 + - path: Sources/SubstrateTypes/MatrixC.swift + blob: a8612ea7b09e6d4d93a4acbeedbe2b3b91776a74 + - path: Sources/SubstrateTypes/MatrixF.swift + blob: ed6e38a78c654562b04074878bf86686da0172c3 + - path: Sources/SubstrateTypes/MatrixO.swift + blob: c10fa0b14d8abd172503d0375dfab9302da32025 + - path: Sources/SubstrateTypes/MatrixT.swift + blob: cd31ca818be6446f8f1cd98c6b912c640a78c613 + - path: Sources/SubstrateTypes/MerkleDomain.swift + blob: 4e7d1888a04b2e9979e62c58b54ba698712e56d5 + - path: Sources/SubstrateTypes/MerkleRoot.swift + blob: 1979925f4646cdbed8785a43e2f2a14a7f9c0cac + - path: Sources/SubstrateTypes/NounType.swift + blob: 940d6c51329a66849e572a95ff8ae51648784f2f + - path: Sources/SubstrateTypes/ORReduce.swift + blob: 3f9f6e6ed30fb233f48457f95047f274bd271745 + - path: Sources/SubstrateTypes/RecallTypes.swift + blob: 7f01133df8495156fab87f1f4e441fb7867b3434 + - path: Sources/SubstrateTypes/Row.swift + blob: d7cf86f774dbccbbdf9daf1f3a5ed9be29fb24e8 + - path: Sources/SubstrateTypes/RowBitmaps.swift + blob: b879191a7076807c6fcd20e54cf806a639e63388 + - path: Sources/SubstrateTypes/RowState.swift + blob: 3f8c772be1fe1e3b44b90a5ea8681d4ce4326db7 + - path: Sources/SubstrateTypes/SimHash.swift + blob: e32a8b685c3db29408ed9e7fb21da4a7da10c467 + - path: Sources/SubstrateTypes/SnapshotId.swift + blob: b476bad15abecba2dacea225e0a7db7f445be777 + - path: Sources/SubstrateTypes/ThreeDBitTensor.swift + blob: 47b53642dda804afe8fbc0d863dc4ccd11d3ec55 + - path: Sources/SubstrateTypes/TimeRange.swift + blob: e1b7fbed94b6505b596ac5701cfd7c693e7d2425 +--- + +# AGENT_MAP: SubstrateTypes + +PURPOSE: layer 1 of the four-package substrate split. Pure data shapes (Row, +Fingerprint256, LatticeAnchor, HLC, RowState/RowBitmaps, AuditEvent) plus +canonical reference compute logically inseparable from those shapes +(Hamming, SimHash, FNV, ORReduce, BitwiseArithmetic, CountVector256). Zero +transcendentals, zero I/O, zero pinned data artifacts. Consumers needing +only substrate-shape (e.g. ConvergenceKit) depend on this package alone. + +DEPS: imports Foundation only (no intra-SDK imports: this is the dependency +floor). Imported by: SubstrateKernel, SubstrateML, SubstrateLib, EngramLib, +IntellectusLib, and the root app package (Package.swift). SubstrateLib +RETAINS the row-state automaton, the verb mechanics, and the full +multi-row substrate object: those are compute, not shape, and stay one +layer up. Rust port in `rust/src/` mirrors every module 1:1 (see DEPS note +in Rust Port and Conformance, DETAILS.md); no fixture directory: agreement +is verified by matched unit-test vectors in each module, not a shared +fixtures/ dir. + +ENTRY POINTS (most callers need only these): +- Row.swift:23 `struct Row`: the substrate's central data shape +- Fingerprint256.swift:43 `struct Fingerprint256`: 256-bit structural coordinate +- LatticeAnchor.swift:12 `struct LatticeAnchor`: topic coordinate (hash of a lattice code + Q-ID) +- HLC.swift:37 `struct HLC`: ordering coordinate across replicas +- Hamming.swift:37 `Hamming.distance(_:_:blocks:)`: structural-similarity primitive +- GSetAuditLog.swift:134 `struct GSetAuditLog`: CRDT audit trail + +## Symbol Table + +### Row shape and lifecycle +- Row.swift:19 `typealias RowId = UUID`: wire-identical to Rust `RowId(u128)` +- Row.swift:23 `struct Row: Sendable`: id, nounType, state, 3 bitmap columns, fingerprint, latticeAnchor, lineageId?, content? +- NounType.swift:12 `enum NounType: UInt8`: drawer=0/tunnel=1/kgFact=2/diaryEntry=3/proposal=4/association=5/learnedReference=6/ambientSample=7; wire-stable, never renumber +- RowState.swift:35 `enum RowState: UInt8`: 10 scale-gapped states (active=0..accepted=3, superseded=16..expired=19, rejected=32/tombstoned=33) +- RowState.swift:63 `enum RowStateCluster: UInt8`: a=active/becoming, b=superseded/historical, c=terminal +- RowState.swift:98 `RowState.cluster: RowStateCluster`: `(rawValue >> 4) & 0x3`; canonical active/retired partition, never re-derive by hand +- RowState.swift:90 `RowState.activeClusterUpperBoundRaw = 16`: the one boundary constant for storage-layer predicates that can't call `cluster` +- RowState.swift:115 `RowState.cluster(ofRawState:)`: classify from a raw byte, nil if undefined +- RowState.swift:122 `enum RowVerb: String`: 12 verbs the (SubstrateLib) automaton accepts +- RowState.swift:137 `enum RowStateError`: `.illegalTransition(RowState, RowVerb)` / `.violatesInvariant(String)` +- RowBitmaps.swift:37 `struct RowBitmaps: Sendable, Hashable, Codable`: adjective/operational/provenance Int64 columns +- RowBitmaps.swift:65 `RowBitmaps.field(_:) -> UInt8`: 36-field/6-bit uniform-grid view over the 3 named Int64 columns; UInt64 cast before shift (avoids Int64 arithmetic-shift sign bleed) +- RowBitmaps.swift:110 `RowBitmaps.fieldValues() -> [(field,value)]`: feeds MatrixO.applyRow / MatrixT.applyPair +- RowBitmaps.swift:151 `struct BitVector216: Sendable, Hashable`: dense 216-bit view; feeds MatrixF +- RowBitmaps.swift:167 `BitVector216.init(rowBitmaps:)`: real-row path, bits 60-71 always zero +- RowBitmaps.swift:191 `BitVector216.init(presenceBytes:)`: raw 27-byte harness/wire path, can set bits 60-71 + +### Lattice anchor (topic coordinate) +- LatticeAnchor.swift:12 `struct LatticeAnchor: Hashable, Sendable`: udcCode: UInt64, qidPointer: UInt64 (0 = null) +- LatticeAnchor.swift:21 `isNull: Bool`: both halves zero +- LatticeAnchor.swift:41 `LatticeAnchor.udc(_:)`: FNV-1a64 hash of a UDC string, null Q-ID +- LatticeAnchor.swift:50 `LatticeAnchor.udcQid(_:qid:)`: hashes both; empty qidString ⇒ same as `udc(_:)` + +### Time coordinates +- HLC.swift:37 `struct HLC: Hashable, Sendable, Codable, Comparable`: (physicalTime, logicalCount, nodeID), lexicographic order +- HLC.swift:58 `HLC.zero`: sentinel earliest value +- HLC.swift:77 `HLC.advanced()`: bump logicalCount, hold physical/node +- HLC.swift:99 `HLC.packed: UInt64` / :108 `init(packed:)`: lossy 8-byte federation form (40-bit ms ≈ 34yr range) +- HLC.swift:206 `HLC.wireBytes` / :224 `init(wireBytes:)`: lossless 16-byte form +- HLC.swift:125 `struct HLCGenerator: Sendable`: per-replica HLC state machine (Kulkarni et al. 2014) +- HLC.swift:139 `HLCGenerator.send(now:)`: local-event timestamp; strictly monotonic per replica +- HLC.swift:156 `HLCGenerator.receive(remote:now:)`: merge rule on message receipt; underlies GSetAuditLog convergence +- TimeRange.swift:11 `struct TimeRange: Sendable, Equatable`: closed [start,end] HLC interval; init preconditions end >= start +- TimeRange.swift:24 `contains(_:) -> Bool`: inclusive both ends +- AsOfCoordinate.swift:18 `enum AsOfCoordinate: Hashable, Sendable, Codable`: `.present` | `.asOf(HLC)`; enum discriminant prevents zero-HLC ⇔ "now" ambiguity +- SnapshotId.swift:15 `struct SnapshotId: Hashable, Sendable, Codable`: UUID wrapper, type-distinct from RowId/drawer/estate ids + +### Fingerprint construction +- Fingerprint256.swift:43 `struct Fingerprint256: Hashable, Sendable, Codable`: block0..block3: UInt64 (256 bits); block0=Bitmap-LSH, block1=Lattice-LSH, block2=Lineage+Temporal, block3=Channel+Source +- Fingerprint256.swift:59 `Fingerprint256.zero`: OR-identity / null-block value +- Fingerprint256.swift:88 `with(bit:set:) -> Fingerprint256`: value-semantic bit set/clear (replaces old mutating setBit) +- Fingerprint256.swift:120 `union(_:) -> Fingerprint256`: bitwise OR +- Fingerprint256.swift:153 `Fingerprint256.fromBits(_:)`: from [Bool] of 256 +- Fingerprint256.swift:166/178 `wireBytes` / `init(wireBytes:)`: 32-byte LE canonical form; `toBytes()`/`fromBytes(_:)` are throws-free aliases +- Fingerprint256.swift:252 `zip4(_:_:)`: pairwise per-block binary op; the ONE unroll every block-wise op in this package (and Hamming/ORReduce/BitwiseArithmetic) must route through +- Fingerprint256.swift:270 `Fingerprint256.reduce4(_:_:)`: block-wise fold over a fingerprint sequence +- Fingerprint256.swift:281 `map4(_:)`: per-block unary op +- Fingerprint256.swift:296 `popcount() -> Int`: sum of 4 block popcounts +- Fingerprint256.swift:313/331 `zip4Batch` / `map4Batch`: vectorize across row arrays, not across one fingerprint's blocks +- HyperplaneFamily.swift:30 `struct Hyperplane: Sendable, Codable, Equatable`: ±1 plane as (positiveMask, negativeMask) bit-pair; 0/0 = sparse zero entry +- HyperplaneFamily.swift:53 `sign(over:) -> Bool`: popcount(v&pos) > popcount(v&neg); tie ⇒ false +- HyperplaneFamily.swift:68 `struct HyperplaneFamily: Sendable, Codable, Equatable`: exactly 64 planes, fixed blockIndex 0..3 + inputBitLength (192 for block0, 64 for 1-3) +- HyperplaneFamily.swift:95 `HyperplaneFamily.generate(seed:blockIndex:inputBitLength:density:)`: deterministic from 32-byte seed; CONSTITUTIONAL: seeds fixed at estate creation, never rotate +- HyperplaneFamily.swift:147 `canonicalHash() -> UInt64`: FNV-1a over canonical wire form, for pairing-audit labels +- HyperplaneFamily.swift:247 `HyperplaneFamily.blockFamilies(baseSeed:density:)`: canonical 4-family generator; fixes prior bug of reusing 1 seed across blocks + assuming uniform 64-bit width +- HyperplaneFamily.swift:261 `diversifiedSeed(base:blockIndex:)` / :273 `expandSeed64(_:)`: per-block seed derivation (SplitMix64-style) +- FloatSimHashPlanes.swift:30 `struct FloatSimHashPlanes: Sendable, Equatable`: materialized ±1 planes for the FLOAT-input SimHash path; lives here so SubstrateKernel (applies) and SubstrateML (generates) both reach it without a layering inversion +- SimHash.swift:37 `SimHash.block(over:family:) -> UInt64`: one 64-bit block: bit k = sign() +- SimHash.swift:52 `SimHash.fingerprint(bitmapInput:latticeInput:lineageTemporalInput:channelSourceInput:families:)`: assembles all 4 blocks; families order MUST be H_0..H_3 +- SimHash.swift:74 `SimHash.fingerprintBatch(...)`: reference loop; hardware backends must match bit-for-bit +- SimHash.swift:102 `SimHash.fingerprint(fromSubhashes:hyperplanes:)`: 4-subhash convenience path +- SimHashInput (SimHash.swift:123): :128 `bitmap(...)`, :140 `lattice(...)`, :156 `lineageTemporal(...)`, :178 `channelSource(...)`: fixed bit-offset input-vector assemblers per cookbook §3.2-3.5; offset mismatch ⇒ silently incompatible fingerprints + +### Fingerprint algebra +- Hamming.swift:21 `typealias HammingDistance = Hamming` +- Hamming.swift:37 `Hamming.distance(_:_:blocks:) -> Int`: XOR+popcount; `.all` fast path is `zip4(^).popcount()` +- Hamming.swift:54 `Hamming.similarity(_:_:blocks:) -> Double`: 1 - distance/max, max = 64*blockCount +- BlockMask.swift:26 `struct BlockMask: OptionSet`: .block0/.block1/.block2/.block3, .all, .none; replaced a Set-per-call allocation hot-path +- BlockMask.swift:42 `blockCount: Int`: popcount of rawValue +- BitwiseArithmetic.swift:31 `intersect(_:_:)`: AND (zip4 &) +- BitwiseArithmetic.swift:45 `difference(_:_:)`: XOR (zip4 ^) +- BitwiseArithmetic.swift:63 `prototype(_:) -> Fingerprint256`: cohort majority vote via CountVector256.fold(...).majorityVote() +- BitwiseArithmetic.swift:77 `indirect enum FingerprintBuilder`: literal/intersect/difference/prototypeOf; `.evaluate()` walks the tree +- ORReduce.swift:31 `ORReduce.reduce(_:) -> Fingerprint256`: OR-fold, zero for empty (identity); delegates to `Fingerprint256.reduce4(_:|:)` +- ORReduce.swift:46 `ORReduce.reduce(_:blocks:defaults:)`: block-subset OR, unselected blocks pass through `defaults` +- CountVector256.swift:41 `struct CountVector256: Sendable, Equatable, Codable`: counts:[UInt32] (256), n:UInt32; wrapping arithmetic (`&+=`) +- CountVector256.swift:52 `CountVector256.zero`: fold/merge identity +- CountVector256.swift:75 `accumulate(_:)`: fold one fingerprint (leaf step) +- CountVector256.swift:94/103 `merge(_:)` / `+`: commutative/associative tree-fold step; order-independent +- CountVector256.swift:120 `majorityVote() -> Fingerprint256`: bit set iff `2*count > n` (strict; exact tie does NOT set); n==0 ⇒ zero +- CountVector256.swift:134 `profile() -> [Float]`: per-bit Bernoulli parameter, not stored, recomputed on demand +- CountVector256.swift:148 `CountVector256.fold(_:)`: reference whole-array accumulate; kernel-layer vectorized backends gate against this +- FNV.swift:25 `FNV.hash64(_:) -> UInt64`: offset 0xCBF29CE484222325, prime 0x100000001B3 +- FNV.swift:40 `FNV.hash32(_:) -> UInt32`: INDEPENDENT hash family, not a truncation of hash64 +- FNV.swift:57 `FNV.hash16(_:) -> UInt16`: low-16 fold of hash64 (NOT a from-scratch FNV variant) + +### Integrity hashing +- MerkleDomain.swift:20 `enum MerkleDomain`: leaf=0x00, interior=0x01, tombstone=0x02, commitment=0x03; conformance-frozen forever +- ContentHash.swift:20 `struct ContentHash: Hashable, Sendable, Codable`: 32-byte SHA-256 leaf-payload digest; private storage, `bytes`/`hexString` accessors +- ContentHash.swift:45 `ContentHash.tombstone`: literal bytes (SHA256 of MerkleDomain.tombstone); literal because layer-1 can't import SubstrateKernel's SHA256 +- MerkleRoot.swift:16 `struct MerkleRoot: Hashable, Sendable, Codable`: 32-byte subtree-summary hash; NOT interchangeable with ContentHash (type-distinct on purpose) +- MerkleRoot.swift:41 `MerkleRoot.empty`: literal bytes (SHA256 of MerkleDomain.interior), same layering reason + +### Audit trail (CRDT) +- AuditEvent.swift:15 `struct AuditEvent: Sendable`: eventID+hlc = idempotence key; before/afterBitmaps, before/afterLatticeAnchor, actor, reason? +- AuditEvent.swift:63 `withReason(_:) -> AuditEvent`: attach a reason post hoc (AuditGate.admit itself is reason-less) +- GSetAuditLog.swift:35 `struct AuditEntry: Hashable, Sendable, Codable`: id = 32-byte content hash (SHA-256 over wire fields); dedupe key +- GSetAuditLog.swift:62 `enum AuditVerb: String`: 9 cookbook verbs + migrate/dreamCompact +- GSetAuditLog.swift:95 `enum AuditValue: Hashable, Sendable, Codable`: bitmap/string/fingerprint/integer; HAND-WRITTEN Codable ⇒ externally-tagged `{"bitmap":42}` shape matching Rust serde, NOT Swift's default `{"bitmap":{"_0":42}}` +- GSetAuditLog.swift:134 `struct GSetAuditLog: Sendable, Codable`: internal store keyed by id (O(1) dedupe); WIRE FORM is a sorted array (`{"entries":[...]}`), not the hash map +- GSetAuditLog.swift:180 `add(_:)`: idempotent insert +- GSetAuditLog.swift:187 `merge(_:)`: CRDT join = set union; commutative/associative/idempotent +- GSetAuditLog.swift:198 `orderedEntries`: full HLC-order replay (drives projection) +- GSetAuditLog.swift:204 `entries(forRow:)`: HLC-order, one row (drives row-state automaton in SubstrateLib) +- GSetAuditLog.swift:212 `entries(since:)`: HLC-order, exclusive cutoff (sync delta) + +### Population-statistics matrices +- MatrixF.swift:24 `struct MatrixF: Sendable, Equatable`: 216-cell Int64 field-presence counts; NO decay +- MatrixF.swift:31 `fieldCount`/`bitsPerField`/`cellCount`: ALIASES onto RowBitmaps' constants (must match by definition) +- MatrixF.swift:76 `applyRow(delta:bitVector:)`: +1 capture / -1 expunge / two calls for mutate +- MatrixC.swift:26 `struct MatrixC: Sendable, Equatable`: 216-cell Float32 marginal probabilities; derived only, NO decay +- MatrixC.swift:67 `MatrixC.derive(from:nRows:)`: Int64→Double→Float32 cast order is the cross-language bit-identity contract; nRows==0 ⇒ all zero +- MatrixO.swift:38 `struct CooccurrenceKey: Hashable, Comparable, Sendable`: packed UInt32 (fieldI|valueI|fieldJ|valueJ, 8 bits each) +- MatrixO.swift:70 `struct MatrixO: Sendable, Equatable`: sorted sparse cell list (NOT a Dictionary: deterministic iteration/serialization); decay half-life 365 days (applied elsewhere, by MatrixDecay) +- MatrixO.swift:165 `applyRow(delta:fieldValues:)`: ALL ordered pairs incl. i==j; NOT required symmetric in storage (unordered pair ⇒ 2 cells) +- MatrixT.swift:43 `struct CausalityKey: Hashable, Comparable, Sendable`: packed UInt64, adds lagBucket (0..7) to CooccurrenceKey's shape +- MatrixT.swift:84 `struct MatrixT: Sendable, Equatable`: ASYMMETRIC by design (source→target ≠ target→source); decay half-life 90 days +- MatrixT.swift:89 `bucketEdgesMinutes = [1,2,4,8,16,32,64,128]`: log-spaced lag buckets +- MatrixT.swift:99 `MatrixT.lagBucket(forMinutes:)`: nil outside [1,256); applyPair is then a no-op +- ThreeDBitTensor.swift:36 `struct ThreeDBitTensor: Sendable`: 6 bit-slices (one per bit position), row-major within each slice; hot-path layout, ~27MiB at 1M rows +- ThreeDBitTensor.swift:71 `setValue(row:field:value:)`: precondition value < 64 (6-bit width, I-6) +- ThreeDBitTensor.swift:107 `scanFieldEquals(field:value:) -> [UInt8]`: O(rowCount/8) per bit-slice, NOT yet word-vectorized +- ThreeDBitTensor.swift:140 `reserveCapacity(_:)`: grow-only, no-op if not larger, preserves existing data + +### Recall wire vocabulary +- RecallTypes.swift:55 `struct RecallScore: Equatable, Sendable`: (RowId, Float32); score meaning is PER-PRIMITIVE, normalize before combining +- RecallTypes.swift:69 `struct DistanceBreakdown: Equatable, Sendable`: lattice/fingerprint/temporal/bitmap contributions, each [0,1] +- RecallTypes.swift:89 `struct RecallResult: Sendable`: rows + breakdown + confidenceInterval? + primitiveName (drives RRF/MMR composition) +- RecallTypes.swift:112 `struct RowProjection: Sendable`: rowId/captureHLC/fingerprint/lattice/bitmaps/rowState; DELIBERATELY omits verbatim content + rung-2 metadata + +## INVARIANTS / GOTCHAS + +- LAYERING FLOOR: this package imports Foundation only. Never add an + intra-SDK import here: that is what would invert the four-package split + (SubstrateTypes → SubstrateKernel → SubstrateML, with SubstrateLib + orchestrating all three). `ContentHash.tombstone` and `MerkleRoot.empty` + are byte literals, NOT computed SHA-256 calls, for exactly this reason; + a SubstrateKernel bridge test verifies each literal against a live hash. +- `Fingerprint256.zip4(_:_:)` is the ONE place the four-block unroll is + expressed. `Hamming.distance`, `ORReduce.reduce`, and + `BitwiseArithmetic.intersect/difference` all delegate to it. Do not + reintroduce a hand-unrolled block0/block1/block2/block3 loop anywhere else. +- `RowBitmaps.field(_:)` casts to `UInt64` before shifting. Do not shift the + raw `Int64`: arithmetic (sign-extending) right shift on a negative Int64 + bleeds the sign bit into the field value. Also guards `shift >= 64` + explicitly; Swift masks shift amounts by 63 and would otherwise compute + the wrong result for fields 11 (shift=66) silently. +- `RowState`'s ten raw values are scale-gapped (0-3 / 16-19 / 32-33) on + purpose so `cluster` is `(raw >> 4) & 0x3`. Never classify active-vs- + retired by a hand-rolled numeric boundary except via the one named + constant `activeClusterUpperBoundRaw`, which exists only for storage + predicates that cannot call `cluster(ofRawState:)`. +- `HyperplaneFamily` seeds are CONSTITUTIONAL: fixed at estate creation, + never rotated. Rotating changes every fingerprint the estate has ever + produced and breaks CRDT convergence across replicas. `blockFamilies` + is the only sanctioned way to derive all four block families from one + base seed: do not reuse one un-diversified seed across blocks. +- `CountVector256.majorityVote()` uses STRICT `2*count > n`; an exact tie + does not set the bit. This convention is identical across every kernel + backend and both language ports: do not change the inequality direction. +- `GSetAuditLog`'s wire format is a sorted array (`{"entries":[...]}`), not + its internal `[id: AuditEntry]` dictionary. `AuditValue`'s Codable is + hand-written to match Rust serde's externally-tagged shape. Do not let + either fall back to Swift's default synthesized Codable. +- `MatrixF`/`MatrixC` have NO decay; `MatrixO` decays at 365-day half-life; + `MatrixT` decays at 90-day half-life. Decay application itself lives + outside this package (MatrixDecay); these types only hold and update raw + counts/derived marginals. +- `MatrixC.derive(from:nRows:)` casts Int64 → Double → Float32, in that + order, on both language ports. This is the cross-language bit-identity + contract for the correlation matrix; do not shortcut through Float32 + division directly. +- `MatrixT` is asymmetric by design: `(i,j)` and `(j,i)` are independent + cells. `MatrixO` is conceptually symmetric but stores both directions + from `applyRow`'s ordered-pair iteration (including the diagonal, i==j). +- `RowId` is a bare `typealias` for `UUID` (wire-identical to Rust's + `RowId(u128)`); `SnapshotId` is a real wrapper struct on purpose, so it is + NOT interchangeable with a bare UUID at the type level. Do not "simplify" + `SnapshotId` into another typealias. +- `ContentHash` and `MerkleRoot` are both 32-byte hashes but are distinct, + non-interchangeable types (payload digest vs. subtree summary). Same + pattern for `RecallScore.score`: meaning is per-primitive, never compare + raw scores across primitives without normalizing first. +- No pinned data artifacts ship in this package (contrast with LatticeLib). + Every value is either caller-supplied (a seed, a row's fields) or a pure + function of caller-supplied input. Reproducibility rests entirely on the + arithmetic being pinned and mirrored in `rust/src/`, checked by matched + unit-test vectors per module, not a shared fixtures directory. +- `Row`, `AuditEvent`, `RowState`/`RowVerb`/`RowStateError`, `NounType`, + and `LatticeAnchor` are pure data only. The row-state automaton + (transition table + validation), the verb implementations, and the + full multi-row substrate object all remain in SubstrateLib. Do not + add behavior to these types here. diff --git a/packages/libs/SubstrateTypes/docs/DETAILS.md b/packages/libs/SubstrateTypes/docs/DETAILS.md new file mode 100644 index 0000000..1fb2cb9 --- /dev/null +++ b/packages/libs/SubstrateTypes/docs/DETAILS.md @@ -0,0 +1,872 @@ +--- +doc: DETAILS +package: SubstrateTypes +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateTypes/AsOfCoordinate.swift + blob: b4433864e36b5803533266c97b153a836ee04adb + - path: Sources/SubstrateTypes/AuditEvent.swift + blob: fbfabeb4eaba4bf43b6e45d8c2b772d994a12b7b + - path: Sources/SubstrateTypes/BitwiseArithmetic.swift + blob: a6270b0f7395c17915c54fdc119e854ca5a44077 + - path: Sources/SubstrateTypes/BlockMask.swift + blob: 904c2179be96d5c20472f0f356947e82e9d8cf37 + - path: Sources/SubstrateTypes/ContentHash.swift + blob: 835744792dc191e824e676d8715c96111b4945ff + - path: Sources/SubstrateTypes/CountVector256.swift + blob: edabb34cb02005135d745ffa6da5541835159927 + - path: Sources/SubstrateTypes/Fingerprint256.swift + blob: 465607fe4164bb0dd94009dbbb20d6e766b973a8 + - path: Sources/SubstrateTypes/FloatSimHashPlanes.swift + blob: b5506e7d12fd153b721f4ca7a52a65a6667ea2f6 + - path: Sources/SubstrateTypes/FNV.swift + blob: f3d8874705ad66f513e49ccbf0c714980a1172e8 + - path: Sources/SubstrateTypes/GSetAuditLog.swift + blob: 1a4452e4c8a147d4fac299a3319b0cb4e70cb795 + - path: Sources/SubstrateTypes/Hamming.swift + blob: dc899742f77434910d51bb3cd830cc9603f56f0a + - path: Sources/SubstrateTypes/HLC.swift + blob: a74442251bd542e8f403029ee475c2ac9d7b91b0 + - path: Sources/SubstrateTypes/HyperplaneFamily.swift + blob: f30c4640629116b1952df48790ed6f5ae67e9025 + - path: Sources/SubstrateTypes/LatticeAnchor.swift + blob: 5f3ba24efa5a85109d20f3b61e6acb17e21cb0a5 + - path: Sources/SubstrateTypes/MatrixC.swift + blob: a8612ea7b09e6d4d93a4acbeedbe2b3b91776a74 + - path: Sources/SubstrateTypes/MatrixF.swift + blob: ed6e38a78c654562b04074878bf86686da0172c3 + - path: Sources/SubstrateTypes/MatrixO.swift + blob: c10fa0b14d8abd172503d0375dfab9302da32025 + - path: Sources/SubstrateTypes/MatrixT.swift + blob: cd31ca818be6446f8f1cd98c6b912c640a78c613 + - path: Sources/SubstrateTypes/MerkleDomain.swift + blob: 4e7d1888a04b2e9979e62c58b54ba698712e56d5 + - path: Sources/SubstrateTypes/MerkleRoot.swift + blob: 1979925f4646cdbed8785a43e2f2a14a7f9c0cac + - path: Sources/SubstrateTypes/NounType.swift + blob: 940d6c51329a66849e572a95ff8ae51648784f2f + - path: Sources/SubstrateTypes/ORReduce.swift + blob: 3f9f6e6ed30fb233f48457f95047f274bd271745 + - path: Sources/SubstrateTypes/RecallTypes.swift + blob: 7f01133df8495156fab87f1f4e441fb7867b3434 + - path: Sources/SubstrateTypes/Row.swift + blob: d7cf86f774dbccbbdf9daf1f3a5ed9be29fb24e8 + - path: Sources/SubstrateTypes/RowBitmaps.swift + blob: b879191a7076807c6fcd20e54cf806a639e63388 + - path: Sources/SubstrateTypes/RowState.swift + blob: 3f8c772be1fe1e3b44b90a5ea8681d4ce4326db7 + - path: Sources/SubstrateTypes/SimHash.swift + blob: e32a8b685c3db29408ed9e7fb21da4a7da10c467 + - path: Sources/SubstrateTypes/SnapshotId.swift + blob: b476bad15abecba2dacea225e0a7db7f445be777 + - path: Sources/SubstrateTypes/ThreeDBitTensor.swift + blob: 47b53642dda804afe8fbc0d863dc4ccd11d3ec55 + - path: Sources/SubstrateTypes/TimeRange.swift + blob: e1b7fbed94b6505b596ac5701cfd7c693e7d2425 +--- + +# SubstrateTypes Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in one order. +First comes the row shape and its lifecycle. Next come the two coordinate +systems a row carries: the lattice anchor and time. Then comes the +fingerprint construction pipeline and its algebra. After that comes +integrity hashing, the audit trail, and the population-statistics +matrices. The recall wire vocabulary closes the document. + +## NounType.swift + +This file provides `NounType`. It is an eight-case enum. It names the kind +of thing a row holds. A row can be a drawer, a tunnel, or a +knowledge-graph fact. It can be a diary entry, a proposal, or an +association. It can also be a learned reference or an ambient sample. + +The enum uses `UInt8` raw values, zero through seven, instead of a string. +A row's noun type is stored in a compact wire format alongside the rest of +the row. That format needs a small fixed-size number, not a string. The +file's opening comment states a rule. These values are wire-stable. They +must never be renumbered. A stored row's noun type is read back by its raw +number. Changing a number would silently reclassify every row already on +disk. + +## RowState.swift + +This file provides three pure-data types. They describe a row's +lifecycle. `RowState` names the ten states a row can be in. `RowVerb` +names the twelve mutations the row-state automaton accepts. +`RowStateError` is the typed failure a rejected transition returns. The +automaton that enforces which transitions are legal is compute, not data. +It stays one layer up, in SubstrateLib. This file only names the states +and verbs the automaton works with. + +The ten states use deliberately spaced raw values. They run zero to +three, then a gap, then sixteen to nineteen, another gap, then thirty-two +and thirty-three. This is not a plain zero-to-nine count. The spacing lets +any consumer recover a state's lifecycle cluster with one shift-and-mask +operation. No lookup table is needed. `RowState.cluster` reads +`(rawValue >> 4) & 0x3`. This classifies a state into one of three +clusters. Cluster A is active or becoming: active, pending, contested, +accepted. Cluster B is superseded or historical: superseded, decayed, +withdrawn, expired. These are retired but revivable. Cluster C is +terminal: rejected, tombstoned. These are retired and final. +`RowState.isActiveCluster` is the convenience most callers want. It gives +a single true-or-false answer to "is this row currently believed." + +`RowState.cluster(ofRawState:)` performs the same classification from a +raw `UInt8`. It works even when code has only read a bare byte out of a +persisted bitmap and has not decoded it yet. +`RowState.activeClusterUpperBoundRaw` names one boundary value: sixteen. +A storage-layer predicate can compare against this directly. One example +is a SQL `WHERE` clause, when it cannot call a Swift function. The file is +explicit about one thing. Code holding a decoded `RowState` should prefer +`cluster` over a hand-rolled numeric comparison. A future state added +inside one of the gaps would silently misclassify under a magic-number +boundary. It would not misclassify under the shift-and-mask. + +`RowState.description` and `RowStateError.description` supply exact +lowercase English strings. Examples are "active", "pending", and the +compact "active --reject-->" form. Downstream parsers depend on these +strings for parity with the Rust port's `Display` implementation. Both are +written as explicit `switch` statements. They do not use the default +`String(describing:)`. Calling that default from inside a type's own +`description` recurses forever. + +## RowBitmaps.swift + +This file provides `RowBitmaps`. It is the named layout over a row's +three bitmap columns. The file also provides `BitVector216`, a dense +216-bit view over that same data, for consumers that want to treat it as +a flat grid. + +A row carries three `Int64` columns: adjective, operational, and +provenance. Each column packs several independently named fields at +specific bit ranges. For some purposes, such as updating the +population-statistics matrices, it helps to think of these three columns +as one uniform grid. That grid holds thirty-six fields of six bits each. +The real fields are not laid out quite that uniformly, though. +`RowBitmaps.field(_:)` performs this translation. Given a field index +from zero to thirty-five, it locates the right column and bit offset. It +returns that field's six-bit value. The function carries two defensive +casts, and the source comment explains both in detail. One is a +shift-amount guard: Swift silently wraps an out-of-range shift instead of +returning zero. The other is a cast to `UInt64` before shifting: +right-shifting a negative `Int64` sign-extends. That would leak the sign +bit into a field's value. + +`RowBitmaps.bit(field:bit:)` reads one bit out of a field's six-bit value. +`RowBitmaps.fieldValues()` returns all thirty-six (field, value) pairs in +order. `MatrixO.applyRow` and `MatrixT.applyPair` consume this list to +update the co-occurrence and causality matrices. `RowBitmaps.bitVector()` +builds a `BitVector216` from a live row. + +`BitVector216` has two constructors for two different sources of the same +216-bit shape. `init(rowBitmaps:)` builds it from a real row, through +`RowBitmaps.field(_:)`. Real rows never set bits above position +seventy-one of the abstract grid. This path is therefore safe for +updating `MatrixF` from row capture and expunge events. +`init(presenceBytes:)` builds it directly from a raw 27-byte pattern. That +pattern may set any of the 216 bits, including ones no real `RowBitmaps` +value can represent. A conformance-test harness vector uses this second +shape. `bit(at:)` and `bit(field:bit:)` read one bit, by absolute index +or by (field, bit) coordinate. + +## Row.swift + +This file provides `Row`, the substrate's central data shape. It also +provides `RowId`, a type alias for `UUID`. Every place the substrate +vocabulary calls for a row identifier uses `RowId`. + +`RowId` is a plain alias, not a wrapper type. A `UUID`'s 16-byte +big-endian wire form is already byte-identical to the Rust port's +`RowId(u128)` newtype. No translation step is needed at the boundary. +Callers can use `UUID`'s existing API directly. + +`Row` itself is a value type. It holds one row's complete current state: +its identifier, noun type, lifecycle state, three bitmap columns, +fingerprint, and lattice anchor. It also holds two optional fields. One is +a lineage identifier, linking a row to the row it was derived from. The +other is the verbatim content. The file is explicit that `Row` is pure +data. It has no logic and no input or output. The full substrate object +that bundles many rows together, with the audit log and the statistics +matrices, remains in SubstrateLib. So do the verb functions that mutate a +row's state. `Row` only says what a row looks like at rest. + +## LatticeAnchor.swift + +This file provides `LatticeAnchor`. It is the sixteen-byte reference a +row carries to its position in the classification lattice. It packs an +eight-byte hash of a lattice code plus an eight-byte hash of a concept +identity, called a Q-ID. Either half can be zero when absent. + +A lattice anchor does not store the lattice code text itself. It stores +a deterministic hash of it. `LatticeAnchor.fnv1a64(_:)` is the private +FNV-1a 64-bit hash used for this purpose. It is the exact FNV-1a +algorithm, matching `FNV.hash64` elsewhere in the package. Two callers +hashing the same code string always agree. The result also matches the +Rust port bit for bit. + +`LatticeAnchor.udc(_:)` builds an anchor from a lattice code string +alone, with a null Q-ID pointer. `LatticeAnchor.udcQid(_:qid:)` builds an +anchor that carries both the lattice code and a specific concept +identity. The file explains why this second form matters. Without it, +every row about a broad subject collapses onto one anchor value. That +loses the distinction between different things classified under the same +broad code. An empty Q-ID string yields the same null pointer as +`udc(_:)`. The two constructors therefore agree when no concept identity +is available. `LatticeAnchor.isNull` reports whether both halves are +zero. This is the anchor's way of saying "unclassified." + +## HLC.swift + +This file provides `HLC`, the Hybrid Logical Clock timestamp. It orders +events across replicas of an estate without needing synchronized wall +clocks. The file also provides `HLCGenerator`, the per-replica state +machine that produces HLC values. + +An `HLC` has three fields, compared in order. `physicalTime` is +milliseconds since the Unix epoch. `logicalCount` is a counter that +advances when physical time does not. `nodeID` is a per-replica +tiebreaker. `HLC.<` compares these three fields in that order. This gives +any two HLC values an unambiguous order. That holds even when two +replicas' clocks briefly agree, or run backward relative to each other. +`HLC.zero` is the earliest possible value, used as a sentinel. +`HLC.advanced()` returns a copy with the logical counter incremented. It +holds physical time and node fixed. + +`HLCGenerator.send(now:)` produces the timestamp for a locally originated +event. If the wall clock has moved forward since the last call, it adopts +the new time and resets the logical counter to zero. Otherwise it holds +the physical time and bumps the logical counter. Either way, every +timestamp this replica emits is strictly greater than the last. +`HLCGenerator.receive(remote:now:)` implements the HLC paper's merge rule +for an event arriving from another replica. It takes the maximum of the +local clock, the remote clock, and the current wall time. It sets the +logical counter according to which input was tied for that maximum. This +is the operation that lets two replicas exchange audit events. They +converge on one shared sense of "what happened before what." The +convergence proof in `GSetAuditLog.swift` depends on this operation. + +`HLC.packed` and `init(packed:)` convert to and from a lossy 8-byte form. +Federation tier-contribution messages use this form. It trades precision, +forty bits of milliseconds, for a smaller wire size. That is roughly +thirty-four years of range. `HLC.wireBytes` and `init(wireBytes:)` are +the lossless 16-byte round trip used elsewhere. The file also carries a +`nodeId`-spelled initializer overload. It exists only so older call sites +written against that casing keep compiling unchanged. + +## TimeRange.swift + +This file provides `TimeRange`, a closed interval `[start, end]` of two +HLC values. + +`TimeRange.init(start:end:)` rejects an interval whose end precedes its +start, with a precondition. An inverted range has no sensible meaning for +any caller. `TimeRange.contains(_:)` reports whether a given HLC falls +within the closed interval. Both endpoints are included. This is the +predicate a caller uses to test "did this event happen during this +window." The file notes that the higher-level primitive consuming these +windows, `MomentSummary`, stays in SubstrateLib. This type is only the +interval shape. + +## AsOfCoordinate.swift + +This file provides `AsOfCoordinate`, a two-case enum. It says whether a +read should return the live state or the state as of a specific HLC. The +first case is `.present`. The second case is `.asOf(hlc)`. + +The file explains the bug this type prevents. Suppose "present" were +represented by a sentinel HLC value such as zero. A caller could not then +distinguish "give me now" from "give me the state at the dawn of time." +Making the choice an explicit enum discriminant removes that ambiguity by +construction. The custom `Codable` conformance encodes the case as an +explicit `"kind"` field, either `"present"` or `"asOf"`. For the second +case, it also encodes the HLC itself. This is not Swift's default enum +encoding. The wire form stays stable and self-describing across the Swift +and Rust legs. + +## SnapshotId.swift + +This file provides `SnapshotId`, a `UUID` wrapper. It identifies one +point-in-time snapshot of an estate. + +Unlike `RowId`, which is a bare type alias, `SnapshotId` is a distinct +struct. The file states the reason. A snapshot identifier must not be +substitutable for a drawer, node, or estate identifier at the type level. +This holds even though all of these are UUIDs underneath. +`init(uuidString:)` and the `uuidString` accessor round-trip through the +standard UUID string form. `Codable` conformance encodes and decodes that +same string form. It throws `SnapshotIdError.invalidUUID` on a malformed +string, rather than crashing. + +## BlockMask.swift + +This file provides `BlockMask`, an `OptionSet`. It selects any subset of +the fingerprint's four 64-bit blocks. + +Before this type existed, callers passed a `Set` to select blocks. +That allocated memory on every call, a measured hot-path cost in +nearest-neighbor search over large row collections. `BlockMask` replaces +that with four fixed bit flags, `.block0` through `.block3`. It adds an +`.all` convenience covering every block, and an empty `.none`. It is a +plain integer under the hood. So membership tests are branchless and need +no allocation. `BlockMask.blockCount` counts how many blocks are +selected, zero through four. `Hamming.similarity` uses this count to +compute the correct maximum-distance denominator for a partial-block +comparison. + +## Fingerprint256.swift + +This file provides `Fingerprint256`, the substrate's 256-bit structural +fingerprint. It also provides the pure combinators that other files in +this package build on. + +A fingerprint is four independent 64-bit blocks: `block0` through +`block3`. Each is a SimHash over a different aspect of a row. One covers +its bitmaps. One covers its lattice position. One covers its lineage and +timing. One covers its channel and source. The file is careful to +distinguish a fingerprint from a lattice anchor. A fingerprint is the +coordinate for structural similarity. A lattice anchor is the coordinate +for topic similarity. The two answer different questions. Neither +substitutes for the other. + +`Fingerprint256.bit(at:)` and `Fingerprint256.with(bit:set:)` give +bit-level read and copy-on-write access across the full 256-bit span. +Both translate a flat index into the right block automatically. +`with(bit:set:)` returns a new value rather than mutating in place. This +matches the package's general preference for value semantics over +in-place mutation. `Fingerprint256.union(_:)` computes the bitwise OR of +two fingerprints: set-union over the bits each one has set. +`Fingerprint256.fromBits(_:)` builds a fingerprint from a 256-element +array of booleans. This is mainly useful for tests, and for converting +from an external bit representation. `wireBytes` and `init(wireBytes:)` +convert to and from the canonical 32-byte little-endian wire form. +`toBytes()` and `fromBytes(_:)` are thin aliases, kept for parity with +call sites that expect a throws-free, optional-returning variant. + +The file's closing section defines a small combinator layer. `zip4(_:_:)` +applies a binary operation to each of the four block pairs between two +fingerprints, in one call. `map4(_:)` applies a unary operation to all +four blocks of one fingerprint. `reduce4(_:_:)` folds a whole sequence of +fingerprints block-wise, with a binary operation. `popcount()` counts set +bits across all four blocks. These four functions exist for a reason. +The same "unroll across four blocks" pattern used to appear, hand-written, +in seven or more places across the codebase. This package's `ORReduce`, +`BitwiseArithmetic`, and `Hamming` were among them. Expressing the unroll +once here means a future change has only one call site to update. There is +no risk that one of the seven repeated copies falls out of sync with the +rest. `zip4Batch(_:_:_:)` and `map4Batch(_:_:)` extend the same idea across +an array of fingerprints. These serve callers vectorizing across many +rows, rather than across one fingerprint's four blocks. + +## HyperplaneFamily.swift + +This file provides `Hyperplane`, one plus-or-minus-one-valued dividing +plane used by SimHash. It also provides `HyperplaneFamily`, the fixed set +of sixty-four such planes that produce one fingerprint block. + +A hyperplane of length *n* would normally be an array of *n* values. Each +value would be plus one, minus one, or zero. Binary inputs make this +arithmetic reduce to counting set bits. So `Hyperplane` instead stores two +bitmasks, `positiveMask` and `negativeMask`. Each has a one wherever the +plane is plus one or minus one, and a zero in both where the plane's +value is zero at that position. This is called a sparse hyperplane. +`Hyperplane.sign(over:)` computes whether the plane's dot product with a +binary input vector is strictly positive. It does this by comparing the +popcount of the input, ANDed with each mask. A tied dot product resolves +to false. The file notes this is vanishingly rare in practice, because +the per-block input has enough bits that an exact tie is unlikely. + +`HyperplaneFamily.generate(seed:blockIndex:inputBitLength:density:)` +builds a family of sixty-four hyperplanes deterministically. It starts +from a 32-byte seed, using a private, non-cryptographic pseudo-random +generator called `HyperplanePRNG`. Determinism here is load-bearing. The +file states that hyperplane seeds are fixed once, at estate creation, and +never rotate. Rotating them would change every fingerprint an estate has +ever computed. It would break cross-replica sync. The `density` parameter +controls what fraction of a plane's positions are active, rather than +zero. At density 1.0, every position is active. The code then takes a +direct path around a floating-point computation. That computation would +otherwise round up past `UInt64.max` and trap. + +`HyperplaneFamily.canonicalHash()` computes a stable 64-bit fingerprint +of a whole family's content. It folds every mask word through FNV-1a. +This labels a pairing arrangement in an audit trail without storing the +full family. `HyperplaneFamily.blockFamilies(baseSeed:density:)` is the +single routine that builds all four block families from one base seed. +Block 0 needs 192 bits; blocks 1 through 3 need 64 bits each. This +routine fixes two mistakes an earlier, separately derived generation path +had made. One mistake was reusing one seed across all four blocks, which +collapsed them into one projection instead of four independent ones. The +other was assuming a uniform 64-bit width for every block, when block 0 +is 192 bits. `diversifiedSeed(base:blockIndex:)` mixes a block index into +the base seed. Each block then gets an independent 32-byte seed. +`expandSeed64(_:)` stretches a 64-bit seed back out to 32 bytes. It does +this through four rounds of a SplitMix64-style mixing function. + +## FloatSimHashPlanes.swift + +This file provides `FloatSimHashPlanes`, the materialized hyperplane set +for the floating-point-input variant of SimHash. It is the float +analogue of `HyperplaneFamily`, which serves the binary-input variant. + +The file explains why this type lives in SubstrateTypes, rather than in +either of the two packages that use it. `SubstrateKernel` applies these +planes, through pure signed-sum-and-sign arithmetic with no randomness. +`SubstrateML` generates them from a seed, using the SplitMix64 random +generator. Placing the shared data shape at the lowest layer lets both +higher packages depend on it without either depending on the other. That +would otherwise invert the SDK's layering. `FloatSimHashPlanes` packs +`256 * dim` sign bits, one per hyperplane-coordinate pair, into a flat +array of 64-bit words. The words are ordered row-major by hyperplane, +matching the draw order the generator produces them in. The planes are +immutable once generated for a given seed and dimensionality. So a caller +materializes one instance per seed-dimension pair. That instance is +reused across every projection under that seed. + +## SimHash.swift + +This file provides `SimHash`, the construction that turns an input +vector and a hyperplane family into one 64-bit fingerprint block. It also +provides `SimHashInput`, the set of helpers that assemble the four +blocks' input vectors from a row's actual fields. + +`SimHash.block(over:family:)` is the core operation. For each of the +family's sixty-four hyperplanes, it sets one output bit. It sets that bit +if the hyperplane's dot product with the input vector is positive. +`SimHash.fingerprint(bitmapInput:latticeInput:lineageTemporalInput:channelSourceInput:families:)` +calls this once per block. It assembles the four results into a complete +`Fingerprint256`. `SimHash.fingerprintBatch(...)` is the same computation, +repeated across many rows, returning results in input order. The file +notes that this reference loop is what a hardware-accelerated backend +elsewhere in the SDK must reproduce bit for bit. That backend vectorizes +the popcount work across many rows at once, for speed, but the result +must match exactly. +`SimHash.fingerprint(fromSubhashes:hyperplanes:)` is a convenience path. +It serves callers that have already reduced each block's input down to +one 64-bit subhash, rather than a full input vector. + +`SimHashInput.bitmap(adjective:operational:provenance:)` concatenates a +row's three bitmap columns into block 0's 192-bit input. +`SimHashInput.lattice(udcPrefixHash:qidDirectHash:qidClosureHash:)` packs +three hash components into block 1's 64-bit input, at fixed bit offsets. +`SimHashInput.lineageTemporal(...)` and `SimHashInput.channelSource(...)` +do the same for blocks 2 and 3. Each function's bit-offset layout is +fixed by the cookbook specification the file cites. A mismatched offset +here would silently produce fingerprints that no longer compare +correctly against ones built the standard way. These helpers exist +precisely so every caller assembles the same layout. + +## Hamming.swift + +This file provides `Hamming`, the substrate's primary +structural-similarity measurement. It counts how many bit positions +differ between two fingerprints. + +`Hamming.distance(_:_:blocks:)` computes this by XORing the two +fingerprints and counting the set bits in the result. That is +`a.zip4(b, ^).popcount()` on the common case of all four blocks. A +per-block loop handles the case where the caller restricts the comparison +to a `BlockMask` subset. Restricting to a subset answers a more specific +question, such as "how similar are these two rows in lattice position +alone, ignoring timing and provenance." `Hamming.similarity(_:_:blocks:)` +converts a distance into a `[0, 1]` score. A score of 1.0 means +identical. A score of 0.0 means maximally distant over the selected +blocks. `HammingDistance` is a type alias, re-exporting `Hamming` under +the name some call sites use. + +## BitwiseArithmetic.swift + +This file provides three fingerprint operations beyond OR-reduction. +They are intersection, symmetric difference, and a cohort prototype. It +also provides `FingerprintBuilder`, a small expression type for composing +them declaratively. + +`BitwiseArithmetic.intersect(_:_:)` computes the bitwise AND of two +fingerprints: the bits both rows share. `BitwiseArithmetic.difference(_:_:)` +computes the bitwise XOR: the bits where the two disagree. Both delegate +to `Fingerprint256.zip4(_:_:)`. `BitwiseArithmetic.prototype(_:)` computes +the bit-for-bit majority vote across a whole cohort of fingerprints. This +is the fingerprint of a "typical" member. It works by folding the cohort +into a `CountVector256` and reading off its majority-vote view. The file +notes this replaced an earlier hand-written per-bit counting loop. That +loop is now an equivalent call into the one canonical cohort-fold +primitive. + +`FingerprintBuilder` is an indirect enum with four cases. One is a +literal fingerprint. One is an intersection of two sub-expressions. One +is a difference of two sub-expressions. One is the prototype of a +fingerprint list. It has one method, `evaluate()`, that walks the tree +and computes the result. This lets a caller build up a compound +fingerprint query as data. One example is "the intersection of Bob's +typical pattern and Amelia's typical pattern." The caller can evaluate it +once, rather than nesting direct function calls. + +## ORReduce.swift + +This file provides `ORReduce`, the substrate's universal aggregation +primitive over collections of fingerprints. + +`ORReduce.reduce(_:)` computes the bitwise OR across a whole sequence of +fingerprints. It returns `Fingerprint256.zero` for an empty sequence, +OR's identity element. It delegates to `Fingerprint256.reduce4(_:|:)`. +`ORReduce.reduce(_:blocks:defaults:)` restricts the reduction to specific +blocks. It fills any block outside the selection from a supplied +default. This helps when a caller wants only the topic-block aggregate +across a cohort, without touching the other three blocks. The file +explains why OR reduction matters beyond simple aggregation. It is +commutative, associative, and idempotent. So it works as the join +operator for a CRDT of fingerprints. Replicas can merge any number of +contributions. They can merge in any order, with any duplication. They +still converge on the same result. It is also a privacy mechanism. Once several fingerprints +are OR-reduced together, a reader can see which structural patterns +appear somewhere in the group. The reader cannot recover which specific +contributor set which bit. + +## CountVector256.swift + +This file provides `CountVector256`, the per-bit counting accumulator. +It underlies cohort-level fingerprint statistics, including the +majority-vote prototype that `BitwiseArithmetic.prototype(_:)` reads from +it. + +The file's opening comment makes the key design argument. A +majority-vote summary does not compose. Suppose node A's majority is +computed from its children, and node B's majority from its children. The +majority of "A's majority and B's majority" is not generally the same as +the true majority over every leaf under both A and B combined. A count +vector does compose. Counts add. Member totals add. The sum is exact +regardless of the order members are folded in. So the count vector, not +the majority-vote fingerprint, is the object worth storing at every level +of a tree. The majority-vote fingerprint is a read-time view, computed +from it whenever a caller actually needs a Hamming-comparable engram. + +`CountVector256.accumulate(_:)` folds one fingerprint into the vector. +For every set bit, its counter increases by one, and the member count +increases by one. `CountVector256.merge(_:)` and its operator form `+` +combine two count vectors by adding element-wise. This is the tree-fold +step. It is safe to apply to a node's children in any order, because +addition is commutative and associative. `CountVector256.majorityVote()` +reads off the fingerprint whose bit `j` is set exactly when a strict +majority of accumulated members had bit `j` set. The rule is +`2 * count > n`. An exact tie at half does not set the bit. The file +calls this convention identical across every kernel backend and both +language ports. `CountVector256.profile()` returns the 256 per-bit +probabilities instead of a threshold. This suits callers that want the +Bernoulli parameter of each bit, rather than a hard yes-or-no answer. The +static `CountVector256.fold(_:)` is the reference implementation of +folding a whole array of fingerprints at once. A vectorized kernel-layer +backend is required to match it exactly. + +## FNV.swift + +This file provides `FNV`, the Fowler-Noll-Vo 1a string hash family. The +substrate uses it everywhere a deterministic string hash is required: +drawer fingerprints, manifest-derived identifiers, and deterministic +tokenization. + +`FNV.hash64(_:)` and `FNV.hash32(_:)` are two independent FNV-1a hashes. +They use different offset bases and primes. The file is explicit that +the 32-bit result is not a truncation of the 64-bit one. It is a wholly +separate computation over the same input bytes. `FNV.hash16(_:)` is +different in kind. It is the low sixteen bits of `hash64(_:)`, not a +from-scratch FNV-1a variant, because FNV-1a has no official 16-bit +definition. The substrate uses it where it needs a compact prefix hash, +such as a lattice sub-hash. + +## MerkleDomain.swift + +This file provides `MerkleDomain`, four one-byte tags. They are +prepended before hashing, to keep different kinds of nodes in the Merkle +content-integrity tree from ever colliding. `leaf` tags a single +drawer's content and vectors. `interior` tags a parent summarizing its +children. `tombstone` tags an expunged payload. `commitment` tags a keyed +HMAC-SHA256 commitment. Domain separation means a leaf hash and an +interior hash can never be mistaken for each other. This holds even if +their underlying bytes happened to coincide, because each was computed +with a different one-byte prefix. The file states these four values are +frozen. They must match exactly between the Swift and Rust legs, +permanently. + +## ContentHash.swift + +This file provides `ContentHash`, the 32-byte SHA-256 digest over one +leaf payload: a drawer's content plus its vectors. + +The type stores its 32 bytes privately, exposed only through the `bytes` +accessor. The fixed size is enforced by construction, not by convention. +`ContentHash.tombstone` is a named constant. It is the SHA-256 hash of +the bare tombstone domain tag byte, used as the sentinel content hash for +an expunged, hard-deleted drawer. The file explains why this constant is +a literal byte array, rather than a value computed at runtime. +SubstrateTypes is the lowest layer. It cannot import SubstrateKernel, +which owns the SHA-256 implementation. Importing it would invert the +SDK's dependency direction. So the precomputed literal stands in instead. A bridge test in +SubstrateKernel checks it against the real computation. `hexString` and +`description` render the bytes as lowercase hex. The custom `Codable` +conformance encodes and decodes that same hex string. It validates the +string's length and character set on decode, rather than trusting +external input. The file is explicit that `ContentHash` is not +interchangeable with `MerkleRoot`, even though both are 32-byte hashes. +One is a payload digest. The other is a subtree summary. The type system +keeps them from being swapped by mistake. + +## MerkleRoot.swift + +This file provides `MerkleRoot`, the 32-byte hash summarizing an +interior node's children in the Merkle content-integrity tree. It is the +counterpart to `ContentHash`'s single-payload digest. + +Its shape mirrors `ContentHash` closely. It uses private byte storage. It +has a `bytes` accessor, hex rendering, and the same `Codable` style. +`MerkleRoot.empty` is the named constant for the hash of +a node with no live children. It is the SHA-256 of the bare interior +domain tag, again stored as a literal, for the same layering reason +`ContentHash.tombstone` is. `MerkleRoot` and `ContentHash` stay two +distinct types, rather than one hash type used for both purposes. That +way, a function that expects a subtree summary cannot accidentally be +handed a single payload's digest. The compiler catches the mistake, +instead of a runtime bug surfacing later. + +## AuditEvent.swift + +This file provides `AuditEvent`, a single recorded mutation of a row. It +holds the before-and-after bitmap and lattice-anchor state. It records +which verb performed the mutation. It records who performed it, and +when. + +`AuditEvent.eventID`, paired with `hlc`, gives every event a compound +key. That key makes replaying the same event twice, after a sync retry +for example, a safe no-op rather than a duplicate. `beforeBitmaps` is optional. A row's very first event is its capture. +That first event has no prior state to record. `reason` is an optional +human-readable explanation. It is threaded from the call site that +performed the mutation. The file notes it is populated for explicit +actions, such as an expunge, but left `nil` for the great majority of +routine mutations. `AuditEvent.withReason(_:)` returns a copy of an event +with a caller-supplied reason attached. This exists because the +structural validator that first produces an event is a pure checker. +That validator, `AuditGate.admit` in SubstrateLib, has no idea why a +mutation happened. The reason is layered on afterward, by the verb that +called it. + +## GSetAuditLog.swift + +This file provides `GSetAuditLog`, the substrate's append-only audit +log. It also provides two supporting types. `AuditEntry` is one immutable +log row. `AuditValue` is a typed field value inside an entry. + +The log is the substrate's source of truth. A row's visible current +state is a projection, computed by replaying its log entries in order. +It is never stored independently of the log. `AuditEntry.id` is a +32-byte SHA-256 content hash, computed over the entry's other fields. +This gives the log a natural way to deduplicate. Two replicas that +independently record the same logical mutation compute the same id. +Once merged, only one copy remains. `AuditValue` is an enum covering four +kinds of value a field change can carry: a bitmap, a string, a +fingerprint, or an integer. It has a hand-written `Codable` conformance. +That conformance encodes each case as a single-key JSON object, such as +`{"bitmap": 42}`. It does not use Swift's default synthesized shape, +`{"bitmap": {"_0": 42}}`. This way the wire format matches what the Rust +port's serde derive produces natively. Both legs agree on one JSON shape. + +`GSetAuditLog` itself stores its entries keyed by content hash +internally, for constant-time deduplication. Its `Codable` conformance +serializes them as a plain array sorted by id instead. The conceptually +correct wire representation of a grow-only set is a set, not a hash map. +The file explains this distinction explicitly, to justify writing custom +`Codable` rather than relying on the default. `GSetAuditLog.add(_:)` +inserts one entry. It is a no-op if the same id is already present. +`GSetAuditLog.merge(_:)` is the CRDT join: the union of two logs' +entries. This is correct regardless of which replica calls it, or in +what order two logs are merged. `GSetAuditLog.orderedEntries` replays +every entry in HLC order. That is the sequence a projection applies to +compute visible state. `GSetAuditLog.entries(forRow:)` and +`GSetAuditLog.entries(since:)` scope that same ordered replay. One scopes +to one row; the other scopes to everything after a cutoff. They serve +the row-state automaton and the sync protocol respectively. The file's +closing comment sketches why this design converges. G-Set merge is set +union. Set union is commutative, associative, and idempotent. HLC gives +any two entries an unambiguous order to replay them in. So any two +replicas that have exchanged all of each other's entries compute +identical visible state. + +## MatrixF.swift + +This file provides `MatrixF`, the field-presence matrix. For every +(field, bit) pair, it counts how many rows in the estate currently have +that bit set. + +`MatrixF` stores 216 `Int64` counts in one flat array. It indexes them +by `field * bitsPerField + bit`. `MatrixF.cellIndex(field:bit:)` computes +that index, with bounds checking. Its layout constants are `fieldCount`, +`bitsPerField`, and `cellCount`. These are aliases onto the identical +constants in `RowBitmaps`. They are kept as aliases, rather than a second +independent definition, so the two types cannot silently drift out of +agreement about the shape of the 216-cell grid they both describe. +`MatrixF.applyRow(delta:bitVector:)` updates every cell whose bit is set +in a `BitVector216`, adding `delta`. That delta is positive one on a +row's capture, negative one on its expunge, or the pair of calls needed +to represent a mutation as a removal followed by an addition. +`MatrixF.totalCount` sums every cell, as a sanity check. +`MatrixF.writeWire(into:)` and `MatrixF.readWire(_:)` serialize the +matrix to and from its canonical 1,728-byte little-endian wire form: +216 cells of eight bytes each. + +## MatrixC.swift + +This file provides `MatrixC`, the correlation matrix. It holds the +marginal probability of each (field, bit) pair, derived from `MatrixF` +rather than updated independently. + +`MatrixC` shares `MatrixF`'s 216-cell shape. It stores `Float` values in +`[0, 1]` instead of raw counts. `MatrixC.derive(from:nRows:)` is its only +way to change. It divides each of `MatrixF`'s counts by the total row +count. The division routes through `Double` before narrowing back to +`Float32`. The file notes this is necessary for cross-language +agreement. Both the Swift and Rust legs perform the identical +cast-divide-cast sequence. IEEE-754 guarantees the same bit pattern +results on both. When there are no rows at all, every cell is zero, +rather than an undefined division result. `MatrixC.writeWire(into:)` and +`MatrixC.readWire(_:)` serialize to and from an 864-byte wire form: +216 cells of four bytes each, as `Float32` bit patterns. + +## MatrixO.swift + +This file provides `MatrixO`, the co-occurrence matrix. It counts how +often each pair of (field, value) settings appears together across rows. +It also provides `CooccurrenceKey`, the packed key identifying one such +pair. + +Each combination pairs one field-value setting against another. Most of +the roughly forty-six thousand possible combinations never occur in a +typical estate. So `MatrixO` stores +only nonzero cells, as a list sorted by `CooccurrenceKey.packed`, rather +than as a dense array or an unordered dictionary. Sorted order makes both +iteration and serialization deterministic across languages. An unordered +dictionary could not guarantee that. `CooccurrenceKey.packed` folds all +four components into one `UInt32`, for compact storage and for the +ordering comparison. `MatrixO.count(_:)` and `MatrixO.increment(_:by:)` +use binary search over the sorted entries. This reads or updates one +cell in logarithmic time. `increment` removes a cell entirely if its +count returns to zero, keeping the canonical form free of dead entries. +`MatrixO.applyRow(delta:fieldValues:)` updates the matrix for one row. It +iterates every ordered pair of the row's field-value settings. This +includes a field paired with itself. It increments each pair's cell by +`delta`. The file notes the matrix is not required to be symmetric in +storage, even though co-occurrence is conceptually symmetric. Each +unordered pair contributes to two distinct cells, (i, j) and (j, i). An +implementation that stores only one direction and infers the other is a +valid optimization. The reference here stores both, for clarity. +`MatrixO.writeWire(into:)` and `MatrixO.readWire(_:)` define the +canonical wire form: an entry count, followed by each entry's packed key +and count. + +## MatrixT.swift + +This file provides `MatrixT`, the temporal causality matrix. It counts +how often a row with one field-value setting is followed by a row with +another field-value setting. This must happen within a bounded time +window. It is the substrate's tool for telling apart two ideas. One idea +is "these things happen together." The other is "one of these things +tends to precede the other." + +`CausalityKey` extends `CooccurrenceKey`'s idea with a fifth component, a +lag bucket. It packs all five into a `UInt64`. +`MatrixT.lagBucket(forMinutes:)` converts a raw time difference in +minutes into one of eight log-spaced buckets. The lower bounds are one, +two, four, eight, sixteen, thirty-two, sixty-four, and one hundred +twenty-eight minutes. It returns `nil` for a difference outside the +supported one-to-two-hundred-fifty-six-minute window. In that case, no +update to the matrix occurs at all. The file states this matrix is +explicitly asymmetric. The cell for "(field i, value i) preceding +(field j, value j)" is tracked completely separately from the reverse +direction. This is unlike `MatrixO`'s conceptually symmetric +co-occurrence. Storage works the same way `MatrixO`'s storage does. +Lookup through `count(_:)` and update through `increment(_:by:)` follow +the same sorted-array, binary-search pattern too. So does wire +serialization. +`MatrixT.applyPair(delta:rowAFieldValues:rowBFieldValues:lagMinutes:)` is +the row-pair version of `MatrixO.applyRow`. Given that row A precedes row +B by a known number of minutes, it increments the cell for every +combination of A's field values against B's field values. This happens +once a valid lag bucket exists for that separation. + +## ThreeDBitTensor.swift + +This file provides `ThreeDBitTensor`, the dense bit-sliced storage +layout. It answers "which rows have field X set to value Y" quickly, +across up to a million rows. + +Rather than storing one 216-bit block per row, a row-major layout, the +tensor keeps six separate bit-slices. There is one slice per bit position +zero through five of a field's six-bit value. Each slice packs one bit +per (row, field) pair. This bit-sliced arrangement is what makes +`scanFieldEquals(field:value:)` fast. Testing whether a whole batch of +rows matches a target value reduces to scanning six flat byte buffers. +That beats reading and unpacking a six-bit value out of every row +individually. `valueAt(row:field:)` and `setValue(row:field:value:)` +provide ordinary cell-level read and write. Each internally loops over +the six bit-slices. `setValue` enforces the six-bit width invariant with +a precondition. `bitSet(row:field:bit:)` and `setBit(row:field:bit:on:)` +are the underlying single-bit primitives both higher-level functions +call. `scanFieldEquals(field:value:)` returns a byte mask, marking every +row matching the target value. `enumerateMatches(_:)` turns that mask +into a plain array of matching row indices. `reserveCapacity(_:)` grows +the tensor to hold more rows. It extends each slice with zero bytes and +leaves all existing data untouched. This is a no-op if the requested +size is not larger than the current one. The file notes the current scan +implementation loops row by row within each bit-slice pass, rather than +operating on whole 64-bit words at once. This is a known optimization +opportunity, not yet taken, for a future vectorized version. + +## RecallTypes.swift + +This file provides the shared wire vocabulary that every recall +primitive and every federation query returns: `RecallScore`, +`DistanceBreakdown`, `RecallResult`, and `RowProjection`. + +These four types live in SubstrateTypes, rather than in whichever +library happens to implement a particular recall strategy. Federation +needs them too. A federated query's response is shaped exactly like a +local recall result. Defining them once here also means no two libraries +can quietly redefine the same shape and drift apart. `RecallScore` pairs +one `RowId` with one `Float32` score. The file is explicit that the +score's meaning is specific to whichever recall primitive produced it. +Vector recall uses cosine similarity. Fingerprint recall uses inverted +Hamming distance. Text recall uses BM25. Any code that combines scores +from different primitives must normalize first. `DistanceBreakdown` +reports how much each of four components contributed to a match: lattice, +fingerprint, temporal, and bitmap. Each is normalized to `[0, 1]`. This +serves two purposes: it explains "why this matched" to a caller, and it +weights Reciprocal Rank Fusion when combining several recall strategies' +results. `RecallResult` bundles a ranked list of scores with an optional +breakdown, an optional confidence interval, and the name of the primitive +that produced it. A composition step downstream then knows which +combination rule to apply. `RowProjection` is the minimal slice of a row +that a recall primitive actually needs to rank candidates. It carries the +row's identifier, capture time, fingerprint, lattice anchor, bitmaps, and +lifecycle state. It deliberately omits the row's verbatim content and any +metadata beyond the bitmaps. The file explains this omission is +intentional. Ranking operates on structure. The heavier verbatim content +is fetched separately, only for the rows that survive ranking. + +## Rust Port and Conformance + +The `rust/` directory mirrors this package, one module per Swift file. +`rust/src/row.rs` matches `Row.swift`. `rust/src/fingerprint256.rs` +matches `Fingerprint256.swift`. The pattern continues through all thirty +files. `rust/src/lib.rs` re-exports the primary types at the crate root, +the same way this package's types are used directly by Swift callers. +There is no separate conformance-fixture directory, unlike LatticeLib's +`rust/tests/fixtures/`. SubstrateTypes ships no pinned data artifacts. So +agreement between the two legs is instead verified by shared test +vectors, embedded directly in each module's own test block. The +illustrative Hamming and Fingerprint256 test vectors, quoted in their +Swift source comments, are one example. These are checked against the +equivalent Rust unit tests. When a function in this package changes, the +corresponding Rust module must change identically. This matters most for +the fingerprint combinators, the Hamming and SimHash arithmetic, and the +FNV hash. Every one of these functions is a cross-platform agreement +contract, not an implementation detail local to one language. diff --git a/packages/libs/SubstrateTypes/docs/OVERVIEW.md b/packages/libs/SubstrateTypes/docs/OVERVIEW.md new file mode 100644 index 0000000..385d5e1 --- /dev/null +++ b/packages/libs/SubstrateTypes/docs/OVERVIEW.md @@ -0,0 +1,225 @@ +--- +doc: OVERVIEW +package: SubstrateTypes +repo: moot-core +authored_commit: b2a5c30b794cf477e18022c55e2fea348614d337 +authored_date: 2026-07-04 +sources: + - path: Sources/SubstrateTypes/AsOfCoordinate.swift + blob: b4433864e36b5803533266c97b153a836ee04adb + - path: Sources/SubstrateTypes/AuditEvent.swift + blob: fbfabeb4eaba4bf43b6e45d8c2b772d994a12b7b + - path: Sources/SubstrateTypes/BitwiseArithmetic.swift + blob: a6270b0f7395c17915c54fdc119e854ca5a44077 + - path: Sources/SubstrateTypes/BlockMask.swift + blob: 904c2179be96d5c20472f0f356947e82e9d8cf37 + - path: Sources/SubstrateTypes/ContentHash.swift + blob: 835744792dc191e824e676d8715c96111b4945ff + - path: Sources/SubstrateTypes/CountVector256.swift + blob: edabb34cb02005135d745ffa6da5541835159927 + - path: Sources/SubstrateTypes/Fingerprint256.swift + blob: 465607fe4164bb0dd94009dbbb20d6e766b973a8 + - path: Sources/SubstrateTypes/FloatSimHashPlanes.swift + blob: b5506e7d12fd153b721f4ca7a52a65a6667ea2f6 + - path: Sources/SubstrateTypes/FNV.swift + blob: f3d8874705ad66f513e49ccbf0c714980a1172e8 + - path: Sources/SubstrateTypes/GSetAuditLog.swift + blob: 1a4452e4c8a147d4fac299a3319b0cb4e70cb795 + - path: Sources/SubstrateTypes/Hamming.swift + blob: dc899742f77434910d51bb3cd830cc9603f56f0a + - path: Sources/SubstrateTypes/HLC.swift + blob: a74442251bd542e8f403029ee475c2ac9d7b91b0 + - path: Sources/SubstrateTypes/HyperplaneFamily.swift + blob: f30c4640629116b1952df48790ed6f5ae67e9025 + - path: Sources/SubstrateTypes/LatticeAnchor.swift + blob: 5f3ba24efa5a85109d20f3b61e6acb17e21cb0a5 + - path: Sources/SubstrateTypes/MatrixC.swift + blob: a8612ea7b09e6d4d93a4acbeedbe2b3b91776a74 + - path: Sources/SubstrateTypes/MatrixF.swift + blob: ed6e38a78c654562b04074878bf86686da0172c3 + - path: Sources/SubstrateTypes/MatrixO.swift + blob: c10fa0b14d8abd172503d0375dfab9302da32025 + - path: Sources/SubstrateTypes/MatrixT.swift + blob: cd31ca818be6446f8f1cd98c6b912c640a78c613 + - path: Sources/SubstrateTypes/MerkleDomain.swift + blob: 4e7d1888a04b2e9979e62c58b54ba698712e56d5 + - path: Sources/SubstrateTypes/MerkleRoot.swift + blob: 1979925f4646cdbed8785a43e2f2a14a7f9c0cac + - path: Sources/SubstrateTypes/NounType.swift + blob: 940d6c51329a66849e572a95ff8ae51648784f2f + - path: Sources/SubstrateTypes/ORReduce.swift + blob: 3f9f6e6ed30fb233f48457f95047f274bd271745 + - path: Sources/SubstrateTypes/RecallTypes.swift + blob: 7f01133df8495156fab87f1f4e441fb7867b3434 + - path: Sources/SubstrateTypes/Row.swift + blob: d7cf86f774dbccbbdf9daf1f3a5ed9be29fb24e8 + - path: Sources/SubstrateTypes/RowBitmaps.swift + blob: b879191a7076807c6fcd20e54cf806a639e63388 + - path: Sources/SubstrateTypes/RowState.swift + blob: 3f8c772be1fe1e3b44b90a5ea8681d4ce4326db7 + - path: Sources/SubstrateTypes/SimHash.swift + blob: e32a8b685c3db29408ed9e7fb21da4a7da10c467 + - path: Sources/SubstrateTypes/SnapshotId.swift + blob: b476bad15abecba2dacea225e0a7db7f445be777 + - path: Sources/SubstrateTypes/ThreeDBitTensor.swift + blob: 47b53642dda804afe8fbc0d863dc4ccd11d3ec55 + - path: Sources/SubstrateTypes/TimeRange.swift + blob: e1b7fbed94b6505b596ac5701cfd7c693e7d2425 +--- + +# SubstrateTypes Overview + +## What This Library Does + +SubstrateTypes defines the smallest shared unit that MOOTx01 stores. That +unit is a row. A row is one recorded observation. A row can be a diary +entry. It can be a fact pulled from a knowledge graph. It can be a +proposal. It can be a sample of ambient context. Every row holds the same +shape. That way, every other library can read it and write it the same +way. SubstrateTypes is where that shape lives. It also defines a few +coordinate systems that ride on every row. + +Two coordinate systems sit on every row. The first is a fingerprint. A +fingerprint is a 256-bit code that captures structural similarity. Rows that +look alike in shape produce fingerprints that share many bits. The second is +a lattice anchor. This is a compact reference to a lattice code. LatticeLib's +FDC engine assigns that code. Rows about the same subject carry related +anchors. A third value, the Hybrid Logical Clock (HLC), timestamps every +change to a row. This lets two devices holding copies of the same estate +agree on the order of events. Neither device needs to trust the other's wall +clock. + +Around these three coordinate systems sit a small set of pure functions. +These include Hamming distance and bitwise arithmetic over fingerprints. +They also include SimHash construction from hyperplane families. Two more +are FNV string hashing and OR-reduction, which folds many fingerprints into +one. Each function is deterministic: the same input always gives the same +output. Call this set "canonical reference compute." These are not heavy +algorithms. They are the exact arithmetic that every consumer of the shapes +must reproduce the same way, on every platform, forever. + +## The Problem It Solves + +MOOTx01 has higher libraries. They rank memories. They detect drift. They +sync an estate across a phone and a laptop. All of them need to talk about +the same row shape. Without one shared definition, each library would +invent its own version of a fingerprint or a timestamp. Those definitions +would drift apart. A comparison between two rows would then depend on +which library produced them. That breaks every guarantee the system makes +about comparing memories. + +SubstrateTypes solves this by being the single, dependency-free home for +these shapes. It ships as the lowest of four packages that together make up +the substrate. SubstrateTypes, this package, holds pure shape and zero +compute beyond canonical reference arithmetic. SubstrateKernel dispatches to +hardware-accelerated backends. SubstrateML holds learned models. +SubstrateLib is the orchestration layer. It still owns the row-state +automaton and the verb mechanics that operate the shapes. SubstrateTypes +depends on none of the other three. A library that only needs to describe a +row's shape depends on SubstrateTypes alone. ConvergenceKit is one example. +It serializes rows for CloudKit. It pulls in no compute it does not need. + +A second problem the package solves is cross-platform agreement. MOOTx01 +estates can federate. A Swift-based Apple client and a Rust-based service +often work on the same estate. Both must compute the identical Hamming +distance for one input. Both must compute the identical fingerprint. Both +must compute the identical hash. The package ships a Rust port in `rust/` +that mirrors every type and function here. Both legs of the SDK share one +arithmetic contract. + +## How It Works + +Every row (`Row`) carries a fixed set of fields. These are an identifier, a +noun type, and a lifecycle state. The noun type says what kind of thing the +row is: a diary entry, a proposal, and so on. A row also carries three +bitmap columns of adjective, operational, and provenance flags. It carries a +fingerprint, a lattice anchor, and optional lineage and content references. +`RowState` and `RowBitmaps` describe the lifecycle and the bitmap layout in +more detail. Both are pure data. The automaton that enforces which state +transitions are legal lives one layer up, in SubstrateLib. + +The fingerprint is built by SimHash, a locality-sensitive hash. A +cryptographic hash scrambles its whole output when one bit of input changes. +SimHash works the opposite way: similar inputs share many output bits. Each +of the fingerprint's four 64-bit blocks compares an input vector against +sixty-four random hyperplanes. A hyperplane is a flat dividing surface, +fixed once at estate creation. The full set of them is recorded in a +`HyperplaneFamily`. + +Once built, fingerprints are compared and combined by a small algebra. +`Hamming.distance` counts differing bits. `BitwiseArithmetic` computes +intersection and symmetric difference. `ORReduce` folds many fingerprints +into one. That merged fingerprint keeps their shared structure. It loses +which original fingerprint set which bit. `CountVector256` accumulates +per-bit counts across a whole cohort. A caller can read off a majority-vote +fingerprint from those counts later. A caller can also read off any other +statistic the same way. + +Every change to a row is recorded, never overwritten. `AuditEvent` is one +recorded mutation. `GSetAuditLog` is a grow-only set of such events: a +conflict-free replicated data type, or CRDT. Two replicas of an estate can +exchange their audit logs in any order. They merge by taking the union of +entries. They land on the same result. Set union does not care about order. +The `HLC` timestamp on every event gives the log a total order to replay. +`TimeRange` and `AsOfCoordinate` let a caller ask what a row looked like at +a past point in that order. + +A separate family of types tracks statistics across a whole estate, rather +than one row at a time. `MatrixF` counts how often each bitmap bit is set +across all rows. `MatrixC` derives the marginal probability of each bit from +`MatrixF`. `MatrixO` counts how often pairs of bitmap values co-occur. +`MatrixT` counts how often one bitmap value precedes another within a time +window. This is the substrate's tool for telling correlation apart from +likely causation. `ThreeDBitTensor` is a dense, bit-sliced storage layout. +It answers "which rows have field X set to value Y" quickly, across a +million rows. + +A last family of types protects data integrity and supports recall. +`ContentHash` and `MerkleRoot` are two distinct fixed-size hashes: one per +leaf payload, one per subtree of children. They are kept as separate types +so the compiler rejects any code that confuses them. `MerkleDomain` supplies +the one-byte tags that keep leaf, interior, and tombstone hashes from ever +colliding. `RecallTypes` defines the wire vocabulary that every recall +primitive returns: `RecallScore`, `RecallResult`, `DistanceBreakdown`, and +`RowProjection`. Every federation query returns this same vocabulary. +Composing results from different recall strategies never needs an ad hoc +translation step. + +## How the Pieces Fit + +Figure 1 shows how the major types connect. It shows a row's own fields, +the fingerprint construction pipeline, and the fingerprint algebra. It also +shows the population-statistics matrices, the audit trail, and the +integrity and recall vocabularies. These last two read from a row without +holding a reference to it. + +![Figure 1. Topology of SubstrateTypes](topology.svg) + +*Figure 1. Topology of SubstrateTypes. A row carries a fingerprint and a +lattice anchor. Hyperplane families and SimHash build the fingerprint. The +fingerprint algebra, made of Hamming distance, bitwise arithmetic, +OR-reduction, and count-vectors, operates on it afterward. The row's +bitmaps feed the population-statistics matrices. Every mutation is recorded +as an audit event. This happens under Hybrid Logical Clock ordering, in the +grow-only audit log. Integrity hashes and the recall wire vocabulary both +read from row data. Neither belongs to the row shape itself.* + +Nothing in this package reaches upward. The row-state automaton stays in +SubstrateLib. So do the verb implementations that call it. So does the +full substrate object that bundles rows with an audit log and the +statistics matrices. SubstrateTypes supplies the vocabulary. SubstrateLib +supplies the behavior. + +## What Ships in the Package + +The package ships thirty Swift source files under `Sources/SubstrateTypes/`. +It also ships a mirrored Rust crate under `rust/src/`. One Rust module +matches each Swift file, by name. There are no pinned data artifacts in +this package. A library such as LatticeLib carries bundled reference tables. +SubstrateTypes carries none, because every value here comes from one of two +places. A caller supplies it directly, such as a hyperplane seed or a row's +own fields. A pure function computes it from caller-supplied input instead. +Examples are a Hamming distance, a SimHash block, and an FNV hash. The +arithmetic itself is pinned, and that is what makes results reproducible. +Conformance tests verify this: they check that the Swift and Rust legs +agree bit for bit. diff --git a/packages/libs/SubstrateTypes/docs/topology.svg b/packages/libs/SubstrateTypes/docs/topology.svg new file mode 100644 index 0000000..455804d --- /dev/null +++ b/packages/libs/SubstrateTypes/docs/topology.svg @@ -0,0 +1,160 @@ + + + + + + + + + + + + + SubstrateTypes: row shape, coordinates, and the views derived from them + + + + HyperplaneFamily + 64 fixed ±1 planes + + + SimHash + sign(<v,h>) per block + + + Fingerprint256 + 4×64-bit structural code + + + Fingerprint algebra + Hamming · Bitwise · OR · CountVector + + + + + + + + LatticeAnchor + topic coordinate + + + Row + the substrate unit + + + Row lifecycle + layout + NounType · RowState · RowBitmaps + + + Temporal coordinates + HLC · TimeRange · AsOfCoordinate + + + + row.fingerprint + + + row.latticeAnchor + + + row.state / .bitmaps + + + + Estate-level views (derived from many rows, not stored on one row) + + + Population matrices + MatrixF · MatrixC · MatrixO · MatrixT + + + Audit trail (CRDT) + AuditEvent · GSetAuditLog + + + Integrity hashes + ContentHash · MerkleRoot + + + Recall vocabulary + RecallResult · RowProjection + + + + + + + + + + + composes / produces + + read by / derived from +