diff --git a/packages/libs/AriaLexiconLib/docs/AGENT_MAP.md b/packages/libs/AriaLexiconLib/docs/AGENT_MAP.md new file mode 100644 index 0000000..3b15a91 --- /dev/null +++ b/packages/libs/AriaLexiconLib/docs/AGENT_MAP.md @@ -0,0 +1,85 @@ +--- +doc: AGENT_MAP +package: AriaLexiconLib +repo: moot-semantics +authored_commit: 021ea704162f86fccea2f030ea9419dacc30a345 +authored_date: 2026-07-04 +sources: + - path: Sources/AriaLexiconLib/Acceptance.swift + blob: 095b16e63587937c22911a7bada283afdf5aea1a + - path: Sources/AriaLexiconLib/Adjective.swift + blob: 4c74c273c5fbf1c863f478de3287133119780936 + - path: Sources/AriaLexiconLib/AriaLexiconLib.swift + blob: 6a7f1d2b66f7fb9b2635befacdd44f3c3212217b + - path: Sources/AriaLexiconLib/Noun.swift + blob: 539bf0460df8d6f60d5e63c11a7706c4c8557eb3 + - path: Sources/AriaLexiconLib/Verb.swift + blob: 7ecf10b17758b23b7652c47879c2b353d28f5965 +--- + +# AGENT_MAP | AriaLexiconLib + +PURPOSE: the reified ARIA grammar, one noun (Drawer), nine verbs, four adjective categories, and the verb-noun acceptance matrix, as pure data. No behavior, no I/O, no dependencies. Canonical prose statement lives outside this package in ARIA_LEXICON.md; this module is that statement made first-class in code so it can be conformance-checked. + +DEPS: imports nothing (zero-dependency, foundational leaf package; Swift stdlib only). Imported by: none currently declare a package/`import` dependency within this SDK checkout. LocusKit (moot-memory/packages/kits/LocusKit) references `AriaLexiconLib`'s types by name in doc comments (Acceptance.swift, Verb.swift, Noun.swift) as the conceptual source of truth for its own Frames/Association/Proposal/LearnedReference/EstateVerbs types, but does NOT declare it as a Package.swift dependency or `import` it as of this commit: treat as documentation-level, not compiled, coupling. Package header names VectorKit and CorpusKit as intended future conformers. Rust port: `rust/` (crate `aria-lexicon-lib`, single file `src/lib.rs`) mirrors every type and the acceptance matrix by hand; its own `#[cfg(test)]` module is the conformance gate (wire-string + matrix parity vs. this Swift package). + +ENTRY POINTS (most callers need only these): +- Acceptance.swift:36 `Acceptance.accepts(_ noun: Noun, _ verb: Verb) -> Bool` | the one call most callers want: "is this operation allowed on this shape" +- Acceptance.swift:14 `Acceptance.verbs(for noun: Noun) -> Set` | full accepted-verb set for a shape +- AriaLexiconLib.swift:15 `AriaLexiconLib.grammar` | the one-sentence rule as a displayable string + +## Symbol Table + +### Module | AriaLexiconLib.swift +- :13 `enum AriaLexiconLib` | empty namespace enum +- :15 `static let grammar: String` | "Every call is one verb applied to a noun, optionally constrained by adjectives." + +### Noun | Noun.swift +- :14 `enum Noun: String, CaseIterable, Sendable, Codable` | 8 storage shapes; rawValue IS the wire string (lowerCamelCase case names, no explicit `=`) +- :15–22 cases: `drawer`, `tunnel`, `kgFact`, `vector`, `diaryEntry`, `proposal`, `association`, `learnedReference` (declaration order = Rust `Noun::ALL` order) +- :27 `static let primary: Noun = .drawer` | the one true noun of the language +- :31 `var role: NounRole` | computed; drawer→primary, kgFact/vector→rung, tunnel/diaryEntry/association→structure, proposal/learnedReference→product +- :45 `enum NounRole: String, CaseIterable, Sendable, Codable` | :47 `primary`, :49 `rung`, :51 `structure`, :53 `product` + +### Verb | Verb.swift +- :9 `enum Verb: String, CaseIterable, Sendable, Codable` | 9 actions, count fixed by spec invariant I-7 +- :10–18 cases: `capture`, `reanchor`, `mutate`, `withdraw`, `expunge`, `recall`, `propose`, `associate`, `learn` (declaration order = Rust `Verb::ALL` order) +- :21 `var flow: Flow` | computed; capture/reanchor/mutate/withdraw/expunge/recall→callerDriven, propose/associate→substrateDriven, learn→groundingDriven +- :37 `enum Flow: String, CaseIterable, Sendable, Codable` | :38 `callerDriven`, :39 `substrateDriven`, :40 `groundingDriven` + +### Adjective | Adjective.swift +- :13 `enum Adjective: String, CaseIterable, Sendable, Codable` | 4 cross-noun categories, count fixed by spec invariant I-8 +- :17 `state` | lifecycle position (active/pending/contested/superseded/decayed/withdrawn/expired/rejected/accepted/tombstoned: VALUES live in LocusKit, not here) +- :20 `trust` | provenance (verbatim/observed/imported/proposed/derived/canonical: values in LocusKit) +- :23 `sensitivity` | exposure level (normal/elevated/restricted/secret: values in LocusKit) +- :26 `exportability` | perimeter crossing (private/public: values in LocusKit) + +### Acceptance matrix | Acceptance.swift +- :10 `enum Acceptance` | namespace for the matrix +- :14 `static func verbs(for noun: Noun) -> Set` | hand-authored per-noun set; SEE ENTRY POINTS +- :16 drawer → {capture, reanchor, mutate, withdraw, expunge, recall} (6, fullest row) +- :18 tunnel → {capture, mutate, withdraw, expunge, recall} (5, no reanchor) +- :20 kgFact → {mutate, withdraw, expunge, recall} (4, no capture/reanchor) +- :22 vector → {} (0: substrate-managed, never verb-addressable directly) +- :24 diaryEntry → {recall} (1, read-only) +- :26 proposal → {propose, mutate, withdraw, expunge, recall} (5; role is "product" but DOES accept propose) +- :28 association → {associate, mutate, expunge, recall} (4; no withdraw) +- :30 learnedReference → {learn, mutate, withdraw, expunge, recall} (5) +- :36 `static func accepts(_ noun: Noun, _ verb: Verb) -> Bool` | `verbs(for:).contains(verb)`; SEE ENTRY POINTS + +### Rust port | rust/src/lib.rs (mirror, not a Swift symbol) +- `Noun`/`NounRole`/`Verb`/`Flow`/`Adjective` | PascalCase enum variants + `as_str()` producing the identical lowerCamelCase wire strings as the Swift rawValues +- `accepted_verbs(Noun) -> Vec` / `accepts(Noun, Verb) -> bool` | same matrix, hand-mirrored +- `#[cfg(test)] mod tests` | conformance gate: verb/adjective counts, full matrix cell-by-cell, wire-string parity, serde round-trip + +## INVARIANTS / GOTCHAS + +- NO BEHAVIOR. This package is pure data (enums, one constant string, one lookup table). Do not add I/O, logging, or business logic here; new domain operations compose the nine existing verbs instead of extending the vocabulary. +- Verb count is pinned at 9 (spec invariant I-7). Adjective category count is pinned at 4 (spec invariant I-8). Both are spec-level invariants, not incidental: changing either is a specification amendment, not a routine edit. +- `Acceptance`'s matrix is HAND-AUTHORED, not derived from `Noun.role` or `Verb.flow`. Do not "simplify" it by deriving from role/flow: the Proposal row (role=`product`, yet accepts `.propose`) and the Association row (role=`structure`, accepts `.associate` but not `.propose`) are real exceptions that a derived rule would get wrong. +- `Adjective` names categories only. The value lists inside each category (the 10 lifecycle states, 6 trust levels, 4 sensitivity levels, 2 exportability levels) live in LocusKit (moot-memory), not here: by design, so the two never fork by being defined twice. Do not add value enums to this file. +- Zero dependencies is load-bearing: this package must stay leaf-level (imports nothing) so any kit, including ones layered far apart, like LocusKit, VectorKit, and CorpusKit, can depend on it without a cycle. +- Swift enum rawValues ARE the wire strings (case names are already lowerCamelCase; no explicit `= "..."` needed). Rust mirrors with PascalCase variants + `as_str()`/`serde(rename_all = "camelCase")` to reproduce the identical strings. Any edit to a Swift case name, or to Rust's `as_str()`/serde attributes, that the other side does not mirror breaks wire conformance silently unless the Rust test suite is rerun. +- Two independent implementations, no shared codegen: editing `Acceptance.swift`, `Noun.swift`, `Verb.swift`, or `Adjective.swift` requires a matching hand-edit to `rust/src/lib.rs` (types, `as_str()`, and `accepted_verbs`/`accepts`) plus a rerun of the Rust `#[cfg(test)]` suite. There is no build step that generates one from the other. +- Canonical source-of-truth order: the prose document ARIA_LEXICON.md (external to this package, not present in this repo checkout) states the grammar; this Swift package is the code reification of that document; the Rust crate is a conformance-gated mirror of this package. When in doubt about intent, ARIA_LEXICON.md wins. +- No compiled consumer exists yet within this SDK checkout: verify current importers before assuming call sites exist. LocusKit's references are doc-comment citations, not `import` statements or Package.swift dependencies, as of this commit. diff --git a/packages/libs/AriaLexiconLib/docs/DETAILS.md b/packages/libs/AriaLexiconLib/docs/DETAILS.md new file mode 100644 index 0000000..55cb1cc --- /dev/null +++ b/packages/libs/AriaLexiconLib/docs/DETAILS.md @@ -0,0 +1,206 @@ +--- +doc: DETAILS +package: AriaLexiconLib +repo: moot-semantics +authored_commit: 021ea704162f86fccea2f030ea9419dacc30a345 +authored_date: 2026-07-04 +sources: + - path: Sources/AriaLexiconLib/Acceptance.swift + blob: 095b16e63587937c22911a7bada283afdf5aea1a + - path: Sources/AriaLexiconLib/Adjective.swift + blob: 4c74c273c5fbf1c863f478de3287133119780936 + - path: Sources/AriaLexiconLib/AriaLexiconLib.swift + blob: 6a7f1d2b66f7fb9b2635befacdd44f3c3212217b + - path: Sources/AriaLexiconLib/Noun.swift + blob: 539bf0460df8d6f60d5e63c11a7706c4c8557eb3 + - path: Sources/AriaLexiconLib/Verb.swift + blob: 7ecf10b17758b23b7652c47879c2b353d28f5965 +--- + +# AriaLexiconLib Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in the order a +reader builds understanding. First comes the module surface and its +one-sentence grammar. Then comes the noun. Then comes the verb. Then comes +the adjective categories. Last comes the acceptance matrix that relates +the noun and the verb. + +## AriaLexiconLib.swift + +This file provides the module surface. It holds the public +`AriaLexiconLib` enum and the one-sentence statement of the grammar. + +Swift has no module-level functions or constants. A library that wants a +single well-known place for a top-level fact uses an empty enum as a +namespace instead. `AriaLexiconLib` is that namespace. It exists only to +hold `grammar`. + +`AriaLexiconLib.grammar` is a constant string: "Every call is one verb +applied to a noun, optionally constrained by adjectives." This matters +because people need to read the rule, not only have types enforce it. A +consumer can print this string in a log, a debugging tool, or a developer +console. It will always match the rule the rest of the package encodes. +It lives in the same file set and the same version as the types that +implement it. + +## Noun.swift + +This file provides `Noun`, the eight storage shapes the substrate +persists. It also provides `NounRole`, how each shape relates to the one +true noun of the language, the Drawer. + +A Drawer is the atomic unit of memory: one row of stored content. The +other seven cases in `Noun` are `tunnel`, `kgFact`, `vector`, +`diaryEntry`, `proposal`, `association`, and `learnedReference`. Each of +these seven is not a separate thing a caller thinks about the way it +thinks about a Drawer. Each is either a different view onto Drawer +content, or a record of something a verb did. The file's opening comment +is explicit. The architecture specification calls all of these "nouns" +loosely, as a storage taxonomy. Only Drawer is a noun in ARIA's grammar +sense. The rest are facets or residue. + +`Noun.primary` is a static constant equal to `.drawer`. It gives code a +named way to ask which case is the real noun. It avoids hard-coding the +case name at every call site. A future reader searching for "primary" +finds the one place that decision is made. + +`Noun.role` is a computed property. It classifies each case into one of +four `NounRole` values. `.primary` covers the Drawer itself. `.rung` +covers a representation of a Drawer's content: `kgFact` and `vector`. +`.structure` covers an edge or event about Drawers: `tunnel`, +`diaryEntry`, and `association`. `.product` covers what a verb leaves +behind: `proposal` and `learnedReference`. This grouping matters because a +caller often cares about a shape's role more than its exact case name. +For example, a caller deciding whether it is safe to delete something +cares about role. A rung can usually be rebuilt from its source Drawer. A +product generally cannot. + +`NounRole` itself is a four-case, string-backed enum: `primary`, `rung`, +`structure`, `product`. Being string-backed means it serializes to a +stable, human-readable form, rather than a raw integer. This matters for +any log or wire format that includes it. + +## Verb.swift + +This file provides `Verb`, the nine actions the substrate supports. It +also provides `Flow`, who initiates each one. + +The nine verbs are `capture`, `reanchor`, `mutate`, `withdraw`, +`expunge`, `recall`, `propose`, `associate`, and `learn`. The file's +opening comment states the design rule directly. This count is fixed by +specification invariant I-7. A new kind of domain operation is expected +to compose these nine, rather than add a tenth. Keeping the verb set +closed lets every kit share one small vocabulary. Otherwise each kit +would grow its own private verbs over time. + +`Verb.flow` is a computed property. It returns a `Flow` value that +answers who calls this verb. Six verbs are `.callerDriven`: `capture`, +`reanchor`, `mutate`, `withdraw`, `expunge`, and `recall`. An application +calls each of these directly and synchronously, when it wants something +to happen. Two verbs are `.substrateDriven`: `propose` and `associate`. +The system's own background processes emit these on their own schedule. +No application code calls them directly. One verb, `learn`, is +`.groundingDriven`. It is the verb that brings authoritative outside +reference material into the system. It differs from both a direct caller +action and an internal background signal. Separating these three flows +matters for anyone writing code against the substrate. A caller-driven +verb needs a call site. A substrate-driven or grounding-driven verb needs +a listener instead. + +`Flow` is a three-case, string-backed enum: `callerDriven`, +`substrateDriven`, `groundingDriven`. + +## Adjective.swift + +This file provides `Adjective`, the four categories that describe any row +of memory, regardless of which noun or verb produced it. + +The four categories are as follows. `state` describes where the row sits +in its lifecycle, such as active, pending, or superseded. `trust` +describes how the content was established, such as verbatim, observed, or +derived. `sensitivity` describes how exposed the content may be. +`exportability` describes whether the content may leave the access +perimeter, such as private or public. The count of four is fixed by +specification invariant I-8, the same way the verb count is fixed by I-7. + +The file's comment explains an important design boundary. `Adjective` +only names the four categories. It does not name the specific values +inside each one. Every row carries some value in each category, no +matter its storage shape. This is why the categories are described as +"cross-noun." How those values are packed and represented is a different +matter. The actual list of lifecycle states, for example, is treated as +a bitmap-layout detail. That detail is reified in a separate kit, +LocusKit, instead of here. Keeping the category names in one place and +the value lists in another means the two cannot silently drift apart by +being maintained twice. Only one of them needs to change when a new +value is added to a category. + +## Acceptance.swift + +This file provides `Acceptance`, the verb-noun acceptance matrix. It is +the one place the package states which of the nine verbs make sense for +each of the eight storage shapes. + +The matrix is hand-authored data. No other rule in the package, neither +`Noun.role` nor `Verb.flow`, derives it. For example, `Proposal`'s role +is `product`. Yet it accepts the verb that creates it, `propose`. It also +accepts the ordinary lifecycle verbs: `mutate`, `withdraw`, `expunge`, +and `recall`. The file's opening comment frames this precisely. In ARIA's +grammar the matrix reads as which actions apply to this shape. It is not +a competition between nouns. It is expressed as data specifically so a +conformance test can check the Swift and Rust implementations agree cell +by cell. + +`Acceptance.verbs(for:)` takes a `Noun` and returns the `Set` it +accepts. A Drawer accepts six verbs. It is the fullest row in the +matrix, since every caller-driven verb applies to it. A Tunnel accepts +five verbs. It omits `reanchor`. A knowledge-graph fact accepts four +verbs. It omits both `capture` and `reanchor`. A fact is not captured +directly the way a Drawer or a Tunnel is. A Vector accepts no verbs. The +comment explains that vectors are substrate-managed. An internal process +derives and maintains them. No caller ever addresses one with a verb +directly. A diary entry accepts only `recall`, since it is a read-only +record. A Proposal accepts `propose` plus the lifecycle verbs. An +Association accepts `associate` plus most of the lifecycle verbs. It +omits `withdraw`. A learned reference accepts `learn` plus the lifecycle +verbs. This function matters because it is the only place any of these +facts is stated. A caller that needs to know whether it can do something +to a shape asks this function. It does not re-derive the answer from role +or flow. Re-deriving would sometimes give the wrong answer, as the +Proposal example shows. + +`Acceptance.accepts(_:_:)` takes a noun and a verb. It returns whether the +verb is a member of that noun's accepted set. It is a thin convenience +over `verbs(for:)`. Most call sites want a yes-or-no answer for one +specific pair, rather than the whole set. Spelling that check inline at +every call site would risk a typo that a shared function cannot make. + +## Rust Port and Conformance + +The `rust/` directory contains the second leg of the library: `lib.rs`, a +single-file crate named `aria-lexicon-lib`. It mirrors every Swift type +as Rust enums: `Noun`, `NounRole`, `Verb`, `Flow`, and `Adjective`. It +also adds `accepted_verbs` and `accepts` as the Rust equivalents of +`Acceptance.verbs(for:)` and `Acceptance.accepts(_:_:)`. Swift's +string-backed enum cases are already lowerCamelCase. They serve as their +own wire strings. Rust instead declares PascalCase variants by +convention. It adds an `as_str()` method, plus `serde` attributes, that +reproduce the identical lowerCamelCase wire strings for JSON. For +example, `Noun::KgFact.as_str()` returns `"kgFact"`. This exactly matches +Swift's `Noun.kgFact.rawValue`. + +The crate's own test module is the conformance suite. It checks the verb +count is nine and the adjective count is four. It checks every wire +string against its expected value. It checks that `serde` serialization +matches `as_str()` for every case. It also re-states the entire +acceptance matrix, cell by cell, including the harder-to-guess cases. One +such case is a Proposal not accepting `associate`. Another is an +Association not accepting `propose`. The two languages cannot share +source code for this vocabulary, so this test module is the only +mechanism that catches the two ports drifting apart. + +Four files must stay in sync: `Acceptance.swift`, `Noun.swift`, +`Verb.swift`, and `Adjective.swift`. Any edit to one of them requires a +matching edit in `rust/src/lib.rs`. It also requires a rerun of both test +suites. Otherwise a silent disagreement ships. diff --git a/packages/libs/AriaLexiconLib/docs/OVERVIEW.md b/packages/libs/AriaLexiconLib/docs/OVERVIEW.md new file mode 100644 index 0000000..4e99ec4 --- /dev/null +++ b/packages/libs/AriaLexiconLib/docs/OVERVIEW.md @@ -0,0 +1,150 @@ +--- +doc: OVERVIEW +package: AriaLexiconLib +repo: moot-semantics +authored_commit: 021ea704162f86fccea2f030ea9419dacc30a345 +authored_date: 2026-07-04 +sources: + - path: Sources/AriaLexiconLib/Acceptance.swift + blob: 095b16e63587937c22911a7bada283afdf5aea1a + - path: Sources/AriaLexiconLib/Adjective.swift + blob: 4c74c273c5fbf1c863f478de3287133119780936 + - path: Sources/AriaLexiconLib/AriaLexiconLib.swift + blob: 6a7f1d2b66f7fb9b2635befacdd44f3c3212217b + - path: Sources/AriaLexiconLib/Noun.swift + blob: 539bf0460df8d6f60d5e63c11a7706c4c8557eb3 + - path: Sources/AriaLexiconLib/Verb.swift + blob: 7ecf10b17758b23b7652c47879c2b353d28f5965 +--- + +# AriaLexiconLib Overview + +## What This Library Does + +AriaLexiconLib names the words every MOOTx01 kit uses to describe an action +on memory. Inside MOOTx01, this fixed vocabulary is called ARIA. ARIA states +one simple grammar. Every call to the substrate is one verb applied to a +noun. A caller may constrain that call with adjectives. AriaLexiconLib turns +this sentence into code. A program can then check whether a verb even makes +sense for a noun before it tries the call. + +The library has one noun: the Drawer. The Drawer is the atomic unit of +memory. The substrate persists other storage shapes too: a knowledge-graph +fact, a vector, or a diary entry. Each of these is either a facet of a +Drawer or something a verb leaves behind. + +The library has nine verbs. Examples include capture, recall, and withdraw. +The library also has four adjective categories. Examples include trust and +sensitivity. An adjective describes a row without changing which noun or +verb it is. A small acceptance matrix records which verbs apply to which +storage shapes. + +## The Problem It Solves + +MOOTx01 is an on-device AI memory system. It stores what an AI observes over +time. It helps the AI recall that memory later. Many separate packages +build on top of this vocabulary. The SDK described at the top of this +repository set calls these packages kits. One kit stores knowledge-graph +facts. Another stores vector embeddings. Another manages the estate's +day-to-day operations. + +If each kit invented its own names for "add a memory" or "remove a +memory," a caller would need to learn a different vocabulary per kit. It +would also become easy to call an operation that makes no sense for a +given storage shape. One example is asking a vector to be captured +directly, instead of letting the substrate manage it. + +AriaLexiconLib fixes the vocabulary once, in one place. It states the +vocabulary as data, rather than as prose scattered across each kit's +documentation. Every kit conforms to it. A kit's API surface uses these +same noun and verb names. Its allowed-operations logic reads its answers +from the acceptance matrix. It does not restate the rule on its own. The +vocabulary also crosses languages. The library ships a Rust port so +non-Swift consumers speak the identical words. Shared tests check the two +ports against each other. + +The canonical statement of the grammar is a prose document named +ARIA_LEXICON.md. That document lives outside this package. AriaLexiconLib +makes that document first class in code. An editor, a compiler, and a +conformance test can all point at the same set of names. + +## How It Works + +The library defines four small pieces of vocabulary. It also defines one +rule that relates two of them. + +`Noun` lists the eight storage shapes the substrate persists. It marks the +Drawer as the one true noun of the language. The other seven cases each +play one of three roles. A "rung" is a representation of a Drawer's +content, such as a vector. A piece of "structure" is an edge or event about +Drawers, such as an association. A "product" is what a verb leaves behind, +such as a proposal. + +`Verb` lists the nine actions the substrate supports: capture, reanchor, +mutate, withdraw, expunge, recall, propose, associate, and learn. Each verb +also names who initiates it. Six verbs are caller-driven. An application +calls each of these directly. Two verbs are substrate-driven. An internal +background process emits each of these on its own schedule. One verb, +`learn`, is grounding-driven. It is how authoritative outside reference +material enters the system. + +`Adjective` lists four categories. Each category describes a row no matter +which noun or verb touched it. State describes where the row sits in its +lifecycle, such as active or superseded. Trust describes how the content +was established, such as verbatim or derived. Sensitivity describes how +exposed the content may be. Exportability describes whether the content +may leave the access perimeter. This library only names the four +categories. The specific values inside each category are a detail of how +the data is packed into storage. For example, the full list of lifecycle +states is such a detail. That detail lives in a separate kit, LocusKit. +Keeping it there means the two kits never drift out of agreement by being +defined twice. + +`Acceptance` is the one rule that connects `Noun` and `Verb`. For each of +the eight nouns, it lists exactly which of the nine verbs make sense. A +Drawer accepts six verbs, including capture and recall. A Vector accepts +none. The substrate manages vectors on its own, so nothing calls a verb on +one directly. This matrix is hand-authored. No other rule in the library +derives it automatically. It is the single place a caller or a test checks +whether an operation is allowed. + +## How the Pieces Fit + +Figure 1 shows how the four vocabulary types and the acceptance matrix +relate to each other and to the Rust port. + +![Figure 1. Topology of AriaLexiconLib](topology.svg) + +*Figure 1. Topology of AriaLexiconLib. `Noun` and `Verb` feed the +`Acceptance` matrix. `Adjective` stands beside them as an independent, +cross-noun category list. The dashed regions mark the Rust mirror and the +downstream packages that conform to this vocabulary. This package does not +depend on them.* + +`AriaLexiconLib.grammar` states the one-sentence rule in plain English. It +is a constant string. A consumer can display or log the rule itself, +rather than reconstruct it from the types. `Noun` and `Verb` are +independent enumerations. Neither refers to the other. +`Acceptance.verbs(for:)` and `Acceptance.accepts(_:_:)` are the only +functions in the package. Both simply look up a fixed table keyed by a +noun. `Adjective` does not participate in the acceptance rule at all. An +adjective always applies to a row, regardless of its noun or the verb that +produced it. + +The Rust port in `rust/` mirrors every type and the acceptance matrix by +hand. The two languages cannot share source code. A build-time conformance +suite inside the Rust crate re-asserts every wire string and every cell of +the acceptance matrix. It checks these against the values recorded here. A +change to either side that the other side misses fails a test. It does not +ship a silent disagreement. + +## What Ships in the Package + +The package ships five Swift source files. It bundles no data files. It +also ships the Rust port in `rust/`. There are no pinned artifacts and no +build-time tooling. Every value in the package is a literal written +directly in the source. The vocabulary is small and fixed by +specification, rather than computed or trained. The verb count is fixed at +nine. The adjective category count is fixed at four. Both counts are +named spec invariants. Growing either number is a specification change, +not a routine code edit. diff --git a/packages/libs/AriaLexiconLib/docs/topology.svg b/packages/libs/AriaLexiconLib/docs/topology.svg new file mode 100644 index 0000000..f6e72c7 --- /dev/null +++ b/packages/libs/AriaLexiconLib/docs/topology.svg @@ -0,0 +1,97 @@ + + + + + + + + + + + + + AriaLexiconLib: the ARIA grammar as data + + + + AriaLexiconLib.grammar + "verb applied to noun, ± adjectives" + + + + Noun + 8 shapes; drawer primary + + + Verb + 9 actions; 3 flows + + + Adjective + 4 categories, cross-noun + + + + Acceptance + verb-noun matrix (hand-authored) + + + + + + + + + + + + + Rust port (conformance-gated mirror) + + + rust/src/lib.rs + aria-lexicon-lib crate + #[cfg(test)] parity gate + + + + + + + + Downstream conformers (outside this package) + + + LocusKit + doc-comment citations only + + + VectorKit / CorpusKit + named future conformers + + + diff --git a/packages/libs/EideticLib/docs/AGENT_MAP.md b/packages/libs/EideticLib/docs/AGENT_MAP.md new file mode 100644 index 0000000..4d177ae --- /dev/null +++ b/packages/libs/EideticLib/docs/AGENT_MAP.md @@ -0,0 +1,63 @@ +--- +doc: AGENT_MAP +package: EideticLib +repo: moot-semantics +authored_commit: 021ea704162f86fccea2f030ea9419dacc30a345 +authored_date: 2026-07-04 +sources: + - path: Sources/EideticLib/EideticLib.swift + blob: 1839588dc98279c1eee2786f1974457a5608742b + - path: Sources/EideticLib/LatticeCodeState.swift + blob: 7580261208b77c7e79363a510920c413054264a1 + - path: Sources/EideticLib/Segmenter.swift + blob: c5313d93bf7ca1cbf577da6ffabc9197b5e788e2 +--- + +# AGENT_MAP: EideticLib + +PURPOSE: thin deterministic facade over LatticeLib's FDC engine (term → Anchor{code, wikidataQID, confidence, dataVersion}) plus two unrelated small utilities: lattice-code shape/known-state classification, and sentence segmentation. Carries no reference data of its own. + +DEPS: imports LatticeLib (FDC.encodeAnchor/isAvailable/dataVersion), NaturalLanguage (Apple-only NLTokenizer sentence path), Foundation. Imported by: CorpusKit (moot-memory kit, `EideticLib.sentences` only) and, per Package.swift header comment, NeuronKit (not present in these four venue repos: likely mootx01-ee/ce, out of scope here). Rust port in rust/ (crate `eidetic-lib`) mirrors lookup, classifyLatticeCode, and the delimiter segmenter exactly; depends on sibling `lattice-lib` crate. No Rust counterpart to the NLTokenizer path (Apple-only, excluded from cross-leg parity by the apple-nlp-accel constitutional constraint C-2). + +ENTRY POINTS (most callers need only these): +- EideticLib.swift:67 `EideticLib.lookup(_ term: String) -> Anchor`: main lookup; fatalError if FDC.isAvailable is false +- EideticLib.swift:118 `EideticLib.lookup(_:recordNovel:)`: same, recordNovel:false = privacy seam (no LatticeLib pool writes) +- EideticLib.swift:40 `EideticLib.classifyLatticeCode(_:knownCodes:) -> LatticeCodeState`: malformed/known/pending +- Segmenter.swift:47 `EideticLib.sentences(_ text: String) -> [Substring]`: platform-routed sentence split + +## Symbol Table + +### Lookup surface: EideticLib.swift +- :20 `enum EideticLib`: namespace; holds no reference data, delegates entirely to LatticeLib.FDC +- :23 `version = "0.1.0"`: module version string +- :40 `classifyLatticeCode(_:knownCodes:) -> LatticeCodeState`: grammar check via LatticeCodeGrammar, then knownCodes membership +- :67 `lookup(_:) -> Anchor`: guards FDC.isAvailable (else fatalError); calls FDC.encodeAnchor(term); resolved→confidence 32, UNRESOLVED→empty code/confidence 0 +- :118 `lookup(_:recordNovel:) -> Anchor`: identical Anchor to lookup(_:); recordNovel:false suppresses LatticeLib novel-token pool accumulation; used by GLK capture intake (EncodeIntake) so private memory content never reaches the pool +- :147 `struct Anchor: Equatable, Sendable, Codable`: code (empty=UNRESOLVED) / wikidataQID (String?) / confidence (UInt8: 0/16/32/48/56=null/low/medium/high/verified; EideticLib only emits 0 or 32) / dataVersion (FDC signatures version); byte-identical JSON shape to Rust `Anchor` + +### Lattice-code state: LatticeCodeState.swift +- :27 `enum LatticeCodeState: Sendable, Hashable, Codable`: .malformed(String) / .known(String) / .pending(String) +- :46 `rawCode: String`: original input regardless of case (storage round-trip without unpacking) +- :56 `isWellFormed: Bool`: false only for .malformed +- :72 `enum LatticeCodeGrammar`: dependency-free; parallel reimplementation of LatticeLib's `Code.isWellFormed(_:)`, kept in agreement by shared conformance tests, NOT shared code +- :77 `maxExtensionDigits = 8`: pinned; matches LatticeLib Code.maxExtensionDigits +- :82 `isWellFormed(_ code:) -> Bool`: 3 ASCII digits + optional `.` + 1–8 ASCII digits; empty/overlong/non-digit extension fails + +### Sentence segmentation: Segmenter.swift (extension EideticLib) +- :47 `sentences(_ text:) -> [Substring]`: Apple: NLTokenizer(unit: .sentence); non-Apple (or NLTokenizer yields nothing on non-empty input): falls back to sentencesByDelimiter; empty input → [] +- :78 `sentencesByDelimiter(_ text:) -> [Substring]`: canonical cross-platform reference; splits on `.` `!` `?` `\n`, terminator stays attached to preceding segment; trailing remainder is its own segment; total-coverage guaranteed (join(segments) == text) for non-empty input +- Relocated 2026-05-27 (F16) from CorpusKit/Sources/CorpusKit/Chunker.swift::sentenceSegments + +## INVARIANTS / GOTCHAS + +- fatalError vs UNRESOLVED are TWO DIFFERENT failure modes, do not conflate: FDC.isAvailable==false (missing/broken artifact bundle, a build defect) → fatalError, process terminates. A term with no signature overlap → UNRESOLVED → Anchor(code: "", confidence: 0), a normal, honest, non-fatal result. Never change the UNRESOLVED path to fatalError or vice versa. +- lookup(_:) and lookup(_:recordNovel:) MUST return byte-identical Anchors for the same term; recordNovel only changes whether LatticeLib's shared novel-token pool is written to. Any divergence in the returned Anchor is a bug. +- confidence is a fixed value, not calibrated: EideticLib always emits exactly 0 (UNRESOLVED) or 32 (resolved) from lookup. 16/48/56 are reserved slots in the substrate provenance confidence value set that this library never produces; do not repurpose them here without an explicit design change. +- classifyLatticeCode takes knownCodes as a caller-supplied parameter, not an owned canon: EideticLib deliberately has no opinion on which codes are "known." Do not add a package-level known-code cache; that decision belongs to the caller (federation/version skew is the whole reason pending exists). +- LatticeCodeGrammar.isWellFormed is a SEPARATE implementation from LatticeLib's Code.isWellFormed, not a wrapper. Changing the grammar in one without the other breaks conformance tests silently until those tests run: always update both plus the shared conformance vectors. +- sentences(_:) is NOT cross-platform-identical by design (Apple NLTokenizer vs delimiter fallback): this is the one place in the EideticLib/LatticeLib family where platform divergence is accepted, because downstream consumers content-address chunks by (sourceID, startOffset, text) under an append-only conflict policy rather than requiring byte-identical segmentation. sentencesByDelimiter(_:) is the only segmentation function under cross-leg parity with Rust. +- Total-coverage invariant for both sentence functions: for non-empty input, concatenating the returned segments must reproduce the input exactly, with no gaps, overlaps, or reordering. Any change to either function must preserve this. +- EideticLib.swift and LatticeCodeState.swift and Segmenter.swift do not call one another: there is no internal pipeline. Do not assume ordering or shared state between the three surfaces. +- Package ships an intentionally empty Sources/EideticLib/Resources/ directory (placeholder so `.process("Resources")` has a target); EideticLib bundles zero reference data: do not add data files here expecting lookup to use them. All classification data lives in LatticeLib's bundle. +- Rust Anchor uses explicit `#[serde(rename = "wikidataQID")]`: auto camelCase would produce "wikidataQid" (lowercase id), breaking Swift/Rust JSON interchange. Do not remove the explicit rename. +- Rust LatticeCodeState wire shape is internally tagged: `{"state": "pending", "code": "999.42"}`: locked by `lattice_code_state_wire_shape_conformance` test; must match Swift's Codable encoding. diff --git a/packages/libs/EideticLib/docs/DETAILS.md b/packages/libs/EideticLib/docs/DETAILS.md new file mode 100644 index 0000000..5ca34c6 --- /dev/null +++ b/packages/libs/EideticLib/docs/DETAILS.md @@ -0,0 +1,235 @@ +--- +doc: DETAILS +package: EideticLib +repo: moot-semantics +authored_commit: 021ea704162f86fccea2f030ea9419dacc30a345 +authored_date: 2026-07-04 +sources: + - path: Sources/EideticLib/EideticLib.swift + blob: 1839588dc98279c1eee2786f1974457a5608742b + - path: Sources/EideticLib/LatticeCodeState.swift + blob: 7580261208b77c7e79363a510920c413054264a1 + - path: Sources/EideticLib/Segmenter.swift + blob: c5313d93bf7ca1cbf577da6ffabc9197b5e788e2 +--- + +# EideticLib Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. The three files stand +independent of one another. They appear here in the order a new reader +would most likely meet them: the lookup surface, the code-state +classifier, and the sentence segmenter. + +## EideticLib.swift + +This file provides the module surface. That surface is the +`EideticLib` enum, its `lookup` functions, `classifyLatticeCode`, and +the `Anchor` result type. + +`EideticLib` is a namespace enum, the same pattern LatticeLib uses for +its own module surface. It holds no reference data. A comment at the +top of the file states this plainly. LatticeLib's FDC runtime owns and +caches the pinned lexicon, frame, and signatures. EideticLib parses +nothing of its own. It simply calls through. + +The two `lookup` overloads share one behavior. They never return a +made-up answer. Suppose `FDC.isAvailable` is false, meaning the FDC +artifacts bundled inside LatticeLib failed to load. Then both overloads +call `fatalError` and stop the process. A comment in the code names a +specific product mandate for this choice. It quotes Bob's board item +seven: "a sentinel identity that persists IS a fabricated identity." +The distinction matters. A term that FDC truly cannot classify returns +UNRESOLVED. This is an empty code, not an error. That empty code is a +true statement about the term. A missing artifact bundle is different. It +is not a fact about any term. It is a broken build. The correct +response is to stop at once, rather than let every caller silently +receive a meaningless answer. + +`version` is the module version string, `"0.1.0"`. Like LatticeLib's +own version constant, it serves one purpose. A consumer can record +which build of EideticLib produced a given answer. + +`classifyLatticeCode(_:knownCodes:)` sorts a candidate lattice code +string into one of `LatticeCodeState`'s three cases. It uses +`LatticeCodeGrammar.isWellFormed(_:)` for the shape check. It uses the +caller-supplied `knownCodes` set for the known-versus-pending +distinction. The known-code set arrives as a parameter, rather than +living inside EideticLib as an owned canon. This keeps EideticLib +decoupled from any single caller's view of which codes currently +resolve to a label. Here is why that matters. Two callers can disagree +about which codes count as known. One might hold a newly synced frame. +The other might hold a stale one. EideticLib does not take a side. + +`lookup(_:)` is the default path. It guards on `FDC.isAvailable`. It +calls `FDC.encodeAnchor(term)`. Then it converts the result to an +`Anchor`. A resolved code is reported at a fixed confidence of 32, or +medium. FDC computes no calibrated score of its own, so EideticLib +always emits this same value. An UNRESOLVED term becomes an `Anchor` +with an empty `code`, a `nil` `wikidataQID`, and confidence 0. That is +the honest "nothing matched" result, never a best-guess fallback. + +`lookup(_:recordNovel:)` is the privacy variant. It produces the exact +same `Anchor` as `lookup(_:)` for the same term. It also threads +`recordNovel` through to `FDC.encodeAnchor(_:recordNovel:)`. When that +flag is `false`, any novel token found while building the term's +concept bag stays out of LatticeLib's shared learning pool. Here is why +this variant exists. The GLK memory-capture intake path classifies +user-supplied memory content. It does this before it decides whether +to store that content. That content must never leak even a single +token into a shared, cross-session pool. This holds even for a +rejected or sensitive capture. Classification runs before the write +decision, not after. + +`Anchor` is a plain `Equatable`, `Sendable`, `Codable` struct with four +fields. `code` is empty when the term is UNRESOLVED. `wikidataQID` is +optional. `confidence` is a `UInt8`. It is drawn from a fixed set of +five values: 0, 16, 32, 48, and 56. These stand for null, low, medium, +high, and verified. EideticLib currently only ever emits 0 or 32. +`dataVersion` is the FDC signatures version that produced the answer. +This lets a caller record provenance. The struct's doc comment notes +that it is byte-identical to the Rust port's `Anchor` under JSON +encoding. That identity is what lets a Swift app and a Rust daemon +exchange anchors through a shared store. + +## LatticeCodeState.swift + +This file provides two things: `LatticeCodeState` and +`LatticeCodeGrammar`. `LatticeCodeState` is the three-state +classification of a candidate lattice code. `LatticeCodeGrammar` is the +grammar check behind it. + +The file's opening comment frames the problem directly. A lattice code +shown to EideticLib falls into one of three states. A malformed code +fails the grammar. A known code is well formed and present in the +caller's current canon. A pending code is well formed, but not yet +present in that canon. Here is why a third state exists at all. FDC +frames carry a version and ship new codes over time. MOOTx01 estates +can federate, so one estate's stored code may predate another estate's +local frame. Rejecting an unrecognized but well-formed code would lose +data for good. A pending code instead round-trips intact through +storage until a newer frame resolves it. + +`LatticeCodeState` conforms to `Sendable`, `Hashable`, and `Codable`. +The `Codable` conformance is not incidental. The file's own comment +calls it "the core invariant of the launch plan's +valid-but-unknown-code requirement": a pending code must survive being +written to storage and read back without alteration. `rawCode` returns +the original string regardless of which case matched, so a storage +layer can persist the raw value without unpacking the enum first. +`isWellFormed` reads `true` for `.known` and `.pending`, and `false` +only for `.malformed`. Tests rely on this to assert that pending codes +are still accepted, just not yet resolved. + +`LatticeCodeGrammar` is a second, separate check. It enforces the same +grammar as LatticeLib's `Code.isWellFormed(_:)`. That grammar requires +three ASCII digits. It allows an optional dot, followed by one to eight +more ASCII digits. + +Here is why the grammar is reimplemented, rather than imported from +LatticeLib. The file's comment gives the reason. EideticLib can check a +code's shape without loading LatticeLib's frame, lexicon, and +signatures at all. A caller who only wants to know whether a string is +shaped like a code pays none of the cost of the full classification +engine. Shared conformance tests keep the two implementations in +agreement. The implementations do not share code. + +`maxExtensionDigits` is a pinned constant, `8`, matching LatticeLib's +own `Code.maxExtensionDigits`. `isWellFormed(_:)` splits the string on +its first `.`. It requires the integer part to be exactly three ASCII +digits. If a decimal part is present, it requires that part to hold one +to eight ASCII digits with no other characters. An empty decimal part, +meaning a trailing dot with nothing after it, fails the check. + +## Segmenter.swift + +This file provides `sentences(_:)` and `sentencesByDelimiter(_:)`, both +declared in an extension on the `EideticLib` enum. + +The file's opening comment draws a direct parallel to LatticeLib's +`WordClassTagger`. That file splits word tagging into two paths. One +path is deterministic: the fast-path table plus an HMM. The other path +is accelerated and Apple-only, built on `NLTagger`. Callers must opt +into the accelerated path. `Segmenter` applies that same shape to +sentence splitting instead of word tagging. `sentencesByDelimiter(_:)` +is the deterministic reference. It behaves the same on every platform. +`sentences(_:)` is the platform-routed entry point most callers should +use. On Apple platforms it defers to `NLTokenizer`. This tool correctly +handles tricky abbreviations. For example, it does not split "Dr." +into two sentences. It also handles locale-aware punctuation and other +edge cases that a naive delimiter split gets wrong. On non-Apple +platforms it falls back to `sentencesByDelimiter`. + +Why is this divergence acceptable here? LatticeLib treats the same +kind of divergence as unacceptable in word tagging. The comment cites +the "apple-nlp-accel constitutional constraint (C-2)." Downstream +consumers of sentence segments key each chunk by a content address. +That address is `(sourceID, startOffset, text)`, not a chunk index. +Two devices might split the same document into slightly different +sentences. Under an append-only conflict policy, this produces a +superset of chunks. It does not create a conflict that needs +reconciling. This is a much weaker consistency rule than the one FDC's +lattice codes carry. Those codes must match byte for byte across +devices, or federated recall fails. + +Both functions guarantee total coverage. For non-empty input, every +character appears in exactly one output segment. Joining the segments +back together reproduces the original text. This holds even in +pathological cases. Consider text that is nothing but delimiters. +Consider also text with no delimiter at all. That case returns as a +single segment holding the whole input. Empty input returns an empty +array from both functions. The file's history comment notes that the +code moved here on 2026-05-27, in a change tagged F16, from +`CorpusKit/Sources/CorpusKit/Chunker.swift`. That move gathered every +text-pipeline stage under LatticeLib and EideticLib. Those stages are +tokenizing, normalizing, stemming, tagging, and now segmenting. The +move ended the old pattern of scattering pipeline pieces across the +kits that consume them. + +`sentences(_:)` builds an `NLTokenizer(unit: .sentence)`, enumerates +sentence-unit ranges over the input, and collects the corresponding +substrings. Suppose the tokenizer produces no segments for non-empty +input. Then the function falls back to returning the whole input as one +segment, which preserves the total-coverage guarantee even in that edge +case. + +`sentencesByDelimiter(_:)` walks the string once. It splits the text +after each terminator: a period, an exclamation point, a question +mark, or a newline. It keeps each terminator attached to the segment +that ends there. Any trailing text after the final terminator becomes +its own segment. This is the function the Rust port's +`segmenter::sentences` mirrors exactly, because Rust has no +NLTokenizer-equivalent acceleration to port. + +## Rust Port and Conformance + +The `rust/` directory holds the second leg. `lib.rs` is the crate +surface, re-exporting `lookup` and `classify`. `anchor.rs` defines +`Anchor`, with an explicit `#[serde(rename = "wikidataQID")]` so the +JSON key matches Swift's `Codable` output exactly, rather than the +auto-derived `wikidataQid`. `lattice_code_state.rs` defines three +items: `LatticeCodeState`, `LatticeCodeGrammar`, and +`classify_lattice_code`. The wire format is tagged as +`{"state": ..., "code": ...}`, matching Swift's encoding. +`segmenter.rs` implements the delimiter algorithm only. There is no +Rust counterpart to the Apple-only `NLTokenizer` path, by design. + +The Rust crate depends on the sibling `lattice-lib` crate for +`Fdc::encode_anchor` and `Fdc::is_available`. This is the same +relationship Swift's `EideticLib` target holds with the LatticeLib +Swift package. `rust::lookup` panics under the identical condition that +Swift's `lookup` treats as a `fatalError`: unavailable FDC artifacts. + +Cross-language conformance runs on two fixtures. +`Tests/SharedVectors/lookup_vectors.json`, schema version 2, holds +twenty-six vectors. It pins expected FDC codes and Wikidata Q-IDs for a +set of inputs, including edge cases such as punctuation-only text. Both +`rust/tests/lookup_conformance_test.rs` and the Swift +`LookupConformanceTests.swift` load that same file and must match it +exactly. `Tests/SharedVectors/word_class_vectors.json` is a second +fixture. LatticeLib's own word-classification conformance surface +shares this fixture too. The EideticLib test target links LatticeLib +directly, so this fixture applies here as well. Suppose you change +`lookup`, `classifyLatticeCode`, +or either segmentation function. Then update both legs, and rerun both +test suites. The fixtures are the contract. diff --git a/packages/libs/EideticLib/docs/OVERVIEW.md b/packages/libs/EideticLib/docs/OVERVIEW.md new file mode 100644 index 0000000..a0135ae --- /dev/null +++ b/packages/libs/EideticLib/docs/OVERVIEW.md @@ -0,0 +1,163 @@ +--- +doc: OVERVIEW +package: EideticLib +repo: moot-semantics +authored_commit: 021ea704162f86fccea2f030ea9419dacc30a345 +authored_date: 2026-07-04 +sources: + - path: Sources/EideticLib/EideticLib.swift + blob: 1839588dc98279c1eee2786f1974457a5608742b + - path: Sources/EideticLib/LatticeCodeState.swift + blob: 7580261208b77c7e79363a510920c413054264a1 + - path: Sources/EideticLib/Segmenter.swift + blob: c5313d93bf7ca1cbf577da6ffabc9197b5e788e2 +--- + +# EideticLib Overview + +## What This Library Does + +EideticLib sits in front of LatticeLib's classification engine. It gives +callers one simple way to classify a term. A caller passes in a term. +EideticLib returns an `Anchor`. + +An `Anchor` holds four fields: a lattice code, a Wikidata Q-ID, a +confidence value, and a data version. A Wikidata Q-ID is a public +identifier for one concept. It works the same way across every +language. For example, `Q144` names the concept "dog." + +EideticLib holds no reference data of its own. Every lookup calls +`FDC.encodeAnchor`. That call is the entry point into LatticeLib's +Frame-Directed Classification engine, known as FDC. + +EideticLib also carries two small utilities. The first checks whether +a lattice code string is well formed and known. The second splits text +into sentences. Neither utility needs the FDC engine to run. All three +surfaces share one goal: give callers a clear answer without asking +them to learn LatticeLib's internals. + +## The Problem It Solves + +LatticeLib's `FDC` type is powerful, but it sits at a low level. It +owns pinned artifacts. It exposes several scoring modes. It returns raw +codes and ancestry chains. Most callers need none of that detail. They +need one function: give me a term, and tell me what it means. EideticLib +is that function. + +EideticLib fixes a confidence scale. It wraps the result in a stable +`Codable` type. It enforces a strict failure policy. A caller never has +to guess whether an empty answer means "no data loaded" or "nothing +matched." + +A second problem concerns privacy of memory content. Some callers +classify text a user typed into memory. That text must never leak into +LatticeLib's shared pool for novel tokens. Even one stray token would +be a problem. EideticLib exposes a `recordNovel` parameter for this +case. When a caller turns this off, the pool learns nothing from that +text. The caller still gets back the exact same anchor. One call path +can then serve both public and private text. + +A third problem concerns the lattice code canon itself. FDC frames +carry a version number. Federated devices can run different frame +versions at the same time. A device might receive a lattice code from +another estate, one user's full memory store. The receiving device's +own frame may not yet recognize that code. Discarding the code would +lose information. Guessing at its meaning would break the classifier's +honesty policy. `classifyLatticeCode` sorts a code into one of three +states: malformed, known, or pending. This lets a storage layer keep a +code it cannot resolve yet. The code settles later, once a newer frame +ships. + +The fourth problem is splitting text into sentences before later +pipeline stages run. Sentence boundaries look simple at first. Then +abbreviations appear, such as "Dr." or "e.g." Locale-specific +punctuation appears too. `Segmenter` solves this the way LatticeLib +solves tokenization. It offers a fast, accurate path on Apple +platforms. It offers a plain, deterministic fallback everywhere else. + +## How It Works + +`EideticLib.lookup(_:)` first checks `FDC.isAvailable`. If the pinned +FDC artifacts failed to load, the function stops the process with +`fatalError`. It never returns an empty or made-up answer instead. A +failed load means the binary shipped without its required data. That +is a build defect. No caller should try to recover from it at runtime. +This is a deliberate stance. MOOTx01's provenance rules treat a +placeholder identity that persists as a fabricated identity. So +EideticLib refuses to invent one. + +Given available artifacts, `lookup` calls `FDC.encodeAnchor(term)`. +This call returns a lattice code and a Wikidata Q-ID. It returns `nil` +for the code when the term is UNRESOLVED. UNRESOLVED is the honest "no +answer" result. LatticeLib returns it rather than guess. EideticLib +packages a resolved result as an `Anchor` with a fixed confidence of +32, meaning medium, because FDC itself produces no calibrated +confidence score. An UNRESOLVED term becomes an `Anchor` with an empty +code, no Q-ID, and confidence 0. + +`lookup(_:recordNovel:)` is the same function with one difference. It +suppresses the pool-recording side effect. Callers use this variant to +classify private memory content. + +`classifyLatticeCode(_:knownCodes:)` runs independent of the FDC +engine. It checks a code string against a small grammar. The grammar +allows three digits, followed by an optional decimal extension of up +to eight digits. It uses `LatticeCodeGrammar`, a rule set that mirrors +LatticeLib's own `Code.isWellFormed(_:)`. A code that fails the grammar +is `.malformed`. A well-formed code present in the caller's +`knownCodes` set is `.known`. A well-formed code absent from that set +is `.pending`. A pending code is still accepted. It is stored and +round-tripped intact, rather than rejected. + +`EideticLib.sentences(_:)` splits text into sentence substrings. On +Apple platforms, it uses `NLTokenizer`. This tool understands +abbreviations and locale-aware boundaries. On every other platform, and +whenever strict cross-platform identity matters, it uses +`sentencesByDelimiter(_:)` instead. That function splits on periods, +exclamation points, question marks, and newlines. It keeps each +terminator attached to the segment it ends. Both functions guarantee +total coverage. Every character of the input appears in exactly one +output segment. A caller can always rebuild the original text by +joining the segments back together. + +## How the Pieces Fit + +EideticLib is not a pipeline the way LatticeLib is. It has three +surfaces: lookup, code-state classification, and sentence +segmentation. These surfaces do not call one another. Each is an +independent, narrow entry point. Each answers one question. Each has +its own link to the rest of the SDK. `lookup` crosses into LatticeLib's +FDC engine. Code classification is a pure, dependency-free grammar +check. Sentence segmentation crosses into Apple's NaturalLanguage +framework, but only on Apple platforms. + +This fan-out shape is also EideticLib's value. It is a single, stable +package. Other libraries and kits import it for a handful of small, +unrelated conveniences. They do not each need to depend on LatticeLib, +NaturalLanguage, and a code-grammar checker separately. CorpusKit, for +example, imports EideticLib only for `sentences(_:)`. CorpusKit uses +this function to split memory text into chunks before further +processing. + +The library keeps the same two-language guarantee as LatticeLib. A +Swift leg serves Apple platforms. A Rust leg in `rust/` mirrors +`lookup`, `classifyLatticeCode`, and `sentencesByDelimiter` exactly. +Shared conformance vectors live in +`Tests/SharedVectors/lookup_vectors.json`. They gate every release, so +both legs agree byte for byte on every case in the fixture. + +## What Ships in the Package + +The package ships three Swift source files under `Sources/EideticLib/`. +It also ships an intentionally empty `Resources/` directory. EideticLib +carries no reference data of its own. LatticeLib owns the pinned +artifacts that `lookup` depends on. The package also ships the Rust +port under `rust/`. + +The Rust crate is named `eidetic-lib`. It depends on the sibling +`lattice-lib` crate. This mirrors how the Swift target depends on the +LatticeLib Swift package. + +Cross-language conformance rests on two checks. The first is +`Tests/SharedVectors/lookup_vectors.json`, with twenty-six vectors. The +second is the set of integration tests in `rust/tests/`. diff --git a/packages/libs/LatticeLib/docs/AGENT_MAP.md b/packages/libs/LatticeLib/docs/AGENT_MAP.md new file mode 100644 index 0000000..14dd1aa --- /dev/null +++ b/packages/libs/LatticeLib/docs/AGENT_MAP.md @@ -0,0 +1,163 @@ +--- +doc: AGENT_MAP +package: LatticeLib +repo: moot-semantics +authored_commit: bbfef540c1b675b3fb9a493e596499b0fdf2e826 +authored_date: 2026-07-04 +sources: + - path: Sources/LatticeLib/Code.swift + blob: e2c8307da3a2821bbabd7055a8f714754289df0f + - path: Sources/LatticeLib/CodeSignature.swift + blob: b9245f8e8a8c913bcdcb27287aab45cd2729abd9 + - path: Sources/LatticeLib/ConceptBag.swift + blob: db18742c50e495682521bd7dd544b3725969b467 + - path: Sources/LatticeLib/FDCFrame.swift + blob: 8d83ab902206ffdb4d3ee9351a6443c90bcead77 + - path: Sources/LatticeLib/FDCMatcher.swift + blob: 330b99e7af1d6366425d4956652a3dd11b1a1c4b + - path: Sources/LatticeLib/FDCRuntime.swift + blob: 2fd34b7657f6888117fb9d4226ef314769d75800 + - path: Sources/LatticeLib/HMMTagger.swift + blob: 45ecd95be83f7416851a80070f85aa05aea7fc4a + - path: Sources/LatticeLib/LatticeLib.swift + blob: 88712d0a73a75d9400741a871719f39be2edc584 + - path: Sources/LatticeLib/Lexicon.swift + blob: 2a51dfbf417b542b60f5b5597e0850957c5b6631 + - path: Sources/LatticeLib/LexRank.swift + blob: 6e3cdb0c19c7a7b68992a6bd79780d15f5aa711e + - path: Sources/LatticeLib/NFKCSubset.swift + blob: fd2753d05e13b1bc430d2688cc78d0395dca58a6 + - path: Sources/LatticeLib/Normalizer.swift + blob: f989b93b3dc55751f21bcca65e8e2892bb4193e7 + - path: Sources/LatticeLib/NovelPoolSubmitter.swift + blob: be91dd71f60a81681c8102567ceeff14ad974526 + - path: Sources/LatticeLib/NovelTokenCache.swift + blob: 70342e766178eb15c14f52b28cb51d9aa11c6890 + - path: Sources/LatticeLib/NovelTokenTaggerChoice.swift + blob: 0c9b84a315ca7b31e4bf626f1830c3eea243d8aa + - path: Sources/LatticeLib/PoolReducer.swift + blob: e0ccb6a314ec39c47d31dab798841af93c4d394a + - path: Sources/LatticeLib/QIDClosure.swift + blob: 11ef3156c12ce27d6b1c1401e3b47a94c3f1bf63 + - path: Sources/LatticeLib/Stemmer.swift + blob: 042fe54236cbe61f1cb4126dc27dde97a3f5cc5d + - path: Sources/LatticeLib/Tokenizer.swift + blob: 82a8b85b5c31163d51194c7f28320305f8c798af + - path: Sources/LatticeLib/WordClass.swift + blob: 9d4851967dca01b266ad0d8f4f35cf4a757d3d7d + - path: Sources/LatticeLib/WordClassTable.swift + blob: 731f6f90c59aae71c73114ede301877d1bf8035f + - path: Sources/LatticeLib/WordClassTagger.swift + blob: 15e27da5cdeca3540654914992e8c78f83ed937a +--- + +# AGENT_MAP: LatticeLib + +PURPOSE: deterministic on-device text→classification-code engine (FDC: Frame-Directed Classification) + shared text primitives. Text → concept bag → scored match against pinned code signatures → decimal-frame descent → code (or nil=UNRESOLVED). + +DEPS: imports SubstrateML (EigenvalueCentrality, build-time LexRank only), NaturalLanguage (LexRank sentence-split; optional NLTagger path), OSLog. Imported by: EideticLib, AriaMcpKit, apps/moot-mgr, tools/seed-generator. Rust port in rust/ mirrors everything; conformance fixtures rust/tests/fixtures/*.json gate byte-identity. + +ENTRY POINTS (most callers need only these): +- FDCRuntime.swift:32 `FDC.encodeAnchor(_ text:) -> (code: String?, conceptQID: String?)`: main runtime classify +- FDCRuntime.swift:45 `FDC.encodeAnchor(_:recordNovel:)`: same, recordNovel:false = privacy seam (no pool writes) +- WordClassTagger.swift:65 `LatticeLib.wordClass(_ token:) -> WordClass`: single-token noun/verb/other + +## Symbol Table + +### Runtime facade: FDCRuntime.swift +- :15 `enum FDC`: artifact-owning facade; loads lexicon+frame+signatures once/process +- :23 `FDC.stopThreshold = 1`: pinned descent cutoff (inert on shallow v1.0 frame) +- :27 `encode(_ text:) -> String?`: code only; nil = UNRESOLVED or artifacts missing +- :32/:45 `encodeAnchor(...)`: code + dominant Q-ID; see ENTRY POINTS +- :50 `isAvailable: Bool`: artifacts loaded? +- :54 `dataVersion: String`: signatures version for provenance; "0.0.0-unavailable" on load failure +- :67 `ancestors(of code:) -> [String]`: frame ancestry, root-first, excl. self +- :84 `label(for code:) -> String?`: human heading; 3-digit codes report parent's label (leaf labels are multi-subject compounds) + +### Matcher: FDCMatcher.swift +- :30 `struct FDCMatcher: Sendable`: Steps 4–5; direct init for custom artifacts/tests +- :49 `enum ScoreMode`: .raw/.idf/.cosine/.idfCosine; runtime ships .idf, direct-init default .raw +- :63 `stopThreshold: Int`: raw-overlap descent cutoff, mode-independent +- :120 `init(lexicon:frame:signatures:stopThreshold:scoreMode:)`: interns terms→dense Int ids (alphabetical order ⇒ Int sort == String sort ⇒ bit-stable float sums); builds inverted index + idf + norms +- :267 `maximumTiedWinnersForClassification = 4`: >4 tied argmax winners ⇒ UNRESOLVED (generic-vocabulary guard) +- :271 `encode(_:)`, :280 `encodeAnchor(_:)`, :296 `encodeAnchor(_:recordNovel:)`: public encode surface + +### Bag construction: ConceptBag.swift +- :25 `typealias ConceptBag = [String: Int]`: conceptID|surface → count +- :27 `enum BagBuilder`: Steps 1–3 +- :37 `bag(_:lexicon:keep:)`: default (keep = [.noun,.verb]); Q-ID lexicon hit bypasses POS filter (named-entity relaxation, deterministic via pinned lexicon) +- :82 `bag(_:lexicon:keep:recordNovel:)`: identical bag; recordNovel:false suppresses pool side effect +- :118 `bag(_:lexicon:keep:taggerChoice:)`: estate-configured tagger threading + +### Text primitives +- Tokenizer.swift:21 `Tokenizer.tokenize(_:) -> [String]`: UAX #29 words, input order +- Normalizer.swift:55 `Normalizer.normalize(_:) -> String`: NFKCSubset.fold then lowercased; fold MUST run first +- NFKCSubset.swift:40 `NFKCSubset.fold(_:)` (internal): ASCII-target compatibility subset; table mirrored verbatim in rust/src/normalizer.rs: edit both or neither +- Stemmer.swift:69 `Stemmer.stem(_:) -> String`: Porter2/Snowball English; <3 chars unchanged +- Stemmer.swift:55 `bundledReferenceCorpus() -> Data?`: conformance corpus accessor + +### Word-class tagging: WordClassTagger.swift (extension LatticeLib) +- :65 `wordClass(_:)`: table fast path (verb set checked BEFORE noun set) → HMM fallback; NEVER NLTagger +- :102 `taggerEnabled(osVersion:minOSVersion:)`: NLTagger OS gate; unparseable minOS fails closed +- :208 `wordClass(_:recordNovel:)`: pool-suppressing variant +- :285 `wordClass(_:tagger:)` / :330 `wordClass(_:tagger:recordNovel:)`: estate-choice dispatch; .nlTagger fails closed to .other below min OS, falls back to HMM on non-Apple +- WordClass.swift:23 `enum WordClass: String`: noun|verb|other; string-backed for wire stability +- NovelTokenTaggerChoice.swift:22 `enum NovelTokenTaggerChoice`: .hmm (default, federatable) | .nlTagger (Apple-only, non-deterministic cross-platform); mirrors PersistenceKit type, bridge by switch +- HMMTagger.swift:70 `enum HMMTagger` (internal): 3-state integer Viterbi; artifact HMMTaggerModel.json (hmm-viterbi-3, MASC 3.0.0 rare-word-trained); tie → lowest state index (noun Apple App Support > XDG; single source of truth for read+write sides +- NovelPoolSubmitter.swift:97 `tableArtifactURL()`: writable merged WordClassTable.json, SIBLING of pool dir +- PoolReducer.swift:154 `PoolReducer.reduce(poolDirectory:tableArtifactURL:now:maxFiles:)`: batch merge; seeds writable artifact from bundle; quarantines malformed/stale-version files; NOUN/VERB only; first-occurrence dedup; archives consumed files (idempotent); DO NOT call from hot path +- PoolReducer.swift:65 `PoolReduceResult` / :105 `PoolReducerError`: result summary / failure classes + +### Frame + codes +- Code.swift:34 `Code.isWellFormed(_:)`: grammar only: 3 digits + optional ≤8-digit decimal ext (:31 maxExtensionDigits=8); known-vs-pending is caller's concern +- Code.swift:58 `Code.integerBase(of:)`: "540.137" → 540 +- FDCFrame.swift:17 `FDCEntry`: code + verbatim label +- FDCFrame.swift:34 `FDCFrame`: versioned code list; ancestry DERIVED, not stored +- FDCFrame.swift:142 `ancestors(of:)`: TWO regimes: integer head is Dewey positional (parent("006")="000"), decimal tail is per-segment (parent("006.6")="006"); NOT a dot-split +- FDCFrame.swift:131 `children(of:)`: immediate children, sorted + +### Build-time only (never on-device) +- Lexicon.swift:30 `CanonicalizationLexicon`: version/language/entries (stem→conceptID); version is part of agreement protocol +- Lexicon.swift:91 `LexiconBuilder.build(_:)`: Wikidata-primary, WordNet-disambiguated; 3-tier deterministic conflict resolution; single-token keys only +- CodeSignature.swift:36 `SignatureAssembler.merge(label:title:article:weights:)`: source weights label 3 / title 2 / article 1 (:13 SourceWeights) +- CodeSignature.swift:52 `accumulateAncestors(ownTerms:ancestorsOf:)`: signature += all ancestors' own terms +- CodeSignature.swift:24 `CodeSignature`: terms + fingerprint (nil until SimHash pass) +- LexRank.swift:25 `LexRank.reduce(_:sentences:)`: top-N central sentences (default :20 =10, cosine threshold :21 =0.1) via SubstrateML.EigenvalueCentrality; deterministic but NOT cross-platform-gated + +### Q-ID closure: QIDClosure.swift +- :45 `QIDClosure.ancestors(of qid:) -> [String]`: transitive P31/P279 closure over pinned QIDClosureEdges.json; BFS, excl. self, sorted by Q-number; memoized per process +- :68 `isAvailable` / :73 `dataVersion`: artifact presence / provenance + +### Module: LatticeLib.swift +- :13 `enum LatticeLib`: namespace; :17 `version = "1.0.0"` bumped with artifact ships + +## INVARIANTS / GOTCHAS + +- DETERMINISM IS THE CONTRACT. encode() is pure over (text, artifact versions). Any change to Tokenizer/Normalizer/NFKCSubset/Stemmer/HMMTagger/FDCMatcher scoring must be mirrored in rust/src/ and pass rust/tests/fixtures/{normalize,tag,fdc}_conformance.json + Swift conformance tests. Same for artifact regeneration. +- nil code = UNRESOLVED, by design: empty bag, no signature overlap, or >4 tied winners. Never "fix" by guessing. +- Verb set checked before noun set in every wordClass overload: a token in both is a verb. Do not reorder. +- HMM (not NLTagger) is the default novel-token tagger on ALL platforms incl. Apple. NLTagger only via explicit .nlTagger estate choice; it breaks cross-platform determinism and federation. +- NFKCSubset table + Rust normalizer.rs are ONE table in two files. Never edit one alone. +- FDCMatcher term interning: IDs assigned in ascending String order: required so sorted-Int iteration == sorted-String iteration ⇒ bit-identical float sums. Do not change assignment order. +- FDCFrame ancestry: integer head ≠ decimal tail regimes. parent("006") is "000" (Dewey positional), not "00". A dot-split reimplementation is wrong. +- stopThreshold compares RAW integer overlap, never the (possibly normalized) score. +- Pinned constants: do not change without a new artifact version + conformance regen: poolSubmitThreshold 50, stopThreshold 1, maxExtensionDigits 8, maximumTiedWinnersForClassification 4, SourceWeights 3/2/1, HMM logScale 1000, LexRank 10/0.1. +- Privacy seams: recordNovel:false variants produce byte-identical results with pool accumulation suppressed: required for user memory content (GLK capture, EideticLib.lookup). sharedNovelCache writes files ONLY when LATTICE_POOL_DIR is set; default submitter is a no-op. +- Bundled WordClassTable.json is a small seed (21 nouns / 18 verbs, v1.0.0): in practice most tokens are novel and route through the HMM until the pool loop grows the writable table. +- Artifacts load once per process (static let); FDC/QIDClosure fail soft (isAvailable=false, encode→nil): check isAvailable in hosts with unusual bundle setups. +- PoolReducer is batch-only (governor idle-tick or operator CLI). Bounded by maxFiles per run; backlog drains across runs. +- WordClassTableCache.swap is the ONLY legal live-update path for tagging behavior; readers get whole-snapshot consistency, and version() observability keys determinism. +- LexRank is build-time-only; it may use floats and NLTokenizer freely. Do not move it into the runtime encode path. diff --git a/packages/libs/LatticeLib/docs/DETAILS.md b/packages/libs/LatticeLib/docs/DETAILS.md new file mode 100644 index 0000000..0dffa77 --- /dev/null +++ b/packages/libs/LatticeLib/docs/DETAILS.md @@ -0,0 +1,603 @@ +--- +doc: DETAILS +package: LatticeLib +repo: moot-semantics +authored_commit: bbfef540c1b675b3fb9a493e596499b0fdf2e826 +authored_date: 2026-07-04 +sources: + - path: Sources/LatticeLib/Code.swift + blob: e2c8307da3a2821bbabd7055a8f714754289df0f + - path: Sources/LatticeLib/CodeSignature.swift + blob: b9245f8e8a8c913bcdcb27287aab45cd2729abd9 + - path: Sources/LatticeLib/ConceptBag.swift + blob: db18742c50e495682521bd7dd544b3725969b467 + - path: Sources/LatticeLib/FDCFrame.swift + blob: 8d83ab902206ffdb4d3ee9351a6443c90bcead77 + - path: Sources/LatticeLib/FDCMatcher.swift + blob: 330b99e7af1d6366425d4956652a3dd11b1a1c4b + - path: Sources/LatticeLib/FDCRuntime.swift + blob: 2fd34b7657f6888117fb9d4226ef314769d75800 + - path: Sources/LatticeLib/HMMTagger.swift + blob: 45ecd95be83f7416851a80070f85aa05aea7fc4a + - path: Sources/LatticeLib/LatticeLib.swift + blob: 88712d0a73a75d9400741a871719f39be2edc584 + - path: Sources/LatticeLib/Lexicon.swift + blob: 2a51dfbf417b542b60f5b5597e0850957c5b6631 + - path: Sources/LatticeLib/LexRank.swift + blob: 6e3cdb0c19c7a7b68992a6bd79780d15f5aa711e + - path: Sources/LatticeLib/NFKCSubset.swift + blob: fd2753d05e13b1bc430d2688cc78d0395dca58a6 + - path: Sources/LatticeLib/Normalizer.swift + blob: f989b93b3dc55751f21bcca65e8e2892bb4193e7 + - path: Sources/LatticeLib/NovelPoolSubmitter.swift + blob: be91dd71f60a81681c8102567ceeff14ad974526 + - path: Sources/LatticeLib/NovelTokenCache.swift + blob: 70342e766178eb15c14f52b28cb51d9aa11c6890 + - path: Sources/LatticeLib/NovelTokenTaggerChoice.swift + blob: 0c9b84a315ca7b31e4bf626f1830c3eea243d8aa + - path: Sources/LatticeLib/PoolReducer.swift + blob: e0ccb6a314ec39c47d31dab798841af93c4d394a + - path: Sources/LatticeLib/QIDClosure.swift + blob: 11ef3156c12ce27d6b1c1401e3b47a94c3f1bf63 + - path: Sources/LatticeLib/Stemmer.swift + blob: 042fe54236cbe61f1cb4126dc27dde97a3f5cc5d + - path: Sources/LatticeLib/Tokenizer.swift + blob: 82a8b85b5c31163d51194c7f28320305f8c798af + - path: Sources/LatticeLib/WordClass.swift + blob: 9d4851967dca01b266ad0d8f4f35cf4a757d3d7d + - path: Sources/LatticeLib/WordClassTable.swift + blob: 731f6f90c59aae71c73114ede301877d1bf8035f + - path: Sources/LatticeLib/WordClassTagger.swift + blob: 15e27da5cdeca3540654914992e8c78f83ed937a +--- + +# LatticeLib Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in pipeline +order. First comes the module surface. Then come the text primitives, the +word-class machinery, and the learning loop. Last comes the classification +engine and its helpers. + +## LatticeLib.swift + +This file provides the module surface. It holds the public `LatticeLib` +enum and the module version. + +Swift has no module-level functions. So libraries often use an empty enum +as a namespace. The `LatticeLib` enum is that namespace. Other files +extend it. The word-class tagging functions in `WordClassTagger.swift` all +live in an extension of this enum. + +`LatticeLib.version` is the module version string. It lets consumers +record which library version produced a result. The version is bumped in +lockstep with the bundled FDC artifacts when new signatures ship. A result +is only reproducible against the exact artifact set that produced it. + +## Tokenizer.swift + +This file provides `Tokenizer`, which splits raw text into words. + +Splitting text into words sounds simple. It is full of edge cases instead: +apostrophes, hyphens, non-Latin scripts. The tokenizer does not invent its +own rules. It follows Unicode Standard Annex #29, the industry-standard +word-boundary specification. This public standard is what lets the Swift +leg and the Rust leg agree byte for byte. Both implement the same +published rules. A shared conformance fixture proves it. + +`Tokenizer.tokenize(_:)` returns the words of a string in input order. It +drops whitespace and punctuation. It works by asking Foundation to +enumerate word substrings. This routes to ICU's word-boundary analyzer. +The Rust port uses the `unicode-segmentation` crate. That crate implements +the same annex. + +## NFKCSubset.swift + +This file provides the internal compatibility-fold table used by the +normalizer. It maps visually equivalent Unicode characters to plain ASCII. +A full-width `Ａ` becomes `A`. The ligature `fi` becomes `fi`. A superscript +`²` becomes `2`. + +Why not use the platform's full Unicode NFKC normalization? The Rust leg +cannot reproduce it byte for byte without a large external dependency. +The two legs must never disagree. So both legs instead implement the same +explicit table. It is defined once here and mirrored verbatim in +`rust/src/normalizer.rs`. Same table plus same algorithm equals identical +output by construction. Every mapping is a strict subset of official NFKC. +Each is limited to mappings whose target is plain ASCII. Those are the +mappings that improve matching for an ASCII-English corpus. + +`fold(_:)` rewrites a string scalar by scalar. It replaces each mapped +character and passes everything else through unchanged. `foldScalar(_:)` +holds the actual mapping. An explicit literal table comes first. Then come +arithmetic range families: full-width forms, super and subscripts, circled +letters, and Roman numerals. These are expressed as offsets, so the table +stays small and easy to check. Canonical recomposition, such as half-width +katakana, is deliberately out of scope. The file documents this so a +future reader does not mistake the omission for a bug. + +## Normalizer.swift + +This file provides `Normalizer`, the canonicalization front door. It runs +before stemming and lexicon lookup. This lets different surface spellings +of the same word collapse to one key. + +`Normalizer.normalize(_:)` applies the `NFKCSubset` fold, then a +Unicode-aware lowercase. The fold runs first for a reason the code spells +out. A full-width `Ａ` must first become `A`. Only then does it fold to +`a`, matching the key that a plain `a` produces. Lowercasing alone would +leave the full-width form distinct. The same word would then split into +two different lexicon keys. The function is deterministic. It is +byte-identical to the Rust port for every input in the shared conformance +fixture. + +## Stemmer.swift + +This file provides `Stemmer`, a hand-ported implementation of the Porter2 +(Snowball English) stemming algorithm. Stemming reduces a word to its +root: "dogs" becomes "dog," and "classification" becomes "classif." The +point is recall. Text about "running" and text about "runs" should reach +the same lexicon entry. + +Porter2 removes English suffixes in five ordered steps. Two regions of the +word, called R1 and R2, guide this process. Vowel-consonant transitions +compute these regions. Most removals only apply inside a region. This +prevents overstripping short words. Two exception lists handle irregular +forms. For example, "skies" becomes "sky" rather than "ski." + +`Stemmer.stem(_:)` runs the full algorithm. First comes preprocessing: +apostrophe marking and consonant-y marking. Then comes region computation, +then steps zero through five. Words shorter than three letters return +unchanged, per the specification. `Stemmer.bundledReferenceCorpus()` +exposes the pinned test corpus, `Resources/SnowballEnglish.json`. The +conformance test uses it to verify a claim. This port and the Rust port, +which uses the `rust-stemmers` crate, must produce byte-identical stems +for every input. + +## WordClass.swift + +This file provides `WordClass`. This is the label produced by tagging Step +1 of the encoder: `.noun`, `.verb`, or `.other`. + +Step 1 keeps only nouns and verbs. They carry the subject matter of a +sentence. `.other` is the discard bucket for everything else: articles, +prepositions, adjectives, punctuation. The enum is backed by strings. So +it serializes to a stable, human-readable JSON form: `"noun"`, `"verb"`, +`"other"`. Both the shared conformance vectors and the Rust port read this +form. + +## WordClassTable.swift + +This file provides the noun and verb fast-path table. It also provides its +live-swappable process cache. The table is the first stop for every +token. A token present in it resolves to its word class in constant time. +No tagger is invoked in that case. + +Three load functions cover the table's life cycle. `loadBundled()` reads +the pinned snapshot shipped in `Resources/WordClassTable.json`. +`loadWritable()` reads the merged artifact that `PoolReducer` maintains +outside the app bundle. `loadWithPrecedence()` is the canonical entry. It +prefers the writable merged artifact, because that one contains learned +tokens. It falls back to the bundled pristine table only when needed. + +`WordClassTableCache` solves a harder problem. It adopts a new table while +the process keeps running. Waiting for a process restart would delay +learning indefinitely after the reducer merges new tokens. The cache +instead publishes an immutable snapshot behind a lock. This snapshot is a +`WordClassTableSnapshot`: the parsed table plus precomputed membership +sets. Readers copy the snapshot reference out. They test membership +outside the lock. A swap can therefore never produce a torn read. A reader +sees the whole old table or the whole new one, never a mix. `swap(_:)` +publishes a new snapshot and bumps a version counter. +`reloadFromPrecedence()` and `reload(fromArtifact:)` re-resolve the table +from disk and publish it. The version counter matters for determinism. +Tagging is deterministic given the input and the table version. The +counter makes that version observable. + +## WordClassTagger.swift + +This file provides the public tagging API, `LatticeLib.wordClass`, in its +four overloads. This is encoder Step 1 as callers see it. + +Every overload shares the same fast path. The token is lowercased. Then it +is checked against the live table snapshot: the verb set first, then the +noun set. A token listed in both resolves to verb. Only a token absent +from the table, a novel token, reaches a tagger. + +`wordClass(_:)` is the default path. Novel tokens go to the deterministic +HMM tagger on all platforms, including Apple. This rule keeps the Swift +and Rust ports bit-identical. Apple's NLTagger is a black box that can +change between OS releases. So it is never in the default path. + +`wordClass(_:tagger:)` threads an explicit estate choice. An estate is one +user's memory store. Each estate picks its novel-token tagger once, at +creation. The `.hmm` choice is the cross-platform baseline. The +`.nlTagger` choice opts in to Apple's NaturalLanguage tagger. It buys +higher accuracy at the cost of cross-platform determinism. +`taggerEnabled(osVersion:minOSVersion:)` gates that choice. The table pins +the NLTagger OS version that seeded it. An older OS fails closed to +`.other` rather than invoke a differently behaving tagger. + +`wordClass(_:recordNovel:)` and `wordClass(_:tagger:recordNovel:)` add +privacy control. By default, a novel token's tag is recorded into the +shared pool cache. This lets the table learn it later. Sometimes the text +being classified is private memory content. Callers then pass +`recordNovel: false`, and the side effect is suppressed. The returned tag +is identical either way. Only the recording changes. The shared cache +itself, `sharedNovelCache`, is wired to a real file writer only when the +`LATTICE_POOL_DIR` environment variable is set. Otherwise the submitter is +a no-op. No deployment leaks tokens to disk by default. + +## HMMTagger.swift + +This file provides the deterministic novel-token tagger. It is a +three-state Hidden Markov Model, over noun, verb, and other. Integer +Viterbi decodes it. + +Why a statistical model here at all? Novel tokens are exactly the words no +table covers. So some generalization is required. Why integer arithmetic, +though? Floating-point rounding can differ across platforms and +compilers. This tagger must produce the same answer on every port. The +model weights are log-likelihoods. They are scaled by one thousand and +rounded to integers at build time. Runtime scoring is pure integer +addition and comparison. + +The observation alphabet is morphological. Each token maps to one +observation, such as "ends in -ing" or "ends in -tion." Other options are +"contains a non-letter" or "plain." `observe(_:)` checks these in a fixed +priority order. Both ports replicate that order exactly. `tag(_:)` then +picks the state with the best initial plus emission weight. For a +one-token sequence, Viterbi reduces to that best-scoring pick. Ties +resolve to the lowest state index. + +The weights ship in the frozen artifact `Resources/HMMTaggerModel.json`. +They are trained on the MASC 3.0.0 Penn Treebank annotation. The training +deliberately uses only the corpus's rare words, at frequency one. The +tagger only ever sees out-of-vocabulary words. Rare words are the best +statistical proxy for those words. This gives the correct noun-leaning +prior for unknown words. Estimating from the full corpus would let +frequent function words dominate instead. It would then wrongly default +unknowns to `.other`. + +## NovelTokenTaggerChoice.swift + +This file provides `NovelTokenTaggerChoice`. This is the two-case enum, +`.hmm` and `.nlTagger`, that selects the novel-token tagger for an estate. + +The type mirrors an identically named enum in PersistenceKit. There, the +choice is stored on the estate configuration. The two packages define the +type independently, so that neither depends on the other. A consumer +bridges between them with a trivial switch. The file documents the +threading rule: pass the choice down as a parameter from the estate +configuration to the call sites. Never pass it through global state. +`.hmm` is the default and the federation-safe choice. `.nlTagger` is +Apple-only. It trades determinism for accuracy. + +## NovelTokenCache.swift + +This file provides the local accumulation cache for novel-token tags. It +also provides two wire-format types, `PoolEntry` and `PoolSubmission`. + +When the tagger classifies a novel token, the result is worth keeping. If +many devices see the same unknown word, the shared table should learn it. +`NovelTokenCache.record(token:wordClass:)` appends the tagged token to a +pending list under a lock. At exactly `poolSubmitThreshold`, fifty +entries, the cache builds a `PoolSubmission`. It is stamped with the table +version, platform, and tagger version. The cache then drains the list. It +hands the payload to an injected submitter closure outside the lock. +Entries below the threshold are kept indefinitely. There is no aging or +cleanup, by contract. + +The submitter is injected at construction. Tests can then assert the +drain without touching the file system. Hosts that never configure a pool +get a no-op this way. The version stamps exist for a reason. The pool +consumer discards submissions made against a stale table version. An +observation is only meaningful relative to the table that failed to +contain it. + +## NovelPoolSubmitter.swift + +This file provides the production submitter factory. It turns a drained +`PoolSubmission` into a JSON file in a local pool directory. + +The pool is a directory of files, not a database or a network call. Files +are cheap, observable, and easy to reason about. A future reducer can +process them in name order. Each file name embeds an ISO 8601 timestamp +plus a short unique suffix. Writes are atomic and fire-and-forget. A +failed write is logged, and the data is discarded for that cycle. It is +never retried and never crashed on. Pool data is recoverable from future +observations. + +`make(poolDirectory:)` returns a submitter closure bound to a directory. +`makeDefault()` binds to the resolved default. `poolDirectory()` resolves +that default. It checks the `LATTICE_POOL_DIR` environment variable first. +Otherwise it uses an Application Support path on Apple platforms, or an +XDG data path elsewhere. This function is public for a reason. The write +side is the submitter. The read side is the governor that triggers the +reducer. The two sides must agree on the path. So this function lives +here as the single source of truth. `tableArtifactURL()` resolves the +sibling location of the writable merged table. That table is +`WordClassTable.json`, next to the pool directory. This is the file the +reducer updates and `WordClassTable.loadWritable()` reads. + +## PoolReducer.swift + +This file provides the batch job that closes the learning loop. It reads +pooled submissions. It merges qualifying novel tokens into the writable +word-class table artifact. + +`PoolReducer.reduce(poolDirectory:tableArtifactURL:now:maxFiles:)` runs in +steps. It first seeds the writable artifact from the bundled table if none +exists. The reducer cannot write into the read-only app bundle otherwise. +It then loads the table and enumerates `pool_*.json` files. It processes +the oldest `maxFiles` of them. Bounding the batch keeps a large backlog +from stalling the caller. The remainder drains on later runs. Each file is +decoded and version-checked. A malformed or stale-version file moves to a +quarantine directory rather than being deleted. This preserves forensic +value. Within the run, the first occurrence of each token wins. Tokens +already in the table are skipped. Only NOUN and VERB tags expand the +table, since an OTHER tag is already represented by absence. Consumed +files move to an archive directory. This makes the reducer idempotent, so +re-running on a drained pool is a no-op. If anything was consumed, the +artifact is rewritten atomically with sorted keys and an advanced +snapshot date. + +`PoolReduceResult` summarizes a run: consumed, quarantined, added, and +skipped counts. `PoolReducerError` covers three failure classes: table +read, table write, and pool directory unreadable. The file warns against +wiring the reducer into the per-token tagging path. Reduction is a batch +operation. A resident governor triggers it on an idle cadence, or an +operator triggers it by command. + +## Lexicon.swift + +This file provides the canonicalization lexicon type and its deterministic +builder. The lexicon is encoder Step 2's reference data. It is a flat map +from `stem(normalize(token))` to a concept identity. + +`CanonicalizationLexicon` is the pinned artifact shape: a version, a +language, and the entries map. The version is part of the agreement +protocol. Two encoders must share it, or their concept bags can diverge. + +`LexiconBuilder.build(_:)` constructs the artifact from two public +sources. Wikidata supplies the concept identities. It maps surface forms +and aliases to Q-IDs, plus the property P8814, which links WordNet synsets +to Q-IDs. WordNet supplies word senses in frequency order. The build +considers every candidate key with a three-tier rule, and the lower tier +wins. First comes a Q-ID reached through a WordNet sense of the word. This +is where WordNet disambiguates: "dog" maps to the animal, not the +sausage, since the animal is the primary sense. Next comes a Q-ID known +only from a Wikidata alias. Last comes a `wn:` synset fallback, used only +when no Q-ID exists for the concept. Within each tier, deterministic +tie-breaks apply in order. The tie-breaks are sense rank, support count, +and lowest Q-number. These make the result order-independent. Same +inputs and the same artifact yield the same output on any machine, byte +for byte. Keys are +derived with the same normalize-then-stem pipeline the runtime uses. So +build-time and runtime keys agree exactly. Only single-token surfaces +become keys, because the runtime looks up one token at a time. + +## ConceptBag.swift + +This file provides encoder Steps 1 through 3. `BagBuilder` turns a block +of text into a concept bag, the weighted map from concept identity to +occurrence count. + +`bag(_:lexicon:keep:)` makes one pass over the tokens. Each token is +normalized and stemmed, then looked up in the lexicon. The token is kept +if its word class is in the keep set, nouns and verbs by default. It is +also kept if it resolves to a Wikidata Q-ID. That second condition is a +deliberate relaxation. Part-of-speech taggers often mislabel named +entities. A proper noun like "Everest" matters more to classification +than most common nouns. Deciding membership from the pinned lexicon keeps +the relaxation deterministic. The lexicon is identical everywhere, while +cross-platform proper-noun tagging is not. A kept token contributes its +concept identity when the lexicon has one. Otherwise it contributes its +bare stemmed surface form. A surface form can still match a signature +carrying the same string. + +`bag(_:lexicon:keep:recordNovel:)` is the privacy variant. It produces a +byte-identical bag. But when `recordNovel` is `false`, novel tokens found +during tagging are not accumulated into the shared pool cache. This is the +seam the memory-capture path uses. Private text never reaches the pool +pipeline this way. `bag(_:lexicon:keep:taggerChoice:)` is the +estate-configuration variant. It threads the estate's explicit tagger +choice into Step 1. The bag is then built consistently with the estate's +other indexed content. + +## Code.swift + +This file provides the FDC code grammar: what makes a code string +well-formed. + +A code is a three-digit integer, from 000 through 999. It may carry an +optional decimal extension of one to eight digits, such as `540` or +`540.137`. Each extension digit subdivides its parent into ten. So the +extension carries leaf resolution under a spine code. The eight-digit cap +bounds the printable form at twelve characters. Tools can then reason +about column widths and string lengths. + +`Code.isWellFormed(_:)` checks the grammar and nothing else. Validity is +purely grammatical by design. A well-formed code is accepted, stored, and +round-tripped even if no term currently encodes to it. Whether a code is +"known" belongs to the caller's own known-code set, not to the grammar. +`Code.integerBase(of:)` extracts the three-digit integer base of a +well-formed code. It returns `nil` for a malformed one. + +## FDCFrame.swift + +This file provides the frame model, the versioned list of codes and +labels, plus the ancestry math over code strings. + +`FDCEntry` is one code with its heading label. The label is preserved +verbatim, including subject markers, because build tooling consumes the +raw heading text. `FDCFrame` is the versioned list. Ancestry is not stored +anywhere. It is derived from the code string itself instead. + +The derivation has two regimes. The file's long comment explains why a +naive dot-split is wrong. The three-digit integer head is a Dewey-style +positional hierarchy. It reads at the hundreds, tens, and units places. +The parent of `006` is `000`. The parent of `510` is `500`. The decimal +tail, though, is a per-segment hierarchy. The parent of `006.6` is `006`. +`decimalParent(of:)` implements both regimes as a pure function of the +string. `ancestors(of:)` walks parents upward. It returns the chain +root-first, so `ancestors("006.6")` is `["000", "006"]`. `children(of:)` +filters the frame's codes by parent identity. It sorts the result for +deterministic output. + +## CodeSignature.swift + +This file provides the build-time signature assembly. It shows how a +code's characteristic term set is put together before it ships as an +artifact. + +Every code in the frame has three source texts: its label, the title of +its reference article, and the article body. `SourceWeights` pins their +relative importance. The label gets a weight of three, the title two, the +article one. The label is the most precise statement of what a code +means. The article body is the broadest. +`SignatureAssembler.merge(label:title:article:weights:)` combines the +three concept bags into one weighted term map. + +`SignatureAssembler.accumulateAncestors(ownTerms:ancestorsOf:)` then adds +every ancestor's own terms into each code's signature. A specific code, +such as `540.1`, thereby carries the full vocabulary of its lineage. So +text that talks about the general subject still supports the specific +code during descent. The result is a `CodeSignature` per code. The +`fingerprint` field stays `nil` until a later SimHash pass computes it, +once the global concept vocabulary is known. All of this is build-time +machinery. The runtime ships only the compact term sets. + +## FDCMatcher.swift + +This file provides encoder Steps 4 and 5. It scores a concept bag against +the signatures. It descends the frame to the most specific supported +code. This file is the heart of the classifier. + +`FDCMatcher.init(lexicon:frame:signatures:stopThreshold:scoreMode:)` builds +the runtime index once. Every term across all signatures is interned. Each +is sorted alphabetically and assigned a dense integer identifier. The hot +path then works entirely with integer keys. Profiling showed this removes +the dominant cost, string hashing, on large imports. The identifier order +is deliberately the alphabetical order. So sorting integer sets visits +terms in the same sequence as sorting the original strings. Floating-point +sums are computed in the same order this way. They stay bit-identical to +the pre-interning implementation. The initializer also builds an inverted +index, from term to codes. It builds document frequencies for IDF +weighting and per-signature norms too. + +`ScoreMode` selects how overlap becomes a score. `.raw` sums the bag +counts of shared terms. `.idf` weights each shared term by its rarity +across signatures, using inverse document frequency. So distinctive +vocabulary counts for more than vocabulary that appears everywhere. +`.cosine` and `.idfCosine` also divide by a signature-size norm. This +removes the advantage of big signatures. The shipped runtime uses `.idf`. +This measurably improved code selection over `.raw` on the v1.0 frame. + +`encode(_:)` and the two `encodeAnchor` overloads are the public entries. +The `recordNovel: false` variant suppresses pool accumulation for private +text. All delegate to one private method, `encodeFromBag(_:)`, which runs +the algorithm. Step 4 collects candidate codes from the inverted index. It +scores each and takes the best. Ties break toward the lowest code, in a +fixed scan order, so the result never depends on hash ordering. Two +guards protect honesty here. An empty or non-matching bag returns `nil`, +meaning UNRESOLVED, never a guess. The other guard is +`maximumTiedWinnersForClassification`, set to four. When more than four +codes tie at the top score, the matcher also returns UNRESOLVED. A wide +tie means the text's +vocabulary is generic. Any single winner would be confidently wrong in +that case. Step 5 then walks down the frame. A child must carry at least +`stopThreshold` raw overlap to be a candidate. This cutoff is +mode-independent by design. The best-scoring child wins, and the walk +repeats until no child qualifies. The deepest reached code is the answer. + +`dominantQID(_:)` computes the other half of the anchor. This is the +highest-counted Wikidata Q-ID in the bag. Ties break toward the lowest +Q-ID. The result is `nil` when the bag has none. This value answers "what +the text is most about." It surfaces in the same pass, so consumers fill +an anchor without re-encoding. + +## FDCRuntime.swift + +This file provides `FDC`, the runtime facade that consumers actually +call. It owns the pinned artifacts. It hides the matcher behind four +static functions. + +There are three runtime artifacts: the lexicon, the frame, and the +compact signatures. They load once per process from the module bundle. +They load together, or not at all. The signatures loaded here are the +compact form, a map from code to term list, because the runtime matcher +uses only term membership. The weighted form +remains a build record. The loaded matcher is configured with +`scoreMode: .idf` and the pinned `stopThreshold` of one. Testing showed +this value is inert on the shallow v1.0 frame. It is pinned for the +contract rather than for tuning. + +`FDC.encode(_:)` returns the lattice code, or `nil` for UNRESOLVED. +`FDC.encodeAnchor(_:)` adds the dominant concept Q-ID. The +`recordNovel: false` overload is the privacy seam for memory capture. +`FDC.isAvailable` reports whether the artifacts loaded. `FDC.dataVersion` +reports the signatures version, so callers can record provenance with +every classification. `FDC.ancestors(of:)` exposes the frame's ancestry +walk. Consumers get the chain this way, without reaching into `FDCFrame` +directly. `FDC.label(for:)` returns a human-readable heading for a code. +For three-digit codes it walks up one level first. Leaf labels at that +depth are often multi-subject compounds. The parent carries the cleaner +single-topic heading that dashboards want instead. + +## QIDClosure.swift + +This file provides the Q-ID ancestor surface. It takes a Wikidata Q-ID. +It returns every ancestor reachable through two relation types. These +are instance-of, property P31, and subclass-of, property P279. + +Consumers use this to generalize a concept, from "dog" up through +"mammal" and "animal," without any network access. The edge graph is a +pinned, offline Wikidata snapshot, `Resources/QIDClosureEdges.json`. Build +tooling produces it, and it is checked in like the other artifacts. The +runtime never queries Wikidata. + +`QIDClosure.ancestors(of:)` computes the transitive closure. An iterative +breadth-first walk over the edge map does the work. It excludes the +queried Q-ID itself. It returns the result sorted numerically by Q-number. +The walk dedupes visited nodes, so even a cycle in the source data would +terminate. Results are memoized per Q-ID for the life of the process. A +lock guards the memo, while the walk itself runs outside the lock. A race +between two threads computing the same Q-ID is harmless. Both produce the +identical pure result. `isAvailable` and `dataVersion` mirror the `FDC` +facade: artifact presence and provenance. + +## LexRank.swift + +This file provides the article-reduction step used at build time. +LexRank selects the most central sentences of a long article before the +encoder runs over it. + +Code signatures are built from reference articles. But a whole article is +noisy. The useful vocabulary concentrates in its most representative +sentences. LexRank finds them with eigenvector centrality, the idea +behind PageRank, over a sentence-similarity graph. Sentences are nodes. +Two sentences connect when their stemmed term vectors are cosine-similar +above a threshold of 0.1. Sentences that many other sentences resemble +score high. + +`LexRank.reduce(_:sentences:)` segments the text. It builds per-sentence +term-frequency vectors with the shared primitives. It assembles the +thresholded similarity graph. It then asks `SubstrateML.EigenvalueCentrality` +for scores. The function returns the top ten sentences by default, joined +in their original order. It returns the text unchanged when the text is +already short enough. This is build-time-only machinery. It never runs on +user devices, so it carries no cross-platform agreement constraint, +though it stays deterministic. It is also the one place the package uses +its single dependency, SubstrateML. + +## Rust Port and Conformance + +The `rust/` directory contains the second leg of the library. Seventeen +source files mirror the Swift implementations, from `tokenizer.rs` +through `fdc_runtime.rs`. The two legs share their pinned artifacts. They +are gated by shared conformance fixtures in `rust/tests/fixtures/`. +These are recorded input-output pairs for normalization, tagging, and +full FDC encoding. Both implementations must reproduce them byte for +byte. When you change either leg, run both test suites. The fixtures are +the contract. diff --git a/packages/libs/LatticeLib/docs/OVERVIEW.md b/packages/libs/LatticeLib/docs/OVERVIEW.md new file mode 100644 index 0000000..c0ab46a --- /dev/null +++ b/packages/libs/LatticeLib/docs/OVERVIEW.md @@ -0,0 +1,181 @@ +--- +doc: OVERVIEW +package: LatticeLib +repo: moot-semantics +authored_commit: bbfef540c1b675b3fb9a493e596499b0fdf2e826 +authored_date: 2026-07-04 +sources: + - path: Sources/LatticeLib/Code.swift + blob: e2c8307da3a2821bbabd7055a8f714754289df0f + - path: Sources/LatticeLib/CodeSignature.swift + blob: b9245f8e8a8c913bcdcb27287aab45cd2729abd9 + - path: Sources/LatticeLib/ConceptBag.swift + blob: db18742c50e495682521bd7dd544b3725969b467 + - path: Sources/LatticeLib/FDCFrame.swift + blob: 8d83ab902206ffdb4d3ee9351a6443c90bcead77 + - path: Sources/LatticeLib/FDCMatcher.swift + blob: 330b99e7af1d6366425d4956652a3dd11b1a1c4b + - path: Sources/LatticeLib/FDCRuntime.swift + blob: 2fd34b7657f6888117fb9d4226ef314769d75800 + - path: Sources/LatticeLib/HMMTagger.swift + blob: 45ecd95be83f7416851a80070f85aa05aea7fc4a + - path: Sources/LatticeLib/LatticeLib.swift + blob: 88712d0a73a75d9400741a871719f39be2edc584 + - path: Sources/LatticeLib/Lexicon.swift + blob: 2a51dfbf417b542b60f5b5597e0850957c5b6631 + - path: Sources/LatticeLib/LexRank.swift + blob: 6e3cdb0c19c7a7b68992a6bd79780d15f5aa711e + - path: Sources/LatticeLib/NFKCSubset.swift + blob: fd2753d05e13b1bc430d2688cc78d0395dca58a6 + - path: Sources/LatticeLib/Normalizer.swift + blob: f989b93b3dc55751f21bcca65e8e2892bb4193e7 + - path: Sources/LatticeLib/NovelPoolSubmitter.swift + blob: be91dd71f60a81681c8102567ceeff14ad974526 + - path: Sources/LatticeLib/NovelTokenCache.swift + blob: 70342e766178eb15c14f52b28cb51d9aa11c6890 + - path: Sources/LatticeLib/NovelTokenTaggerChoice.swift + blob: 0c9b84a315ca7b31e4bf626f1830c3eea243d8aa + - path: Sources/LatticeLib/PoolReducer.swift + blob: e0ccb6a314ec39c47d31dab798841af93c4d394a + - path: Sources/LatticeLib/QIDClosure.swift + blob: 11ef3156c12ce27d6b1c1401e3b47a94c3f1bf63 + - path: Sources/LatticeLib/Stemmer.swift + blob: 042fe54236cbe61f1cb4126dc27dde97a3f5cc5d + - path: Sources/LatticeLib/Tokenizer.swift + blob: 82a8b85b5c31163d51194c7f28320305f8c798af + - path: Sources/LatticeLib/WordClass.swift + blob: 9d4851967dca01b266ad0d8f4f35cf4a757d3d7d + - path: Sources/LatticeLib/WordClassTable.swift + blob: 731f6f90c59aae71c73114ede301877d1bf8035f + - path: Sources/LatticeLib/WordClassTagger.swift + blob: 15e27da5cdeca3540654914992e8c78f83ed937a +--- + +# LatticeLib Overview + +## What This Library Does + +LatticeLib reads a piece of text. It answers one question: what subject is +this about? The answer is a lattice code. A lattice code is a structured +classification code. It gives a memory a place in an organized hierarchy of +subjects. Take the text "the dog chased a ball in the park." It encodes to +a code in the animals region of the hierarchy. + +MOOTx01 is an on-device AI memory system. It stores what an AI observes over +time. It helps the AI recall that memory later. Every memory that enters +the system passes through LatticeLib first. LatticeLib assigns each memory +a code. That code becomes part of the memory's filing address. Memories +about the same subject cluster together this way. This holds even when the +memories share no exact words. + +The engine inside LatticeLib is called FDC. FDC stands for Frame-Directed +Classification. A frame is a fixed, versioned list of classification codes +and their subject labels. FDC directs each piece of text to its best +position in that frame. + +## The Problem It Solves + +Two devices must file the same text under the same code. MOOTx01 estates +can federate. Federation lets separate devices share and compare memories. +Say one device filed "dog" under animals. Say another device filed it +under sports. Shared recall would then fall apart. Classification must be +deterministic for this reason. The same input must always produce the same +output. This must hold on every platform, every time. + +Cloud classifiers cannot make this promise. They change without notice. +They need a network connection. They see private text. LatticeLib avoids +all three problems. It ships every piece of reference data it needs as +pinned artifacts. A pinned artifact is a data file built once and given a +version. It is checked into the repository and never changed at runtime. +The library runs entirely on the device. The same text against the same +artifact versions always yields the same code. This is the library's +agreement property. + +The library keeps that promise across two independent implementations. A +Swift leg serves Apple platforms. A Rust leg, in `rust/`, serves everything +else. Shared conformance fixtures gate every release. These fixtures are +recorded input and output pairs. Both legs must reproduce them exactly. + +## How It Works + +Classification runs in five steps. The first three steps turn text into a +concept bag. A concept bag is a table. It maps each concept found in the +text to the number of times it appears. The last two steps match that bag +against the frame. + +Step 1 tags each word and keeps only nouns and verbs. Other words carry +little subject meaning. Articles and adjectives fall into this group, so +they are dropped. A word the tagger has never seen is called a novel +token. A small, deterministic Hidden Markov Model tags novel tokens. This +step therefore never guesses differently on different platforms. + +Step 2 canonicalizes each kept word. First the word is normalized. Odd +Unicode forms fold to plain letters in this step. Next the word is +stemmed, so "dogs" becomes "dog." The stemmed word is then looked up in +the canonicalization lexicon. The lexicon is a pinned artifact. It maps +about seventy thousand word roots to concept identities. A concept +identity is usually a Wikidata Q-ID. This is a public, language-neutral +identifier, such as `Q144` for "dog." Synonyms collapse onto one identity. +This is what lets two devices agree. + +Step 3 accumulates the counts. The result is the concept bag. + +Step 4 scores the bag against code signatures. A code signature is the set +of concepts that marks one classification code. It is prepared ahead of +time from reference articles. Some codes share concepts with the bag. +Those codes earn scores. Rarer shared concepts earn more. The best-scoring +code wins. Suppose nothing overlaps. Suppose too many codes tie instead. +The library then returns UNRESOLVED rather than guess. + +Step 5 descends the frame. Starting from the winning code, the matcher +walks down to child codes. It keeps walking while the text still supports +the extra detail. It returns the deepest code the text still supports. + +## How the Pieces Fit + +Figure 1 shows the library's topology. It shows the major parts and how +data moves between them. + +![Figure 1. Topology of LatticeLib](topology.svg) + +*Figure 1. Topology of LatticeLib. Text flows left to right. It moves +through the shared text primitives into the concept bag. From there it +moves through the matcher to a code. Dashed regions mark the pinned +artifacts and the offline novel-token learning loop.* + +The runtime entry point is the `FDC` enum. Consumers call +`FDC.encodeAnchor(text)`. This returns two values: the lattice code and the +dominant concept Q-ID of the text. The three runtime artifacts are the +lexicon, the frame, and the signatures. Each loads once per process. + +Shared text primitives serve both the runtime encoder and the build-time +tools. These primitives are `Tokenizer`, `Normalizer`, `Stemmer`, and the +word-class tagger. Build-time tools produce the pinned artifacts. These +tools are `LexiconBuilder`, `SignatureAssembler`, and `LexRank`. They never +run on user devices. + +One slow feedback loop improves the tagger over time. Novel tokens +accumulate in a local cache. When fifty gather, the cache writes them to a +local pool directory. A reducer later merges qualifying tokens into a +writable copy of the word-class table. The running process then adopts +the new table with an atomic swap. Classification stays deterministic +because determinism is always relative to a table version. The version +advances only at that swap. + +Privacy is built into the seams. Sometimes the text being classified is +private memory content. Callers then pass `recordNovel: false`. No token +of that text is ever written to the pool. The pool itself is off by +default. It activates only when a host sets the `LATTICE_POOL_DIR` +environment variable. + +## What Ships in the Package + +The package ships three things: the Swift sources, the Rust port in +`rust/`, and the pinned artifacts. The artifacts live in +`Sources/LatticeLib/Resources/`. They are the frame (1,075 codes), the +code signatures (1,071 entries), and the lexicon (about 70,000 entries). +They also include the Q-ID ancestor graph (about 39,000 nodes), the +trained tagger model, the word-class table, and the stemmer conformance +corpus. Each artifact carries its own version string. The library reports +the versions it used. Every classification is reproducible this way, with +its provenance included. diff --git a/packages/libs/LatticeLib/docs/topology.svg b/packages/libs/LatticeLib/docs/topology.svg new file mode 100644 index 0000000..e81bfe7 --- /dev/null +++ b/packages/libs/LatticeLib/docs/topology.svg @@ -0,0 +1,130 @@ + + + + + + + + + + + + + LatticeLib: text in, lattice code out + + + + Input text + FDC.encodeAnchor + + + Text primitives + Tokenizer · Normalizer · Stemmer + + + Word-class tagger + table fast path + HMM + + + BagBuilder + concept bag + + + FDCMatcher + score + descend + + + Lattice code + + concept Q-ID + + + + + + + + + + + Pinned artifacts (versioned, read-only, loaded once per process) + + + Lexicon + ~70k stems → Q-IDs + + + FDC frame + 1,075 codes + labels + + + Code signatures + 1,071 term sets + + + QID closure + P31/P279 graph + + + + + + + + + Novel-token learning loop (offline, opt-in via LATTICE_POOL_DIR) + + + NovelTokenCache + drains at 50 entries + + + NovelPoolSubmitter + JSON files → pool dir + + + PoolReducer + merge nouns/verbs + + + WordClassTable + live atomic swap + + + + + + + +