From 021ea704162f86fccea2f030ea9419dacc30a345 Mon Sep 17 00:00:00 2001 From: Fable Date: Sat, 4 Jul 2026 12:50:53 -0500 Subject: [PATCH 1/3] =?UTF-8?q?docs(latticelib):=20SDK=20docs=20pilot=20?= =?UTF-8?q?=E2=80=94=20OVERVIEW,=20DETAILS,=20AGENT=5FMAP,=20topology.svg?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pilot deliverables for the Moot-SDK documentation initiative. Front-matter pins authored_commit bbfef540 and per-source blob hashes for docs-scrub staleness detection. Voice: Turabian at 8th grade; AGENT_MAP is token-optimized and voice-exempt. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Pzkqg3bpbbhfuWRsYomJ3o --- packages/libs/LatticeLib/docs/AGENT_MAP.md | 163 ++++++ packages/libs/LatticeLib/docs/DETAILS.md | 575 +++++++++++++++++++++ packages/libs/LatticeLib/docs/OVERVIEW.md | 171 ++++++ packages/libs/LatticeLib/docs/topology.svg | 130 +++++ 4 files changed, 1039 insertions(+) create mode 100644 packages/libs/LatticeLib/docs/AGENT_MAP.md create mode 100644 packages/libs/LatticeLib/docs/DETAILS.md create mode 100644 packages/libs/LatticeLib/docs/OVERVIEW.md create mode 100644 packages/libs/LatticeLib/docs/topology.svg diff --git a/packages/libs/LatticeLib/docs/AGENT_MAP.md b/packages/libs/LatticeLib/docs/AGENT_MAP.md new file mode 100644 index 0000000..cd5f87c --- /dev/null +++ b/packages/libs/LatticeLib/docs/AGENT_MAP.md @@ -0,0 +1,163 @@ +--- +doc: AGENT_MAP +package: LatticeLib +repo: moot-semantics +authored_commit: bbfef540c1b675b3fb9a493e596499b0fdf2e826 +authored_date: 2026-07-04 +sources: + - path: Sources/LatticeLib/Code.swift + blob: e2c8307da3a2821bbabd7055a8f714754289df0f + - path: Sources/LatticeLib/CodeSignature.swift + blob: b9245f8e8a8c913bcdcb27287aab45cd2729abd9 + - path: Sources/LatticeLib/ConceptBag.swift + blob: db18742c50e495682521bd7dd544b3725969b467 + - path: Sources/LatticeLib/FDCFrame.swift + blob: 8d83ab902206ffdb4d3ee9351a6443c90bcead77 + - path: Sources/LatticeLib/FDCMatcher.swift + blob: 330b99e7af1d6366425d4956652a3dd11b1a1c4b + - path: Sources/LatticeLib/FDCRuntime.swift + blob: 2fd34b7657f6888117fb9d4226ef314769d75800 + - path: Sources/LatticeLib/HMMTagger.swift + blob: 45ecd95be83f7416851a80070f85aa05aea7fc4a + - path: Sources/LatticeLib/LatticeLib.swift + blob: 88712d0a73a75d9400741a871719f39be2edc584 + - path: Sources/LatticeLib/Lexicon.swift + blob: 2a51dfbf417b542b60f5b5597e0850957c5b6631 + - path: Sources/LatticeLib/LexRank.swift + blob: 6e3cdb0c19c7a7b68992a6bd79780d15f5aa711e + - path: Sources/LatticeLib/NFKCSubset.swift + blob: fd2753d05e13b1bc430d2688cc78d0395dca58a6 + - path: Sources/LatticeLib/Normalizer.swift + blob: f989b93b3dc55751f21bcca65e8e2892bb4193e7 + - path: Sources/LatticeLib/NovelPoolSubmitter.swift + blob: be91dd71f60a81681c8102567ceeff14ad974526 + - path: Sources/LatticeLib/NovelTokenCache.swift + blob: 70342e766178eb15c14f52b28cb51d9aa11c6890 + - path: Sources/LatticeLib/NovelTokenTaggerChoice.swift + blob: 0c9b84a315ca7b31e4bf626f1830c3eea243d8aa + - path: Sources/LatticeLib/PoolReducer.swift + blob: e0ccb6a314ec39c47d31dab798841af93c4d394a + - path: Sources/LatticeLib/QIDClosure.swift + blob: 11ef3156c12ce27d6b1c1401e3b47a94c3f1bf63 + - path: Sources/LatticeLib/Stemmer.swift + blob: 042fe54236cbe61f1cb4126dc27dde97a3f5cc5d + - path: Sources/LatticeLib/Tokenizer.swift + blob: 82a8b85b5c31163d51194c7f28320305f8c798af + - path: Sources/LatticeLib/WordClass.swift + blob: 9d4851967dca01b266ad0d8f4f35cf4a757d3d7d + - path: Sources/LatticeLib/WordClassTable.swift + blob: 731f6f90c59aae71c73114ede301877d1bf8035f + - path: Sources/LatticeLib/WordClassTagger.swift + blob: 15e27da5cdeca3540654914992e8c78f83ed937a +--- + +# AGENT_MAP — LatticeLib + +PURPOSE: deterministic on-device text→classification-code engine (FDC: Frame-Directed Classification) + shared text primitives. Text → concept bag → scored match against pinned code signatures → decimal-frame descent → code (or nil=UNRESOLVED). + +DEPS: imports SubstrateML (EigenvalueCentrality, build-time LexRank only), NaturalLanguage (LexRank sentence-split; optional NLTagger path), OSLog. Imported by: EideticLib, AriaMcpKit, apps/moot-mgr, tools/seed-generator. Rust port in rust/ mirrors everything; conformance fixtures rust/tests/fixtures/*.json gate byte-identity. + +ENTRY POINTS (most callers need only these): +- FDCRuntime.swift:32 `FDC.encodeAnchor(_ text:) -> (code: String?, conceptQID: String?)` — main runtime classify +- FDCRuntime.swift:45 `FDC.encodeAnchor(_:recordNovel:)` — same, recordNovel:false = privacy seam (no pool writes) +- WordClassTagger.swift:65 `LatticeLib.wordClass(_ token:) -> WordClass` — single-token noun/verb/other + +## Symbol Table + +### Runtime facade — FDCRuntime.swift +- :15 `enum FDC` — artifact-owning facade; loads lexicon+frame+signatures once/process +- :23 `FDC.stopThreshold = 1` — pinned descent cutoff (inert on shallow v1.0 frame) +- :27 `encode(_ text:) -> String?` — code only; nil = UNRESOLVED or artifacts missing +- :32/:45 `encodeAnchor(...)` — code + dominant Q-ID; see ENTRY POINTS +- :50 `isAvailable: Bool` — artifacts loaded? +- :54 `dataVersion: String` — signatures version for provenance; "0.0.0-unavailable" on load failure +- :67 `ancestors(of code:) -> [String]` — frame ancestry, root-first, excl. self +- :84 `label(for code:) -> String?` — human heading; 3-digit codes report parent's label (leaf labels are multi-subject compounds) + +### Matcher — FDCMatcher.swift +- :30 `struct FDCMatcher: Sendable` — Steps 4–5; direct init for custom artifacts/tests +- :49 `enum ScoreMode` — .raw/.idf/.cosine/.idfCosine; runtime ships .idf, direct-init default .raw +- :63 `stopThreshold: Int` — raw-overlap descent cutoff, mode-independent +- :120 `init(lexicon:frame:signatures:stopThreshold:scoreMode:)` — interns terms→dense Int ids (alphabetical order ⇒ Int sort == String sort ⇒ bit-stable float sums); builds inverted index + idf + norms +- :267 `maximumTiedWinnersForClassification = 4` — >4 tied argmax winners ⇒ UNRESOLVED (generic-vocabulary guard) +- :271 `encode(_:)`, :280 `encodeAnchor(_:)`, :296 `encodeAnchor(_:recordNovel:)` — public encode surface + +### Bag construction — ConceptBag.swift +- :25 `typealias ConceptBag = [String: Int]` — conceptID|surface → count +- :27 `enum BagBuilder` — Steps 1–3 +- :37 `bag(_:lexicon:keep:)` — default (keep = [.noun,.verb]); Q-ID lexicon hit bypasses POS filter (named-entity relaxation, deterministic via pinned lexicon) +- :82 `bag(_:lexicon:keep:recordNovel:)` — identical bag; recordNovel:false suppresses pool side effect +- :118 `bag(_:lexicon:keep:taggerChoice:)` — estate-configured tagger threading + +### Text primitives +- Tokenizer.swift:21 `Tokenizer.tokenize(_:) -> [String]` — UAX #29 words, input order +- Normalizer.swift:55 `Normalizer.normalize(_:) -> String` — NFKCSubset.fold then lowercased; fold MUST run first +- NFKCSubset.swift:40 `NFKCSubset.fold(_:)` (internal) — ASCII-target compatibility subset; table mirrored verbatim in rust/src/normalizer.rs — edit both or neither +- Stemmer.swift:69 `Stemmer.stem(_:) -> String` — Porter2/Snowball English; <3 chars unchanged +- Stemmer.swift:55 `bundledReferenceCorpus() -> Data?` — conformance corpus accessor + +### Word-class tagging — WordClassTagger.swift (extension LatticeLib) +- :65 `wordClass(_:)` — table fast path (verb set checked BEFORE noun set) → HMM fallback; NEVER NLTagger +- :102 `taggerEnabled(osVersion:minOSVersion:)` — NLTagger OS gate; unparseable minOS fails closed +- :208 `wordClass(_:recordNovel:)` — pool-suppressing variant +- :285 `wordClass(_:tagger:)` / :330 `wordClass(_:tagger:recordNovel:)` — estate-choice dispatch; .nlTagger fails closed to .other below min OS, falls back to HMM on non-Apple +- WordClass.swift:23 `enum WordClass: String` — noun|verb|other; string-backed for wire stability +- NovelTokenTaggerChoice.swift:22 `enum NovelTokenTaggerChoice` — .hmm (default, federatable) | .nlTagger (Apple-only, non-deterministic cross-platform); mirrors PersistenceKit type, bridge by switch +- HMMTagger.swift:70 `enum HMMTagger` (internal) — 3-state integer Viterbi; artifact HMMTaggerModel.json (hmm-viterbi-3, MASC 3.0.0 rare-word-trained); tie → lowest state index (noun Apple App Support > XDG; single source of truth for read+write sides +- NovelPoolSubmitter.swift:97 `tableArtifactURL()` — writable merged WordClassTable.json, SIBLING of pool dir +- PoolReducer.swift:154 `PoolReducer.reduce(poolDirectory:tableArtifactURL:now:maxFiles:)` — batch merge; seeds writable artifact from bundle; quarantines malformed/stale-version files; NOUN/VERB only; first-occurrence dedup; archives consumed files (idempotent); DO NOT call from hot path +- PoolReducer.swift:65 `PoolReduceResult` / :105 `PoolReducerError` — result summary / failure classes + +### Frame + codes +- Code.swift:34 `Code.isWellFormed(_:)` — grammar only: 3 digits + optional ≤8-digit decimal ext (:31 maxExtensionDigits=8); known-vs-pending is caller's concern +- Code.swift:58 `Code.integerBase(of:)` — "540.137" → 540 +- FDCFrame.swift:17 `FDCEntry` — code + verbatim label +- FDCFrame.swift:34 `FDCFrame` — versioned code list; ancestry DERIVED, not stored +- FDCFrame.swift:142 `ancestors(of:)` — TWO regimes: integer head is Dewey positional (parent("006")="000"), decimal tail is per-segment (parent("006.6")="006"); NOT a dot-split +- FDCFrame.swift:131 `children(of:)` — immediate children, sorted + +### Build-time only (never on-device) +- Lexicon.swift:30 `CanonicalizationLexicon` — version/language/entries (stem→conceptID); version is part of agreement protocol +- Lexicon.swift:91 `LexiconBuilder.build(_:)` — Wikidata-primary, WordNet-disambiguated; 3-tier deterministic conflict resolution; single-token keys only +- CodeSignature.swift:36 `SignatureAssembler.merge(label:title:article:weights:)` — source weights label 3 / title 2 / article 1 (:13 SourceWeights) +- CodeSignature.swift:52 `accumulateAncestors(ownTerms:ancestorsOf:)` — signature += all ancestors' own terms +- CodeSignature.swift:24 `CodeSignature` — terms + fingerprint (nil until SimHash pass) +- LexRank.swift:25 `LexRank.reduce(_:sentences:)` — top-N central sentences (default :20 =10, cosine threshold :21 =0.1) via SubstrateML.EigenvalueCentrality; deterministic but NOT cross-platform-gated + +### Q-ID closure — QIDClosure.swift +- :45 `QIDClosure.ancestors(of qid:) -> [String]` — transitive P31/P279 closure over pinned QIDClosureEdges.json; BFS, excl. self, sorted by Q-number; memoized per process +- :68 `isAvailable` / :73 `dataVersion` — artifact presence / provenance + +### Module — LatticeLib.swift +- :13 `enum LatticeLib` — namespace; :17 `version = "1.0.0"` bumped with artifact ships + +## INVARIANTS / GOTCHAS + +- DETERMINISM IS THE CONTRACT. encode() is pure over (text, artifact versions). Any change to Tokenizer/Normalizer/NFKCSubset/Stemmer/HMMTagger/FDCMatcher scoring must be mirrored in rust/src/ and pass rust/tests/fixtures/{normalize,tag,fdc}_conformance.json + Swift conformance tests. Same for artifact regeneration. +- nil code = UNRESOLVED, by design: empty bag, no signature overlap, or >4 tied winners. Never "fix" by guessing. +- Verb set checked before noun set in every wordClass overload — a token in both is a verb. Do not reorder. +- HMM (not NLTagger) is the default novel-token tagger on ALL platforms incl. Apple. NLTagger only via explicit .nlTagger estate choice; it breaks cross-platform determinism and federation. +- NFKCSubset table + Rust normalizer.rs are ONE table in two files. Never edit one alone. +- FDCMatcher term interning: IDs assigned in ascending String order — required so sorted-Int iteration == sorted-String iteration ⇒ bit-identical float sums. Do not change assignment order. +- FDCFrame ancestry: integer head ≠ decimal tail regimes. parent("006") is "000" (Dewey positional), not "00". A dot-split reimplementation is wrong. +- stopThreshold compares RAW integer overlap, never the (possibly normalized) score. +- Pinned constants — do not change without a new artifact version + conformance regen: poolSubmitThreshold 50, stopThreshold 1, maxExtensionDigits 8, maximumTiedWinnersForClassification 4, SourceWeights 3/2/1, HMM logScale 1000, LexRank 10/0.1. +- Privacy seams: recordNovel:false variants produce byte-identical results with pool accumulation suppressed — required for user memory content (GLK capture, EideticLib.lookup). sharedNovelCache writes files ONLY when LATTICE_POOL_DIR is set; default submitter is a no-op. +- Bundled WordClassTable.json is a small seed (21 nouns / 18 verbs, v1.0.0): in practice most tokens are novel and route through the HMM until the pool loop grows the writable table. +- Artifacts load once per process (static let); FDC/QIDClosure fail soft (isAvailable=false, encode→nil) — check isAvailable in hosts with unusual bundle setups. +- PoolReducer is batch-only (governor idle-tick or operator CLI). Bounded by maxFiles per run; backlog drains across runs. +- WordClassTableCache.swap is the ONLY legal live-update path for tagging behavior; readers get whole-snapshot consistency, and version() observability keys determinism. +- LexRank is build-time-only; it may use floats and NLTokenizer freely. Do not move it into the runtime encode path. diff --git a/packages/libs/LatticeLib/docs/DETAILS.md b/packages/libs/LatticeLib/docs/DETAILS.md new file mode 100644 index 0000000..3d4c2e5 --- /dev/null +++ b/packages/libs/LatticeLib/docs/DETAILS.md @@ -0,0 +1,575 @@ +--- +doc: DETAILS +package: LatticeLib +repo: moot-semantics +authored_commit: bbfef540c1b675b3fb9a493e596499b0fdf2e826 +authored_date: 2026-07-04 +sources: + - path: Sources/LatticeLib/Code.swift + blob: e2c8307da3a2821bbabd7055a8f714754289df0f + - path: Sources/LatticeLib/CodeSignature.swift + blob: b9245f8e8a8c913bcdcb27287aab45cd2729abd9 + - path: Sources/LatticeLib/ConceptBag.swift + blob: db18742c50e495682521bd7dd544b3725969b467 + - path: Sources/LatticeLib/FDCFrame.swift + blob: 8d83ab902206ffdb4d3ee9351a6443c90bcead77 + - path: Sources/LatticeLib/FDCMatcher.swift + blob: 330b99e7af1d6366425d4956652a3dd11b1a1c4b + - path: Sources/LatticeLib/FDCRuntime.swift + blob: 2fd34b7657f6888117fb9d4226ef314769d75800 + - path: Sources/LatticeLib/HMMTagger.swift + blob: 45ecd95be83f7416851a80070f85aa05aea7fc4a + - path: Sources/LatticeLib/LatticeLib.swift + blob: 88712d0a73a75d9400741a871719f39be2edc584 + - path: Sources/LatticeLib/Lexicon.swift + blob: 2a51dfbf417b542b60f5b5597e0850957c5b6631 + - path: Sources/LatticeLib/LexRank.swift + blob: 6e3cdb0c19c7a7b68992a6bd79780d15f5aa711e + - path: Sources/LatticeLib/NFKCSubset.swift + blob: fd2753d05e13b1bc430d2688cc78d0395dca58a6 + - path: Sources/LatticeLib/Normalizer.swift + blob: f989b93b3dc55751f21bcca65e8e2892bb4193e7 + - path: Sources/LatticeLib/NovelPoolSubmitter.swift + blob: be91dd71f60a81681c8102567ceeff14ad974526 + - path: Sources/LatticeLib/NovelTokenCache.swift + blob: 70342e766178eb15c14f52b28cb51d9aa11c6890 + - path: Sources/LatticeLib/NovelTokenTaggerChoice.swift + blob: 0c9b84a315ca7b31e4bf626f1830c3eea243d8aa + - path: Sources/LatticeLib/PoolReducer.swift + blob: e0ccb6a314ec39c47d31dab798841af93c4d394a + - path: Sources/LatticeLib/QIDClosure.swift + blob: 11ef3156c12ce27d6b1c1401e3b47a94c3f1bf63 + - path: Sources/LatticeLib/Stemmer.swift + blob: 042fe54236cbe61f1cb4126dc27dde97a3f5cc5d + - path: Sources/LatticeLib/Tokenizer.swift + blob: 82a8b85b5c31163d51194c7f28320305f8c798af + - path: Sources/LatticeLib/WordClass.swift + blob: 9d4851967dca01b266ad0d8f4f35cf4a757d3d7d + - path: Sources/LatticeLib/WordClassTable.swift + blob: 731f6f90c59aae71c73114ede301877d1bf8035f + - path: Sources/LatticeLib/WordClassTagger.swift + blob: 15e27da5cdeca3540654914992e8c78f83ed937a +--- + +# LatticeLib Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in pipeline order: +the module surface, the text primitives, the word-class machinery, the +learning loop, and finally the classification engine and its helpers. + +## LatticeLib.swift + +This file provides the module surface: the public `LatticeLib` enum and the +module version. + +Swift has no module-level functions, so libraries often use an empty enum as +a namespace. The `LatticeLib` enum is that namespace. Other files extend it; +the word-class tagging functions in `WordClassTagger.swift` all live in an +extension of this enum. + +`LatticeLib.version` is the module version string. It exists so that +consumers can record which library version produced a result. The version is +bumped in lockstep with the bundled FDC artifacts when new signatures ship, +because a result is only reproducible against the exact artifact set that +produced it. + +## Tokenizer.swift + +This file provides `Tokenizer`, which splits raw text into words. + +Splitting text into words sounds simple but is full of edge cases: +apostrophes, hyphens, non-Latin scripts. Rather than invent rules, the +tokenizer follows Unicode Standard Annex #29, the industry-standard +word-boundary specification. Following a public standard is what allows the +Swift leg and the Rust leg to agree byte for byte: both implement the same +published rules, and a shared conformance fixture proves it. + +`Tokenizer.tokenize(_:)` returns the words of a string in input order. +Whitespace and punctuation are dropped. It works by asking Foundation to +enumerate word substrings, which routes to ICU's word-boundary analyzer. The +Rust port uses the `unicode-segmentation` crate, which implements the same +annex. + +## NFKCSubset.swift + +This file provides the internal compatibility-fold table used by the +normalizer. It maps visually equivalent Unicode characters to plain ASCII: a +full-width `Ａ` becomes `A`, the ligature `fi` becomes `fi`, a superscript `²` +becomes `2`. + +Why not use the platform's full Unicode NFKC normalization? Because the Rust +leg cannot reproduce it byte for byte without a large external dependency, +and the two legs must never disagree. Instead, both legs implement the same +explicit table, defined once here and mirrored verbatim in +`rust/src/normalizer.rs`. Same table plus same algorithm equals identical +output by construction. Every mapping is a strict subset of official NFKC, +limited to mappings whose target is plain ASCII, because those are the ones +that improve matching for an ASCII-English corpus. + +`fold(_:)` rewrites a string scalar by scalar, replacing each mapped +character and passing everything else through unchanged. `foldScalar(_:)` +holds the actual mapping: an explicit literal table first, then arithmetic +range families (full-width forms, super/subscripts, circled letters, Roman +numerals) expressed as offsets so the table stays small and auditable. +Canonical recomposition (for example, half-width katakana) is deliberately +out of scope; the file documents this so a future reader does not mistake the +omission for a bug. + +## Normalizer.swift + +This file provides `Normalizer`, the canonicalization front door. It runs +before stemming and lexicon lookup so that different surface spellings of the +same word collapse to one key. + +`Normalizer.normalize(_:)` applies the `NFKCSubset` fold and then a +Unicode-aware lowercase. The fold runs first for a reason the code spells +out: a full-width `Ａ` must first become `A` and then fold to `a`, matching +the key that a plain `a` produces. Lowercasing alone would leave the +full-width form distinct, and the same word would split into two different +lexicon keys. The function is deterministic and byte-identical to the Rust +port for every input in the shared conformance fixture. + +## Stemmer.swift + +This file provides `Stemmer`, a hand-ported implementation of the Porter2 +(Snowball English) stemming algorithm. Stemming reduces a word to its root: +"dogs" becomes "dog," "classification" becomes "classif." The point is +recall: text about "running" and text about "runs" should reach the same +lexicon entry. + +Porter2 removes English suffixes in five ordered steps, guided by two regions +of the word (R1 and R2) computed from vowel-consonant transitions. Most +removals only apply when the suffix sits inside a region, which prevents +overstripping short words. Two exception lists handle irregular forms, such +as "skies" becoming "sky" rather than "ski." + +`Stemmer.stem(_:)` runs the full algorithm: preprocessing (apostrophe and +consonant-y marking), region computation, then steps 0 through 5. Words +shorter than three letters return unchanged, per the specification. +`Stemmer.bundledReferenceCorpus()` exposes the pinned test corpus +(`Resources/SnowballEnglish.json`) so the conformance test can verify that +this port and the Rust port (which uses the `rust-stemmers` crate) produce +byte-identical stems for every input. + +## WordClass.swift + +This file provides `WordClass`, the label produced by tagging Step 1 of the +encoder: `.noun`, `.verb`, or `.other`. + +Step 1 keeps only nouns and verbs because they carry the subject matter of a +sentence. `.other` is the discard bucket for everything else — articles, +prepositions, adjectives, punctuation. The enum is backed by strings so it +serializes to a stable, human-readable JSON form (`"noun"`, `"verb"`, +`"other"`) that the shared conformance vectors and the Rust port both read. + +## WordClassTable.swift + +This file provides the noun/verb fast-path table and its live-swappable +process cache. The table is the first stop for every token: a token present +in it resolves to its word class in constant time, with no tagger invoked. + +Three load functions cover the table's life cycle. `loadBundled()` reads the +pinned snapshot shipped in `Resources/WordClassTable.json`. +`loadWritable()` reads the merged artifact that `PoolReducer` maintains +outside the app bundle. `loadWithPrecedence()` is the canonical entry: it +prefers the writable merged artifact, because that one contains learned +tokens, and falls back to the bundled pristine table. + +`WordClassTableCache` solves a harder problem: adopting a new table while the +process is running. After the reducer merges new tokens, waiting for a +process restart would delay learning indefinitely. The cache instead +publishes an immutable snapshot (`WordClassTableSnapshot`, the parsed table +plus precomputed membership sets) behind a lock. Readers copy the snapshot +reference out and test membership outside the lock, so a swap can never +produce a torn read: a reader sees the whole old table or the whole new one, +never a mix. `swap(_:)` publishes a new snapshot and bumps a version counter. +`reloadFromPrecedence()` and `reload(fromArtifact:)` re-resolve the table +from disk and publish it. The version counter matters for determinism: +tagging is deterministic given (input, table version), and the counter makes +the version observable. + +## WordClassTagger.swift + +This file provides the public tagging API, `LatticeLib.wordClass`, in its +four overloads. This is encoder Step 1 as callers see it. + +Every overload shares the same fast path. The token is lowercased, then +checked against the live table snapshot: the verb set first, then the noun +set, so a token listed in both resolves to verb. Only a token absent from the +table — a novel token — reaches a tagger. + +`wordClass(_:)` is the default path. Novel tokens go to the deterministic +HMM tagger on all platforms, including Apple. This is the rule that keeps +the Swift and Rust ports bit-identical: Apple's NLTagger is a black box that +can change between OS releases, so it is never in the default path. + +`wordClass(_:tagger:)` threads an explicit estate choice. An estate is one +user's memory store; each estate picks its novel-token tagger once, at +creation. The `.hmm` choice is the cross-platform baseline. The `.nlTagger` +choice opts in to Apple's NaturalLanguage tagger for higher accuracy at the +cost of cross-platform determinism, and it is gated by +`taggerEnabled(osVersion:minOSVersion:)`: the table pins the NLTagger OS +version that seeded it, and an older OS fails closed to `.other` rather than +invoke a differently-behaving tagger. + +`wordClass(_:recordNovel:)` and `wordClass(_:tagger:recordNovel:)` add +privacy control. By default a novel token's tag is recorded into the shared +pool cache so the table can learn it later. When the text being classified is +private memory content, callers pass `recordNovel: false` and the side effect +is suppressed. The returned tag is identical either way; only the recording +changes. The shared cache itself (`sharedNovelCache`) is wired to a real file +writer only when the `LATTICE_POOL_DIR` environment variable is set; +otherwise the submitter is a no-op, so no deployment leaks tokens to disk by +default. + +## HMMTagger.swift + +This file provides the deterministic novel-token tagger: a three-state Hidden +Markov Model (noun, verb, other) decoded with integer Viterbi. + +Why a statistical model here at all? Because novel tokens are exactly the +words no table covers, so some generalization is required. And why integer +arithmetic? Because floating-point rounding can differ across platforms and +compilers, and this tagger must produce the same answer on every port. The +model weights are log-likelihoods scaled by 1,000 and rounded to integers at +build time; runtime scoring is pure integer addition and comparison. + +The observation alphabet is morphological: each token maps to one observation +such as "ends in -ing," "ends in -tion," "contains a non-letter," or "plain." +`observe(_:)` checks these in a fixed priority order that both ports +replicate exactly. `tag(_:)` then picks the state with the best initial plus +emission weight; for a one-token sequence, Viterbi reduces to that argmax, +with ties resolved to the lowest state index. + +The weights ship in the frozen artifact `Resources/HMMTaggerModel.json`, +trained on the MASC 3.0.0 Penn Treebank annotation. The training deliberately +uses only the corpus's rare words (frequency 1). The tagger only ever sees +out-of-vocabulary words, and rare words are the best statistical proxy for +those, which gives the correct noun-leaning prior for unknown words. +Estimating from the full corpus would let frequent function words dominate +and wrongly default unknowns to `.other`. + +## NovelTokenTaggerChoice.swift + +This file provides `NovelTokenTaggerChoice`, the two-case enum (`.hmm`, +`.nlTagger`) that selects the novel-token tagger for an estate. + +The type mirrors an identically named enum in PersistenceKit, where the +choice is stored on the estate configuration. The two packages define it +independently so that neither depends on the other; a consumer bridges +between them with a trivial switch. The file documents the threading rule: +pass the choice down as a parameter from the estate configuration to the +call sites, never through global state. `.hmm` is the default and the +federation-safe choice; `.nlTagger` is Apple-only and trades determinism for +accuracy. + +## NovelTokenCache.swift + +This file provides the local accumulation cache for novel-token tags, plus +the two wire-format types `PoolEntry` and `PoolSubmission`. + +When the tagger classifies a novel token, the result is worth keeping: if +many devices see the same unknown word, the shared table should learn it. +`NovelTokenCache.record(token:wordClass:)` appends the tagged token to a +pending list under a lock. At exactly `poolSubmitThreshold` (50) entries, the +cache builds a `PoolSubmission` — stamped with the table version, platform, +and tagger version — drains the list, and hands the payload to an injected +submitter closure outside the lock. Entries below the threshold are kept +indefinitely; there is no aging or cleanup, by contract. + +The submitter is injected at construction so tests can assert the drain +without touching the file system, and so hosts that never configure a pool +get a no-op. The version stamps exist because the pool consumer discards +submissions made against a stale table version; an observation is only +meaningful relative to the table that failed to contain it. + +## NovelPoolSubmitter.swift + +This file provides the production submitter factory: it turns a drained +`PoolSubmission` into a JSON file in a local pool directory. + +The pool is a directory of files rather than a database or a network call. +Files are cheap, observable, and easy to reason about; a future reducer can +process them in name order because each file name embeds an ISO 8601 +timestamp plus a short unique suffix. Writes are atomic and fire-and-forget: +a failed write is logged and the data is discarded for that cycle, never +retried and never crashed on, because pool data is recoverable from future +observations. + +`make(poolDirectory:)` returns a submitter closure bound to a directory. +`makeDefault()` binds to the resolved default. `poolDirectory()` resolves +that default: the `LATTICE_POOL_DIR` environment variable when set, otherwise +an Application Support path on Apple platforms or an XDG data path elsewhere. +It is public because the write side (submitter) and the read side (the +governor that triggers the reducer) must agree on the path, so it lives here +as the single source of truth. `tableArtifactURL()` resolves the sibling +location of the writable merged table — `WordClassTable.json` next to the +pool directory — which is the file the reducer updates and +`WordClassTable.loadWritable()` reads. + +## PoolReducer.swift + +This file provides the batch job that closes the learning loop: it reads +pooled submissions and merges qualifying novel tokens into the writable +word-class table artifact. + +`PoolReducer.reduce(poolDirectory:tableArtifactURL:now:maxFiles:)` runs in +steps. It first seeds the writable artifact from the bundled table if none +exists, because the reducer cannot write into the read-only app bundle. It +then loads the table, enumerates `pool_*.json` files, and processes the +oldest `maxFiles` of them; bounding the batch keeps a large backlog from +stalling the caller, and the remainder drains on later runs. Each file is +decoded and version-checked: a malformed or stale-version file is moved to a +quarantine directory rather than deleted, preserving forensic value. Within +the run, the first occurrence of each token wins, tokens already in the table +are skipped, and only NOUN and VERB tags expand the table — an OTHER tag is +already represented by absence. Consumed files move to an archive directory, +which makes the reducer idempotent: re-running on a drained pool is a no-op. +If anything was consumed, the artifact is rewritten atomically with sorted +keys and an advanced snapshot date. + +`PoolReduceResult` summarizes a run (consumed, quarantined, added, skipped), +and `PoolReducerError` covers the three failure classes (table read, table +write, pool directory unreadable). The file warns against wiring the reducer +into the per-token tagging path: reduction is a batch operation, triggered by +a resident governor on an idle cadence or by an operator command. + +## Lexicon.swift + +This file provides the canonicalization lexicon type and its deterministic +builder. The lexicon is encoder Step 2's reference data: a flat map from +`stem(normalize(token))` to a concept identity. + +`CanonicalizationLexicon` is the pinned artifact shape: a version, a +language, and the entries map. The version is part of the agreement protocol: +two encoders must share it, or their concept bags can diverge. + +`LexiconBuilder.build(_:)` constructs the artifact from two public sources. +Wikidata supplies the concept identities: surface forms and aliases mapped to +Q-IDs, plus the property (P8814) that links WordNet synsets to Q-IDs. +WordNet supplies word senses in frequency order. The build considers every +candidate key with a three-tier rule, lower tier winning: first a Q-ID +reached through a WordNet sense of the word (this is where WordNet +disambiguates — "dog" maps to the animal, not the sausage, because the animal +is the primary sense); then a Q-ID known only from a Wikidata alias; and +last a `wn:` synset fallback, used only when no Q-ID exists for the concept. +Within each tier, explicit deterministic tie-breaks (sense rank, support +count, lowest Q-number) make the result order-independent: same inputs, same +artifact, byte for byte, on any machine. Keys are derived with the same +normalize-then-stem pipeline the runtime uses, so build-time and runtime keys +agree exactly. Only single-token surfaces become keys, because the runtime +looks up one token at a time. + +## ConceptBag.swift + +This file provides encoder Steps 1 through 3: `BagBuilder` turns a block of +text into a concept bag, the weighted map from concept identity to +occurrence count. + +`bag(_:lexicon:keep:)` makes one pass over the tokens. Each token is +normalized and stemmed, then looked up in the lexicon. The token is kept if +its word class is in the keep set (nouns and verbs by default) — or if it +resolves to a Wikidata Q-ID. That second condition is a deliberate +relaxation: part-of-speech taggers often mislabel named entities, and a +proper noun like "Everest" matters more to classification than most common +nouns. Deciding membership from the pinned lexicon keeps the relaxation +deterministic, because the lexicon is identical everywhere, while +cross-platform proper-noun tagging is not. A kept token contributes its +concept identity when the lexicon has one, or its bare stemmed surface form +otherwise; a surface form can still match a signature carrying the same +string. + +`bag(_:lexicon:keep:recordNovel:)` is the privacy variant. It produces a +byte-identical bag, but when `recordNovel` is `false`, novel tokens +encountered during tagging are not accumulated into the shared pool cache. +This is the seam the memory-capture path uses so that private text never +reaches the pool pipeline. `bag(_:lexicon:keep:taggerChoice:)` is the +estate-configuration variant: it threads the estate's explicit tagger choice +into Step 1 so the bag is built consistently with the estate's other +indexed content. + +## Code.swift + +This file provides the FDC code grammar: what makes a code string +well-formed. + +A code is a three-digit integer (000 through 999) with an optional decimal +extension of one to eight digits, such as `540` or `540.137`. Each extension +digit subdivides its parent into ten, so the extension carries leaf +resolution under a spine code. The eight-digit cap bounds the printable form +at twelve characters so tools can reason about column widths and string +lengths. + +`Code.isWellFormed(_:)` checks the grammar and nothing else. Validity is +purely grammatical by design: a well-formed code is accepted, stored, and +round-tripped even if no term currently encodes to it, because whether a code +is "known" belongs to the caller's own known-code set, not to the grammar. +`Code.integerBase(of:)` extracts the three-digit integer base of a +well-formed code, or `nil` for a malformed one. + +## FDCFrame.swift + +This file provides the frame model — the versioned list of codes and labels — +and the ancestry math over code strings. + +`FDCEntry` is one code with its heading label, preserved verbatim including +subject markers, because build tooling consumes the raw heading text. +`FDCFrame` is the versioned list. Ancestry is not stored anywhere; it is +derived from the code string itself. + +The derivation has two regimes, and the file's long comment explains why a +naive dot-split is wrong. The three-digit integer head is a Dewey-style +positional hierarchy read at the hundreds, tens, and units places: the parent +of `006` is `000`, and the parent of `510` is `500`. The decimal tail is a +per-segment hierarchy: the parent of `006.6` is `006`. `decimalParent(of:)` +implements both regimes as a pure function of the string. `ancestors(of:)` +walks parents upward and returns the chain root-first, so +`ancestors("006.6")` is `["000", "006"]`. `children(of:)` filters the frame's +codes by parent identity and sorts the result for deterministic output. + +## CodeSignature.swift + +This file provides the build-time signature assembly: how a code's +characteristic term set is put together before it ships as an artifact. + +Every code in the frame has three source texts: its label, the title of its +reference article, and the article body. `SourceWeights` pins their relative +importance — label 3, title 2, article 1 — because the label is the most +precise statement of what a code means and the article body the broadest. +`SignatureAssembler.merge(label:title:article:weights:)` combines the three +concept bags into one weighted term map. + +`SignatureAssembler.accumulateAncestors(ownTerms:ancestorsOf:)` then adds +every ancestor's own terms into each code's signature. A specific code such +as `540.1` thereby carries the full vocabulary of its lineage, so text that +talks about the general subject still supports the specific code during +descent. The result is a `CodeSignature` per code; the `fingerprint` field +stays `nil` until a later SimHash pass computes it, once the global concept +vocabulary is known. All of this is build-time machinery: the runtime ships +only the compact term sets. + +## FDCMatcher.swift + +This file provides encoder Steps 4 and 5: scoring a concept bag against the +signatures and descending the frame to the most specific supported code. It +is the heart of the classifier. + +`FDCMatcher.init(lexicon:frame:signatures:stopThreshold:scoreMode:)` builds +the runtime index once. Every term across all signatures is interned: sorted +alphabetically and assigned a dense integer identifier. The hot path then +works entirely with integer keys, which profiling showed removes the dominant +cost (string hashing) on large imports. The identifier order is deliberately +the alphabetical order, so sorting integer sets visits terms in the same +sequence as sorting the original strings — floating-point sums are computed +in the same order and stay bit-identical to the pre-interning implementation. +The initializer also builds an inverted index (term to codes), document +frequencies for IDF weighting, and per-signature norms. + +`ScoreMode` selects how overlap becomes a score. `.raw` sums the bag counts +of shared terms. `.idf` weights each shared term by its rarity across +signatures (inverse document frequency), so distinctive vocabulary counts for +more than vocabulary that appears everywhere. `.cosine` and `.idfCosine` +additionally divide by a signature-size norm to remove the advantage of big +signatures. The shipped runtime uses `.idf`, which measurably improved code +selection over `.raw` on the v1.0 frame. + +`encode(_:)` and the two `encodeAnchor` overloads are the public entries; the +`recordNovel: false` variant suppresses pool accumulation for private text. +All delegate to one private method, `encodeFromBag(_:)`, which runs the +algorithm. Step 4 collects candidate codes from the inverted index, scores +each, and takes the best, breaking ties toward the lowest code in a fixed +scan order so the result never depends on hash ordering. Two guards protect +honesty: an empty or non-matching bag returns `nil` (UNRESOLVED, never a +guess), and when more than `maximumTiedWinnersForClassification` (4) codes +tie at the top score, the matcher also returns UNRESOLVED, because a wide tie +means the text's vocabulary is generic and any single winner would be +confidently wrong. Step 5 then walks down the frame: a child must carry at +least `stopThreshold` raw overlap to be a candidate (the cutoff is +mode-independent by design), the best-scoring child wins, and the walk +repeats until no child qualifies. The deepest reached code is the answer. + +`dominantQID(_:)` computes the other half of the anchor: the +highest-counted Wikidata Q-ID in the bag, ties broken toward the lowest +Q-ID, or `nil` when the bag has none. This is "what the text is most about," +surfaced in the same pass so consumers fill an anchor without re-encoding. + +## FDCRuntime.swift + +This file provides `FDC`, the runtime facade that consumers actually call. +It owns the pinned artifacts and hides the matcher behind four static +functions. + +The three runtime artifacts — lexicon, frame, and compact signatures — load +once per process from the module bundle, together, or not at all. The +signatures loaded here are the compact form (code to term list) because the +runtime matcher uses only term membership; the weighted form remains a build +record. The loaded matcher is configured with `scoreMode: .idf` and the +pinned `stopThreshold` of 1 — a value that testing showed is inert on the +shallow v1.0 frame, pinned for the contract rather than for tuning. + +`FDC.encode(_:)` returns the lattice code or `nil` for UNRESOLVED. +`FDC.encodeAnchor(_:)` adds the dominant concept Q-ID; the +`recordNovel: false` overload is the privacy seam for memory capture. +`FDC.isAvailable` reports whether the artifacts loaded, and +`FDC.dataVersion` reports the signatures version so callers can record +provenance with every classification. `FDC.ancestors(of:)` exposes the +frame's ancestry walk so consumers get the chain without reaching into +`FDCFrame` directly. `FDC.label(for:)` returns a human-readable heading for +a code; for three-digit codes it walks up one level first, because leaf +labels at that depth are often multi-subject compounds while the parent +carries the cleaner single-topic heading that dashboards want. + +## QIDClosure.swift + +This file provides the Q-ID ancestor surface: given a Wikidata Q-ID, it +returns every ancestor reachable through instance-of (P31) and subclass-of +(P279) relations. + +Consumers use this to generalize a concept — from "dog" up through "mammal" +and "animal" — without any network access. The edge graph is a pinned, +offline Wikidata snapshot (`Resources/QIDClosureEdges.json`), produced by +build tooling and checked in like the other artifacts. The runtime never +queries Wikidata. + +`QIDClosure.ancestors(of:)` computes the transitive closure with an +iterative breadth-first walk over the edge map, excluding the queried Q-ID +itself, and returns the result sorted numerically by Q-number. The walk +dedupes visited nodes, so even a cycle in the source data would terminate. +Results are memoized per Q-ID for the life of the process; the memo is +guarded by a lock, while the walk itself runs outside the lock, and a race +between two threads computing the same Q-ID is harmless because both produce +the identical pure result. `isAvailable` and `dataVersion` mirror the +`FDC` facade: artifact-presence and provenance. + +## LexRank.swift + +This file provides the article-reduction step used at build time: LexRank, +which selects the most central sentences of a long article before the +encoder runs over it. + +Code signatures are built from reference articles, but a whole article is +noisy; the useful vocabulary is concentrated in its most representative +sentences. LexRank finds them with eigenvector centrality — the idea behind +PageRank — over a sentence-similarity graph: sentences are nodes, and two +sentences connect when their stemmed term vectors are cosine-similar above a +threshold (0.1). Sentences that many other sentences resemble score high. + +`LexRank.reduce(_:sentences:)` segments the text, builds per-sentence +term-frequency vectors with the shared primitives, assembles the thresholded +similarity graph, and asks `SubstrateML.EigenvalueCentrality` for scores. It +returns the top N sentences (default 10) joined in their original order, or +the text unchanged when it is already short enough. This is build-time-only +machinery: it never runs on user devices, so it has no cross-platform +agreement constraint, though it is still deterministic. It is also the one +place the package uses its single dependency, SubstrateML. + +## Rust Port and Conformance + +The `rust/` directory contains the second leg of the library: seventeen +source files mirroring the Swift implementations, from `tokenizer.rs` through +`fdc_runtime.rs`. The two legs share their pinned artifacts and are gated by +shared conformance fixtures in `rust/tests/fixtures/` — recorded +input-output pairs for normalization, tagging, and full FDC encoding that +both implementations must reproduce byte for byte. When you change either +leg, run both test suites; the fixtures are the contract. diff --git a/packages/libs/LatticeLib/docs/OVERVIEW.md b/packages/libs/LatticeLib/docs/OVERVIEW.md new file mode 100644 index 0000000..bd14625 --- /dev/null +++ b/packages/libs/LatticeLib/docs/OVERVIEW.md @@ -0,0 +1,171 @@ +--- +doc: OVERVIEW +package: LatticeLib +repo: moot-semantics +authored_commit: bbfef540c1b675b3fb9a493e596499b0fdf2e826 +authored_date: 2026-07-04 +sources: + - path: Sources/LatticeLib/Code.swift + blob: e2c8307da3a2821bbabd7055a8f714754289df0f + - path: Sources/LatticeLib/CodeSignature.swift + blob: b9245f8e8a8c913bcdcb27287aab45cd2729abd9 + - path: Sources/LatticeLib/ConceptBag.swift + blob: db18742c50e495682521bd7dd544b3725969b467 + - path: Sources/LatticeLib/FDCFrame.swift + blob: 8d83ab902206ffdb4d3ee9351a6443c90bcead77 + - path: Sources/LatticeLib/FDCMatcher.swift + blob: 330b99e7af1d6366425d4956652a3dd11b1a1c4b + - path: Sources/LatticeLib/FDCRuntime.swift + blob: 2fd34b7657f6888117fb9d4226ef314769d75800 + - path: Sources/LatticeLib/HMMTagger.swift + blob: 45ecd95be83f7416851a80070f85aa05aea7fc4a + - path: Sources/LatticeLib/LatticeLib.swift + blob: 88712d0a73a75d9400741a871719f39be2edc584 + - path: Sources/LatticeLib/Lexicon.swift + blob: 2a51dfbf417b542b60f5b5597e0850957c5b6631 + - path: Sources/LatticeLib/LexRank.swift + blob: 6e3cdb0c19c7a7b68992a6bd79780d15f5aa711e + - path: Sources/LatticeLib/NFKCSubset.swift + blob: fd2753d05e13b1bc430d2688cc78d0395dca58a6 + - path: Sources/LatticeLib/Normalizer.swift + blob: f989b93b3dc55751f21bcca65e8e2892bb4193e7 + - path: Sources/LatticeLib/NovelPoolSubmitter.swift + blob: be91dd71f60a81681c8102567ceeff14ad974526 + - path: Sources/LatticeLib/NovelTokenCache.swift + blob: 70342e766178eb15c14f52b28cb51d9aa11c6890 + - path: Sources/LatticeLib/NovelTokenTaggerChoice.swift + blob: 0c9b84a315ca7b31e4bf626f1830c3eea243d8aa + - path: Sources/LatticeLib/PoolReducer.swift + blob: e0ccb6a314ec39c47d31dab798841af93c4d394a + - path: Sources/LatticeLib/QIDClosure.swift + blob: 11ef3156c12ce27d6b1c1401e3b47a94c3f1bf63 + - path: Sources/LatticeLib/Stemmer.swift + blob: 042fe54236cbe61f1cb4126dc27dde97a3f5cc5d + - path: Sources/LatticeLib/Tokenizer.swift + blob: 82a8b85b5c31163d51194c7f28320305f8c798af + - path: Sources/LatticeLib/WordClass.swift + blob: 9d4851967dca01b266ad0d8f4f35cf4a757d3d7d + - path: Sources/LatticeLib/WordClassTable.swift + blob: 731f6f90c59aae71c73114ede301877d1bf8035f + - path: Sources/LatticeLib/WordClassTagger.swift + blob: 15e27da5cdeca3540654914992e8c78f83ed937a +--- + +# LatticeLib Overview + +## What This Library Does + +LatticeLib reads a piece of text and answers one question: what subject is +this about? It answers with a lattice code. A lattice code is a structured +classification code that gives a memory a place in an organized hierarchy of +subjects. For example, the text "the dog chased a ball in the park" encodes +to a code in the animals region of the hierarchy. + +MOOTx01 is an on-device AI memory system. It stores what an AI observes over +time and helps the AI recall it later. Every memory that enters the system +passes through LatticeLib. The code that LatticeLib assigns becomes part of +the memory's filing address, so memories about the same subject cluster +together even when they share no exact words. + +The engine inside LatticeLib is called FDC, which stands for Frame-Directed +Classification. A frame is a fixed, versioned list of classification codes +and their subject labels. FDC directs every piece of text to its best +position in that frame. + +## The Problem It Solves + +Two devices must file the same text under the same code. MOOTx01 estates can +federate, which means separate devices share and compare memories. If one +device filed "dog" under animals and another filed it under sports, shared +recall would fall apart. Classification therefore has to be deterministic: the +same input must produce the same output on every platform, every time. + +Cloud classifiers cannot promise this. They change without notice, they +require a network, and they see private text. LatticeLib instead ships every +piece of reference data it needs as pinned artifacts. A pinned artifact is a +data file that is built once, given a version, checked into the repository, +and never modified at runtime. The library runs entirely on the device. The +same text against the same artifact versions yields the same code — the +library's agreement property. + +The library keeps that promise across two independent implementations. A +Swift leg serves Apple platforms, and a Rust leg (in `rust/`) serves +everything else. Shared conformance fixtures — recorded input and output +pairs that both legs must reproduce exactly — gate every release. + +## How It Works + +Classification runs in five steps. The first three turn text into a concept +bag. A concept bag is a table that maps each concept found in the text to the +number of times it appears. The last two match that bag against the frame. + +Step 1 tags each word and keeps only nouns and verbs. Articles, adjectives, +and everything else carry little subject meaning, so they are dropped. A word +the tagger has never seen is called a novel token. Novel tokens are tagged by +a small, deterministic Hidden Markov Model, so this step never guesses +differently on different platforms. + +Step 2 canonicalizes each kept word. The word is normalized (odd Unicode +forms are folded to plain letters), stemmed (reduced to its root, so "dogs" +becomes "dog"), and looked up in the canonicalization lexicon. The lexicon is +a pinned artifact that maps roughly seventy thousand word roots to concept +identities. A concept identity is usually a Wikidata Q-ID — a public, +language-neutral identifier such as `Q144` for "dog." Synonyms collapse onto +one identity, which is what lets two devices agree. + +Step 3 accumulates the counts. The result is the concept bag. + +Step 4 scores the bag against code signatures. A code signature is the set of +concepts that characterizes one classification code, prepared ahead of time +from reference articles. Codes whose signatures share concepts with the bag +earn scores; rarer shared concepts earn more. The best-scoring code wins. If +nothing overlaps, or too many codes tie, the library returns UNRESOLVED +rather than guess. + +Step 5 descends the frame. Starting from the winning code, the matcher walks +down to child codes while the text still supports the extra specificity, and +returns the deepest supported code. + +## How the Pieces Fit + +Figure 1 shows the library's topology — its major parts and how data moves +between them. + +![Figure 1. Topology of LatticeLib](topology.svg) + +*Figure 1. Topology of LatticeLib. Text flows left to right through the +shared text primitives into the concept bag, then through the matcher to a +code. Dashed regions mark the pinned artifacts and the offline novel-token +learning loop.* + +The runtime entry point is the `FDC` enum. Consumers call +`FDC.encodeAnchor(text)` and receive two values: the lattice code and the +dominant concept Q-ID of the text. Loading of the three runtime artifacts +(lexicon, frame, signatures) happens once per process. + +The shared text primitives — `Tokenizer`, `Normalizer`, `Stemmer`, and the +word-class tagger — serve both the runtime encoder and the build-time tools. +Build-time tools (`LexiconBuilder`, `SignatureAssembler`, `LexRank`) produce +the pinned artifacts; they never run on user devices. + +One slow feedback loop improves the tagger over time. Novel tokens accumulate +in a local cache. When fifty gather, they are written to a local pool +directory. A reducer later merges qualifying tokens into a writable copy of +the word-class table, and the running process adopts the new table with an +atomic swap. Classification stays deterministic because determinism is always +relative to a table version, and the version advances only at that swap. + +Privacy is engineered into the seams. When the text being classified is +private memory content, callers pass `recordNovel: false`, and no token of it +is ever written to the pool. The pool itself is off by default; it activates +only when a host sets the `LATTICE_POOL_DIR` environment variable. + +## What Ships in the Package + +The package ships the Swift sources, the Rust port in `rust/`, and the pinned +artifacts in `Sources/LatticeLib/Resources/`: the frame (1,075 codes), the +code signatures (1,071 entries), the lexicon (about 70,000 entries), the +Q-ID ancestor graph (about 39,000 nodes), the trained tagger model, the +word-class table, and the stemmer conformance corpus. Each artifact carries +its own version string, and the library reports the versions it used, so +every classification is reproducible provenance included. diff --git a/packages/libs/LatticeLib/docs/topology.svg b/packages/libs/LatticeLib/docs/topology.svg new file mode 100644 index 0000000..e81bfe7 --- /dev/null +++ b/packages/libs/LatticeLib/docs/topology.svg @@ -0,0 +1,130 @@ + + + + + + + + + + + + + LatticeLib: text in, lattice code out + + + + Input text + FDC.encodeAnchor + + + Text primitives + Tokenizer · Normalizer · Stemmer + + + Word-class tagger + table fast path + HMM + + + BagBuilder + concept bag + + + FDCMatcher + score + descend + + + Lattice code + + concept Q-ID + + + + + + + + + + + Pinned artifacts (versioned, read-only, loaded once per process) + + + Lexicon + ~70k stems → Q-IDs + + + FDC frame + 1,075 codes + labels + + + Code signatures + 1,071 term sets + + + QID closure + P31/P279 graph + + + + + + + + + Novel-token learning loop (offline, opt-in via LATTICE_POOL_DIR) + + + NovelTokenCache + drains at 50 entries + + + NovelPoolSubmitter + JSON files → pool dir + + + PoolReducer + merge nouns/verbs + + + WordClassTable + live atomic swap + + + + + + + + From ae4006b83240f5093e1313fae4af687db37eefd8 Mon Sep 17 00:00:00 2001 From: Fable Date: Sat, 4 Jul 2026 16:06:52 -0500 Subject: [PATCH 2/3] =?UTF-8?q?docs(sdk):=20package=20docs=20fan-out=20?= =?UTF-8?q?=E2=80=94=20OVERVIEW,=20DETAILS,=20AGENT=5FMAP,=20topology=20pe?= =?UTF-8?q?r=20package?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3-4 of the Moot-SDK documentation initiative. Voice: Turabian at 8th grade per the LatticeLib pilot exemplar. Front-matter pins per-repo authored_commit and per-source git blob hashes for docs-scrub. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Pzkqg3bpbbhfuWRsYomJ3o --- .../libs/AriaLexiconLib/docs/AGENT_MAP.md | 85 +++++++ packages/libs/AriaLexiconLib/docs/DETAILS.md | 196 ++++++++++++++++ packages/libs/AriaLexiconLib/docs/OVERVIEW.md | 140 +++++++++++ .../libs/AriaLexiconLib/docs/topology.svg | 97 ++++++++ packages/libs/EideticLib/docs/AGENT_MAP.md | 63 +++++ packages/libs/EideticLib/docs/DETAILS.md | 222 ++++++++++++++++++ packages/libs/EideticLib/docs/OVERVIEW.md | 148 ++++++++++++ 7 files changed, 951 insertions(+) create mode 100644 packages/libs/AriaLexiconLib/docs/AGENT_MAP.md create mode 100644 packages/libs/AriaLexiconLib/docs/DETAILS.md create mode 100644 packages/libs/AriaLexiconLib/docs/OVERVIEW.md create mode 100644 packages/libs/AriaLexiconLib/docs/topology.svg create mode 100644 packages/libs/EideticLib/docs/AGENT_MAP.md create mode 100644 packages/libs/EideticLib/docs/DETAILS.md create mode 100644 packages/libs/EideticLib/docs/OVERVIEW.md diff --git a/packages/libs/AriaLexiconLib/docs/AGENT_MAP.md b/packages/libs/AriaLexiconLib/docs/AGENT_MAP.md new file mode 100644 index 0000000..3fe1f42 --- /dev/null +++ b/packages/libs/AriaLexiconLib/docs/AGENT_MAP.md @@ -0,0 +1,85 @@ +--- +doc: AGENT_MAP +package: AriaLexiconLib +repo: moot-semantics +authored_commit: 021ea704162f86fccea2f030ea9419dacc30a345 +authored_date: 2026-07-04 +sources: + - path: Sources/AriaLexiconLib/Acceptance.swift + blob: 095b16e63587937c22911a7bada283afdf5aea1a + - path: Sources/AriaLexiconLib/Adjective.swift + blob: 4c74c273c5fbf1c863f478de3287133119780936 + - path: Sources/AriaLexiconLib/AriaLexiconLib.swift + blob: 6a7f1d2b66f7fb9b2635befacdd44f3c3212217b + - path: Sources/AriaLexiconLib/Noun.swift + blob: 539bf0460df8d6f60d5e63c11a7706c4c8557eb3 + - path: Sources/AriaLexiconLib/Verb.swift + blob: 7ecf10b17758b23b7652c47879c2b353d28f5965 +--- + +# AGENT_MAP — AriaLexiconLib + +PURPOSE: the reified ARIA grammar — one noun (Drawer), nine verbs, four adjective categories, and the verb-noun acceptance matrix, as pure data. No behavior, no I/O, no dependencies. Canonical prose statement lives outside this package in ARIA_LEXICON.md; this module is that statement made first-class in code so it can be conformance-checked. + +DEPS: imports nothing (zero-dependency, foundational leaf package; Swift stdlib only). Imported by: none currently declare a package/`import` dependency within this SDK checkout. LocusKit (moot-memory/packages/kits/LocusKit) references `AriaLexiconLib`'s types by name in doc comments (Acceptance.swift, Verb.swift, Noun.swift) as the conceptual source of truth for its own Frames/Association/Proposal/LearnedReference/EstateVerbs types, but does NOT declare it as a Package.swift dependency or `import` it as of this commit — treat as documentation-level, not compiled, coupling. Package header names VectorKit and CorpusKit as intended future conformers. Rust port: `rust/` (crate `aria-lexicon-lib`, single file `src/lib.rs`) mirrors every type and the acceptance matrix by hand; its own `#[cfg(test)]` module is the conformance gate (wire-string + matrix parity vs. this Swift package). + +ENTRY POINTS (most callers need only these): +- Acceptance.swift:36 `Acceptance.accepts(_ noun: Noun, _ verb: Verb) -> Bool` — the one call most callers want: "is this operation allowed on this shape" +- Acceptance.swift:14 `Acceptance.verbs(for noun: Noun) -> Set` — full accepted-verb set for a shape +- AriaLexiconLib.swift:15 `AriaLexiconLib.grammar` — the one-sentence rule as a displayable string + +## Symbol Table + +### Module — AriaLexiconLib.swift +- :13 `enum AriaLexiconLib` — empty namespace enum +- :15 `static let grammar: String` — "Every call is one verb applied to a noun, optionally constrained by adjectives." + +### Noun — Noun.swift +- :14 `enum Noun: String, CaseIterable, Sendable, Codable` — 8 storage shapes; rawValue IS the wire string (lowerCamelCase case names, no explicit `=`) +- :15–22 cases: `drawer`, `tunnel`, `kgFact`, `vector`, `diaryEntry`, `proposal`, `association`, `learnedReference` (declaration order = Rust `Noun::ALL` order) +- :27 `static let primary: Noun = .drawer` — the one true noun of the language +- :31 `var role: NounRole` — computed; drawer→primary, kgFact/vector→rung, tunnel/diaryEntry/association→structure, proposal/learnedReference→product +- :45 `enum NounRole: String, CaseIterable, Sendable, Codable` — :47 `primary`, :49 `rung`, :51 `structure`, :53 `product` + +### Verb — Verb.swift +- :9 `enum Verb: String, CaseIterable, Sendable, Codable` — 9 actions, count fixed by spec invariant I-7 +- :10–18 cases: `capture`, `reanchor`, `mutate`, `withdraw`, `expunge`, `recall`, `propose`, `associate`, `learn` (declaration order = Rust `Verb::ALL` order) +- :21 `var flow: Flow` — computed; capture/reanchor/mutate/withdraw/expunge/recall→callerDriven, propose/associate→substrateDriven, learn→groundingDriven +- :37 `enum Flow: String, CaseIterable, Sendable, Codable` — :38 `callerDriven`, :39 `substrateDriven`, :40 `groundingDriven` + +### Adjective — Adjective.swift +- :13 `enum Adjective: String, CaseIterable, Sendable, Codable` — 4 cross-noun categories, count fixed by spec invariant I-8 +- :17 `state` — lifecycle position (active/pending/contested/superseded/decayed/withdrawn/expired/rejected/accepted/tombstoned — VALUES live in LocusKit, not here) +- :20 `trust` — provenance (verbatim/observed/imported/proposed/derived/canonical — values in LocusKit) +- :23 `sensitivity` — exposure level (normal/elevated/restricted/secret — values in LocusKit) +- :26 `exportability` — perimeter crossing (private/public — values in LocusKit) + +### Acceptance matrix — Acceptance.swift +- :10 `enum Acceptance` — namespace for the matrix +- :14 `static func verbs(for noun: Noun) -> Set` — hand-authored per-noun set; SEE ENTRY POINTS +- :16 drawer → {capture, reanchor, mutate, withdraw, expunge, recall} (6, fullest row) +- :18 tunnel → {capture, mutate, withdraw, expunge, recall} (5, no reanchor) +- :20 kgFact → {mutate, withdraw, expunge, recall} (4, no capture/reanchor) +- :22 vector → {} (0 — substrate-managed, never verb-addressable directly) +- :24 diaryEntry → {recall} (1, read-only) +- :26 proposal → {propose, mutate, withdraw, expunge, recall} (5; role is "product" but DOES accept propose) +- :28 association → {associate, mutate, expunge, recall} (4; no withdraw) +- :30 learnedReference → {learn, mutate, withdraw, expunge, recall} (5) +- :36 `static func accepts(_ noun: Noun, _ verb: Verb) -> Bool` — `verbs(for:).contains(verb)`; SEE ENTRY POINTS + +### Rust port — rust/src/lib.rs (mirror, not a Swift symbol) +- `Noun`/`NounRole`/`Verb`/`Flow`/`Adjective` — PascalCase enum variants + `as_str()` producing the identical lowerCamelCase wire strings as the Swift rawValues +- `accepted_verbs(Noun) -> Vec` / `accepts(Noun, Verb) -> bool` — same matrix, hand-mirrored +- `#[cfg(test)] mod tests` — conformance gate: verb/adjective counts, full matrix cell-by-cell, wire-string parity, serde round-trip + +## INVARIANTS / GOTCHAS + +- NO BEHAVIOR. This package is pure data (enums, one constant string, one lookup table). Do not add I/O, logging, or business logic here; new domain operations compose the nine existing verbs instead of extending the vocabulary. +- Verb count is pinned at 9 (spec invariant I-7). Adjective category count is pinned at 4 (spec invariant I-8). Both are spec-level invariants, not incidental — changing either is a specification amendment, not a routine edit. +- `Acceptance`'s matrix is HAND-AUTHORED, not derived from `Noun.role` or `Verb.flow`. Do not "simplify" it by deriving from role/flow — the Proposal row (role=`product`, yet accepts `.propose`) and the Association row (role=`structure`, accepts `.associate` but not `.propose`) are real exceptions that a derived rule would get wrong. +- `Adjective` names categories only. The value lists inside each category (the 10 lifecycle states, 6 trust levels, 4 sensitivity levels, 2 exportability levels) live in LocusKit (moot-memory), not here — by design, so the two never fork by being defined twice. Do not add value enums to this file. +- Zero dependencies is load-bearing: this package must stay leaf-level (imports nothing) so any kit — including ones layered far apart, like LocusKit, VectorKit, and CorpusKit — can depend on it without a cycle. +- Swift enum rawValues ARE the wire strings (case names are already lowerCamelCase; no explicit `= "..."` needed). Rust mirrors with PascalCase variants + `as_str()`/`serde(rename_all = "camelCase")` to reproduce the identical strings. Any edit to a Swift case name, or to Rust's `as_str()`/serde attributes, that the other side does not mirror breaks wire conformance silently unless the Rust test suite is rerun. +- Two independent implementations, no shared codegen: editing `Acceptance.swift`, `Noun.swift`, `Verb.swift`, or `Adjective.swift` requires a matching hand-edit to `rust/src/lib.rs` (types, `as_str()`, and `accepted_verbs`/`accepts`) plus a rerun of the Rust `#[cfg(test)]` suite. There is no build step that generates one from the other. +- Canonical source-of-truth order: the prose document ARIA_LEXICON.md (external to this package, not present in this repo checkout) states the grammar; this Swift package is the code reification of that document; the Rust crate is a conformance-gated mirror of this package. When in doubt about intent, ARIA_LEXICON.md wins. +- No compiled consumer exists yet within this SDK checkout — verify current importers before assuming call sites exist. LocusKit's references are doc-comment citations, not `import` statements or Package.swift dependencies, as of this commit. diff --git a/packages/libs/AriaLexiconLib/docs/DETAILS.md b/packages/libs/AriaLexiconLib/docs/DETAILS.md new file mode 100644 index 0000000..63c2832 --- /dev/null +++ b/packages/libs/AriaLexiconLib/docs/DETAILS.md @@ -0,0 +1,196 @@ +--- +doc: DETAILS +package: AriaLexiconLib +repo: moot-semantics +authored_commit: 021ea704162f86fccea2f030ea9419dacc30a345 +authored_date: 2026-07-04 +sources: + - path: Sources/AriaLexiconLib/Acceptance.swift + blob: 095b16e63587937c22911a7bada283afdf5aea1a + - path: Sources/AriaLexiconLib/Adjective.swift + blob: 4c74c273c5fbf1c863f478de3287133119780936 + - path: Sources/AriaLexiconLib/AriaLexiconLib.swift + blob: 6a7f1d2b66f7fb9b2635befacdd44f3c3212217b + - path: Sources/AriaLexiconLib/Noun.swift + blob: 539bf0460df8d6f60d5e63c11a7706c4c8557eb3 + - path: Sources/AriaLexiconLib/Verb.swift + blob: 7ecf10b17758b23b7652c47879c2b353d28f5965 +--- + +# AriaLexiconLib Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. Files appear here in the order a +reader builds understanding: the module surface and its one-sentence +grammar, then the noun, then the verb, then the adjective categories, and +finally the acceptance matrix that relates the noun and the verb. + +## AriaLexiconLib.swift + +This file provides the module surface: the public `AriaLexiconLib` enum and +the one-sentence statement of the grammar. + +Swift has no module-level functions or constants, so a library that wants a +single well-known place for a top-level fact uses an empty enum as a +namespace. `AriaLexiconLib` is that namespace; it exists only to hold +`grammar`. + +`AriaLexiconLib.grammar` is a constant string: "Every call is one verb +applied to a noun, optionally constrained by adjectives." This matters +because the rule is meant to be read by people, not only enforced by types. +A consumer can print this string in a log, a debugging tool, or a developer +console, and it will always match the rule the rest of the package encodes, +because it lives in the same file set and the same version as the types +that implement it. + +## Noun.swift + +This file provides `Noun`, the eight storage shapes the substrate persists, +and `NounRole`, how each shape relates to the one true noun of the +language, the Drawer. + +A Drawer is the atomic unit of memory: one row of stored content. The other +seven cases in `Noun` — `tunnel`, `kgFact`, `vector`, `diaryEntry`, +`proposal`, `association`, and `learnedReference` — are not separate things +a caller thinks about the way it thinks about a Drawer. Each is either a +different view onto Drawer content or a record of something a verb did. +The file's opening comment is explicit that the architecture specification +calls all of these "nouns" loosely, as a storage taxonomy, but only Drawer +is a noun in ARIA's grammar sense; the rest are facets or residue. + +`Noun.primary` is a static constant equal to `.drawer`. It gives code a +named way to ask "which case is the real noun" without hard-coding the +case name at every call site, so a future reader searching for "primary" +finds the one place that decision is made. + +`Noun.role` is a computed property that classifies each case into one of +four `NounRole` values: `.primary` for the Drawer itself, `.rung` for a +representation of a Drawer's content (`kgFact` and `vector`), `.structure` +for an edge or event about Drawers (`tunnel`, `diaryEntry`, `association`), +and `.product` for what a verb leaves behind (`proposal`, +`learnedReference`). This grouping matters because a caller reasoning about +a storage shape — for example, deciding whether it is safe to delete — +often cares about the shape's role more than its exact case name; a rung +can usually be rebuilt from its source Drawer, while a product generally +cannot. + +`NounRole` itself is a four-case, string-backed enum +(`primary`, `rung`, `structure`, `product`). Being string-backed means it +serializes to a stable, human-readable form rather than a raw integer, which +matters for any log or wire format that includes it. + +## Verb.swift + +This file provides `Verb`, the nine actions the substrate supports, and +`Flow`, who initiates each one. + +The nine verbs are `capture`, `reanchor`, `mutate`, `withdraw`, `expunge`, +`recall`, `propose`, `associate`, and `learn`. The file's opening comment +states the design rule directly: this count is fixed by specification +invariant I-7, and a new kind of domain operation is expected to compose +these nine rather than add a tenth. Keeping the verb set closed is what +lets every kit share one small vocabulary instead of each kit growing its +own private verbs over time. + +`Verb.flow` is a computed property returning a `Flow` value that answers +"who calls this verb." Six verbs — `capture`, `reanchor`, `mutate`, +`withdraw`, `expunge`, and `recall` — are `.callerDriven`: an application +calls them directly, synchronously, when it wants something to happen. Two +verbs, `propose` and `associate`, are `.substrateDriven`: they are emitted +by the system's own background processes on their own schedule, not called +directly by application code. One verb, `learn`, is `.groundingDriven`: it +is the verb that brings authoritative outside reference material into the +system, distinct from both a direct caller action and an internal +background signal. Separating these three flows matters for anyone writing +code against the substrate, because a caller-driven verb needs a call site, +while a substrate-driven or grounding-driven verb needs a listener instead. + +`Flow` is a three-case, string-backed enum (`callerDriven`, +`substrateDriven`, `groundingDriven`). + +## Adjective.swift + +This file provides `Adjective`, the four categories that describe any row +of memory regardless of which noun or verb produced it. + +The four categories are `state` (where the row sits in its lifecycle, such +as active, pending, or superseded), `trust` (how the content was +established, such as verbatim, observed, or derived), `sensitivity` (how +exposed the content may be), and `exportability` (whether the content may +leave the access perimeter — private or public). The count of four is +fixed by specification invariant I-8, the same way the verb count is fixed +by I-7. + +The file's comment explains an important design boundary: `Adjective` only +names the four categories, not the specific values inside each one. Every +row carries some value in each category no matter its storage shape, which +is why the categories are described as "cross-noun." But how those values +are packed and represented — the actual list of lifecycle states, for +example — is treated as a bitmap-layout detail, and that detail is reified +in a separate kit, LocusKit, instead of here. Keeping the category names in +one place and the value lists in another means the two cannot silently +drift apart by being maintained twice; only one of them needs to change +when a new value is added to a category. + +## Acceptance.swift + +This file provides `Acceptance`, the verb-noun acceptance matrix: the one +place the package states which of the nine verbs make sense for each of +the eight storage shapes. + +The matrix is hand-authored data, not derived from `Noun.role` or +`Verb.flow`. For example, `Proposal`'s role is "product," yet it does +accept the verb that creates it, `propose`, along with the ordinary +lifecycle verbs (`mutate`, `withdraw`, `expunge`, `recall`). The file's +opening comment frames this precisely: in ARIA's grammar the matrix reads +as "which actions apply to this shape," not as a competition between +nouns, and it is expressed as data specifically so that a conformance test +can check the Swift and Rust implementations agree cell by cell. + +`Acceptance.verbs(for:)` takes a `Noun` and returns the `Set` it +accepts. A Drawer accepts six verbs (every caller-driven verb except none +— it is the fullest row in the matrix). A Tunnel accepts five, omitting +`reanchor`. A knowledge-graph fact accepts four, omitting both `capture` +and `reanchor`, because a fact is not captured directly the way a Drawer +or Tunnel is. A Vector accepts none: the comment explains that vectors are +substrate-managed, meaning an internal process derives and maintains them, +so no caller ever addresses one with a verb directly. A diary entry accepts +only `recall`, since it is a read-only record. A Proposal accepts +`propose` plus the lifecycle verbs, and an Association accepts `associate` +plus most of the lifecycle verbs (it omits `withdraw`). A learned reference +accepts `learn` plus the lifecycle verbs. This function matters because it +is the only place any of these facts is stated; a caller that needs to know +"can I do X to a Y" asks this function instead of re-deriving the answer +from role or flow, which would sometimes give the wrong answer, as the +Proposal example shows. + +`Acceptance.accepts(_:_:)` takes a noun and a verb and returns whether the +verb is a member of that noun's accepted set. It is a thin convenience over +`verbs(for:)`, provided because most call sites want a yes-or-no answer for +one specific pair rather than the whole set, and spelling that check inline +at every call site would risk a typo that a shared function cannot make. + +## Rust Port and Conformance + +The `rust/` directory contains the second leg of the library, `lib.rs`, a +single-file crate named `aria-lexicon-lib`. It mirrors every Swift type: +`Noun`, `NounRole`, `Verb`, `Flow`, and `Adjective` as Rust enums, plus +`accepted_verbs` and `accepts` as the Rust equivalents of +`Acceptance.verbs(for:)` and `Acceptance.accepts(_:_:)`. Swift's string-backed +enum cases are already lowerCamelCase and serve as their own wire strings. +Rust instead declares PascalCase variants by convention and adds an +`as_str()` method, plus `serde` attributes, that reproduce the identical +lowerCamelCase wire strings for JSON — for example, `Noun::KgFact.as_str()` +returns `"kgFact"`, matching Swift's `Noun.kgFact.rawValue` exactly. + +The crate's own test module is the conformance suite: it checks the verb +count is nine and the adjective count is four, checks every wire string +against its expected value, checks that `serde` serialization matches +`as_str()` for every case, and re-states the entire acceptance matrix cell +by cell, including the harder-to-guess cases such as a Proposal not +accepting `associate` and an Association not accepting `propose`. Because +the two languages cannot share source code for this vocabulary, this test +module is the only mechanism that catches the two ports drifting apart: +when you change `Acceptance.swift`, `Noun.swift`, `Verb.swift`, or +`Adjective.swift`, you must make the matching edit in `rust/src/lib.rs` and +rerun both test suites, or a silent disagreement ships. diff --git a/packages/libs/AriaLexiconLib/docs/OVERVIEW.md b/packages/libs/AriaLexiconLib/docs/OVERVIEW.md new file mode 100644 index 0000000..cf49e4c --- /dev/null +++ b/packages/libs/AriaLexiconLib/docs/OVERVIEW.md @@ -0,0 +1,140 @@ +--- +doc: OVERVIEW +package: AriaLexiconLib +repo: moot-semantics +authored_commit: 021ea704162f86fccea2f030ea9419dacc30a345 +authored_date: 2026-07-04 +sources: + - path: Sources/AriaLexiconLib/Acceptance.swift + blob: 095b16e63587937c22911a7bada283afdf5aea1a + - path: Sources/AriaLexiconLib/Adjective.swift + blob: 4c74c273c5fbf1c863f478de3287133119780936 + - path: Sources/AriaLexiconLib/AriaLexiconLib.swift + blob: 6a7f1d2b66f7fb9b2635befacdd44f3c3212217b + - path: Sources/AriaLexiconLib/Noun.swift + blob: 539bf0460df8d6f60d5e63c11a7706c4c8557eb3 + - path: Sources/AriaLexiconLib/Verb.swift + blob: 7ecf10b17758b23b7652c47879c2b353d28f5965 +--- + +# AriaLexiconLib Overview + +## What This Library Does + +AriaLexiconLib names the words that every MOOTx01 kit uses to describe an +action on memory. Inside MOOTx01, that fixed vocabulary is called ARIA. ARIA +states a simple grammar: every call to the substrate is one verb applied to +a noun, optionally constrained by adjectives. AriaLexiconLib turns that +sentence into code, so a program can check whether a given verb is even +meaningful for a given noun before it tries the call. + +The library has one noun, the Drawer, which is the atomic unit of memory. +Every other storage shape the substrate persists — a knowledge-graph fact, a +vector, a diary entry — is either a facet of a Drawer or something a verb +leaves behind. The library has nine verbs, such as capture, recall, and +withdraw, and four adjective categories, such as trust and sensitivity, that +describe a row without changing which noun or verb it is. A small acceptance +matrix records which verbs apply to which storage shapes. + +## The Problem It Solves + +MOOTx01 is an on-device AI memory system. It stores what an AI observes over +time and helps the AI recall it later. Many separate packages — the SDK +described at the top of this repository set calls them kits — build on top +of this vocabulary: one kit stores knowledge-graph facts, another stores +vector embeddings, another manages the estate's day-to-day operations. If +each kit invented its own names for "add a memory" or "remove a memory," a +caller would need to learn a different vocabulary per kit, and it would +become easy to call an operation that does not make sense for a given +storage shape, such as asking a vector to be captured directly instead of +letting the substrate manage it. + +AriaLexiconLib fixes the vocabulary once, in one place, as data rather than +as prose scattered across each kit's documentation. Every kit conforms to +it: a kit's API surface uses these same noun and verb names, and its +allowed-operations logic reads its answers from the acceptance matrix +instead of restating the rule. The vocabulary also crosses languages: the +library ships a Rust port so that non-Swift consumers speak the identical +words, checked one against the other by shared tests. + +The canonical statement of the grammar is a prose document, ARIA_LEXICON.md, +maintained outside this package. AriaLexiconLib is that document made +first-class in code, so an editor, a compiler, and a conformance test can +all point at the same set of names. + +## How It Works + +The library defines four small pieces of vocabulary and one rule that +relates two of them. + +`Noun` lists the eight storage shapes the substrate persists, and marks the +Drawer as the one true noun of the language. The other seven are either a +"rung" (a representation of a Drawer's content, such as a vector), a piece +of "structure" (an edge or event about Drawers, such as an association), or +a "product" (what a verb leaves behind, such as a proposal). + +`Verb` lists the nine actions the substrate supports: capture, reanchor, +mutate, withdraw, expunge, recall, propose, associate, and learn. Each verb +also names who initiates it. Six verbs are caller-driven, meaning an +application calls them directly. Two are substrate-driven, meaning an +internal background process emits them on its own schedule. One, +`learn`, is grounding-driven: it is how authoritative outside reference +material enters the system. + +`Adjective` lists the four categories that describe a row no matter which +noun or verb touched it: state (where the row sits in its lifecycle, such as +active or superseded), trust (how the content was established, such as +verbatim or derived), sensitivity (how exposed the content may be), and +exportability (whether the content may leave the access perimeter). This +library only names the four categories. The specific values inside each +category — for example, the full list of lifecycle states — are a detail of +how the data is packed into storage, and that detail lives in a separate +kit, LocusKit, so the two never drift out of agreement with each other by +being defined twice. + +`Acceptance` is the one rule that connects `Noun` and `Verb`. For each of +the eight nouns, it lists exactly which of the nine verbs make sense. A +Drawer accepts six verbs, including capture and recall. A Vector accepts +none, because the substrate manages vectors on its own; nothing calls a verb +on one directly. This matrix is hand-authored, not derived automatically +from any other rule in the library, so it is the single place a caller or a +test checks "is this operation allowed here." + +## How the Pieces Fit + +Figure 1 shows how the four vocabulary types and the acceptance matrix +relate to each other and to the Rust port. + +![Figure 1. Topology of AriaLexiconLib](topology.svg) + +*Figure 1. Topology of AriaLexiconLib. `Noun` and `Verb` feed the +`Acceptance` matrix; `Adjective` stands beside them as an independent, +cross-noun category list. The dashed regions mark the Rust mirror and the +downstream packages that conform to this vocabulary without this package +depending on them.* + +`AriaLexiconLib.grammar` states the one-sentence rule in plain English, as a +constant string, so a consumer can display or log the rule itself rather +than reconstruct it from the types. `Noun` and `Verb` are independent +enumerations; neither refers to the other. `Acceptance.verbs(for:)` and +`Acceptance.accepts(_:_:)` are the only functions in the package, and both +simply look up a fixed table keyed by a noun. `Adjective` does not +participate in the acceptance rule at all — an adjective always applies to +a row regardless of its noun or the verb that produced it. + +The Rust port in `rust/` mirrors every type and the acceptance matrix by +hand, because the two languages cannot share source code. A build-time +conformance suite inside the Rust crate re-asserts every wire string and +every cell of the acceptance matrix against the values recorded here, so a +change to either side that the other side misses fails a test rather than +shipping a silent disagreement. + +## What Ships in the Package + +The package ships five Swift source files, no bundled data files, and the +Rust port in `rust/`. There are no pinned artifacts and no build-time +tooling: every value in the package is a literal written directly in the +source, because the vocabulary is small and fixed by specification rather +than computed or trained. The verb count is fixed at nine and the adjective +category count at four; both counts are named spec invariants, so growing +either number is a specification change, not a routine code edit. diff --git a/packages/libs/AriaLexiconLib/docs/topology.svg b/packages/libs/AriaLexiconLib/docs/topology.svg new file mode 100644 index 0000000..f6e72c7 --- /dev/null +++ b/packages/libs/AriaLexiconLib/docs/topology.svg @@ -0,0 +1,97 @@ + + + + + + + + + + + + + AriaLexiconLib: the ARIA grammar as data + + + + AriaLexiconLib.grammar + "verb applied to noun, ± adjectives" + + + + Noun + 8 shapes; drawer primary + + + Verb + 9 actions; 3 flows + + + Adjective + 4 categories, cross-noun + + + + Acceptance + verb-noun matrix (hand-authored) + + + + + + + + + + + + + Rust port (conformance-gated mirror) + + + rust/src/lib.rs + aria-lexicon-lib crate + #[cfg(test)] parity gate + + + + + + + + Downstream conformers (outside this package) + + + LocusKit + doc-comment citations only + + + VectorKit / CorpusKit + named future conformers + + + diff --git a/packages/libs/EideticLib/docs/AGENT_MAP.md b/packages/libs/EideticLib/docs/AGENT_MAP.md new file mode 100644 index 0000000..203cca4 --- /dev/null +++ b/packages/libs/EideticLib/docs/AGENT_MAP.md @@ -0,0 +1,63 @@ +--- +doc: AGENT_MAP +package: EideticLib +repo: moot-semantics +authored_commit: 021ea704162f86fccea2f030ea9419dacc30a345 +authored_date: 2026-07-04 +sources: + - path: Sources/EideticLib/EideticLib.swift + blob: 1839588dc98279c1eee2786f1974457a5608742b + - path: Sources/EideticLib/LatticeCodeState.swift + blob: 7580261208b77c7e79363a510920c413054264a1 + - path: Sources/EideticLib/Segmenter.swift + blob: c5313d93bf7ca1cbf577da6ffabc9197b5e788e2 +--- + +# AGENT_MAP — EideticLib + +PURPOSE: thin deterministic facade over LatticeLib's FDC engine (term → Anchor{code, wikidataQID, confidence, dataVersion}) plus two unrelated small utilities: lattice-code shape/known-state classification, and sentence segmentation. Carries no reference data of its own. + +DEPS: imports LatticeLib (FDC.encodeAnchor/isAvailable/dataVersion), NaturalLanguage (Apple-only NLTokenizer sentence path), Foundation. Imported by: CorpusKit (moot-memory kit, `EideticLib.sentences` only) and, per Package.swift header comment, NeuronKit (not present in these four venue repos — likely mootx01-ee/ce, out of scope here). Rust port in rust/ (crate `eidetic-lib`) mirrors lookup, classifyLatticeCode, and the delimiter segmenter exactly; depends on sibling `lattice-lib` crate. No Rust counterpart to the NLTokenizer path (Apple-only, excluded from cross-leg parity by the apple-nlp-accel constitutional constraint C-2). + +ENTRY POINTS (most callers need only these): +- EideticLib.swift:67 `EideticLib.lookup(_ term: String) -> Anchor` — main lookup; fatalError if FDC.isAvailable is false +- EideticLib.swift:118 `EideticLib.lookup(_:recordNovel:)` — same, recordNovel:false = privacy seam (no LatticeLib pool writes) +- EideticLib.swift:40 `EideticLib.classifyLatticeCode(_:knownCodes:) -> LatticeCodeState` — malformed/known/pending +- Segmenter.swift:47 `EideticLib.sentences(_ text: String) -> [Substring]` — platform-routed sentence split + +## Symbol Table + +### Lookup surface — EideticLib.swift +- :20 `enum EideticLib` — namespace; holds no reference data, delegates entirely to LatticeLib.FDC +- :23 `version = "0.1.0"` — module version string +- :40 `classifyLatticeCode(_:knownCodes:) -> LatticeCodeState` — grammar check via LatticeCodeGrammar, then knownCodes membership +- :67 `lookup(_:) -> Anchor` — guards FDC.isAvailable (else fatalError); calls FDC.encodeAnchor(term); resolved→confidence 32, UNRESOLVED→empty code/confidence 0 +- :118 `lookup(_:recordNovel:) -> Anchor` — identical Anchor to lookup(_:); recordNovel:false suppresses LatticeLib novel-token pool accumulation; used by GLK capture intake (EncodeIntake) so private memory content never reaches the pool +- :147 `struct Anchor: Equatable, Sendable, Codable` — code (empty=UNRESOLVED) / wikidataQID (String?) / confidence (UInt8: 0/16/32/48/56=null/low/medium/high/verified; EideticLib only emits 0 or 32) / dataVersion (FDC signatures version); byte-identical JSON shape to Rust `Anchor` + +### Lattice-code state — LatticeCodeState.swift +- :27 `enum LatticeCodeState: Sendable, Hashable, Codable` — .malformed(String) / .known(String) / .pending(String) +- :46 `rawCode: String` — original input regardless of case (storage round-trip without unpacking) +- :56 `isWellFormed: Bool` — false only for .malformed +- :72 `enum LatticeCodeGrammar` — dependency-free; parallel reimplementation of LatticeLib's `Code.isWellFormed(_:)`, kept in agreement by shared conformance tests, NOT shared code +- :77 `maxExtensionDigits = 8` — pinned; matches LatticeLib Code.maxExtensionDigits +- :82 `isWellFormed(_ code:) -> Bool` — 3 ASCII digits + optional `.` + 1–8 ASCII digits; empty/overlong/non-digit extension fails + +### Sentence segmentation — Segmenter.swift (extension EideticLib) +- :47 `sentences(_ text:) -> [Substring]` — Apple: NLTokenizer(unit: .sentence); non-Apple (or NLTokenizer yields nothing on non-empty input): falls back to sentencesByDelimiter; empty input → [] +- :78 `sentencesByDelimiter(_ text:) -> [Substring]` — canonical cross-platform reference; splits on `.` `!` `?` `\n`, terminator stays attached to preceding segment; trailing remainder is its own segment; total-coverage guaranteed (join(segments) == text) for non-empty input +- Relocated 2026-05-27 (F16) from CorpusKit/Sources/CorpusKit/Chunker.swift::sentenceSegments + +## INVARIANTS / GOTCHAS + +- fatalError vs UNRESOLVED are TWO DIFFERENT failure modes, do not conflate: FDC.isAvailable==false (missing/broken artifact bundle, a build defect) → fatalError, process terminates. A term with no signature overlap → UNRESOLVED → Anchor(code: "", confidence: 0), a normal, honest, non-fatal result. Never change the UNRESOLVED path to fatalError or vice versa. +- lookup(_:) and lookup(_:recordNovel:) MUST return byte-identical Anchors for the same term; recordNovel only changes whether LatticeLib's shared novel-token pool is written to. Any divergence in the returned Anchor is a bug. +- confidence is a fixed value, not calibrated: EideticLib always emits exactly 0 (UNRESOLVED) or 32 (resolved) from lookup. 16/48/56 are reserved slots in the substrate provenance confidence value set that this library never produces; do not repurpose them here without an explicit design change. +- classifyLatticeCode takes knownCodes as a caller-supplied parameter, not an owned canon — EideticLib deliberately has no opinion on which codes are "known." Do not add a package-level known-code cache; that decision belongs to the caller (federation/version skew is the whole reason pending exists). +- LatticeCodeGrammar.isWellFormed is a SEPARATE implementation from LatticeLib's Code.isWellFormed, not a wrapper. Changing the grammar in one without the other breaks conformance tests silently until those tests run — always update both plus the shared conformance vectors. +- sentences(_:) is NOT cross-platform-identical by design (Apple NLTokenizer vs delimiter fallback) — this is the one place in the EideticLib/LatticeLib family where platform divergence is accepted, because downstream consumers content-address chunks by (sourceID, startOffset, text) under an append-only conflict policy rather than requiring byte-identical segmentation. sentencesByDelimiter(_:) is the only segmentation function under cross-leg parity with Rust. +- Total-coverage invariant for both sentence functions: for non-empty input, concatenating the returned segments must reproduce the input exactly, with no gaps, overlaps, or reordering. Any change to either function must preserve this. +- EideticLib.swift and LatticeCodeState.swift and Segmenter.swift do not call one another — there is no internal pipeline. Do not assume ordering or shared state between the three surfaces. +- Package ships an intentionally empty Sources/EideticLib/Resources/ directory (placeholder so `.process("Resources")` has a target); EideticLib bundles zero reference data — do not add data files here expecting lookup to use them. All classification data lives in LatticeLib's bundle. +- Rust Anchor uses explicit `#[serde(rename = "wikidataQID")]` — auto camelCase would produce "wikidataQid" (lowercase id), breaking Swift/Rust JSON interchange. Do not remove the explicit rename. +- Rust LatticeCodeState wire shape is internally tagged: `{"state": "pending", "code": "999.42"}` — locked by `lattice_code_state_wire_shape_conformance` test; must match Swift's Codable encoding. diff --git a/packages/libs/EideticLib/docs/DETAILS.md b/packages/libs/EideticLib/docs/DETAILS.md new file mode 100644 index 0000000..1a1a362 --- /dev/null +++ b/packages/libs/EideticLib/docs/DETAILS.md @@ -0,0 +1,222 @@ +--- +doc: DETAILS +package: EideticLib +repo: moot-semantics +authored_commit: 021ea704162f86fccea2f030ea9419dacc30a345 +authored_date: 2026-07-04 +sources: + - path: Sources/EideticLib/EideticLib.swift + blob: 1839588dc98279c1eee2786f1974457a5608742b + - path: Sources/EideticLib/LatticeCodeState.swift + blob: 7580261208b77c7e79363a510920c413054264a1 + - path: Sources/EideticLib/Segmenter.swift + blob: c5313d93bf7ca1cbf577da6ffabc9197b5e788e2 +--- + +# EideticLib Details + +This document walks through every source file in the package. Read +`OVERVIEW.md` first for the big picture. The three files are +independent of one another, so they appear here in the order a new +reader would most likely meet them: the lookup surface, the code-state +classifier, and the sentence segmenter. + +## EideticLib.swift + +This file provides the module surface: the `EideticLib` enum, its +`lookup` functions, `classifyLatticeCode`, and the `Anchor` result +type. + +`EideticLib` is a namespace enum, the same pattern LatticeLib uses for +its own module surface. It holds no reference data. The comment at the +top of the file states this explicitly: LatticeLib's FDC runtime owns +and caches the pinned lexicon, frame, and signatures; EideticLib parses +nothing of its own and simply calls through. + +The two `lookup` overloads share one behavior: they never return a +fabricated answer. If `FDC.isAvailable` is false — meaning the FDC +artifacts bundled inside LatticeLib failed to load — both overloads +call `fatalError` and terminate the process. A comment in the code +ties this choice to a specific product mandate: "a sentinel identity +that persists IS a fabricated identity" (Bob's board item 7). The +distinction matters. A term that FDC genuinely cannot classify returns +UNRESOLVED — an empty code, not an error — because that is a true +statement about the term. A missing artifact bundle is not a fact +about any term; it is a broken build, and the correct response is to +stop immediately rather than let every caller silently receive +meaningless answers. + +`version` is the module version string, `"0.1.0"`. Like LatticeLib's +version constant, it exists so a consumer can record which build of +EideticLib produced a given answer. + +`classifyLatticeCode(_:knownCodes:)` sorts a candidate lattice code +string into one of `LatticeCodeState`'s three cases, using +`LatticeCodeGrammar.isWellFormed(_:)` for the shape check and the +caller-supplied `knownCodes` set for the known/pending distinction. +Passing the known-code set in as a parameter, rather than having +EideticLib own a canon, keeps EideticLib decoupled from any particular +caller's view of which codes currently resolve to a label. WHY this +matters: two callers can disagree about which codes are "known" (a +newly synced frame versus a stale one) without EideticLib taking a +side. + +`lookup(_:)` is the default path. It guards on `FDC.isAvailable`, calls +`FDC.encodeAnchor(term)`, and converts the result to an `Anchor`. A +resolved code is reported at confidence 32 (medium), the fixed value +this library always emits, because FDC itself computes no calibrated +score. An UNRESOLVED term becomes an `Anchor` with an empty `code`, a +`nil` `wikidataQID`, and confidence 0 — the honest "nothing matched" +result, never a best-guess fallback. + +`lookup(_:recordNovel:)` is the privacy variant. It produces the exact +same `Anchor` as `lookup(_:)` for the same term, but threads +`recordNovel` through to `FDC.encodeAnchor(_:recordNovel:)`. When +`false`, any novel token encountered while building the term's concept +bag is not written into LatticeLib's shared learning pool. WHY this +exists: the GLK (memory-capture) intake path classifies user-supplied +memory content before deciding whether to store it, and that content +must never leak a single token into a shared, cross-session pool — +even a rejected or sensitive capture must leak nothing, because +classification happens before the write decision, not after. + +`Anchor` is a plain `Equatable`, `Sendable`, `Codable` struct with four +fields: `code` (empty string means UNRESOLVED), `wikidataQID` +(optional), `confidence` (a `UInt8` drawn from the fixed set 0/16/32/48/56 +— null/low/medium/high/verified — of which EideticLib currently only +ever emits 0 or 32), and `dataVersion` (the FDC signatures version that +produced the answer, letting a caller record provenance). The struct's +doc comment notes it is byte-identical to the Rust port's `Anchor` +under JSON encoding, which is what lets a Swift app and a Rust daemon +exchange anchors through a shared store. + +## LatticeCodeState.swift + +This file provides `LatticeCodeState`, the three-state classification +of a candidate lattice code, and `LatticeCodeGrammar`, the grammar +check behind it. + +The file's opening comment frames the problem directly: a lattice code +shown to EideticLib is either malformed (fails the grammar), known +(well-formed and present in the caller's current canon), or pending +(well-formed, but not yet in the caller's canon). WHY a third state is +needed at all: FDC frames are versioned and ship new codes over time, +and MOOTx01 estates can federate, meaning one estate's stored code may +predate another estate's local frame. Rejecting an unrecognized but +well-formed code would lose data permanently; a pending code instead +round-trips intact through storage until a newer frame resolves it. + +`LatticeCodeState` conforms to `Sendable`, `Hashable`, and `Codable`. +The `Codable` conformance is not incidental — it is, per the file's own +comment, "the core invariant of the launch plan's valid-but-unknown-code +requirement": a pending code must survive being written to and read +back from storage without alteration. `rawCode` returns the original +string regardless of which case matched, so a storage layer can +persist the raw value without unpacking the enum first. `isWellFormed` +is `true` for `.known` and `.pending` and `false` only for +`.malformed`, which is useful in tests asserting that pending codes are +still accepted, just not yet resolved. + +`LatticeCodeGrammar` is a second, independent implementation of the +same grammar LatticeLib's `Code.isWellFormed(_:)` enforces: three ASCII +digits, optionally followed by a dot and one to eight further ASCII +digits. WHY it is reimplemented here rather than imported from +LatticeLib: the file's comment explains that EideticLib can validate a +code's shape without loading LatticeLib's frame, lexicon, and +signatures at all — a caller who only wants to know "is this string +shaped like a code" pays none of the cost of the classification engine. +The two implementations are kept in agreement by shared conformance +tests, not by shared code. + +`maxExtensionDigits` is a pinned constant, `8`, matching LatticeLib's +own `Code.maxExtensionDigits`. `isWellFormed(_:)` splits the string on +its first `.`, requires the integer part to be exactly three ASCII +digits, and — if a decimal part is present — requires it to be one to +eight ASCII digits with no other characters. An empty decimal part +(a trailing dot with nothing after it) fails the check. + +## Segmenter.swift + +This file provides `sentences(_:)` and `sentencesByDelimiter(_:)`, both +declared in an extension on the `EideticLib` enum. + +The file's opening comment draws an explicit parallel to LatticeLib's +`WordClassTagger`: that file splits word tagging into a deterministic +reference path (the fast-path table plus HMM) and an accelerated, +Apple-only path (`NLTagger`, opt-in). `Segmenter` applies the same +shape to sentence splitting instead of word tagging. +`sentencesByDelimiter(_:)` is the deterministic reference, +identical on every platform. `sentences(_:)` is the platform-routed +entry point most callers should use: on Apple platforms it defers to +`NLTokenizer`, which correctly handles abbreviations ("Dr."), locale-aware +punctuation, and other edge cases a naive delimiter split gets wrong; on +non-Apple platforms it falls back to `sentencesByDelimiter`. + +WHY platform divergence here is acceptable, when LatticeLib treats +platform divergence in word tagging as unacceptable: the comment cites +the "apple-nlp-accel constitutional constraint (C-2)." Downstream +consumers of sentence segments key each chunk by `(sourceID, +startOffset, text)` — a content address — rather than by chunk index. +Two devices producing slightly different sentence boundaries for the +same document therefore produce a superset of chunks under an +append-only conflict policy, not a pair of conflicting writes that need +reconciling. This is a materially weaker consistency requirement than +FDC's lattice codes, which must be byte-identical across devices for +federated recall to work at all. + +Both functions guarantee total coverage: for non-empty input, every +character appears in exactly one output segment, and joining the +segments reproduces the original text. This holds even in pathological +cases — text that is nothing but delimiters, or text with no delimiter +at all, which returns the entire input as a single segment. Empty +input returns an empty array from both functions. The file's history +comment notes the code was relocated here in 2026-05-27 (a change +tagged F16) from `CorpusKit/Sources/CorpusKit/Chunker.swift`, to +centralize every linguistic-pipeline stage — tokenizing, normalizing, +stemming, tagging, and now segmenting — behind LatticeLib and +EideticLib rather than scattering pieces of the pipeline across +consuming kits. + +`sentences(_:)` builds an `NLTokenizer(unit: .sentence)`, enumerates +sentence-unit ranges over the input, and collects the corresponding +substrings. If the tokenizer produces no segments for non-empty input, +the function falls back to returning the whole input as one segment, +preserving the total-coverage guarantee even in that edge case. + +`sentencesByDelimiter(_:)` walks the string once, splitting after every +`.`, `!`, `?`, or `\n`, and keeping the terminator attached to the +segment that ends there. Any trailing text after the final terminator +becomes its own segment. This is the function the Rust port's +`segmenter::sentences` mirrors exactly, because Rust has no +NLTokenizer-equivalent acceleration to port. + +## Rust Port and Conformance + +The `rust/` directory holds the second leg: `lib.rs` (the crate surface, +re-exporting `lookup` and `classify`), `anchor.rs` (`Anchor`, with an +explicit `#[serde(rename = "wikidataQID")]` so the JSON key matches +Swift's `Codable` output exactly rather than the auto-derived +`wikidataQid`), `lattice_code_state.rs` (`LatticeCodeState`, +`LatticeCodeGrammar`, and `classify_lattice_code`, wire-tagged as +`{"state": ..., "code": ...}` to match Swift's encoding), and +`segmenter.rs` (the delimiter algorithm only — there is no Rust +counterpart to the Apple-only `NLTokenizer` path, by design). + +The Rust crate depends on the sibling `lattice-lib` crate for +`Fdc::encode_anchor` and `Fdc::is_available`, the same relationship +Swift's `EideticLib` target has to the LatticeLib Swift package. +`rust::lookup` panics under the identical condition Swift's `lookup` +treats as a `fatalError`: unavailable FDC artifacts. + +Cross-language conformance runs on two fixtures. +`Tests/SharedVectors/lookup_vectors.json` (schema version 2, 26 vectors) +pins expected FDC codes and Wikidata Q-IDs for a set of inputs, +including edge cases such as punctuation-only text; both +`rust/tests/lookup_conformance_test.rs` and the Swift +`LookupConformanceTests.swift` load the same file and must match it +exactly. `Tests/SharedVectors/word_class_vectors.json` is a second +fixture, shared with LatticeLib's own word-classification conformance +surface, exercised here because the EideticLib test target links +LatticeLib directly. When you change `lookup`, `classifyLatticeCode`, +or either segmentation function, update both legs and rerun both test +suites; the fixtures are the contract. diff --git a/packages/libs/EideticLib/docs/OVERVIEW.md b/packages/libs/EideticLib/docs/OVERVIEW.md new file mode 100644 index 0000000..56250e0 --- /dev/null +++ b/packages/libs/EideticLib/docs/OVERVIEW.md @@ -0,0 +1,148 @@ +--- +doc: OVERVIEW +package: EideticLib +repo: moot-semantics +authored_commit: 021ea704162f86fccea2f030ea9419dacc30a345 +authored_date: 2026-07-04 +sources: + - path: Sources/EideticLib/EideticLib.swift + blob: 1839588dc98279c1eee2786f1974457a5608742b + - path: Sources/EideticLib/LatticeCodeState.swift + blob: 7580261208b77c7e79363a510920c413054264a1 + - path: Sources/EideticLib/Segmenter.swift + blob: c5313d93bf7ca1cbf577da6ffabc9197b5e788e2 +--- + +# EideticLib Overview + +## What This Library Does + +EideticLib is a thin, deterministic front door onto LatticeLib's +classification engine. A caller hands it a term and gets back an +`Anchor`: a lattice code, the dominant Wikidata Q-ID of the term, a +confidence value, and the data version that produced the answer. A +Wikidata Q-ID is a public, language-neutral identifier for one +concept, such as `Q144` for "dog." EideticLib holds no reference data +of its own. Every lookup delegates to `FDC.encodeAnchor`, the runtime +entry point of LatticeLib's Frame-Directed Classification (FDC) +engine. + +Alongside lookup, EideticLib carries two small, self-contained +utilities that other parts of the SDK need: a check for whether a +lattice code string is well-formed and known, and a function that +splits text into sentences. Neither utility depends on the FDC engine. +All three surfaces share one purpose: give callers a deterministic +answer without asking them to understand LatticeLib's internals. + +## The Problem It Solves + +LatticeLib's `FDC` type is a powerful but low-level engine: it owns +pinned artifacts, exposes several scoring modes, and returns raw +codes and ancestry chains. Most callers do not need any of that. They +need one function: give me a term, tell me what it is about. EideticLib +is that function. It fixes a confidence scale, wraps the result in a +stable `Codable` type, and enforces a strict failure policy so callers +never have to guess whether an empty answer means "no data loaded" or +"nothing matched." + +A second, related problem is memory content privacy. Some callers +classify user-supplied memory content — private text that must never +leak into LatticeLib's shared novel-token learning pool, even by +accident. EideticLib exposes a `recordNovel` parameter that turns that +accumulation off while returning the exact same anchor, so a capture +path can classify text safely without a second, divergent code path. + +A third problem concerns the lattice code canon itself. FDC frames are +versioned, and federated devices can be running against different +frame versions at different times. A device might receive a lattice +code from another estate — one user's complete memory store — that its +own local frame does not yet recognize. Discarding that code would +lose information; guessing at its meaning would violate the classifier's +honesty. `classifyLatticeCode` sorts a code string into one of three +states — malformed, known, or pending — so storage layers can accept and +round-trip a code they cannot yet resolve, and settle it later when a +newer frame ships. + +The fourth problem is splitting text into sentences before other +pipeline stages run. Sentence boundaries look simple until abbreviations +("Dr.", "e.g.") and locale-specific punctuation appear. `Segmenter` +answers this the same way LatticeLib answers tokenization: a fast, +higher-accuracy path on Apple platforms, and a plain, deterministic +fallback everywhere else. + +## How It Works + +`EideticLib.lookup(_:)` first checks `FDC.isAvailable`. If the pinned +FDC artifacts failed to load, the function terminates the process with +`fatalError` rather than returning an empty or fabricated answer. A +failed load means the binary shipped without its required data, which +is a build defect, not a condition any caller should recover from at +runtime. This is a deliberate stance: MOOTx01's provenance rules treat +a persistent placeholder identity as a fabricated identity, so +EideticLib refuses to manufacture one. + +Given available artifacts, `lookup` calls `FDC.encodeAnchor(term)`, +which returns a lattice code and a Wikidata Q-ID, or `nil` for the +code when the term is UNRESOLVED — the honest "no answer" result that +LatticeLib returns rather than guess. EideticLib packages a resolved +result as an `Anchor` with a fixed confidence of 32 (medium), because +FDC produces no calibrated confidence score of its own. An UNRESOLVED +term becomes an `Anchor` with an empty code, no Q-ID, and confidence 0. +`lookup(_:recordNovel:)` is the same function with the pool-recording +side effect suppressed, for callers classifying private memory content. + +`classifyLatticeCode(_:knownCodes:)` runs independently of the FDC +engine. It checks the code string against a small grammar — three +digits, an optional decimal extension of up to eight digits — using +`LatticeCodeGrammar`, a self-contained rule set that mirrors LatticeLib's +own `Code.isWellFormed(_:)`. A code that fails the grammar is +`.malformed`. A well-formed code present in the caller-supplied +`knownCodes` set is `.known`. A well-formed code absent from that set +is `.pending`: accepted, storable, and round-tripped intact rather than +rejected. + +`EideticLib.sentences(_:)` splits text into sentence substrings. On +Apple platforms it uses `NLTokenizer`, which understands abbreviations +and locale-aware boundaries. On every other platform, and whenever +strict cross-platform identity is required, `sentencesByDelimiter(_:)` +splits on periods, exclamation points, question marks, and newlines, +keeping the terminator attached to the segment it ends. Both functions +guarantee total coverage: every character of the input appears in +exactly one output segment, so a caller can always reconstruct the +original text by joining the segments back together. + +## How the Pieces Fit + +EideticLib is not a pipeline the way LatticeLib is. Its three surfaces +— lookup, code-state classification, and sentence segmentation — do not +call one another. Each is an independent, narrow entry point that +answers one question, and each has its own relationship to the rest of +the SDK: `lookup` crosses into LatticeLib's FDC engine, code +classification is a pure, dependency-free grammar check, and sentence +segmentation crosses into Apple's NaturalLanguage framework only on +Apple platforms. + +That fan-out shape is also EideticLib's value: it is a single, stable +package that other libraries and kits import for a handful of small, +unrelated conveniences, so they do not each have to depend on +LatticeLib, NaturalLanguage, and a code-grammar checker separately. +CorpusKit, for example, imports EideticLib only for `sentences(_:)`, +to segment memory text into chunks before further processing. + +The library keeps the same two-language guarantee as LatticeLib. A +Swift leg serves Apple platforms, and a Rust leg in `rust/` mirrors +`lookup`, `classifyLatticeCode`, and `sentencesByDelimiter` exactly. +Shared conformance vectors in `Tests/SharedVectors/lookup_vectors.json` +gate every release so both legs agree byte for byte on every case in +the fixture. + +## What Ships in the Package + +The package ships three Swift source files under `Sources/EideticLib/`, +an intentionally empty `Resources/` directory (EideticLib carries no +reference data of its own; LatticeLib owns the pinned artifacts that +`lookup` depends on), and the Rust port under `rust/`. The Rust crate, +`eidetic-lib`, depends on the sibling `lattice-lib` crate the same way +the Swift target depends on the LatticeLib Swift package. Cross-language +conformance is enforced by `Tests/SharedVectors/lookup_vectors.json` +(26 vectors) and the integration tests in `rust/tests/`. From 508897c5695efc411b76e15b211b2cfd86a28db7 Mon Sep 17 00:00:00 2001 From: Fable Date: Sat, 4 Jul 2026 22:15:01 -0500 Subject: [PATCH 3/3] =?UTF-8?q?docs(sdk):=20rewrite=20prose=20to=20the=20f?= =?UTF-8?q?our=20immutable=20laws=20=E2=80=94=20FK=20<=3D=208.0,=20no=20em?= =?UTF-8?q?-dashes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full rewrite of OVERVIEW.md and DETAILS.md per package: Turabian/Chicago, reading level 8.0 or less, no sentence over 2 commas outside simple lists, no 'not X, but Y' or 'it is not X, it is Y' forms. AGENT_MAP and topology.svg scrubbed of em-dashes only. Front matter unchanged. Gated by scripts/docs-style-lint.py (mootx01-ee): 65/65 pass; docs-scrub clean. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Pzkqg3bpbbhfuWRsYomJ3o --- .../libs/AriaLexiconLib/docs/AGENT_MAP.md | 76 +- packages/libs/AriaLexiconLib/docs/DETAILS.md | 288 ++++--- packages/libs/AriaLexiconLib/docs/OVERVIEW.md | 180 ++-- packages/libs/EideticLib/docs/AGENT_MAP.md | 64 +- packages/libs/EideticLib/docs/DETAILS.md | 331 ++++---- packages/libs/EideticLib/docs/OVERVIEW.md | 235 +++--- packages/libs/LatticeLib/docs/AGENT_MAP.md | 156 ++-- packages/libs/LatticeLib/docs/DETAILS.md | 794 +++++++++--------- packages/libs/LatticeLib/docs/OVERVIEW.md | 182 ++-- 9 files changed, 1196 insertions(+), 1110 deletions(-) diff --git a/packages/libs/AriaLexiconLib/docs/AGENT_MAP.md b/packages/libs/AriaLexiconLib/docs/AGENT_MAP.md index 3fe1f42..3b15a91 100644 --- a/packages/libs/AriaLexiconLib/docs/AGENT_MAP.md +++ b/packages/libs/AriaLexiconLib/docs/AGENT_MAP.md @@ -17,69 +17,69 @@ sources: blob: 7ecf10b17758b23b7652c47879c2b353d28f5965 --- -# AGENT_MAP — AriaLexiconLib +# AGENT_MAP | AriaLexiconLib -PURPOSE: the reified ARIA grammar — one noun (Drawer), nine verbs, four adjective categories, and the verb-noun acceptance matrix, as pure data. No behavior, no I/O, no dependencies. Canonical prose statement lives outside this package in ARIA_LEXICON.md; this module is that statement made first-class in code so it can be conformance-checked. +PURPOSE: the reified ARIA grammar, one noun (Drawer), nine verbs, four adjective categories, and the verb-noun acceptance matrix, as pure data. No behavior, no I/O, no dependencies. Canonical prose statement lives outside this package in ARIA_LEXICON.md; this module is that statement made first-class in code so it can be conformance-checked. -DEPS: imports nothing (zero-dependency, foundational leaf package; Swift stdlib only). Imported by: none currently declare a package/`import` dependency within this SDK checkout. LocusKit (moot-memory/packages/kits/LocusKit) references `AriaLexiconLib`'s types by name in doc comments (Acceptance.swift, Verb.swift, Noun.swift) as the conceptual source of truth for its own Frames/Association/Proposal/LearnedReference/EstateVerbs types, but does NOT declare it as a Package.swift dependency or `import` it as of this commit — treat as documentation-level, not compiled, coupling. Package header names VectorKit and CorpusKit as intended future conformers. Rust port: `rust/` (crate `aria-lexicon-lib`, single file `src/lib.rs`) mirrors every type and the acceptance matrix by hand; its own `#[cfg(test)]` module is the conformance gate (wire-string + matrix parity vs. this Swift package). +DEPS: imports nothing (zero-dependency, foundational leaf package; Swift stdlib only). Imported by: none currently declare a package/`import` dependency within this SDK checkout. LocusKit (moot-memory/packages/kits/LocusKit) references `AriaLexiconLib`'s types by name in doc comments (Acceptance.swift, Verb.swift, Noun.swift) as the conceptual source of truth for its own Frames/Association/Proposal/LearnedReference/EstateVerbs types, but does NOT declare it as a Package.swift dependency or `import` it as of this commit: treat as documentation-level, not compiled, coupling. Package header names VectorKit and CorpusKit as intended future conformers. Rust port: `rust/` (crate `aria-lexicon-lib`, single file `src/lib.rs`) mirrors every type and the acceptance matrix by hand; its own `#[cfg(test)]` module is the conformance gate (wire-string + matrix parity vs. this Swift package). ENTRY POINTS (most callers need only these): -- Acceptance.swift:36 `Acceptance.accepts(_ noun: Noun, _ verb: Verb) -> Bool` — the one call most callers want: "is this operation allowed on this shape" -- Acceptance.swift:14 `Acceptance.verbs(for noun: Noun) -> Set` — full accepted-verb set for a shape -- AriaLexiconLib.swift:15 `AriaLexiconLib.grammar` — the one-sentence rule as a displayable string +- Acceptance.swift:36 `Acceptance.accepts(_ noun: Noun, _ verb: Verb) -> Bool` | the one call most callers want: "is this operation allowed on this shape" +- Acceptance.swift:14 `Acceptance.verbs(for noun: Noun) -> Set` | full accepted-verb set for a shape +- AriaLexiconLib.swift:15 `AriaLexiconLib.grammar` | the one-sentence rule as a displayable string ## Symbol Table -### Module — AriaLexiconLib.swift -- :13 `enum AriaLexiconLib` — empty namespace enum -- :15 `static let grammar: String` — "Every call is one verb applied to a noun, optionally constrained by adjectives." +### Module | AriaLexiconLib.swift +- :13 `enum AriaLexiconLib` | empty namespace enum +- :15 `static let grammar: String` | "Every call is one verb applied to a noun, optionally constrained by adjectives." -### Noun — Noun.swift -- :14 `enum Noun: String, CaseIterable, Sendable, Codable` — 8 storage shapes; rawValue IS the wire string (lowerCamelCase case names, no explicit `=`) +### Noun | Noun.swift +- :14 `enum Noun: String, CaseIterable, Sendable, Codable` | 8 storage shapes; rawValue IS the wire string (lowerCamelCase case names, no explicit `=`) - :15–22 cases: `drawer`, `tunnel`, `kgFact`, `vector`, `diaryEntry`, `proposal`, `association`, `learnedReference` (declaration order = Rust `Noun::ALL` order) -- :27 `static let primary: Noun = .drawer` — the one true noun of the language -- :31 `var role: NounRole` — computed; drawer→primary, kgFact/vector→rung, tunnel/diaryEntry/association→structure, proposal/learnedReference→product -- :45 `enum NounRole: String, CaseIterable, Sendable, Codable` — :47 `primary`, :49 `rung`, :51 `structure`, :53 `product` +- :27 `static let primary: Noun = .drawer` | the one true noun of the language +- :31 `var role: NounRole` | computed; drawer→primary, kgFact/vector→rung, tunnel/diaryEntry/association→structure, proposal/learnedReference→product +- :45 `enum NounRole: String, CaseIterable, Sendable, Codable` | :47 `primary`, :49 `rung`, :51 `structure`, :53 `product` -### Verb — Verb.swift -- :9 `enum Verb: String, CaseIterable, Sendable, Codable` — 9 actions, count fixed by spec invariant I-7 +### Verb | Verb.swift +- :9 `enum Verb: String, CaseIterable, Sendable, Codable` | 9 actions, count fixed by spec invariant I-7 - :10–18 cases: `capture`, `reanchor`, `mutate`, `withdraw`, `expunge`, `recall`, `propose`, `associate`, `learn` (declaration order = Rust `Verb::ALL` order) -- :21 `var flow: Flow` — computed; capture/reanchor/mutate/withdraw/expunge/recall→callerDriven, propose/associate→substrateDriven, learn→groundingDriven -- :37 `enum Flow: String, CaseIterable, Sendable, Codable` — :38 `callerDriven`, :39 `substrateDriven`, :40 `groundingDriven` +- :21 `var flow: Flow` | computed; capture/reanchor/mutate/withdraw/expunge/recall→callerDriven, propose/associate→substrateDriven, learn→groundingDriven +- :37 `enum Flow: String, CaseIterable, Sendable, Codable` | :38 `callerDriven`, :39 `substrateDriven`, :40 `groundingDriven` -### Adjective — Adjective.swift -- :13 `enum Adjective: String, CaseIterable, Sendable, Codable` — 4 cross-noun categories, count fixed by spec invariant I-8 -- :17 `state` — lifecycle position (active/pending/contested/superseded/decayed/withdrawn/expired/rejected/accepted/tombstoned — VALUES live in LocusKit, not here) -- :20 `trust` — provenance (verbatim/observed/imported/proposed/derived/canonical — values in LocusKit) -- :23 `sensitivity` — exposure level (normal/elevated/restricted/secret — values in LocusKit) -- :26 `exportability` — perimeter crossing (private/public — values in LocusKit) +### Adjective | Adjective.swift +- :13 `enum Adjective: String, CaseIterable, Sendable, Codable` | 4 cross-noun categories, count fixed by spec invariant I-8 +- :17 `state` | lifecycle position (active/pending/contested/superseded/decayed/withdrawn/expired/rejected/accepted/tombstoned: VALUES live in LocusKit, not here) +- :20 `trust` | provenance (verbatim/observed/imported/proposed/derived/canonical: values in LocusKit) +- :23 `sensitivity` | exposure level (normal/elevated/restricted/secret: values in LocusKit) +- :26 `exportability` | perimeter crossing (private/public: values in LocusKit) -### Acceptance matrix — Acceptance.swift -- :10 `enum Acceptance` — namespace for the matrix -- :14 `static func verbs(for noun: Noun) -> Set` — hand-authored per-noun set; SEE ENTRY POINTS +### Acceptance matrix | Acceptance.swift +- :10 `enum Acceptance` | namespace for the matrix +- :14 `static func verbs(for noun: Noun) -> Set` | hand-authored per-noun set; SEE ENTRY POINTS - :16 drawer → {capture, reanchor, mutate, withdraw, expunge, recall} (6, fullest row) - :18 tunnel → {capture, mutate, withdraw, expunge, recall} (5, no reanchor) - :20 kgFact → {mutate, withdraw, expunge, recall} (4, no capture/reanchor) -- :22 vector → {} (0 — substrate-managed, never verb-addressable directly) +- :22 vector → {} (0: substrate-managed, never verb-addressable directly) - :24 diaryEntry → {recall} (1, read-only) - :26 proposal → {propose, mutate, withdraw, expunge, recall} (5; role is "product" but DOES accept propose) - :28 association → {associate, mutate, expunge, recall} (4; no withdraw) - :30 learnedReference → {learn, mutate, withdraw, expunge, recall} (5) -- :36 `static func accepts(_ noun: Noun, _ verb: Verb) -> Bool` — `verbs(for:).contains(verb)`; SEE ENTRY POINTS +- :36 `static func accepts(_ noun: Noun, _ verb: Verb) -> Bool` | `verbs(for:).contains(verb)`; SEE ENTRY POINTS -### Rust port — rust/src/lib.rs (mirror, not a Swift symbol) -- `Noun`/`NounRole`/`Verb`/`Flow`/`Adjective` — PascalCase enum variants + `as_str()` producing the identical lowerCamelCase wire strings as the Swift rawValues -- `accepted_verbs(Noun) -> Vec` / `accepts(Noun, Verb) -> bool` — same matrix, hand-mirrored -- `#[cfg(test)] mod tests` — conformance gate: verb/adjective counts, full matrix cell-by-cell, wire-string parity, serde round-trip +### Rust port | rust/src/lib.rs (mirror, not a Swift symbol) +- `Noun`/`NounRole`/`Verb`/`Flow`/`Adjective` | PascalCase enum variants + `as_str()` producing the identical lowerCamelCase wire strings as the Swift rawValues +- `accepted_verbs(Noun) -> Vec` / `accepts(Noun, Verb) -> bool` | same matrix, hand-mirrored +- `#[cfg(test)] mod tests` | conformance gate: verb/adjective counts, full matrix cell-by-cell, wire-string parity, serde round-trip ## INVARIANTS / GOTCHAS - NO BEHAVIOR. This package is pure data (enums, one constant string, one lookup table). Do not add I/O, logging, or business logic here; new domain operations compose the nine existing verbs instead of extending the vocabulary. -- Verb count is pinned at 9 (spec invariant I-7). Adjective category count is pinned at 4 (spec invariant I-8). Both are spec-level invariants, not incidental — changing either is a specification amendment, not a routine edit. -- `Acceptance`'s matrix is HAND-AUTHORED, not derived from `Noun.role` or `Verb.flow`. Do not "simplify" it by deriving from role/flow — the Proposal row (role=`product`, yet accepts `.propose`) and the Association row (role=`structure`, accepts `.associate` but not `.propose`) are real exceptions that a derived rule would get wrong. -- `Adjective` names categories only. The value lists inside each category (the 10 lifecycle states, 6 trust levels, 4 sensitivity levels, 2 exportability levels) live in LocusKit (moot-memory), not here — by design, so the two never fork by being defined twice. Do not add value enums to this file. -- Zero dependencies is load-bearing: this package must stay leaf-level (imports nothing) so any kit — including ones layered far apart, like LocusKit, VectorKit, and CorpusKit — can depend on it without a cycle. +- Verb count is pinned at 9 (spec invariant I-7). Adjective category count is pinned at 4 (spec invariant I-8). Both are spec-level invariants, not incidental: changing either is a specification amendment, not a routine edit. +- `Acceptance`'s matrix is HAND-AUTHORED, not derived from `Noun.role` or `Verb.flow`. Do not "simplify" it by deriving from role/flow: the Proposal row (role=`product`, yet accepts `.propose`) and the Association row (role=`structure`, accepts `.associate` but not `.propose`) are real exceptions that a derived rule would get wrong. +- `Adjective` names categories only. The value lists inside each category (the 10 lifecycle states, 6 trust levels, 4 sensitivity levels, 2 exportability levels) live in LocusKit (moot-memory), not here: by design, so the two never fork by being defined twice. Do not add value enums to this file. +- Zero dependencies is load-bearing: this package must stay leaf-level (imports nothing) so any kit, including ones layered far apart, like LocusKit, VectorKit, and CorpusKit, can depend on it without a cycle. - Swift enum rawValues ARE the wire strings (case names are already lowerCamelCase; no explicit `= "..."` needed). Rust mirrors with PascalCase variants + `as_str()`/`serde(rename_all = "camelCase")` to reproduce the identical strings. Any edit to a Swift case name, or to Rust's `as_str()`/serde attributes, that the other side does not mirror breaks wire conformance silently unless the Rust test suite is rerun. - Two independent implementations, no shared codegen: editing `Acceptance.swift`, `Noun.swift`, `Verb.swift`, or `Adjective.swift` requires a matching hand-edit to `rust/src/lib.rs` (types, `as_str()`, and `accepted_verbs`/`accepts`) plus a rerun of the Rust `#[cfg(test)]` suite. There is no build step that generates one from the other. - Canonical source-of-truth order: the prose document ARIA_LEXICON.md (external to this package, not present in this repo checkout) states the grammar; this Swift package is the code reification of that document; the Rust crate is a conformance-gated mirror of this package. When in doubt about intent, ARIA_LEXICON.md wins. -- No compiled consumer exists yet within this SDK checkout — verify current importers before assuming call sites exist. LocusKit's references are doc-comment citations, not `import` statements or Package.swift dependencies, as of this commit. +- No compiled consumer exists yet within this SDK checkout: verify current importers before assuming call sites exist. LocusKit's references are doc-comment citations, not `import` statements or Package.swift dependencies, as of this commit. diff --git a/packages/libs/AriaLexiconLib/docs/DETAILS.md b/packages/libs/AriaLexiconLib/docs/DETAILS.md index 63c2832..55cb1cc 100644 --- a/packages/libs/AriaLexiconLib/docs/DETAILS.md +++ b/packages/libs/AriaLexiconLib/docs/DETAILS.md @@ -21,176 +21,186 @@ sources: This document walks through every source file in the package. Read `OVERVIEW.md` first for the big picture. Files appear here in the order a -reader builds understanding: the module surface and its one-sentence -grammar, then the noun, then the verb, then the adjective categories, and -finally the acceptance matrix that relates the noun and the verb. +reader builds understanding. First comes the module surface and its +one-sentence grammar. Then comes the noun. Then comes the verb. Then comes +the adjective categories. Last comes the acceptance matrix that relates +the noun and the verb. ## AriaLexiconLib.swift -This file provides the module surface: the public `AriaLexiconLib` enum and -the one-sentence statement of the grammar. +This file provides the module surface. It holds the public +`AriaLexiconLib` enum and the one-sentence statement of the grammar. -Swift has no module-level functions or constants, so a library that wants a +Swift has no module-level functions or constants. A library that wants a single well-known place for a top-level fact uses an empty enum as a -namespace. `AriaLexiconLib` is that namespace; it exists only to hold -`grammar`. +namespace instead. `AriaLexiconLib` is that namespace. It exists only to +hold `grammar`. `AriaLexiconLib.grammar` is a constant string: "Every call is one verb applied to a noun, optionally constrained by adjectives." This matters -because the rule is meant to be read by people, not only enforced by types. -A consumer can print this string in a log, a debugging tool, or a developer -console, and it will always match the rule the rest of the package encodes, -because it lives in the same file set and the same version as the types -that implement it. +because people need to read the rule, not only have types enforce it. A +consumer can print this string in a log, a debugging tool, or a developer +console. It will always match the rule the rest of the package encodes. +It lives in the same file set and the same version as the types that +implement it. ## Noun.swift -This file provides `Noun`, the eight storage shapes the substrate persists, -and `NounRole`, how each shape relates to the one true noun of the -language, the Drawer. +This file provides `Noun`, the eight storage shapes the substrate +persists. It also provides `NounRole`, how each shape relates to the one +true noun of the language, the Drawer. -A Drawer is the atomic unit of memory: one row of stored content. The other -seven cases in `Noun` — `tunnel`, `kgFact`, `vector`, `diaryEntry`, -`proposal`, `association`, and `learnedReference` — are not separate things -a caller thinks about the way it thinks about a Drawer. Each is either a -different view onto Drawer content or a record of something a verb did. -The file's opening comment is explicit that the architecture specification -calls all of these "nouns" loosely, as a storage taxonomy, but only Drawer -is a noun in ARIA's grammar sense; the rest are facets or residue. +A Drawer is the atomic unit of memory: one row of stored content. The +other seven cases in `Noun` are `tunnel`, `kgFact`, `vector`, +`diaryEntry`, `proposal`, `association`, and `learnedReference`. Each of +these seven is not a separate thing a caller thinks about the way it +thinks about a Drawer. Each is either a different view onto Drawer +content, or a record of something a verb did. The file's opening comment +is explicit. The architecture specification calls all of these "nouns" +loosely, as a storage taxonomy. Only Drawer is a noun in ARIA's grammar +sense. The rest are facets or residue. `Noun.primary` is a static constant equal to `.drawer`. It gives code a -named way to ask "which case is the real noun" without hard-coding the -case name at every call site, so a future reader searching for "primary" +named way to ask which case is the real noun. It avoids hard-coding the +case name at every call site. A future reader searching for "primary" finds the one place that decision is made. -`Noun.role` is a computed property that classifies each case into one of -four `NounRole` values: `.primary` for the Drawer itself, `.rung` for a -representation of a Drawer's content (`kgFact` and `vector`), `.structure` -for an edge or event about Drawers (`tunnel`, `diaryEntry`, `association`), -and `.product` for what a verb leaves behind (`proposal`, -`learnedReference`). This grouping matters because a caller reasoning about -a storage shape — for example, deciding whether it is safe to delete — -often cares about the shape's role more than its exact case name; a rung -can usually be rebuilt from its source Drawer, while a product generally -cannot. - -`NounRole` itself is a four-case, string-backed enum -(`primary`, `rung`, `structure`, `product`). Being string-backed means it -serializes to a stable, human-readable form rather than a raw integer, which -matters for any log or wire format that includes it. +`Noun.role` is a computed property. It classifies each case into one of +four `NounRole` values. `.primary` covers the Drawer itself. `.rung` +covers a representation of a Drawer's content: `kgFact` and `vector`. +`.structure` covers an edge or event about Drawers: `tunnel`, +`diaryEntry`, and `association`. `.product` covers what a verb leaves +behind: `proposal` and `learnedReference`. This grouping matters because a +caller often cares about a shape's role more than its exact case name. +For example, a caller deciding whether it is safe to delete something +cares about role. A rung can usually be rebuilt from its source Drawer. A +product generally cannot. + +`NounRole` itself is a four-case, string-backed enum: `primary`, `rung`, +`structure`, `product`. Being string-backed means it serializes to a +stable, human-readable form, rather than a raw integer. This matters for +any log or wire format that includes it. ## Verb.swift -This file provides `Verb`, the nine actions the substrate supports, and -`Flow`, who initiates each one. - -The nine verbs are `capture`, `reanchor`, `mutate`, `withdraw`, `expunge`, -`recall`, `propose`, `associate`, and `learn`. The file's opening comment -states the design rule directly: this count is fixed by specification -invariant I-7, and a new kind of domain operation is expected to compose -these nine rather than add a tenth. Keeping the verb set closed is what -lets every kit share one small vocabulary instead of each kit growing its -own private verbs over time. - -`Verb.flow` is a computed property returning a `Flow` value that answers -"who calls this verb." Six verbs — `capture`, `reanchor`, `mutate`, -`withdraw`, `expunge`, and `recall` — are `.callerDriven`: an application -calls them directly, synchronously, when it wants something to happen. Two -verbs, `propose` and `associate`, are `.substrateDriven`: they are emitted -by the system's own background processes on their own schedule, not called -directly by application code. One verb, `learn`, is `.groundingDriven`: it -is the verb that brings authoritative outside reference material into the -system, distinct from both a direct caller action and an internal -background signal. Separating these three flows matters for anyone writing -code against the substrate, because a caller-driven verb needs a call site, -while a substrate-driven or grounding-driven verb needs a listener instead. - -`Flow` is a three-case, string-backed enum (`callerDriven`, -`substrateDriven`, `groundingDriven`). +This file provides `Verb`, the nine actions the substrate supports. It +also provides `Flow`, who initiates each one. + +The nine verbs are `capture`, `reanchor`, `mutate`, `withdraw`, +`expunge`, `recall`, `propose`, `associate`, and `learn`. The file's +opening comment states the design rule directly. This count is fixed by +specification invariant I-7. A new kind of domain operation is expected +to compose these nine, rather than add a tenth. Keeping the verb set +closed lets every kit share one small vocabulary. Otherwise each kit +would grow its own private verbs over time. + +`Verb.flow` is a computed property. It returns a `Flow` value that +answers who calls this verb. Six verbs are `.callerDriven`: `capture`, +`reanchor`, `mutate`, `withdraw`, `expunge`, and `recall`. An application +calls each of these directly and synchronously, when it wants something +to happen. Two verbs are `.substrateDriven`: `propose` and `associate`. +The system's own background processes emit these on their own schedule. +No application code calls them directly. One verb, `learn`, is +`.groundingDriven`. It is the verb that brings authoritative outside +reference material into the system. It differs from both a direct caller +action and an internal background signal. Separating these three flows +matters for anyone writing code against the substrate. A caller-driven +verb needs a call site. A substrate-driven or grounding-driven verb needs +a listener instead. + +`Flow` is a three-case, string-backed enum: `callerDriven`, +`substrateDriven`, `groundingDriven`. ## Adjective.swift This file provides `Adjective`, the four categories that describe any row -of memory regardless of which noun or verb produced it. - -The four categories are `state` (where the row sits in its lifecycle, such -as active, pending, or superseded), `trust` (how the content was -established, such as verbatim, observed, or derived), `sensitivity` (how -exposed the content may be), and `exportability` (whether the content may -leave the access perimeter — private or public). The count of four is -fixed by specification invariant I-8, the same way the verb count is fixed -by I-7. - -The file's comment explains an important design boundary: `Adjective` only -names the four categories, not the specific values inside each one. Every -row carries some value in each category no matter its storage shape, which -is why the categories are described as "cross-noun." But how those values -are packed and represented — the actual list of lifecycle states, for -example — is treated as a bitmap-layout detail, and that detail is reified -in a separate kit, LocusKit, instead of here. Keeping the category names in -one place and the value lists in another means the two cannot silently -drift apart by being maintained twice; only one of them needs to change -when a new value is added to a category. +of memory, regardless of which noun or verb produced it. + +The four categories are as follows. `state` describes where the row sits +in its lifecycle, such as active, pending, or superseded. `trust` +describes how the content was established, such as verbatim, observed, or +derived. `sensitivity` describes how exposed the content may be. +`exportability` describes whether the content may leave the access +perimeter, such as private or public. The count of four is fixed by +specification invariant I-8, the same way the verb count is fixed by I-7. + +The file's comment explains an important design boundary. `Adjective` +only names the four categories. It does not name the specific values +inside each one. Every row carries some value in each category, no +matter its storage shape. This is why the categories are described as +"cross-noun." How those values are packed and represented is a different +matter. The actual list of lifecycle states, for example, is treated as +a bitmap-layout detail. That detail is reified in a separate kit, +LocusKit, instead of here. Keeping the category names in one place and +the value lists in another means the two cannot silently drift apart by +being maintained twice. Only one of them needs to change when a new +value is added to a category. ## Acceptance.swift -This file provides `Acceptance`, the verb-noun acceptance matrix: the one -place the package states which of the nine verbs make sense for each of -the eight storage shapes. +This file provides `Acceptance`, the verb-noun acceptance matrix. It is +the one place the package states which of the nine verbs make sense for +each of the eight storage shapes. -The matrix is hand-authored data, not derived from `Noun.role` or -`Verb.flow`. For example, `Proposal`'s role is "product," yet it does -accept the verb that creates it, `propose`, along with the ordinary -lifecycle verbs (`mutate`, `withdraw`, `expunge`, `recall`). The file's -opening comment frames this precisely: in ARIA's grammar the matrix reads -as "which actions apply to this shape," not as a competition between -nouns, and it is expressed as data specifically so that a conformance test -can check the Swift and Rust implementations agree cell by cell. +The matrix is hand-authored data. No other rule in the package, neither +`Noun.role` nor `Verb.flow`, derives it. For example, `Proposal`'s role +is `product`. Yet it accepts the verb that creates it, `propose`. It also +accepts the ordinary lifecycle verbs: `mutate`, `withdraw`, `expunge`, +and `recall`. The file's opening comment frames this precisely. In ARIA's +grammar the matrix reads as which actions apply to this shape. It is not +a competition between nouns. It is expressed as data specifically so a +conformance test can check the Swift and Rust implementations agree cell +by cell. `Acceptance.verbs(for:)` takes a `Noun` and returns the `Set` it -accepts. A Drawer accepts six verbs (every caller-driven verb except none -— it is the fullest row in the matrix). A Tunnel accepts five, omitting -`reanchor`. A knowledge-graph fact accepts four, omitting both `capture` -and `reanchor`, because a fact is not captured directly the way a Drawer -or Tunnel is. A Vector accepts none: the comment explains that vectors are -substrate-managed, meaning an internal process derives and maintains them, -so no caller ever addresses one with a verb directly. A diary entry accepts -only `recall`, since it is a read-only record. A Proposal accepts -`propose` plus the lifecycle verbs, and an Association accepts `associate` -plus most of the lifecycle verbs (it omits `withdraw`). A learned reference -accepts `learn` plus the lifecycle verbs. This function matters because it -is the only place any of these facts is stated; a caller that needs to know -"can I do X to a Y" asks this function instead of re-deriving the answer -from role or flow, which would sometimes give the wrong answer, as the +accepts. A Drawer accepts six verbs. It is the fullest row in the +matrix, since every caller-driven verb applies to it. A Tunnel accepts +five verbs. It omits `reanchor`. A knowledge-graph fact accepts four +verbs. It omits both `capture` and `reanchor`. A fact is not captured +directly the way a Drawer or a Tunnel is. A Vector accepts no verbs. The +comment explains that vectors are substrate-managed. An internal process +derives and maintains them. No caller ever addresses one with a verb +directly. A diary entry accepts only `recall`, since it is a read-only +record. A Proposal accepts `propose` plus the lifecycle verbs. An +Association accepts `associate` plus most of the lifecycle verbs. It +omits `withdraw`. A learned reference accepts `learn` plus the lifecycle +verbs. This function matters because it is the only place any of these +facts is stated. A caller that needs to know whether it can do something +to a shape asks this function. It does not re-derive the answer from role +or flow. Re-deriving would sometimes give the wrong answer, as the Proposal example shows. -`Acceptance.accepts(_:_:)` takes a noun and a verb and returns whether the -verb is a member of that noun's accepted set. It is a thin convenience over -`verbs(for:)`, provided because most call sites want a yes-or-no answer for -one specific pair rather than the whole set, and spelling that check inline -at every call site would risk a typo that a shared function cannot make. +`Acceptance.accepts(_:_:)` takes a noun and a verb. It returns whether the +verb is a member of that noun's accepted set. It is a thin convenience +over `verbs(for:)`. Most call sites want a yes-or-no answer for one +specific pair, rather than the whole set. Spelling that check inline at +every call site would risk a typo that a shared function cannot make. ## Rust Port and Conformance -The `rust/` directory contains the second leg of the library, `lib.rs`, a -single-file crate named `aria-lexicon-lib`. It mirrors every Swift type: -`Noun`, `NounRole`, `Verb`, `Flow`, and `Adjective` as Rust enums, plus -`accepted_verbs` and `accepts` as the Rust equivalents of -`Acceptance.verbs(for:)` and `Acceptance.accepts(_:_:)`. Swift's string-backed -enum cases are already lowerCamelCase and serve as their own wire strings. -Rust instead declares PascalCase variants by convention and adds an -`as_str()` method, plus `serde` attributes, that reproduce the identical -lowerCamelCase wire strings for JSON — for example, `Noun::KgFact.as_str()` -returns `"kgFact"`, matching Swift's `Noun.kgFact.rawValue` exactly. - -The crate's own test module is the conformance suite: it checks the verb -count is nine and the adjective count is four, checks every wire string -against its expected value, checks that `serde` serialization matches -`as_str()` for every case, and re-states the entire acceptance matrix cell -by cell, including the harder-to-guess cases such as a Proposal not -accepting `associate` and an Association not accepting `propose`. Because -the two languages cannot share source code for this vocabulary, this test -module is the only mechanism that catches the two ports drifting apart: -when you change `Acceptance.swift`, `Noun.swift`, `Verb.swift`, or -`Adjective.swift`, you must make the matching edit in `rust/src/lib.rs` and -rerun both test suites, or a silent disagreement ships. +The `rust/` directory contains the second leg of the library: `lib.rs`, a +single-file crate named `aria-lexicon-lib`. It mirrors every Swift type +as Rust enums: `Noun`, `NounRole`, `Verb`, `Flow`, and `Adjective`. It +also adds `accepted_verbs` and `accepts` as the Rust equivalents of +`Acceptance.verbs(for:)` and `Acceptance.accepts(_:_:)`. Swift's +string-backed enum cases are already lowerCamelCase. They serve as their +own wire strings. Rust instead declares PascalCase variants by +convention. It adds an `as_str()` method, plus `serde` attributes, that +reproduce the identical lowerCamelCase wire strings for JSON. For +example, `Noun::KgFact.as_str()` returns `"kgFact"`. This exactly matches +Swift's `Noun.kgFact.rawValue`. + +The crate's own test module is the conformance suite. It checks the verb +count is nine and the adjective count is four. It checks every wire +string against its expected value. It checks that `serde` serialization +matches `as_str()` for every case. It also re-states the entire +acceptance matrix, cell by cell, including the harder-to-guess cases. One +such case is a Proposal not accepting `associate`. Another is an +Association not accepting `propose`. The two languages cannot share +source code for this vocabulary, so this test module is the only +mechanism that catches the two ports drifting apart. + +Four files must stay in sync: `Acceptance.swift`, `Noun.swift`, +`Verb.swift`, and `Adjective.swift`. Any edit to one of them requires a +matching edit in `rust/src/lib.rs`. It also requires a rerun of both test +suites. Otherwise a silent disagreement ships. diff --git a/packages/libs/AriaLexiconLib/docs/OVERVIEW.md b/packages/libs/AriaLexiconLib/docs/OVERVIEW.md index cf49e4c..4e99ec4 100644 --- a/packages/libs/AriaLexiconLib/docs/OVERVIEW.md +++ b/packages/libs/AriaLexiconLib/docs/OVERVIEW.md @@ -21,84 +21,92 @@ sources: ## What This Library Does -AriaLexiconLib names the words that every MOOTx01 kit uses to describe an -action on memory. Inside MOOTx01, that fixed vocabulary is called ARIA. ARIA -states a simple grammar: every call to the substrate is one verb applied to -a noun, optionally constrained by adjectives. AriaLexiconLib turns that -sentence into code, so a program can check whether a given verb is even -meaningful for a given noun before it tries the call. - -The library has one noun, the Drawer, which is the atomic unit of memory. -Every other storage shape the substrate persists — a knowledge-graph fact, a -vector, a diary entry — is either a facet of a Drawer or something a verb -leaves behind. The library has nine verbs, such as capture, recall, and -withdraw, and four adjective categories, such as trust and sensitivity, that -describe a row without changing which noun or verb it is. A small acceptance -matrix records which verbs apply to which storage shapes. +AriaLexiconLib names the words every MOOTx01 kit uses to describe an action +on memory. Inside MOOTx01, this fixed vocabulary is called ARIA. ARIA states +one simple grammar. Every call to the substrate is one verb applied to a +noun. A caller may constrain that call with adjectives. AriaLexiconLib turns +this sentence into code. A program can then check whether a verb even makes +sense for a noun before it tries the call. + +The library has one noun: the Drawer. The Drawer is the atomic unit of +memory. The substrate persists other storage shapes too: a knowledge-graph +fact, a vector, or a diary entry. Each of these is either a facet of a +Drawer or something a verb leaves behind. + +The library has nine verbs. Examples include capture, recall, and withdraw. +The library also has four adjective categories. Examples include trust and +sensitivity. An adjective describes a row without changing which noun or +verb it is. A small acceptance matrix records which verbs apply to which +storage shapes. ## The Problem It Solves MOOTx01 is an on-device AI memory system. It stores what an AI observes over -time and helps the AI recall it later. Many separate packages — the SDK -described at the top of this repository set calls them kits — build on top -of this vocabulary: one kit stores knowledge-graph facts, another stores -vector embeddings, another manages the estate's day-to-day operations. If -each kit invented its own names for "add a memory" or "remove a memory," a -caller would need to learn a different vocabulary per kit, and it would -become easy to call an operation that does not make sense for a given -storage shape, such as asking a vector to be captured directly instead of -letting the substrate manage it. - -AriaLexiconLib fixes the vocabulary once, in one place, as data rather than -as prose scattered across each kit's documentation. Every kit conforms to -it: a kit's API surface uses these same noun and verb names, and its -allowed-operations logic reads its answers from the acceptance matrix -instead of restating the rule. The vocabulary also crosses languages: the -library ships a Rust port so that non-Swift consumers speak the identical -words, checked one against the other by shared tests. - -The canonical statement of the grammar is a prose document, ARIA_LEXICON.md, -maintained outside this package. AriaLexiconLib is that document made -first-class in code, so an editor, a compiler, and a conformance test can -all point at the same set of names. +time. It helps the AI recall that memory later. Many separate packages +build on top of this vocabulary. The SDK described at the top of this +repository set calls these packages kits. One kit stores knowledge-graph +facts. Another stores vector embeddings. Another manages the estate's +day-to-day operations. + +If each kit invented its own names for "add a memory" or "remove a +memory," a caller would need to learn a different vocabulary per kit. It +would also become easy to call an operation that makes no sense for a +given storage shape. One example is asking a vector to be captured +directly, instead of letting the substrate manage it. + +AriaLexiconLib fixes the vocabulary once, in one place. It states the +vocabulary as data, rather than as prose scattered across each kit's +documentation. Every kit conforms to it. A kit's API surface uses these +same noun and verb names. Its allowed-operations logic reads its answers +from the acceptance matrix. It does not restate the rule on its own. The +vocabulary also crosses languages. The library ships a Rust port so +non-Swift consumers speak the identical words. Shared tests check the two +ports against each other. + +The canonical statement of the grammar is a prose document named +ARIA_LEXICON.md. That document lives outside this package. AriaLexiconLib +makes that document first class in code. An editor, a compiler, and a +conformance test can all point at the same set of names. ## How It Works -The library defines four small pieces of vocabulary and one rule that -relates two of them. +The library defines four small pieces of vocabulary. It also defines one +rule that relates two of them. -`Noun` lists the eight storage shapes the substrate persists, and marks the -Drawer as the one true noun of the language. The other seven are either a -"rung" (a representation of a Drawer's content, such as a vector), a piece -of "structure" (an edge or event about Drawers, such as an association), or -a "product" (what a verb leaves behind, such as a proposal). +`Noun` lists the eight storage shapes the substrate persists. It marks the +Drawer as the one true noun of the language. The other seven cases each +play one of three roles. A "rung" is a representation of a Drawer's +content, such as a vector. A piece of "structure" is an edge or event about +Drawers, such as an association. A "product" is what a verb leaves behind, +such as a proposal. `Verb` lists the nine actions the substrate supports: capture, reanchor, mutate, withdraw, expunge, recall, propose, associate, and learn. Each verb -also names who initiates it. Six verbs are caller-driven, meaning an -application calls them directly. Two are substrate-driven, meaning an -internal background process emits them on its own schedule. One, -`learn`, is grounding-driven: it is how authoritative outside reference +also names who initiates it. Six verbs are caller-driven. An application +calls each of these directly. Two verbs are substrate-driven. An internal +background process emits each of these on its own schedule. One verb, +`learn`, is grounding-driven. It is how authoritative outside reference material enters the system. -`Adjective` lists the four categories that describe a row no matter which -noun or verb touched it: state (where the row sits in its lifecycle, such as -active or superseded), trust (how the content was established, such as -verbatim or derived), sensitivity (how exposed the content may be), and -exportability (whether the content may leave the access perimeter). This -library only names the four categories. The specific values inside each -category — for example, the full list of lifecycle states — are a detail of -how the data is packed into storage, and that detail lives in a separate -kit, LocusKit, so the two never drift out of agreement with each other by -being defined twice. +`Adjective` lists four categories. Each category describes a row no matter +which noun or verb touched it. State describes where the row sits in its +lifecycle, such as active or superseded. Trust describes how the content +was established, such as verbatim or derived. Sensitivity describes how +exposed the content may be. Exportability describes whether the content +may leave the access perimeter. This library only names the four +categories. The specific values inside each category are a detail of how +the data is packed into storage. For example, the full list of lifecycle +states is such a detail. That detail lives in a separate kit, LocusKit. +Keeping it there means the two kits never drift out of agreement by being +defined twice. `Acceptance` is the one rule that connects `Noun` and `Verb`. For each of the eight nouns, it lists exactly which of the nine verbs make sense. A Drawer accepts six verbs, including capture and recall. A Vector accepts -none, because the substrate manages vectors on its own; nothing calls a verb -on one directly. This matrix is hand-authored, not derived automatically -from any other rule in the library, so it is the single place a caller or a -test checks "is this operation allowed here." +none. The substrate manages vectors on its own, so nothing calls a verb on +one directly. This matrix is hand-authored. No other rule in the library +derives it automatically. It is the single place a caller or a test checks +whether an operation is allowed. ## How the Pieces Fit @@ -108,33 +116,35 @@ relate to each other and to the Rust port. ![Figure 1. Topology of AriaLexiconLib](topology.svg) *Figure 1. Topology of AriaLexiconLib. `Noun` and `Verb` feed the -`Acceptance` matrix; `Adjective` stands beside them as an independent, +`Acceptance` matrix. `Adjective` stands beside them as an independent, cross-noun category list. The dashed regions mark the Rust mirror and the -downstream packages that conform to this vocabulary without this package -depending on them.* - -`AriaLexiconLib.grammar` states the one-sentence rule in plain English, as a -constant string, so a consumer can display or log the rule itself rather -than reconstruct it from the types. `Noun` and `Verb` are independent -enumerations; neither refers to the other. `Acceptance.verbs(for:)` and -`Acceptance.accepts(_:_:)` are the only functions in the package, and both -simply look up a fixed table keyed by a noun. `Adjective` does not -participate in the acceptance rule at all — an adjective always applies to -a row regardless of its noun or the verb that produced it. +downstream packages that conform to this vocabulary. This package does not +depend on them.* + +`AriaLexiconLib.grammar` states the one-sentence rule in plain English. It +is a constant string. A consumer can display or log the rule itself, +rather than reconstruct it from the types. `Noun` and `Verb` are +independent enumerations. Neither refers to the other. +`Acceptance.verbs(for:)` and `Acceptance.accepts(_:_:)` are the only +functions in the package. Both simply look up a fixed table keyed by a +noun. `Adjective` does not participate in the acceptance rule at all. An +adjective always applies to a row, regardless of its noun or the verb that +produced it. The Rust port in `rust/` mirrors every type and the acceptance matrix by -hand, because the two languages cannot share source code. A build-time -conformance suite inside the Rust crate re-asserts every wire string and -every cell of the acceptance matrix against the values recorded here, so a -change to either side that the other side misses fails a test rather than -shipping a silent disagreement. +hand. The two languages cannot share source code. A build-time conformance +suite inside the Rust crate re-asserts every wire string and every cell of +the acceptance matrix. It checks these against the values recorded here. A +change to either side that the other side misses fails a test. It does not +ship a silent disagreement. ## What Ships in the Package -The package ships five Swift source files, no bundled data files, and the -Rust port in `rust/`. There are no pinned artifacts and no build-time -tooling: every value in the package is a literal written directly in the -source, because the vocabulary is small and fixed by specification rather -than computed or trained. The verb count is fixed at nine and the adjective -category count at four; both counts are named spec invariants, so growing -either number is a specification change, not a routine code edit. +The package ships five Swift source files. It bundles no data files. It +also ships the Rust port in `rust/`. There are no pinned artifacts and no +build-time tooling. Every value in the package is a literal written +directly in the source. The vocabulary is small and fixed by +specification, rather than computed or trained. The verb count is fixed at +nine. The adjective category count is fixed at four. Both counts are +named spec invariants. Growing either number is a specification change, +not a routine code edit. diff --git a/packages/libs/EideticLib/docs/AGENT_MAP.md b/packages/libs/EideticLib/docs/AGENT_MAP.md index 203cca4..4d177ae 100644 --- a/packages/libs/EideticLib/docs/AGENT_MAP.md +++ b/packages/libs/EideticLib/docs/AGENT_MAP.md @@ -13,39 +13,39 @@ sources: blob: c5313d93bf7ca1cbf577da6ffabc9197b5e788e2 --- -# AGENT_MAP — EideticLib +# AGENT_MAP: EideticLib PURPOSE: thin deterministic facade over LatticeLib's FDC engine (term → Anchor{code, wikidataQID, confidence, dataVersion}) plus two unrelated small utilities: lattice-code shape/known-state classification, and sentence segmentation. Carries no reference data of its own. -DEPS: imports LatticeLib (FDC.encodeAnchor/isAvailable/dataVersion), NaturalLanguage (Apple-only NLTokenizer sentence path), Foundation. Imported by: CorpusKit (moot-memory kit, `EideticLib.sentences` only) and, per Package.swift header comment, NeuronKit (not present in these four venue repos — likely mootx01-ee/ce, out of scope here). Rust port in rust/ (crate `eidetic-lib`) mirrors lookup, classifyLatticeCode, and the delimiter segmenter exactly; depends on sibling `lattice-lib` crate. No Rust counterpart to the NLTokenizer path (Apple-only, excluded from cross-leg parity by the apple-nlp-accel constitutional constraint C-2). +DEPS: imports LatticeLib (FDC.encodeAnchor/isAvailable/dataVersion), NaturalLanguage (Apple-only NLTokenizer sentence path), Foundation. Imported by: CorpusKit (moot-memory kit, `EideticLib.sentences` only) and, per Package.swift header comment, NeuronKit (not present in these four venue repos: likely mootx01-ee/ce, out of scope here). Rust port in rust/ (crate `eidetic-lib`) mirrors lookup, classifyLatticeCode, and the delimiter segmenter exactly; depends on sibling `lattice-lib` crate. No Rust counterpart to the NLTokenizer path (Apple-only, excluded from cross-leg parity by the apple-nlp-accel constitutional constraint C-2). ENTRY POINTS (most callers need only these): -- EideticLib.swift:67 `EideticLib.lookup(_ term: String) -> Anchor` — main lookup; fatalError if FDC.isAvailable is false -- EideticLib.swift:118 `EideticLib.lookup(_:recordNovel:)` — same, recordNovel:false = privacy seam (no LatticeLib pool writes) -- EideticLib.swift:40 `EideticLib.classifyLatticeCode(_:knownCodes:) -> LatticeCodeState` — malformed/known/pending -- Segmenter.swift:47 `EideticLib.sentences(_ text: String) -> [Substring]` — platform-routed sentence split +- EideticLib.swift:67 `EideticLib.lookup(_ term: String) -> Anchor`: main lookup; fatalError if FDC.isAvailable is false +- EideticLib.swift:118 `EideticLib.lookup(_:recordNovel:)`: same, recordNovel:false = privacy seam (no LatticeLib pool writes) +- EideticLib.swift:40 `EideticLib.classifyLatticeCode(_:knownCodes:) -> LatticeCodeState`: malformed/known/pending +- Segmenter.swift:47 `EideticLib.sentences(_ text: String) -> [Substring]`: platform-routed sentence split ## Symbol Table -### Lookup surface — EideticLib.swift -- :20 `enum EideticLib` — namespace; holds no reference data, delegates entirely to LatticeLib.FDC -- :23 `version = "0.1.0"` — module version string -- :40 `classifyLatticeCode(_:knownCodes:) -> LatticeCodeState` — grammar check via LatticeCodeGrammar, then knownCodes membership -- :67 `lookup(_:) -> Anchor` — guards FDC.isAvailable (else fatalError); calls FDC.encodeAnchor(term); resolved→confidence 32, UNRESOLVED→empty code/confidence 0 -- :118 `lookup(_:recordNovel:) -> Anchor` — identical Anchor to lookup(_:); recordNovel:false suppresses LatticeLib novel-token pool accumulation; used by GLK capture intake (EncodeIntake) so private memory content never reaches the pool -- :147 `struct Anchor: Equatable, Sendable, Codable` — code (empty=UNRESOLVED) / wikidataQID (String?) / confidence (UInt8: 0/16/32/48/56=null/low/medium/high/verified; EideticLib only emits 0 or 32) / dataVersion (FDC signatures version); byte-identical JSON shape to Rust `Anchor` - -### Lattice-code state — LatticeCodeState.swift -- :27 `enum LatticeCodeState: Sendable, Hashable, Codable` — .malformed(String) / .known(String) / .pending(String) -- :46 `rawCode: String` — original input regardless of case (storage round-trip without unpacking) -- :56 `isWellFormed: Bool` — false only for .malformed -- :72 `enum LatticeCodeGrammar` — dependency-free; parallel reimplementation of LatticeLib's `Code.isWellFormed(_:)`, kept in agreement by shared conformance tests, NOT shared code -- :77 `maxExtensionDigits = 8` — pinned; matches LatticeLib Code.maxExtensionDigits -- :82 `isWellFormed(_ code:) -> Bool` — 3 ASCII digits + optional `.` + 1–8 ASCII digits; empty/overlong/non-digit extension fails - -### Sentence segmentation — Segmenter.swift (extension EideticLib) -- :47 `sentences(_ text:) -> [Substring]` — Apple: NLTokenizer(unit: .sentence); non-Apple (or NLTokenizer yields nothing on non-empty input): falls back to sentencesByDelimiter; empty input → [] -- :78 `sentencesByDelimiter(_ text:) -> [Substring]` — canonical cross-platform reference; splits on `.` `!` `?` `\n`, terminator stays attached to preceding segment; trailing remainder is its own segment; total-coverage guaranteed (join(segments) == text) for non-empty input +### Lookup surface: EideticLib.swift +- :20 `enum EideticLib`: namespace; holds no reference data, delegates entirely to LatticeLib.FDC +- :23 `version = "0.1.0"`: module version string +- :40 `classifyLatticeCode(_:knownCodes:) -> LatticeCodeState`: grammar check via LatticeCodeGrammar, then knownCodes membership +- :67 `lookup(_:) -> Anchor`: guards FDC.isAvailable (else fatalError); calls FDC.encodeAnchor(term); resolved→confidence 32, UNRESOLVED→empty code/confidence 0 +- :118 `lookup(_:recordNovel:) -> Anchor`: identical Anchor to lookup(_:); recordNovel:false suppresses LatticeLib novel-token pool accumulation; used by GLK capture intake (EncodeIntake) so private memory content never reaches the pool +- :147 `struct Anchor: Equatable, Sendable, Codable`: code (empty=UNRESOLVED) / wikidataQID (String?) / confidence (UInt8: 0/16/32/48/56=null/low/medium/high/verified; EideticLib only emits 0 or 32) / dataVersion (FDC signatures version); byte-identical JSON shape to Rust `Anchor` + +### Lattice-code state: LatticeCodeState.swift +- :27 `enum LatticeCodeState: Sendable, Hashable, Codable`: .malformed(String) / .known(String) / .pending(String) +- :46 `rawCode: String`: original input regardless of case (storage round-trip without unpacking) +- :56 `isWellFormed: Bool`: false only for .malformed +- :72 `enum LatticeCodeGrammar`: dependency-free; parallel reimplementation of LatticeLib's `Code.isWellFormed(_:)`, kept in agreement by shared conformance tests, NOT shared code +- :77 `maxExtensionDigits = 8`: pinned; matches LatticeLib Code.maxExtensionDigits +- :82 `isWellFormed(_ code:) -> Bool`: 3 ASCII digits + optional `.` + 1–8 ASCII digits; empty/overlong/non-digit extension fails + +### Sentence segmentation: Segmenter.swift (extension EideticLib) +- :47 `sentences(_ text:) -> [Substring]`: Apple: NLTokenizer(unit: .sentence); non-Apple (or NLTokenizer yields nothing on non-empty input): falls back to sentencesByDelimiter; empty input → [] +- :78 `sentencesByDelimiter(_ text:) -> [Substring]`: canonical cross-platform reference; splits on `.` `!` `?` `\n`, terminator stays attached to preceding segment; trailing remainder is its own segment; total-coverage guaranteed (join(segments) == text) for non-empty input - Relocated 2026-05-27 (F16) from CorpusKit/Sources/CorpusKit/Chunker.swift::sentenceSegments ## INVARIANTS / GOTCHAS @@ -53,11 +53,11 @@ ENTRY POINTS (most callers need only these): - fatalError vs UNRESOLVED are TWO DIFFERENT failure modes, do not conflate: FDC.isAvailable==false (missing/broken artifact bundle, a build defect) → fatalError, process terminates. A term with no signature overlap → UNRESOLVED → Anchor(code: "", confidence: 0), a normal, honest, non-fatal result. Never change the UNRESOLVED path to fatalError or vice versa. - lookup(_:) and lookup(_:recordNovel:) MUST return byte-identical Anchors for the same term; recordNovel only changes whether LatticeLib's shared novel-token pool is written to. Any divergence in the returned Anchor is a bug. - confidence is a fixed value, not calibrated: EideticLib always emits exactly 0 (UNRESOLVED) or 32 (resolved) from lookup. 16/48/56 are reserved slots in the substrate provenance confidence value set that this library never produces; do not repurpose them here without an explicit design change. -- classifyLatticeCode takes knownCodes as a caller-supplied parameter, not an owned canon — EideticLib deliberately has no opinion on which codes are "known." Do not add a package-level known-code cache; that decision belongs to the caller (federation/version skew is the whole reason pending exists). -- LatticeCodeGrammar.isWellFormed is a SEPARATE implementation from LatticeLib's Code.isWellFormed, not a wrapper. Changing the grammar in one without the other breaks conformance tests silently until those tests run — always update both plus the shared conformance vectors. -- sentences(_:) is NOT cross-platform-identical by design (Apple NLTokenizer vs delimiter fallback) — this is the one place in the EideticLib/LatticeLib family where platform divergence is accepted, because downstream consumers content-address chunks by (sourceID, startOffset, text) under an append-only conflict policy rather than requiring byte-identical segmentation. sentencesByDelimiter(_:) is the only segmentation function under cross-leg parity with Rust. +- classifyLatticeCode takes knownCodes as a caller-supplied parameter, not an owned canon: EideticLib deliberately has no opinion on which codes are "known." Do not add a package-level known-code cache; that decision belongs to the caller (federation/version skew is the whole reason pending exists). +- LatticeCodeGrammar.isWellFormed is a SEPARATE implementation from LatticeLib's Code.isWellFormed, not a wrapper. Changing the grammar in one without the other breaks conformance tests silently until those tests run: always update both plus the shared conformance vectors. +- sentences(_:) is NOT cross-platform-identical by design (Apple NLTokenizer vs delimiter fallback): this is the one place in the EideticLib/LatticeLib family where platform divergence is accepted, because downstream consumers content-address chunks by (sourceID, startOffset, text) under an append-only conflict policy rather than requiring byte-identical segmentation. sentencesByDelimiter(_:) is the only segmentation function under cross-leg parity with Rust. - Total-coverage invariant for both sentence functions: for non-empty input, concatenating the returned segments must reproduce the input exactly, with no gaps, overlaps, or reordering. Any change to either function must preserve this. -- EideticLib.swift and LatticeCodeState.swift and Segmenter.swift do not call one another — there is no internal pipeline. Do not assume ordering or shared state between the three surfaces. -- Package ships an intentionally empty Sources/EideticLib/Resources/ directory (placeholder so `.process("Resources")` has a target); EideticLib bundles zero reference data — do not add data files here expecting lookup to use them. All classification data lives in LatticeLib's bundle. -- Rust Anchor uses explicit `#[serde(rename = "wikidataQID")]` — auto camelCase would produce "wikidataQid" (lowercase id), breaking Swift/Rust JSON interchange. Do not remove the explicit rename. -- Rust LatticeCodeState wire shape is internally tagged: `{"state": "pending", "code": "999.42"}` — locked by `lattice_code_state_wire_shape_conformance` test; must match Swift's Codable encoding. +- EideticLib.swift and LatticeCodeState.swift and Segmenter.swift do not call one another: there is no internal pipeline. Do not assume ordering or shared state between the three surfaces. +- Package ships an intentionally empty Sources/EideticLib/Resources/ directory (placeholder so `.process("Resources")` has a target); EideticLib bundles zero reference data: do not add data files here expecting lookup to use them. All classification data lives in LatticeLib's bundle. +- Rust Anchor uses explicit `#[serde(rename = "wikidataQID")]`: auto camelCase would produce "wikidataQid" (lowercase id), breaking Swift/Rust JSON interchange. Do not remove the explicit rename. +- Rust LatticeCodeState wire shape is internally tagged: `{"state": "pending", "code": "999.42"}`: locked by `lattice_code_state_wire_shape_conformance` test; must match Swift's Codable encoding. diff --git a/packages/libs/EideticLib/docs/DETAILS.md b/packages/libs/EideticLib/docs/DETAILS.md index 1a1a362..5ca34c6 100644 --- a/packages/libs/EideticLib/docs/DETAILS.md +++ b/packages/libs/EideticLib/docs/DETAILS.md @@ -16,207 +16,220 @@ sources: # EideticLib Details This document walks through every source file in the package. Read -`OVERVIEW.md` first for the big picture. The three files are -independent of one another, so they appear here in the order a new -reader would most likely meet them: the lookup surface, the code-state +`OVERVIEW.md` first for the big picture. The three files stand +independent of one another. They appear here in the order a new reader +would most likely meet them: the lookup surface, the code-state classifier, and the sentence segmenter. ## EideticLib.swift -This file provides the module surface: the `EideticLib` enum, its -`lookup` functions, `classifyLatticeCode`, and the `Anchor` result -type. +This file provides the module surface. That surface is the +`EideticLib` enum, its `lookup` functions, `classifyLatticeCode`, and +the `Anchor` result type. `EideticLib` is a namespace enum, the same pattern LatticeLib uses for -its own module surface. It holds no reference data. The comment at the -top of the file states this explicitly: LatticeLib's FDC runtime owns -and caches the pinned lexicon, frame, and signatures; EideticLib parses -nothing of its own and simply calls through. - -The two `lookup` overloads share one behavior: they never return a -fabricated answer. If `FDC.isAvailable` is false — meaning the FDC -artifacts bundled inside LatticeLib failed to load — both overloads -call `fatalError` and terminate the process. A comment in the code -ties this choice to a specific product mandate: "a sentinel identity -that persists IS a fabricated identity" (Bob's board item 7). The -distinction matters. A term that FDC genuinely cannot classify returns -UNRESOLVED — an empty code, not an error — because that is a true -statement about the term. A missing artifact bundle is not a fact -about any term; it is a broken build, and the correct response is to -stop immediately rather than let every caller silently receive -meaningless answers. +its own module surface. It holds no reference data. A comment at the +top of the file states this plainly. LatticeLib's FDC runtime owns and +caches the pinned lexicon, frame, and signatures. EideticLib parses +nothing of its own. It simply calls through. + +The two `lookup` overloads share one behavior. They never return a +made-up answer. Suppose `FDC.isAvailable` is false, meaning the FDC +artifacts bundled inside LatticeLib failed to load. Then both overloads +call `fatalError` and stop the process. A comment in the code names a +specific product mandate for this choice. It quotes Bob's board item +seven: "a sentinel identity that persists IS a fabricated identity." +The distinction matters. A term that FDC truly cannot classify returns +UNRESOLVED. This is an empty code, not an error. That empty code is a +true statement about the term. A missing artifact bundle is different. It +is not a fact about any term. It is a broken build. The correct +response is to stop at once, rather than let every caller silently +receive a meaningless answer. `version` is the module version string, `"0.1.0"`. Like LatticeLib's -version constant, it exists so a consumer can record which build of -EideticLib produced a given answer. +own version constant, it serves one purpose. A consumer can record +which build of EideticLib produced a given answer. `classifyLatticeCode(_:knownCodes:)` sorts a candidate lattice code -string into one of `LatticeCodeState`'s three cases, using -`LatticeCodeGrammar.isWellFormed(_:)` for the shape check and the -caller-supplied `knownCodes` set for the known/pending distinction. -Passing the known-code set in as a parameter, rather than having -EideticLib own a canon, keeps EideticLib decoupled from any particular -caller's view of which codes currently resolve to a label. WHY this -matters: two callers can disagree about which codes are "known" (a -newly synced frame versus a stale one) without EideticLib taking a -side. - -`lookup(_:)` is the default path. It guards on `FDC.isAvailable`, calls -`FDC.encodeAnchor(term)`, and converts the result to an `Anchor`. A -resolved code is reported at confidence 32 (medium), the fixed value -this library always emits, because FDC itself computes no calibrated -score. An UNRESOLVED term becomes an `Anchor` with an empty `code`, a -`nil` `wikidataQID`, and confidence 0 — the honest "nothing matched" -result, never a best-guess fallback. +string into one of `LatticeCodeState`'s three cases. It uses +`LatticeCodeGrammar.isWellFormed(_:)` for the shape check. It uses the +caller-supplied `knownCodes` set for the known-versus-pending +distinction. The known-code set arrives as a parameter, rather than +living inside EideticLib as an owned canon. This keeps EideticLib +decoupled from any single caller's view of which codes currently +resolve to a label. Here is why that matters. Two callers can disagree +about which codes count as known. One might hold a newly synced frame. +The other might hold a stale one. EideticLib does not take a side. + +`lookup(_:)` is the default path. It guards on `FDC.isAvailable`. It +calls `FDC.encodeAnchor(term)`. Then it converts the result to an +`Anchor`. A resolved code is reported at a fixed confidence of 32, or +medium. FDC computes no calibrated score of its own, so EideticLib +always emits this same value. An UNRESOLVED term becomes an `Anchor` +with an empty `code`, a `nil` `wikidataQID`, and confidence 0. That is +the honest "nothing matched" result, never a best-guess fallback. `lookup(_:recordNovel:)` is the privacy variant. It produces the exact -same `Anchor` as `lookup(_:)` for the same term, but threads -`recordNovel` through to `FDC.encodeAnchor(_:recordNovel:)`. When -`false`, any novel token encountered while building the term's concept -bag is not written into LatticeLib's shared learning pool. WHY this -exists: the GLK (memory-capture) intake path classifies user-supplied -memory content before deciding whether to store it, and that content -must never leak a single token into a shared, cross-session pool — -even a rejected or sensitive capture must leak nothing, because -classification happens before the write decision, not after. +same `Anchor` as `lookup(_:)` for the same term. It also threads +`recordNovel` through to `FDC.encodeAnchor(_:recordNovel:)`. When that +flag is `false`, any novel token found while building the term's +concept bag stays out of LatticeLib's shared learning pool. Here is why +this variant exists. The GLK memory-capture intake path classifies +user-supplied memory content. It does this before it decides whether +to store that content. That content must never leak even a single +token into a shared, cross-session pool. This holds even for a +rejected or sensitive capture. Classification runs before the write +decision, not after. `Anchor` is a plain `Equatable`, `Sendable`, `Codable` struct with four -fields: `code` (empty string means UNRESOLVED), `wikidataQID` -(optional), `confidence` (a `UInt8` drawn from the fixed set 0/16/32/48/56 -— null/low/medium/high/verified — of which EideticLib currently only -ever emits 0 or 32), and `dataVersion` (the FDC signatures version that -produced the answer, letting a caller record provenance). The struct's -doc comment notes it is byte-identical to the Rust port's `Anchor` -under JSON encoding, which is what lets a Swift app and a Rust daemon +fields. `code` is empty when the term is UNRESOLVED. `wikidataQID` is +optional. `confidence` is a `UInt8`. It is drawn from a fixed set of +five values: 0, 16, 32, 48, and 56. These stand for null, low, medium, +high, and verified. EideticLib currently only ever emits 0 or 32. +`dataVersion` is the FDC signatures version that produced the answer. +This lets a caller record provenance. The struct's doc comment notes +that it is byte-identical to the Rust port's `Anchor` under JSON +encoding. That identity is what lets a Swift app and a Rust daemon exchange anchors through a shared store. ## LatticeCodeState.swift -This file provides `LatticeCodeState`, the three-state classification -of a candidate lattice code, and `LatticeCodeGrammar`, the grammar -check behind it. - -The file's opening comment frames the problem directly: a lattice code -shown to EideticLib is either malformed (fails the grammar), known -(well-formed and present in the caller's current canon), or pending -(well-formed, but not yet in the caller's canon). WHY a third state is -needed at all: FDC frames are versioned and ship new codes over time, -and MOOTx01 estates can federate, meaning one estate's stored code may -predate another estate's local frame. Rejecting an unrecognized but -well-formed code would lose data permanently; a pending code instead -round-trips intact through storage until a newer frame resolves it. +This file provides two things: `LatticeCodeState` and +`LatticeCodeGrammar`. `LatticeCodeState` is the three-state +classification of a candidate lattice code. `LatticeCodeGrammar` is the +grammar check behind it. + +The file's opening comment frames the problem directly. A lattice code +shown to EideticLib falls into one of three states. A malformed code +fails the grammar. A known code is well formed and present in the +caller's current canon. A pending code is well formed, but not yet +present in that canon. Here is why a third state exists at all. FDC +frames carry a version and ship new codes over time. MOOTx01 estates +can federate, so one estate's stored code may predate another estate's +local frame. Rejecting an unrecognized but well-formed code would lose +data for good. A pending code instead round-trips intact through +storage until a newer frame resolves it. `LatticeCodeState` conforms to `Sendable`, `Hashable`, and `Codable`. -The `Codable` conformance is not incidental — it is, per the file's own -comment, "the core invariant of the launch plan's valid-but-unknown-code -requirement": a pending code must survive being written to and read -back from storage without alteration. `rawCode` returns the original -string regardless of which case matched, so a storage layer can -persist the raw value without unpacking the enum first. `isWellFormed` -is `true` for `.known` and `.pending` and `false` only for -`.malformed`, which is useful in tests asserting that pending codes are -still accepted, just not yet resolved. - -`LatticeCodeGrammar` is a second, independent implementation of the -same grammar LatticeLib's `Code.isWellFormed(_:)` enforces: three ASCII -digits, optionally followed by a dot and one to eight further ASCII -digits. WHY it is reimplemented here rather than imported from -LatticeLib: the file's comment explains that EideticLib can validate a +The `Codable` conformance is not incidental. The file's own comment +calls it "the core invariant of the launch plan's +valid-but-unknown-code requirement": a pending code must survive being +written to storage and read back without alteration. `rawCode` returns +the original string regardless of which case matched, so a storage +layer can persist the raw value without unpacking the enum first. +`isWellFormed` reads `true` for `.known` and `.pending`, and `false` +only for `.malformed`. Tests rely on this to assert that pending codes +are still accepted, just not yet resolved. + +`LatticeCodeGrammar` is a second, separate check. It enforces the same +grammar as LatticeLib's `Code.isWellFormed(_:)`. That grammar requires +three ASCII digits. It allows an optional dot, followed by one to eight +more ASCII digits. + +Here is why the grammar is reimplemented, rather than imported from +LatticeLib. The file's comment gives the reason. EideticLib can check a code's shape without loading LatticeLib's frame, lexicon, and -signatures at all — a caller who only wants to know "is this string -shaped like a code" pays none of the cost of the classification engine. -The two implementations are kept in agreement by shared conformance -tests, not by shared code. +signatures at all. A caller who only wants to know whether a string is +shaped like a code pays none of the cost of the full classification +engine. Shared conformance tests keep the two implementations in +agreement. The implementations do not share code. `maxExtensionDigits` is a pinned constant, `8`, matching LatticeLib's own `Code.maxExtensionDigits`. `isWellFormed(_:)` splits the string on -its first `.`, requires the integer part to be exactly three ASCII -digits, and — if a decimal part is present — requires it to be one to -eight ASCII digits with no other characters. An empty decimal part -(a trailing dot with nothing after it) fails the check. +its first `.`. It requires the integer part to be exactly three ASCII +digits. If a decimal part is present, it requires that part to hold one +to eight ASCII digits with no other characters. An empty decimal part, +meaning a trailing dot with nothing after it, fails the check. ## Segmenter.swift This file provides `sentences(_:)` and `sentencesByDelimiter(_:)`, both declared in an extension on the `EideticLib` enum. -The file's opening comment draws an explicit parallel to LatticeLib's -`WordClassTagger`: that file splits word tagging into a deterministic -reference path (the fast-path table plus HMM) and an accelerated, -Apple-only path (`NLTagger`, opt-in). `Segmenter` applies the same -shape to sentence splitting instead of word tagging. -`sentencesByDelimiter(_:)` is the deterministic reference, -identical on every platform. `sentences(_:)` is the platform-routed -entry point most callers should use: on Apple platforms it defers to -`NLTokenizer`, which correctly handles abbreviations ("Dr."), locale-aware -punctuation, and other edge cases a naive delimiter split gets wrong; on -non-Apple platforms it falls back to `sentencesByDelimiter`. - -WHY platform divergence here is acceptable, when LatticeLib treats -platform divergence in word tagging as unacceptable: the comment cites +The file's opening comment draws a direct parallel to LatticeLib's +`WordClassTagger`. That file splits word tagging into two paths. One +path is deterministic: the fast-path table plus an HMM. The other path +is accelerated and Apple-only, built on `NLTagger`. Callers must opt +into the accelerated path. `Segmenter` applies that same shape to +sentence splitting instead of word tagging. `sentencesByDelimiter(_:)` +is the deterministic reference. It behaves the same on every platform. +`sentences(_:)` is the platform-routed entry point most callers should +use. On Apple platforms it defers to `NLTokenizer`. This tool correctly +handles tricky abbreviations. For example, it does not split "Dr." +into two sentences. It also handles locale-aware punctuation and other +edge cases that a naive delimiter split gets wrong. On non-Apple +platforms it falls back to `sentencesByDelimiter`. + +Why is this divergence acceptable here? LatticeLib treats the same +kind of divergence as unacceptable in word tagging. The comment cites the "apple-nlp-accel constitutional constraint (C-2)." Downstream -consumers of sentence segments key each chunk by `(sourceID, -startOffset, text)` — a content address — rather than by chunk index. -Two devices producing slightly different sentence boundaries for the -same document therefore produce a superset of chunks under an -append-only conflict policy, not a pair of conflicting writes that need -reconciling. This is a materially weaker consistency requirement than -FDC's lattice codes, which must be byte-identical across devices for -federated recall to work at all. - -Both functions guarantee total coverage: for non-empty input, every -character appears in exactly one output segment, and joining the -segments reproduces the original text. This holds even in pathological -cases — text that is nothing but delimiters, or text with no delimiter -at all, which returns the entire input as a single segment. Empty -input returns an empty array from both functions. The file's history -comment notes the code was relocated here in 2026-05-27 (a change -tagged F16) from `CorpusKit/Sources/CorpusKit/Chunker.swift`, to -centralize every linguistic-pipeline stage — tokenizing, normalizing, -stemming, tagging, and now segmenting — behind LatticeLib and -EideticLib rather than scattering pieces of the pipeline across -consuming kits. +consumers of sentence segments key each chunk by a content address. +That address is `(sourceID, startOffset, text)`, not a chunk index. +Two devices might split the same document into slightly different +sentences. Under an append-only conflict policy, this produces a +superset of chunks. It does not create a conflict that needs +reconciling. This is a much weaker consistency rule than the one FDC's +lattice codes carry. Those codes must match byte for byte across +devices, or federated recall fails. + +Both functions guarantee total coverage. For non-empty input, every +character appears in exactly one output segment. Joining the segments +back together reproduces the original text. This holds even in +pathological cases. Consider text that is nothing but delimiters. +Consider also text with no delimiter at all. That case returns as a +single segment holding the whole input. Empty input returns an empty +array from both functions. The file's history comment notes that the +code moved here on 2026-05-27, in a change tagged F16, from +`CorpusKit/Sources/CorpusKit/Chunker.swift`. That move gathered every +text-pipeline stage under LatticeLib and EideticLib. Those stages are +tokenizing, normalizing, stemming, tagging, and now segmenting. The +move ended the old pattern of scattering pipeline pieces across the +kits that consume them. `sentences(_:)` builds an `NLTokenizer(unit: .sentence)`, enumerates sentence-unit ranges over the input, and collects the corresponding -substrings. If the tokenizer produces no segments for non-empty input, -the function falls back to returning the whole input as one segment, -preserving the total-coverage guarantee even in that edge case. - -`sentencesByDelimiter(_:)` walks the string once, splitting after every -`.`, `!`, `?`, or `\n`, and keeping the terminator attached to the -segment that ends there. Any trailing text after the final terminator -becomes its own segment. This is the function the Rust port's +substrings. Suppose the tokenizer produces no segments for non-empty +input. Then the function falls back to returning the whole input as one +segment, which preserves the total-coverage guarantee even in that edge +case. + +`sentencesByDelimiter(_:)` walks the string once. It splits the text +after each terminator: a period, an exclamation point, a question +mark, or a newline. It keeps each terminator attached to the segment +that ends there. Any trailing text after the final terminator becomes +its own segment. This is the function the Rust port's `segmenter::sentences` mirrors exactly, because Rust has no NLTokenizer-equivalent acceleration to port. ## Rust Port and Conformance -The `rust/` directory holds the second leg: `lib.rs` (the crate surface, -re-exporting `lookup` and `classify`), `anchor.rs` (`Anchor`, with an -explicit `#[serde(rename = "wikidataQID")]` so the JSON key matches -Swift's `Codable` output exactly rather than the auto-derived -`wikidataQid`), `lattice_code_state.rs` (`LatticeCodeState`, -`LatticeCodeGrammar`, and `classify_lattice_code`, wire-tagged as -`{"state": ..., "code": ...}` to match Swift's encoding), and -`segmenter.rs` (the delimiter algorithm only — there is no Rust -counterpart to the Apple-only `NLTokenizer` path, by design). +The `rust/` directory holds the second leg. `lib.rs` is the crate +surface, re-exporting `lookup` and `classify`. `anchor.rs` defines +`Anchor`, with an explicit `#[serde(rename = "wikidataQID")]` so the +JSON key matches Swift's `Codable` output exactly, rather than the +auto-derived `wikidataQid`. `lattice_code_state.rs` defines three +items: `LatticeCodeState`, `LatticeCodeGrammar`, and +`classify_lattice_code`. The wire format is tagged as +`{"state": ..., "code": ...}`, matching Swift's encoding. +`segmenter.rs` implements the delimiter algorithm only. There is no +Rust counterpart to the Apple-only `NLTokenizer` path, by design. The Rust crate depends on the sibling `lattice-lib` crate for -`Fdc::encode_anchor` and `Fdc::is_available`, the same relationship -Swift's `EideticLib` target has to the LatticeLib Swift package. -`rust::lookup` panics under the identical condition Swift's `lookup` -treats as a `fatalError`: unavailable FDC artifacts. +`Fdc::encode_anchor` and `Fdc::is_available`. This is the same +relationship Swift's `EideticLib` target holds with the LatticeLib +Swift package. `rust::lookup` panics under the identical condition that +Swift's `lookup` treats as a `fatalError`: unavailable FDC artifacts. Cross-language conformance runs on two fixtures. -`Tests/SharedVectors/lookup_vectors.json` (schema version 2, 26 vectors) -pins expected FDC codes and Wikidata Q-IDs for a set of inputs, -including edge cases such as punctuation-only text; both +`Tests/SharedVectors/lookup_vectors.json`, schema version 2, holds +twenty-six vectors. It pins expected FDC codes and Wikidata Q-IDs for a +set of inputs, including edge cases such as punctuation-only text. Both `rust/tests/lookup_conformance_test.rs` and the Swift -`LookupConformanceTests.swift` load the same file and must match it +`LookupConformanceTests.swift` load that same file and must match it exactly. `Tests/SharedVectors/word_class_vectors.json` is a second -fixture, shared with LatticeLib's own word-classification conformance -surface, exercised here because the EideticLib test target links -LatticeLib directly. When you change `lookup`, `classifyLatticeCode`, -or either segmentation function, update both legs and rerun both test -suites; the fixtures are the contract. +fixture. LatticeLib's own word-classification conformance surface +shares this fixture too. The EideticLib test target links LatticeLib +directly, so this fixture applies here as well. Suppose you change +`lookup`, `classifyLatticeCode`, +or either segmentation function. Then update both legs, and rerun both +test suites. The fixtures are the contract. diff --git a/packages/libs/EideticLib/docs/OVERVIEW.md b/packages/libs/EideticLib/docs/OVERVIEW.md index 56250e0..a0135ae 100644 --- a/packages/libs/EideticLib/docs/OVERVIEW.md +++ b/packages/libs/EideticLib/docs/OVERVIEW.md @@ -17,132 +17,147 @@ sources: ## What This Library Does -EideticLib is a thin, deterministic front door onto LatticeLib's -classification engine. A caller hands it a term and gets back an -`Anchor`: a lattice code, the dominant Wikidata Q-ID of the term, a -confidence value, and the data version that produced the answer. A -Wikidata Q-ID is a public, language-neutral identifier for one -concept, such as `Q144` for "dog." EideticLib holds no reference data -of its own. Every lookup delegates to `FDC.encodeAnchor`, the runtime -entry point of LatticeLib's Frame-Directed Classification (FDC) -engine. - -Alongside lookup, EideticLib carries two small, self-contained -utilities that other parts of the SDK need: a check for whether a -lattice code string is well-formed and known, and a function that -splits text into sentences. Neither utility depends on the FDC engine. -All three surfaces share one purpose: give callers a deterministic -answer without asking them to understand LatticeLib's internals. +EideticLib sits in front of LatticeLib's classification engine. It gives +callers one simple way to classify a term. A caller passes in a term. +EideticLib returns an `Anchor`. + +An `Anchor` holds four fields: a lattice code, a Wikidata Q-ID, a +confidence value, and a data version. A Wikidata Q-ID is a public +identifier for one concept. It works the same way across every +language. For example, `Q144` names the concept "dog." + +EideticLib holds no reference data of its own. Every lookup calls +`FDC.encodeAnchor`. That call is the entry point into LatticeLib's +Frame-Directed Classification engine, known as FDC. + +EideticLib also carries two small utilities. The first checks whether +a lattice code string is well formed and known. The second splits text +into sentences. Neither utility needs the FDC engine to run. All three +surfaces share one goal: give callers a clear answer without asking +them to learn LatticeLib's internals. ## The Problem It Solves -LatticeLib's `FDC` type is a powerful but low-level engine: it owns -pinned artifacts, exposes several scoring modes, and returns raw -codes and ancestry chains. Most callers do not need any of that. They -need one function: give me a term, tell me what it is about. EideticLib -is that function. It fixes a confidence scale, wraps the result in a -stable `Codable` type, and enforces a strict failure policy so callers -never have to guess whether an empty answer means "no data loaded" or -"nothing matched." - -A second, related problem is memory content privacy. Some callers -classify user-supplied memory content — private text that must never -leak into LatticeLib's shared novel-token learning pool, even by -accident. EideticLib exposes a `recordNovel` parameter that turns that -accumulation off while returning the exact same anchor, so a capture -path can classify text safely without a second, divergent code path. - -A third problem concerns the lattice code canon itself. FDC frames are -versioned, and federated devices can be running against different -frame versions at different times. A device might receive a lattice -code from another estate — one user's complete memory store — that its -own local frame does not yet recognize. Discarding that code would -lose information; guessing at its meaning would violate the classifier's -honesty. `classifyLatticeCode` sorts a code string into one of three -states — malformed, known, or pending — so storage layers can accept and -round-trip a code they cannot yet resolve, and settle it later when a -newer frame ships. - -The fourth problem is splitting text into sentences before other -pipeline stages run. Sentence boundaries look simple until abbreviations -("Dr.", "e.g.") and locale-specific punctuation appear. `Segmenter` -answers this the same way LatticeLib answers tokenization: a fast, -higher-accuracy path on Apple platforms, and a plain, deterministic -fallback everywhere else. +LatticeLib's `FDC` type is powerful, but it sits at a low level. It +owns pinned artifacts. It exposes several scoring modes. It returns raw +codes and ancestry chains. Most callers need none of that detail. They +need one function: give me a term, and tell me what it means. EideticLib +is that function. + +EideticLib fixes a confidence scale. It wraps the result in a stable +`Codable` type. It enforces a strict failure policy. A caller never has +to guess whether an empty answer means "no data loaded" or "nothing +matched." + +A second problem concerns privacy of memory content. Some callers +classify text a user typed into memory. That text must never leak into +LatticeLib's shared pool for novel tokens. Even one stray token would +be a problem. EideticLib exposes a `recordNovel` parameter for this +case. When a caller turns this off, the pool learns nothing from that +text. The caller still gets back the exact same anchor. One call path +can then serve both public and private text. + +A third problem concerns the lattice code canon itself. FDC frames +carry a version number. Federated devices can run different frame +versions at the same time. A device might receive a lattice code from +another estate, one user's full memory store. The receiving device's +own frame may not yet recognize that code. Discarding the code would +lose information. Guessing at its meaning would break the classifier's +honesty policy. `classifyLatticeCode` sorts a code into one of three +states: malformed, known, or pending. This lets a storage layer keep a +code it cannot resolve yet. The code settles later, once a newer frame +ships. + +The fourth problem is splitting text into sentences before later +pipeline stages run. Sentence boundaries look simple at first. Then +abbreviations appear, such as "Dr." or "e.g." Locale-specific +punctuation appears too. `Segmenter` solves this the way LatticeLib +solves tokenization. It offers a fast, accurate path on Apple +platforms. It offers a plain, deterministic fallback everywhere else. ## How It Works `EideticLib.lookup(_:)` first checks `FDC.isAvailable`. If the pinned -FDC artifacts failed to load, the function terminates the process with -`fatalError` rather than returning an empty or fabricated answer. A -failed load means the binary shipped without its required data, which -is a build defect, not a condition any caller should recover from at -runtime. This is a deliberate stance: MOOTx01's provenance rules treat -a persistent placeholder identity as a fabricated identity, so -EideticLib refuses to manufacture one. - -Given available artifacts, `lookup` calls `FDC.encodeAnchor(term)`, -which returns a lattice code and a Wikidata Q-ID, or `nil` for the -code when the term is UNRESOLVED — the honest "no answer" result that -LatticeLib returns rather than guess. EideticLib packages a resolved -result as an `Anchor` with a fixed confidence of 32 (medium), because -FDC produces no calibrated confidence score of its own. An UNRESOLVED -term becomes an `Anchor` with an empty code, no Q-ID, and confidence 0. -`lookup(_:recordNovel:)` is the same function with the pool-recording -side effect suppressed, for callers classifying private memory content. - -`classifyLatticeCode(_:knownCodes:)` runs independently of the FDC -engine. It checks the code string against a small grammar — three -digits, an optional decimal extension of up to eight digits — using -`LatticeCodeGrammar`, a self-contained rule set that mirrors LatticeLib's -own `Code.isWellFormed(_:)`. A code that fails the grammar is -`.malformed`. A well-formed code present in the caller-supplied +FDC artifacts failed to load, the function stops the process with +`fatalError`. It never returns an empty or made-up answer instead. A +failed load means the binary shipped without its required data. That +is a build defect. No caller should try to recover from it at runtime. +This is a deliberate stance. MOOTx01's provenance rules treat a +placeholder identity that persists as a fabricated identity. So +EideticLib refuses to invent one. + +Given available artifacts, `lookup` calls `FDC.encodeAnchor(term)`. +This call returns a lattice code and a Wikidata Q-ID. It returns `nil` +for the code when the term is UNRESOLVED. UNRESOLVED is the honest "no +answer" result. LatticeLib returns it rather than guess. EideticLib +packages a resolved result as an `Anchor` with a fixed confidence of +32, meaning medium, because FDC itself produces no calibrated +confidence score. An UNRESOLVED term becomes an `Anchor` with an empty +code, no Q-ID, and confidence 0. + +`lookup(_:recordNovel:)` is the same function with one difference. It +suppresses the pool-recording side effect. Callers use this variant to +classify private memory content. + +`classifyLatticeCode(_:knownCodes:)` runs independent of the FDC +engine. It checks a code string against a small grammar. The grammar +allows three digits, followed by an optional decimal extension of up +to eight digits. It uses `LatticeCodeGrammar`, a rule set that mirrors +LatticeLib's own `Code.isWellFormed(_:)`. A code that fails the grammar +is `.malformed`. A well-formed code present in the caller's `knownCodes` set is `.known`. A well-formed code absent from that set -is `.pending`: accepted, storable, and round-tripped intact rather than -rejected. +is `.pending`. A pending code is still accepted. It is stored and +round-tripped intact, rather than rejected. `EideticLib.sentences(_:)` splits text into sentence substrings. On -Apple platforms it uses `NLTokenizer`, which understands abbreviations -and locale-aware boundaries. On every other platform, and whenever -strict cross-platform identity is required, `sentencesByDelimiter(_:)` -splits on periods, exclamation points, question marks, and newlines, -keeping the terminator attached to the segment it ends. Both functions -guarantee total coverage: every character of the input appears in -exactly one output segment, so a caller can always reconstruct the -original text by joining the segments back together. +Apple platforms, it uses `NLTokenizer`. This tool understands +abbreviations and locale-aware boundaries. On every other platform, and +whenever strict cross-platform identity matters, it uses +`sentencesByDelimiter(_:)` instead. That function splits on periods, +exclamation points, question marks, and newlines. It keeps each +terminator attached to the segment it ends. Both functions guarantee +total coverage. Every character of the input appears in exactly one +output segment. A caller can always rebuild the original text by +joining the segments back together. ## How the Pieces Fit -EideticLib is not a pipeline the way LatticeLib is. Its three surfaces -— lookup, code-state classification, and sentence segmentation — do not -call one another. Each is an independent, narrow entry point that -answers one question, and each has its own relationship to the rest of -the SDK: `lookup` crosses into LatticeLib's FDC engine, code -classification is a pure, dependency-free grammar check, and sentence -segmentation crosses into Apple's NaturalLanguage framework only on -Apple platforms. - -That fan-out shape is also EideticLib's value: it is a single, stable -package that other libraries and kits import for a handful of small, -unrelated conveniences, so they do not each have to depend on -LatticeLib, NaturalLanguage, and a code-grammar checker separately. -CorpusKit, for example, imports EideticLib only for `sentences(_:)`, -to segment memory text into chunks before further processing. +EideticLib is not a pipeline the way LatticeLib is. It has three +surfaces: lookup, code-state classification, and sentence +segmentation. These surfaces do not call one another. Each is an +independent, narrow entry point. Each answers one question. Each has +its own link to the rest of the SDK. `lookup` crosses into LatticeLib's +FDC engine. Code classification is a pure, dependency-free grammar +check. Sentence segmentation crosses into Apple's NaturalLanguage +framework, but only on Apple platforms. + +This fan-out shape is also EideticLib's value. It is a single, stable +package. Other libraries and kits import it for a handful of small, +unrelated conveniences. They do not each need to depend on LatticeLib, +NaturalLanguage, and a code-grammar checker separately. CorpusKit, for +example, imports EideticLib only for `sentences(_:)`. CorpusKit uses +this function to split memory text into chunks before further +processing. The library keeps the same two-language guarantee as LatticeLib. A -Swift leg serves Apple platforms, and a Rust leg in `rust/` mirrors +Swift leg serves Apple platforms. A Rust leg in `rust/` mirrors `lookup`, `classifyLatticeCode`, and `sentencesByDelimiter` exactly. -Shared conformance vectors in `Tests/SharedVectors/lookup_vectors.json` -gate every release so both legs agree byte for byte on every case in -the fixture. +Shared conformance vectors live in +`Tests/SharedVectors/lookup_vectors.json`. They gate every release, so +both legs agree byte for byte on every case in the fixture. ## What Ships in the Package -The package ships three Swift source files under `Sources/EideticLib/`, -an intentionally empty `Resources/` directory (EideticLib carries no -reference data of its own; LatticeLib owns the pinned artifacts that -`lookup` depends on), and the Rust port under `rust/`. The Rust crate, -`eidetic-lib`, depends on the sibling `lattice-lib` crate the same way -the Swift target depends on the LatticeLib Swift package. Cross-language -conformance is enforced by `Tests/SharedVectors/lookup_vectors.json` -(26 vectors) and the integration tests in `rust/tests/`. +The package ships three Swift source files under `Sources/EideticLib/`. +It also ships an intentionally empty `Resources/` directory. EideticLib +carries no reference data of its own. LatticeLib owns the pinned +artifacts that `lookup` depends on. The package also ships the Rust +port under `rust/`. + +The Rust crate is named `eidetic-lib`. It depends on the sibling +`lattice-lib` crate. This mirrors how the Swift target depends on the +LatticeLib Swift package. + +Cross-language conformance rests on two checks. The first is +`Tests/SharedVectors/lookup_vectors.json`, with twenty-six vectors. The +second is the set of integration tests in `rust/tests/`. diff --git a/packages/libs/LatticeLib/docs/AGENT_MAP.md b/packages/libs/LatticeLib/docs/AGENT_MAP.md index cd5f87c..14dd1aa 100644 --- a/packages/libs/LatticeLib/docs/AGENT_MAP.md +++ b/packages/libs/LatticeLib/docs/AGENT_MAP.md @@ -51,113 +51,113 @@ sources: blob: 15e27da5cdeca3540654914992e8c78f83ed937a --- -# AGENT_MAP — LatticeLib +# AGENT_MAP: LatticeLib PURPOSE: deterministic on-device text→classification-code engine (FDC: Frame-Directed Classification) + shared text primitives. Text → concept bag → scored match against pinned code signatures → decimal-frame descent → code (or nil=UNRESOLVED). DEPS: imports SubstrateML (EigenvalueCentrality, build-time LexRank only), NaturalLanguage (LexRank sentence-split; optional NLTagger path), OSLog. Imported by: EideticLib, AriaMcpKit, apps/moot-mgr, tools/seed-generator. Rust port in rust/ mirrors everything; conformance fixtures rust/tests/fixtures/*.json gate byte-identity. ENTRY POINTS (most callers need only these): -- FDCRuntime.swift:32 `FDC.encodeAnchor(_ text:) -> (code: String?, conceptQID: String?)` — main runtime classify -- FDCRuntime.swift:45 `FDC.encodeAnchor(_:recordNovel:)` — same, recordNovel:false = privacy seam (no pool writes) -- WordClassTagger.swift:65 `LatticeLib.wordClass(_ token:) -> WordClass` — single-token noun/verb/other +- FDCRuntime.swift:32 `FDC.encodeAnchor(_ text:) -> (code: String?, conceptQID: String?)`: main runtime classify +- FDCRuntime.swift:45 `FDC.encodeAnchor(_:recordNovel:)`: same, recordNovel:false = privacy seam (no pool writes) +- WordClassTagger.swift:65 `LatticeLib.wordClass(_ token:) -> WordClass`: single-token noun/verb/other ## Symbol Table -### Runtime facade — FDCRuntime.swift -- :15 `enum FDC` — artifact-owning facade; loads lexicon+frame+signatures once/process -- :23 `FDC.stopThreshold = 1` — pinned descent cutoff (inert on shallow v1.0 frame) -- :27 `encode(_ text:) -> String?` — code only; nil = UNRESOLVED or artifacts missing -- :32/:45 `encodeAnchor(...)` — code + dominant Q-ID; see ENTRY POINTS -- :50 `isAvailable: Bool` — artifacts loaded? -- :54 `dataVersion: String` — signatures version for provenance; "0.0.0-unavailable" on load failure -- :67 `ancestors(of code:) -> [String]` — frame ancestry, root-first, excl. self -- :84 `label(for code:) -> String?` — human heading; 3-digit codes report parent's label (leaf labels are multi-subject compounds) - -### Matcher — FDCMatcher.swift -- :30 `struct FDCMatcher: Sendable` — Steps 4–5; direct init for custom artifacts/tests -- :49 `enum ScoreMode` — .raw/.idf/.cosine/.idfCosine; runtime ships .idf, direct-init default .raw -- :63 `stopThreshold: Int` — raw-overlap descent cutoff, mode-independent -- :120 `init(lexicon:frame:signatures:stopThreshold:scoreMode:)` — interns terms→dense Int ids (alphabetical order ⇒ Int sort == String sort ⇒ bit-stable float sums); builds inverted index + idf + norms -- :267 `maximumTiedWinnersForClassification = 4` — >4 tied argmax winners ⇒ UNRESOLVED (generic-vocabulary guard) -- :271 `encode(_:)`, :280 `encodeAnchor(_:)`, :296 `encodeAnchor(_:recordNovel:)` — public encode surface - -### Bag construction — ConceptBag.swift -- :25 `typealias ConceptBag = [String: Int]` — conceptID|surface → count -- :27 `enum BagBuilder` — Steps 1–3 -- :37 `bag(_:lexicon:keep:)` — default (keep = [.noun,.verb]); Q-ID lexicon hit bypasses POS filter (named-entity relaxation, deterministic via pinned lexicon) -- :82 `bag(_:lexicon:keep:recordNovel:)` — identical bag; recordNovel:false suppresses pool side effect -- :118 `bag(_:lexicon:keep:taggerChoice:)` — estate-configured tagger threading +### Runtime facade: FDCRuntime.swift +- :15 `enum FDC`: artifact-owning facade; loads lexicon+frame+signatures once/process +- :23 `FDC.stopThreshold = 1`: pinned descent cutoff (inert on shallow v1.0 frame) +- :27 `encode(_ text:) -> String?`: code only; nil = UNRESOLVED or artifacts missing +- :32/:45 `encodeAnchor(...)`: code + dominant Q-ID; see ENTRY POINTS +- :50 `isAvailable: Bool`: artifacts loaded? +- :54 `dataVersion: String`: signatures version for provenance; "0.0.0-unavailable" on load failure +- :67 `ancestors(of code:) -> [String]`: frame ancestry, root-first, excl. self +- :84 `label(for code:) -> String?`: human heading; 3-digit codes report parent's label (leaf labels are multi-subject compounds) + +### Matcher: FDCMatcher.swift +- :30 `struct FDCMatcher: Sendable`: Steps 4–5; direct init for custom artifacts/tests +- :49 `enum ScoreMode`: .raw/.idf/.cosine/.idfCosine; runtime ships .idf, direct-init default .raw +- :63 `stopThreshold: Int`: raw-overlap descent cutoff, mode-independent +- :120 `init(lexicon:frame:signatures:stopThreshold:scoreMode:)`: interns terms→dense Int ids (alphabetical order ⇒ Int sort == String sort ⇒ bit-stable float sums); builds inverted index + idf + norms +- :267 `maximumTiedWinnersForClassification = 4`: >4 tied argmax winners ⇒ UNRESOLVED (generic-vocabulary guard) +- :271 `encode(_:)`, :280 `encodeAnchor(_:)`, :296 `encodeAnchor(_:recordNovel:)`: public encode surface + +### Bag construction: ConceptBag.swift +- :25 `typealias ConceptBag = [String: Int]`: conceptID|surface → count +- :27 `enum BagBuilder`: Steps 1–3 +- :37 `bag(_:lexicon:keep:)`: default (keep = [.noun,.verb]); Q-ID lexicon hit bypasses POS filter (named-entity relaxation, deterministic via pinned lexicon) +- :82 `bag(_:lexicon:keep:recordNovel:)`: identical bag; recordNovel:false suppresses pool side effect +- :118 `bag(_:lexicon:keep:taggerChoice:)`: estate-configured tagger threading ### Text primitives -- Tokenizer.swift:21 `Tokenizer.tokenize(_:) -> [String]` — UAX #29 words, input order -- Normalizer.swift:55 `Normalizer.normalize(_:) -> String` — NFKCSubset.fold then lowercased; fold MUST run first -- NFKCSubset.swift:40 `NFKCSubset.fold(_:)` (internal) — ASCII-target compatibility subset; table mirrored verbatim in rust/src/normalizer.rs — edit both or neither -- Stemmer.swift:69 `Stemmer.stem(_:) -> String` — Porter2/Snowball English; <3 chars unchanged -- Stemmer.swift:55 `bundledReferenceCorpus() -> Data?` — conformance corpus accessor - -### Word-class tagging — WordClassTagger.swift (extension LatticeLib) -- :65 `wordClass(_:)` — table fast path (verb set checked BEFORE noun set) → HMM fallback; NEVER NLTagger -- :102 `taggerEnabled(osVersion:minOSVersion:)` — NLTagger OS gate; unparseable minOS fails closed -- :208 `wordClass(_:recordNovel:)` — pool-suppressing variant -- :285 `wordClass(_:tagger:)` / :330 `wordClass(_:tagger:recordNovel:)` — estate-choice dispatch; .nlTagger fails closed to .other below min OS, falls back to HMM on non-Apple -- WordClass.swift:23 `enum WordClass: String` — noun|verb|other; string-backed for wire stability -- NovelTokenTaggerChoice.swift:22 `enum NovelTokenTaggerChoice` — .hmm (default, federatable) | .nlTagger (Apple-only, non-deterministic cross-platform); mirrors PersistenceKit type, bridge by switch -- HMMTagger.swift:70 `enum HMMTagger` (internal) — 3-state integer Viterbi; artifact HMMTaggerModel.json (hmm-viterbi-3, MASC 3.0.0 rare-word-trained); tie → lowest state index (noun [String]`: UAX #29 words, input order +- Normalizer.swift:55 `Normalizer.normalize(_:) -> String`: NFKCSubset.fold then lowercased; fold MUST run first +- NFKCSubset.swift:40 `NFKCSubset.fold(_:)` (internal): ASCII-target compatibility subset; table mirrored verbatim in rust/src/normalizer.rs: edit both or neither +- Stemmer.swift:69 `Stemmer.stem(_:) -> String`: Porter2/Snowball English; <3 chars unchanged +- Stemmer.swift:55 `bundledReferenceCorpus() -> Data?`: conformance corpus accessor + +### Word-class tagging: WordClassTagger.swift (extension LatticeLib) +- :65 `wordClass(_:)`: table fast path (verb set checked BEFORE noun set) → HMM fallback; NEVER NLTagger +- :102 `taggerEnabled(osVersion:minOSVersion:)`: NLTagger OS gate; unparseable minOS fails closed +- :208 `wordClass(_:recordNovel:)`: pool-suppressing variant +- :285 `wordClass(_:tagger:)` / :330 `wordClass(_:tagger:recordNovel:)`: estate-choice dispatch; .nlTagger fails closed to .other below min OS, falls back to HMM on non-Apple +- WordClass.swift:23 `enum WordClass: String`: noun|verb|other; string-backed for wire stability +- NovelTokenTaggerChoice.swift:22 `enum NovelTokenTaggerChoice`: .hmm (default, federatable) | .nlTagger (Apple-only, non-deterministic cross-platform); mirrors PersistenceKit type, bridge by switch +- HMMTagger.swift:70 `enum HMMTagger` (internal): 3-state integer Viterbi; artifact HMMTaggerModel.json (hmm-viterbi-3, MASC 3.0.0 rare-word-trained); tie → lowest state index (noun Apple App Support > XDG; single source of truth for read+write sides -- NovelPoolSubmitter.swift:97 `tableArtifactURL()` — writable merged WordClassTable.json, SIBLING of pool dir -- PoolReducer.swift:154 `PoolReducer.reduce(poolDirectory:tableArtifactURL:now:maxFiles:)` — batch merge; seeds writable artifact from bundle; quarantines malformed/stale-version files; NOUN/VERB only; first-occurrence dedup; archives consumed files (idempotent); DO NOT call from hot path -- PoolReducer.swift:65 `PoolReduceResult` / :105 `PoolReducerError` — result summary / failure classes +- NovelTokenCache.swift:86 `final class NovelTokenCache`: locked pending list; drains at exactly :91 `poolSubmitThreshold = 50`; submitter injected, called outside lock +- NovelTokenCache.swift:30 `PoolEntry` / :42 `PoolSubmission`: wire format (snake_case keys; tags "NOUN"/"VERB"/"OTHER") +- NovelPoolSubmitter.swift:49 `make(poolDirectory:)` / :63 `makeDefault()`: file-writing submitter factory; fire-and-forget, no retry +- NovelPoolSubmitter.swift:81 `poolDirectory()`: LATTICE_POOL_DIR > Apple App Support > XDG; single source of truth for read+write sides +- NovelPoolSubmitter.swift:97 `tableArtifactURL()`: writable merged WordClassTable.json, SIBLING of pool dir +- PoolReducer.swift:154 `PoolReducer.reduce(poolDirectory:tableArtifactURL:now:maxFiles:)`: batch merge; seeds writable artifact from bundle; quarantines malformed/stale-version files; NOUN/VERB only; first-occurrence dedup; archives consumed files (idempotent); DO NOT call from hot path +- PoolReducer.swift:65 `PoolReduceResult` / :105 `PoolReducerError`: result summary / failure classes ### Frame + codes -- Code.swift:34 `Code.isWellFormed(_:)` — grammar only: 3 digits + optional ≤8-digit decimal ext (:31 maxExtensionDigits=8); known-vs-pending is caller's concern -- Code.swift:58 `Code.integerBase(of:)` — "540.137" → 540 -- FDCFrame.swift:17 `FDCEntry` — code + verbatim label -- FDCFrame.swift:34 `FDCFrame` — versioned code list; ancestry DERIVED, not stored -- FDCFrame.swift:142 `ancestors(of:)` — TWO regimes: integer head is Dewey positional (parent("006")="000"), decimal tail is per-segment (parent("006.6")="006"); NOT a dot-split -- FDCFrame.swift:131 `children(of:)` — immediate children, sorted +- Code.swift:34 `Code.isWellFormed(_:)`: grammar only: 3 digits + optional ≤8-digit decimal ext (:31 maxExtensionDigits=8); known-vs-pending is caller's concern +- Code.swift:58 `Code.integerBase(of:)`: "540.137" → 540 +- FDCFrame.swift:17 `FDCEntry`: code + verbatim label +- FDCFrame.swift:34 `FDCFrame`: versioned code list; ancestry DERIVED, not stored +- FDCFrame.swift:142 `ancestors(of:)`: TWO regimes: integer head is Dewey positional (parent("006")="000"), decimal tail is per-segment (parent("006.6")="006"); NOT a dot-split +- FDCFrame.swift:131 `children(of:)`: immediate children, sorted ### Build-time only (never on-device) -- Lexicon.swift:30 `CanonicalizationLexicon` — version/language/entries (stem→conceptID); version is part of agreement protocol -- Lexicon.swift:91 `LexiconBuilder.build(_:)` — Wikidata-primary, WordNet-disambiguated; 3-tier deterministic conflict resolution; single-token keys only -- CodeSignature.swift:36 `SignatureAssembler.merge(label:title:article:weights:)` — source weights label 3 / title 2 / article 1 (:13 SourceWeights) -- CodeSignature.swift:52 `accumulateAncestors(ownTerms:ancestorsOf:)` — signature += all ancestors' own terms -- CodeSignature.swift:24 `CodeSignature` — terms + fingerprint (nil until SimHash pass) -- LexRank.swift:25 `LexRank.reduce(_:sentences:)` — top-N central sentences (default :20 =10, cosine threshold :21 =0.1) via SubstrateML.EigenvalueCentrality; deterministic but NOT cross-platform-gated +- Lexicon.swift:30 `CanonicalizationLexicon`: version/language/entries (stem→conceptID); version is part of agreement protocol +- Lexicon.swift:91 `LexiconBuilder.build(_:)`: Wikidata-primary, WordNet-disambiguated; 3-tier deterministic conflict resolution; single-token keys only +- CodeSignature.swift:36 `SignatureAssembler.merge(label:title:article:weights:)`: source weights label 3 / title 2 / article 1 (:13 SourceWeights) +- CodeSignature.swift:52 `accumulateAncestors(ownTerms:ancestorsOf:)`: signature += all ancestors' own terms +- CodeSignature.swift:24 `CodeSignature`: terms + fingerprint (nil until SimHash pass) +- LexRank.swift:25 `LexRank.reduce(_:sentences:)`: top-N central sentences (default :20 =10, cosine threshold :21 =0.1) via SubstrateML.EigenvalueCentrality; deterministic but NOT cross-platform-gated -### Q-ID closure — QIDClosure.swift -- :45 `QIDClosure.ancestors(of qid:) -> [String]` — transitive P31/P279 closure over pinned QIDClosureEdges.json; BFS, excl. self, sorted by Q-number; memoized per process -- :68 `isAvailable` / :73 `dataVersion` — artifact presence / provenance +### Q-ID closure: QIDClosure.swift +- :45 `QIDClosure.ancestors(of qid:) -> [String]`: transitive P31/P279 closure over pinned QIDClosureEdges.json; BFS, excl. self, sorted by Q-number; memoized per process +- :68 `isAvailable` / :73 `dataVersion`: artifact presence / provenance -### Module — LatticeLib.swift -- :13 `enum LatticeLib` — namespace; :17 `version = "1.0.0"` bumped with artifact ships +### Module: LatticeLib.swift +- :13 `enum LatticeLib`: namespace; :17 `version = "1.0.0"` bumped with artifact ships ## INVARIANTS / GOTCHAS - DETERMINISM IS THE CONTRACT. encode() is pure over (text, artifact versions). Any change to Tokenizer/Normalizer/NFKCSubset/Stemmer/HMMTagger/FDCMatcher scoring must be mirrored in rust/src/ and pass rust/tests/fixtures/{normalize,tag,fdc}_conformance.json + Swift conformance tests. Same for artifact regeneration. - nil code = UNRESOLVED, by design: empty bag, no signature overlap, or >4 tied winners. Never "fix" by guessing. -- Verb set checked before noun set in every wordClass overload — a token in both is a verb. Do not reorder. +- Verb set checked before noun set in every wordClass overload: a token in both is a verb. Do not reorder. - HMM (not NLTagger) is the default novel-token tagger on ALL platforms incl. Apple. NLTagger only via explicit .nlTagger estate choice; it breaks cross-platform determinism and federation. - NFKCSubset table + Rust normalizer.rs are ONE table in two files. Never edit one alone. -- FDCMatcher term interning: IDs assigned in ascending String order — required so sorted-Int iteration == sorted-String iteration ⇒ bit-identical float sums. Do not change assignment order. +- FDCMatcher term interning: IDs assigned in ascending String order: required so sorted-Int iteration == sorted-String iteration ⇒ bit-identical float sums. Do not change assignment order. - FDCFrame ancestry: integer head ≠ decimal tail regimes. parent("006") is "000" (Dewey positional), not "00". A dot-split reimplementation is wrong. - stopThreshold compares RAW integer overlap, never the (possibly normalized) score. -- Pinned constants — do not change without a new artifact version + conformance regen: poolSubmitThreshold 50, stopThreshold 1, maxExtensionDigits 8, maximumTiedWinnersForClassification 4, SourceWeights 3/2/1, HMM logScale 1000, LexRank 10/0.1. -- Privacy seams: recordNovel:false variants produce byte-identical results with pool accumulation suppressed — required for user memory content (GLK capture, EideticLib.lookup). sharedNovelCache writes files ONLY when LATTICE_POOL_DIR is set; default submitter is a no-op. +- Pinned constants: do not change without a new artifact version + conformance regen: poolSubmitThreshold 50, stopThreshold 1, maxExtensionDigits 8, maximumTiedWinnersForClassification 4, SourceWeights 3/2/1, HMM logScale 1000, LexRank 10/0.1. +- Privacy seams: recordNovel:false variants produce byte-identical results with pool accumulation suppressed: required for user memory content (GLK capture, EideticLib.lookup). sharedNovelCache writes files ONLY when LATTICE_POOL_DIR is set; default submitter is a no-op. - Bundled WordClassTable.json is a small seed (21 nouns / 18 verbs, v1.0.0): in practice most tokens are novel and route through the HMM until the pool loop grows the writable table. -- Artifacts load once per process (static let); FDC/QIDClosure fail soft (isAvailable=false, encode→nil) — check isAvailable in hosts with unusual bundle setups. +- Artifacts load once per process (static let); FDC/QIDClosure fail soft (isAvailable=false, encode→nil): check isAvailable in hosts with unusual bundle setups. - PoolReducer is batch-only (governor idle-tick or operator CLI). Bounded by maxFiles per run; backlog drains across runs. - WordClassTableCache.swap is the ONLY legal live-update path for tagging behavior; readers get whole-snapshot consistency, and version() observability keys determinism. - LexRank is build-time-only; it may use floats and NLTokenizer freely. Do not move it into the runtime encode path. diff --git a/packages/libs/LatticeLib/docs/DETAILS.md b/packages/libs/LatticeLib/docs/DETAILS.md index 3d4c2e5..0dffa77 100644 --- a/packages/libs/LatticeLib/docs/DETAILS.md +++ b/packages/libs/LatticeLib/docs/DETAILS.md @@ -54,522 +54,550 @@ sources: # LatticeLib Details This document walks through every source file in the package. Read -`OVERVIEW.md` first for the big picture. Files appear here in pipeline order: -the module surface, the text primitives, the word-class machinery, the -learning loop, and finally the classification engine and its helpers. +`OVERVIEW.md` first for the big picture. Files appear here in pipeline +order. First comes the module surface. Then come the text primitives, the +word-class machinery, and the learning loop. Last comes the classification +engine and its helpers. ## LatticeLib.swift -This file provides the module surface: the public `LatticeLib` enum and the -module version. +This file provides the module surface. It holds the public `LatticeLib` +enum and the module version. -Swift has no module-level functions, so libraries often use an empty enum as -a namespace. The `LatticeLib` enum is that namespace. Other files extend it; -the word-class tagging functions in `WordClassTagger.swift` all live in an -extension of this enum. +Swift has no module-level functions. So libraries often use an empty enum +as a namespace. The `LatticeLib` enum is that namespace. Other files +extend it. The word-class tagging functions in `WordClassTagger.swift` all +live in an extension of this enum. -`LatticeLib.version` is the module version string. It exists so that -consumers can record which library version produced a result. The version is -bumped in lockstep with the bundled FDC artifacts when new signatures ship, -because a result is only reproducible against the exact artifact set that -produced it. +`LatticeLib.version` is the module version string. It lets consumers +record which library version produced a result. The version is bumped in +lockstep with the bundled FDC artifacts when new signatures ship. A result +is only reproducible against the exact artifact set that produced it. ## Tokenizer.swift This file provides `Tokenizer`, which splits raw text into words. -Splitting text into words sounds simple but is full of edge cases: -apostrophes, hyphens, non-Latin scripts. Rather than invent rules, the -tokenizer follows Unicode Standard Annex #29, the industry-standard -word-boundary specification. Following a public standard is what allows the -Swift leg and the Rust leg to agree byte for byte: both implement the same -published rules, and a shared conformance fixture proves it. +Splitting text into words sounds simple. It is full of edge cases instead: +apostrophes, hyphens, non-Latin scripts. The tokenizer does not invent its +own rules. It follows Unicode Standard Annex #29, the industry-standard +word-boundary specification. This public standard is what lets the Swift +leg and the Rust leg agree byte for byte. Both implement the same +published rules. A shared conformance fixture proves it. -`Tokenizer.tokenize(_:)` returns the words of a string in input order. -Whitespace and punctuation are dropped. It works by asking Foundation to -enumerate word substrings, which routes to ICU's word-boundary analyzer. The -Rust port uses the `unicode-segmentation` crate, which implements the same -annex. +`Tokenizer.tokenize(_:)` returns the words of a string in input order. It +drops whitespace and punctuation. It works by asking Foundation to +enumerate word substrings. This routes to ICU's word-boundary analyzer. +The Rust port uses the `unicode-segmentation` crate. That crate implements +the same annex. ## NFKCSubset.swift This file provides the internal compatibility-fold table used by the -normalizer. It maps visually equivalent Unicode characters to plain ASCII: a -full-width `Ａ` becomes `A`, the ligature `fi` becomes `fi`, a superscript `²` -becomes `2`. - -Why not use the platform's full Unicode NFKC normalization? Because the Rust -leg cannot reproduce it byte for byte without a large external dependency, -and the two legs must never disagree. Instead, both legs implement the same -explicit table, defined once here and mirrored verbatim in +normalizer. It maps visually equivalent Unicode characters to plain ASCII. +A full-width `Ａ` becomes `A`. The ligature `fi` becomes `fi`. A superscript +`²` becomes `2`. + +Why not use the platform's full Unicode NFKC normalization? The Rust leg +cannot reproduce it byte for byte without a large external dependency. +The two legs must never disagree. So both legs instead implement the same +explicit table. It is defined once here and mirrored verbatim in `rust/src/normalizer.rs`. Same table plus same algorithm equals identical -output by construction. Every mapping is a strict subset of official NFKC, -limited to mappings whose target is plain ASCII, because those are the ones -that improve matching for an ASCII-English corpus. - -`fold(_:)` rewrites a string scalar by scalar, replacing each mapped -character and passing everything else through unchanged. `foldScalar(_:)` -holds the actual mapping: an explicit literal table first, then arithmetic -range families (full-width forms, super/subscripts, circled letters, Roman -numerals) expressed as offsets so the table stays small and auditable. -Canonical recomposition (for example, half-width katakana) is deliberately -out of scope; the file documents this so a future reader does not mistake the -omission for a bug. +output by construction. Every mapping is a strict subset of official NFKC. +Each is limited to mappings whose target is plain ASCII. Those are the +mappings that improve matching for an ASCII-English corpus. + +`fold(_:)` rewrites a string scalar by scalar. It replaces each mapped +character and passes everything else through unchanged. `foldScalar(_:)` +holds the actual mapping. An explicit literal table comes first. Then come +arithmetic range families: full-width forms, super and subscripts, circled +letters, and Roman numerals. These are expressed as offsets, so the table +stays small and easy to check. Canonical recomposition, such as half-width +katakana, is deliberately out of scope. The file documents this so a +future reader does not mistake the omission for a bug. ## Normalizer.swift This file provides `Normalizer`, the canonicalization front door. It runs -before stemming and lexicon lookup so that different surface spellings of the -same word collapse to one key. +before stemming and lexicon lookup. This lets different surface spellings +of the same word collapse to one key. -`Normalizer.normalize(_:)` applies the `NFKCSubset` fold and then a +`Normalizer.normalize(_:)` applies the `NFKCSubset` fold, then a Unicode-aware lowercase. The fold runs first for a reason the code spells -out: a full-width `Ａ` must first become `A` and then fold to `a`, matching -the key that a plain `a` produces. Lowercasing alone would leave the -full-width form distinct, and the same word would split into two different -lexicon keys. The function is deterministic and byte-identical to the Rust -port for every input in the shared conformance fixture. +out. A full-width `Ａ` must first become `A`. Only then does it fold to +`a`, matching the key that a plain `a` produces. Lowercasing alone would +leave the full-width form distinct. The same word would then split into +two different lexicon keys. The function is deterministic. It is +byte-identical to the Rust port for every input in the shared conformance +fixture. ## Stemmer.swift This file provides `Stemmer`, a hand-ported implementation of the Porter2 -(Snowball English) stemming algorithm. Stemming reduces a word to its root: -"dogs" becomes "dog," "classification" becomes "classif." The point is -recall: text about "running" and text about "runs" should reach the same -lexicon entry. - -Porter2 removes English suffixes in five ordered steps, guided by two regions -of the word (R1 and R2) computed from vowel-consonant transitions. Most -removals only apply when the suffix sits inside a region, which prevents -overstripping short words. Two exception lists handle irregular forms, such -as "skies" becoming "sky" rather than "ski." - -`Stemmer.stem(_:)` runs the full algorithm: preprocessing (apostrophe and -consonant-y marking), region computation, then steps 0 through 5. Words -shorter than three letters return unchanged, per the specification. -`Stemmer.bundledReferenceCorpus()` exposes the pinned test corpus -(`Resources/SnowballEnglish.json`) so the conformance test can verify that -this port and the Rust port (which uses the `rust-stemmers` crate) produce -byte-identical stems for every input. +(Snowball English) stemming algorithm. Stemming reduces a word to its +root: "dogs" becomes "dog," and "classification" becomes "classif." The +point is recall. Text about "running" and text about "runs" should reach +the same lexicon entry. + +Porter2 removes English suffixes in five ordered steps. Two regions of the +word, called R1 and R2, guide this process. Vowel-consonant transitions +compute these regions. Most removals only apply inside a region. This +prevents overstripping short words. Two exception lists handle irregular +forms. For example, "skies" becomes "sky" rather than "ski." + +`Stemmer.stem(_:)` runs the full algorithm. First comes preprocessing: +apostrophe marking and consonant-y marking. Then comes region computation, +then steps zero through five. Words shorter than three letters return +unchanged, per the specification. `Stemmer.bundledReferenceCorpus()` +exposes the pinned test corpus, `Resources/SnowballEnglish.json`. The +conformance test uses it to verify a claim. This port and the Rust port, +which uses the `rust-stemmers` crate, must produce byte-identical stems +for every input. ## WordClass.swift -This file provides `WordClass`, the label produced by tagging Step 1 of the -encoder: `.noun`, `.verb`, or `.other`. +This file provides `WordClass`. This is the label produced by tagging Step +1 of the encoder: `.noun`, `.verb`, or `.other`. -Step 1 keeps only nouns and verbs because they carry the subject matter of a -sentence. `.other` is the discard bucket for everything else — articles, -prepositions, adjectives, punctuation. The enum is backed by strings so it -serializes to a stable, human-readable JSON form (`"noun"`, `"verb"`, -`"other"`) that the shared conformance vectors and the Rust port both read. +Step 1 keeps only nouns and verbs. They carry the subject matter of a +sentence. `.other` is the discard bucket for everything else: articles, +prepositions, adjectives, punctuation. The enum is backed by strings. So +it serializes to a stable, human-readable JSON form: `"noun"`, `"verb"`, +`"other"`. Both the shared conformance vectors and the Rust port read this +form. ## WordClassTable.swift -This file provides the noun/verb fast-path table and its live-swappable -process cache. The table is the first stop for every token: a token present -in it resolves to its word class in constant time, with no tagger invoked. +This file provides the noun and verb fast-path table. It also provides its +live-swappable process cache. The table is the first stop for every +token. A token present in it resolves to its word class in constant time. +No tagger is invoked in that case. -Three load functions cover the table's life cycle. `loadBundled()` reads the -pinned snapshot shipped in `Resources/WordClassTable.json`. +Three load functions cover the table's life cycle. `loadBundled()` reads +the pinned snapshot shipped in `Resources/WordClassTable.json`. `loadWritable()` reads the merged artifact that `PoolReducer` maintains -outside the app bundle. `loadWithPrecedence()` is the canonical entry: it +outside the app bundle. `loadWithPrecedence()` is the canonical entry. It prefers the writable merged artifact, because that one contains learned -tokens, and falls back to the bundled pristine table. - -`WordClassTableCache` solves a harder problem: adopting a new table while the -process is running. After the reducer merges new tokens, waiting for a -process restart would delay learning indefinitely. The cache instead -publishes an immutable snapshot (`WordClassTableSnapshot`, the parsed table -plus precomputed membership sets) behind a lock. Readers copy the snapshot -reference out and test membership outside the lock, so a swap can never -produce a torn read: a reader sees the whole old table or the whole new one, -never a mix. `swap(_:)` publishes a new snapshot and bumps a version counter. +tokens. It falls back to the bundled pristine table only when needed. + +`WordClassTableCache` solves a harder problem. It adopts a new table while +the process keeps running. Waiting for a process restart would delay +learning indefinitely after the reducer merges new tokens. The cache +instead publishes an immutable snapshot behind a lock. This snapshot is a +`WordClassTableSnapshot`: the parsed table plus precomputed membership +sets. Readers copy the snapshot reference out. They test membership +outside the lock. A swap can therefore never produce a torn read. A reader +sees the whole old table or the whole new one, never a mix. `swap(_:)` +publishes a new snapshot and bumps a version counter. `reloadFromPrecedence()` and `reload(fromArtifact:)` re-resolve the table -from disk and publish it. The version counter matters for determinism: -tagging is deterministic given (input, table version), and the counter makes -the version observable. +from disk and publish it. The version counter matters for determinism. +Tagging is deterministic given the input and the table version. The +counter makes that version observable. ## WordClassTagger.swift This file provides the public tagging API, `LatticeLib.wordClass`, in its four overloads. This is encoder Step 1 as callers see it. -Every overload shares the same fast path. The token is lowercased, then -checked against the live table snapshot: the verb set first, then the noun -set, so a token listed in both resolves to verb. Only a token absent from the -table — a novel token — reaches a tagger. +Every overload shares the same fast path. The token is lowercased. Then it +is checked against the live table snapshot: the verb set first, then the +noun set. A token listed in both resolves to verb. Only a token absent +from the table, a novel token, reaches a tagger. `wordClass(_:)` is the default path. Novel tokens go to the deterministic -HMM tagger on all platforms, including Apple. This is the rule that keeps -the Swift and Rust ports bit-identical: Apple's NLTagger is a black box that -can change between OS releases, so it is never in the default path. +HMM tagger on all platforms, including Apple. This rule keeps the Swift +and Rust ports bit-identical. Apple's NLTagger is a black box that can +change between OS releases. So it is never in the default path. `wordClass(_:tagger:)` threads an explicit estate choice. An estate is one -user's memory store; each estate picks its novel-token tagger once, at -creation. The `.hmm` choice is the cross-platform baseline. The `.nlTagger` -choice opts in to Apple's NaturalLanguage tagger for higher accuracy at the -cost of cross-platform determinism, and it is gated by -`taggerEnabled(osVersion:minOSVersion:)`: the table pins the NLTagger OS -version that seeded it, and an older OS fails closed to `.other` rather than -invoke a differently-behaving tagger. +user's memory store. Each estate picks its novel-token tagger once, at +creation. The `.hmm` choice is the cross-platform baseline. The +`.nlTagger` choice opts in to Apple's NaturalLanguage tagger. It buys +higher accuracy at the cost of cross-platform determinism. +`taggerEnabled(osVersion:minOSVersion:)` gates that choice. The table pins +the NLTagger OS version that seeded it. An older OS fails closed to +`.other` rather than invoke a differently behaving tagger. `wordClass(_:recordNovel:)` and `wordClass(_:tagger:recordNovel:)` add -privacy control. By default a novel token's tag is recorded into the shared -pool cache so the table can learn it later. When the text being classified is -private memory content, callers pass `recordNovel: false` and the side effect -is suppressed. The returned tag is identical either way; only the recording -changes. The shared cache itself (`sharedNovelCache`) is wired to a real file -writer only when the `LATTICE_POOL_DIR` environment variable is set; -otherwise the submitter is a no-op, so no deployment leaks tokens to disk by -default. +privacy control. By default, a novel token's tag is recorded into the +shared pool cache. This lets the table learn it later. Sometimes the text +being classified is private memory content. Callers then pass +`recordNovel: false`, and the side effect is suppressed. The returned tag +is identical either way. Only the recording changes. The shared cache +itself, `sharedNovelCache`, is wired to a real file writer only when the +`LATTICE_POOL_DIR` environment variable is set. Otherwise the submitter is +a no-op. No deployment leaks tokens to disk by default. ## HMMTagger.swift -This file provides the deterministic novel-token tagger: a three-state Hidden -Markov Model (noun, verb, other) decoded with integer Viterbi. - -Why a statistical model here at all? Because novel tokens are exactly the -words no table covers, so some generalization is required. And why integer -arithmetic? Because floating-point rounding can differ across platforms and -compilers, and this tagger must produce the same answer on every port. The -model weights are log-likelihoods scaled by 1,000 and rounded to integers at -build time; runtime scoring is pure integer addition and comparison. - -The observation alphabet is morphological: each token maps to one observation -such as "ends in -ing," "ends in -tion," "contains a non-letter," or "plain." -`observe(_:)` checks these in a fixed priority order that both ports -replicate exactly. `tag(_:)` then picks the state with the best initial plus -emission weight; for a one-token sequence, Viterbi reduces to that argmax, -with ties resolved to the lowest state index. - -The weights ship in the frozen artifact `Resources/HMMTaggerModel.json`, -trained on the MASC 3.0.0 Penn Treebank annotation. The training deliberately -uses only the corpus's rare words (frequency 1). The tagger only ever sees -out-of-vocabulary words, and rare words are the best statistical proxy for -those, which gives the correct noun-leaning prior for unknown words. -Estimating from the full corpus would let frequent function words dominate -and wrongly default unknowns to `.other`. +This file provides the deterministic novel-token tagger. It is a +three-state Hidden Markov Model, over noun, verb, and other. Integer +Viterbi decodes it. + +Why a statistical model here at all? Novel tokens are exactly the words no +table covers. So some generalization is required. Why integer arithmetic, +though? Floating-point rounding can differ across platforms and +compilers. This tagger must produce the same answer on every port. The +model weights are log-likelihoods. They are scaled by one thousand and +rounded to integers at build time. Runtime scoring is pure integer +addition and comparison. + +The observation alphabet is morphological. Each token maps to one +observation, such as "ends in -ing" or "ends in -tion." Other options are +"contains a non-letter" or "plain." `observe(_:)` checks these in a fixed +priority order. Both ports replicate that order exactly. `tag(_:)` then +picks the state with the best initial plus emission weight. For a +one-token sequence, Viterbi reduces to that best-scoring pick. Ties +resolve to the lowest state index. + +The weights ship in the frozen artifact `Resources/HMMTaggerModel.json`. +They are trained on the MASC 3.0.0 Penn Treebank annotation. The training +deliberately uses only the corpus's rare words, at frequency one. The +tagger only ever sees out-of-vocabulary words. Rare words are the best +statistical proxy for those words. This gives the correct noun-leaning +prior for unknown words. Estimating from the full corpus would let +frequent function words dominate instead. It would then wrongly default +unknowns to `.other`. ## NovelTokenTaggerChoice.swift -This file provides `NovelTokenTaggerChoice`, the two-case enum (`.hmm`, -`.nlTagger`) that selects the novel-token tagger for an estate. +This file provides `NovelTokenTaggerChoice`. This is the two-case enum, +`.hmm` and `.nlTagger`, that selects the novel-token tagger for an estate. -The type mirrors an identically named enum in PersistenceKit, where the -choice is stored on the estate configuration. The two packages define it -independently so that neither depends on the other; a consumer bridges -between them with a trivial switch. The file documents the threading rule: -pass the choice down as a parameter from the estate configuration to the -call sites, never through global state. `.hmm` is the default and the -federation-safe choice; `.nlTagger` is Apple-only and trades determinism for -accuracy. +The type mirrors an identically named enum in PersistenceKit. There, the +choice is stored on the estate configuration. The two packages define the +type independently, so that neither depends on the other. A consumer +bridges between them with a trivial switch. The file documents the +threading rule: pass the choice down as a parameter from the estate +configuration to the call sites. Never pass it through global state. +`.hmm` is the default and the federation-safe choice. `.nlTagger` is +Apple-only. It trades determinism for accuracy. ## NovelTokenCache.swift -This file provides the local accumulation cache for novel-token tags, plus -the two wire-format types `PoolEntry` and `PoolSubmission`. +This file provides the local accumulation cache for novel-token tags. It +also provides two wire-format types, `PoolEntry` and `PoolSubmission`. -When the tagger classifies a novel token, the result is worth keeping: if +When the tagger classifies a novel token, the result is worth keeping. If many devices see the same unknown word, the shared table should learn it. `NovelTokenCache.record(token:wordClass:)` appends the tagged token to a -pending list under a lock. At exactly `poolSubmitThreshold` (50) entries, the -cache builds a `PoolSubmission` — stamped with the table version, platform, -and tagger version — drains the list, and hands the payload to an injected -submitter closure outside the lock. Entries below the threshold are kept -indefinitely; there is no aging or cleanup, by contract. - -The submitter is injected at construction so tests can assert the drain -without touching the file system, and so hosts that never configure a pool -get a no-op. The version stamps exist because the pool consumer discards -submissions made against a stale table version; an observation is only -meaningful relative to the table that failed to contain it. +pending list under a lock. At exactly `poolSubmitThreshold`, fifty +entries, the cache builds a `PoolSubmission`. It is stamped with the table +version, platform, and tagger version. The cache then drains the list. It +hands the payload to an injected submitter closure outside the lock. +Entries below the threshold are kept indefinitely. There is no aging or +cleanup, by contract. + +The submitter is injected at construction. Tests can then assert the +drain without touching the file system. Hosts that never configure a pool +get a no-op this way. The version stamps exist for a reason. The pool +consumer discards submissions made against a stale table version. An +observation is only meaningful relative to the table that failed to +contain it. ## NovelPoolSubmitter.swift -This file provides the production submitter factory: it turns a drained +This file provides the production submitter factory. It turns a drained `PoolSubmission` into a JSON file in a local pool directory. -The pool is a directory of files rather than a database or a network call. -Files are cheap, observable, and easy to reason about; a future reducer can -process them in name order because each file name embeds an ISO 8601 -timestamp plus a short unique suffix. Writes are atomic and fire-and-forget: -a failed write is logged and the data is discarded for that cycle, never -retried and never crashed on, because pool data is recoverable from future +The pool is a directory of files, not a database or a network call. Files +are cheap, observable, and easy to reason about. A future reducer can +process them in name order. Each file name embeds an ISO 8601 timestamp +plus a short unique suffix. Writes are atomic and fire-and-forget. A +failed write is logged, and the data is discarded for that cycle. It is +never retried and never crashed on. Pool data is recoverable from future observations. `make(poolDirectory:)` returns a submitter closure bound to a directory. `makeDefault()` binds to the resolved default. `poolDirectory()` resolves -that default: the `LATTICE_POOL_DIR` environment variable when set, otherwise -an Application Support path on Apple platforms or an XDG data path elsewhere. -It is public because the write side (submitter) and the read side (the -governor that triggers the reducer) must agree on the path, so it lives here -as the single source of truth. `tableArtifactURL()` resolves the sibling -location of the writable merged table — `WordClassTable.json` next to the -pool directory — which is the file the reducer updates and -`WordClassTable.loadWritable()` reads. +that default. It checks the `LATTICE_POOL_DIR` environment variable first. +Otherwise it uses an Application Support path on Apple platforms, or an +XDG data path elsewhere. This function is public for a reason. The write +side is the submitter. The read side is the governor that triggers the +reducer. The two sides must agree on the path. So this function lives +here as the single source of truth. `tableArtifactURL()` resolves the +sibling location of the writable merged table. That table is +`WordClassTable.json`, next to the pool directory. This is the file the +reducer updates and `WordClassTable.loadWritable()` reads. ## PoolReducer.swift -This file provides the batch job that closes the learning loop: it reads -pooled submissions and merges qualifying novel tokens into the writable +This file provides the batch job that closes the learning loop. It reads +pooled submissions. It merges qualifying novel tokens into the writable word-class table artifact. `PoolReducer.reduce(poolDirectory:tableArtifactURL:now:maxFiles:)` runs in steps. It first seeds the writable artifact from the bundled table if none -exists, because the reducer cannot write into the read-only app bundle. It -then loads the table, enumerates `pool_*.json` files, and processes the -oldest `maxFiles` of them; bounding the batch keeps a large backlog from -stalling the caller, and the remainder drains on later runs. Each file is -decoded and version-checked: a malformed or stale-version file is moved to a -quarantine directory rather than deleted, preserving forensic value. Within -the run, the first occurrence of each token wins, tokens already in the table -are skipped, and only NOUN and VERB tags expand the table — an OTHER tag is -already represented by absence. Consumed files move to an archive directory, -which makes the reducer idempotent: re-running on a drained pool is a no-op. -If anything was consumed, the artifact is rewritten atomically with sorted -keys and an advanced snapshot date. - -`PoolReduceResult` summarizes a run (consumed, quarantined, added, skipped), -and `PoolReducerError` covers the three failure classes (table read, table -write, pool directory unreadable). The file warns against wiring the reducer -into the per-token tagging path: reduction is a batch operation, triggered by -a resident governor on an idle cadence or by an operator command. +exists. The reducer cannot write into the read-only app bundle otherwise. +It then loads the table and enumerates `pool_*.json` files. It processes +the oldest `maxFiles` of them. Bounding the batch keeps a large backlog +from stalling the caller. The remainder drains on later runs. Each file is +decoded and version-checked. A malformed or stale-version file moves to a +quarantine directory rather than being deleted. This preserves forensic +value. Within the run, the first occurrence of each token wins. Tokens +already in the table are skipped. Only NOUN and VERB tags expand the +table, since an OTHER tag is already represented by absence. Consumed +files move to an archive directory. This makes the reducer idempotent, so +re-running on a drained pool is a no-op. If anything was consumed, the +artifact is rewritten atomically with sorted keys and an advanced +snapshot date. + +`PoolReduceResult` summarizes a run: consumed, quarantined, added, and +skipped counts. `PoolReducerError` covers three failure classes: table +read, table write, and pool directory unreadable. The file warns against +wiring the reducer into the per-token tagging path. Reduction is a batch +operation. A resident governor triggers it on an idle cadence, or an +operator triggers it by command. ## Lexicon.swift This file provides the canonicalization lexicon type and its deterministic -builder. The lexicon is encoder Step 2's reference data: a flat map from -`stem(normalize(token))` to a concept identity. +builder. The lexicon is encoder Step 2's reference data. It is a flat map +from `stem(normalize(token))` to a concept identity. `CanonicalizationLexicon` is the pinned artifact shape: a version, a -language, and the entries map. The version is part of the agreement protocol: -two encoders must share it, or their concept bags can diverge. - -`LexiconBuilder.build(_:)` constructs the artifact from two public sources. -Wikidata supplies the concept identities: surface forms and aliases mapped to -Q-IDs, plus the property (P8814) that links WordNet synsets to Q-IDs. -WordNet supplies word senses in frequency order. The build considers every -candidate key with a three-tier rule, lower tier winning: first a Q-ID -reached through a WordNet sense of the word (this is where WordNet -disambiguates — "dog" maps to the animal, not the sausage, because the animal -is the primary sense); then a Q-ID known only from a Wikidata alias; and -last a `wn:` synset fallback, used only when no Q-ID exists for the concept. -Within each tier, explicit deterministic tie-breaks (sense rank, support -count, lowest Q-number) make the result order-independent: same inputs, same -artifact, byte for byte, on any machine. Keys are derived with the same -normalize-then-stem pipeline the runtime uses, so build-time and runtime keys -agree exactly. Only single-token surfaces become keys, because the runtime -looks up one token at a time. +language, and the entries map. The version is part of the agreement +protocol. Two encoders must share it, or their concept bags can diverge. + +`LexiconBuilder.build(_:)` constructs the artifact from two public +sources. Wikidata supplies the concept identities. It maps surface forms +and aliases to Q-IDs, plus the property P8814, which links WordNet synsets +to Q-IDs. WordNet supplies word senses in frequency order. The build +considers every candidate key with a three-tier rule, and the lower tier +wins. First comes a Q-ID reached through a WordNet sense of the word. This +is where WordNet disambiguates: "dog" maps to the animal, not the +sausage, since the animal is the primary sense. Next comes a Q-ID known +only from a Wikidata alias. Last comes a `wn:` synset fallback, used only +when no Q-ID exists for the concept. Within each tier, deterministic +tie-breaks apply in order. The tie-breaks are sense rank, support count, +and lowest Q-number. These make the result order-independent. Same +inputs and the same artifact yield the same output on any machine, byte +for byte. Keys are +derived with the same normalize-then-stem pipeline the runtime uses. So +build-time and runtime keys agree exactly. Only single-token surfaces +become keys, because the runtime looks up one token at a time. ## ConceptBag.swift -This file provides encoder Steps 1 through 3: `BagBuilder` turns a block of -text into a concept bag, the weighted map from concept identity to +This file provides encoder Steps 1 through 3. `BagBuilder` turns a block +of text into a concept bag, the weighted map from concept identity to occurrence count. `bag(_:lexicon:keep:)` makes one pass over the tokens. Each token is -normalized and stemmed, then looked up in the lexicon. The token is kept if -its word class is in the keep set (nouns and verbs by default) — or if it -resolves to a Wikidata Q-ID. That second condition is a deliberate -relaxation: part-of-speech taggers often mislabel named entities, and a -proper noun like "Everest" matters more to classification than most common -nouns. Deciding membership from the pinned lexicon keeps the relaxation -deterministic, because the lexicon is identical everywhere, while +normalized and stemmed, then looked up in the lexicon. The token is kept +if its word class is in the keep set, nouns and verbs by default. It is +also kept if it resolves to a Wikidata Q-ID. That second condition is a +deliberate relaxation. Part-of-speech taggers often mislabel named +entities. A proper noun like "Everest" matters more to classification +than most common nouns. Deciding membership from the pinned lexicon keeps +the relaxation deterministic. The lexicon is identical everywhere, while cross-platform proper-noun tagging is not. A kept token contributes its -concept identity when the lexicon has one, or its bare stemmed surface form -otherwise; a surface form can still match a signature carrying the same -string. +concept identity when the lexicon has one. Otherwise it contributes its +bare stemmed surface form. A surface form can still match a signature +carrying the same string. `bag(_:lexicon:keep:recordNovel:)` is the privacy variant. It produces a -byte-identical bag, but when `recordNovel` is `false`, novel tokens -encountered during tagging are not accumulated into the shared pool cache. -This is the seam the memory-capture path uses so that private text never -reaches the pool pipeline. `bag(_:lexicon:keep:taggerChoice:)` is the -estate-configuration variant: it threads the estate's explicit tagger choice -into Step 1 so the bag is built consistently with the estate's other -indexed content. +byte-identical bag. But when `recordNovel` is `false`, novel tokens found +during tagging are not accumulated into the shared pool cache. This is the +seam the memory-capture path uses. Private text never reaches the pool +pipeline this way. `bag(_:lexicon:keep:taggerChoice:)` is the +estate-configuration variant. It threads the estate's explicit tagger +choice into Step 1. The bag is then built consistently with the estate's +other indexed content. ## Code.swift This file provides the FDC code grammar: what makes a code string well-formed. -A code is a three-digit integer (000 through 999) with an optional decimal -extension of one to eight digits, such as `540` or `540.137`. Each extension -digit subdivides its parent into ten, so the extension carries leaf -resolution under a spine code. The eight-digit cap bounds the printable form -at twelve characters so tools can reason about column widths and string -lengths. +A code is a three-digit integer, from 000 through 999. It may carry an +optional decimal extension of one to eight digits, such as `540` or +`540.137`. Each extension digit subdivides its parent into ten. So the +extension carries leaf resolution under a spine code. The eight-digit cap +bounds the printable form at twelve characters. Tools can then reason +about column widths and string lengths. `Code.isWellFormed(_:)` checks the grammar and nothing else. Validity is -purely grammatical by design: a well-formed code is accepted, stored, and -round-tripped even if no term currently encodes to it, because whether a code -is "known" belongs to the caller's own known-code set, not to the grammar. +purely grammatical by design. A well-formed code is accepted, stored, and +round-tripped even if no term currently encodes to it. Whether a code is +"known" belongs to the caller's own known-code set, not to the grammar. `Code.integerBase(of:)` extracts the three-digit integer base of a -well-formed code, or `nil` for a malformed one. +well-formed code. It returns `nil` for a malformed one. ## FDCFrame.swift -This file provides the frame model — the versioned list of codes and labels — -and the ancestry math over code strings. +This file provides the frame model, the versioned list of codes and +labels, plus the ancestry math over code strings. -`FDCEntry` is one code with its heading label, preserved verbatim including -subject markers, because build tooling consumes the raw heading text. -`FDCFrame` is the versioned list. Ancestry is not stored anywhere; it is -derived from the code string itself. +`FDCEntry` is one code with its heading label. The label is preserved +verbatim, including subject markers, because build tooling consumes the +raw heading text. `FDCFrame` is the versioned list. Ancestry is not stored +anywhere. It is derived from the code string itself instead. -The derivation has two regimes, and the file's long comment explains why a +The derivation has two regimes. The file's long comment explains why a naive dot-split is wrong. The three-digit integer head is a Dewey-style -positional hierarchy read at the hundreds, tens, and units places: the parent -of `006` is `000`, and the parent of `510` is `500`. The decimal tail is a -per-segment hierarchy: the parent of `006.6` is `006`. `decimalParent(of:)` -implements both regimes as a pure function of the string. `ancestors(of:)` -walks parents upward and returns the chain root-first, so -`ancestors("006.6")` is `["000", "006"]`. `children(of:)` filters the frame's -codes by parent identity and sorts the result for deterministic output. +positional hierarchy. It reads at the hundreds, tens, and units places. +The parent of `006` is `000`. The parent of `510` is `500`. The decimal +tail, though, is a per-segment hierarchy. The parent of `006.6` is `006`. +`decimalParent(of:)` implements both regimes as a pure function of the +string. `ancestors(of:)` walks parents upward. It returns the chain +root-first, so `ancestors("006.6")` is `["000", "006"]`. `children(of:)` +filters the frame's codes by parent identity. It sorts the result for +deterministic output. ## CodeSignature.swift -This file provides the build-time signature assembly: how a code's -characteristic term set is put together before it ships as an artifact. +This file provides the build-time signature assembly. It shows how a +code's characteristic term set is put together before it ships as an +artifact. -Every code in the frame has three source texts: its label, the title of its -reference article, and the article body. `SourceWeights` pins their relative -importance — label 3, title 2, article 1 — because the label is the most -precise statement of what a code means and the article body the broadest. -`SignatureAssembler.merge(label:title:article:weights:)` combines the three -concept bags into one weighted term map. +Every code in the frame has three source texts: its label, the title of +its reference article, and the article body. `SourceWeights` pins their +relative importance. The label gets a weight of three, the title two, the +article one. The label is the most precise statement of what a code +means. The article body is the broadest. +`SignatureAssembler.merge(label:title:article:weights:)` combines the +three concept bags into one weighted term map. `SignatureAssembler.accumulateAncestors(ownTerms:ancestorsOf:)` then adds -every ancestor's own terms into each code's signature. A specific code such -as `540.1` thereby carries the full vocabulary of its lineage, so text that -talks about the general subject still supports the specific code during -descent. The result is a `CodeSignature` per code; the `fingerprint` field -stays `nil` until a later SimHash pass computes it, once the global concept -vocabulary is known. All of this is build-time machinery: the runtime ships -only the compact term sets. +every ancestor's own terms into each code's signature. A specific code, +such as `540.1`, thereby carries the full vocabulary of its lineage. So +text that talks about the general subject still supports the specific +code during descent. The result is a `CodeSignature` per code. The +`fingerprint` field stays `nil` until a later SimHash pass computes it, +once the global concept vocabulary is known. All of this is build-time +machinery. The runtime ships only the compact term sets. ## FDCMatcher.swift -This file provides encoder Steps 4 and 5: scoring a concept bag against the -signatures and descending the frame to the most specific supported code. It -is the heart of the classifier. +This file provides encoder Steps 4 and 5. It scores a concept bag against +the signatures. It descends the frame to the most specific supported +code. This file is the heart of the classifier. `FDCMatcher.init(lexicon:frame:signatures:stopThreshold:scoreMode:)` builds -the runtime index once. Every term across all signatures is interned: sorted -alphabetically and assigned a dense integer identifier. The hot path then -works entirely with integer keys, which profiling showed removes the dominant -cost (string hashing) on large imports. The identifier order is deliberately -the alphabetical order, so sorting integer sets visits terms in the same -sequence as sorting the original strings — floating-point sums are computed -in the same order and stay bit-identical to the pre-interning implementation. -The initializer also builds an inverted index (term to codes), document -frequencies for IDF weighting, and per-signature norms. - -`ScoreMode` selects how overlap becomes a score. `.raw` sums the bag counts -of shared terms. `.idf` weights each shared term by its rarity across -signatures (inverse document frequency), so distinctive vocabulary counts for -more than vocabulary that appears everywhere. `.cosine` and `.idfCosine` -additionally divide by a signature-size norm to remove the advantage of big -signatures. The shipped runtime uses `.idf`, which measurably improved code -selection over `.raw` on the v1.0 frame. - -`encode(_:)` and the two `encodeAnchor` overloads are the public entries; the -`recordNovel: false` variant suppresses pool accumulation for private text. -All delegate to one private method, `encodeFromBag(_:)`, which runs the -algorithm. Step 4 collects candidate codes from the inverted index, scores -each, and takes the best, breaking ties toward the lowest code in a fixed -scan order so the result never depends on hash ordering. Two guards protect -honesty: an empty or non-matching bag returns `nil` (UNRESOLVED, never a -guess), and when more than `maximumTiedWinnersForClassification` (4) codes -tie at the top score, the matcher also returns UNRESOLVED, because a wide tie -means the text's vocabulary is generic and any single winner would be -confidently wrong. Step 5 then walks down the frame: a child must carry at -least `stopThreshold` raw overlap to be a candidate (the cutoff is -mode-independent by design), the best-scoring child wins, and the walk +the runtime index once. Every term across all signatures is interned. Each +is sorted alphabetically and assigned a dense integer identifier. The hot +path then works entirely with integer keys. Profiling showed this removes +the dominant cost, string hashing, on large imports. The identifier order +is deliberately the alphabetical order. So sorting integer sets visits +terms in the same sequence as sorting the original strings. Floating-point +sums are computed in the same order this way. They stay bit-identical to +the pre-interning implementation. The initializer also builds an inverted +index, from term to codes. It builds document frequencies for IDF +weighting and per-signature norms too. + +`ScoreMode` selects how overlap becomes a score. `.raw` sums the bag +counts of shared terms. `.idf` weights each shared term by its rarity +across signatures, using inverse document frequency. So distinctive +vocabulary counts for more than vocabulary that appears everywhere. +`.cosine` and `.idfCosine` also divide by a signature-size norm. This +removes the advantage of big signatures. The shipped runtime uses `.idf`. +This measurably improved code selection over `.raw` on the v1.0 frame. + +`encode(_:)` and the two `encodeAnchor` overloads are the public entries. +The `recordNovel: false` variant suppresses pool accumulation for private +text. All delegate to one private method, `encodeFromBag(_:)`, which runs +the algorithm. Step 4 collects candidate codes from the inverted index. It +scores each and takes the best. Ties break toward the lowest code, in a +fixed scan order, so the result never depends on hash ordering. Two +guards protect honesty here. An empty or non-matching bag returns `nil`, +meaning UNRESOLVED, never a guess. The other guard is +`maximumTiedWinnersForClassification`, set to four. When more than four +codes tie at the top score, the matcher also returns UNRESOLVED. A wide +tie means the text's +vocabulary is generic. Any single winner would be confidently wrong in +that case. Step 5 then walks down the frame. A child must carry at least +`stopThreshold` raw overlap to be a candidate. This cutoff is +mode-independent by design. The best-scoring child wins, and the walk repeats until no child qualifies. The deepest reached code is the answer. -`dominantQID(_:)` computes the other half of the anchor: the -highest-counted Wikidata Q-ID in the bag, ties broken toward the lowest -Q-ID, or `nil` when the bag has none. This is "what the text is most about," -surfaced in the same pass so consumers fill an anchor without re-encoding. +`dominantQID(_:)` computes the other half of the anchor. This is the +highest-counted Wikidata Q-ID in the bag. Ties break toward the lowest +Q-ID. The result is `nil` when the bag has none. This value answers "what +the text is most about." It surfaces in the same pass, so consumers fill +an anchor without re-encoding. ## FDCRuntime.swift -This file provides `FDC`, the runtime facade that consumers actually call. -It owns the pinned artifacts and hides the matcher behind four static -functions. - -The three runtime artifacts — lexicon, frame, and compact signatures — load -once per process from the module bundle, together, or not at all. The -signatures loaded here are the compact form (code to term list) because the -runtime matcher uses only term membership; the weighted form remains a build -record. The loaded matcher is configured with `scoreMode: .idf` and the -pinned `stopThreshold` of 1 — a value that testing showed is inert on the -shallow v1.0 frame, pinned for the contract rather than for tuning. - -`FDC.encode(_:)` returns the lattice code or `nil` for UNRESOLVED. -`FDC.encodeAnchor(_:)` adds the dominant concept Q-ID; the +This file provides `FDC`, the runtime facade that consumers actually +call. It owns the pinned artifacts. It hides the matcher behind four +static functions. + +There are three runtime artifacts: the lexicon, the frame, and the +compact signatures. They load once per process from the module bundle. +They load together, or not at all. The signatures loaded here are the +compact form, a map from code to term list, because the runtime matcher +uses only term membership. The weighted form +remains a build record. The loaded matcher is configured with +`scoreMode: .idf` and the pinned `stopThreshold` of one. Testing showed +this value is inert on the shallow v1.0 frame. It is pinned for the +contract rather than for tuning. + +`FDC.encode(_:)` returns the lattice code, or `nil` for UNRESOLVED. +`FDC.encodeAnchor(_:)` adds the dominant concept Q-ID. The `recordNovel: false` overload is the privacy seam for memory capture. -`FDC.isAvailable` reports whether the artifacts loaded, and -`FDC.dataVersion` reports the signatures version so callers can record -provenance with every classification. `FDC.ancestors(of:)` exposes the -frame's ancestry walk so consumers get the chain without reaching into -`FDCFrame` directly. `FDC.label(for:)` returns a human-readable heading for -a code; for three-digit codes it walks up one level first, because leaf -labels at that depth are often multi-subject compounds while the parent -carries the cleaner single-topic heading that dashboards want. +`FDC.isAvailable` reports whether the artifacts loaded. `FDC.dataVersion` +reports the signatures version, so callers can record provenance with +every classification. `FDC.ancestors(of:)` exposes the frame's ancestry +walk. Consumers get the chain this way, without reaching into `FDCFrame` +directly. `FDC.label(for:)` returns a human-readable heading for a code. +For three-digit codes it walks up one level first. Leaf labels at that +depth are often multi-subject compounds. The parent carries the cleaner +single-topic heading that dashboards want instead. ## QIDClosure.swift -This file provides the Q-ID ancestor surface: given a Wikidata Q-ID, it -returns every ancestor reachable through instance-of (P31) and subclass-of -(P279) relations. - -Consumers use this to generalize a concept — from "dog" up through "mammal" -and "animal" — without any network access. The edge graph is a pinned, -offline Wikidata snapshot (`Resources/QIDClosureEdges.json`), produced by -build tooling and checked in like the other artifacts. The runtime never -queries Wikidata. - -`QIDClosure.ancestors(of:)` computes the transitive closure with an -iterative breadth-first walk over the edge map, excluding the queried Q-ID -itself, and returns the result sorted numerically by Q-number. The walk -dedupes visited nodes, so even a cycle in the source data would terminate. -Results are memoized per Q-ID for the life of the process; the memo is -guarded by a lock, while the walk itself runs outside the lock, and a race -between two threads computing the same Q-ID is harmless because both produce -the identical pure result. `isAvailable` and `dataVersion` mirror the -`FDC` facade: artifact-presence and provenance. +This file provides the Q-ID ancestor surface. It takes a Wikidata Q-ID. +It returns every ancestor reachable through two relation types. These +are instance-of, property P31, and subclass-of, property P279. + +Consumers use this to generalize a concept, from "dog" up through +"mammal" and "animal," without any network access. The edge graph is a +pinned, offline Wikidata snapshot, `Resources/QIDClosureEdges.json`. Build +tooling produces it, and it is checked in like the other artifacts. The +runtime never queries Wikidata. + +`QIDClosure.ancestors(of:)` computes the transitive closure. An iterative +breadth-first walk over the edge map does the work. It excludes the +queried Q-ID itself. It returns the result sorted numerically by Q-number. +The walk dedupes visited nodes, so even a cycle in the source data would +terminate. Results are memoized per Q-ID for the life of the process. A +lock guards the memo, while the walk itself runs outside the lock. A race +between two threads computing the same Q-ID is harmless. Both produce the +identical pure result. `isAvailable` and `dataVersion` mirror the `FDC` +facade: artifact presence and provenance. ## LexRank.swift -This file provides the article-reduction step used at build time: LexRank, -which selects the most central sentences of a long article before the +This file provides the article-reduction step used at build time. +LexRank selects the most central sentences of a long article before the encoder runs over it. -Code signatures are built from reference articles, but a whole article is -noisy; the useful vocabulary is concentrated in its most representative -sentences. LexRank finds them with eigenvector centrality — the idea behind -PageRank — over a sentence-similarity graph: sentences are nodes, and two -sentences connect when their stemmed term vectors are cosine-similar above a -threshold (0.1). Sentences that many other sentences resemble score high. - -`LexRank.reduce(_:sentences:)` segments the text, builds per-sentence -term-frequency vectors with the shared primitives, assembles the thresholded -similarity graph, and asks `SubstrateML.EigenvalueCentrality` for scores. It -returns the top N sentences (default 10) joined in their original order, or -the text unchanged when it is already short enough. This is build-time-only -machinery: it never runs on user devices, so it has no cross-platform -agreement constraint, though it is still deterministic. It is also the one -place the package uses its single dependency, SubstrateML. +Code signatures are built from reference articles. But a whole article is +noisy. The useful vocabulary concentrates in its most representative +sentences. LexRank finds them with eigenvector centrality, the idea +behind PageRank, over a sentence-similarity graph. Sentences are nodes. +Two sentences connect when their stemmed term vectors are cosine-similar +above a threshold of 0.1. Sentences that many other sentences resemble +score high. + +`LexRank.reduce(_:sentences:)` segments the text. It builds per-sentence +term-frequency vectors with the shared primitives. It assembles the +thresholded similarity graph. It then asks `SubstrateML.EigenvalueCentrality` +for scores. The function returns the top ten sentences by default, joined +in their original order. It returns the text unchanged when the text is +already short enough. This is build-time-only machinery. It never runs on +user devices, so it carries no cross-platform agreement constraint, +though it stays deterministic. It is also the one place the package uses +its single dependency, SubstrateML. ## Rust Port and Conformance -The `rust/` directory contains the second leg of the library: seventeen -source files mirroring the Swift implementations, from `tokenizer.rs` through -`fdc_runtime.rs`. The two legs share their pinned artifacts and are gated by -shared conformance fixtures in `rust/tests/fixtures/` — recorded -input-output pairs for normalization, tagging, and full FDC encoding that -both implementations must reproduce byte for byte. When you change either -leg, run both test suites; the fixtures are the contract. +The `rust/` directory contains the second leg of the library. Seventeen +source files mirror the Swift implementations, from `tokenizer.rs` +through `fdc_runtime.rs`. The two legs share their pinned artifacts. They +are gated by shared conformance fixtures in `rust/tests/fixtures/`. +These are recorded input-output pairs for normalization, tagging, and +full FDC encoding. Both implementations must reproduce them byte for +byte. When you change either leg, run both test suites. The fixtures are +the contract. diff --git a/packages/libs/LatticeLib/docs/OVERVIEW.md b/packages/libs/LatticeLib/docs/OVERVIEW.md index bd14625..c0ab46a 100644 --- a/packages/libs/LatticeLib/docs/OVERVIEW.md +++ b/packages/libs/LatticeLib/docs/OVERVIEW.md @@ -55,117 +55,127 @@ sources: ## What This Library Does -LatticeLib reads a piece of text and answers one question: what subject is -this about? It answers with a lattice code. A lattice code is a structured -classification code that gives a memory a place in an organized hierarchy of -subjects. For example, the text "the dog chased a ball in the park" encodes -to a code in the animals region of the hierarchy. +LatticeLib reads a piece of text. It answers one question: what subject is +this about? The answer is a lattice code. A lattice code is a structured +classification code. It gives a memory a place in an organized hierarchy of +subjects. Take the text "the dog chased a ball in the park." It encodes to +a code in the animals region of the hierarchy. MOOTx01 is an on-device AI memory system. It stores what an AI observes over -time and helps the AI recall it later. Every memory that enters the system -passes through LatticeLib. The code that LatticeLib assigns becomes part of -the memory's filing address, so memories about the same subject cluster -together even when they share no exact words. +time. It helps the AI recall that memory later. Every memory that enters +the system passes through LatticeLib first. LatticeLib assigns each memory +a code. That code becomes part of the memory's filing address. Memories +about the same subject cluster together this way. This holds even when the +memories share no exact words. -The engine inside LatticeLib is called FDC, which stands for Frame-Directed +The engine inside LatticeLib is called FDC. FDC stands for Frame-Directed Classification. A frame is a fixed, versioned list of classification codes -and their subject labels. FDC directs every piece of text to its best +and their subject labels. FDC directs each piece of text to its best position in that frame. ## The Problem It Solves -Two devices must file the same text under the same code. MOOTx01 estates can -federate, which means separate devices share and compare memories. If one -device filed "dog" under animals and another filed it under sports, shared -recall would fall apart. Classification therefore has to be deterministic: the -same input must produce the same output on every platform, every time. - -Cloud classifiers cannot promise this. They change without notice, they -require a network, and they see private text. LatticeLib instead ships every -piece of reference data it needs as pinned artifacts. A pinned artifact is a -data file that is built once, given a version, checked into the repository, -and never modified at runtime. The library runs entirely on the device. The -same text against the same artifact versions yields the same code — the -library's agreement property. +Two devices must file the same text under the same code. MOOTx01 estates +can federate. Federation lets separate devices share and compare memories. +Say one device filed "dog" under animals. Say another device filed it +under sports. Shared recall would then fall apart. Classification must be +deterministic for this reason. The same input must always produce the same +output. This must hold on every platform, every time. + +Cloud classifiers cannot make this promise. They change without notice. +They need a network connection. They see private text. LatticeLib avoids +all three problems. It ships every piece of reference data it needs as +pinned artifacts. A pinned artifact is a data file built once and given a +version. It is checked into the repository and never changed at runtime. +The library runs entirely on the device. The same text against the same +artifact versions always yields the same code. This is the library's +agreement property. The library keeps that promise across two independent implementations. A -Swift leg serves Apple platforms, and a Rust leg (in `rust/`) serves -everything else. Shared conformance fixtures — recorded input and output -pairs that both legs must reproduce exactly — gate every release. +Swift leg serves Apple platforms. A Rust leg, in `rust/`, serves everything +else. Shared conformance fixtures gate every release. These fixtures are +recorded input and output pairs. Both legs must reproduce them exactly. ## How It Works -Classification runs in five steps. The first three turn text into a concept -bag. A concept bag is a table that maps each concept found in the text to the -number of times it appears. The last two match that bag against the frame. - -Step 1 tags each word and keeps only nouns and verbs. Articles, adjectives, -and everything else carry little subject meaning, so they are dropped. A word -the tagger has never seen is called a novel token. Novel tokens are tagged by -a small, deterministic Hidden Markov Model, so this step never guesses -differently on different platforms. - -Step 2 canonicalizes each kept word. The word is normalized (odd Unicode -forms are folded to plain letters), stemmed (reduced to its root, so "dogs" -becomes "dog"), and looked up in the canonicalization lexicon. The lexicon is -a pinned artifact that maps roughly seventy thousand word roots to concept -identities. A concept identity is usually a Wikidata Q-ID — a public, -language-neutral identifier such as `Q144` for "dog." Synonyms collapse onto -one identity, which is what lets two devices agree. +Classification runs in five steps. The first three steps turn text into a +concept bag. A concept bag is a table. It maps each concept found in the +text to the number of times it appears. The last two steps match that bag +against the frame. + +Step 1 tags each word and keeps only nouns and verbs. Other words carry +little subject meaning. Articles and adjectives fall into this group, so +they are dropped. A word the tagger has never seen is called a novel +token. A small, deterministic Hidden Markov Model tags novel tokens. This +step therefore never guesses differently on different platforms. + +Step 2 canonicalizes each kept word. First the word is normalized. Odd +Unicode forms fold to plain letters in this step. Next the word is +stemmed, so "dogs" becomes "dog." The stemmed word is then looked up in +the canonicalization lexicon. The lexicon is a pinned artifact. It maps +about seventy thousand word roots to concept identities. A concept +identity is usually a Wikidata Q-ID. This is a public, language-neutral +identifier, such as `Q144` for "dog." Synonyms collapse onto one identity. +This is what lets two devices agree. Step 3 accumulates the counts. The result is the concept bag. -Step 4 scores the bag against code signatures. A code signature is the set of -concepts that characterizes one classification code, prepared ahead of time -from reference articles. Codes whose signatures share concepts with the bag -earn scores; rarer shared concepts earn more. The best-scoring code wins. If -nothing overlaps, or too many codes tie, the library returns UNRESOLVED -rather than guess. +Step 4 scores the bag against code signatures. A code signature is the set +of concepts that marks one classification code. It is prepared ahead of +time from reference articles. Some codes share concepts with the bag. +Those codes earn scores. Rarer shared concepts earn more. The best-scoring +code wins. Suppose nothing overlaps. Suppose too many codes tie instead. +The library then returns UNRESOLVED rather than guess. -Step 5 descends the frame. Starting from the winning code, the matcher walks -down to child codes while the text still supports the extra specificity, and -returns the deepest supported code. +Step 5 descends the frame. Starting from the winning code, the matcher +walks down to child codes. It keeps walking while the text still supports +the extra detail. It returns the deepest code the text still supports. ## How the Pieces Fit -Figure 1 shows the library's topology — its major parts and how data moves -between them. +Figure 1 shows the library's topology. It shows the major parts and how +data moves between them. ![Figure 1. Topology of LatticeLib](topology.svg) -*Figure 1. Topology of LatticeLib. Text flows left to right through the -shared text primitives into the concept bag, then through the matcher to a -code. Dashed regions mark the pinned artifacts and the offline novel-token -learning loop.* +*Figure 1. Topology of LatticeLib. Text flows left to right. It moves +through the shared text primitives into the concept bag. From there it +moves through the matcher to a code. Dashed regions mark the pinned +artifacts and the offline novel-token learning loop.* The runtime entry point is the `FDC` enum. Consumers call -`FDC.encodeAnchor(text)` and receive two values: the lattice code and the -dominant concept Q-ID of the text. Loading of the three runtime artifacts -(lexicon, frame, signatures) happens once per process. - -The shared text primitives — `Tokenizer`, `Normalizer`, `Stemmer`, and the -word-class tagger — serve both the runtime encoder and the build-time tools. -Build-time tools (`LexiconBuilder`, `SignatureAssembler`, `LexRank`) produce -the pinned artifacts; they never run on user devices. - -One slow feedback loop improves the tagger over time. Novel tokens accumulate -in a local cache. When fifty gather, they are written to a local pool -directory. A reducer later merges qualifying tokens into a writable copy of -the word-class table, and the running process adopts the new table with an -atomic swap. Classification stays deterministic because determinism is always -relative to a table version, and the version advances only at that swap. - -Privacy is engineered into the seams. When the text being classified is -private memory content, callers pass `recordNovel: false`, and no token of it -is ever written to the pool. The pool itself is off by default; it activates -only when a host sets the `LATTICE_POOL_DIR` environment variable. +`FDC.encodeAnchor(text)`. This returns two values: the lattice code and the +dominant concept Q-ID of the text. The three runtime artifacts are the +lexicon, the frame, and the signatures. Each loads once per process. + +Shared text primitives serve both the runtime encoder and the build-time +tools. These primitives are `Tokenizer`, `Normalizer`, `Stemmer`, and the +word-class tagger. Build-time tools produce the pinned artifacts. These +tools are `LexiconBuilder`, `SignatureAssembler`, and `LexRank`. They never +run on user devices. + +One slow feedback loop improves the tagger over time. Novel tokens +accumulate in a local cache. When fifty gather, the cache writes them to a +local pool directory. A reducer later merges qualifying tokens into a +writable copy of the word-class table. The running process then adopts +the new table with an atomic swap. Classification stays deterministic +because determinism is always relative to a table version. The version +advances only at that swap. + +Privacy is built into the seams. Sometimes the text being classified is +private memory content. Callers then pass `recordNovel: false`. No token +of that text is ever written to the pool. The pool itself is off by +default. It activates only when a host sets the `LATTICE_POOL_DIR` +environment variable. ## What Ships in the Package -The package ships the Swift sources, the Rust port in `rust/`, and the pinned -artifacts in `Sources/LatticeLib/Resources/`: the frame (1,075 codes), the -code signatures (1,071 entries), the lexicon (about 70,000 entries), the -Q-ID ancestor graph (about 39,000 nodes), the trained tagger model, the -word-class table, and the stemmer conformance corpus. Each artifact carries -its own version string, and the library reports the versions it used, so -every classification is reproducible provenance included. +The package ships three things: the Swift sources, the Rust port in +`rust/`, and the pinned artifacts. The artifacts live in +`Sources/LatticeLib/Resources/`. They are the frame (1,075 codes), the +code signatures (1,071 entries), and the lexicon (about 70,000 entries). +They also include the Q-ID ancestor graph (about 39,000 nodes), the +trained tagger model, the word-class table, and the stemmer conformance +corpus. Each artifact carries its own version string. The library reports +the versions it used. Every classification is reproducible this way, with +its provenance included.