Repository navigation
test(conformance): clean-room harness proving the public packages against a reference oracle (P10) - #32
Merged
Conversation
…es against a reference oracle (P10) Five proofs run from the public package surface only (`bun run conformance:check` after `bun run build`), over vectors generated from the reference implementation @ 31b62f3: parse and classify over the nine parser-contract cases and the golden ISRC; anchor build and term id; createAtomsWithUris encode, decode and receipt linkage; ladder differential (same classification, same input, same IID and rung) between the declarative ladders and the reference walker; canonicalization parity against the spec corpus. Every disagreement is an explicit entry in conformance/known-divergences.ts with the side that must change, a one-sentence reason, the retirement condition and both output fingerprints; a listed case asserts the divergence, so an entry fails as soon as the two sides agree. Classes: typed Wikidata rungs the train does not carry at this base (retire with the typed-wd stack); identity-field order in two-field gen1 recipes (reference adopts the public builder); public-builder defects carried as follow-ups (sameAs evidence and derived recipe values such as geo7 and textHash omitted from identity envelopes; over-cap URL fallback); the no-identifier fallback envelope the reference builds and the public builder refuses (a ruling); parse strictness where the reference is laxer than the corpus grammar; the odd-length hex encode case. Not a workspace and in no tarball; no package source changes. 981 pass, 23 reasoned todos for capabilities the public surface does not expose, 0 fail.
…o proof can pass vacuously (P10 fix round) Review found two ways the harness could stay green while proving nothing: a generic undefined-to-null normalisation let a malformed parser result (an empty object) pass as a rejection and let an undefined train value equal an explicit null in the oracle; and the loaders validated only the provenance hash, so removing 129 of 131 canonicalization vectors, duplicating an id, or pointing a divergence entry at a vector that no longer exists all went unnoticed. The recorded agreement flag on canonicalization vectors was never read. - Comparison helpers assert the shape of every public result before any field is projected; an unexpected shape fails the case. The only remaining undefined mapping is the documented rejection sentinel of two public APIs, mapped once at their adapters with a shape check and a comment. - conformance/vectors/manifest.json (committed data generated from the oracle) records the case count and sorted ids per proof; loaders assert count and id-set equality, reject duplicates, and every divergence entry must resolve to a loaded case. Each canonicalization corpus is asserted present with its count; each vector's agreement flag is checked against the computed reference-vs-corpus relation and the oracle's disagreements list. - harness-integrity.test.ts carries the reviewer's mutations as permanent regressions through the same helpers the proofs use; the README's runtime enumeration table and the excluded typed-wd accounting (34 anchor and ladder vectors plus 36 parser-corpus cases) are asserted, not hand-written. Gates: conformance:check 1004 pass, 23 reasoned todos, 0 fail across seven files; biome clean; pack:dry-run carries no harness file; the 110 divergence entries and their fingerprints are unchanged.
…ng, not a reference defect The three parse entries said the reference parser must tighten to the corpus grammar. An audit of the reference's consumers showed its lenient parse is deliberate: raw values are canonicalized after parsing, and a strict parser changes behaviour at several write and identity paths. The entries now name the pending ruling and the two ways it can retire them; fingerprints are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Child PR of the v1.1.0-alpha umbrella (#15).
Current (train)
Nothing on the train runs the published packages against the reference implementation end to end, so the repo cannot show where they disagree.
Need
A root harness that proves the public packages against the reference and records every disagreement as data that cannot go stale. No package source changes, no new dependency, nothing in any tarball.
Reconciliation
Five proofs run under
bun run conformance:checkafterbun run build, over the built public exports only:buildIidAnchorbytes, thencalculateAtomId.createAtomsWithUriscalldata decoded, hashed, and linked across synthetic event logs.deriveIntuitionIdagainst the reference walker, structure included, for 37 classifications.The oracle is JSON recovered from reference implementation @ 31b62f3, checked in under
conformance/vectors/; tests never load reference code. A committed manifest pins each vector file's count and ids, so a thinned corpus fails at load.Every disagreement is an entry in
conformance/known-divergences.ts: the side that changes, a reason, the retirement condition, and fingerprints of both outputs. A listed case asserts the divergence, so the entry fails the moment the sides agree or either output drifts. 110 entries, byside:Typed Wikidata identity (#28 to #30) is excluded at this base: 70 vectors are held out (7 anchor, 27 ladder, 36 parser), and the 59 ladder entries record bare
wdfiring where the reference skips a dormant typed rung; both reconcile when that stack lands. The 23test.todocases name the missing public capability: 14 candidate overrides, 9 unexported validators.Held out
Evidence
bun run conformance:check: 1004 pass, 23 todo, 0 fail (1027 tests, 7 files): 725 agreements and 110 divergences across the five proofs, plus 23 integrity self-tests and 146 from the existingiidconformance file.bunx biome check conformance: clean.bun run pack:dry-run: no harness file in any tarball.registry:checkandguard:supply-chainpass. No file underpackages/changes, so typecheck is not exercised.Review notes
Base is
update/v1.1.0-alpha. Readconformance/known-divergences.tsfirst, thenconformance/README.mdfor how each proof works and how an entry retires. Runbun run build && bun run conformance:check. Merge before the typed-wd stack; rerun after each of #28 to #30 and retire entries that start agreeing, since a child PR that changes output trips the matching entries by design.