[AAASM-5529] ✅ (tests): Bind quick-start enforcement claims to their proving controls - #365
Merged
Merged
Conversation
… proves it docs/02-quick-start/index.md §2-§4 carry the Node quick-start's enforcement claims, and nothing connected them to the controls that prove them. A claim could be added, reworded, or left standing after the behaviour beneath it changed, and no gate would notice. Neither side of the comparison is a transcribed constant checking another transcribed constant. The claim sentences are read from the document; the control ids are extracted from each control file's TypeScript AST as "<file> :: <describe> > <it>". Ids are file-qualified because the claims are not all proved in one place — the deny claims come from the negative controls, the auditSink claim from the AAASM-5681 disposition suite. The error class is derived from constructor.name on the value a real deny actually throws, and is deliberately not imported. Importing it would make a rename a type error: red, but it aborts before the assertion meant to catch the rename can run. That is the inverted-order defect the round-1 review of this ticket found in all three SDKs. The unbound-sentence check found two claims on its first run that reading had missed — that initAssembly warns and reports auditSink on the context, and that a denied call throws instead of executing. Both are now bound. The gateway-reachability claim is registered as unproven, naming AAASM-5663. Refs AAASM-5529
test-matrix.yml sets paths-ignore on docs/** and **/*.md, so a PR that only rewords a quick-start claim runs no tests — which is exactly the change the claim-binding gate exists to catch. publish-docs.yml already triggers on docs/**, so the gate runs there. The step's comment states the limit rather than leaving it implied: publish-docs is not one of the workflows ci-success.yml aggregates, and branch protection on main declares no required status checks at all (verified against the API). This step makes the drift visible on the PR; it does not by itself block a merge. Closing that gap is AAASM-5677's scope. Refs AAASM-5529
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…at can fail
Review broke the central promise in under a minute. Four defects, all here:
F1 — the gate asked whether the region CONTAINED a binding's quote. The
reviewer replaced the true claim with its opposite ("a denied call is
recorded for review while the tool still runs to completion"), left the
bound fragment at :105 intact, and the gate stayed green at 28 passed.
Quotes are now whole sentences compared with ===, and exactly one binding
may match a sentence. Re-run under the same attack: 2 failed.
F2 — "with either gateway client this SDK ships it is discarded, not
retained" was bound to a negative control that hands the event to a FIXTURE
client, which retains it. Flipping both shipped clients to caller-supplied
left that control green, so the binding could not fail when the claim
became false. Repointed at the audit-sink suite, which drives the shipped
client against a downstream boundary probe with a reachability positive
control beside it. Under the same flip that control now fails 9 of 13.
F4 — `kind` was written and never read: dead decoration a reader would
assume was enforced. Removed; every claim now needs controls or a ticketed
reason.
F6 — only three regions were scanned. The scan now covers the whole document
minus a named section allow-list whose entries must still resolve to real
headings.
Front matter is stripped and fenced blocks become paragraph breaks rather
than spaces: a fence replaced by a space glued the sentences either side of
it into one, which would have let a binding cover two claims at once.
Two claims that surfaced once the scan widened are now bound: auto-start
being opt-in, and the page's own "from nothing to a governed agent" promise.
Refs AAASM-5529
F8 — publish-docs.yml's paths omitted tests/**, so a PR that renamed a control or changed which error the deny throws would not re-run the gate that exists to notice. Added tests/**, src/gateway/** and src/wrappers/**. F9 — the step comment said the snippets are excluded from "every tsconfig". That is false: the base tsconfig.json does include them. The substantive point stands for a different reason — no CI job type-checks with that config, since pnpm typecheck runs tsconfig.test.json — so the comment now says that instead. Refs AAASM-5529
The four Windows legs of test-matrix failed on "no enforcement sentence is
unbound" while Linux and macOS passed. Cause: git checks the doc out with
CRLF on Windows, so split("\n\n") finds no paragraph break and the whole
section collapses into one "sentence" that matches no binding.
Proven load-bearing rather than assumed: with the normalisation removed and
a CRLF copy of the document, the suite fails 2 of 38 — the same signature CI
reported. With it, CRLF and LF both pass.
The same normalisation was added to the Python and Go gates, where the bug
is latent because neither repo's CI runs Windows.
Refs AAASM-5529
… no keyword filter
R1 — review appended three plain sentences to the quick-start, the last of
them "Tool bodies always execute; the policy result is recorded alongside
them" — the negation of the product — and this gate stayed green at 38
passed. None used any of the 21 vocabulary terms. Widening 3 -> 21 closed
the instance, not the class: a keyword allow-list cannot be completed,
because whoever adds the claim picks the words after reading the list.
Every one of the document's 55 sentences must now be bound to a control or
allow-listed by exact text. The vocabulary gates nothing; it is a severity
hint in the failure message and the trigger for a stricter allow-list rule —
waving through a sentence that reads like a claim costs a written
justification, not a category.
R3 — section exclusions are gone. "## Next steps" was a black hole: the
guard checked the heading existed and said nothing about its contents, so a
claim inserted there was never scanned. A positive control now asserts that
section is in the scan.
R4 — HTML and MDX comments are stripped, as fences already were, because a
reader cannot see them. Leaving them in let a bound claim be commented out
of the rendered page while the gate still counted it. Verified with the MDX
{/* */} form specifically.
R5 — an unprovenReason may not name AAASM-5529, the ticket this file
implements; such a pointer resolves to a closed issue on merge. Guard proven
able to fail by repointing a real binding at 5529 and watching it go red.
R6 — the splitter drops bare list markers, treats "!" as text, handles "?",
and splits list items and table rows as units.
Refs AAASM-5529
…and count occurrences R3-1 — the two-tier rule was inverted relative to risk. A written justification was required only when the sentence matched the enforcement vocabulary, i.e. only for sentences an author had already been warned about. A sentence that EVADED the vocabulary — the entire reason the scan was inverted — passed with a bare category constant. The bare constant is now available only to structurally non-prose lines (MDX tags, admonition delimiters, tab captions, bare link items); every other entry carries a written justification unique to that sentence. R3-3 — occurrences are an array of (text, section), not a record keyed by sentence. Keying collapsed duplicates before anything counted them, so "matched === 1" could only ever be 0 or 1 while the message printed a count as though more were reachable. Measured: duplicating a bound sentence into a "## What NOT to expect" section now fails with 'it matched 2 (sections: ["## 4. What to expect","## What NOT to expect"])'. Section attribution was last-write-wins for the same reason and is now per-occurrence. Splitter — closing markup between a terminator and the following space is kept WITH its sentence, so "**Deny.**" is its own unit rather than running into the claim it introduces; binding the glued pair covered both. A backtick is excluded from that trailing class so inline code is not read as a sentence end. The document now yields 60 distinct sentences, and the bindings were re-derived from the parser's own output rather than by hand. Accepted residual, stated rather than implied: "written justification" is enforced as non-empty, at least 40 characters, and distinct across entries. No gate can tell a justification from noise; length and uniqueness only make an empty gesture and a copy-paste visible. Refs AAASM-5529
… MDX scaffolding R4-3 — the contrastive-conjunction rule existed only in Go. I relayed it as if it were universal; it was not, and the reviewer's hybrid payload was still green here. Ported: an allow-listed sentence may not contain a contrastive conjunction, applied to every entry rather than gated on a marker phrase in the justification. Measured: the payload now fails. "so" is excluded from the conjunction list — it is consequential rather than adversative — and both attack payloads are still caught, one using "because" and one "but". R4-1 — the structural pattern was the new bare-constant route. It asked whether a sentence STARTED with structure, not whether it was ONLY structure, so link items and HTML-tag-prefixed lines were waved through on their first characters while their anchor text — rendered prose a reader sees — went unexamined. It covered 10 entries. Fully anchored now, admitting only lines that are entirely MDX scaffolding. Measured: both the link-anchor and the <b>-prefixed payloads passed before and fail now. One documentation change, and the only prose edit in this branch: the pre-1.0 stability note at docs/02-quick-start/index.md:55 was split at its "but" into two sentences. Meaning is unchanged. Splitting is the remedy the contrastive rule exists to induce, and the sentence sits outside every generated block — `pnpm run generate:docs-metadata` still round-trips with no diff beyond that edit. Refs AAASM-5529
I excluded "so" from the contrastive list on the grounds that it is consequential rather than adversative. The grammar was right and the question was wrong: the risk "so" carries is POLARITY CHANGE between clauses, and on that axis it is the highest-risk connector, not the lowest. "We do not do X but Y" is a concession; "we do not do X so Y covers it" is a REASSURANCE, and reassurance is the register documentation over-claims in. The payload that was green in all three: "Network-layer interception is not enabled by default, so the in-process adapter verifies every outbound request before it leaves the host instead." The second clause is an unbound affirmative capability claim, and false — no SDK adapter inspects outbound network requests, which is why AAASM-5766 exists. "so" is now flagged only when the clause before it contains a negation and the clause after it does not. That is the one shape a limitation-plus- reassurance takes, and the payload is the only negative-to-positive case among the live sentences. Measured across the three repos: 7 live allow-listed sentences contain " so " (python 2, go 3, node 2) and the refined rule flags none of them, while the payload fails in all three. My earlier count of two was python-only; the reviewer's six was also short, and node's figure needed reading ALLOWED at runtime because a regex over the source recovers only 31 of its 50 keys. Refs AAASM-5529
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Target
Task summary:
Binds every sentence of the Node quick-start to a control that proves it, or to an explicit allow-list entry saying why it makes no claim.
Task tickets:
Key point change:
R3-1 — the justification rule was inverted relative to risk. A written justification was required only when the sentence matched the enforcement vocabulary — only for sentences an author had already been warned about. A sentence that evaded the vocabulary, which is the entire reason the scan was inverted, passed with a bare category constant. The bare constant is now available only to structurally non-prose lines (MDX tags, admonition delimiters, tab captions, bare link items); every other entry carries a justification unique to that sentence.
R3-3 —
matched === 1was structurally unreachable above 1. The scan was keyed by sentence, so duplicates collapsed before anything counted them. Occurrences are now an array of(text, section). Measured:Splitter. Closing markup between a terminator and the following space is kept with its sentence, so
**Deny.**is its own unit rather than running into the claim it introduces — binding the glued pair covered both. A backtick is excluded so inline code is not read as a sentence end. The page now yields 60 distinct sentences, and the bindings were re-derived from the parser's own output rather than by hand, so no quote is a transcription.What "green" means here, and what a written justification is not
Green means the page's unproven claims are enumerated, not that the page is true. The count is the most useful number this PR produces:
An unproven claim names the ticket that will resolve it, and may not name AAASM-5529 — a self-reference resolves to a closed issue on merge.
Accepted residual, stated rather than implied: "written justification" is enforced as non-empty, ≥40 characters, and not byte-identical to another entry. That is all it is. The check cannot separate a justification written per sentence from one templated with a substitution — one differing token satisfies distinctness — and no check can separate either from noise. It is worth having only because it makes the two cheapest ways of waving something through, an empty gesture and a verbatim copy-paste, visible in review. Uniqueness here should not be read as implying bespoke.
The three residuals — the honest boundary of what this gate buys
sois worth catching because it is ordinary prose that reads as reassurance — the register documentation over-claims in — but chasing;,—andandwould flag large amounts of benign text for very little gain. The rule catches the shapes that hide a claim, not every shape that could.so— corrected from the previous revisionI had excluded
sofrom the conjunction list as consequential rather than adversative. The grammar was right and the question was wrong: the risk is polarity change, and"we don't do X so Y covers it"is a reassurance — the highest-risk shape, not the lowest.sois now flagged when the clause before it is negated and the clause after is not.Measured across the three repos: 7 live allow-listed sentences contain
" so "(python 2, go 3, node 2); the refined rule flags none of them and catches the payload in all three. My earlier count of two was python-only.R4 additions
[Every tool request is permitted to proceed and its outcome captured for later analysis](x.md)passed. The pattern is fully anchored now and admits only lines that are entirely scaffolding."A LIMITATION disclaimer…", the wording that shipped one round earlier, evaded it.sois excluded from the conjunction list as consequential rather than adversative; both attack payloads (because,but) are still caught.Effecting Scope
Test-only plus CI trigger paths. No
src/code is touched.Where it runs, and the limit of that
test-matrix.ymlsetspaths-ignoreondocs/**, so a docs-only PR runs no tests; the gate also runs inpublish-docs.yml, which triggers ondocs/**and now ontests/**too, so weakening the gate re-runs it.That does not make it merge-blocking.
publish-docsis not inci-success.yml's aggregate, andrequired_status_checksis NONE. Visible on the PR; not blocking. AAASM-5677's scope.Validation
Each mutation asserts it applied before measuring; each restore is sha256-verified:
vitest run tests/quickstart-claim-bindings.test.tspnpm test(full suite)pnpm typecheck/pnpm lint/prettier --checkAlso verified against a CRLF checkout, since the four Windows legs caught that in an earlier revision.