Repository navigation
Add negative controls for the conformance runner - #293
Open
astrogilda wants to merge 4 commits into
Open
astrogilda wants to merge 4 commits into
astrogilda wants to merge 4 commits into
Conversation
Issue 53 lists what a conformance suite has to get right. Requirements 2, 3, 4, 5, 7, 9 and 11 are about the gate rather than any one Guardian, so each gets a control here: a stub Guardian or a corpus the runner must refuse, and seven runner mutants that at least one control must catch. The runner is standard library only and runs as a child process. It posts signed envelopes over HTTP and scores by section 17.1 code, never by message text. The same vectors run against the reference Guardian, where the four known gaps (signature, missing signature, repeated request_id and the skew window) are strict expected failures that cite issue 184. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
The test job in deploy-pages.yml has no Bun, so the reference Guardian group in tests/test_negative_controls.py skips there. The agt job already installs Bun and the tree's dependencies, so it now runs that file too, with ACS_REQUIRE_REFERENCE_GUARDIAN set so the group fails rather than skips when the Guardian cannot start. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
The runner's key derivation matches the parameters pull request 272 adds to section 10, so the docstring now says so instead of describing them as this runner's own choice. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
Each strict xfail reason now names the section whose MUST the reference Guardian fails, so the marker reads as a known failure rather than as an optional case. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
astrogilda
requested review from
GangGreenTemperTatum,
afogel,
bar-capsule,
mamicidal and
rocklambros
as code owners
October 17, 2026 15:06
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
@rocklambros, this is the tests/ file promised on #53, for the suite that #19 gave the Spec workstream of @bar-capsule and @afogel.
tests/test_negative_controls.pycovers requirements 2, 3, 4, 5, 7, 9 and 11 with subjects a small runner must refuse, like an allow-everything Guardian. Its MUTANTS list holds seven single-line weakenings of that runner, one per guard, and a control catches each.The reference Guardian fails four section 10 MUSTs today, on signatures, replay and the skew window, and those vectors are strict xfails citing #184.
Which issue does this implement
Part of #33; refs #53.
I tested this
At 9ecc0bf, pytest gives 939 passed and 7 skipped, and the strict mkdocs build passes. The skips are the seven reference Guardian cases, and the Bun job agt in reference-implementation.yml runs them: 3 passed, 4 xfailed.