feat(pr-workflow): add the /mms-evidence orchestrator, runners, and publish gate - #84
Draft
MajorLift wants to merge 62 commits into
Draft
feat(pr-workflow): add the /mms-evidence orchestrator, runners, and publish gate#84MajorLift wants to merge 62 commits into
/mms-evidence orchestrator, runners, and publish gate#84MajorLift wants to merge 62 commits into
Conversation
Replaces #56, whose skill.md shipped frontmatter with no body at all — the installer's bodyAfterFrontmatter() returned empty, so an agent loading it got a description and six references it had no instruction to read. pr-validate: for a PR's specific falsifiable claim, name the observation that would prove the claim false, gather it, and publish it into the PR body. Drives the AEP harness (visual_validation, perf_validation) as the primary engine, backed by a catalog of complementary lanes, a trustworthiness gate that rejects vacuous passes, and a publishing flow with an audience-reachability rule for re-hosted artifacts. falsifying-test: the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. Its own falsifier is a base-commit failure for the wrong reason (import error, missing fixture, unrelated red), which looks identical in an exit code and proves nothing. pr-validate calls it as the engine behind lane B3. hooks/pr-evidence-gate.py enforces the trustworthiness gate at emit time, blocking an outward-facing write whose body carries an unbacked verdict, an untracked deferral, a CI restatement, a bare or truncated identifier, a mutable ref, a dump-as-resolver, a link-only or data-only exhibit, or a step waiver. It polices `gh api` body writes as well as the porcelain, since a PATCH to a comment is the same publish with a different spelling.
Over-scrubbed. The audience is the MetaMask org, and the ticket is the evidence for the claim the item makes — that the LaunchDarkly provisioning blocker covers only the prod-flag half of that lane. Without it the example is an assertion. The scrub line is personal references, not org-internal ones.
MajorLift
changed the title
pr-validate and falsifying-test skills
MajorLift
marked this pull request as draft
July 30, 2026 14:03
`pr-validate`'s other engines are placed by subject — `react-render-proof` in `performance`, `memory-leak-hunt` in `stability`, `supply-chain-audit` in `security`. This one was placed by its caller instead. Writing a test that fails on the base commit and passes on the branch is a testing technique, and `testing/` already holds techniques of that kind (`e2e-flakiness-patterns`, `test-layer-placement`, `performance-testing`), while `pr-workflow/` is uniformly PR-lifecycle stages. Both references to it are by name rather than path, so nothing needed updating.
MajorLift
marked this pull request as ready for review
July 30, 2026 18:15
`D6` covers claims where a PR hand-writes an artifact that restates an existing source — a type, schema, vendored constant, or checked-in policy. Both arms sit at the same commit and differ by a substitution rather than a ref, so there is no build or merge boundary to confound the result.
Contributor
Author
Context budgetWhat this PR costs an agent, measured from an install rather than read from the diff. Three tiers, and only the first is unavoidable.
Frontmatter is the only tier paid unconditionally — every agent loads it on every run once the skill is installed, used or not, because it is what the agent reads to decide relevance. The 28 skills across the eleven open skill PRs sit at a median of ~1,716 tokens selected and ~1,860 with references followed. All are within the 1,536-character description budget. Selected is paid only when the agent picks the skill. + refs & knowledge is the ceiling if every bundled reference is then read; it is a worst case, not an expectation. Method
These figures are pinned to the commit above and drift on every push; #96 tracks automating them. |
`pr-validate` cost ~9,059 tokens the moment an agent selected it — 5.6× the median of the 26 skills in the open PRs — and a quarter of that was the AEP local-run procedure, which most validations never touch. The skill's own Sufficiency section tells you to prefer a lighter lane; the body charged you for the heavy one regardless. Preflight, run mechanics, and teardown move to `references/aep-local-run.md`, which the body already linked twice and which did not exist. The link was dangling — the same defect class the `knowledge/` guard catches, on a path nothing checks. Publishing keeps the decisions (surface by ownership, post complete once, falsifier-forward, scrub) and points at `references/evidence-publishing.md` for the mechanics it already documents in full. Body 34,431 → 25,736 bytes, so a selected skill is ~6,812 tokens installed rather than ~9,059. Nothing is lost: it sits behind the same progressive disclosure boundary as the other seven references, read when an AEP run is actually warranted. Description trimmed 1,147 → 885 characters. It was over the 1,024 ceiling that #47 enforces, so it would have failed that check on merge.
Trimmed to 885 characters to fit a 1,024 budget that turned out to be unverified — no operator observed rejects or truncates a longer description, and several over 1,024 load today. Back to 1,147, under the 1,536 budget. The description is the discovery surface, so the 262 characters were trigger cues: the subcommand list and the phrasings that route a request here rather than to another skill. Cutting them made the skill harder to select, which is a functional loss and not a cosmetic one. The body restructure is unaffected — that removed duplication behind a reference, which costs nothing at selection time.
2af1db1 carries the reasoning but not the change: its `git add` was chained into a command the commit guard rejected, so the commit was created from an already staged tree and landed empty — same 885-character description as its parent. This applies it. Back to 1,147 characters, under the 1,536 budget. The 262 characters are trigger cues — the subcommand list and the phrasings that route a request here rather than to a sibling skill — so losing them made the skill harder to select, which is a functional loss rather than a cosmetic one.
Two of the three assumptions in the old name were false. The skill runs in the inner loop against uncommitted changes, and on a symptom with no claim and no PR at all, so `pr-` narrowed it to one of three modes. `validate` named an activity where the deliverable is an artifact, and read as a checklist exercise — the posture the skill spends its opening section arguing against. `evidence` names what it produces, and covers a refutation as naturally as a confirmation. The verdicts are proven, refuted, and inconclusive; a name promising proof would make two of those read as failure. The description is rewritten rather than search-replaced. It now states all three modes, since the old one described only the PR case and so under-selected for the other two, and it names the trigger as `mms-evidence` — the form the installer actually emits. Eleven skills across the repo still promise the unprefixed `/<name>` in their descriptions while installing prefixed; this corrects the one being renamed.
MajorLift
changed the title
pr-validate and falsifying-test skillsevidence and falsifying-test skills
MajorLift
changed the title
evidence and falsifying-test skills/mms-evidence orchestrator and falsifying-test skill
The engine table still routed the memory-leak category to `/memory-leak-hunt`, which was renamed to `/memory-leak` — the rename missed the PR that performed it. Two `[[snake_case]]` entries were wiki links to a private authoring vault; they render as literal brackets here and resolve for no reader. One had a real counterpart in `references/` and now links it; the other pointed at a file that does not exist and is dropped rather than left dangling. `falsifying-test` named its evidence categories as `B3` and `B7`. Those are addresses into `evidence-catalog.md`, not names, so both now use the category name and link the catalog.
`race-condition-proof` is now `race-condition-repro` and `react-render-proof` is now `react-render-delta`. Both are named here as engines, in the catalog, the engine table, and the sibling reference — none of which the renaming branches can reach.
The catalog is 41 full lane specs with no summary, so a reader arriving from a link lands mid-document with no way to see the shape of it. Adds a generated "Lanes at a glance" table: family, count, and every lane id with its title. Two placement bugs: - `C9` sat inside the `# D. Build output` section, so scanning family C missed the lane backing `memory-leak`, and scanning D found a stranger. - `B7` sat between `B3` and `B4`. All 41 lanes now read in order. Also removes six pointers into a private authoring vault — four inline `exogram` references and two full `exogram-daemon/...` paths. They resolve for no reader of a public repository. Every substantive claim they were attached to is kept; only the dangling pointer is dropped. `memory-leak-hunt` updated to `memory-leak`.
The catalog had 41 lanes and none for how long a build takes. Family D covered build *output* — size, chunks, policy, permissions, variants — and `C5` covers runtime, so a toolchain change had no category to publish into even though a skill for measuring one is specified in #102. `D7` is the dev-loop half: paired A/B, cold and warm as separate numbers, with the four confounds that each return a favourable result when uncontrolled — warm cache leaking into the cold arm, worker-pool startup amortised away, core count that does not transfer off the measuring machine, and watch rebuilds presented as cold builds. `G6` is the CI half, in family G because its dominant confound is a process one: `get-requirements.yml` skips jobs when build output matches base, so a measured speedup is often a skipped job. Family D is retitled from "Build output" to "Build", since it now covers both. The at-a-glance index is regenerated rather than hand-patched — it is derived from the headings, and hand-editing it is how it drifts.
This was referenced Jul 31, 2026
An 18-comment trial run met none of this skill's output requirements. The cause was structural, not behavioural: `VALIDATION_RUN_START` and the in-situ capture rule occurred zero times in skill.md and only in a reference costing ~5x the body to open, described there as "image re-hosting and the privacy scrub". The publish gate checks none of them either. All three layers failed open. Moves six non-negotiables and the canonical output shape into the body, where they load with the work: 1. Ship an artifact the reader can check without trusting you. Pasted terminal text is indistinguishable from invented terminal text — running the check justifies your belief, not the reader's. 2. `proven` requires execution; reading gives shape, never power. Run arm B against your own probe: one that passes with the mechanism deleted is measuring something else. 3. No "what would close it" section — that is an unfinished run formatted to look finished. Imperative-mood prose means the artifact does not exist. 4. Write to the reviewer who arrives, not whoever commissioned the run. 5. Delete findings whose entire content is test quality, unless critical. 6. Route privacy and security findings to the private tracker. Derived from eight postmortems in exogram-core; the reference keeps the full recipe.
A recipe returns as many answers as it has operators, and any output an operator
retypes carries the operator's provenance rather than the measurement's. This
ships the mechanism instead.
Runs arm A, mutates one line, runs arm B, restores the source, and writes
`falsify-<label>.{json,md}` plus both raw logs itself — nothing is transcribed.
The exit code is the verdict, so CI gates on it directly: 0 falsifying, 1 vacuous,
2 arm A already failing, 3 usage error.
Every artifact pins HEAD, node version, yarn.lock hash, and the tracked-change
count, so two operators either produce comparable results or visibly do not.
Verified against both outcomes on metamask-extension at 796685ce7b7: the perps
coalescing suite reports `falsifying` (10 passed, 2 failed under mutation), and
the token-search suite reports `vacuous` (3 passed both arms, so it does not
test the abort it appears to test).
…o retype them `retention-scan.py` and `policy-audit.py` already do the analysis well; both print to stdout, which makes the operator the capture device and returns provenance to whoever pasted the output. This wraps any command so the tool writes the artifact. Emits <label>.log verbatim, plus .json and an attachable .md that quotes the log rather than summarising it, with HEAD, tracked-change count, node, python, and yarn.lock hash pinned in each. The wrapped exit code passes through for CI. `--verdict` is stated by the caller, never inferred from the exit code. The first run of this script proved why: it labelled a policy audit "pass" while the output listed sixteen newly granted capabilities, because policy-audit.py exits 0 regardless. With no --verdict it now says "ran to completion, no verdict asserted". Verified on both scripts against real PRs, including a deliberately wrong invocation — which produces an artifact containing the traceback rather than a fabricated finding.
A memoization claim is a claim about a count, and reselect publishes the count. This generates a probe, runs it, deletes it, and writes the artifact, so the number never passes through an operator's hands. Three conditions; the middle one discriminates. A selector built on narrowed input selectors is unmoved by a write it does not read, while one taking `state.metamask` wholesale recomputes on every unrelated write in the app. Verified against both shapes on main, so the runner is shown to distinguish them rather than only to report success: getWalletsWithAccounts 1 / 1 / 6 narrowed selectRampsControllerState 1 / 6 / 11 recomputes on unrelated writes That completes runner coverage for the catalog lanes with engine skills: B3 and B7 via falsify-probe, C4 here, C9 and D3 via capture around their existing analysis scripts.
Whether a hand-written type agrees with the source it restates is a question only the compiler can settle. Arm A typechecks the baseline, arm B applies the substitution, and the finding is the error diff. Source is restored on exit, including on interrupt. Carries the warning the lane most needs: a silent arm B is not proof of agreement. Indexing and `.match()` compile against `string` and `string[]` alike, so without `--probe` injecting a deliberately-typed sink, the lane reports false clean on exactly the divergence it exists to find. When arm A already fails it stops and says nothing was established, alongside the module/export error count — but it does not classify from that ratio. The first real run had 124 of 280 errors as install artifacts while tripping no majority rule, because other codes are downstream of the same missing types. A threshold there would be a number I could not justify, so it reports the breakdown and leaves the judgement with the operator.
Gating on arm A being error-free was wrong: the finding was always the diff, so a single unrelated pre-existing error — a local work-in-progress file, in the run that exposed this — vetoed the whole lane. Baseline errors are now subtracted and only errors new under substitution count. Verified end to end on shared/lib/transactions-controller-utils.ts, where the local `LogWithTopicsArray` declares `topics?: string[]` against an upstream `topics?: string`: substitution alone 1 -> 1 errors, 0 new silent substitution + typed sink 1 -> 2 errors, 1 new TS2322 at the sink That is the lane's central caution demonstrated rather than asserted. Indexing and `.match()` compile against both shapes, so the obvious probe reports a false clean; only a deliberately-typed sink surfaces the divergence. A silent arm B means the probe was too weak, not that the types agree. Also fixes a `grep -c ... || echo 0` double-fire that produced "0\n0" and an integer-comparison error — the same shape already fixed once in this script's tracked-change count.
…lished Phase 0 of the /attest command. Everything greppable is checked before a model is asked for judgement, because a model asked "is this good evidence?" answers from inside the frame that produced the text. Marker pair, canonical header, verdict line, environment pin, a captured artifact, no "what would close it", no first-person process narration, and `proven` only where an execution artifact exists. Check 5 carries the weight: if every character of the output is one the operator typed, the run published an assertion. `--reference` compares capture density against a known-good artifact. Verified in both directions. A retracted run-1 comment is BLOCKED on three checks — no captured artifact, a "what would close it" section, and an unearned `proven`. A runner-produced artifact passes all eight. Building it reproduced two bugs it exists to catch: `hasre -i '<pat>'` passed `-i` as the pattern, so three checks silently grepped for the literal string and returned false passes; and an over-escaped backtick made the environment-pin check never match. Both found by running the gate against a file whose expected verdict was already known.
7 tasks
This was referenced Aug 2, 2026
Four runs measured `$SHA^..$SHA` on branches of five to twenty-nine commits and produced clean artifacts for a fraction of each change — same runner, same green run, nothing in the output distinguishing it from a finished measurement. So the range is now a non-negotiable, with the compare endpoint's `merge_base_commit.sha` named because `.base.sha` is the base branch tip and moves. Two rules alongside it, from the same batch: what a run could not see is a finding to state rather than a gap to omit, and the label on a number is caller-stated for the same reason the verdict is — a probe counting distinct context values published under a fixed "renders" heading passes every check while naming the wrong quantity.
…on it took A mutation runner echoed its `--replace` argument into the artifact, so the two could never disagree. They did: an `awk -v` assignment escape-processed the value and wrote a different line than the one requested, narrowing a regex meant to be widened. The suite ran the same test count in both arms, a different test failed than the one targeted, and the run reported power over a mechanism it never touched. Nothing in the artifact could have shown it.
MajorLift
marked this pull request as draft
August 3, 2026 09:42
MajorLift
added a commit
that referenced
this pull request
Aug 3, 2026
Each encodes a failure that actually shipped, rather than a principle that sounded right. `unmeasured-join` targets the costliest one — a conclusion assembled from true facts through a step nobody measured, which survives review because the facts check out and the join reads as prose. `instrument-check` requires a positive and a negative control before a measurement counts, after a mutation runner reported a falsification for a line it never wrote. `scope-of-search` makes a negative carry the pattern that produced it, after a grep keyed on a variable name declared absent what its own output printed two blocks later. `coverage-partition` replaces "the test has power" with which cases guard which mechanism. `selection-audit` separates a count from the rule that produced it, after two different selections landed on the same number and the match closed the question. `distinguishing-observation` and `observability-gap` point the same discipline at debugging: design the observation that separates candidates rather than confirms the favourite, and establish what signal exists on a path before reading more of it. The `evidence` links are forward references — that skill ships in #84 and is not on main yet.
4 tasks
Step 6 asked for both runs shown and left the operator to produce them, which is the shape that yields a retyped paste indistinguishable from output nobody ran. The run workflow already takes `ref` and `baseline`, executes the same command at both commits, and attaches the artifacts to a URL a reader can open — so the step names it. The larger risk was confusion with `falsify-probe.sh`, which shares the two-arm shape and answers a different question: base-against-branch asks whether a test is connected to the reported bug, one-commit-with-a-mutated-line asks whether it notices the mechanism going away. A test can pass either and fail the other. The skill now states the distinction in a table rather than leaving two similarly named things to be conflated, and notes that the runner's guards mechanise this skill's own falsifier — a red arm that ran fewer tests, or failed to load, is refused rather than counted.
Two things in this PR shared a root and a two-arm shape while answering different questions: the skill compares a base commit against a branch to show a test is connected to the reported bug, and `falsify-probe.sh` mutates one line at one commit to show a suite notices the mechanism going away. A test can satisfy either and fail the other, so the names had to stop rhyming. `red-on-base` names the skill's own discipline — the test must be red on the base commit — and shares no root with the runner. The evidence category keeps its name: a falsifying regression test is the artifact, and this is the procedure that produces one. The runner's stderr described its own result as "a falsifying test", which was the same collision inside the output of the thing causing it; it now says what it proves and points at the other experiment by name.
MajorLift
changed the title
/mms-evidence orchestrator and falsifying-test skill/mms-evidence orchestrator and red-on-base skill
Every other engine `evidence` names ships in its own pull request — `memory-leak`, `race-condition-repro`, `supply-chain-audit`, `react-render-delta`, `agent-run-cost`. This one was the sole exception, and the reason was authoring order rather than design: it was written alongside the orchestrator before the split between the instrument and the reasoning that points it existed as a concept. It lands in #106 with the other reasoning skills, whose substance is the same kind — what counts as proof, and how a proof can look right while testing the wrong thing. What stays here is the machinery: the runners, the run workflow, the gate, the hooks. The B3 engine cell now names a skill that arrives in #106, which is a dangling name in a table rather than a broken link, and resolves whichever order the two merge.
MajorLift
added a commit
that referenced
this pull request
Aug 3, 2026
Its substance is the same as everything else here: what counts as proof, and how a proof can look right while testing the wrong thing. Its falsifier — a base commit that fails for the wrong reason produces an identical exit code and proves nothing — is the same move `unmeasured-join` and `scope-of-search` make on different material. It sat in #84 because it was written alongside the orchestrator, before the instrument and the reasoning that points it were separate ideas. Every other engine already ships in its own pull request. Renamed from `falsifying-test` before the move, because it and `falsify-probe.sh` shared a root while answering different questions — base-against-branch asks whether a test is connected to the reported bug, one-commit-with-a-mutated-line asks whether a suite notices the mechanism going away.
MajorLift
changed the title
/mms-evidence orchestrator and red-on-base skill/mms-evidence orchestrator, runners, and publish gate
Eleven checks tested properties of the comment text. Text can be perfect and land somewhere nobody will read it, and that is what happened: across one register of published runs, 22 of 27 comments went onto pull requests that had already merged when they were posted — median 22 days after the merge, one 178 days after. The gate passed every one, because no property of a comment reveals the state of its destination. Check 12 takes `--target owner/repo#N` and blocks on anything that is not open. Omitting the target fails rather than passes: an unchecked destination is the condition that produced all 22. Without `gh` on PATH it reports UNVERIFIED and still refuses, since the point is that silence here is indistinguishable from success.
3 tasks
The CI workflow sources runners by `skills_ref`, and every run this week pointed at a fork branch. Four fixes were made there and never reached this one: the mutation now travels through `ENVIRON` instead of an escape-processed `awk -v` assignment, the artifact reports the line read back off disk beside the line requested, `--expect-fail` turns a red arm in the wrong place into its own outcome, and `--metric` lets the caller name what a probe counted. So the defect this PR's own description cites as the reason instruments must report their effect was, until now, still live in the instrument this PR ships. The probe comes across too, with the import fix that made it resolve at the destination the workflow copies it to. Two copies of the same scripts on two branches, edited in both directions — `attest-gate.sh` had a check the fork lacked, so it stays as it is here.
Check 9 is called "verdict matches artifact" and compares verdict words. Nothing compared the numbers, and prose drifting from the exhibit beside it is the most common way one of these goes wrong. Found by building a demonstration artifact to test this gate: the prose read "0 errors over 48 skills" directly above an exhibit reading "47 skill(s) checked", and named a warning class with zero instances in the output it was describing. Every other check passed. Two independent readers caught both, which is the argument for moving it into the layer that always runs rather than the one that costs money and sometimes never reports. Narrow on purpose, because a noisy check is an ignored one. Two-plus digits only, and only those absent from every fenced block; whole URLs, issue refs, versions, dates, SHAs, file:line citations, hyphenated identifiers and regex quantifiers are excluded as references rather than measurements. Each exclusion was added after a control run flagged something that was not a figure. Across eight real artifacts it flags one, correctly: a verdict line quoting an author's "730 tests" beside an exhibit measuring 731, where nothing distinguishes the cited figure from the measured one.
With `gh` absent the check printed UNVERIFIED and exited 0, so on a machine without it — running locally — the destination check announced that it had not run and the gate reported clean. A control that cannot run is indistinguishable from one that passed unless the exit code says otherwise.
Renamed on the security-domain branch; installs as `mms-lavamoat-policy`.
A diligence comment renders no verdict and deliberately does not use the Validation Run envelope. That exemption meant it was checked by nothing: this gate only knew the Validation Run shape, and `pr-evidence-gate.py` by design does not trip on a body claiming no verdict. So every rule the diligence skills state about their own output — including "runtime claims need a runtime artifact" — had no execution path. It showed. A lavamoat comment shipped with no marker pair, an `npm pack` specifier set no reader could fetch, and two bare integers traceable to nothing. `--diligence` swaps the four envelope checks for that contract's own — its marker pair, its header, permalinks pinned to a tag or SHA rather than a branch head, and a runtime claim check asking for the thing a `/blob/` link cannot witness. 3, 8 and 9 report SKIP with the reason rather than passing silently, since a check that cannot fail should not read as a check that passed. Everything downstream of the envelope is shared, because those defects are shared. Run against the comment that prompted this, it fails 1, 5 and 13 and passes the rest.
This repository is public. Naming a private repository here discloses its existence,
its owner and roughly its contents to every reader — and a prohibition naming it
("do not re-host to X, it is private") discloses exactly as much as a recommendation
would. Four such references were doing that, and the guidance survives without them:
the rule is audience-reachability, which is stated directly rather than by example.
Two memory-file citations offered as "source of truth" pointed into a private repo, so
a reader was told to follow a rule whose justification they cannot open. The reasoning
is inlined; the pointer is gone.
The publish-surface snippet hardcoded a GitHub username, which decided the destination
for whoever ran it. Now derived from `gh api user --jq .login`, and the surrounding
prose is second-person rather than first — a shared skill has no "my PRs".
`/attest` is no longer linked to a personal repository. That leaves it named but not
resolvable, which is honest and is the smaller problem; the workflow depending on a
command nobody else has is tracked separately.
The re-hosting section named a personal S3 bucket, its region, its prefix layout, and stated that anonymous `GetObject` is enabled under `public/*`. That is a live unauthenticated endpoint advertised, with its structure, to every reader of a public repository — a larger disclosure than the repository names removed alongside it, and one that reads as configuration rather than as a secret, which is why it survived two passes over this file. Now `EVIDENCE_BUCKET` and `EVIDENCE_REGION` from the environment. The requirements the bucket must satisfy — anonymous GetObject under `public/*`, listing disabled — are stated, because those are the load-bearing part; the name never was. The G5 lane likewise named a private test fork, which carried both the org and a personal handle. Now "your own test fork".
…setup Removing the hardcoded username left `gh pr view --jq --arg me "$ME"`, which is not a thing gh supports — its built-in filter takes no --arg and the command dies with "accepts at most 1 arg(s)". Piped to real jq instead, and checked against both branches of the logic: a PR authored by someone else resolves to "skip", one authored by the caller to "body". Replacing the named bucket with `EVIDENCE_BUCKET` removed a working default and put nothing in its place, so the section told you to configure a bucket without saying what "conforming" meant. The policy is now stated: anonymous `s3:GetObject` under `public/*`, public-access blocks off for that bucket, `s3:ListBucket` to nobody. With a note that an org-owned bucket beats a personal one, since artifact links outlive their publisher.
A results section reached a public PR whose entire content was hand-typed to look like
terminal output. Three independent things had to hold for that, and all three did.
The gate is model-invoked, so it can be skipped: the publish and the gate ran as two
statements rather than one chain, and the verdict was read after the write.
The hook that fires on the publish call carried a SECOND, narrower copy of the rules —
keyed on verdict tokens — so a comment rendering no verdict satisfied neither copy.
Two rule sets means the weaker one governs whatever falls between them. The hook now
delegates to `attest-gate.sh`: one rule set, invoked by construction rather than by
choice, in the mode the body's markers imply. It fails CLOSED once it has identified a
body it is about to publish — an enforcement point that waves things through when it
cannot find its rules is not one.
And check 5 in `--diligence` had been rewritten as a phrase denylist ("npm pack",
"complete specifier set"), which is precisely the regression its own comment records as
having shipped four times: every property of plaintext is forgeable by whatever emits
the plaintext. It is a medium test again — if the artifact shows a command or a run
result, it owes the reader something fetchable. `/blob/` links are excluded, because a
permalink to a `.json` file satisfied a naive extension test and was the specific reason
the hand-typed section passed.
Comment-update URLs carry the comment id, not the issue's, so check 12 was asking
whether pull #5177261620 was open. Resolved through the API instead.
Four-arm verified: blocks the exact command and body that shipped; ignores `ls`; ignores
a `gh` read with no body write; refuses when the gate is unreachable.
`mms-evidence` named `/attest` as the gate its output passes and shipped no such thing. Anyone installing the skill got phase 0 as a script and a reference to a command only its author had, which is half a publish path for everyone else. Phase 0 is `attest-gate.sh`, already here. Phase 1 is three briefs sent to fresh instances — frame, coverage, and how it reads to a stranger — written out in `references/dispatched-passes.md` so dispatch does not depend on commands that live elsewhere. The check table in `references/phase-0-checks.md` is generated from the gate rather than retyped, so it cannot drift from what runs. Two things the port makes explicit that the original left to discipline. The gate must be the same shell chain as the publish, because running both and reading the verdict after the write is how a blocked artifact reached a public PR. And softening a check to fit the case in hand is called out as an anti-pattern: if the new version could be satisfied by better prose alone, it is no longer the check.
Three copies of this hook were on one machine and the oldest was the one wired into settings — no `gh api` matcher, no delegation to `attest-gate.sh`. Every publish through that path went ungated, and nothing noticed, because a gate that blocks nothing looks exactly like a gate with nothing to block. `gate-controls.sh` is the thing that would have noticed. Six arms: three publish routes that must block, three inputs that must pass. It copies the hook somewhere with no sibling `scripts/` so `_find_gate()` resolves the way it does in production rather than the way it does in a checkout — the difference matters, and testing the checkout copy is how the deployed one stayed broken. Both halves earned their place immediately. The negative arm caught that delegating to attest-gate ran it over EVERY published body, so an ordinary reply was judged as a failed validation run; wiring that would have blocked every normal comment. The gate now applies only to bodies carrying an artifact marker or a verdict line. And the positive arm caught itself: the enrichment probe was a single sentence, which that rule correctly ignores, so the arm had been passing because attest-gate blocked the body for an unrelated reason. A probe that fires for the wrong reason reports a working rule. The `enrichment` class is ported forward from the older copy, with the constant it depends on — it existed in the deployed version and in neither newer one.
The six arms prove the script blocks what it should. They say nothing about whether anything calls it, and those are different questions — a hook that is unwired, or wired to a path that no longer exists, is indistinguishable from a hook with nothing to block. One session ran start to finish with every PreToolUse hook inert: 306 certification markers written, none enforcing anything, and the ritual read as compliance. The check enumerates config roots rather than trusting `$HOME`. Its first version did trust it, found one settings file, reported it as "the" wiring and never looked at the second — because `$HOME` here points at a per-account directory rather than the login home. That is the same defect one level up, caught only because two configs were known to exist and one was missing from the output. It stops short of claiming liveness, and says so: a settings file naming an existing file is not proof the running session loaded it. Only a command the gate must block, issued in a session and observed to be blocked, shows that.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
evidencetopr-workflow: an orchestrator, six runners, a CI workflow that executes them, a publish gate, and an optional emit-time hook.A reviewer takes the author's tested locally, works on trust. Capture skills replay a flow and photograph the screen — but a clean screenshot is also what a silently-swallowed failure produces, and
pr-readiness-checkproves tests are present, not that behaviour is correct.evidencenames the observation that would prove a claim false, goes looking for it, and publishes what it finds where review happens.The rule the skill enforces: cite what exists, capture what you ran
Terminal text pasted into a comment is indistinguishable from terminal text invented for it. Both are plaintext; both carry the author's assertion rather than a measurement. A runner emits clean stdout, clean stdout formats into a fenced block, and a fenced block looks like evidence — so a gate that checks the text can never separate them.
The reader has to be able to check a measurement without going through the author. That requires a different medium, not better wording. The one exception is line-level citation to something externally verifiable: a permalink resolves, so an audit built from permalinks is checkable even though it is prose.
Two consequences make up most of the diff.
The measurement runs in CI.
assets/evidence-run.ymlexecutes any runner against any target repository at a pinned SHA. A local run's only witness is its author, andactions/runs/<id>is itself the artifact. Every failure class this suite has shipped was local-environment — a helper script in/tmp, a probe deleted after the run, an absolute path, a contended host producing numbers that had to be retracted — and none is expressible in CI. The workflow lives in one repository and measures others, so nothing needs installing in the repo under review.Six runners write the artifact, so nothing is retyped.
falsify-probe.shselector-recompute.shrender-count.shtsc-substitution.shcapture.shattest-gate.shEach emits JSON, a log and a markdown block, and each states in its own footer whether a reader can verify it. Run locally, that footer reads "no reader-verifiable capture" — the artifact discloses its own weakness rather than leaving it for a gate to catch.
The publish gate
attest-gate.shruns eleven checks:marker pair,canonical header,verdict line,environment pinned,captured artifact,no prescriptions,no process narration,verdict is earned,verdict matches artifact,floats something for review,disclaimer present and early.Check 5 is the one that matters. It requires an image, a run URL, a gist or a hosted log — then rejects three forgeries that pass a naive version: a placeholder in a shown command, a local path in a shown command, and an A/B pair rendered as one command line printed twice with different output.
Nine non-negotiables, in
skill.mdrather than a referenceA requirement you have to fetch is advisory. Four of these were added by runs that failed while producing evidence with the earlier version:
$SHA^..$SHAon a twenty-six-commit branch is a twenty-sixth of the change and looks exactly like a finished measurement — same runner, same green run, same artifactawk -vassignment escape-processed a replacement, wrote a different line than requested, failed a different test than targeted, ran the full suite, and reported a falsification for a mechanism it never touchedThe other five cover the artifact requirement,
provenrequiring execution, the absence of a "what would close it" section, writing to the reviewer rather than the commissioner, and routing privacy and security findings to the private tracker instead of publishing them.The evidence catalog — 41 lanes across 7 families
The taxonomy lives in
references/evidence-catalog.md(commit-pinned, 34 KB). Each family answers a different kind of question, so the family follows from what the claim asserts.A1 visual_validation— before/after of a reachable surfaceB3 falsifying test— fails on base, passes on branchC4 render & selector proof— WDYR counts,.recomputations()D3 LavaMoat policy diff— newly granted capabilitiesE1 Sentry links— before/after, in situF1 state migration—changedKeys, old→new stateG5 CI-workflow change— run on a test forkBolded lanes carry a claim on their own; the rest corroborate. Five lanes have their own engine skills, each in its own pull request:
B3→red-on-base(#106),B7→race-condition-repro(#97),C4→react-render-delta(#43),C9→memory-leak(#81),D3→supply-chain-auditandlavamoat-policy(#83).A known gap: every family covers product behaviour. None covers agent behaviour — whether a skill improves what an agent produces. Picking a lane cannot answer "does this guidance work", which is why reviewing skills PRs is a different problem from reviewing code.
hooks/pr-evidence-gate.pyA
PreToolUse:Bashhook enforcing the trustworthiness gate at emit time, blocking an outward-facing write whose body carries an unbacked verdict, an untracked deferral, a CI restatement, a bare or truncated identifier, a mutable/blob/<branch>/ref, a raw dump standing in as the resolver, a link-only or data-only exhibit, or a step waiver.gh apibody writes, not just the porcelain (-f body=,-F body=@file,--input). A PATCH to a comment is the same publish with a different spelling; a porcelain-only matcher is a hole, not a gate.references/evidence-gate-setup.md.evidence-trustworthiness.mdstates which of its nineteen items the hook enforces mechanically and which stay reader-applied: it sees vocabulary, not semantics, and cannot tell whether a screenshot shows the resolving UI.Showcase — the falsifying-test discipline applied in this repo
Two PRs here were validated with it, so the output is checkable without leaving
MetaMask/skills.#94 — deliver knowledge a skill cites from another domain. Claim: four skills cite
testing-layers.md, which lives in another domain, so the reference cannot resolve for any consumer. Tests written against that claim, run at both refs:#87 — guard that installed knowledge references resolve. A test failing on base for the wrong reason proves nothing — an import error and a real assertion exit identically — so the base failure is read, not counted:
The message names the defect rather than a missing fixture, and the run reports 0 import or fixture errors.
What it caught that a green suite did not. A description-budget check added during this work passed while asserting nothing: the regex anchored with
^and no multiline flag, so it matched zero files. Addingmthen made$match at every line end, catching the plain frontmatter form and silently missing the folded>-form — most of the corpus. Both passes looked identical in the runner. Only running the check against a known violation exposed it.Validation runs
Trial runs against merged
metamask-extensionPRs nobody flagged. Every claim was re-verified against the real diff before posting. Clean results are included on purpose — a skill that only ever reports problems cannot be calibrated.Each comment carries a trial-run disclaimer and links back here for feedback.
Reviewing this
hooks/besideskill.mdis rejected by the structural linter until #99 adds it toBUNDLE_DIRS, so this fails CI onmaintoday through no fault of its contentsdomains/pr-workflow/skills/evidence/—skill.md, ninereferences/, sixscripts/,assets/evidence-run.yml,hooks/pr-evidence-gate.py. Experimental, GPG-signedpr-workflowvstesting), and AEP scope — the AEP repo is internal, so the skill points at its README for stack bring-up rather than reproducing setup hereevidenceis the heaviest skill in the catalogue at ~6,800 tokens once selected. The AEP run procedure sits behindreferences/aep-local-run.mdrather than inline, since most validations use a lighter laneThis branch alone does not work
The installer resolves one source directory. It does not follow
## Relatedas a dependency edge, does not fetch other refs, and reports a dangling reference as a warning rather than an error — because forward references across open pull requests are expected. So a checkout of this branch installs only the skills on it, with every cross-branch reference dead.To get a tree that actually runs:
Merges clean onto
main. Yields all elevenpr-workflowskills plusevidence, and lints at 0 errors. Three## Relatednames still dangle —react-render-delta,race-condition-repro,lavamoat-policy— because those engines ship in #43, #97 and #83.Worth knowing that the combination is also where defects surface: an error caught only with #99 and this set together — a description advertising a command the installer does not emit — passes on every branch individually.
Draft: these files are pending removal
The scripts, dispatcher and probe on this branch have moved to a standalone runners repository (
v0.1.0). That repository is the source; the copies here are temporary. They remain only so this branch can be run and tested on its own while the extracted repository is reviewed.Once that review lands, they come out and the skill references the runners by pinned tag.
Why the interim needs watching. Two copies of these files is the arrangement that already failed: CI executed the runners from a branch by ref, four fixes landed on the copy it was reading, and none reached the copy under review — including a mutation runner that rewrote the line it was asked to apply, reported the full suite as having run, and published a falsification for a mechanism it never touched. That fix is cited in this description and was, until it was reconciled, still live in the code this branch ships.
So while both copies exist:
Compares the eight files that must agree and names the one that must not —
evidence-run.ymltargets a flat layout there and a nested one here. ReportsDRIFTEDwith a diff, or8 file(s) agree. Currently agreeing.What is still open, and it is layout rather than content.
hooks/pr-evidence-gate.pyis a Claude Code hook rather than a runner, so it fits neither repository cleanly and may end up in a third place. The skill text, the catalog, the non-negotiables and the showcase do not depend on any of that — they are reviewable now, and comments on them are wanted.