From c635aa8823a5260ac082b3a1a734a9241e4a44b3 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Thu, 30 Jul 2026 08:04:06 -0400 Subject: [PATCH 01/63] feat(pr-workflow): add pr-validate and falsifying-test skills MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces #56, whose skill.md shipped frontmatter with no body at all — the installer's bodyAfterFrontmatter() returned empty, so an agent loading it got a description and six references it had no instruction to read. pr-validate: for a PR's specific falsifiable claim, name the observation that would prove the claim false, gather it, and publish it into the PR body. Drives the AEP harness (visual_validation, perf_validation) as the primary engine, backed by a catalog of complementary lanes, a trustworthiness gate that rejects vacuous passes, and a publishing flow with an audience-reachability rule for re-hosted artifacts. falsifying-test: the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. Its own falsifier is a base-commit failure for the wrong reason (import error, missing fixture, unrelated red), which looks identical in an exit code and proves nothing. pr-validate calls it as the engine behind lane B3. hooks/pr-evidence-gate.py enforces the trustworthiness gate at emit time, blocking an outward-facing write whose body carries an unbacked verdict, an untracked deferral, a CI restatement, a bare or truncated identifier, a mutable ref, a dump-as-resolver, a link-only or data-only exhibit, or a step waiver. It polices `gh api` body writes as well as the porcelain, since a PATCH to a comment is the same publish with a different spelling. --- .../skills/falsifying-test/skill.md | 86 ++++ .../pr-validate/hooks/pr-evidence-gate.py | 390 ++++++++++++++++++ .../references/claim-extraction.md | 63 +++ .../references/evidence-catalog.md | 254 ++++++++++++ .../references/evidence-gate-setup.md | 58 +++ .../references/evidence-publishing.md | 291 +++++++++++++ .../references/evidence-trustworthiness.md | 43 ++ .../pr-validate/references/lane-assertions.md | 26 ++ .../pr-validate/references/worked-examples.md | 30 ++ .../pr-workflow/skills/pr-validate/skill.md | 305 ++++++++++++++ 10 files changed, 1546 insertions(+) create mode 100644 domains/pr-workflow/skills/falsifying-test/skill.md create mode 100755 domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py create mode 100644 domains/pr-workflow/skills/pr-validate/references/claim-extraction.md create mode 100644 domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md create mode 100644 domains/pr-workflow/skills/pr-validate/references/evidence-gate-setup.md create mode 100644 domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md create mode 100644 domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md create mode 100644 domains/pr-workflow/skills/pr-validate/references/lane-assertions.md create mode 100644 domains/pr-workflow/skills/pr-validate/references/worked-examples.md create mode 100644 domains/pr-workflow/skills/pr-validate/skill.md diff --git a/domains/pr-workflow/skills/falsifying-test/skill.md b/domains/pr-workflow/skills/falsifying-test/skill.md new file mode 100644 index 00000000..922991f5 --- /dev/null +++ b/domains/pr-workflow/skills/falsifying-test/skill.md @@ -0,0 +1,86 @@ +--- +name: falsifying-test +description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /falsifying-test, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by pr-validate as the engine behind its falsifying regression test evidence category. +maturity: experimental +--- + +# /falsifying-test + +Reach for this on **every bug-fix PR**. A test that passes on the branch proves the branch is +green. A test that **fails on base and passes on the branch** proves the change is causally +connected to the reported bug. Only the second is evidence, and the gap between them is where +this skill lives. + +> **Falsifier.** A test that fails on base for a reason unrelated to the bug. A missing import, +> a fixture the base commit doesn't have, a helper introduced by the branch, an unrelated +> pre-existing failure — every one produces a red run and a non-zero exit code that looks +> exactly like a correct falsification. **The exit code is not the evidence; the assertion +> message is.** + +## Method + +1. **Write the test against the reported behaviour, not the diff.** Start from the issue's + reproduction. A test derived from reading the fix tends to assert the fix's mechanism and + will pass on base the moment the mechanism is reachable by other means — or fail on base + for structural reasons rather than behavioural ones. + +2. **Run it on base FIRST, and read the failure output.** Not the exit code — the message. It + must fail on the **assertion that encodes the bug**: an expected value that differs, a state + that wasn't reached, an event that didn't fire. If base fails with a + `ModuleNotFoundError`, a syntax error, or a helper that doesn't exist yet, you have not + falsified anything; you have discovered that the test can't run there. + +3. **Pin the base explicitly.** Use the PR's actual merge-base, not whatever `main` points at + today. `main` moves; a re-run weeks later against a drifted `main` is a different + experiment and may fail for reasons that have nothing to do with the fix. + +4. **Make the test runnable on base.** When the test needs a helper or fixture the branch + introduces, split it: land the scaffolding in a form that exists on both sides, or inline + the setup so the test file is self-contained. If that's impossible, say so and downgrade the + claim — a test that *cannot* run on base gives a branch-only pass, which is a weaker piece + of evidence and should not be presented as a falsifying one. + +5. **Confirm it fails for one reason, not several.** If base has unrelated failures in the same + file or suite, scope the run to the new test (by name/path) so the red is attributable. A + suite that was already red proves nothing about your assertion. + +6. **Show both runs.** Base: the assertion failure, verbatim. Branch: the pass. Same command, + same filter, both commits identified. Captured terminal output beats a transcription — + retyped output is a self-report, and a real capture has caught errors that careful prose + missed. + +7. **Pair it with the issue.** The PR's `Fixes #N` plus a test named for the behaviour makes + the causal chain checkable by a reader who runs nothing. + +## When you can't write one + +This is a finding, not a gap to paper over. If no test fails on base, one of these is true: + +- **The bug isn't where the fix is.** The most common case, and the reason to run this check + before review rather than after. +- **The reported behaviour isn't reproducible in the harness** — timing, environment, or a + real-device dependency. Say which, and reach for a different evidence category (a + deterministic interleaving test for ordering bugs, an e2e trace for environment-dependent + ones). +- **The fix is a refactor or hardening change, not a bug fix.** Fine — then the PR's claim + should say that, and this category doesn't apply. + +State which one. "No test added" with no explanation reads as an omission; the diagnosis is +useful information about the change. + +## Output + +``` +Falsifying test — (Fixes #N) + base FAIL + branch PASS + command: + scoped: +``` + +## Related + +- `pr-validate` — packages this skill's output as its B3 evidence category; B7 (deterministic + interleaving) is the sibling for concurrency and temporal-ordering bugs. +- `react-render-proof` — the same before/after discipline applied to a measured quantity + rather than a boolean. diff --git a/domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py b/domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py new file mode 100755 index 00000000..2bcd4651 --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py @@ -0,0 +1,390 @@ +#!/usr/bin/env python3 +""" +Emit-time evidence gate (PreToolUse:Bash). + +Blocks outward-facing `gh pr|issue edit|create|comment` — and `gh api` body +writes, which bypass the porcelain — whose body contains, in a validation-scoped +paragraph, a claim that the trustworthiness gate would reject. Rationale: an +unbacked "confirmed / verified / proven / observed / ingested / ✅" launders an +unverified assertion as fact under the author's name, and an untracked "remains +pending" decays to never. + +The trustworthiness gate is the checklist; THIS is the trigger that runs it. +Each class below implements a numbered item of `references/evidence-trustworthiness.md`. + +Contract: reads PreToolUse JSON on stdin. Exit 0 = allow. Exit 2 = block +(stderr shown to the model). Fails OPEN on anything it cannot parse, so it +never bricks unrelated Bash commands. +""" +import json +import os +import re +import sys + + +def _out_allow(): + sys.exit(0) + + +def _block(msg): + sys.stderr.write(msg) + sys.exit(2) + + +# Outward-facing gh write surfaces. The porcelain set is wider than +# `gh pr edit|create` because the same unbacked verdict launders identically +# through a PR comment or an issue body. `gh api` is included because a PATCH +# to .../comments/ is the same publish with a different spelling — a gate +# that cannot see the write it is meant to police is not a gate. +GH_PORCELAIN = re.compile(r"\bgh\s+(?:pr|issue)\s+(?:edit|create|comment)\b") +GH_API = re.compile(r"\bgh\s+api\b") + + +def main(): + try: + payload = json.load(sys.stdin) + except Exception: + _out_allow() + + if payload.get("tool_name") != "Bash": + _out_allow() + + cmd = (payload.get("tool_input") or {}).get("command", "") + + is_porcelain = bool(GH_PORCELAIN.search(cmd)) + is_api = bool(GH_API.search(cmd)) and re.search(r"(?:-F|-f|--field|--raw-field)\s+body=|--input\b", cmd) + if not (is_porcelain or is_api): + _out_allow() + if is_porcelain and "--body" not in cmd: # covers --body and --body-file + _out_allow() + + body = _extract_body(cmd) + if not body: + _out_allow() # can't read it -> don't block; nothing to scan + + violations = _scan(body) + if not violations: + _out_allow() + + lines = [ + "EVIDENCE GATE (PreToolUse) — blocked outward-facing GitHub write.", + "", + "Each finding names the trustworthiness-gate item it violates. Fix by", + "attaching the missing artifact in the SAME block, or by downgrading the", + "claim (⚠️ inconclusive / remove it). Do not rephrase around the check.", + "", + ] + for v in violations[:12]: + need = NEEDS.get(v.get("kind", "verdict"), "ARTIFACT") + lines.append(f' • [{v["kind"]}] "{v["token"]}"') + lines.append(f' needs: {need}') + lines.append(f' in: {v["snippet"]}') + if len(violations) > 12: + lines.append(f" … and {len(violations) - 12} more.") + lines += [ + "", + "If the evidence exists on disk, BIND it: every collected artifact the", + "claim rests on gets referenced or re-hosted before the write.", + ] + _block("\n".join(lines) + "\n") + + +NEEDS = { + "verdict": "an inspectable ARTIFACT (https:// permalink, /blob//, or a *.test.ts ref)", + "observation": "an OBSERVATION artifact (screenshot/recording/log/JSON/permalink) — " + "a /blob/ code link witnesses code, not runtime behavior", + "deferral": "a co-located TRACKER (#issue, issues/pull URL, 'triage', 'tracked in')", + "ci-restatement": "removal — a validation surface carries zero CI references. " + "The Checks tab already shows them; cite CI only as the revert " + "lane's outcome, never as 'green at head'", + "inflated-verdict": "a downgraded verdict — 'live-proven' co-located with " + "'not exercised' is inflated; borrowed evidence never " + "upgrades an uncaptured lane", + "bare-identifier": "a resolving link for the id (permalink or absolute-windowed " + "query) OR the re-hosted capture showing it", + "truncated-identifier": "the FULL identifier, quoted verbatim — an ellipsized id " + "cannot be grepped against any artifact, and a co-located " + "resolver does not excuse it", + "mutable-ref": "a commit-pinned permalink (/blob//…#Lx-Ly) — a branch ref " + "can be rewritten after review", + "dump-resolver": "a reader-native exhibit — a live link or a visual. A raw " + "log/JSON/HAR dump is appendix-only, never the exhibit a claim rests on", + "link-only-exhibit": "an embedded visual of the linked view ALONGSIDE the permalink — " + "link-only defers validation behind click + auth + query rendering", + "data-only-exhibit": "an in-environment capture (the resolving UI with its query, " + "project/environment selectors and time window in-frame) — " + "quoted data alone carries no liveness provenance", + "step-waiver": "a per-step ⏳ + tracker whose blocker is that step's OWN unmet " + "precondition — an impossibility argument is not a discharge", +} + + +def _extract_body(cmd): + # 1) --body-file / --input + m = re.search(r"--(?:body-file|input)[=\s]+(?:'([^']+)'|\"([^\"]+)\"|(\S+))", cmd) + if m: + path = m.group(1) or m.group(2) or m.group(3) + try: + with open(os.path.expanduser(path), "r", encoding="utf-8") as fh: + raw = fh.read() + except Exception: + return "" + # `gh api --input` takes a JSON file; pull .body out of it. + try: + obj = json.loads(raw) + if isinstance(obj, dict) and isinstance(obj.get("body"), str): + return obj["body"] + except Exception: + pass + return raw + # 2) gh api -F body=@ / --field body=@ + m = re.search(r"(?:-F|--field|--raw-field)\s+body=@(?:'([^']+)'|\"([^\"]+)\"|(\S+))", cmd) + if m: + path = m.group(1) or m.group(2) or m.group(3) + try: + with open(os.path.expanduser(path), "r", encoding="utf-8") as fh: + return fh.read() + except Exception: + return "" + # 3) --body "$(cat <<'EOF' ... EOF)" heredoc + m = re.search(r"<<-?'?EOF'?\s*\n(.*?)\n\s*EOF", cmd, re.DOTALL) + if m: + return m.group(1) + # 4) --body '...' / --body "..." / gh api -f body='...' + m = re.search(r"(?:--body|(?:-f|--field|--raw-field)\s+body=)[=\s]*'((?:[^']|'\\'')*)'", cmd, re.DOTALL) + if m: + return m.group(1) + m = re.search(r'(?:--body|(?:-f|--field|--raw-field)\s+body=)[=\s]*"(.*?)"', cmd, re.DOTALL) + if m: + return m.group(1) + return "" + + +# ── item 1/5: verdict claims ──────────────────────────────────────────────── +VERDICT = re.compile( + r"(?i)(?:\bcapture[ds]?\s+confirm\w*|\bconfirm(?:s|ed)\b|\bverif(?:y|ies|ied)\b" + r"|\bproven\b|\bobserved\b|\bingested\b|\bdemonstrat(?:e|es|ed)\b" + r"|\blive-proven\b|\bsuccessful\b|\bvalidated\b" + r"|does not drop\b|✅)" +) +ARTIFACT = re.compile( + r"(?i)(?:https?://\S+|actions/runs/\d+|/blob/|\bjob/\d+" + r"|`?[\w./-]*\.(?:test|spec)\.[tj]sx?(?::\d+)?`?)" +) +# ── item 2: runtime observation claims ───────────────────────────────────── +OBSERVATION = re.compile( + r"(?i)(?:\brendered\b|byte-identical(?:ly)?|\bsnapshot\s+shows?\b" + r"|\bscreenshots?\s+show\w*|\breproduc(?:ed|es)\b" + r"|\bstill\s+(?:shown|shows|fails|failing|raises)\b" + r"|\bin\s+a\s+(?:real|live)\s+browser\b|\blive\s+test\s+build\b" + r"|\bin\s+two\s+independent\s+runs\b|\bworks\s+as\s+described\b)" +) +OBS_ARTIFACT = re.compile( + r"(?i)(?:!\[|/ instead of /blob// ───────── +MUTABLE_REF = re.compile( + r"(?i)https?://github\.com/[\w.-]+/[\w.-]+/blob/(?![0-9a-f]{7,40}[/#])[\w.-]+/" +) +# ── item 13: dump-as-resolver ────────────────────────────────────────────── +DUMP_LINK = re.compile(r"(?i)https?://\S+\.(?:log|json|har|txt)\b") +IMAGE_EMBED = re.compile( + r"(?i)(?:!\[|.*?", + "", body, flags=re.DOTALL) + violations = [] + section = "" + for block in re.split(r"(?m)^(?=\s*#{1,6}\s)", body): + hm = re.match(r"\s*#{1,6}\s*(.+)", block) + if hm: + section = hm.group(1) + section_in_scope = bool(SCOPE_HEADING.search(section)) + for para in re.split(r"\n\s*\n", block): + scan_lines = [] + for ln in para.splitlines(): + s = ln.strip() + if re.match(r"-\s*\[[ xX]\]", s): # checklist item + continue + if s.startswith(">"): # blockquote (bot NOTE) + continue + if s.startswith("_Status key"): # legend + continue + if s.startswith("#"): # heading line + continue + scan_lines.append(ln) + chunk = "\n".join(scan_lines) + if not chunk.strip(): + continue + if not (section_in_scope or SCOPE_PARA.search(chunk)): + continue + # A markdown table row is its own claim unit — scan each row so an + # artifact two rows down cannot excuse a bare row. + units = chunk.splitlines() if chunk.lstrip().startswith("|") else [chunk] + for unit in units: + _scan_unit(unit, violations) + return violations + + +def _add(violations, kind, token, unit): + violations.append({ + "kind": kind, + "token": token, + "snippet": re.sub(r"\s+", " ", unit.strip())[:120], + }) + + +def _positive_verdict(unit): + """A non-negated verdict token in this unit, or None.""" + for m in VERDICT.finditer(unit): + if not _negated(unit, m.start()): + return m.group(0) + return None + + +def _scan_unit(unit, violations): + # ── VERDICT: excused by a co-located inspectable artifact. + if not ARTIFACT.search(unit): + tok = _positive_verdict(unit) + if tok: + _add(violations, "verdict", tok, unit) + + # ── OBSERVATION: needs an observation-class artifact. A /blob/ code + # permalink does NOT excuse it. + if not OBS_ARTIFACT.search(unit): + for m in OBSERVATION.finditer(unit): + if _negated(unit, m.start()): + continue + _add(violations, "observation", m.group(0), unit) + break + + # ── DEFERRAL: excused by a co-located tracker, NOT by an artifact. + if not TRACKER.search(unit): + dm = DEFERRAL.search(unit) + if dm: + _add(violations, "deferral", dm.group(0), unit) + + # ── CI RESTATEMENT (item 11): unconditional in validation scope. No + # verdict co-location required, no "beyond-CI"/"as context" excuse — + # a carve-out here is an instruction to phrase every violation as the + # exception. + cm = CI_RESTATEMENT.search(unit) + if cm: + _add(violations, "ci-restatement", cm.group(0), unit) + + # ── INFLATED VERDICT (item 11): proof language co-located with an + # admission the surface was not exercised. + nm = NOT_EXERCISED.search(unit) + if nm and _positive_verdict(unit): + _add(violations, "inflated-verdict", nm.group(0), unit) + + # ── STEP WAIVER (item 14): an impossibility argument never discharges a + # lane derived from an executable Manual testing step. + sm = STEP_WAIVER.search(unit) + if sm: + _add(violations, "step-waiver", sm.group(0), unit) + + # ── TRUNCATED IDENTIFIER (item 16): a co-located resolver does NOT + # excuse — the resolver resolves the full id, not the fragment the + # reader holds. Hash-equality prose is exempt. + if not HASH_EQUALITY.search(unit): + tm = TRUNCATED_ID.search(unit) + if tm: + _add(violations, "truncated-identifier", tm.group(0), unit) + + # ── BARE IDENTIFIER (item 12): an id with no resolving link and no + # re-hosted capture is a digging assignment. + if not RESOLVER.search(unit) and not OBS_ARTIFACT.search(unit): + bm = BARE_ID.search(unit) + if bm: + _add(violations, "bare-identifier", bm.group(0), unit) + + # ── MUTABLE REF (item 16): pin evidence links to a SHA. + mm = MUTABLE_REF.search(unit) + if mm: + _add(violations, "mutable-ref", mm.group(0)[:60], unit) + + # ── DUMP RESOLVER (item 13): a positive verdict whose only resolver is a + # raw dump behind a link. The digging moved a hop away, it did not + # disappear. + if _positive_verdict(unit) and DUMP_LINK.search(unit) and not IMAGE_EMBED.search(unit) \ + and not LIVE_LINK.search(unit): + _add(violations, "dump-resolver", DUMP_LINK.search(unit).group(0)[:60], unit) + + # ── LINK-ONLY EXHIBIT (item 15): a live permalink defers validation + # behind click + auth + query rendering. Needs the visual too. + if _positive_verdict(unit) and LIVE_LINK.search(unit) and not IMAGE_EMBED.search(unit): + _add(violations, "link-only-exhibit", LIVE_LINK.search(unit).group(0)[:60], unit) + + # ── DATA-ONLY EXHIBIT (item 17): telemetry claim with neither a visual + # nor a live link carries no liveness provenance — extracted data is + # indistinguishable from data typed by hand. + if _positive_verdict(unit) and TELEMETRY_VOCAB.search(unit) \ + and not IMAGE_EMBED.search(unit) and not LIVE_LINK.search(unit): + _add(violations, "data-only-exhibit", TELEMETRY_VOCAB.search(unit).group(0), unit) + + +def _negated(text, pos): + """A verdict token preceded by a negator is a hedge, not a claim.""" + pre = text[max(0, pos - 16):pos].lower() + if re.search(r"\b(not|never|no|isn't|aren't|cannot|can't|without|un|yet)\s*$", pre): + return True + # 'unverified' / 'unproven' — negator fused onto the token + if pre.endswith("un"): + return True + return False + + +if __name__ == "__main__": + main() diff --git a/domains/pr-workflow/skills/pr-validate/references/claim-extraction.md b/domains/pr-workflow/skills/pr-validate/references/claim-extraction.md new file mode 100644 index 00000000..fdd90417 --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/claim-extraction.md @@ -0,0 +1,63 @@ +# Claim extraction + +The linchpin of pr-validate: before choosing any lane, turn the PR into a **falsifiable, surface-specific claim**. Every lane is only as good as the claim it tests. A vague claim ("improves perf", "fixes the bug") can't be proven or refuted; a sharp claim names the precondition, action, observable outcome, and what would disprove it. + +## Read these, in order + +1. **PR body** — Description (what/why), `Fixes #N`, Manual testing steps, the Before/After intent. +2. **Linked issue(s)** — the bug report / acceptance criteria; "Steps to reproduce" and "Expected vs actual" are the claim in the reporter's words. +3. **The diff** (`gh pr diff`) — what actually changed: which surfaces, controllers, modules. The claim must be anchored to what the code can do, not only what the body promises. +4. **Labels / type** — bug vs feat vs perf vs refactor changes the claim shape (see [special cases](#special-cases)). + +## Extraction steps + +1. **Asserted change** — what does the PR say it does? (body + issue) +2. **Anchor to the diff** — which surface/module changed? Reconcile intent with the diff. If the body promises X but the diff can't deliver X, **flag the drift** — that's a finding, not a claim. +3. **Phrase as falsifiable** — `Given , when , then .` The outcome must be observable and checkable. Replace vague verbs (improve / fix / handle / support) with the concrete observable. +4. **Pin the surface + reachability** — exact screen / API / metric. Reachable in the default fixture, or does it need state seeding, a feature flag, or a fallback surface? + - **A surface need not be a screen.** A pipeline's job graph, a build artifact, a policy file, a telemetry shape, or a harness's determinism are all legitimate surfaces with their own falsifiers. Do not force a user-visible observable onto a claim that does not have one — routing a CI or build claim through a product effect is the *wrong* bar, not a stricter one. + - **When the changed code is the automation, the PR's own run may not exercise it.** A CI-config diff commonly skips the very path it edits (build reuse, `needs-*` resolution, event-type conditions). Execute the changed workflow where its trigger conditions hold — a test fork, a branch whose name satisfies the condition — with the failure state forced. Name that substitution explicitly; a claim about *this* repo's pipeline is not proven by a run on another. +5. **Classify the type** → routes to lanes via the matching guide: visible UI · non-visible perf · telemetry · persisted-state · build-output · behavior-no-UI. +6. **Decompose mixed claims** — a PR that changes UI *and* shifts a metric is two claims; validate each. + +## Claim Card (output) + +``` +Claim: Given , when , then . +Surface: (reachable? seed / flag / fallback: …) +Type: → lanes +Falsifier: +Baseline: +``` + +One card per claim. For a refactor, the claim is a **negation** (see below). + +## Claim quality bar + +A good claim is **falsifiable** (observable outcome + clear falsifier), **surface-specific** (names the exact screen/API/metric, not "the app"), **diff-anchored** (the changed code can plausibly produce it), **bounded** (one behavior, one precondition), and **measurable** where quantitative (a number + threshold, not "faster"). + +## Anti-patterns → refinements + +| Vague claim | Refined | +|---|---| +| "Improves performance" | "Opening the Activity tab: TBT drops below 200ms (was >600ms)" — name the interaction, metric, threshold | +| "Fixes the bug" | "With privacy mode on, the Perps tab balance is masked" — observable behavior + precondition + surface | +| "Refactor, no behavior change" | Negation claim: "behavior of `` is unchanged" → prove via falsifying-test-stays-green / snapshot / identical output, **not** a screenshot | +| "Adds a null check" (restates the diff) | "No crash when `` is null on ``" — the behavior, not the code | +| Body promises X, diff does Y | Not a claim — **flag the drift** to the author | + +## Special cases + +- **Refactor / no-op:** the claim is "nothing observable changed." Falsifier = any behavior/output diff. Lanes: regression test stays green, snapshot diff empty, bundle/output identical (D1/D2), benchmark within noise. A passing screenshot proves nothing here. +- **Bug fix:** the strongest claim form ships its own falsifier — a test that fails on `main` and passes on the branch (catalog **B3**). Extract the claim straight from the issue's "Expected vs actual." +- **Perf:** always quantify — metric + interaction + threshold + baseline. Without a number it isn't falsifiable. +- **Persisted-state / migration:** claim = "upgrading from `` preserves `` and applies ``." Falsifier = corrupted/lost state. Baseline = a profile from the prior version (catalog **F1**). +- **Flag-gated:** two claims, one per flag state (catalog **F5**). + +## Worked examples + +- **Visible (#42683):** body "privacy mode doesn't hide the Perps balance"; issue: expected masked, actual visible; diff touches the Perps balance component. → **Claim:** *Given privacy mode on, when I open the Perps tab, the balance is masked.* **Surface:** Perps tab (gated → fallback: Shield entry modal). **Type:** visible → A1/B1. **Falsifier:** balance digits visible under privacy mode. **Baseline:** same flow on base reproduces the bug. +- **Perf:** body "defer Rive wasm at startup"; diff: dynamic `import()` of the Rive runtime. → **Claim:** *On cold start of the home view, the Rive wasm chunk is not requested until the animation surface mounts.* **Surface:** startup network + chunk graph. **Type:** perf → A2/C6/D2. **Falsifier:** the chunk appears in the cold-start waterfall. **Baseline:** base requests it at startup. +- **Migration:** diff adds migration NNN. → **Claim:** *Loading a profile from `` applies migration NNN; `changedKeys = {}`; all other state intact.* **Type:** state → F1. **Falsifier:** an untouched controller mutated, or migrated state malformed. **Baseline:** a prior-version profile. + +A sharp claim is also a good recipe **proof target** (ADR-0058): precondition → action → observable maps to pre-conditions → assertions → screenshot points. Extraction pays off in both lanes. diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md b/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md new file mode 100644 index 00000000..b34a5e20 --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md @@ -0,0 +1,254 @@ +# Evidence catalog + +The menu of evidence kinds for validating a MetaMask **extension** PR, with **what each proves**, **how to capture it (verified against the live repo)**, and **when to reach for it**. AEP is the primary autonomous engine; the rest are complementary. The skill's job is to **match evidence to the claim** and to **proactively suggest kinds the author didn't think of**. + +Pick the evidence that would **falsify the claim if it were false**. Prefer a lane that yields an artifact a reviewer can independently re-check (a link, an image, a number, a replayable trace) over prose. Don't run the whole menu — match, then capture. Capture commands are written against the `metamask-extension` checkout; verify script names against its `package.json` (they drift). + +Legend: **first-class lanes** are `##`-headed; closely-related variants are sub-bullets. Capture marked *(manual)* has no repo helper — it's a DevTools/CDP action. + +--- + +# A. AEP harness (primary, autonomous) + +## A1. visual_validation — before/after screenshots +- **Proves:** a visible UI change on the real surface. Deterministic state seed + agent navigation; PNG artifacts in `evidenceBundle.artifactRefs`. +- **Capture:** `taskClass: visual_validation`, `payload.prUrl` + `description` hint. See the *Preflight* and *Run mechanics* sections of [skill.md](../skill.md). +- **Reach for it:** anything a human would screenshot for the PR's `### After`. + +## A2. perf_validation — falsifiable network/static/smoke assertions +- **Proves:** non-visible behavior (hover-preload, no double-fetch, chunk membership, smoke boot). CDP netlog / phase segmentation / source-map membership. +- **Capture:** `taskClass: perf_validation` (needs a `yarn webpack --test` build). Confirm the perf-validation graph is registered in your AEP checkout; falls back to C6/D2 manually if it isn't present. + +## A3. AEP bundle byproducts (free with any run) +- Test results (`executionResult`/`checkResults`), diff stats, automated `reviewResult` findings, and the **LangSmith trace** of the run. Include the relevant subset; link the trace for auditability. + +--- + +# B. Behavior & flow proof + +## B1. Visual before/after via the `mm` CLI (`visual-testing`) +- **Proves:** UI behavior on a real headed build, with controlled state/network. Defers to the public `visual-testing` skill. +- **Capture:** `yarn build:test:webpack` → `dist/chrome`; `yarn mm launch` → `mm describe-screen` / `mm screenshot` / `mm click` / `mm type` / `mm navigate`. README: `test/e2e/playwright/llm-workflow/`. + - **Degraded-path:** `mm mock-network` to force error/slow responses (session-scoped; add after launch, before the action; can't intercept pre-launch startup). + - **a11y / DOM:** `mm accessibility-snapshot` and `mm cdp` (per the `visual-testing` skill; `a11yRef`s are ephemeral — re-describe after navigation). + +## B2. E2E trace + video (Playwright / Selenium) +- **Proves:** a full flow works, replayably. The strongest "it works end-to-end" artifact. +- **Capture (Playwright):** `yarn playwright test `; trace is `'on'` by default (`playwright.config.ts`), video is `'off'` (enable in config if needed). View: `yarn test:e2e:pw:report`. Artifacts under `public/playwright/`. +- **Capture (Selenium):** `yarn test:e2e:single --browser chrome|firefox|all [--retries n]`; screenshots auto-captured on failure to `test/test-results/e2e/`. + +## B3. Falsifying regression test ⭐ +- **Proves — strongest single proof a fix targets the bug:** a new test that **fails on `main` and passes on the branch**. Show both runs. + - **Engine: the `falsifying-test` skill.** +- **Capture:** add the test, run it on the PR branch (pass) and on the PR's **merge-base** (fail) — pin the base, don't use whatever `main` points at today. Pair with the PR's `Fixes #N`. **Read the base failure's message, not its exit code:** it must fail on the assertion that encodes the bug. A `ModuleNotFoundError`, a missing fixture, or an unrelated pre-existing red produces an identical non-zero exit and falsifies nothing. +- **Reach for it:** every bug-fix PR. If you can't write a test that fails on main, question whether the fix addresses the reported bug. + +## B7. Deterministic interleaving test (concurrency / temporal-ordering) ⭐ +- **Proves:** an ordering guarantee under interleaving — retry, cancellation, supersession, debounce, locks, queues, async state machines — where the correctness *is* the ordering under races, not a value. +- **Capture:** force each race deterministically — `jest.useFakeTimers()` + `advanceTimersByTimeAsync(DELAY)` to fire the delayed action at a known point; `Promise.all([opA, opB])` to overlap operations; `advanceTimersByTimeAsync(0)` to step to a precise interleaving point; then assert the ordering/cancellation outcome for **each** guarantee, including asymmetric ones (one path canceled → its recovery event `.not.toHaveBeenCalled()`; another must complete → `.toHaveBeenCalledWith(...)`). Corroborate with transition telemetry; for the integration path, a live forced-race capture (C8/CDP, the #44610 technique). +- **Trust-gate:** the test must **actually interleave** — time advanced into the pending window, the superseding op injected *during* it. A sequential run exercises no race and is a vacuous green. Verify the interleaving, not just the assertion. + +## B4. Component / Storybook visual +- **Proves:** a component renders across states/props in isolation. +- **Capture:** `.storybook/` present; `yarn storybook` (port 6006), `yarn storybook:build`, `yarn test-storybook` (visual + a11y via `@storybook/addon-a11y`). Jest snapshot diffs for serialized output. + +## B5. Accessibility (a11y) +- **Proves:** no a11y regression / an a11y improvement. +- **Capture:** `yarn test-storybook` (Storybook a11y addon) for components; `mm accessibility-snapshot` for live flows. (No axe-core in the e2e suite — don't claim it.) + +## B6. Flaky-stability rerun +- **Proves:** a flow/test is not flaky (or that a fix removed flakiness). +- **Capture:** Playwright retries `1` on CI / `0` local (`playwright.config.ts`); Selenium `--retries n`; benchmarks default `--retries 2`. Run N× and report the pass rate. See `e2e-flakiness-patterns`. +- Sub: jest snapshot diffs; a unit run for just the changed module (`yarn test:unit `); fuzz/property tests for parsers/encoders. + +--- + +# C. Performance & render + +## C1. Startup / custom traces + phase segmentation +- **Proves:** which startup phase moved (init → FirstRender → interactive), per named span. +- **Capture:** `shared/lib/trace.ts` `TraceName` enum (UIStartup, LoadScripts, FirstRender, …); read in test/debug via `window.stateHooks.getCustomTraces()`. LCP fallback mark: `performance.mark('mm-hero-painted')`. `driver.collectMetrics()` aggregates paint/navigation/long-task/custom traces in e2e. + +## C2. Web vitals — INP / FCP / LCP / CLS +- **Proves:** a user-centric metric moved. `ui/helpers/utils/web-vitals.ts` via `web-vitals/attribution` (attribution names the causing element). +- **Capture:** `window.stateHooks.getWebVitalsMetrics()` (test/debug) → `{inp, fcp, lcp, cls, *Rating}`. Thresholds: INP good<200/poor>500, FCP<1800/3000, LCP<2500/4000, CLS<0.1/0.25. +- **Caveat:** **INP fires on all pages; FCP/LCP/CLS do not fire on popup pages** (sidepanel/E2E only). For extensions, INP is the high-value runtime metric. + +## C3. Long-task / TBT +- **Proves:** main-thread blocking during an interaction dropped. This is where **TBT** lives (the web-vitals lib lane does *not* collect TBT). +- **Capture:** `ui/helpers/utils/performance-observers.ts`; `window.stateHooks.getLongTaskMetricsWithTBT()` → `{count, totalDuration, maxDuration, tbt, tbtRating}`. TBT good<200 / needs-improvement<600 / poor>600. Sampled 10% prod / 100% test. + +## C4. React render & selector proof + - **Engine: the `react-render-proof` skill.** Delegate the measurement to it; it runs the source/delivery/metric gates, derives the needle from real build output, repeats the capture, and returns a band (or "not resolvable at this n" with an MDE). pr-validate packages the result. +- **Proves:** a component/selector stopped over-rendering (cascade-amplification before/after). +- **Capture:** WDYR via `ENABLE_WHY_DID_YOU_RENDER` (`.metamaskrc` or env) — wired in `app/scripts/development/wdyr.ts` (`trackAllPureComponents`); console logs each unnecessary re-render. `yarn devtools:react` for the Profiler flame graph. Selectors use `reselect`'s `createSelector`, which **does expose a real `.recomputations()` counter** — read it (sample on an interval if the count should visibly climb) rather than injecting a log into the selector body; an injected log is an authored claim, a library API is an observation. *(This entry previously said there was no built-in counter. There is.)* +- **Bar:** the delivery check comes before the number. An arm whose manipulation cannot be observed in the built bundle produces a null indistinguishable from "small effect" — and reports as the second. + +## C5. Benchmark A/B +- **Proves:** a startup/journey/interaction timing moved, with a distribution not one sample. +- **Capture:** `yarn test:e2e:benchmark` (`test/e2e/benchmarks/run-benchmark.ts`); presets in `shared/constants/benchmarks.ts` (`startupStandardHome`, `sendTransactions`, `swap`, `dappPageLoad`, …). +- **Caveat:** the rolling baseline (`MetaMask/extension_benchmark_stats`) can **silently freeze** behind a green check (the `store-benchmark-stats` step is `continue-on-error`; happened 2026-04-02, PR #42947). Prefer a **paired A/B** (build both refs now, compare directly) over the stored baseline. +- **Treatment check first** — before trusting any delta, confirm the mechanism under test is actually active in each arm (split chunk present in head and absent in base; the span emitted; the flag evaluated). An arm without the treatment delivered is a no-op, not a control (2026-07-22, #42795). +- **A null needs its power stated** — "no change" and "underpowered" print the same result. When the run-to-run spread exceeds the effect under test, report **not resolvable at this n** and name the smallest detectable effect; never let it read as "no effect". Correcting a known bias (discarding a warm-up, alternating the starting arm) removes *that* bias and nothing more — it is not a trust gate, and the confounds you did not enumerate (thermal drift, background load, ordering within a round) stay live. + +### Capturing an authenticated view (the in-situ requirement) + +Headless Chrome's `--screenshot` cannot set cookies, so an authenticated dashboard +(Grafana/Tempo, Sentry Discover, an internal panel) screenshots as a login page. Drive +Chrome over CDP instead — inject the session cookie, navigate, capture: + +```bash +COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN= \ + cdp-shot "" out.png 25000 1500 2400 +``` + +- **Deep-link to the exact view** so the capture and the reader's verification path are the + same URL (Grafana: `/explore?schemaVersion=1&panes=`). +- **Wait generously** — a trace waterfall or Discover table renders well after `load`. +- **Capture tall + `captureBeyondViewport`**, then crop; the interesting span is usually + below the fold, and cropping after the fact beats guessing a viewport. +- **Crop out the chrome that identifies the operator** (profile avatar, org switcher) + before the image leaves the machine. +- **Keep the trace/query id, timestamp, and result count in frame** — that is what makes + the exhibit reproducible rather than decorative. +- Never echo the cookie value, never commit it, never pass it to a subagent. + +## C6. DevTools / CDP profiling *(manual)* +- **Proves:** a flame-chart hot path shrank, a request was removed/deferred, frame rate held, or it holds on slow hardware. +- **Capture (manual via DevTools or `mm cdp`):** performance profile / flame chart; network waterfall (HAR) + request-count delta; **CPU throttling** (CDP `Emulation.setCPUThrottlingRate` — *no repo helper*, set it in DevTools); **animation/Rive FPS / dropped frames** (DevTools rendering FPS meter — *no repo helper*); JS coverage for dead-code. + +## C7. Memory stability over a flow *(manual)* +- **Proves:** a leak is fixed across repeated interactions (not one snapshot): retained heap stays flat, detached DOM nodes / listeners don't accumulate. +- **Capture:** DevTools heap snapshots before/after N cycles of the flow; compare retained size + detached nodes. +- Sub: redux dispatch/action count per interaction; network payload bytes; forced-reflow / layout-thrash count (DevTools Performance). + +## C8. Same-window app + DevTools capture *(manual)* +- **Proves:** the UI behavior **and** its internal evidence (console log, network row, storage state) in **one frame** — cause and effect temporally correlated in a single artifact. Two separate captures can't prove they came from the same run; one frame can. Canonical use: "the toast does NOT appear *while* the console shows the silent-handling path executed". +- **Capture (macOS, OS-level — Playwright `recordVideo` sees only the page viewport, never DevTools):** + 1. Tab-target DevTools: launch Chrome with `--auto-open-devtools-for-tabs` so DevTools opens **docked in the same window** (dock side persists per profile; set once via the DevTools ⋮ menu if a fresh profile defaults to undocked). + 2. MV3 **service-worker console has no dockable host** — open its dedicated inspector (`chrome://extensions` → *Inspect views: service worker*) and tile it flush beside the app window: `osascript -e 'tell application "Google Chrome" to set bounds of front window to {x, y, w, h}'` (the SW inspector is a Chrome window too and tiles the same way; CDP `Browser.setWindowBounds` also works per `windowId`).\ + 3. Record the union region, not a single window: stills `screencapture -x -R out.png`; video `screencapture -v -V -R out.mov`, then ffmpeg two-pass palette → GIF (recipe in [evidence-publishing](evidence-publishing.md)). First use prompts for macOS Screen Recording permission for the terminal. +- **Legibility rule:** console text dies in GIF downscale. Keep the GIF ≥720px wide, and pair it with (a) a full-res PNG of the same frame and (b) a text dump of the console via CDP (`Runtime.consoleAPICalled` on the SW target, `npx mm cdp` or a 20-line ws script) so the log lines are quotable/searchable. +- **Trust note:** arrange windows *before* triggering the behavior so the recording shows trigger → console line → UI (non-)reaction as one continuous take; a post-hoc composite of separate captures is exactly what this lane exists to avoid. + +--- + +# D. Build output + +## C9. Retention-path analysis — memory leak from code ⭐ *(static; lead for leak claims)* +- **Engine: the `memory-leak-hunt` skill.** For a memory-leak claim, delegate the analysis to `memory-leak-hunt` — it runs Phase-1 static pairing (and Phase-2 heap investigation if a primitive can't be paired) and returns the paired/unpaired sites + verdict. pr-validate keeps **memory leak** as the evidence category: it invokes the skill on the diff and packages the result (in-situ scan capture, plus the lifecycle test / retainer graph if Phase 2 ran) as the category's evidence. The lane spec below is the method that skill implements. +- **Proves:** "X is retained past its lifecycle boundary" / "collection Y grows unboundedly" — argued from code, no runtime needed. This is the lane that works at **review time** (does this PR *introduce* retention?) and leads fix-side validation (does the fix *break* the retention path?). C7 is the runtime corroborator, not the lead — leaks need many cycles to exceed noise. +- **Capture — the holder → held → boundary triple, per suspect:** (1) the **holder** (listener, closure, module singleton, accumulating collection, timer); (2) the **held set** — the *specific* objects pinned (list the closure's captures; note when a closure links two objects' GC); (3) the **outlived boundary** (`destroy()`, stream close, instance replacement, request completion). Method: **pair every acquire with its release site** (`on`↔`removeListener`, push↔drain, assign↔null) — the absence of the pair, cited at the acquire site, IS the finding. Four canonical shapes: unbounded accumulator (defeated guard, no drain) · stale-instance listeners on replacement · unremoved listener + capture set · retention past `destroy()`. +- **Scope to the diff, or you invent findings.** Classify every flagged primitive as *introduced by this PR* (in the added lines) vs *pre-existing* (already in the file). Charge only the introduced ones to the PR; report pre-existing un-paired primitives separately and uncharged. On extension#40684 the two new stream listeners each had a `removeListener` on `onStreamClosed` (the exact fix a reviewer suggested) and the new pending-request Map had its `.delete` — no leak introduced — while three pre-existing un-torn-down listeners were surfaced but left uncharged, matching how the human/bot reviewers treated them in-thread. This lane *is* the retention review automated; a heap snapshot (C7) is warranted only for an introduced primitive it cannot pair. +- **Corroborate:** a falsifying lifecycle test (force the boundary, assert release — listener count zero, singleton nulled, collection drained); C7 heap-over-flow with the **retainer graph naming the same path** the static argument named. +- **Trust-gate:** the triple must be specific ("this listener holds `patchStore` after `patchStore.destroy()`", not "might leak"); distinguish **bounded staleness vs unbounded growth** (severity differs); attribute **introduced vs pre-existing** honestly. + +## D1. Bundle-size diff +- **Proves:** the build grew/shrank by a measured amount. Use the bundle-size CI output or a local build size comparison. + +## D2. Chunk membership / source-map +- **Proves:** a module moved to the intended (lazy) chunk and no longer ships on the critical path. Requires the webpack build. Mirrors AEP `perf-chunks`. + +## D3. LavaMoat policy / supply-chain capability diff + - **Engine: the `supply-chain-audit` skill** (umbrella — lockfile/manifest diff, advisories, Socket Security, install scripts), which delegates capability grants to **`lavamoat-policy-diligence`** (per-grant call-site justification). Delegate the dependency change to the umbrella; it returns a disposition per lane. pr-validate keeps **supply-chain capability diff** as the evidence category and packages the output. Note the lanes are independent: a clean policy diff does not mean a safe dependency, and a known CVE never appears as a new grant. +- **Proves:** a dependency change (bump/add/lockfile) grants **no *unjustified* new capability** — the supply-chain-risk lane. Note the bar: for a bump the policy *will* change, so "empty diff" is the WRONG test; the right test is **every new grant is justified by the dep's function**. The framing generalizes past LavaMoat to any capability-containment mechanism. +- **Capture:** **Prefer the CI-generated policy whenever one is available.** `@metamaskbot update-policies` regenerates the policy files from a real run of the code and `validate-lavamoat-policies` fails the build on drift, so the committed policy on a bot-run PR *is* the authoritative artifact — diff that. Regenerating locally when a current CI policy exists only re-does a machine that is already trusted, and a local run's provenance is weaker (your node/OS/lockfile resolution, not CI's). **Local regen is the fallback**, for when the bot hasn't run yet, the branch is unpushed, or you need a variant CI didn't cover: `yarn webpack:lavamoat:policy:build` (`:mv2` / `:mv3` for variants) over `lavamoat/webpack/build/policy.json` (+ `policy-override.json`). Either way, `git diff` the policy across **all 8 variants** (mv{2,3}/{main,beta,flask,experimental}) — a grant can appear in one and not others. Then audit **grant-by-grant**: new **globals** (`fetch`, `importScripts`, `WebAssembly`) / **builtins** (`fs`, `child_process`) on a dep that shouldn't need them, new **packages** edges to powerful APIs, or an identifier substitution (`pkgC>name` replacing `pkgB>pkgA>name` = possible dep swap). Falsifier = a surprising grant ("I wonder what it's using this for"). Guide: lavamoat.github.io/guides/policy-diff/. `allowScripts` in `package.json` gates install scripts. + +## D4. Manifest permissions diff +- **Proves:** no permission/host-permission scope creep. +- **Capture:** `git diff app/manifest/v3/_base.json app/manifest/v2/_base.json` (+ `chrome.json`/`firefox.json`). Flag new sensitive perms (webRequest, broad host patterns). + +## D5. Build-variant matrix +- **Proves:** the change works across build types, not just main. +- **Capture:** `yarn build:test:flask` / `:beta` / `:mv2` (`ENABLE_MV3=false`, Firefox). Run the relevant lane per variant when behavior is build-type-gated. + +--- + +# E. Production telemetry + +## E1. Sentry query links (before/after) +- **Proves:** error-rate / transaction count / latency moved in prod. A link a reviewer opens beats a chart screenshot. +- **Capture:** Sentry MCP (`search_events`/`search_issues`) → hand the discover/dashboard link with the before/after window, scoped to the release. Projects: `metamask` = prod, `metamask-performance` = CI. +- **Boundary:** PRs that *add/change span instrumentation* (volume/quota) → `/sentry-quota`, not this lane. +- **Perf-PR promotion (standard, not just complementary):** for a **performance-focused PR**, the main-branch Sentry **trend** for the affected metric across the PR's merge (before/after the merge commit's release) is **standalone lead evidence** — CI already sends every main/release `startupPowerUserHome` / journey benchmark to Sentry, so the trend is a real before/after on the actual metric, continuously tracked, with no local run. Prefer it over a local paired A/B when a clean merge-boundary window exists: it sidesteps the stale committed-baseline trap (`benchmark-baseline-staleness-paired-ab`). Still bound to the trust gate — a **windowed, release-scoped, one-click-resolvable** trend link with the merge boundary visible, never a prose "looks fine." A local interleaved paired A/B (C5) remains the precision complement when the merge window is noisy or the metric CI doesn't track (selector-eval count, re-render count, INP-on-typing — none of which CI captures). + +## E2. Tempo distributed traces +- **Proves:** a span/transaction now appears / is shaped correctly (e.g. background-RPC tracing). Link the trace + note the release. + +## E3. Sentry error-event / breadcrumb shape +- **Proves:** an instrumentation PR captures the intended error-event state / breadcrumbs (relevant after the Sentry-v10 error-event capture changes). Show the captured event payload. + +--- + +# F. Extension integrity (high-stakes, extension-specific) + +## F1. State migration / upgrade ⭐ +- **Proves:** a persisted-state change doesn't corrupt existing users. +- **Capture:** migrations in `app/scripts/migrations/NNN.ts`, runner `app/scripts/lib/migrator/`; scaffold with `./development/generate-migration.sh NNN`. The `NNN.test.js` asserts `meta.version` and that the `changedKeys` Set covers only mutated controllers — i.e. untouched state is preserved. Run it; show old-state-in / new-state-out. + +## F2. Vault / keyring round-trip +- **Proves:** no key/vault corruption; encrypt→decrypt is lossless. +- **Capture:** `app/scripts/lib/encryptor-factory.ts` (`@metamask/browser-passworder`, PBKDF2). E2E: `test/e2e/dist/vault-decryption-chrome.spec.ts`; `test/e2e/tests/vault-corruption/`. Storage-size via `getFileSize` on the encrypted blob. + +## F3. Transaction simulation / gas +- **Proves:** tx behavior/balance-changes/gas are correct before submit. +- **Capture:** `app/scripts/lib/transaction/containers/enforced-simulations.ts`; e2e `test/e2e/tests/simulation-details/`; mock `test/e2e/tests/confirmations/mocks/simulation.ts` (returns `gasUsed`, `callTrace`, `stateDiff`, token balance changes). TX_SENTINEL_URL in `shared/constants/transaction.ts`. + +## F4. Provider / dapp connectivity +- **Proves:** dapp integration works (injection, connect, requests). +- **Capture:** `yarn dapp` (serves `@metamask/test-dapp` on :8080); EIP-6963 `test/e2e/provider/eip-6963.spec.js`; multi-provider `test/e2e/multi-injected-provider/`; EIP-1193 reconnect tests under `test/e2e/tests/mm-connect/`. + +## F5. Feature-flag matrix (on/off) +- **Proves:** correct behavior in both remote-flag states (the Perps-gating class of bug). +- **Capture:** remote-feature-flag-controller (`app/scripts/lib/update-remote-feature-flags.ts`); flags come from `client-config.api.cx.metamask.io/v1/flags` — **not** `.metamaskrc`. In e2e, mock the response (see `test/e2e/tests/remote-feature-flag/`) to force each state; read via `uiState.metamask.remoteFeatureFlags`. + +## F6. Snaps / multichain execution +- **Proves:** snap behavior across multichain (e.g. `snap_startTrace`/`snap_endTrace`). +- **Capture:** `test/e2e/flask/snaps/preinstalled-example.spec.ts` (the snap-trace test), broader `test/e2e/snaps/`. Build flask (`yarn build:test:flask`). + +## F7. i18n usage +- **Proves:** no hardcoded strings; locales resolve. +- **Capture:** `yarn verify-locales` (`development/verify-locale-strings.js`); locales in `app/_locales/`. `yarn verify-locales:fix` to auto-fix. + +## F8. SES lockdown / runtime containment ⭐ +- **Proves:** the runtime defenses are **actually in force in the shipped artifact** — SES `lockdown()` and its taming levels, LavaMoat global scuttling, Snow's anti-escape hooks, Snaps compartments. Distinct from D3: D3 is the build-time *policy* (what a package may reach), this is whether containment *holds at runtime*. A correct policy ships alongside a lockdown that silently failed, and no policy diff would show it. +- **Capture:** `Runtime.evaluate` over CDP against the **built variant under discussion** — `Object.isFrozen(Object.prototype)`; a scuttled global throws while an exception-list global still resolves; `typeof SNOW === 'function'`; the `lockdown({…})` options as they appear *in the bundle*. Pair a positive with a negative — a check that only confirms the permitted case passes in a completely unlocked environment. +- **Bar — three divergences make this a lane, not a checkbox:** (1) the `lockdown()` call is wrapped in `try/catch` that logs to Sentry and **continues unlocked** (added for Firefox v56 contentscript injection), so it is a runtime assertion, never a guaranteed precondition; (2) **scuttling is off entirely in DEV builds** (`shouldScuttle = entryTask !== BUILD_TARGETS.DEV`); (3) **TEST builds widen the scuttling exception list** for chromedriver (`Proxy`, `ret_nodes`, `browser`, `chrome`, `indexedDB`). So **a green e2e run is evidence about a wider-open global than users get** — always state which build variant produced the evidence. +- **Reach for it:** any change touching the lockdown call site or its ordering (lockdown must precede untrusted code), the scuttling exception list, a taming level, compartment boundaries, or a `@lavamoat/snow` bump (Snow is patched in-repo — re-read the patch; see `supply-chain-audit`'s patch lane). + +--- + +# G. CI, review & process + +- **G1. CI check links** — `gh pr checks `; link the full suite (AEP's bundle is often `partial`). Always worth a one-line "all green" + link. +- **G2. Coverage delta** — `yarn test:unit:coverage` → `coverage/unit/` (and `yarn test:unit:webpack:coverage`); `codecov.yml`. Proves the new code is exercised. +- **G3. Automated-reviewer output** — independent bot (e.g. cursor[bot]) found nothing blocking. Complements, never replaces, behavior evidence. +- **G4. Manual reproduction steps** — human-followable steps that reproduce the fixed behavior; populates the PR template's Manual testing steps. +- **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: in a fork you control, push to a branch literally named **`main`** (or `stable`) — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets configured on that fork (the benchmark jobs need the Infura and test-account secrets; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo. + +--- + +# Matching guide (claim → lanes) + +| The PR claims… | Lead with | Corroborate | +|---|---|---| +| a visible UI behavior | A1 / B1 visual | B2 recording for motion; B5 a11y | +| a fixed bug (any) | **B3 falsifying test** | A1/B1 if visible; E1 if it errored | +| preload / no-double-fetch / lazy-load | A2 perf | C6 netlog, D2 chunk | +| a render/over-render fix | C4 WDYR/profiler | C1 traces | +| interaction responsiveness | C2 INP, C3 TBT | C6 profile | +| startup/load timing | C5 benchmark (paired) | C1 phase traces, C2 FCP/LCP | +| smaller/cleaner bundle | D1 size | D2 chunk | +| a memory leak fixed / introduced | **C9 retention-path from code** (holder → held → boundary) | C7 heap-over-flow + retainer graph; falsifying lifecycle test | +| an error/crash fixed | E1 Sentry rate→0 | B3 test, A1 if visible | +| a dep change is safe | D3 LavaMoat + D4 manifest | D1 size; supply-chain-audit's patch/resolutions/ignore lanes | +| runtime containment / SES / scuttling | **F8 runtime containment** (on the shipped variant) | D3 policy; E1 for `Lockdown failed` events | +| persisted-state change | **F1 migration** | F2 vault | +| tx/confirmation behavior | F3 simulation | B2 e2e | +| dapp/provider behavior | F4 connectivity | B2 e2e | +| flag-gated behavior | F5 flag matrix | A1/B1 per state | +| snap behavior | F6 snaps | E2 trace | +| copy/localization | F7 i18n | A1 visual | +| CI workflow behavior | **G5 fork run** (branch named `main`) | G1 checks, G4 repro steps | + +Run the cheapest lane that yields an independently re-checkable artifact, confirm the claim holds, then escalate. Don't over-instrument a one-line copy fix; don't under-prove a startup-latency or migration claim with a single screenshot. diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-gate-setup.md b/domains/pr-workflow/skills/pr-validate/references/evidence-gate-setup.md new file mode 100644 index 00000000..f1df5b9c --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-gate-setup.md @@ -0,0 +1,58 @@ +# Evidence gate — setup (optional, Claude Code only) + +`hooks/pr-evidence-gate.py` is an **optional** mechanical enforcement of the disciplines documented in [`evidence-trustworthiness.md`](./evidence-trustworthiness.md). It is a Claude Code `PreToolUse:Bash` hook: before an outward-facing write runs, it scans the body for a validation-scoped claim the trustworthiness gate would reject, and blocks the write if it finds one. + +**Surfaces policed:** the `gh pr|issue edit|create|comment` porcelain (`--body`, `--body-file`) *and* `gh api` body writes (`-f body=…`, `-F body=@file`, `--input file.json`) — a PATCH to a comment is the same publish with a different spelling, so a porcelain-only matcher is a hole rather than a gate. Read-only `gh api` calls pass through untouched. + +**Classes enforced:** `verdict`, `observation`, `deferral`, `ci-restatement`, `inflated-verdict`, `bare-identifier`, `truncated-identifier`, `mutable-ref`, `dump-resolver`, `link-only-exhibit`, `data-only-exhibit`, `step-waiver` — each implementing a numbered item of [`evidence-trustworthiness.md`](./evidence-trustworthiness.md). What the hook cannot see (whether a screenshot shows the resolving UI, whether a deferral's blocker matches its step, quotation fidelity) stays reader-applied. + +The hook is **Claude-Code-specific**. Other operators (Cursor, Codex, plain review) don't get the mechanical gate — for them the same disciplines apply as *documentation*, self-enforced by reading `evidence-trustworthiness.md`. The hook is not required to use the skill; it just moves the checklist from "remember to run it" to "runs automatically at emit time." + +It **fails open**: anything it cannot parse (non-`gh` command, unreadable body, malformed JSON) is allowed through, so it never bricks unrelated Bash commands. It uses the Python 3 standard library only (`json`, `re`, `sys`) — no dependencies to install. + +## Wire it up (Claude Code `settings.json`) + +Add a `PreToolUse` hook with matcher `Bash` that runs the script with `python3`. Put this in your user `~/.claude/settings.json` or a project `.claude/settings.json`: + +```json +{ + "hooks": { + "PreToolUse": [ + { + "matcher": "Bash", + "hooks": [ + { + "type": "command", + "command": "python3 /absolute/path/to/pr-validate/hooks/pr-evidence-gate.py" + } + ] + } + ] + } +} +``` + +Resolve the path to wherever `pr-validate` lives on disk. Note that `tools/install` copies only the `references`/`scripts`/`assets`/`adapters` bundles into `~/.claude/skills/mms-pr-validate/` — the `hooks/` directory is **not** part of the installed bundle. Point the `command` at your checked-out skills repo instead: + +``` +/domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py +``` + +**When it blocks:** the hook exits `2` and prints the reason (which claim, what artifact/tracker it needs) to stderr. Claude Code surfaces that to the model, which self-corrects — attaches the missing artifact/tracker or downgrades the verdict — and re-posts. No manual intervention needed. + +## Two other setup requirements the skill needs + +These are independent of the hook; the skill needs them whether or not you install the gate. + +1. **`gh pr comment` must be permitted — pick a grant model.** pr-validate posts its evidence bundle as a PR review comment (`gh pr edit` if publishing into your own PR body). Four options, in descending order of standing safety: + + | Model | How | Tradeoff | + |---|---|---| + | **`ask` (recommended)** | `"Bash(gh pr comment:*)"` in `permissions.ask` | Per-post confirmation prompt. Combined with this hook (content gate) and a draft-confirm habit, that's three independent layers. | + | **`allow` + hook** | same pattern in `permissions.allow`, hook wired | Frictionless posting; safety rests entirely on the hook and your draft discipline. Only sensible where the hook is actually installed — not for operators without hook support. | + | **Allowlisted wrapper** | keep raw `gh pr comment` denied; allowlist a small script that takes `--repo`/`--pr`/`--body-file`, checks preconditions (canonical header present), and is the only sanctioned path | Tightest scoping — the raw verb stays blocked; costs a script to maintain. | + | **No grant — manual post** | the model prepares the body file; you run `gh pr comment --repo --body-file ` yourself | Zero standing grant; you are the bottleneck. The universal fallback, and the only option on operators with no permission system. | + + Avoid a bare **deny** on the comment verbs if you use this skill: it hard-blocks the publish step with no prompt, which reads as a mysterious failure mid-run. + +2. **Image re-hosting needs your own public evidence repo.** Screenshots and recordings captured locally must be re-hosted to a public URL before a reviewer can see them (see items 8–9 in `evidence-trustworthiness.md`). This repo is **yours to provide** — set it to a public repo you control, referenced here as ``. There is no shared/default host: parameterize it in your own configuration and push captures there, then reference the resulting raw URLs in the PR comment. Do not hardcode someone else's host. diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md b/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md new file mode 100644 index 00000000..b1a46e1c --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md @@ -0,0 +1,291 @@ +# Publishing the evidence bundle to a PR body + +How to take run artifacts + complementary evidence and write a clean, idempotent, reviewer-familiar section into the PR body — **matching AEP's own format** so a re-run replaces in place instead of stacking duplicates. + +Canonical source for the format: `packages/github/src/pr-body-builder.ts` (`upsertVisualValidationSection`) in the [AEP repo](https://github.com/MetaMask/metamask-autonomous-engineering-platform). Mirror it. + +> **Publishing is public and outward-facing. Always render the section and get explicit confirmation before writing the PR body. Use `publishEvidence: false` on the run; this manual flow is the only publish path.** + +## Step 1 — Re-host images (artifacts are localhost) + +Control-plane artifact URLs (`localhost:3000/v1/runs/:id/artifacts/:name`) won't render on GitHub. Re-host each artifact and link the hosted URL. + +**Host: an object store or repo whose read access matches your audience.** Configure it once and +reuse it; the examples below assume an S3 bucket exposed through an environment variable: + +```bash +# set these to a bucket you control whose `public/` prefix allows anonymous GetObject +EVIDENCE_BUCKET= +EVIDENCE_BASE="https://$EVIDENCE_BUCKET.s3..amazonaws.com" +``` + +``` +s3://$EVIDENCE_BUCKET/public/metamask/pr-// +$EVIDENCE_BASE/public/metamask/pr-// +``` + +Allow anonymous `GetObject` under `public/*` but not bucket listing, so the prefix is not +browsable — link individual files, and don't promise readers an index. + +**Do NOT re-host to a personal repo.** A personal private repo returns 404 for every reader but +its owner, so every raw link to it is dead on arrival. + +The test is **audience-reachability, not public-vs-private.** An org repo that is private but +readable by colleagues is fine for an internal-audience link. A personal repo is unreachable by +colleagues *and* by the public, so it fails for every audience. + +- Path convention: `pr-//` keeps runs from colliding. +- **Verify unauthenticated before shipping**: `curl -s -o /dev/null -w "%{http_code}"` on each + published URL. A 200 from your own browser proves nothing — you are logged in. + +```bash +RUN_ID=; PR=; CP=localhost:3000 +BUCKET="$EVIDENCE_BUCKET" +BASE="$EVIDENCE_BASE" +for name in ; do + curl -fsS "$CP/v1/runs/$RUN_ID/artifacts/$name" -o "/tmp/$name" + key="public/metamask/pr-$PR/$RUN_ID/$name" + aws s3 cp "/tmp/$name" "s3://$BUCKET/$key" --only-show-errors + url="$BASE/$key" + # the link is not shippable until it resolves WITHOUT credentials + code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 25 "$url") + [ "$code" = "200" ] || { echo "UNREACHABLE ($code): $url" >&2; exit 1; } + echo "$url" +done +``` + +No base64 round-trip and no 1 MB contents-API ceiling — the ceiling silently truncated a +1.7 MB gif to **0 bytes** on one run, and the loop reported success. Size-check anything you +transfer by another route. + +Files >1MB exceed `ARG_MAX` for an inline `-f content=` — use `gh api -F content=@` (write the base64 to a file first). For GIFs, re-host the same way. + +## Step 2 — Build the section (canonical header + mirror AEP) + +**Canonical header (2026-07-21):** every validation-run output — a PR comment *or* the PR-body section — leads with the exact literal `## 🧪 Validation Run`. Never reworded, never demoted to `###`: the constant string is the identifiability anchor, exactly like Copilot's fixed `## Pull request overview`. `hooks/pr-evidence-gate.py` blocks any `gh` write whose body has a validation/verification/evidence heading or AEP marker without this literal. + +Marker pairs, used so re-runs replace idempotently: + +- Whole section: `` … `` +- AEP status block (nested): `` … `` +- Screenshots block: `` … `` + +AEP prefers to inject screenshots into the PR template's `### **After**` section (replacing the `` placeholder), falling back to a `### Screenshots` block inside the status block when there's no After scaffold. Do the same. + +Section shape: + +```markdown + +## 🧪 Validation Run + +**Verdict:** ✅ proven — **Claim:** +head `` · · lanes: + + + + +### AEP Visual Validation + +**✅ Passed** + + + +
Validation details + +** — . " lines> + +
+ +Run `` · [LangSmith trace]() + + +``` + +Verdict icon: `✅` Passed, `❌` Failed, `ℹ️` otherwise. For perf, retitle the nested block `### AEP Perf Validation` and put `M/M assertions proven` in the headline. When AEP's *service* publishes its own `## AEP Visual Validation` block (publishEvidence:true, not the local flow), leave that block's heading alone — the demotion to `###` applies to hand-assembled bundles under the canonical header. + +Screenshots block (injected into `### After`, or appended under `### Screenshots`): + +```markdown + +
+ +<artifact-name> + +[Open full-size image]() + +
+ +``` + +`
` so reviewers see evidence without a click. One block per image; before/after read top-to-bottom. + +## Step 3 — Choose the surface by ownership, then publish + +**Publish surface depends on your relationship to the PR.** Determine it FIRST: + +```bash +PR=; REPO=MetaMask/metamask-extension +ME=$(gh api user --jq .login) +SURFACE=$(gh pr view "$PR" --repo "$REPO" --json author,commits --jq --arg me "$ME" ' + if .author.login==$me then "body" + elif ([.commits[] | select(.authors[].login==$me) + | select([.authors[].login] | map(select(.!=$me and .!="Copilot" and (test("claude|anthropic")|not))) | length == 0)] | length) > 0 + then "comment" else "skip" end') +``` + +- `body` — I authored the PR → upsert into the PR body (below). Validation is + part of my own claim. +- `comment` — not author but I have a solo commit (no HUMAN co-author) → post a + `gh pr comment` under the canonical `## 🧪 Validation Run` header. Never edit + someone else's PR body. +- `skip` — my only commits are co-authored with a human (review/pairing) OR I + have no commits → **do not publish**. Not my PR to validate outward. + +### Publish the script that produced a computed artifact, next to the artifact + +Any number you derived rather than read off a tool — a hash comparison, a count, a delta, a +statistic — is only as trustworthy as the reader's ability to re-run it. **A prose `method:` field +is not provenance.** Reviewers discount computed figures from an agent by default, and correctly: +on extension#45024 a reviewer dismissed a policy-identity check as *"we know LLMs are really bad at +this"*. It had in fact been a deterministic `sha256`, not the model counting — but the script lived +in a throwaway `python3 - <<'PY'` heredoc, so nothing could show that. The objection was +unanswerable because of how the evidence was packaged, not because of what it said. + +So: + +- **Write the script to a file, never an inline heredoc**, when its output will be published. The + heredoc survives only in the transcript, which the reader does not have. +- **Publish the script alongside its output**, and cross-reference: the artifact carries + `provenance: { script, script_sha256, command }`; the comment links the artifact. +- **Include the exact command** with its inputs (PR ref, head SHA), so the run is reproducible + rather than merely described. +- **Verify the round trip** — fetch the published script anonymously, hash it, and confirm it + matches `script_sha256`. A link that 200s is not proof the bytes are the ones you ran. +- Prefer a script that takes arguments and is re-runnable against a different PR. A one-off that + only works on your paths is weak provenance even when published. + +State plainly what the script does and does not do (`no model judgement; not a count of '+' +characters`) — that sentence is what actually retires the reviewer's prior. + +### Before any of the commands below: show the body in the response + +Every publish path here uses `--body-file`, so the **permission prompt displays a file path, not +the content**. The user is then asked to authorize publishing something under their name that they +cannot read, and the correct answer to that is no. + +**Paste the complete body inline in the response first, then run the command.** For an edit, also +say what changed relative to what is currently live. "I've drafted it, shall I post?" with a path +instead of the text is incomplete — pointing at `/tmp/validation-run.md` is the same failure as the +prompt itself. If the body is too long to show comfortably, that is a signal to trim it. +(Three consecutive denials on extension#45024, 2026-07-30, all from this.) + +### body surface (I own the PR) +```bash +gh pr view "$PR" --json body -q .body > /tmp/pr-body.md +# Replace the region between VALIDATION_RUN markers if present, else append. +# (Legacy bodies: replace the AEP_VISUAL_VALIDATION region and re-wrap it under +# the canonical "## 🧪 Validation Run" header + VALIDATION_RUN markers.) +# Replace the region between AEP_SCREENSHOTS markers if present, else inject after +# the "### **After**" heading (replacing the [screenshots/recordings] placeholder). +# ...edit /tmp/pr-body.md... +gh pr edit "$PR" --body-file /tmp/pr-body.md +``` + +### comment surface (I contributed but don't own) +```bash +# Same canonical "## 🧪 Validation Run" header + bundle; post as a comment. +gh pr comment "$PR" --repo "$REPO" --body-file /tmp/validation-run.md +``` + +Idempotency: because both regions are marker-delimited, re-running replaces them — never append a second copy. If the markers are absent (human-authored body), append the status block at the end and inject screenshots into `### After` when that heading exists. + +## Step 4 — Privacy scrub (before writing) + +Failure summaries and agent narratives leak the dev environment. Before publishing, strip: + +- Absolute local paths (`/Users//…`, `~/Code/…`) → describe the surface, not the path. +- The username anywhere it appears. +- `localhost` / `127.0.0.1` URLs → must be re-hosted public URLs only. +- Internal hostnames, JFrog/registry URLs, tokens. + +A failed run still must not publish raw — either omit the section or publish a scrubbed `❌ Failed` summary, with confirmation. + +## Recordings → GIF (for flows/motion a still can't prove) + +The platform can't collect video (artifact regex = png/jpg/log/txt). Capture out-of-band: + +1. In a **built** PR checkout (mm's fixture infra is required — a bare `dist/chrome` won't boot), write a preload `/tmp/patch-record.mjs` that monkey-patches `playwright-core`'s `chromium.launchPersistentContext` to inject `recordVideo: { dir }`. Resolve the module via `createRequire(/package.json)` so the patch hits the same module instance the `mm` daemon uses. +2. `NODE_OPTIONS="--import /tmp/patch-record.mjs" npx mm launch --state onboarding` → drive the flow (or let it sit) → `npx mm stop` flushes the `.webm`. States: `default | onboarding | custom`. +3. Convert with `ffmpeg` two-pass palette (better color than single-pass): + ```bash + ffmpeg -i in.webm -vf "fps=12,scale=480:-1:flags=lanczos,palettegen" -y /tmp/pal.png + ffmpeg -i in.webm -i /tmp/pal.png -lavfi "fps=12,scale=480:-1:flags=lanczos[x];[x][1:v]paletteuse" -y out.gif + ``` + webm/mp4 don't render inline in GitHub PR bodies; GIF does. +4. Re-host the GIF (Step 1) and embed like a screenshot. + +**Same-window app + DevTools (lane C8):** when the claim needs UI + console/network in one frame (e.g. "no toast *while* the log shows the silent path ran"), skip `recordVideo` entirely — it can't see DevTools. Use the OS-level region capture in [evidence-catalog C8](evidence-catalog.md): dock tab DevTools with `--auto-open-devtools-for-tabs`, tile the SW inspector window via `osascript`/CDP `Browser.setWindowBounds`, then `screencapture -v -V -R` → same ffmpeg GIF recipe. Publish the GIF + one full-res PNG + the CDP console text dump (GIF downscale makes log lines illegible on their own). + +## Re-validation runs: delta-first presentation, every verdict re-earned (2026-07-21) + +The common loop — a run refutes a claim, the author pushes a fix, `/pr-validate` re-runs at the new head — gets a **delta report**, not a second full bundle: + +- **Presentation is delta-only.** Full exhibits only for lanes whose outcome changed (flipped verdict / new lane / new residual). Unchanged lanes collapse to a `Prior run | This run` ledger, each row with a fresh run-log link from the new head plus one link to the prior run's comment for the full exhibits — and say so ("unchanged rows re-run at ``; full exhibits in the prior run"). +- **Evidence is never delta.** Evidence is head-pinned: re-run every automated lane at the new head and re-earn every verdict with a fresh artifact. "Unchanged" is a conclusion from the re-run, never a carried-over assumption (the stale-baseline trap at report level). Re-running is cheap — the falsifier harness already exists from the first run. +- Same canonical header + markers; the meta line names the fix commit and links the prior run. Comments: one per run, chronological, each linking its predecessor. PR-body section: replaced in place via markers. +- New head → **new hosted artifact directory keyed to the fix commit** (`pr-/fix-/`), commit-pinned raw URLs; never overwrite a prior run's published files. +- Residuals the fix intentionally leaves get their own row/section — don't round a fixed-with-residual claim up to fully proven. + + +## Lead with a lane-status ledger (no silent absence) + +The published section must **enumerate every lane the claim type calls for and give each an explicit status** — never render only the lanes you happen to have and let the rest be silently absent. An unmarked gap is indistinguishable from a lane that ran and came back empty; the reader (and you, on the next pass) can't tell "no evidence because none needed" from "no evidence because not done." This is the vacuous-pass trap at the publish layer — carry the run's `✅/❌/⚠️` verdict into the PR body, don't leave it in the internal report-back. + +Open the evidence section with a ledger: + +```markdown +| Lane | Status | Evidence | +|---|---|---| +| B3 falsifying test | ✅ proven | 32/32 head, 3/32 reverted | +| E1 Sentry before/after | ✅ proven | [discover](…) — distinct trace ids | +| A1 visual | ➖ N/A | background change, no UI surface | +| C6 CDP netlog | ⏳ not-captured | — | +``` + +Status vocabulary: `✅ proven` (link) · `⚠️ inconclusive` (name what's missing) · `➖ N/A` (reason) · `⏳ not-captured`. Mirror the `N/A — ` convention the `### Screenshots` block already uses for no-UI PRs. Never upgrade a `⏳`/`⚠️` to a pass by omission. + +**Sibling-PR parity:** when a set of PRs shares a claim shape (same program, same author, "root the X traces"), their ledgers must match lane-for-lane. A lane present on one and absent on another is either added or explicitly marked `➖ N/A — ` — a bar that silently drifts between siblings is a finding (postmortem 2026-07-17, #43929/#43930). + +## Non-visual & multi-lane evidence + +Screenshots are only one lane. Most claims (perf, telemetry, state, build) publish as **text/links/tables**, not images. Put them in the same verdict-first section so a reviewer sees one coherent bundle, not scattered comments. + +Per-lane rendering: + +- **Sentry / Tempo (E1/E2):** a markdown link to the discover/trace query with the before/after window baked in, plus the headline numbers inline (`errors: 1.2% → 0.0% over 24h post-release`). Link, not screenshot — reviewers re-run it. +- **Benchmark / web-vitals / TBT (C2/C3/C5):** a small before/after table (metric · base · head · Δ · threshold). State it's a **paired A/B** if the stored baseline was bypassed. +- **Migration (F1):** the `changedKeys` set + a before/after state-shape snippet, and a link to the migration-test run. +- **Bundle / chunk / LavaMoat / manifest (D1–D4):** the diff or size delta in a fenced block; for policy/manifest, the actual `git diff` (or "diff empty — no new capability"). +- **Trace artifacts (B2):** link the Playwright trace-viewer report / attach the `trace.zip`; don't paste raw. + +Multi-claim PRs get one sub-block per claim under the status section, each with its own ✅/❌/⚠️ verdict — mirror the Claim Cards. Keep the visual block (markers + `### After` injection) for the image lanes; render the rest as text beneath it. + +**Per-scenario presentation (2026-07-21):** the same applies one level down — when the evidence spans multiple test scenarios (flag-on vs flag-off, control vs treatment in an A/B falsifier, numbered manual-testing steps), give each scenario its **own sub-section**: a heading naming the scenario in observation terms, one line on what it tests plus its verdict, and that scenario's artifacts co-located under it. Never bunch all scenarios' artifacts into one large evidence dump — the reviewer verifies "under condition X, artifact shows Y" one condition at a time, and a merged block destroys that mapping even when every artifact is real. For long artifact sets use a `
` block *per scenario*, not a merge. + +## Artifact contract (ADR-0058 alignment) + +To stay interoperable with the recipe-based verification system (MetaMask/decisions#173), shape the bundle like its reviewer-visible contract where practical: a `summary.json` (claim → verdict → evidence refs), a `trace.json` (the run/assertion log), and an artifact manifest (names + media types), with screenshots/video as the confidence layer. Publishing then becomes "render `summary.json` into the PR section." This keeps pr-validate's output and a recipe's output the same shape — see [lane-assertions.md](lane-assertions.md). Don't hand-roll a divergent format. + +## Checklist before you publish + +- [ ] Section opens with a **lane-status ledger** — every claim-required lane marked `✅`/`⚠️`/`➖ N/A`/`⏳`; no lane silently absent (and sibling PRs' ledgers match lane-for-lane) +- [ ] `evidenceBundle.artifactRefs` non-empty with expected media (not a vacuous pass) +- [ ] Each lane passed the [trustworthiness gate](evidence-trustworthiness.md) (shows the claimed surface, signal > noise, could-have-failed) +- [ ] Multi-scenario evidence rendered **per scenario** (own heading + verdict + co-located artifacts), not bunched into one block +- [ ] **Automated-process voice, no first person** — published validation output never says "I ran/captured/verified"; attribute to the process ("Automated validation ran…", "the harness captured…") so readers know the evidence is machine-generated, not a manual account under the author's name +- [ ] Every image/GIF re-hosted to your configured evidence host; no localhost/local-path URLs in the body +- [ ] **Every published link curl'd unauthenticated and returning 200** — never a personal private repo +- [ ] Work cited by **PR link** rather than tracking-ticket id, unless the ticket's own content (an RCA, a spec) is the referent +- [ ] Narrative scrubbed of username/paths/internal hosts +- [ ] Marker pairs present so the upsert is idempotent +- [ ] Section rendered and **confirmed by the user** diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md b/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md new file mode 100644 index 00000000..8933a8de --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md @@ -0,0 +1,43 @@ +# Evidence trustworthiness (anti-reward-hacking) + +A green result is not proof. An agent — or an eager run — can produce evidence that *looks* like it validates the claim but doesn't. Before believing or publishing any lane, run it through this gate. It extends the vacuous-pass trap to all lanes; the Claim Card's **Falsifier** is the anchor: trustworthy evidence is evidence that *could* have shown the falsifier and didn't. + +> **Which items are mechanically enforced.** Several items below close with an *"Emit-time trigger: `pr-evidence-gate.py` class …"* note. Every such class is implemented in [`hooks/pr-evidence-gate.py`](../hooks/pr-evidence-gate.py), which runs as a `PreToolUse:Bash` hook and blocks the write: `verdict`, `observation`, `deferral`, `ci-restatement`, `inflated-verdict`, `bare-identifier`, `truncated-identifier`, `mutable-ref`, `dump-resolver`, `link-only-exhibit`, `data-only-exhibit`, `step-waiver`. It polices both the `gh pr|issue edit|create|comment` porcelain and `gh api` body writes, since a PATCH to a comment is the same publish with a different spelling. +> +> **What stays procedural.** The hook sees vocabulary, not semantics. It cannot tell whether an embedded screenshot actually shows the resolving UI's chrome (item 17), whether a deferral's stated blocker matches the step's own precondition (item 14), or whether inline data is a faithful quotation rather than a transcription (item 16). Those remain reader-applied. Setup: [evidence-gate-setup](evidence-gate-setup.md). + +## The gate (per lane, before publish) + +1. **Non-empty & expected media** — the bundle has artifacts of the expected kind. Zero artifacts = not a pass (the vacuous-pass guard). +2. **Shows the claimed surface** — the screenshot/recording is the Claim Card surface in the asserted state — not a loading spinner, an error toast, the wrong screen, or a pre-action frame. Eyeball it. +3. **Exercises the changed code** — the test/flow actually hits the diff. For a test: it **fails on `main`** (catalog B3). For a flow: the changed component/route is on the path. A green test that never imports the changed module proves nothing. +4. **Signal exceeds noise — and a null states its power** — a perf delta must be beyond run-to-run variance (paired A/B, multiple iterations); a 3% move on a noisy metric is not evidence. The same bar applies in reverse: when the spread is wider than the effect being looked for, the finding is **"not resolvable at this sample size"**, never "no change" — an underpowered run and a true null print the same word, and reporting the word alone lets the reader infer the stronger claim. State the smallest effect the design could have detected. + - **Removing a bias is not establishing validity.** Correcting a flaw you found (discarding a warm-up, alternating the starting arm, pinning CPU governor) removes *that* bias and licenses no more than that. It is not a trust gate, because a trust gate names how the evidence could **still** be vacuous — residual risk, not completed work. List what remains uncontrolled (thermal drift, background load, ordering within a round); an unenumerated confound reads as a nonexistent one. + - **When correcting an overclaim, cut the certainty, not the evidence.** A falsifier that actually fired is the strongest thing on the page — downgrade the conclusion around it, don't delete it with the overclaim. +5. **Could have failed** — the assertion has a reachable failure mode. Always-true assertions (`expect(true)`, a screenshot with no assertion, a Sentry query with no time bound) can't falsify anything. +6. **Right baseline** — "before" is the actual base ref / prior version / pre-window, not a stale or mismatched comparison. +7. **Artifacts are independent & honestly labeled** — checksum every capture set (`md5 *`). Byte-identical files across supposedly independent runs/cases cannot stand as separate observations: either explain the identity in the artifact bundle (deterministic fixture rendering) with per-run provenance that *does* differ (the harness state dump, timestamps, a manifest), or re-capture at distinct moments. Labels must describe the observation, not the interpretation — a file named for the state it *should* show under the claim (`steady-state`, `no-toast`) misleads when the capture shows the refutation. +8. **The finding ships with its artifacts** — a findings comment (including a refutation shared privately) carries functional links to the re-hosted observation artifacts at *draft* time, not descriptions of artifacts that exist only on the capturing machine. "Would need re-hosting" is not a reason to omit: re-hosting is the procedure ([evidence-publishing](evidence-publishing.md) Step 1). Code permalinks + a runnable repro are corroboration, not a substitute for the observation itself. +9. **Signal is surfaced — least-effort validation** — evidence is judged at the reader's eyes, not the author's disk: signal the reader must excavate from a mountain of attached data is, for evidence purposes, no evidence. Every published exhibit leads with a one-line pointer — *what to open, where to look, what it should show*. Deltas are presented **as** deltas (annotated side-by-side, diff, before→after crop of the differing region), never two full captures for the reader to compare by eye; if the claim is "no visual change," publish one image plus the hash-equality line, never N identical-looking copies as separate exhibits. Bulk artifacts (MB-scale JSON, full logs) are excerpted inline to the discriminating lines, with the full file linked as appendix. Emit-time test, per exhibit: can a reader who did not run the session confirm the claim in ~30 seconds from what is directly visible? If not, restructure the presentation — attaching more data cannot fix it. Coverage is the converse constraint (2026-07-21): this item governs *form*, never column-set minimalism — a valid, relevant dimension is never omitted because it duplicates another's signal (redundant corroboration costs a skippable glance; an omitted column is unfalsifiable and reads as cherry-picking). Exclusion requires invalidity (metric void on this surface, e.g. TTFB on `chrome-extension://` pages) or irrelevance (different claim/different data → sibling exhibit, not a column), each stated in a one-line disposition. +10. **Parallel exhibits are format-uniform** — sibling exhibits (table rows, per-scenario blocks, the legs of an A/B pair) carry the same evidence format and quality. If one row links its artifact inline, every row does; if one scenario gets an annotated timeline, action-log provenance, and co-located full-res/raw links, every scenario does. The bar is the **best sibling**: when the presentation standard improves mid-session, re-normalize the whole document up to it before publish — never apply the improvement only to the exhibit being produced (append-only drafting). Any asymmetry carries an explicit stated reason co-located with the weaker exhibit ("close-event variant unit-uncoverable", "manual-only trigger"); an unexplained format gap reads as an evidence gap — the reader cannot tell an unlinked artifact from a missing one, and inconsistency spends credibility on *every* exhibit, including the strong ones. Emit-time test: enumerate the sibling sets, diff each against the best-formatted member, and for every deviation either normalize it or state the reason. +11. **Lanes derive from the Manual testing steps — a CI-green row is not a lane** — the Validation Run's rows are generated top-down from the claim and the PR's own **Manual testing steps**, never bottom-up from whatever links already exist. For each step the claim depends on, the lane's payload is the **captured output of executing that step** (step "in Discover, group by `trace`" → a Discover permalink / **linked** trace-id table showing N rounds → N distinct `trace_id`s, per item 12), or an honest ⏳ naming the missing capture with a tracker. A row restating CI ("tests green at head `` in [CI run]") duplicates the Checks tab and is deleted — and a validation surface carries **zero** CI references, full stop: no `actions/runs` links, no "green at head" clauses, no "as context (only)" retention. The earlier carve-out here ("a CI link is admissible as context on a beyond-CI row") was itself the next costume: within a day all four sibling bodies (extension#43928–#43931) shipped restatements phrased as the exception — rows *leading* with "green at head … in [Unit tests CI]", the same link repeated 3× per body, the remediated row keeping it re-labeled "as context only" — while the gate's excuse regex matched the mere word "revert", so vocabulary, not evidence, discharged the class. The revert lane cites the revert **outcome** (which blocks failed, at which commit); its green-at-head half is the Checks tab's information and is omitted. A carve-out in an emit-time gate is an instruction to generation to phrase every violation as the exception — deliberate exceptions route through the human, never through an excuse predicate. Borrowed evidence — a sibling PR's capture, a unit falsifier standing in for the named live surface — never upgrades an uncaptured lane to ✅: "mechanism live-proven" co-located with "was not exercised" is an inflated verdict; downgrade it. Emit-time trigger: `pr-evidence-gate.py` classes `ci-restatement` (unconditional since 2026-07-21: any CI link / CI-green phrase in validation scope fires — no verdict co-location required, no beyond-CI excuse) and `inflated-verdict`, with the shipped extension#43928 rows and the carve-out-blessed "as context" shape as regression cases (2026-07-21). +12. **Identifiers resolve in one click — a bare id is a digging assignment** — trace ids, event ids, run ids, SHAs are *pointers into a system*, not evidence. Publishing a bunch of raw trace ids hands the reviewer the job of reconstructing project/environment/time window and querying Sentry themselves — it fails item 9's ~30-second test by construction (item 9 makes the signal *findable*; this item makes it *checkable*). Every identifier published as evidence is either hyperlinked to its resolving surface (the Sentry trace/event permalink, or an absolute-windowed Discover query pre-filtered to exactly those ids) or accompanied by the re-hosted captured output (query-result rows / envelope excerpt showing the discriminating fields) — ideally both. Special case that produced the rule: ids captured **locally** (mockttp forwarder, envelope intercept) never reached Sentry, so no permalink can exist — the re-hosted capture is the *only* admissible form, and pasting the id fragments plus a re-run recipe is the "spec necessary / output sufficient" violation wearing ids as decoration (extension#43931 Validation row, 2026-07-21). Rule of construction: when any item in this gate blesses an evidence class by name ("trace-id table", "envelope log"), it means the class's *resolvable instance*, never its bare tokens — a blessed class name is otherwise the next costume. Emit-time trigger: `pr-evidence-gate.py` class `bare-identifier`; converse-of-gate note: the prior gate *whitelisted* `trace_ids?` as beyond-CI payload and its own fix-message recommended "trace-id table" unqualified — second occurrence of "audit the gate for whitelists of the violating shape." +13. **Terminal exhibits are reader-native — a live link or a visual; a dump behind a link is still an opaque reference** — item 12 makes every pointer resolve in one click; this item constrains what it may resolve *to*. A positive verdict's terminal artifact is one of the two media a reviewer natively consumes: a **live link into the resolving system** (Sentry trace/event permalink, absolute-windowed Discover query pre-filtered to the claim) or a **visual capture** (screenshot/recording, annotated or cropped to the discriminating region). Raw files (`.log`/`.json`/`.har`, MB-scale dumps) are **appendix-only** — linked once for auditability, never the exhibit a claim rests on: a link whose target is a raw dump passes item 12 and fails item 9 one click later; the digging moved a hop away, it did not disappear (extension#43931 *second* remediation, 2026-07-21: the `bare-identifier` fix shipped a ✅ row whose sole resolver was a re-hosted ~70KB run log). Two corollaries: (a) **the gate items are conjunctive** — a fix for the newest item must re-pass all prior items; satisfying resolvability with an artifact that fails legibility is the generator's next costume; (b) **ascertain the terminal medium at step zero and pick the capture lane that can produce it** — a local intercept (mockttp envelope forwarder) can never yield a live Sentry permalink, so for Sentry-observable claims it is the supplementary falsifier lane and live ingest (dev build → `SENTRY_DSN_DEV`/test-metamask) is primary, precisely because it terminates in permalinks + screenshots; choosing a lane that cannot produce the terminal medium silently displaces it. Emit-time trigger: `pr-evidence-gate.py` class `dump-resolver`, with the remediated extension#43931 row as the regression case (2026-07-21). +14. **Manual testing steps are the validation contract — steps present ⇒ live evidence definitionally required; an impossibility waiver contradicting an executable step is invalid** — the PR's own **Manual testing steps** are the author's assertion that the claimed behavior *is* live-observable, and how: each numbered step is an executability proof (a step a human reviewer can run, `/pr-validate` can run) and its text is the capture spec. Item 11 derives the lanes from the steps top-down; this item closes the other side of the hatch — a lane derived from a step may not then be *waived by argument*. Emit-time procedure: build the per-step coverage map (step → executed-output artifact); for any step without one, the only admissible state is a **per-step** ⏳ + tracker whose blocker is that step's *own* unmet precondition, checked against the step text. Three waiver-inflation patterns from the producing instance (extension#43228/#42869/#44538, 2026-07-21 — all three shipped articulate impossibility rationales in the same body whose Manual testing steps asserted the opposite): (a) **borrowed impossibility** — the excuse imported from a different mechanism or sibling PR (#43228 waived its live lane citing the async remote-flag read race, which is #44538's mechanism; #43228's overrides are build-time env vars, and its steps 1–4 are directly executable in a dev build); (b) **lane-limitation universalized** — one harness's gap stated as global impossibility (#42869: "the e2e harness emits no error `event` envelope, so … not capturable pre-merge" — the step says *dev build with Sentry enabled, trigger an error*, which ingests error events into test-metamask without the e2e harness); (c) **blocked-scope inflation** — a genuinely blocked precondition of one half of the claim expanded to waive the whole lane (#44538: LaunchDarkly provisioning (tracked internally) blocks only the *prod-flag* half; the step's own text names the dev-injectable alternative — "or inject it into persisted `RemoteFeatureFlagController` state"). If a step is *truly* non-executable, the waiver is still inadmissible alone: the Manual testing steps are then wrong and are corrected in the same edit — a document may not simultaneously instruct a reviewer to observe X and declare X unobservable. The honest-⏳ lane blessed throughout this gate is for *not yet done*, never for *argued away*: an eloquent impossibility rationale is the cheapest token sequence that satisfies every prior item (no fake capture, no CI link, no bare id, no dump) — the generator's costume for the coverage axis. Emit-time trigger: `pr-evidence-gate.py` class `step-waiver` ("not demonstrable / capturable / observable", "not separately captured", "not attached", "rests entirely/solely on the falsifiers/unit/revert" — unconditional in validation scope; no tracker or artifact excuses it), with the three shipped waiver paragraphs as regression cases and the gate verified-blocking on all three live bodies (2026-07-21). Detection gap: the per-step consistency check (does a deferral's blocker match the step's own precondition?) stays procedural — the gate sees vocabulary, not step semantics. +15. **The exhibit lives in the body — link AND visual; a live link alone is the verification path, not the exhibit** — item 13 blessed the terminal media as a *disjunction* (live link OR visual), and generation took the cheaper disjunct: a Discover permalink is producible from the API token alone, a screenshot needs a browser session — so extension#44540's live-ingestion exhibit shipped as a permalink + prose counts, with nothing in the PR body a reader could look at (2026-07-21: "only sentry link and not screenshot that makes it immediately obvious how evidence validates pr"). A live link defers validation behind **click + auth + query rendering + column interpretation** — the dump-resolver displacement one hop further, with the mountain now behind a login: it fails item 9's ~30-second test at the moment of the click, and for any reader *without* Sentry org access (most PR reviewers) a link-only exhibit degrades to a bare identifier (item 12) behind an auth wall. The repaired rule is a **conjunction**: a positive verdict's headline exhibit is an **embedded visual** — screenshot/recording of the linked resolving view (Discover result rows, trace waterfall), cropped/annotated to the discriminating region, captioned with what it should show — **and** the co-located live permalink (absolute-windowed) as the independent-verification path. Neither substitutes for the other: link-only hides the exhibit; visual-only is independently unverifiable. The 2026-07-16 clause "screenshots ride along when a browser session is available; the API token alone yields links + JSON, which is the automatable minimum" was the self-authored escape hatch of this axis (family: the "as context" carve-out, the honest-⏳ waiver): the *automatable minimum* got promoted to the shipped standard because it was the cheapest compliant artifact. A capture lane that cannot screenshot its resolving view is a lane gap to fix before publish (drive a browser session to the Discover URL), never a licensed downgrade — deliberate exceptions route through the human. Emit-time trigger: `pr-evidence-gate.py` class `link-only-exhibit` (non-negated verdict + `sentry.io` link + no image/recording embed in the unit), with the shipped #44540 paragraph as the regression case and the prior suite's permalink-only ALLOW cases flipped/augmented — third occurrence of "an ALLOW case containing the violating tokens is a specification of the next costume." Detection gaps: verdict co-location is required, so a no-verdict link-only paragraph evades mechanically; the visual-without-link converse stays procedural under item 12. +16. **The audit chain is mechanical — quote, don't transcribe; pin, don't point** — an exhibit's inline data must be a **verbatim, greppable excerpt** of the artifact (full-length identifiers, raw capture lines quoted exactly), and every repo-hosted artifact link must be a **commit-pinned, line-anchored permalink** (`/blob//…#Lx-Ly`) to the discriminating lines. Producing real artifacts and then hand-transcribing digests severs the claim→artifact bridge at every link: an ellipsized id (`24b1e2da…`) cannot be grepped against any artifact even when the log is linked in the same block; a reformatted data block cannot be distinguished from confabulation without redoing the dig, so it reads as *claims in the form of data*; a branch-ref `/blob//` link is a mutable pointer whose target can be rewritten after review (not tamper-evident); a file-level link without line anchors lands the reader at the top of a 1,000-line dump. The producing instance (extension#43929 validation comment, 2026-07-21) had every number substantiated by four re-hosted run logs — real, included, resolvable — and still drew "still no immediately auditable evidence just claims in the form of data": each prior item individually near-passed while the exhibit↔artifact binding stayed **editorial** (transcription + file link) instead of **mechanical** (quotation + pinned line anchor). Emit-time procedure: for every inline datum, quote the raw capture line it comes from (fenced, verbatim, full ids) and anchor it (`#L`); pin every evidence link to the SHA (press `y` on the GitHub file view). Emit-time trigger: `pr-evidence-gate.py` classes `truncated-identifier` (a co-located resolver does NOT excuse it — the resolver resolves the full id, not the fragment the reader holds; hash-equality prose exempt) and `mutable-ref`, plus the closed **surface hole**: the shipped comment was published via `gh api` PATCH, which the porcelain-only matcher (`gh pr|issue edit|create|comment`) never saw — the gate now scans `gh api` body writes (`-F body=@file`, `-f body=…`, `--input`); fifth occurrence of the converse-of-gate rule, one level down: audit the gate for *spellings of the write it cannot see*, not just tokens it excuses. Detection gaps: the line-anchor half of pinning and the paraphrase-vs-quotation judgment stay procedural — the gate sees ellipses and branch refs, not editorial fidelity. +17. **Evidence is captured in its environment — data alone is insufficient even when correct** — item 16 makes the data trustworthy as *transcription* (verbatim, greppable, pinned); this item polices what transcription can never carry: **liveness provenance**. A quoted `EVIDENCE trace_id=…` line, a re-hosted gist, a hand-assembled id table can all be correct and still show nothing about *where they came from* — extracted data is indistinguishable from data typed by hand, so it cannot make it immediately apparent that the evidence was captured **live** from a **functioning** system. The exhibit for a system-of-record-observable claim therefore includes an **in-environment capture**: a screenshot/recording of the resolving system's own UI (the Sentry Discover/trace view with the query, project/environment selectors, absolute time window, and result rows all in-frame) — the environmental chrome is not decoration, it *is* the provenance: it shows the query really ran, in the real dashboard, over the real window, and returned these rows. Correctness was never the failing dimension (2026-07-21: "just the data is insufficient even if correct — it needs to be immediately apparent that evidence was captured live and is functional"). Relation to prior items: item 15's link+visual conjunction fired only when a `sentry.io` link was present, and item 13's `NATIVE_MEDIUM` blessed an inline fenced excerpt as a terminal medium — so a no-link, quoted-data exhibit (the fidelity-remediated shape: full ids, verbatim excerpts, pinned line anchors, zero environment captures) passed the whole regime while carrying zero liveness provenance. The joint rule after this item: a telemetry-observable positive verdict always carries the in-environment visual (plus the live permalink per item 15); quoted excerpts, gists, and data files are appendix beside it, never the exhibit. Emit-time trigger: `pr-evidence-gate.py` class `data-only-exhibit` (non-negated verdict + telemetry-observation vocabulary + no image/recording embed + no sentry link — with a sentry link, `link-only-exhibit` already fires), with the re-hosted-gist ALLOW case flipped (fifth occurrence of "the ALLOW case was the next costume's spec") and the #43929 quoted-excerpt shape as a regression case. Detection gaps: vocabulary-scoped (telemetry-observation terms, not bare code tokens like `trace.test.ts`), so a claim phrased entirely without them evades mechanically; and the gate cannot see whether an embedded image actually shows the environment's chrome — screenshot content stays procedural (item 2's "eyeball it" applies: the capture must show the *resolving UI*, not a cropped data region indistinguishable from a spreadsheet). +18. **"Successful" is an evidence predicate, not a run status — and the default Sentry exhibit is fixed in advance** — a validation run may be scored/reported "successful"/"validated" only when its published surface already carries, for every Sentry-observable lane, the default exhibit pair: an **in-environment Sentry-UI screenshot** (item 17) **plus the co-located live permalink** (item 15). Completed runs, green falsifiers, staged drafts, and honest ⏳ lanes do not confer success — a run without the pair is at most "run-complete, evidence-owed." The default recipe needs no per-PR ascertainment: for Sentry, **generally capture actual screenshots from the Sentry UI and attach the link** — that pair is step zero's pre-computed answer for any Sentry-observable claim, never the terminus of axis-by-axis escalation. Capture-first ordering: the capture executes before any rule/gate/postmortem authoring may close a validation session — writing a new rule or gate class discharges nothing (2026-07-22: ten postmortems and 17 gate items shipped while zero Sentry-UI screenshots did; every "successful" run was claims-only, because success was assigned by run-completion and meta-work substituted for capture work). Emit-time trigger: `pr-evidence-gate.py` `VERDICT` vocabulary now includes the status spellings `successful`/`validated`/`live-proven`, so a claims-only unit scoring itself successful blocks like any bare "confirmed." Detection gap: the gate fires only on re-emit — already-shipped "successful" surfaces are audited by backward re-score, enumerated from live state, never from the ledger (the discharge-granularity rule applies to success statuses verbatim). + +## Lane-specific traps + +- **Visual:** spinner/skeleton mistaken for the loaded state; the toggle (privacy/redaction) not actually flipped; a cached screenshot from a prior run; the fallback surface shown without saying so. +- **Perf / benchmark:** stale frozen baseline (catalog C5 caveat); single sample; warm-vs-cold mismatch; measuring a different interaction than the claim. +- **Test:** snapshot regenerated to match the bug (`--updateSnapshot` masking a regression); the test mocks out the changed path; it passes on `main` too (so it's not a regression test). +- **Telemetry:** query window excludes the release; the error regrouped under a different fingerprint; sample-rate makes "0 events" meaningless. +- **Migration:** only the happy path asserted; `changedKeys` not checked against actual mutations; no real prior-version fixture. +- **Coverage:** a line covered ≠ a behavior asserted (executed but never checked). + +## When evidence fails the gate + +Don't publish it. Either re-capture correctly, **downgrade the verdict to ⚠️ inconclusive** and name what's missing, or — if the evidence shows the claim is false — switch to the [refutation path](SKILL.md). Never round a weak pass up to "proven." diff --git a/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md b/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md new file mode 100644 index 00000000..31316c66 --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md @@ -0,0 +1,26 @@ +# Lane → declarative assertion mapping (ADR-0058 bridge) + +Maps each evidence-catalog lane to a declarative assertion form, so a Claim Card can be expressed as an ADR-0058 recipe (pre-conditions → proof targets → assertions → screenshot points) where possible — and so we know which lanes are **CDP-expressible** vs **out-of-band**. State/log assertions give determinism; screenshots/video give reviewer confidence. See [[../ITERATION]] items 9–11 and MetaMask/decisions#173. + +| Lane | Assertion form | Expressible as a CDP recipe action? | +|---|---|---| +| A1 / B1 visual | screenshot at a proof point + (optional) DOM/a11y assertion | **yes** — Chrome CDP | +| B2 e2e | the spec's own assertions; trace.zip as artifact | yes — it *is* a driver | +| B3 falsifying test | test exit code: fail@`main`, pass@branch | out-of-band (test runner) | +| C1 startup traces | `stateHooks.getCustomTraces()[name] < threshold` | **yes** — `Runtime.evaluate` | +| C2 web-vitals | `stateHooks.getWebVitalsMetrics().inp < 200` | **yes** | +| C3 long-task / TBT | `stateHooks.getLongTaskMetricsWithTBT().tbt < 200` | **yes** | +| C4 render (WDYR) | console-log assertion: 0 unnecessary re-renders | partial — needs console capture | +| C5 benchmark | metric delta vs paired baseline > threshold | out-of-band (benchmark runner) | +| C6 DevTools/CDP | netlog: request absent/present; profile metric | **yes** | +| D1 / D2 bundle/chunk | static: chunk-manifest membership / size delta | out-of-band (build artifact) | +| D3 LavaMoat | static: `policy.json` diff empty / justified | out-of-band (git diff) | +| D4 manifest | static: permissions diff empty | out-of-band | +| E1 / E2 Sentry/Tempo | external query link (before/after window) | out-of-band (dashboard) | +| F1 migration | `changedKeys == expected` + state shape valid | out-of-band (migration test) | +| F3 simulation | `simulationData.{gasUsed,stateDiff}` matches | **yes** — `Runtime.evaluate` on state | +| F5 flag matrix | the same assertion repeated per `remoteFeatureFlags` state | **yes** | +| F7 i18n | static: `verify-locales` exit 0 | out-of-band | +| F8 runtime containment | `Object.isFrozen(Object.prototype)`; scuttled global throws + exception resolves; `typeof SNOW` | **yes** — `Runtime.evaluate`, but only against the SHIPPED build variant (dev is unscuttled, test's exception list is wider) | + +**Takeaway.** UI-state and runtime-metric lanes (A/B-visual, C1–C3/C6, F3/F5) map cleanly to CDP recipe assertions — ADR-0058's sweet spot. Static (D, F7), test-runner (B3, C5, F1), and dashboard (E) lanes are **out-of-band**: the recipe should *reference* them as proof targets without executing them. That out-of-band reference is precisely the **non-UI scaling gap** raised in review on decisions#173 — a recipe schema that admits out-of-band assertion references (not only CDP actions) closes it. This table is the proposed taxonomy to contribute back (ITERATION item 10). diff --git a/domains/pr-workflow/skills/pr-validate/references/worked-examples.md b/domains/pr-workflow/skills/pr-validate/references/worked-examples.md new file mode 100644 index 00000000..e98a5f1f --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/worked-examples.md @@ -0,0 +1,30 @@ +# Worked examples (end-to-end) + +Full runs: claim → lanes → capture → trust-gate → publish. The visual case is in SKILL.md; these cover the non-visual claim shapes. + +## Perf — "defer Rive wasm at startup" +- **Claim:** on cold start of the home view, the Rive wasm chunk isn't requested until the animation surface mounts. **Surface:** startup network + chunk graph. **Falsifier:** the chunk appears in the cold-start waterfall. **Baseline:** base requests it at startup. +- **Lanes:** A2 `perf_validation` (primary) → D2 chunk membership + C6 CDP netlog (corroborate). +- **Capture:** paired build of base vs head (`yarn webpack --test`); CDP netlog over cold start for each; source-map chunk membership of the Rive runtime. +- **Trust gate:** cold-vs-cold (not warm); the chunk truly absent (not deferred by a few ms); the netlog covers the whole startup window. +- **Publish:** before/after request list + a chunk-membership table in the PR body. No screenshot needed. + +## Migration — "add migration NNN" +- **Claim:** loading a profile from `` applies NNN; `changedKeys = {X, Y}`; all other state intact. **Falsifier:** an untouched controller mutated / malformed state. **Baseline:** a prior-version profile. +- **Lanes:** F1 migration test (primary) → F2 vault round-trip (if the vault is touched). +- **Capture:** run `NNN.test.js` (old-state-in → new-state-out); assert `changedKeys`; load a real prior-version profile and confirm boot. +- **Trust gate:** the test asserts more than the happy path; `changedKeys` matches the actual mutations; the fixture is a real prior profile, not synthetic. +- **Publish:** the `changedKeys` assertion + before/after state shape; link the test run. + +## Flag-gated — "Perps banner behind a remote feature flag" +- **Claim (×2):** flag on → banner shows; flag off → banner absent. **Surface:** home/Perps. **Falsifier:** banner state ≠ flag state. **Baseline:** each flag state is its own baseline. +- **Lanes:** F5 flag matrix → A1/B1 visual per state. +- **Capture:** mock the client-config response for each flag state; screenshot each. +- **Trust gate:** the flag is actually toggled (read `remoteFeatureFlags`); two distinct states are shown, not the same frame twice. +- **Publish:** a two-up before/after (flag off / flag on) in the PR body. + +## Refactor / no-op — "extract a hook, no behavior change" +- **Claim (negation):** behavior of `` is unchanged. **Falsifier:** any output/behavior diff. **Baseline:** base behavior. +- **Lanes:** B3 regression suite stays green + B4 snapshot diff empty + D1 bundle within noise. +- **Trust gate:** snapshots were *not* regenerated to hide a diff; the tests actually cover the surface; bundle delta is within noise, not "small but real". +- **Publish:** "no behavior change — regression suite green, snapshots unchanged, bundle ±0"; link CI. A passing screenshot is not evidence here. diff --git a/domains/pr-workflow/skills/pr-validate/skill.md b/domains/pr-workflow/skills/pr-validate/skill.md new file mode 100644 index 00000000..bb8e20f9 --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/skill.md @@ -0,0 +1,305 @@ +--- +name: pr-validate +description: Validate a MetaMask PR with objective evidence — primarily the Autonomous Engineering Platform (AEP) harness (visual_validation for visible UI behavior, perf_validation for non-visible perf behavior), backed by complementary evidence (Sentry query links, screenshots, screen recordings→GIF, DevTools/CDP output, bundle-size, web-vitals, test/CI results). Drives the AEP local stack end to end: preflight (postgres + temporal + worker + control-plane) → submit POST /v1/tasks → poll GET /v1/runs/:id → fetch artifacts → assemble an evidence bundle → publish to the PR body (re-hosting images to a public repo, scrubbing local paths). Match evidence to the PR's specific falsifiable claim, not a fixed checklist. Triggers on /pr-validate, /pr-validate visual, /pr-validate perf, /pr-validate preflight, /pr-validate status, /pr-validate evidence, /pr-validate plan, or when the user mentions validating/proving a PR, AEP / visual validation / perf validation, capturing evidence for a PR, before/after screenshots, a screen recording or GIF for a PR, attaching Sentry links or DevTools output as proof, or publishing an evidence bundle to a PR body. +maturity: experimental +--- + +# /pr-validate + +Prove a PR does what it claims with **objective, reviewer-grade evidence**. The primary engine is the **Autonomous Engineering Platform (AEP)** harness run locally — `visual_validation` for visible UI behavior, `perf_validation` for non-visible perf behavior — augmented by whatever complementary evidence the claim demands (Sentry query links, screenshots, screen recordings, DevTools/CDP output, bundle/web-vitals/test results). + +This skill **executes**: it brings up the local stack, runs the harness, captures artifacts, assembles the bundle, and — **only with confirmation** — publishes to the public PR body. The platform decides what passes; the model does not declare victory. See the AEP creed: *"Tests prove the code compiles. Screenshots prove the user actually sees the fix."* + +> **Hard rule — demonstrate, don't claim.** Every verdict must *demonstrate that the **ticket objective** is achieved* with an inspectable artifact (link / screenshot / screen recording / run / Sentry query / CDP capture) — never explain or claim in prose that it is achieved. Anchor to the linked issue's objective, not just the PR body's self-description. An unbacked "objective achieved" narrative is a vacuous pass → report **⚠️ inconclusive** and name what's missing; never upgrade prose to **✅ proven**. + +## Principles + +Twenty rules the rest of this skill implements. When a situation isn't covered below, decide by these. + +**What you may claim** +- **Falsifiability** — name the observation that would disprove the claim, then go looking for it. A review that cannot fail is not a review. +- **Falsifier coverage is not exhaustive — human intervention point.** There is no fixed checklist, so there is no completeness guarantee: the falsifiers found are bounded by claim-extraction quality and by what the reviewer thought to test. "No falsifier fired" is not "no falsifier exists." A human judges whether the falsifier chosen matches the claim's actual risk, and whether a mixed or high-stakes claim needed more than one — this skill closes the falsifiers it finds, it does not attest that it found all of them. +- **Diff-anchored** — the claim is what the code *can* do, not what the PR body promises. Drift between them is a finding, not a claim. +- **Surface-specific, and a surface need not be a screen** — a job graph, a build artifact, a policy file, or a telemetry shape are all legitimate surfaces with their own falsifiers. + +**What counts as evidence** +- **Demonstrate, don't claim** — a verdict shows the objective met with an inspectable artifact. Prose asserting it is a vacuous pass. +- **In situ** — present output on the tool's own surface (the run page, the Discover view, the trace waterfall, the console). Retyping output into the report launders evidence into claim: verbatim text proves nothing about provenance, and a transcription is a place to be selective without noticing you are being selective. +- **Reproducibility of assertions** — the bar is not that the reader *can* re-run it (a working link is the floor) but that they *needn't*: the exhibit is complete enough — the numbers, the window, the method, the control — that reading it makes the result near-certain. The re-issuable link/query is a backstop for the skeptic, offered second, never the headline. + +**Why believe it** +- **No vacuous passes** — green is not proof. Assert non-empty artifacts; ask whether the assertion *could* have failed and whether the test exercises the changed code. +- **Check the instrument, not just the result** — measurement design can manufacture a finding. Verify the treatment is actually delivered in each arm before interpreting any delta. +- **Removing a bias is not establishing validity** — correcting a flaw you found licenses only that correction. A trust gate names how the evidence could *still* be vacuous; if the sentence describes work you did rather than risk that remains, it is not a gate. +- **A null states its power** — when the spread exceeds the effect under test, report *not resolvable at this n* and name the smallest detectable effect. Never let it read as "no effect". +- **Premises are claims** — probe the *because* ("unavailable", "access-limited", "can't be done here") as hard as the verdict. A false premise silently justifies the wrong method, and "unavailable" is the highest-suspicion premise because it licenses weaker evidence. +- **Recompute stated counts** against the source before publishing. A number true of an earlier draft's scope is the commonest stale fact. + +**When to stop** +- **Stop at the falsifier** — match the bar to the claim's risk; evidence past the closed falsifier is noise. +- **Defer to CI** where CI already covers it, unless the coverage is itself the point. +- **State what was not covered** — steps that could not be automated are recorded as open, with the reason. A report listing only successes reads the same as one where nothing was checked. + +**How it is handled** +- **Refutation is a successful validation** — report it, localize it, hand back the repro. Don't fix, and don't publish a failure to someone else's PR unprompted. +- **Publish surface follows ownership** — the PR body when you authored it; a comment when validating someone else's. +- **Scrub before publishing, confirm before any public write** — local paths and usernames leak through failure summaries; one PR's approval does not carry to the next. +- **Isolate concurrent runs** — colliding ports, artifact dirs, or upload paths cross-contaminate evidence *silently*. That is an integrity failure, not flakiness. + +## The core move: match evidence to the claim + +A PR makes a **falsifiable claim** ("privacy mode now hides the Perps balance"; "hovering the asset row preloads the chart with no double-fetch"; "this cuts startup http.client time"). Validation = pick the evidence that would **falsify that claim if it were false**, then run it. Do not run a fixed checklist. + +**Step 1 — extract the claim.** Read the PR (`gh pr view`, `gh pr diff`) *and the linked issue*, then write a **Claim Card** — the linchpin; every lane is only as good as the claim. Full rubric, anti-patterns, and special cases (refactor/no-op, bug-fix, perf, migration, flag-gated): **[references/claim-extraction.md](references/claim-extraction.md).** + +``` +Claim: Given , when , then . +Surface: (reachable? seed / flag / fallback: …) +Type: → lanes <…> +Falsifier: +Baseline: +``` + +A claim must be falsifiable, surface-specific, **anchored to the diff** (if the body promises X but the diff can't deliver it, flag the drift — that's a finding, not a claim), bounded, and quantified where it's a perf claim. Decompose a mixed PR into one card per claim. + +**Step 2 — match each claim to lanes** from the [evidence catalog](references/evidence-catalog.md): + +| Claim shape | Primary lane | Complementary | +|---|---|---| +| Visible UI change (layout, copy, show/hide, theme) | **A1 `visual_validation`** / B1 mm-CLI — before/after screenshots | recording→GIF for motion; B5 a11y | +| A bug fix (any kind) | **⭐ B3 falsifying test** — fails on `main`, passes on the branch | A1/B1 if visible; E1 if it errored | +| Non-visible perf (preload, no-double-fetch, lazy-load, chunk) | **A2 `perf_validation`** — falsifiable assertions | C6 CDP netlog, D2 chunk membership | +| Render / over-render | **C4 WDYR + `devtools:react`** | C1 startup traces | +| Interaction responsiveness / startup timing | **C2 INP · C3 TBT · C5 benchmark (paired A/B)** | C1 phase traces, C6 profile | +| Telemetry / error-rate / latency in prod | **E1 Sentry links** (before/after) | E2 Tempo; span-volume → `/sentry-quota` | +| Bundle / build output | **D1 size · D2 chunk membership** | — | +| A dependency change is safe | **D3 LavaMoat policy + D4 manifest diff** | D1 size | +| Runtime containment still holds | **F8 SES lockdown / scuttling, on the shipped variant** | D3 policy | +| Persisted-state change | **⭐ F1 migration** (`changedKeys`, old→new state) | F2 vault round-trip | +| Tx / dapp / flag / snap / i18n behavior | **F3 sim · F4 provider · F5 flag matrix · F6 snaps · F7 i18n** | B2 e2e trace | +| Behavior with no UI | **B3 test + G4 repro** | G1 CI checks | + +Lane IDs (A1, B3, …) index [references/evidence-catalog.md](references/evidence-catalog.md) — the full menu with verified capture commands and the complete matching guide. When a PR mixes claims (a UI fix that also shifts a metric), run more than one lane and assemble them into one bundle. + +## When to use + +- **Prove a PR** before requesting review or merge — produce the before/after a reviewer expects. +- **Re-validate** after a force-push or a requested change. +- **Back a perf/telemetry claim** with numbers and links, not prose. +- **Assemble + publish** an evidence bundle from a run you already have (`evidence` subcommand). + +Not for code-correctness review (use `/review`, `/code-review`) or span-quota review (use `/sentry-quota`). This skill proves *behavior*, not code quality. + +## Subcommands + +| Invocation | Behavior | +|---|---| +| `/pr-validate ` | **Flagship.** Read the PR → state the claim → pick lanes → [preflight](#preflight) → run AEP lane(s) + gather complementary evidence → assemble bundle → **propose** the PR-body section and confirm before publishing. | +| `/pr-validate plan ` | Dry run: read the PR, state the claim, recommend lanes + targeting hints. No stack, no run. Cheap first step when unsure. | +| `/pr-validate visual ` | AEP `visual_validation` only. | +| `/pr-validate perf ` | AEP `perf_validation` only — check the graph is present first, see [caveat](#perf_validation-caveat). | +| `/pr-validate preflight` | Health-check the local stack; bring up what's down. No run. | +| `/pr-validate status ` | Poll `GET /v1/runs/:id`; print stage timeline + `evidenceBundle.artifactRefs`. | +| `/pr-validate evidence [--run ]` | Assemble + publish a bundle from an existing run and/or complementary sources (Sentry/screens/devtools). No new AEP run. | +| `/pr-validate lane ` | Run a single [catalog](references/evidence-catalog.md) lane by id (e.g. `lane F1`, `lane C3`, `lane D3`) — for the non-AEP lanes where you know the claim type. | +| `/pr-validate compare ` | Paired A/B for a perf or refactor claim: build base + head, capture the lane on both, diff. Avoids the stale-baseline trap (catalog C5). **Per-arm treatment check first:** verify the mechanism under test is actually active in each arm (chunk split present, span emitted, flag evaluated) before interpreting deltas — a null arm without delivered treatment is a no-op, not a control (2026-07-22, #42795 bisect lesson). | + +`` is a number or URL on `MetaMask/metamask-extension` unless another repo is given. Every variant runs Step 1 (extract the Claim Card) first — the claim decides the lane, even when you named one. + +## Preflight + +The AEP harness runs as a local stack: postgres, a temporal server, a worker, and a control plane. Bring-up steps, required Node version, registry auth, and environment are documented in the [AEP repository](https://github.com/MetaMask/metamask-autonomous-engineering-platform) itself — follow its README rather than a copy here, which drifts. Health-check first and bring up only what is down. + +Fast checks: + +```bash +curl -fsS localhost:3000/health >/dev/null && echo "control-plane up" || echo "control-plane DOWN" +curl -fsS localhost:8233 >/dev/null && echo "temporal UI up" || echo "temporal DOWN" +docker ps --format '{{.Names}}' | grep -E 'aep-postgres|aep-temporal' +``` + +If the control plane answers on `localhost:3000/health`, the stack is ready and you can skip to *Run mechanics*. + +## Teardown + +The stack is the heaviest thing this skill starts — postgres + temporal + a Node worker + control-plane — and the worker holds a live Claude session while the autonomous run itself spends tokens. It is **on-demand, not resident**: bring it up for the validation window, **tear it down when the run(s) finish**. Left up, it's the single largest reclaimable footprint on a shared host and quietly keeps a Claude seat warm. + +- **If your host wraps the stack in a service manager**, use its own down command — it stops the services and removes the `--rm` postgres/temporal containers, so state resets on the next bring-up (fine, each run is fresh anyway). +- **Otherwise:** stop the `yarn dev:*` processes and remove the postgres/temporal containers. +- **Tear down on every exit path** — pass, refutation, *or* abort. A failed or abandoned run leaves the stack up exactly as much as a passing one; the usual leak is walking away after a refutation without stopping it. + +## Run mechanics (submit → poll → fetch) + +The control-plane is a thin REST shell. Submit a PR-validation task, poll the run, pull artifacts from the evidence bundle. + +```bash +CP=localhost:3000 +PR="https://github.com/MetaMask/metamask-extension/pull/" + +# Submit (publishEvidence:false ALWAYS for local runs — the platform otherwise +# writes to the public PR body even on failure, leaking local paths/usernames) +RUN_ID=$(curl -fsS -X POST "$CP/v1/tasks" -H 'content-type: application/json' -d '{ + "repo": "MetaMask/metamask-extension", + "title": "Visual validation — PR #", + "taskClass": "visual_validation", + "externalRef": "'"$PR"'", + "payload": { "prUrl": "'"$PR"'", "description": "", "publishEvidence": false } +}' | node -e 'process.stdin.on("data",d=>console.log(JSON.parse(d).runId||JSON.parse(d).id))') + +# Poll +curl -fsS "$CP/v1/runs/$RUN_ID" | node -e 'const r=JSON.parse(require("fs").readFileSync(0));console.log(r.status); (r.evidenceBundle?.artifactRefs||[]).forEach(a=>console.log(a.name,a.mediaType))' + +# Fetch an artifact +curl -fsS "$CP/v1/runs/$RUN_ID/artifacts/" -o /tmp/ +``` + +- `taskClass`: `visual_validation` or `perf_validation`. The worker auto-enriches the payload from `prUrl` (pulls headSha, base, diff, files, linked issues via the GitHub app) — you only supply `prUrl` + a `description` targeting hint. +- The **targeting hint** (`payload.description`) is how you steer the agent to the surface under test. Be specific: which screen, which control, what to toggle. For hard-to-reach surfaces, name the reachable fallback (e.g. the Shield entry modal stands in for the Perps tutorial modal, which is gated in the default fixture). +- Artifact regex allows **png/jpg/log/txt only** — no video. Screen recordings need the side-channel recipe (catalog + publishing reference). + +### Concurrent runs (multiple agents / parallel lanes) + +Five shared resources need per-run isolation on one machine — collisions cross-contaminate evidence *silently* (wrong session's logs attributed to a run), which is an integrity failure, not flakiness: **(1)** CDP debug ports — derive per run, never hardcode; **(2)** e2e harness service ports (anvil/proxy/fixture/mocha) — one e2e run at a time per worktree, one worktree per agent, and never rebuild `dist/` in a worktree with an active run; **(3)** artifact dirs — per-run namespaces; `test-artifacts/` is per-worktree shared state, harvest failure artifacts before the next run overwrites the same test-title dir; **(4)** evidence-repo uploads — run-scoped paths (`pr-//`), retry-with-fresh-sha on 409, never overwrite another run's published files; **(5)** commit-pinning — pin only after your own final upload lands, verifying your files exist at that sha. Safe to share: registry auth, a read-only `dist/`, the AEP stack itself. + +### Trust the evidence (anti-reward-hacking) + +A green result is not proof. The vacuous-pass trap is the floor: if `promptCrafter` errors, the chain "passes" via skip with **zero artifacts** — a pass is only real if `evidenceBundle.artifactRefs` is non-empty with the expected media. Beyond that, every lane must clear a trustworthiness gate before you believe or publish it: **does the artifact show the *claimed* surface** (not a spinner/wrong screen), **does the test exercise the *changed* code** (fails on `main`), **does the signal exceed noise**, **could the assertion have failed**? The Claim Card's Falsifier is the anchor. Full gate + per-lane traps: **[references/evidence-trustworthiness.md](references/evidence-trustworthiness.md).** + +### perf_validation caveat + +The `perf-validation/` graph writes falsifiable network/static/smoke assertions and gives the tester deterministic `.aep/` helpers (CDP netlog, phase segmentation, source-map chunk membership). Two constraints worth knowing before a perf run: + +- It requires a `yarn webpack --test` build first — the browserify `build:test` has no code splitting, so `import()` never hits the network there. +- Temporal caps activity results at ~2MB, so artifact refs must be content-free; only `evidenceBundle` carries base64. + +**Check the graph is present in your AEP checkout before relying on it.** It is newer than the visual-validation graph and may not be in every version — if it isn't registered, perf runs silently won't dispatch, and the fallback is manual DevTools/CDP capture (see the catalog). + +## Complementary evidence + +AEP is primary but rarely sufficient alone. Pull whatever the claim needs — **and proactively suggest evidence the PR author likely didn't think of**. The catalog is grouped into 7 families; full menu with verified capture commands and "what it proves": **[references/evidence-catalog.md](references/evidence-catalog.md).** Families: + +- **A. AEP** — `visual_validation` / `perf_validation` / bundle byproducts (primary autonomous engine). +- **B. Behavior & flow** — mm-CLI visual, E2E trace+video, **⭐ falsifying regression test** (fails on main, passes on branch — the strongest bug proof), Storybook/component, a11y, flaky-stability rerun. +- **C. Performance & render** — startup/custom traces, web-vitals (**INP/FCP/LCP/CLS** via `stateHooks`), long-task **TBT** (separate observer), React render/selector (WDYR), benchmark A/B (paired), DevTools/CDP profiling, memory-over-flow, **same-window app+DevTools capture** (C8 — UI + console evidence in one frame, OS-level region recording). +- **D. Build output** — bundle-size, chunk membership, **LavaMoat policy diff**, manifest permissions diff, build-variant matrix. + (Runtime containment — SES lockdown, scuttling, Snow — is **F8**, not D: D is what the build *permits*, F8 is what the running artifact *enforces*.) +- **E. Production telemetry** — Sentry links (span-volume → `/sentry-quota`), Tempo traces, error-event shape. +- **F. Extension integrity** — **⭐ state migration**, vault/keyring, tx simulation, provider/dapp, feature-flag matrix, snaps, i18n. +- **G. CI/review/process** — check links, coverage delta, reviewer bot, manual repro. + +Screen recordings (motion a still can't prove): `mm` + a Playwright `recordVideo` preload → `ffmpeg` two-pass palette GIF (webm/mp4 don't render inline). See [references/evidence-publishing.md](references/evidence-publishing.md). + +## Sufficiency — how much is enough + +Match the bar to the claim; stop when the claim's falsifier is closed. Don't over-instrument a copy fix; don't under-prove a high-stakes claim. + +- **One lead lane that closes the falsifier** is enough for low-risk, single-claim PRs (a copy fix → one screenshot; a bug fix → the falsifying test). +- **Weigh AEP's cost before reaching for it.** A `visual_validation`/`perf_validation` run spins the full stack *and* burns autonomous-agent tokens — by far the most expensive lane. Use it when the claim genuinely needs autonomous capture of a reachable surface; when a lighter lane closes the same falsifier (a single `mm` screenshot, a falsifying test, a CDP capture, an artifact CI already produced), prefer it and skip the stack. Whenever you do start it, tear it down after (see [Teardown](#teardown)). +- **Lead + one corroborator** for perf/telemetry (a number *and* its source) and for anything user-facing that also moves a metric. **For a perf-targeting PR the lead lane is the measured impact itself** — a paired A/B benchmark at the current head (C5) or equivalent — never mechanism evidence alone (chunk membership, netlog exclusion prove the improvement is *possible*, not that it *happened*). A perf PR also always carries correctness + non-regression lanes: changed-surface tests green at head, affected flows exercised, neutral profile within noise. (2026-07-22, #42795 lesson.) +- **Lead + integrity lane** for high-stakes surfaces regardless of size: persisted-state (migration + vault), money (tx simulation), permissions (LavaMoat + manifest), runtime containment (SES lockdown / scuttling), security/keyring. Size-S doesn't lower the bar here. +- **Per-claim** for mixed PRs — each Claim Card needs its own closed falsifier; a strong UI proof doesn't cover the metric it also shifts. +- **Rely on CI for routine coverage — don't re-collect what CI already establishes.** Lint, build, typecheck, the full test suite, changelog validation: CI is the authoritative source; **cite the check result** (e.g. "423 pass / 0 fail at head") instead of re-running it locally. Spend independent evidence only on (a) the claim's load-bearing falsifier, (b) specifically important/noteworthy areas (security, money, permissions, the exact changed surface), or (c) where the trust-gate warns a green result could be vacuous/misattributed. This is the economy counterpart to *"don't trust green blindly"*: that gate polices the **claim-critical** lane; this rule spares the **routine** coverage — re-collecting what CI covers is bundle noise. (#9628: cited CI's pass matrix for build/test, ran independent evidence only for the load-bearing homogeneity + resolution lanes.) + +Stop when each claim has one trustworthy artifact that would have shown its falsifier. More evidence past that is noise. + +## Publishing the evidence bundle + +**Public, outward-facing action — always confirm the rendered section with the user before writing the PR body.** Match AEP's own format so the section is idempotent and reviewer-familiar. Full recipe (markers, image re-hosting, recordings, the `### After` injection, privacy scrub): **[references/evidence-publishing.md](references/evidence-publishing.md).** Essentials: + +- **Canonical header — every validation output leads with the exact literal `## 🧪 Validation Run`.** Same string in a PR comment and in the PR-body section, never reworded or demoted — the constancy is what makes it scannable/Ctrl-F-able, like Copilot's fixed `## Pull request overview`. Line 2 is the meta line: `**Verdict:** ✅ proven — **Claim:** ` then `head \`\` · · lanes: `. Enforced mechanically by `hooks/pr-evidence-gate.py` (a validation/verification/evidence heading or AEP marker without the literal blocks the `gh` write). +- **Post complete, once — and know which regime the surface is in.** Comments are **push** (audience notified once at post time; edits are silent): hold until every planned lane is present or consciously dropped, and put substantive additions or changed verdicts in a **new comment referencing the original**, never a silent edit. The PR **body** is **pull** (consulted at review time): the idempotent marker upsert on re-validation at a new head is correct there. Typo-level comment edits are fine. +- **Falsifier-forward.** After the meta line, foreground **what would have falsified the claim and how each falsifier is closed** — the falsifier is the load-bearing content, not a footnote. Structure the body as "what would make this false → the evidence that rules it out," not a lane inventory with a `falsifiers closed` line buried at the bottom. The reviewer should see the disproof attempt first. +- **Don't restate CI results.** Lint/build/typecheck/test/changelog outcomes are already on the PR's Checks tab — the reviewer sees them. Cite a CI result in the comment only to **highlight something specific** they'd otherwise miss; otherwise reference "green in Checks" or omit it. Restating "423 pass / 0 fail" is bundle noise (the display-side counterpart to the catalog's *rely on CI* collection rule). +- **Re-host images first.** Control-plane artifact URLs are `localhost` and won't render on GitHub. Re-host each artifact somewhere **your readers can reach unauthenticated**, then link the hosted URL — see [evidence-publishing.md](references/evidence-publishing.md) for the host choice and the mandatory unauthenticated `curl` check. A personal repo or a private bucket fails this for every reader but you. +- **Use idempotency markers** so a re-run replaces in place: wrap the whole section in `` … ``; inside it, AEP's own `` for the status block and `` for images, injected into the PR template's `### **After**` section (replacing the `` placeholder) when present. +- **Verdict-first, lanes nested:** under the canonical header, hand-assembled AEP blocks demote to `### AEP Visual Validation` (leave AEP's own service-published `##` blocks untouched) with `**✅ Passed**` / `**❌ Failed**` / `ℹ️`, the long narrative in `
Validation details`, a meta line `Run \`\` · [LangSmith trace](…)`. +- **Scrub** local paths and your username from any narrative before publishing — failure summaries leak them. + +## Validation output format + +When reporting back (before publishing), lead with the verdict and the claim it tests: + +``` +PR # — +Claim: <the falsifiable behavior under test> +Verdict: ✅ proven / ❌ refuted / ⚠️ inconclusive (vacuous pass — 0 artifacts) +Evidence: + - visual_validation run <id> — N screenshots (before/after <surface>) + - perf_validation run <id> — M/M assertions proven + - Sentry: <before/after link> +Artifacts: <local paths or re-hosted URLs> +Next: publish to PR body? (y/N) +``` + +If a lane comes back inconclusive, say so and name what's missing — never upgrade a vacuous pass to "proven". + +### When validation refutes the claim (❌) + +A refutation is a *successful* validation — the skill did its job. Report it constructively, do **not** publish a public "Failed" section to the author's PR unprompted: + +- **Lead with the falsifier you hit:** "Claim refuted — under privacy mode the Perps balance is still visible (screenshot)." Show the evidence that disproves it. +- **Localize:** which lane, which surface, the exact observation vs the expected. Tie it to the diff if you can see why. +- **Separate refuted from inconclusive:** refuted = evidence shows the claim is false; inconclusive = evidence couldn't be captured / was untrustworthy (trust-gate fail). Don't conflate. +- **Hand back, don't fix:** this skill proves behavior; fixing is the author's loop (or a `bug_fix`/`pr_feedback` run). Offer the repro, not a patch. +- Surface privately first; only post to the PR if the author asks or it's your own PR. + +## Safety & privacy + +- **`publishEvidence: false` on every local submit.** Publish manually, only after a real pass, only with confirmation. +- **Re-host before linking** — never put a `localhost` URL or a local file path in a public PR body. +- **Scrub** usernames/paths from narratives. Failure summaries are the usual leak. +- **Don't trust green blindly** — assert non-empty `artifactRefs` (vacuous-pass trap). +- **Confirm before any PR-body write.** One PR's approval doesn't carry to the next. + +## Worked example + +PR claims privacy mode now hides the Perps balance (the demo bug #42683): +1. `gh pr view` → claim = "with privacy mode on, the Perps tab balance is masked like everywhere else." +2. Lane = `visual_validation` (visible). Preflight stack. +3. Submit with `description: "Onboard, enable privacy mode in Settings, open the Perps tab, confirm the balance is masked. If the Perps tutorial modal blocks, use the Shield entry modal as the reachable surface."` + `publishEvidence:false`. +4. Poll to completion; assert `artifactRefs` has the before/after pair (not a vacuous skip). +5. Fetch the two PNGs; re-host them to your configured evidence host; assemble the `AEP_VISUAL_VALIDATION` section with the hosted URLs injected into the template's `### After`. +6. Show the rendered section; on confirm, upsert the PR body. + +End-to-end examples for **non-visual** claims (perf, migration, flag-gated, refactor/no-op): **[references/worked-examples.md](references/worked-examples.md).** + +## Positioning: AEP vs recipes vs pr-validate + +Three adjacent things; keep the boundary clear so they compose instead of collide: + +- **AEP** — governed *fleet orchestration*: sandboxes, Temporal, autonomous runs at scale. The heavy engine. +- **ADR-0058 recipes** ([decisions#173](https://github.com/MetaMask/decisions/pull/173)) — a *dev-machine inner-loop* proof artifact: a declarative per-PR recipe run against the live app over CDP, emitting `summary.json`/`trace.json`/manifest. +- **pr-validate** (this skill) — the *claim→evidence methodology + taxonomy* both draw on. The Claim Card is the bridge from a PR's claim to the right proof target; the [evidence catalog](references/evidence-catalog.md) is the lane vocabulary; [lane-assertions.md](references/lane-assertions.md) maps each lane to a recipe assertion (and flags the out-of-band, non-UI lanes — the gap raised in review on decisions#173). + +pr-validate is the one a human drives; it can dispatch an AEP run or author a recipe as its capture step. + +## Workflow integration + +Where pr-validate sits in the PR lifecycle (see the public `pr-workflow` siblings): + +- **After `create-pr`, before `pr-review-queue`:** validate the claim, attach the bundle, *then* request review — reviewers get the before/after up front. +- **On force-push / requested-change:** re-run the affected lane(s); re-validation keeps a stale evidence section honest. +- **`/triage` push items:** a `push`-state PR isn't done until its claim is proven; pr-validate produces the evidence that lets it move. +- **Not a CI gate** (same scope line as ADR-0058) — it's the author's inner loop, complementing unit/e2e, not replacing them. + +## Boundaries + +- **Executes, with a confirmation gate on publish.** It runs the harness and captures evidence autonomously; it does not write to the public PR body without showing you the section first. +- **Local-only AEP.** No hosted instance. The skill drives the local stack. +- **Proves behavior, not code.** Pair with `/review` / `/code-review` for correctness and `/sentry-quota` for span-volume risk. +- **No persisted state.** Each run is fresh. To keep a validation record, ask — nothing is written by default. + +## Related + +- [references/claim-extraction.md](references/claim-extraction.md) — Step 1: turn a PR into a falsifiable Claim Card. +- [references/evidence-catalog.md](references/evidence-catalog.md) — the full menu of evidence kinds, verified capture commands, and what each proves. +- [references/evidence-trustworthiness.md](references/evidence-trustworthiness.md) — the anti-reward-hacking gate before believing/publishing a lane. +- [references/evidence-publishing.md](references/evidence-publishing.md) — PR-body format, non-visual/multi-lane rendering, image re-hosting, recordings→GIF, privacy scrub, ADR-0058 artifact contract. +- [references/worked-examples.md](references/worked-examples.md) — end-to-end runs for perf / migration / flag-gated / refactor claims. +- [references/lane-assertions.md](references/lane-assertions.md) — lane → declarative recipe-assertion mapping (ADR-0058 bridge). +- [MetaMask/metamask-autonomous-engineering-platform](https://github.com/MetaMask/metamask-autonomous-engineering-platform) — the AEP repo: stack bring-up in its README, plus `docs/demo-runbook.md`, `packages/agent-chain/src/graphs/{visual,perf}-validation/`, and `packages/github/src/pr-body-builder.ts` (the canonical PR-body format this skill mirrors). +- `MetaMask/decisions#173` — ADR-0058 Recipe-Based Verification (the adjacent inner-loop proof system). +- `/sentry-quota` — sibling skill for span-volume PR review; `/review`, `/code-review` — code correctness. +- `/memory-leak-hunt` — the engine behind the **memory leak** evidence category (C9). pr-validate delegates retention analysis to it and packages the verdict; it also runs standalone. +- [[reference_aep_local_run]] — the source memory this skill encodes. +- [[reference_sentry_project_topology]] — Sentry project mapping for the telemetry-evidence lane. From 26a7daf83dfa30128cd847019866d40a037746c2 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 08:41:58 -0400 Subject: [PATCH 02/63] Restore the MetaMask-planning link in the step-waiver item MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Over-scrubbed. The audience is the MetaMask org, and the ticket is the evidence for the claim the item makes — that the LaunchDarkly provisioning blocker covers only the prod-flag half of that lane. Without it the example is an assertion. The scrub line is personal references, not org-internal ones. --- .../skills/pr-validate/references/evidence-trustworthiness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md b/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md index 8933a8de..482f01e5 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md @@ -23,7 +23,7 @@ A green result is not proof. An agent — or an eager run — can produce eviden 11. **Lanes derive from the Manual testing steps — a CI-green row is not a lane** — the Validation Run's rows are generated top-down from the claim and the PR's own **Manual testing steps**, never bottom-up from whatever links already exist. For each step the claim depends on, the lane's payload is the **captured output of executing that step** (step "in Discover, group by `trace`" → a Discover permalink / **linked** trace-id table showing N rounds → N distinct `trace_id`s, per item 12), or an honest ⏳ naming the missing capture with a tracker. A row restating CI ("tests green at head `<sha>` in [CI run]") duplicates the Checks tab and is deleted — and a validation surface carries **zero** CI references, full stop: no `actions/runs` links, no "green at head" clauses, no "as context (only)" retention. The earlier carve-out here ("a CI link is admissible as context on a beyond-CI row") was itself the next costume: within a day all four sibling bodies (extension#43928–#43931) shipped restatements phrased as the exception — rows *leading* with "green at head … in [Unit tests CI]", the same link repeated 3× per body, the remediated row keeping it re-labeled "as context only" — while the gate's excuse regex matched the mere word "revert", so vocabulary, not evidence, discharged the class. The revert lane cites the revert **outcome** (which blocks failed, at which commit); its green-at-head half is the Checks tab's information and is omitted. A carve-out in an emit-time gate is an instruction to generation to phrase every violation as the exception — deliberate exceptions route through the human, never through an excuse predicate. Borrowed evidence — a sibling PR's capture, a unit falsifier standing in for the named live surface — never upgrades an uncaptured lane to ✅: "mechanism live-proven" co-located with "was not exercised" is an inflated verdict; downgrade it. Emit-time trigger: `pr-evidence-gate.py` classes `ci-restatement` (unconditional since 2026-07-21: any CI link / CI-green phrase in validation scope fires — no verdict co-location required, no beyond-CI excuse) and `inflated-verdict`, with the shipped extension#43928 rows and the carve-out-blessed "as context" shape as regression cases (2026-07-21). 12. **Identifiers resolve in one click — a bare id is a digging assignment** — trace ids, event ids, run ids, SHAs are *pointers into a system*, not evidence. Publishing a bunch of raw trace ids hands the reviewer the job of reconstructing project/environment/time window and querying Sentry themselves — it fails item 9's ~30-second test by construction (item 9 makes the signal *findable*; this item makes it *checkable*). Every identifier published as evidence is either hyperlinked to its resolving surface (the Sentry trace/event permalink, or an absolute-windowed Discover query pre-filtered to exactly those ids) or accompanied by the re-hosted captured output (query-result rows / envelope excerpt showing the discriminating fields) — ideally both. Special case that produced the rule: ids captured **locally** (mockttp forwarder, envelope intercept) never reached Sentry, so no permalink can exist — the re-hosted capture is the *only* admissible form, and pasting the id fragments plus a re-run recipe is the "spec necessary / output sufficient" violation wearing ids as decoration (extension#43931 Validation row, 2026-07-21). Rule of construction: when any item in this gate blesses an evidence class by name ("trace-id table", "envelope log"), it means the class's *resolvable instance*, never its bare tokens — a blessed class name is otherwise the next costume. Emit-time trigger: `pr-evidence-gate.py` class `bare-identifier`; converse-of-gate note: the prior gate *whitelisted* `trace_ids?` as beyond-CI payload and its own fix-message recommended "trace-id table" unqualified — second occurrence of "audit the gate for whitelists of the violating shape." 13. **Terminal exhibits are reader-native — a live link or a visual; a dump behind a link is still an opaque reference** — item 12 makes every pointer resolve in one click; this item constrains what it may resolve *to*. A positive verdict's terminal artifact is one of the two media a reviewer natively consumes: a **live link into the resolving system** (Sentry trace/event permalink, absolute-windowed Discover query pre-filtered to the claim) or a **visual capture** (screenshot/recording, annotated or cropped to the discriminating region). Raw files (`.log`/`.json`/`.har`, MB-scale dumps) are **appendix-only** — linked once for auditability, never the exhibit a claim rests on: a link whose target is a raw dump passes item 12 and fails item 9 one click later; the digging moved a hop away, it did not disappear (extension#43931 *second* remediation, 2026-07-21: the `bare-identifier` fix shipped a ✅ row whose sole resolver was a re-hosted ~70KB run log). Two corollaries: (a) **the gate items are conjunctive** — a fix for the newest item must re-pass all prior items; satisfying resolvability with an artifact that fails legibility is the generator's next costume; (b) **ascertain the terminal medium at step zero and pick the capture lane that can produce it** — a local intercept (mockttp envelope forwarder) can never yield a live Sentry permalink, so for Sentry-observable claims it is the supplementary falsifier lane and live ingest (dev build → `SENTRY_DSN_DEV`/test-metamask) is primary, precisely because it terminates in permalinks + screenshots; choosing a lane that cannot produce the terminal medium silently displaces it. Emit-time trigger: `pr-evidence-gate.py` class `dump-resolver`, with the remediated extension#43931 row as the regression case (2026-07-21). -14. **Manual testing steps are the validation contract — steps present ⇒ live evidence definitionally required; an impossibility waiver contradicting an executable step is invalid** — the PR's own **Manual testing steps** are the author's assertion that the claimed behavior *is* live-observable, and how: each numbered step is an executability proof (a step a human reviewer can run, `/pr-validate` can run) and its text is the capture spec. Item 11 derives the lanes from the steps top-down; this item closes the other side of the hatch — a lane derived from a step may not then be *waived by argument*. Emit-time procedure: build the per-step coverage map (step → executed-output artifact); for any step without one, the only admissible state is a **per-step** ⏳ + tracker whose blocker is that step's *own* unmet precondition, checked against the step text. Three waiver-inflation patterns from the producing instance (extension#43228/#42869/#44538, 2026-07-21 — all three shipped articulate impossibility rationales in the same body whose Manual testing steps asserted the opposite): (a) **borrowed impossibility** — the excuse imported from a different mechanism or sibling PR (#43228 waived its live lane citing the async remote-flag read race, which is #44538's mechanism; #43228's overrides are build-time env vars, and its steps 1–4 are directly executable in a dev build); (b) **lane-limitation universalized** — one harness's gap stated as global impossibility (#42869: "the e2e harness emits no error `event` envelope, so … not capturable pre-merge" — the step says *dev build with Sentry enabled, trigger an error*, which ingests error events into test-metamask without the e2e harness); (c) **blocked-scope inflation** — a genuinely blocked precondition of one half of the claim expanded to waive the whole lane (#44538: LaunchDarkly provisioning (tracked internally) blocks only the *prod-flag* half; the step's own text names the dev-injectable alternative — "or inject it into persisted `RemoteFeatureFlagController` state"). If a step is *truly* non-executable, the waiver is still inadmissible alone: the Manual testing steps are then wrong and are corrected in the same edit — a document may not simultaneously instruct a reviewer to observe X and declare X unobservable. The honest-⏳ lane blessed throughout this gate is for *not yet done*, never for *argued away*: an eloquent impossibility rationale is the cheapest token sequence that satisfies every prior item (no fake capture, no CI link, no bare id, no dump) — the generator's costume for the coverage axis. Emit-time trigger: `pr-evidence-gate.py` class `step-waiver` ("not demonstrable / capturable / observable", "not separately captured", "not attached", "rests entirely/solely on the falsifiers/unit/revert" — unconditional in validation scope; no tracker or artifact excuses it), with the three shipped waiver paragraphs as regression cases and the gate verified-blocking on all three live bodies (2026-07-21). Detection gap: the per-step consistency check (does a deferral's blocker match the step's own precondition?) stays procedural — the gate sees vocabulary, not step semantics. +14. **Manual testing steps are the validation contract — steps present ⇒ live evidence definitionally required; an impossibility waiver contradicting an executable step is invalid** — the PR's own **Manual testing steps** are the author's assertion that the claimed behavior *is* live-observable, and how: each numbered step is an executability proof (a step a human reviewer can run, `/pr-validate` can run) and its text is the capture spec. Item 11 derives the lanes from the steps top-down; this item closes the other side of the hatch — a lane derived from a step may not then be *waived by argument*. Emit-time procedure: build the per-step coverage map (step → executed-output artifact); for any step without one, the only admissible state is a **per-step** ⏳ + tracker whose blocker is that step's *own* unmet precondition, checked against the step text. Three waiver-inflation patterns from the producing instance (extension#43228/#42869/#44538, 2026-07-21 — all three shipped articulate impossibility rationales in the same body whose Manual testing steps asserted the opposite): (a) **borrowed impossibility** — the excuse imported from a different mechanism or sibling PR (#43228 waived its live lane citing the async remote-flag read race, which is #44538's mechanism; #43228's overrides are build-time env vars, and its steps 1–4 are directly executable in a dev build); (b) **lane-limitation universalized** — one harness's gap stated as global impossibility (#42869: "the e2e harness emits no error `event` envelope, so … not capturable pre-merge" — the step says *dev build with Sentry enabled, trigger an error*, which ingests error events into test-metamask without the e2e harness); (c) **blocked-scope inflation** — a genuinely blocked precondition of one half of the claim expanded to waive the whole lane (#44538: LaunchDarkly provisioning [#7482](https://github.com/MetaMask/MetaMask-planning/issues/7482) blocks only the *prod-flag* half; the step's own text names the dev-injectable alternative — "or inject it into persisted `RemoteFeatureFlagController` state"). If a step is *truly* non-executable, the waiver is still inadmissible alone: the Manual testing steps are then wrong and are corrected in the same edit — a document may not simultaneously instruct a reviewer to observe X and declare X unobservable. The honest-⏳ lane blessed throughout this gate is for *not yet done*, never for *argued away*: an eloquent impossibility rationale is the cheapest token sequence that satisfies every prior item (no fake capture, no CI link, no bare id, no dump) — the generator's costume for the coverage axis. Emit-time trigger: `pr-evidence-gate.py` class `step-waiver` ("not demonstrable / capturable / observable", "not separately captured", "not attached", "rests entirely/solely on the falsifiers/unit/revert" — unconditional in validation scope; no tracker or artifact excuses it), with the three shipped waiver paragraphs as regression cases and the gate verified-blocking on all three live bodies (2026-07-21). Detection gap: the per-step consistency check (does a deferral's blocker match the step's own precondition?) stays procedural — the gate sees vocabulary, not step semantics. 15. **The exhibit lives in the body — link AND visual; a live link alone is the verification path, not the exhibit** — item 13 blessed the terminal media as a *disjunction* (live link OR visual), and generation took the cheaper disjunct: a Discover permalink is producible from the API token alone, a screenshot needs a browser session — so extension#44540's live-ingestion exhibit shipped as a permalink + prose counts, with nothing in the PR body a reader could look at (2026-07-21: "only sentry link and not screenshot that makes it immediately obvious how evidence validates pr"). A live link defers validation behind **click + auth + query rendering + column interpretation** — the dump-resolver displacement one hop further, with the mountain now behind a login: it fails item 9's ~30-second test at the moment of the click, and for any reader *without* Sentry org access (most PR reviewers) a link-only exhibit degrades to a bare identifier (item 12) behind an auth wall. The repaired rule is a **conjunction**: a positive verdict's headline exhibit is an **embedded visual** — screenshot/recording of the linked resolving view (Discover result rows, trace waterfall), cropped/annotated to the discriminating region, captioned with what it should show — **and** the co-located live permalink (absolute-windowed) as the independent-verification path. Neither substitutes for the other: link-only hides the exhibit; visual-only is independently unverifiable. The 2026-07-16 clause "screenshots ride along when a browser session is available; the API token alone yields links + JSON, which is the automatable minimum" was the self-authored escape hatch of this axis (family: the "as context" carve-out, the honest-⏳ waiver): the *automatable minimum* got promoted to the shipped standard because it was the cheapest compliant artifact. A capture lane that cannot screenshot its resolving view is a lane gap to fix before publish (drive a browser session to the Discover URL), never a licensed downgrade — deliberate exceptions route through the human. Emit-time trigger: `pr-evidence-gate.py` class `link-only-exhibit` (non-negated verdict + `sentry.io` link + no image/recording embed in the unit), with the shipped #44540 paragraph as the regression case and the prior suite's permalink-only ALLOW cases flipped/augmented — third occurrence of "an ALLOW case containing the violating tokens is a specification of the next costume." Detection gaps: verdict co-location is required, so a no-verdict link-only paragraph evades mechanically; the visual-without-link converse stays procedural under item 12. 16. **The audit chain is mechanical — quote, don't transcribe; pin, don't point** — an exhibit's inline data must be a **verbatim, greppable excerpt** of the artifact (full-length identifiers, raw capture lines quoted exactly), and every repo-hosted artifact link must be a **commit-pinned, line-anchored permalink** (`/blob/<sha>/…#Lx-Ly`) to the discriminating lines. Producing real artifacts and then hand-transcribing digests severs the claim→artifact bridge at every link: an ellipsized id (`24b1e2da…`) cannot be grepped against any artifact even when the log is linked in the same block; a reformatted data block cannot be distinguished from confabulation without redoing the dig, so it reads as *claims in the form of data*; a branch-ref `/blob/<branch>/` link is a mutable pointer whose target can be rewritten after review (not tamper-evident); a file-level link without line anchors lands the reader at the top of a 1,000-line dump. The producing instance (extension#43929 validation comment, 2026-07-21) had every number substantiated by four re-hosted run logs — real, included, resolvable — and still drew "still no immediately auditable evidence just claims in the form of data": each prior item individually near-passed while the exhibit↔artifact binding stayed **editorial** (transcription + file link) instead of **mechanical** (quotation + pinned line anchor). Emit-time procedure: for every inline datum, quote the raw capture line it comes from (fenced, verbatim, full ids) and anchor it (`#L<n>`); pin every evidence link to the SHA (press `y` on the GitHub file view). Emit-time trigger: `pr-evidence-gate.py` classes `truncated-identifier` (a co-located resolver does NOT excuse it — the resolver resolves the full id, not the fragment the reader holds; hash-equality prose exempt) and `mutable-ref`, plus the closed **surface hole**: the shipped comment was published via `gh api` PATCH, which the porcelain-only matcher (`gh pr|issue edit|create|comment`) never saw — the gate now scans `gh api` body writes (`-F body=@file`, `-f body=…`, `--input`); fifth occurrence of the converse-of-gate rule, one level down: audit the gate for *spellings of the write it cannot see*, not just tokens it excuses. Detection gaps: the line-anchor half of pinning and the paraphrase-vs-quotation judgment stay procedural — the gate sees ellipses and branch refs, not editorial fidelity. 17. **Evidence is captured in its environment — data alone is insufficient even when correct** — item 16 makes the data trustworthy as *transcription* (verbatim, greppable, pinned); this item polices what transcription can never carry: **liveness provenance**. A quoted `EVIDENCE trace_id=…` line, a re-hosted gist, a hand-assembled id table can all be correct and still show nothing about *where they came from* — extracted data is indistinguishable from data typed by hand, so it cannot make it immediately apparent that the evidence was captured **live** from a **functioning** system. The exhibit for a system-of-record-observable claim therefore includes an **in-environment capture**: a screenshot/recording of the resolving system's own UI (the Sentry Discover/trace view with the query, project/environment selectors, absolute time window, and result rows all in-frame) — the environmental chrome is not decoration, it *is* the provenance: it shows the query really ran, in the real dashboard, over the real window, and returned these rows. Correctness was never the failing dimension (2026-07-21: "just the data is insufficient even if correct — it needs to be immediately apparent that evidence was captured live and is functional"). Relation to prior items: item 15's link+visual conjunction fired only when a `sentry.io` link was present, and item 13's `NATIVE_MEDIUM` blessed an inline fenced excerpt as a terminal medium — so a no-link, quoted-data exhibit (the fidelity-remediated shape: full ids, verbatim excerpts, pinned line anchors, zero environment captures) passed the whole regime while carrying zero liveness provenance. The joint rule after this item: a telemetry-observable positive verdict always carries the in-environment visual (plus the live permalink per item 15); quoted excerpts, gists, and data files are appendix beside it, never the exhibit. Emit-time trigger: `pr-evidence-gate.py` class `data-only-exhibit` (non-negated verdict + telemetry-observation vocabulary + no image/recording embed + no sentry link — with a sentry link, `link-only-exhibit` already fires), with the re-hosted-gist ALLOW case flipped (fifth occurrence of "the ALLOW case was the next costume's spec") and the #43929 quoted-excerpt shape as a regression case. Detection gaps: vocabulary-scoped (telemetry-observation terms, not bare code tokens like `trace.test.ts`), so a claim phrased entirely without them evades mechanically; and the gate cannot see whether an embedded image actually shows the environment's chrome — screenshot content stays procedural (item 2's "eyeball it" applies: the capture must show the *resolving UI*, not a cropped data region indistinguishable from a spreadsheet). From a66e72ad2fec9c0fb798f3824b6efde84894173a Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 13:21:53 -0400 Subject: [PATCH 03/63] Move `falsifying-test` to the `testing` domain MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `pr-validate`'s other engines are placed by subject — `react-render-proof` in `performance`, `memory-leak-hunt` in `stability`, `supply-chain-audit` in `security`. This one was placed by its caller instead. Writing a test that fails on the base commit and passes on the branch is a testing technique, and `testing/` already holds techniques of that kind (`e2e-flakiness-patterns`, `test-layer-placement`, `performance-testing`), while `pr-workflow/` is uniformly PR-lifecycle stages. Both references to it are by name rather than path, so nothing needed updating. --- domains/{pr-workflow => testing}/skills/falsifying-test/skill.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename domains/{pr-workflow => testing}/skills/falsifying-test/skill.md (100%) diff --git a/domains/pr-workflow/skills/falsifying-test/skill.md b/domains/testing/skills/falsifying-test/skill.md similarity index 100% rename from domains/pr-workflow/skills/falsifying-test/skill.md rename to domains/testing/skills/falsifying-test/skill.md From 78a93b0062f690aefabcad9cf989b10d14951ff7 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 14:38:06 -0400 Subject: [PATCH 04/63] Add `D6` substitution A/B lane and sync `pr-validate` references MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `D6` covers claims where a PR hand-writes an artifact that restates an existing source — a type, schema, vendored constant, or checked-in policy. Both arms sit at the same commit and differ by a substitution rather than a ref, so there is no build or merge boundary to confound the result. --- .../references/evidence-catalog.md | 44 +++++++--- .../references/evidence-publishing.md | 81 ++++++++++++------- .../references/evidence-trustworthiness.md | 11 ++- .../pr-validate/references/lane-assertions.md | 2 +- .../pr-workflow/skills/pr-validate/skill.md | 68 ++++++++++------ 5 files changed, 133 insertions(+), 73 deletions(-) diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md b/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md index b34a5e20..8c8b7680 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md @@ -2,7 +2,7 @@ The menu of evidence kinds for validating a MetaMask **extension** PR, with **what each proves**, **how to capture it (verified against the live repo)**, and **when to reach for it**. AEP is the primary autonomous engine; the rest are complementary. The skill's job is to **match evidence to the claim** and to **proactively suggest kinds the author didn't think of**. -Pick the evidence that would **falsify the claim if it were false**. Prefer a lane that yields an artifact a reviewer can independently re-check (a link, an image, a number, a replayable trace) over prose. Don't run the whole menu — match, then capture. Capture commands are written against the `metamask-extension` checkout; verify script names against its `package.json` (they drift). +Pick the evidence that would **falsify the claim if it were false**. Prefer a lane that yields an artifact a reviewer can independently re-check (a link, an image, a number, a replayable trace) over prose. Don't run the whole menu — match, then capture. Capture commands cite `~/Code/metamask/metamask-extension`; verify script names against its `package.json` (they drift). Legend: **first-class lanes** are `##`-headed; closely-related variants are sub-bullets. Capture marked *(manual)* has no repo helper — it's a DevTools/CDP action. @@ -12,12 +12,12 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- ## A1. visual_validation — before/after screenshots - **Proves:** a visible UI change on the real surface. Deterministic state seed + agent navigation; PNG artifacts in `evidenceBundle.artifactRefs`. -- **Capture:** `taskClass: visual_validation`, `payload.prUrl` + `description` hint. See the *Preflight* and *Run mechanics* sections of [skill.md](../skill.md). +- **Capture:** `taskClass: visual_validation`, `payload.prUrl` + `description` hint. See [aep-local-run.md](aep-local-run.md). - **Reach for it:** anything a human would screenshot for the PR's `### After`. ## A2. perf_validation — falsifiable network/static/smoke assertions - **Proves:** non-visible behavior (hover-preload, no double-fetch, chunk membership, smoke boot). CDP netlog / phase segmentation / source-map membership. -- **Capture:** `taskClass: perf_validation` (needs a `yarn webpack --test` build). Confirm the perf-validation graph is registered in your AEP checkout; falls back to C6/D2 manually if it isn't present. +- **Capture:** `taskClass: perf_validation` (local/uncommitted graph; needs `yarn webpack --test`). Falls back to C6/D2 manually if the graph isn't present. ## A3. AEP bundle byproducts (free with any run) - Test results (`executionResult`/`checkResults`), diff stats, automated `reviewResult` findings, and the **LangSmith trace** of the run. Include the relevant subset; link the trace for auditability. @@ -44,7 +44,8 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- - **Reach for it:** every bug-fix PR. If you can't write a test that fails on main, question whether the fix addresses the reported bug. ## B7. Deterministic interleaving test (concurrency / temporal-ordering) ⭐ -- **Proves:** an ordering guarantee under interleaving — retry, cancellation, supersession, debounce, locks, queues, async state machines — where the correctness *is* the ordering under races, not a value. +- **Engine:** `race-condition-proof` — run it rather than hand-rolling the harness. +- **Proves:** an ordering guarantee under interleaving — retry, cancellation, supersession, debounce, locks, queues, async state machines — where the correctness *is* the ordering under races, not a value. Full category: `exogram-daemon/artifacts/evidence-taxonomy/category-concurrency-temporal-ordering.md`. - **Capture:** force each race deterministically — `jest.useFakeTimers()` + `advanceTimersByTimeAsync(DELAY)` to fire the delayed action at a known point; `Promise.all([opA, opB])` to overlap operations; `advanceTimersByTimeAsync(0)` to step to a precise interleaving point; then assert the ordering/cancellation outcome for **each** guarantee, including asymmetric ones (one path canceled → its recovery event `.not.toHaveBeenCalled()`; another must complete → `.toHaveBeenCalledWith(...)`). Corroborate with transition telemetry; for the integration path, a live forced-race capture (C8/CDP, the #44610 technique). - **Trust-gate:** the test must **actually interleave** — time advanced into the pending window, the superseding op injected *during* it. A sequential run exercises no race and is a vacuous green. Verify the interleaving, not just the assertion. @@ -72,7 +73,7 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- ## C2. Web vitals — INP / FCP / LCP / CLS - **Proves:** a user-centric metric moved. `ui/helpers/utils/web-vitals.ts` via `web-vitals/attribution` (attribution names the causing element). - **Capture:** `window.stateHooks.getWebVitalsMetrics()` (test/debug) → `{inp, fcp, lcp, cls, *Rating}`. Thresholds: INP good<200/poor>500, FCP<1800/3000, LCP<2500/4000, CLS<0.1/0.25. -- **Caveat:** **INP fires on all pages; FCP/LCP/CLS do not fire on popup pages** (sidepanel/E2E only). For extensions, INP is the high-value runtime metric. +- **Caveat:** **INP fires on all pages; FCP/LCP/CLS do not fire on popup pages** (sidepanel/E2E only). For extensions, INP is the high-value runtime metric (per exogram `web-vitals-runtime-metrics`). ## C3. Long-task / TBT - **Proves:** main-thread blocking during an interaction dropped. This is where **TBT** lives (the web-vitals lib lane does *not* collect TBT). @@ -80,16 +81,16 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- ## C4. React render & selector proof - **Engine: the `react-render-proof` skill.** Delegate the measurement to it; it runs the source/delivery/metric gates, derives the needle from real build output, repeats the capture, and returns a band (or "not resolvable at this n" with an MDE). pr-validate packages the result. -- **Proves:** a component/selector stopped over-rendering (cascade-amplification before/after). +- **Proves:** a component/selector stopped over-rendering (cascade-amplification before/after — exogram `react-redux-performance`). - **Capture:** WDYR via `ENABLE_WHY_DID_YOU_RENDER` (`.metamaskrc` or env) — wired in `app/scripts/development/wdyr.ts` (`trackAllPureComponents`); console logs each unnecessary re-render. `yarn devtools:react` for the Profiler flame graph. Selectors use `reselect`'s `createSelector`, which **does expose a real `.recomputations()` counter** — read it (sample on an interval if the count should visibly climb) rather than injecting a log into the selector body; an injected log is an authored claim, a library API is an observation. *(This entry previously said there was no built-in counter. There is.)* - **Bar:** the delivery check comes before the number. An arm whose manipulation cannot be observed in the built bundle produces a null indistinguishable from "small effect" — and reports as the second. ## C5. Benchmark A/B - **Proves:** a startup/journey/interaction timing moved, with a distribution not one sample. - **Capture:** `yarn test:e2e:benchmark` (`test/e2e/benchmarks/run-benchmark.ts`); presets in `shared/constants/benchmarks.ts` (`startupStandardHome`, `sendTransactions`, `swap`, `dappPageLoad`, …). -- **Caveat:** the rolling baseline (`MetaMask/extension_benchmark_stats`) can **silently freeze** behind a green check (the `store-benchmark-stats` step is `continue-on-error`; happened 2026-04-02, PR #42947). Prefer a **paired A/B** (build both refs now, compare directly) over the stored baseline. +- **Caveat:** the rolling baseline (`MetaMask/extension_benchmark_stats`) can **silently freeze** behind a green check (the `store-benchmark-stats` step is `continue-on-error`; happened 2026-04-02, PR #42947). Prefer a **paired A/B** (build both refs now, compare directly) over the stored baseline. See exogram `benchmark-baseline-staleness-paired-ab`. - **Treatment check first** — before trusting any delta, confirm the mechanism under test is actually active in each arm (split chunk present in head and absent in base; the span emitted; the flag evaluated). An arm without the treatment delivered is a no-op, not a control (2026-07-22, #42795). -- **A null needs its power stated** — "no change" and "underpowered" print the same result. When the run-to-run spread exceeds the effect under test, report **not resolvable at this n** and name the smallest detectable effect; never let it read as "no effect". Correcting a known bias (discarding a warm-up, alternating the starting arm) removes *that* bias and nothing more — it is not a trust gate, and the confounds you did not enumerate (thermal drift, background load, ordering within a round) stay live. +- **A null needs its power stated** — "no change" and "underpowered" print the same result. When the run-to-run spread exceeds the effect under test, report **not resolvable at this n** and name the smallest detectable effect; never let it read as "no effect". Correcting a known bias (discarding a warm-up, alternating the starting arm) removes *that* bias and nothing more — it is not a trust gate, and the confounds you did not enumerate (thermal drift, background load, ordering within a round) stay live. See exogram `removing-a-bias-is-not-establishing-validity` (2026-07-24). ### Capturing an authenticated view (the in-situ requirement) @@ -137,7 +138,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ ## C9. Retention-path analysis — memory leak from code ⭐ *(static; lead for leak claims)* - **Engine: the `memory-leak-hunt` skill.** For a memory-leak claim, delegate the analysis to `memory-leak-hunt` — it runs Phase-1 static pairing (and Phase-2 heap investigation if a primitive can't be paired) and returns the paired/unpaired sites + verdict. pr-validate keeps **memory leak** as the evidence category: it invokes the skill on the diff and packages the result (in-situ scan capture, plus the lifecycle test / retainer graph if Phase 2 ran) as the category's evidence. The lane spec below is the method that skill implements. -- **Proves:** "X is retained past its lifecycle boundary" / "collection Y grows unboundedly" — argued from code, no runtime needed. This is the lane that works at **review time** (does this PR *introduce* retention?) and leads fix-side validation (does the fix *break* the retention path?). C7 is the runtime corroborator, not the lead — leaks need many cycles to exceed noise. +- **Proves:** "X is retained past its lifecycle boundary" / "collection Y grows unboundedly" — argued from code, no runtime needed. This is the lane that works at **review time** (does this PR *introduce* retention?) and leads fix-side validation (does the fix *break* the retention path?). C7 is the runtime corroborator, not the lead — leaks need many cycles to exceed noise. Full category: `exogram-daemon/artifacts/evidence-taxonomy/category-memory-retention-from-code.md`. - **Capture — the holder → held → boundary triple, per suspect:** (1) the **holder** (listener, closure, module singleton, accumulating collection, timer); (2) the **held set** — the *specific* objects pinned (list the closure's captures; note when a closure links two objects' GC); (3) the **outlived boundary** (`destroy()`, stream close, instance replacement, request completion). Method: **pair every acquire with its release site** (`on`↔`removeListener`, push↔drain, assign↔null) — the absence of the pair, cited at the acquire site, IS the finding. Four canonical shapes: unbounded accumulator (defeated guard, no drain) · stale-instance listeners on replacement · unremoved listener + capture set · retention past `destroy()`. - **Scope to the diff, or you invent findings.** Classify every flagged primitive as *introduced by this PR* (in the added lines) vs *pre-existing* (already in the file). Charge only the introduced ones to the PR; report pre-existing un-paired primitives separately and uncharged. On extension#40684 the two new stream listeners each had a `removeListener` on `onStreamClosed` (the exact fix a reviewer suggested) and the new pending-request Map had its `.delete` — no leak introduced — while three pre-existing un-torn-down listeners were surfaced but left uncharged, matching how the human/bot reviewers treated them in-thread. This lane *is* the retention review automated; a heap snapshot (C7) is warranted only for an introduced primitive it cannot pair. - **Corroborate:** a falsifying lifecycle test (force the boundary, assert release — listener count zero, singleton nulled, collection drained); C7 heap-over-flow with the **retainer graph naming the same path** the static argument named. @@ -151,7 +152,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ ## D3. LavaMoat policy / supply-chain capability diff - **Engine: the `supply-chain-audit` skill** (umbrella — lockfile/manifest diff, advisories, Socket Security, install scripts), which delegates capability grants to **`lavamoat-policy-diligence`** (per-grant call-site justification). Delegate the dependency change to the umbrella; it returns a disposition per lane. pr-validate keeps **supply-chain capability diff** as the evidence category and packages the output. Note the lanes are independent: a clean policy diff does not mean a safe dependency, and a known CVE never appears as a new grant. -- **Proves:** a dependency change (bump/add/lockfile) grants **no *unjustified* new capability** — the supply-chain-risk lane. Note the bar: for a bump the policy *will* change, so "empty diff" is the WRONG test; the right test is **every new grant is justified by the dep's function**. The framing generalizes past LavaMoat to any capability-containment mechanism. +- **Proves:** a dependency change (bump/add/lockfile) grants **no *unjustified* new capability** — the supply-chain-risk lane. Note the bar: for a bump the policy *will* change, so "empty diff" is the WRONG test; the right test is **every new grant is justified by the dep's function**. Full category (trust-boundary framing, generalizes past LavaMoat to any capability-containment mechanism): `exogram-daemon/artifacts/evidence-taxonomy/category-supply-chain-capability-diff.md`. - **Capture:** **Prefer the CI-generated policy whenever one is available.** `@metamaskbot update-policies` regenerates the policy files from a real run of the code and `validate-lavamoat-policies` fails the build on drift, so the committed policy on a bot-run PR *is* the authoritative artifact — diff that. Regenerating locally when a current CI policy exists only re-does a machine that is already trusted, and a local run's provenance is weaker (your node/OS/lockfile resolution, not CI's). **Local regen is the fallback**, for when the bot hasn't run yet, the branch is unpushed, or you need a variant CI didn't cover: `yarn webpack:lavamoat:policy:build` (`:mv2` / `:mv3` for variants) over `lavamoat/webpack/build/policy.json` (+ `policy-override.json`). Either way, `git diff` the policy across **all 8 variants** (mv{2,3}/{main,beta,flask,experimental}) — a grant can appear in one and not others. Then audit **grant-by-grant**: new **globals** (`fetch`, `importScripts`, `WebAssembly`) / **builtins** (`fs`, `child_process`) on a dep that shouldn't need them, new **packages** edges to powerful APIs, or an identifier substitution (`pkgC>name` replacing `pkgB>pkgA>name` = possible dep swap). Falsifier = a surprising grant ("I wonder what it's using this for"). Guide: lavamoat.github.io/guides/policy-diff/. `allowScripts` in `package.json` gates install scripts. ## D4. Manifest permissions diff @@ -162,6 +163,25 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **Proves:** the change works across build types, not just main. - **Capture:** `yarn build:test:flask` / `:beta` / `:mv2` (`ENABLE_MV3=false`, Firefox). Run the relevant lane per variant when behavior is build-type-gated. +## D6. Authored-vs-authoritative substitution A/B ⭐ *(fixed head; lead for "the artifact restates a source" claims)* +- **Proves:** whether an artifact the PR *hand-wrote* agrees with the source it restates — a type vs the value's real type, a hand-maintained schema vs the generated one, a vendored constant vs the upstream export, a checked-in policy vs `update-policies` output. The finding is the **delta in a checker's output**, not a reading of the diff. +- **Shape:** both arms sit at the **same commit**; they differ by a *substitution*, not by a ref — so there is no build, no rebase, and no merge boundary to confound. + - **Arm A** — the PR as written, run through the checker. Must be **silent**. A non-empty Arm A means the instrument is broken and Arm B is unreadable (see trustworthiness gate item 19). + - **Arm B** — same tree, with the authored artifact replaced by the **derived** equivalent, exercised exactly as the real code exercises it. Every new diagnostic is a disagreement the authored version concealed. +- **Capture (TypeScript worked example — extension#44397, 2026-07-30):** + ```bash + # Arm A — baseline. Expect zero errors. + NODE_OPTIONS='--max-old-space-size=9216' npx tsc -p tsconfig.json --noEmit + # Arm B — probe files that substitute the derived type and call it as the caller does. + mkdir -p app/scripts/derive-probe && cp probe-*.ts app/scripts/derive-probe/ + NODE_OPTIONS='--max-old-space-size=9216' npx tsc -p tsconfig.json --noEmit # diagnostics = the findings + rm -rf app/scripts/derive-probe + ``` + One probe per claim, each naming the authoritative source in a header comment and calling the derived type the way the real call site does. Keep the probes as the artifact — they are the re-runnable falsifier. +- **Why it finds what review and CI miss:** the authored artifact compiles, so CI is green *by construction*. In a partially-migrated repo the asymmetry is structural — with `checkJs` off, a type written for a function whose callers are still `.js` is checked against nothing, and drifts silently forever. Those boundaries are where the lane pays. +- **Traps:** (a) **a substitution can fail for the wrong reason** — a diagnostic on an earlier property short-circuits the one under test, and counting exit codes reads that as confirmation; assert on the *specific* diagnostic, and re-probe with the earlier cause neutralised (`NonNullable<…>`, a targeted assertion) to isolate each claim. Same hazard as B3's "fails on base for the wrong reason." (b) **no authoritative source may exist** — an unshipped package's types, a lib not in tsconfig `lib`, a genuinely new boundary the repo owns. Hand-writing is then *correct*; report it as a cleared falsifier, not a finding. +- **Pairs with:** [lane-assertions.md](lane-assertions.md) for the recipe form; D3 when the substituted artifact is a LavaMoat policy. + --- # E. Production telemetry @@ -224,7 +244,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **G2. Coverage delta** — `yarn test:unit:coverage` → `coverage/unit/` (and `yarn test:unit:webpack:coverage`); `codecov.yml`. Proves the new code is exercised. - **G3. Automated-reviewer output** — independent bot (e.g. cursor[bot]) found nothing blocking. Complements, never replaces, behavior evidence. - **G4. Manual reproduction steps** — human-followable steps that reproduce the fixed behavior; populates the PR template's Manual testing steps. -- **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-<X>=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: in a fork you control, push to a branch literally named **`main`** (or `stable`) — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets configured on that fork (the benchmark jobs need the Infura and test-account secrets; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo. +- **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-<X>=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: push to a branch literally named **`main`** (or `stable`) on `consensys-test/metamask-extension-test-majorlift` — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets on the fork (`INFURA_PROJECT_ID`, `TEST_SRP_*` for benchmarks; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo. Detail: `exogram-daemon/memory/ci-workflow-pr-self-validation-gap.md`. --- @@ -242,6 +262,8 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ | a memory leak fixed / introduced | **C9 retention-path from code** (holder → held → boundary) | C7 heap-over-flow + retainer graph; falsifying lifecycle test | | an error/crash fixed | E1 Sentry rate→0 | B3 test, A1 if visible | | a dep change is safe | D3 LavaMoat + D4 manifest | D1 size; supply-chain-audit's patch/resolutions/ignore lanes | +| a mechanical migration / "rename-only" refactor | **D6 substitution A/B** (authored artifact vs its authoritative source) | B3 if behavior-visible; D1 for accidental output change | +| a hand-written type/schema/policy restates a source | **D6 substitution A/B** | G1 checks (as the *premise*: it compiles, which is why nobody noticed) | | runtime containment / SES / scuttling | **F8 runtime containment** (on the shipped variant) | D3 policy; E1 for `Lockdown failed` events | | persisted-state change | **F1 migration** | F2 vault | | tx/confirmation behavior | F3 simulation | B2 e2e | diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md b/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md index b1a46e1c..6f031e99 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md @@ -2,7 +2,7 @@ How to take run artifacts + complementary evidence and write a clean, idempotent, reviewer-familiar section into the PR body — **matching AEP's own format** so a re-run replaces in place instead of stacking duplicates. -Canonical source for the format: `packages/github/src/pr-body-builder.ts` (`upsertVisualValidationSection`) in the [AEP repo](https://github.com/MetaMask/metamask-autonomous-engineering-platform). Mirror it. +Canonical source for the format: `~/Code/metamask/metamask-autonomous-engineering-platform/packages/github/src/pr-body-builder.ts` (`upsertVisualValidationSection`). Mirror it. > **Publishing is public and outward-facing. Always render the section and get explicit confirmation before writing the PR body. Use `publishEvidence: false` on the run; this manual flow is the only publish path.** @@ -10,29 +10,24 @@ Canonical source for the format: `packages/github/src/pr-body-builder.ts` (`upse Control-plane artifact URLs (`localhost:3000/v1/runs/:id/artifacts/:name`) won't render on GitHub. Re-host each artifact and link the hosted URL. -**Host: an object store or repo whose read access matches your audience.** Configure it once and -reuse it; the examples below assume an S3 bucket exposed through an environment variable: - -```bash -# set these to a bucket you control whose `public/` prefix allows anonymous GetObject -EVIDENCE_BUCKET=<your-bucket> -EVIDENCE_BASE="https://$EVIDENCE_BUCKET.s3.<region>.amazonaws.com" -``` +**Host: the S3 bucket `majorlift-artifacts-share`, prefix `public/`.** ``` -s3://$EVIDENCE_BUCKET/public/metamask/pr-<n>/<run-id>/<artifact-name> -$EVIDENCE_BASE/public/metamask/pr-<n>/<run-id>/<artifact-name> +s3://majorlift-artifacts-share/public/metamask/pr-<n>/<run-id>/<artifact-name> +https://majorlift-artifacts-share.s3.us-west-1.amazonaws.com/public/metamask/pr-<n>/<run-id>/<artifact-name> ``` -Allow anonymous `GetObject` under `public/*` but not bucket listing, so the prefix is not +Anonymous `GetObject` is allowed under `public/*`; bucket listing is not, so the prefix is not browsable — link individual files, and don't promise readers an index. -**Do NOT re-host to a personal repo.** A personal private repo returns 404 for every reader but -its owner, so every raw link to it is dead on arrival. +**Do NOT re-host to `MajorLift/metamask-extension-skills`.** It is a **personal private** repo: +every raw link to it returns 404 for every reader but its owner. That was the previous target +here, and this file simultaneously said links to it were unreachable — guidance that instructed +you to publish dead links. Verified live in a published artifact. -The test is **audience-reachability, not public-vs-private.** An org repo that is private but -readable by colleagues is fine for an internal-audience link. A personal repo is unreachable by -colleagues *and* by the public, so it fails for every audience. +The test is **audience-reachability, not public-vs-private.** A `MetaMask/*` org repo is private +but readable by colleagues, so an internal-audience link to one is fine. A `MajorLift/*` personal +repo is unreachable by colleagues *and* by the public, so it fails for every audience. - Path convention: `pr-<n>/<run-id>/<artifact-name>` keeps runs from colliding. - **Verify unauthenticated before shipping**: `curl -s -o /dev/null -w "%{http_code}"` on each @@ -40,8 +35,8 @@ colleagues *and* by the public, so it fails for every audience. ```bash RUN_ID=<id>; PR=<n>; CP=localhost:3000 -BUCKET="$EVIDENCE_BUCKET" -BASE="$EVIDENCE_BASE" +BUCKET=majorlift-artifacts-share +BASE="https://$BUCKET.s3.us-west-1.amazonaws.com" for name in <artifactName1> <artifactName2>; do curl -fsS "$CP/v1/runs/$RUN_ID/artifacts/$name" -o "/tmp/$name" key="public/metamask/pr-$PR/$RUN_ID/$name" @@ -109,9 +104,9 @@ Screenshots block (injected into `### After`, or appended under `### Screenshots <!-- AEP_SCREENSHOTS_START --> <details open><summary><artifact-name></summary> -<img alt="<artifact-name>" src="<hosted artifact URL>" width="420" /> +<img alt="<artifact-name>" src="<raw.githubusercontent URL>" width="420" /> -[Open full-size image](<hosted artifact URL>) +[Open full-size image](<raw URL>) </details> <!-- AEP_SCREENSHOTS_END --> @@ -121,15 +116,15 @@ Screenshots block (injected into `### After`, or appended under `### Screenshots ## Step 3 — Choose the surface by ownership, then publish -**Publish surface depends on your relationship to the PR.** Determine it FIRST: +**Publish surface depends on my relationship to the PR** (see exogram +`pr-validate-publish-surface-by-ownership`). Determine it FIRST: ```bash PR=<n>; REPO=MetaMask/metamask-extension -ME=$(gh api user --jq .login) -SURFACE=$(gh pr view "$PR" --repo "$REPO" --json author,commits --jq --arg me "$ME" ' - if .author.login==$me then "body" - elif ([.commits[] | select(.authors[].login==$me) - | select([.authors[].login] | map(select(.!=$me and .!="Copilot" and (test("claude|anthropic")|not))) | length == 0)] | length) > 0 +SURFACE=$(gh pr view "$PR" --repo "$REPO" --json author,commits --jq ' + if .author.login=="MajorLift" then "body" + elif ([.commits[] | select(.authors[].login=="MajorLift") + | select([.authors[].login] | map(select(.!="MajorLift" and .!="Copilot" and (test("claude|anthropic")|not))) | length == 0)] | length) > 0 then "comment" else "skip" end') ``` @@ -236,6 +231,7 @@ The common loop — a run refutes a claim, the author pushes a fix, `/pr-validat - New head → **new hosted artifact directory keyed to the fix commit** (`pr-<n>/fix-<sha>/`), commit-pinned raw URLs; never overwrite a prior run's published files. - Residuals the fix intentionally leaves get their own row/section — don't round a fixed-with-residual claim up to fully proven. +Source of truth: `exogram-core/memory/pr-validate-revalidation-delta-reports.md`. ## Lead with a lane-status ledger (no silent absence) @@ -270,7 +266,34 @@ Per-lane rendering: Multi-claim PRs get one sub-block per claim under the status section, each with its own ✅/❌/⚠️ verdict — mirror the Claim Cards. Keep the visual block (markers + `### After` injection) for the image lanes; render the rest as text beneath it. -**Per-scenario presentation (2026-07-21):** the same applies one level down — when the evidence spans multiple test scenarios (flag-on vs flag-off, control vs treatment in an A/B falsifier, numbered manual-testing steps), give each scenario its **own sub-section**: a heading naming the scenario in observation terms, one line on what it tests plus its verdict, and that scenario's artifacts co-located under it. Never bunch all scenarios' artifacts into one large evidence dump — the reviewer verifies "under condition X, artifact shows Y" one condition at a time, and a merged block destroys that mapping even when every artifact is real. For long artifact sets use a `<details>` block *per scenario*, not a merge. +### One comment per evidence *kind*, not one comment per PR (2026-07-30) + +Sub-blocks are for several claims **of the same kind**. When a PR draws two different +kinds — say an executed Validation Run *and* a read-level capability triage — they get +**separate comments**, each with its own header, its own marker pair, and its own format. + +| | Validation Run | LavaMoat policy diligence | +|---|---|---| +| header | `## 🧪 Validation Run` | `## 🔒 LavaMoat Grants — <pkg> <old> → <new>` | +| markers | `VALIDATION_RUN_*` | `LAVAMOAT_DILIGENCE_*` | +| opens on | `**Verdict:** ✅/⚠️/❌` | the finding; **no verdict at all** | +| body | lane ledger, artifacts per lane | deny candidates, enumeration folded | +| audience | whoever owns the PR's claim | whoever owns the policy | + +Merging them forces one frame onto both. A read-level triage has no run to verdict, so it +would land as `⚠️ inconclusive` on a header promising a run; and a `⏳ not-captured` lane +needs a tracker it does not have. The marker pairs also collide — a re-run replacing the +`VALIDATION_RUN` region would silently eat the diligence output sharing it. + +**So: choose the format from the evidence kind, not from this document's default.** The +canonical `## 🧪 Validation Run` header applies when a run produced artifacts. An engine +skill that defines its own output contract (`lavamoat-policy-diligence`) publishes in that +contract. `hooks/pr-evidence-gate.py` enforces the canonical literal only on bodies that +*claim* validation/evidence framing — a diligence comment that renders no verdict does not +trip it, which is the tell that the two are different artifacts rather than one with a +different skin. + +**Per-scenario presentation (2026-07-21):** the same applies one level down — when the evidence spans multiple test scenarios (flag-on vs flag-off, control vs treatment in an A/B falsifier, numbered manual-testing steps), give each scenario its **own sub-section**: a heading naming the scenario in observation terms, one line on what it tests plus its verdict, and that scenario's artifacts co-located under it. Never bunch all scenarios' artifacts into one large evidence dump — the reviewer verifies "under condition X, artifact shows Y" one condition at a time, and a merged block destroys that mapping even when every artifact is real. For long artifact sets use a `<details>` block *per scenario*, not a merge. (Preference: exogram-core `memory/pr-validate-present-scenarios-separately.md`; instance #44610.) ## Artifact contract (ADR-0058 alignment) @@ -283,7 +306,7 @@ To stay interoperable with the recipe-based verification system (MetaMask/decisi - [ ] Each lane passed the [trustworthiness gate](evidence-trustworthiness.md) (shows the claimed surface, signal > noise, could-have-failed) - [ ] Multi-scenario evidence rendered **per scenario** (own heading + verdict + co-located artifacts), not bunched into one block - [ ] **Automated-process voice, no first person** — published validation output never says "I ran/captured/verified"; attribute to the process ("Automated validation ran…", "the harness captured…") so readers know the evidence is machine-generated, not a manual account under the author's name -- [ ] Every image/GIF re-hosted to your configured evidence host; no localhost/local-path URLs in the body +- [ ] Every image/GIF re-hosted to `majorlift-artifacts-share/public/…`; no localhost/local-path URLs in the body - [ ] **Every published link curl'd unauthenticated and returning 200** — never a personal private repo - [ ] Work cited by **PR link** rather than tracking-ticket id, unless the ticket's own content (an RCA, a spec) is the referent - [ ] Narrative scrubbed of username/paths/internal hosts diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md b/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md index 482f01e5..06478a68 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md @@ -2,10 +2,6 @@ A green result is not proof. An agent — or an eager run — can produce evidence that *looks* like it validates the claim but doesn't. Before believing or publishing any lane, run it through this gate. It extends the vacuous-pass trap to all lanes; the Claim Card's **Falsifier** is the anchor: trustworthy evidence is evidence that *could* have shown the falsifier and didn't. -> **Which items are mechanically enforced.** Several items below close with an *"Emit-time trigger: `pr-evidence-gate.py` class …"* note. Every such class is implemented in [`hooks/pr-evidence-gate.py`](../hooks/pr-evidence-gate.py), which runs as a `PreToolUse:Bash` hook and blocks the write: `verdict`, `observation`, `deferral`, `ci-restatement`, `inflated-verdict`, `bare-identifier`, `truncated-identifier`, `mutable-ref`, `dump-resolver`, `link-only-exhibit`, `data-only-exhibit`, `step-waiver`. It polices both the `gh pr|issue edit|create|comment` porcelain and `gh api` body writes, since a PATCH to a comment is the same publish with a different spelling. -> -> **What stays procedural.** The hook sees vocabulary, not semantics. It cannot tell whether an embedded screenshot actually shows the resolving UI's chrome (item 17), whether a deferral's stated blocker matches the step's own precondition (item 14), or whether inline data is a faithful quotation rather than a transcription (item 16). Those remain reader-applied. Setup: [evidence-gate-setup](evidence-gate-setup.md). - ## The gate (per lane, before publish) 1. **Non-empty & expected media** — the bundle has artifacts of the expected kind. Zero artifacts = not a pass (the vacuous-pass guard). @@ -13,7 +9,7 @@ A green result is not proof. An agent — or an eager run — can produce eviden 3. **Exercises the changed code** — the test/flow actually hits the diff. For a test: it **fails on `main`** (catalog B3). For a flow: the changed component/route is on the path. A green test that never imports the changed module proves nothing. 4. **Signal exceeds noise — and a null states its power** — a perf delta must be beyond run-to-run variance (paired A/B, multiple iterations); a 3% move on a noisy metric is not evidence. The same bar applies in reverse: when the spread is wider than the effect being looked for, the finding is **"not resolvable at this sample size"**, never "no change" — an underpowered run and a true null print the same word, and reporting the word alone lets the reader infer the stronger claim. State the smallest effect the design could have detected. - **Removing a bias is not establishing validity.** Correcting a flaw you found (discarding a warm-up, alternating the starting arm, pinning CPU governor) removes *that* bias and licenses no more than that. It is not a trust gate, because a trust gate names how the evidence could **still** be vacuous — residual risk, not completed work. List what remains uncontrolled (thermal drift, background load, ordering within a round); an unenumerated confound reads as a nonexistent one. - - **When correcting an overclaim, cut the certainty, not the evidence.** A falsifier that actually fired is the strongest thing on the page — downgrade the conclusion around it, don't delete it with the overclaim. + - **When correcting an overclaim, cut the certainty, not the evidence.** A falsifier that actually caught something is the strongest thing on the page — downgrade the conclusion around it, don't delete it with the overclaim. 5. **Could have failed** — the assertion has a reachable failure mode. Always-true assertions (`expect(true)`, a screenshot with no assertion, a Sentry query with no time bound) can't falsify anything. 6. **Right baseline** — "before" is the actual base ref / prior version / pre-window, not a stale or mismatched comparison. 7. **Artifacts are independent & honestly labeled** — checksum every capture set (`md5 *`). Byte-identical files across supposedly independent runs/cases cannot stand as separate observations: either explain the identity in the artifact bundle (deterministic fixture rendering) with per-run provenance that *does* differ (the harness state dump, timestamps, a manifest), or re-capture at distinct moments. Labels must describe the observation, not the interpretation — a file named for the state it *should* show under the claim (`steady-state`, `no-toast`) misleads when the capture shows the refutation. @@ -25,15 +21,18 @@ A green result is not proof. An agent — or an eager run — can produce eviden 13. **Terminal exhibits are reader-native — a live link or a visual; a dump behind a link is still an opaque reference** — item 12 makes every pointer resolve in one click; this item constrains what it may resolve *to*. A positive verdict's terminal artifact is one of the two media a reviewer natively consumes: a **live link into the resolving system** (Sentry trace/event permalink, absolute-windowed Discover query pre-filtered to the claim) or a **visual capture** (screenshot/recording, annotated or cropped to the discriminating region). Raw files (`.log`/`.json`/`.har`, MB-scale dumps) are **appendix-only** — linked once for auditability, never the exhibit a claim rests on: a link whose target is a raw dump passes item 12 and fails item 9 one click later; the digging moved a hop away, it did not disappear (extension#43931 *second* remediation, 2026-07-21: the `bare-identifier` fix shipped a ✅ row whose sole resolver was a re-hosted ~70KB run log). Two corollaries: (a) **the gate items are conjunctive** — a fix for the newest item must re-pass all prior items; satisfying resolvability with an artifact that fails legibility is the generator's next costume; (b) **ascertain the terminal medium at step zero and pick the capture lane that can produce it** — a local intercept (mockttp envelope forwarder) can never yield a live Sentry permalink, so for Sentry-observable claims it is the supplementary falsifier lane and live ingest (dev build → `SENTRY_DSN_DEV`/test-metamask) is primary, precisely because it terminates in permalinks + screenshots; choosing a lane that cannot produce the terminal medium silently displaces it. Emit-time trigger: `pr-evidence-gate.py` class `dump-resolver`, with the remediated extension#43931 row as the regression case (2026-07-21). 14. **Manual testing steps are the validation contract — steps present ⇒ live evidence definitionally required; an impossibility waiver contradicting an executable step is invalid** — the PR's own **Manual testing steps** are the author's assertion that the claimed behavior *is* live-observable, and how: each numbered step is an executability proof (a step a human reviewer can run, `/pr-validate` can run) and its text is the capture spec. Item 11 derives the lanes from the steps top-down; this item closes the other side of the hatch — a lane derived from a step may not then be *waived by argument*. Emit-time procedure: build the per-step coverage map (step → executed-output artifact); for any step without one, the only admissible state is a **per-step** ⏳ + tracker whose blocker is that step's *own* unmet precondition, checked against the step text. Three waiver-inflation patterns from the producing instance (extension#43228/#42869/#44538, 2026-07-21 — all three shipped articulate impossibility rationales in the same body whose Manual testing steps asserted the opposite): (a) **borrowed impossibility** — the excuse imported from a different mechanism or sibling PR (#43228 waived its live lane citing the async remote-flag read race, which is #44538's mechanism; #43228's overrides are build-time env vars, and its steps 1–4 are directly executable in a dev build); (b) **lane-limitation universalized** — one harness's gap stated as global impossibility (#42869: "the e2e harness emits no error `event` envelope, so … not capturable pre-merge" — the step says *dev build with Sentry enabled, trigger an error*, which ingests error events into test-metamask without the e2e harness); (c) **blocked-scope inflation** — a genuinely blocked precondition of one half of the claim expanded to waive the whole lane (#44538: LaunchDarkly provisioning [#7482](https://github.com/MetaMask/MetaMask-planning/issues/7482) blocks only the *prod-flag* half; the step's own text names the dev-injectable alternative — "or inject it into persisted `RemoteFeatureFlagController` state"). If a step is *truly* non-executable, the waiver is still inadmissible alone: the Manual testing steps are then wrong and are corrected in the same edit — a document may not simultaneously instruct a reviewer to observe X and declare X unobservable. The honest-⏳ lane blessed throughout this gate is for *not yet done*, never for *argued away*: an eloquent impossibility rationale is the cheapest token sequence that satisfies every prior item (no fake capture, no CI link, no bare id, no dump) — the generator's costume for the coverage axis. Emit-time trigger: `pr-evidence-gate.py` class `step-waiver` ("not demonstrable / capturable / observable", "not separately captured", "not attached", "rests entirely/solely on the falsifiers/unit/revert" — unconditional in validation scope; no tracker or artifact excuses it), with the three shipped waiver paragraphs as regression cases and the gate verified-blocking on all three live bodies (2026-07-21). Detection gap: the per-step consistency check (does a deferral's blocker match the step's own precondition?) stays procedural — the gate sees vocabulary, not step semantics. 15. **The exhibit lives in the body — link AND visual; a live link alone is the verification path, not the exhibit** — item 13 blessed the terminal media as a *disjunction* (live link OR visual), and generation took the cheaper disjunct: a Discover permalink is producible from the API token alone, a screenshot needs a browser session — so extension#44540's live-ingestion exhibit shipped as a permalink + prose counts, with nothing in the PR body a reader could look at (2026-07-21: "only sentry link and not screenshot that makes it immediately obvious how evidence validates pr"). A live link defers validation behind **click + auth + query rendering + column interpretation** — the dump-resolver displacement one hop further, with the mountain now behind a login: it fails item 9's ~30-second test at the moment of the click, and for any reader *without* Sentry org access (most PR reviewers) a link-only exhibit degrades to a bare identifier (item 12) behind an auth wall. The repaired rule is a **conjunction**: a positive verdict's headline exhibit is an **embedded visual** — screenshot/recording of the linked resolving view (Discover result rows, trace waterfall), cropped/annotated to the discriminating region, captioned with what it should show — **and** the co-located live permalink (absolute-windowed) as the independent-verification path. Neither substitutes for the other: link-only hides the exhibit; visual-only is independently unverifiable. The 2026-07-16 clause "screenshots ride along when a browser session is available; the API token alone yields links + JSON, which is the automatable minimum" was the self-authored escape hatch of this axis (family: the "as context" carve-out, the honest-⏳ waiver): the *automatable minimum* got promoted to the shipped standard because it was the cheapest compliant artifact. A capture lane that cannot screenshot its resolving view is a lane gap to fix before publish (drive a browser session to the Discover URL), never a licensed downgrade — deliberate exceptions route through the human. Emit-time trigger: `pr-evidence-gate.py` class `link-only-exhibit` (non-negated verdict + `sentry.io` link + no image/recording embed in the unit), with the shipped #44540 paragraph as the regression case and the prior suite's permalink-only ALLOW cases flipped/augmented — third occurrence of "an ALLOW case containing the violating tokens is a specification of the next costume." Detection gaps: verdict co-location is required, so a no-verdict link-only paragraph evades mechanically; the visual-without-link converse stays procedural under item 12. -16. **The audit chain is mechanical — quote, don't transcribe; pin, don't point** — an exhibit's inline data must be a **verbatim, greppable excerpt** of the artifact (full-length identifiers, raw capture lines quoted exactly), and every repo-hosted artifact link must be a **commit-pinned, line-anchored permalink** (`/blob/<sha>/…#Lx-Ly`) to the discriminating lines. Producing real artifacts and then hand-transcribing digests severs the claim→artifact bridge at every link: an ellipsized id (`24b1e2da…`) cannot be grepped against any artifact even when the log is linked in the same block; a reformatted data block cannot be distinguished from confabulation without redoing the dig, so it reads as *claims in the form of data*; a branch-ref `/blob/<branch>/` link is a mutable pointer whose target can be rewritten after review (not tamper-evident); a file-level link without line anchors lands the reader at the top of a 1,000-line dump. The producing instance (extension#43929 validation comment, 2026-07-21) had every number substantiated by four re-hosted run logs — real, included, resolvable — and still drew "still no immediately auditable evidence just claims in the form of data": each prior item individually near-passed while the exhibit↔artifact binding stayed **editorial** (transcription + file link) instead of **mechanical** (quotation + pinned line anchor). Emit-time procedure: for every inline datum, quote the raw capture line it comes from (fenced, verbatim, full ids) and anchor it (`#L<n>`); pin every evidence link to the SHA (press `y` on the GitHub file view). Emit-time trigger: `pr-evidence-gate.py` classes `truncated-identifier` (a co-located resolver does NOT excuse it — the resolver resolves the full id, not the fragment the reader holds; hash-equality prose exempt) and `mutable-ref`, plus the closed **surface hole**: the shipped comment was published via `gh api` PATCH, which the porcelain-only matcher (`gh pr|issue edit|create|comment`) never saw — the gate now scans `gh api` body writes (`-F body=@file`, `-f body=…`, `--input`); fifth occurrence of the converse-of-gate rule, one level down: audit the gate for *spellings of the write it cannot see*, not just tokens it excuses. Detection gaps: the line-anchor half of pinning and the paraphrase-vs-quotation judgment stay procedural — the gate sees ellipses and branch refs, not editorial fidelity. +16. **The audit chain is mechanical — quote, don't transcribe; pin, don't point** — an exhibit's inline data must be a **verbatim, greppable excerpt** of the artifact (full-length identifiers, raw capture lines quoted exactly), and every repo-hosted artifact link must be a **commit-pinned, line-anchored permalink** (`/blob/<sha>/…#Lx-Ly`) to the discriminating lines. Producing real artifacts and then hand-transcribing digests severs the claim→artifact bridge at every link: an ellipsized id (`24b1e2da…`) cannot be grepped against any artifact even when the log is linked in the same block; a reformatted data block cannot be distinguished from confabulation without redoing the dig, so it reads as *claims in the form of data*; a branch-ref `/blob/<branch>/` link is a mutable pointer whose target can be rewritten after review (not tamper-evident); a file-level link without line anchors lands the reader at the top of a 1,000-line dump. The producing instance (extension#43929 validation comment, 2026-07-21) had every number substantiated by four re-hosted run logs — real, included, resolvable — and still drew "still no immediately auditable evidence just claims in the form of data": each prior item individually near-passed while the exhibit↔artifact binding stayed **editorial** (transcription + file link) instead of **mechanical** (quotation + pinned line anchor). Emit-time procedure: for every inline datum, quote the raw capture line it comes from (fenced, verbatim, full ids) and anchor it (`#L<n>`); pin every evidence link to the SHA (press `y` on the GitHub file view). Emit-time trigger: `pr-evidence-gate.py` classes `truncated-identifier` (a co-located resolver does NOT excuse it — the resolver resolves the full id, not the fragment the reader holds; hash-equality prose exempt) and `mutable-ref`, plus the **surface hole** — the shipped comment was published via `gh api` PATCH, which the porcelain-only matcher (`gh pr|issue edit|create|comment`) never saw; fifth occurrence of the converse-of-gate rule, one level down: audit the gate for *spellings of the write it cannot see*, not just tokens it excuses. **CORRECTION 2026-07-30 — this hole was recorded as closed and is not.** Verified against the deployed `hooks/pr-evidence-gate.py` (259 lines): line 47 is the only command matcher, `\bgh\s+(?:pr|issue)\s+(?:edit|create|comment)\b`, so `gh api` body writes are still invisible; and the file implements essentially one check (verdict-needs-artifact), **not** the ~9 classes named across items 11–18 (`ci-restatement`, `bare-identifier`, `dump-resolver`, `link-only-exhibit`, `data-only-exhibit`, `step-waiver`, `truncated-identifier`, `mutable-ref`, `inflated-verdict`). Treat every "Emit-time trigger: `pr-evidence-gate.py` class …" line in this document as **specified, not implemented**, until re-verified in the code — a doc asserting a class the code lacks retires the vigilance it claims to replace, which is the failure this very item warns about. Consequence observed the same day: 14 unlinked `path:line` references shipped across 12 review comments via `gh api`, with the gate both classless for that shape and unwired in `settings.json`. Detection gaps: the line-anchor half of pinning and the paraphrase-vs-quotation judgment stay procedural — the gate sees ellipses and branch refs, not editorial fidelity. 17. **Evidence is captured in its environment — data alone is insufficient even when correct** — item 16 makes the data trustworthy as *transcription* (verbatim, greppable, pinned); this item polices what transcription can never carry: **liveness provenance**. A quoted `EVIDENCE trace_id=…` line, a re-hosted gist, a hand-assembled id table can all be correct and still show nothing about *where they came from* — extracted data is indistinguishable from data typed by hand, so it cannot make it immediately apparent that the evidence was captured **live** from a **functioning** system. The exhibit for a system-of-record-observable claim therefore includes an **in-environment capture**: a screenshot/recording of the resolving system's own UI (the Sentry Discover/trace view with the query, project/environment selectors, absolute time window, and result rows all in-frame) — the environmental chrome is not decoration, it *is* the provenance: it shows the query really ran, in the real dashboard, over the real window, and returned these rows. Correctness was never the failing dimension (2026-07-21: "just the data is insufficient even if correct — it needs to be immediately apparent that evidence was captured live and is functional"). Relation to prior items: item 15's link+visual conjunction fired only when a `sentry.io` link was present, and item 13's `NATIVE_MEDIUM` blessed an inline fenced excerpt as a terminal medium — so a no-link, quoted-data exhibit (the fidelity-remediated shape: full ids, verbatim excerpts, pinned line anchors, zero environment captures) passed the whole regime while carrying zero liveness provenance. The joint rule after this item: a telemetry-observable positive verdict always carries the in-environment visual (plus the live permalink per item 15); quoted excerpts, gists, and data files are appendix beside it, never the exhibit. Emit-time trigger: `pr-evidence-gate.py` class `data-only-exhibit` (non-negated verdict + telemetry-observation vocabulary + no image/recording embed + no sentry link — with a sentry link, `link-only-exhibit` already fires), with the re-hosted-gist ALLOW case flipped (fifth occurrence of "the ALLOW case was the next costume's spec") and the #43929 quoted-excerpt shape as a regression case. Detection gaps: vocabulary-scoped (telemetry-observation terms, not bare code tokens like `trace.test.ts`), so a claim phrased entirely without them evades mechanically; and the gate cannot see whether an embedded image actually shows the environment's chrome — screenshot content stays procedural (item 2's "eyeball it" applies: the capture must show the *resolving UI*, not a cropped data region indistinguishable from a spreadsheet). 18. **"Successful" is an evidence predicate, not a run status — and the default Sentry exhibit is fixed in advance** — a validation run may be scored/reported "successful"/"validated" only when its published surface already carries, for every Sentry-observable lane, the default exhibit pair: an **in-environment Sentry-UI screenshot** (item 17) **plus the co-located live permalink** (item 15). Completed runs, green falsifiers, staged drafts, and honest ⏳ lanes do not confer success — a run without the pair is at most "run-complete, evidence-owed." The default recipe needs no per-PR ascertainment: for Sentry, **generally capture actual screenshots from the Sentry UI and attach the link** — that pair is step zero's pre-computed answer for any Sentry-observable claim, never the terminus of axis-by-axis escalation. Capture-first ordering: the capture executes before any rule/gate/postmortem authoring may close a validation session — writing a new rule or gate class discharges nothing (2026-07-22: ten postmortems and 17 gate items shipped while zero Sentry-UI screenshots did; every "successful" run was claims-only, because success was assigned by run-completion and meta-work substituted for capture work). Emit-time trigger: `pr-evidence-gate.py` `VERDICT` vocabulary now includes the status spellings `successful`/`validated`/`live-proven`, so a claims-only unit scoring itself successful blocks like any bare "confirmed." Detection gap: the gate fires only on re-emit — already-shipped "successful" surfaces are audited by backward re-score, enumerated from live state, never from the ledger (the discharge-granularity rule applies to success statuses verbatim). +19. **A substitution A/B is readable only if the unmodified arm is silent — and only if each diagnostic fires for the reason claimed** — the substitution lane (catalog D6) derives its finding from the *delta* in a checker's output between the PR as written (Arm A) and the PR with one authored artifact replaced by its authoritative equivalent (Arm B). Two ways that delta lies, both of which look like a confirmed finding. (a) **A noisy Arm A destroys attribution.** If the unmodified tree already emits diagnostics, nothing in Arm B is attributable to the substitution — the reader cannot tell a concealed disagreement from ambient breakage, and "N errors in Arm B" is then a count, not a finding. Publish Arm A's result explicitly (`0 errors`, verbatim) as the delivery check; if it is non-empty, the instrument is broken and the lane is **inconclusive**, not a pass — fix the baseline (pin the toolchain, raise the heap, exclude the unrelated project) or drop the lane. This is the substitution analogue of item 1's vacuous-pass guard: item 1 asks whether the artifact exists, this asks whether the *comparison* means anything. (b) **A diagnostic can fire for the wrong reason.** A checker reports the first failure it reaches, so an earlier cause short-circuits the claim under test and an exit-code read scores it as confirmation — the same hazard item 3 polices for tests that fail on base for an import error rather than the bug. Producing instance (extension#44397, 2026-07-30): a probe asserting a hand-written provider return type was unsound errored on *nullability* one property earlier, and the return-type claim — re-probed with the nullability neutralised via `NonNullable<…>` — turned out to be **sound**, i.e. a finding that would have shipped as real. Emit-time procedure: for every substitution claim, assert on the *specific* expected diagnostic (code + message + line), not on non-zero exit; where an earlier cause intervenes, isolate it and re-run; and report the claims the re-probe **cleared** alongside the ones it confirmed — a substitution sweep that only ever confirms is indistinguishable from one that never isolated anything. Corollary on the negative case: when no authoritative source exists (a package that ships no types, a lib absent from tsconfig `lib`, a boundary the repo genuinely owns), hand-writing is *correct* — record it as a cleared falsifier with the reason, never as an unreported non-finding. Detection gap: procedural — a gate can see whether Arm A's result is published, not whether the diagnostic it cites is the one the claim needs. + ## Lane-specific traps - **Visual:** spinner/skeleton mistaken for the loaded state; the toggle (privacy/redaction) not actually flipped; a cached screenshot from a prior run; the fallback surface shown without saying so. - **Perf / benchmark:** stale frozen baseline (catalog C5 caveat); single sample; warm-vs-cold mismatch; measuring a different interaction than the claim. - **Test:** snapshot regenerated to match the bug (`--updateSnapshot` masking a regression); the test mocks out the changed path; it passes on `main` too (so it's not a regression test). +- **Substitution A/B (D6):** a non-silent Arm A (attribution destroyed); a diagnostic that fires one property earlier than the claim (isolate and re-probe); a substitution the checker never reaches because the caller is untyped JS or `any`; treating "no authoritative source exists" as a null result rather than a cleared falsifier. - **Telemetry:** query window excludes the release; the error regrouped under a different fingerprint; sample-rate makes "0 events" meaningless. - **Migration:** only the happy path asserted; `changedKeys` not checked against actual mutations; no real prior-version fixture. - **Coverage:** a line covered ≠ a behavior asserted (executed but never checked). diff --git a/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md b/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md index 31316c66..89622327 100644 --- a/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md +++ b/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md @@ -23,4 +23,4 @@ Maps each evidence-catalog lane to a declarative assertion form, so a Claim Card | F7 i18n | static: `verify-locales` exit 0 | out-of-band | | F8 runtime containment | `Object.isFrozen(Object.prototype)`; scuttled global throws + exception resolves; `typeof SNOW` | **yes** — `Runtime.evaluate`, but only against the SHIPPED build variant (dev is unscuttled, test's exception list is wider) | -**Takeaway.** UI-state and runtime-metric lanes (A/B-visual, C1–C3/C6, F3/F5) map cleanly to CDP recipe assertions — ADR-0058's sweet spot. Static (D, F7), test-runner (B3, C5, F1), and dashboard (E) lanes are **out-of-band**: the recipe should *reference* them as proof targets without executing them. That out-of-band reference is precisely the **non-UI scaling gap** raised in review on decisions#173 — a recipe schema that admits out-of-band assertion references (not only CDP actions) closes it. This table is the proposed taxonomy to contribute back (ITERATION item 10). +**Takeaway.** UI-state and runtime-metric lanes (A/B-visual, C1–C3/C6, F3/F5) map cleanly to CDP recipe assertions — ADR-0058's sweet spot. Static (D, F7), test-runner (B3, C5, F1), and dashboard (E) lanes are **out-of-band**: the recipe should *reference* them as proof targets without executing them. That out-of-band reference is precisely the **non-UI scaling gap** MajorLift's review of #173 flagged — a recipe schema that admits out-of-band assertion references (not only CDP actions) closes it. This table is the proposed taxonomy to contribute back (ITERATION item 10). diff --git a/domains/pr-workflow/skills/pr-validate/skill.md b/domains/pr-workflow/skills/pr-validate/skill.md index bb8e20f9..660ef602 100644 --- a/domains/pr-workflow/skills/pr-validate/skill.md +++ b/domains/pr-workflow/skills/pr-validate/skill.md @@ -1,7 +1,6 @@ --- name: pr-validate description: Validate a MetaMask PR with objective evidence — primarily the Autonomous Engineering Platform (AEP) harness (visual_validation for visible UI behavior, perf_validation for non-visible perf behavior), backed by complementary evidence (Sentry query links, screenshots, screen recordings→GIF, DevTools/CDP output, bundle-size, web-vitals, test/CI results). Drives the AEP local stack end to end: preflight (postgres + temporal + worker + control-plane) → submit POST /v1/tasks → poll GET /v1/runs/:id → fetch artifacts → assemble an evidence bundle → publish to the PR body (re-hosting images to a public repo, scrubbing local paths). Match evidence to the PR's specific falsifiable claim, not a fixed checklist. Triggers on /pr-validate, /pr-validate visual, /pr-validate perf, /pr-validate preflight, /pr-validate status, /pr-validate evidence, /pr-validate plan, or when the user mentions validating/proving a PR, AEP / visual validation / perf validation, capturing evidence for a PR, before/after screenshots, a screen recording or GIF for a PR, attaching Sentry links or DevTools output as proof, or publishing an evidence bundle to a PR body. -maturity: experimental --- # /pr-validate @@ -18,7 +17,7 @@ Twenty rules the rest of this skill implements. When a situation isn't covered b **What you may claim** - **Falsifiability** — name the observation that would disprove the claim, then go looking for it. A review that cannot fail is not a review. -- **Falsifier coverage is not exhaustive — human intervention point.** There is no fixed checklist, so there is no completeness guarantee: the falsifiers found are bounded by claim-extraction quality and by what the reviewer thought to test. "No falsifier fired" is not "no falsifier exists." A human judges whether the falsifier chosen matches the claim's actual risk, and whether a mixed or high-stakes claim needed more than one — this skill closes the falsifiers it finds, it does not attest that it found all of them. +- **Falsifier coverage is not exhaustive — human intervention point.** There is no fixed checklist, so there is no completeness guarantee: the falsifiers found are bounded by claim-extraction quality and by what the reviewer thought to test. "Nothing turned up" is not "there is nothing to find." A human judges whether the falsifier chosen matches the claim's actual risk, and whether a mixed or high-stakes claim needed more than one — this skill closes the falsifiers it finds, it does not attest that it found all of them. - **Diff-anchored** — the claim is what the code *can* do, not what the PR body promises. Drift between them is a finding, not a claim. - **Surface-specific, and a surface need not be a screen** — a job graph, a build artifact, a policy file, or a telemetry shape are all legitimate surfaces with their own falsifiers. @@ -74,10 +73,10 @@ A claim must be falsifiable, surface-specific, **anchored to the diff** (if the | Telemetry / error-rate / latency in prod | **E1 Sentry links** (before/after) | E2 Tempo; span-volume → `/sentry-quota` | | Bundle / build output | **D1 size · D2 chunk membership** | — | | A dependency change is safe | **D3 LavaMoat policy + D4 manifest diff** | D1 size | -| Runtime containment still holds | **F8 SES lockdown / scuttling, on the shipped variant** | D3 policy | | Persisted-state change | **⭐ F1 migration** (`changedKeys`, old→new state) | F2 vault round-trip | | Tx / dapp / flag / snap / i18n behavior | **F3 sim · F4 provider · F5 flag matrix · F6 snaps · F7 i18n** | B2 e2e trace | | Behavior with no UI | **B3 test + G4 repro** | G1 CI checks | +| Mechanical migration / "rename-only" refactor; a hand-written type/schema/constant/policy that restates a source | **⭐ D6 substitution A/B** at a fixed head (`compare` arm kind `substitution`) | B3 if behavior-visible; D1 for accidental output change | Lane IDs (A1, B3, …) index [references/evidence-catalog.md](references/evidence-catalog.md) — the full menu with verified capture commands and the complete matching guide. When a PR mixes claims (a UI fix that also shifts a metric), run more than one lane and assemble them into one bundle. @@ -97,35 +96,43 @@ Not for code-correctness review (use `/review`, `/code-review`) or span-quota re | `/pr-validate <pr>` | **Flagship.** Read the PR → state the claim → pick lanes → [preflight](#preflight) → run AEP lane(s) + gather complementary evidence → assemble bundle → **propose** the PR-body section and confirm before publishing. | | `/pr-validate plan <pr>` | Dry run: read the PR, state the claim, recommend lanes + targeting hints. No stack, no run. Cheap first step when unsure. | | `/pr-validate visual <pr>` | AEP `visual_validation` only. | -| `/pr-validate perf <pr>` | AEP `perf_validation` only — check the graph is present first, see [caveat](#perf_validation-caveat). | +| `/pr-validate perf <pr>` | AEP `perf_validation` only (local/uncommitted graph — see [caveat](#perf_validation-caveat)). | | `/pr-validate preflight` | Health-check the local stack; bring up what's down. No run. | | `/pr-validate status <run-id>` | Poll `GET /v1/runs/:id`; print stage timeline + `evidenceBundle.artifactRefs`. | | `/pr-validate evidence <pr> [--run <id>]` | Assemble + publish a bundle from an existing run and/or complementary sources (Sentry/screens/devtools). No new AEP run. | | `/pr-validate lane <id> <pr>` | Run a single [catalog](references/evidence-catalog.md) lane by id (e.g. `lane F1`, `lane C3`, `lane D3`) — for the non-AEP lanes where you know the claim type. | -| `/pr-validate compare <pr>` | Paired A/B for a perf or refactor claim: build base + head, capture the lane on both, diff. Avoids the stale-baseline trap (catalog C5). **Per-arm treatment check first:** verify the mechanism under test is actually active in each arm (chunk split present, span emitted, flag evaluated) before interpreting deltas — a null arm without delivered treatment is a no-op, not a control (2026-07-22, #42795 bisect lesson). | +| `/pr-validate compare <pr>` | Paired A/B for a perf or refactor claim. Two arm kinds — pick by what the claim varies: **`ref`** (default) builds base + head, captures the lane on both, diffs; avoids the stale-baseline trap (catalog C5). **`substitution`** holds a **fixed head** and varies one *artifact* instead of the ref — replace the PR's hand-written type/schema/constant/policy with the authoritative equivalent and diff a checker's output (catalog D6); no build, no rebase, no merge boundary. **Per-arm checks first, one per kind:** for `ref`, verify the mechanism under test is actually active in each arm (chunk split present, span emitted, flag evaluated) — a null arm without delivered treatment is a no-op, not a control (2026-07-22, #42795 bisect lesson). For `substitution`, verify the unmodified arm is **silent** and that each diagnostic fires for the reason claimed — a noisy Arm A destroys attribution, and a diagnostic tripping one property early scores as a confirmation it isn't (trustworthiness gate item 19; 2026-07-30, #44397). | `<pr>` is a number or URL on `MetaMask/metamask-extension` unless another repo is given. Every variant runs Step 1 (extract the Claim Card) first — the claim decides the lane, even when you named one. ## Preflight -The AEP harness runs as a local stack: postgres, a temporal server, a worker, and a control plane. Bring-up steps, required Node version, registry auth, and environment are documented in the [AEP repository](https://github.com/MetaMask/metamask-autonomous-engineering-platform) itself — follow its README rather than a copy here, which drifts. Health-check first and bring up only what is down. +The hosted AEP doesn't resolve (`aep.dev.web3factory.consensys.net` is dead as of 2026-06). Everything runs locally. Health-check, then bring up only what's down. **Full procedure + every gotcha: [references/aep-local-run.md](references/aep-local-run.md).** Skim it before a first run in a session — each bullet there cost a failed run. Fast checks: ```bash +AEP=~/Code/metamask/metamask-autonomous-engineering-platform curl -fsS localhost:3000/health >/dev/null && echo "control-plane up" || echo "control-plane DOWN" curl -fsS localhost:8233 >/dev/null && echo "temporal UI up" || echo "temporal DOWN" -docker ps --format '{{.Names}}' | grep -E 'aep-postgres|aep-temporal' +docker ps --format '{{.Names}}' | grep -E 'mm-aep-postgres-dev|mm-aep-temporal-dev' ``` -If the control plane answers on `localhost:3000/health`, the stack is ready and you can skip to *Run mechanics*. +Bring-up order (each in its own shell; details + env in the reference): +1. `yarn dev:postgres` (docker `postgres:16-alpine`, `mm-aep-postgres-dev`, port 5432) +2. `yarn dev:temporal` (temporal dev server; UI on 8233) +3. `yarn db:migrate` +4. **worker** — `yarn dev:worker` on **Node ≥ 24.13**, env `ANTHROPIC_API_KEY=host-subscription`, `CLAUDE_CODE_EXECUTABLE=~/.local/bin/claude`, `GITHUB_TOKEN="$(gh auth token)"`, `SANDBOX_PROVIDER=local` (needs JFrog `npm login` first; relies on uncommitted local patches) +5. `yarn dev:control-plane` (`localhost:3000`) + +If any of the local patches (`local-sandbox-adapter.ts` timeout, `claude-agent-runner.ts` auth, the `perf-validation/` graph) are missing from the working tree, the reference says how to restore them — `git status` in the AEP repo should show them modified/untracked. ## Teardown The stack is the heaviest thing this skill starts — postgres + temporal + a Node worker + control-plane — and the worker holds a live Claude session while the autonomous run itself spends tokens. It is **on-demand, not resident**: bring it up for the validation window, **tear it down when the run(s) finish**. Left up, it's the single largest reclaimable footprint on a shared host and quietly keeps a Claude seat warm. -- **If your host wraps the stack in a service manager**, use its own down command — it stops the services and removes the `--rm` postgres/temporal containers, so state resets on the next bring-up (fine, each run is fresh anyway). -- **Otherwise:** stop the `yarn dev:*` processes and remove the postgres/temporal containers. +- **On a host managed by `aep-stack` (systemd):** `aep-stack up` to preflight, **`aep-stack down` when done** — stops the services; the `--rm` postgres/temporal containers are removed, so state resets on the next `up` (fine — each run is fresh anyway). +- **Otherwise:** stop the `yarn dev:*` processes and `docker rm -f mm-aep-postgres-dev mm-aep-temporal-dev`. - **Tear down on every exit path** — pass, refutation, *or* abort. A failed or abandoned run leaves the stack up exactly as much as a passing one; the usual leak is walking away after a refutation without stopping it. ## Run mechanics (submit → poll → fetch) @@ -159,7 +166,7 @@ curl -fsS "$CP/v1/runs/$RUN_ID/artifacts/<artifactName>" -o /tmp/<artifactName> ### Concurrent runs (multiple agents / parallel lanes) -Five shared resources need per-run isolation on one machine — collisions cross-contaminate evidence *silently* (wrong session's logs attributed to a run), which is an integrity failure, not flakiness: **(1)** CDP debug ports — derive per run, never hardcode; **(2)** e2e harness service ports (anvil/proxy/fixture/mocha) — one e2e run at a time per worktree, one worktree per agent, and never rebuild `dist/` in a worktree with an active run; **(3)** artifact dirs — per-run namespaces; `test-artifacts/` is per-worktree shared state, harvest failure artifacts before the next run overwrites the same test-title dir; **(4)** evidence-repo uploads — run-scoped paths (`pr-<n>/<run-id>/`), retry-with-fresh-sha on 409, never overwrite another run's published files; **(5)** commit-pinning — pin only after your own final upload lands, verifying your files exist at that sha. Safe to share: registry auth, a read-only `dist/`, the AEP stack itself. +Five shared resources need per-run isolation on one machine — collisions cross-contaminate evidence *silently* (wrong session's logs attributed to a run), which is an integrity failure, not flakiness: **(1)** CDP debug ports — derive per run, never hardcode; **(2)** e2e harness service ports (anvil/proxy/fixture/mocha) — one e2e run at a time per worktree, one worktree per agent (`wt new`), and never rebuild `dist/` in a worktree with an active run; **(3)** artifact dirs — per-run namespaces; `test-artifacts/` is per-worktree shared state, harvest failure artifacts before the next run overwrites the same test-title dir; **(4)** evidence-repo uploads — run-scoped paths (`pr-<n>/<run-id>/`), retry-with-fresh-sha on 409, never overwrite another run's published files; **(5)** commit-pinning — pin only after your own final upload lands, verifying your files exist at that sha. Safe to share: JFrog login, a read-only `dist/`, the AEP stack itself. ### Trust the evidence (anti-reward-hacking) @@ -167,12 +174,7 @@ A green result is not proof. The vacuous-pass trap is the floor: if `promptCraft ### perf_validation caveat -The `perf-validation/` graph writes falsifiable network/static/smoke assertions and gives the tester deterministic `.aep/` helpers (CDP netlog, phase segmentation, source-map chunk membership). Two constraints worth knowing before a perf run: - -- It requires a `yarn webpack --test` build first — the browserify `build:test` has no code splitting, so `import()` never hits the network there. -- Temporal caps activity results at ~2MB, so artifact refs must be content-free; only `evidenceBundle` carries base64. - -**Check the graph is present in your AEP checkout before relying on it.** It is newer than the visual-validation graph and may not be in every version — if it isn't registered, perf runs silently won't dispatch, and the fallback is manual DevTools/CDP capture (see the catalog). +The `perf-validation/` graph is **uncommitted local AEP work** (added 2026-06-11). It writes falsifiable network/static/smoke assertions and gives the tester deterministic `.aep/` helpers (CDP netlog, phase segmentation, source-map chunk membership). It requires a `yarn webpack --test` build first (the browserify `build:test` has no code splitting, so `import()` never hits the network there). Temporal caps activity results at ~2MB — artifact refs must be content-free; only `evidenceBundle` carries base64. If the graph isn't in the working tree, perf runs won't register — fall back to manual DevTools/CDP capture (catalog). ## Complementary evidence @@ -182,7 +184,6 @@ AEP is primary but rarely sufficient alone. Pull whatever the claim needs — ** - **B. Behavior & flow** — mm-CLI visual, E2E trace+video, **⭐ falsifying regression test** (fails on main, passes on branch — the strongest bug proof), Storybook/component, a11y, flaky-stability rerun. - **C. Performance & render** — startup/custom traces, web-vitals (**INP/FCP/LCP/CLS** via `stateHooks`), long-task **TBT** (separate observer), React render/selector (WDYR), benchmark A/B (paired), DevTools/CDP profiling, memory-over-flow, **same-window app+DevTools capture** (C8 — UI + console evidence in one frame, OS-level region recording). - **D. Build output** — bundle-size, chunk membership, **LavaMoat policy diff**, manifest permissions diff, build-variant matrix. - (Runtime containment — SES lockdown, scuttling, Snow — is **F8**, not D: D is what the build *permits*, F8 is what the running artifact *enforces*.) - **E. Production telemetry** — Sentry links (span-volume → `/sentry-quota`), Tempo traces, error-event shape. - **F. Extension integrity** — **⭐ state migration**, vault/keyring, tx simulation, provider/dapp, feature-flag matrix, snaps, i18n. - **G. CI/review/process** — check links, coverage delta, reviewer bot, manual repro. @@ -196,7 +197,7 @@ Match the bar to the claim; stop when the claim's falsifier is closed. Don't ove - **One lead lane that closes the falsifier** is enough for low-risk, single-claim PRs (a copy fix → one screenshot; a bug fix → the falsifying test). - **Weigh AEP's cost before reaching for it.** A `visual_validation`/`perf_validation` run spins the full stack *and* burns autonomous-agent tokens — by far the most expensive lane. Use it when the claim genuinely needs autonomous capture of a reachable surface; when a lighter lane closes the same falsifier (a single `mm` screenshot, a falsifying test, a CDP capture, an artifact CI already produced), prefer it and skip the stack. Whenever you do start it, tear it down after (see [Teardown](#teardown)). - **Lead + one corroborator** for perf/telemetry (a number *and* its source) and for anything user-facing that also moves a metric. **For a perf-targeting PR the lead lane is the measured impact itself** — a paired A/B benchmark at the current head (C5) or equivalent — never mechanism evidence alone (chunk membership, netlog exclusion prove the improvement is *possible*, not that it *happened*). A perf PR also always carries correctness + non-regression lanes: changed-surface tests green at head, affected flows exercised, neutral profile within noise. (2026-07-22, #42795 lesson.) -- **Lead + integrity lane** for high-stakes surfaces regardless of size: persisted-state (migration + vault), money (tx simulation), permissions (LavaMoat + manifest), runtime containment (SES lockdown / scuttling), security/keyring. Size-S doesn't lower the bar here. +- **Lead + integrity lane** for high-stakes surfaces regardless of size: persisted-state (migration + vault), money (tx simulation), permissions (LavaMoat + manifest), security/keyring. Size-S doesn't lower the bar here. - **Per-claim** for mixed PRs — each Claim Card needs its own closed falsifier; a strong UI proof doesn't cover the metric it also shifts. - **Rely on CI for routine coverage — don't re-collect what CI already establishes.** Lint, build, typecheck, the full test suite, changelog validation: CI is the authoritative source; **cite the check result** (e.g. "423 pass / 0 fail at head") instead of re-running it locally. Spend independent evidence only on (a) the claim's load-bearing falsifier, (b) specifically important/noteworthy areas (security, money, permissions, the exact changed surface), or (c) where the trust-gate warns a green result could be vacuous/misattributed. This is the economy counterpart to *"don't trust green blindly"*: that gate polices the **claim-critical** lane; this rule spares the **routine** coverage — re-collecting what CI covers is bundle noise. (#9628: cited CI's pass matrix for build/test, ran independent evidence only for the load-bearing homogeneity + resolution lanes.) @@ -207,10 +208,10 @@ Stop when each claim has one trustworthy artifact that would have shown its fals **Public, outward-facing action — always confirm the rendered section with the user before writing the PR body.** Match AEP's own format so the section is idempotent and reviewer-familiar. Full recipe (markers, image re-hosting, recordings, the `### After` injection, privacy scrub): **[references/evidence-publishing.md](references/evidence-publishing.md).** Essentials: - **Canonical header — every validation output leads with the exact literal `## 🧪 Validation Run`.** Same string in a PR comment and in the PR-body section, never reworded or demoted — the constancy is what makes it scannable/Ctrl-F-able, like Copilot's fixed `## Pull request overview`. Line 2 is the meta line: `**Verdict:** ✅ proven — **Claim:** <one-liner>` then `head \`<sha>\` · <date> · lanes: <list>`. Enforced mechanically by `hooks/pr-evidence-gate.py` (a validation/verification/evidence heading or AEP marker without the literal blocks the `gh` write). -- **Post complete, once — and know which regime the surface is in.** Comments are **push** (audience notified once at post time; edits are silent): hold until every planned lane is present or consciously dropped, and put substantive additions or changed verdicts in a **new comment referencing the original**, never a silent edit. The PR **body** is **pull** (consulted at review time): the idempotent marker upsert on re-validation at a new head is correct there. Typo-level comment edits are fine. +- **Post complete, once — and know which regime the surface is in.** Comments are **push** (audience notified once at post time; edits are silent): hold until every planned lane is present or consciously dropped, and put substantive additions or changed verdicts in a **new comment referencing the original**, never a silent edit. The PR **body** is **pull** (consulted at review time): the idempotent marker upsert on re-validation at a new head is correct there. Typo-level comment edits are fine. (Decision: `exogram-core/decisions/2026-07-23-publish-complete-bundles.md`; framework: Reprise `push-pull-artifact-edit-regimes`.) - **Falsifier-forward.** After the meta line, foreground **what would have falsified the claim and how each falsifier is closed** — the falsifier is the load-bearing content, not a footnote. Structure the body as "what would make this false → the evidence that rules it out," not a lane inventory with a `falsifiers closed` line buried at the bottom. The reviewer should see the disproof attempt first. - **Don't restate CI results.** Lint/build/typecheck/test/changelog outcomes are already on the PR's Checks tab — the reviewer sees them. Cite a CI result in the comment only to **highlight something specific** they'd otherwise miss; otherwise reference "green in Checks" or omit it. Restating "423 pass / 0 fail" is bundle noise (the display-side counterpart to the catalog's *rely on CI* collection rule). -- **Re-host images first.** Control-plane artifact URLs are `localhost` and won't render on GitHub. Re-host each artifact somewhere **your readers can reach unauthenticated**, then link the hosted URL — see [evidence-publishing.md](references/evidence-publishing.md) for the host choice and the mandatory unauthenticated `curl` check. A personal repo or a private bucket fails this for every reader but you. +- **Re-host images first.** Control-plane artifact URLs are `localhost` and won't render on GitHub. Push to the public `MajorLift/metamask-extension-skills` repo, branch `aep-evidence`, via the contents API; link the `raw.githubusercontent.com` URLs. - **Use idempotency markers** so a re-run replaces in place: wrap the whole section in `<!-- VALIDATION_RUN_START -->` … `<!-- VALIDATION_RUN_END -->`; inside it, AEP's own `<!-- AEP_VISUAL_VALIDATION_START/END -->` for the status block and `<!-- AEP_SCREENSHOTS_START/END -->` for images, injected into the PR template's `### **After**` section (replacing the `<!-- [screenshots/recordings] -->` placeholder) when present. - **Verdict-first, lanes nested:** under the canonical header, hand-assembled AEP blocks demote to `### AEP Visual Validation` (leave AEP's own service-published `##` blocks untouched) with `**✅ Passed**` / `**❌ Failed**` / `ℹ️`, the long narrative in `<details><summary>Validation details</summary>`, a meta line `Run \`<id>\` · [LangSmith trace](…)`. - **Scrub** local paths and your username from any narrative before publishing — failure summaries leak them. @@ -258,7 +259,7 @@ PR claims privacy mode now hides the Perps balance (the demo bug #42683): 2. Lane = `visual_validation` (visible). Preflight stack. 3. Submit with `description: "Onboard, enable privacy mode in Settings, open the Perps tab, confirm the balance is masked. If the Perps tutorial modal blocks, use the Shield entry modal as the reachable surface."` + `publishEvidence:false`. 4. Poll to completion; assert `artifactRefs` has the before/after pair (not a vacuous skip). -5. Fetch the two PNGs; re-host them to your configured evidence host; assemble the `AEP_VISUAL_VALIDATION` section with the hosted URLs injected into the template's `### After`. +5. Fetch the two PNGs; re-host to `aep-evidence`; assemble the `AEP_VISUAL_VALIDATION` section with the `raw.githubusercontent` URLs injected into the template's `### After`. 6. Show the rendered section; on confirm, upsert the PR body. End-to-end examples for **non-visual** claims (perf, migration, flag-gated, refactor/no-op): **[references/worked-examples.md](references/worked-examples.md).** @@ -269,7 +270,7 @@ Three adjacent things; keep the boundary clear so they compose instead of collid - **AEP** — governed *fleet orchestration*: sandboxes, Temporal, autonomous runs at scale. The heavy engine. - **ADR-0058 recipes** ([decisions#173](https://github.com/MetaMask/decisions/pull/173)) — a *dev-machine inner-loop* proof artifact: a declarative per-PR recipe run against the live app over CDP, emitting `summary.json`/`trace.json`/manifest. -- **pr-validate** (this skill) — the *claim→evidence methodology + taxonomy* both draw on. The Claim Card is the bridge from a PR's claim to the right proof target; the [evidence catalog](references/evidence-catalog.md) is the lane vocabulary; [lane-assertions.md](references/lane-assertions.md) maps each lane to a recipe assertion (and flags the out-of-band, non-UI lanes — the gap raised in review on decisions#173). +- **pr-validate** (this skill) — the *claim→evidence methodology + taxonomy* both draw on. The Claim Card is the bridge from a PR's claim to the right proof target; the [evidence catalog](references/evidence-catalog.md) is the lane vocabulary; [lane-assertions.md](references/lane-assertions.md) maps each lane to a recipe assertion (and flags the out-of-band, non-UI lanes — the gap MajorLift's #173 review raised). pr-validate is the one a human drives; it can dispatch an AEP run or author a recipe as its capture step. @@ -287,7 +288,7 @@ Where pr-validate sits in the PR lifecycle (see the public `pr-workflow` sibling - **Executes, with a confirmation gate on publish.** It runs the harness and captures evidence autonomously; it does not write to the public PR body without showing you the section first. - **Local-only AEP.** No hosted instance. The skill drives the local stack. - **Proves behavior, not code.** Pair with `/review` / `/code-review` for correctness and `/sentry-quota` for span-volume risk. -- **No persisted state.** Each run is fresh. To keep a validation record, ask — nothing is written by default. +- **No persisted state.** Each run is fresh. To keep a validation record, ask — it can go to `exogram-daemon/`, but nothing writes by default. ## Related @@ -297,9 +298,24 @@ Where pr-validate sits in the PR lifecycle (see the public `pr-workflow` sibling - [references/evidence-publishing.md](references/evidence-publishing.md) — PR-body format, non-visual/multi-lane rendering, image re-hosting, recordings→GIF, privacy scrub, ADR-0058 artifact contract. - [references/worked-examples.md](references/worked-examples.md) — end-to-end runs for perf / migration / flag-gated / refactor claims. - [references/lane-assertions.md](references/lane-assertions.md) — lane → declarative recipe-assertion mapping (ADR-0058 bridge). -- [MetaMask/metamask-autonomous-engineering-platform](https://github.com/MetaMask/metamask-autonomous-engineering-platform) — the AEP repo: stack bring-up in its README, plus `docs/demo-runbook.md`, `packages/agent-chain/src/graphs/{visual,perf}-validation/`, and `packages/github/src/pr-body-builder.ts` (the canonical PR-body format this skill mirrors). +- [references/aep-local-run.md](references/aep-local-run.md) — full local-stack bring-up + every gotcha. +- `~/Code/metamask/metamask-autonomous-engineering-platform` — the AEP repo (`docs/demo-runbook.md`, `packages/agent-chain/src/graphs/{visual,perf}-validation/`, `packages/github/src/pr-body-builder.ts`). - `MetaMask/decisions#173` — ADR-0058 Recipe-Based Verification (the adjacent inner-loop proof system). - `/sentry-quota` — sibling skill for span-volume PR review; `/review`, `/code-review` — code correctness. -- `/memory-leak-hunt` — the engine behind the **memory leak** evidence category (C9). pr-validate delegates retention analysis to it and packages the verdict; it also runs standalone. +- **Engine skills — delegate the analysis, package the result.** Each owns a category in + [references/evidence-catalog.md](references/evidence-catalog.md); all run standalone too. + + | category | engine | + |---|---| + | B3 falsifying regression test | `/falsifying-test` | + | B7 deterministic interleaving (concurrency / ordering) | `/race-condition-proof` | + | C4 React render & selector proof | `/react-render-proof` | + | C9 memory leak | `/memory-leak-hunt` | + | D supply-chain / dependency change | `/supply-chain-audit` → delegates capability grants to `/lavamoat-policy-diligence` | + + **An engine that defines its own output contract publishes in it.** `lavamoat-policy-diligence` + is the live case: read-level triage, no verdict, its own header and marker pair. Do not + re-frame it as a Validation Run — see *One comment per evidence kind* in + [references/evidence-publishing.md](references/evidence-publishing.md). - [[reference_aep_local_run]] — the source memory this skill encodes. - [[reference_sentry_project_topology]] — Sentry project mapping for the telemetry-evidence lane. From 36ea9c8f48a5a725de830133c7b08963b4239aca Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 16:53:30 -0400 Subject: [PATCH 05/63] Move the AEP run procedure behind a reference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `pr-validate` cost ~9,059 tokens the moment an agent selected it — 5.6× the median of the 26 skills in the open PRs — and a quarter of that was the AEP local-run procedure, which most validations never touch. The skill's own Sufficiency section tells you to prefer a lighter lane; the body charged you for the heavy one regardless. Preflight, run mechanics, and teardown move to `references/aep-local-run.md`, which the body already linked twice and which did not exist. The link was dangling — the same defect class the `knowledge/` guard catches, on a path nothing checks. Publishing keeps the decisions (surface by ownership, post complete once, falsifier-forward, scrub) and points at `references/evidence-publishing.md` for the mechanics it already documents in full. Body 34,431 → 25,736 bytes, so a selected skill is ~6,812 tokens installed rather than ~9,059. Nothing is lost: it sits behind the same progressive disclosure boundary as the other seven references, read when an AEP run is actually warranted. Description trimmed 1,147 → 885 characters. It was over the 1,024 ceiling that #47 enforces, so it would have failed that check on merge. --- .../pr-validate/references/aep-local-run.md | 81 +++++++++++++ .../pr-workflow/skills/pr-validate/skill.md | 108 +++++------------- 2 files changed, 108 insertions(+), 81 deletions(-) create mode 100644 domains/pr-workflow/skills/pr-validate/references/aep-local-run.md diff --git a/domains/pr-workflow/skills/pr-validate/references/aep-local-run.md b/domains/pr-workflow/skills/pr-validate/references/aep-local-run.md new file mode 100644 index 00000000..cccb6b6e --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/aep-local-run.md @@ -0,0 +1,81 @@ +# Running AEP locally + +Everything the local Autonomous Engineering Platform run needs: bring-up, submit, poll, +fetch artifacts, tear down. The hosted instance (`aep.dev.web3factory.consensys.net`) has +not resolved since 2026-06, so local is the only path. + +The skill body links here rather than carrying this inline. An AEP run is the heaviest +lane in the catalog and most validations do not need it — a falsifying test, a single +screenshot, or an artifact CI already produced usually closes the same falsifier. Read +this when you have decided an AEP run is warranted. + +Every bullet below cost a failed run at least once. + +## Preflight — bring up what is down + +The hosted AEP doesn't resolve (`aep.dev.web3factory.consensys.net` is dead as of 2026-06). Everything runs locally. Health-check, then bring up only what's down. + +Fast checks: + +```bash +AEP=~/Code/metamask/metamask-autonomous-engineering-platform +curl -fsS localhost:3000/health >/dev/null && echo "control-plane up" || echo "control-plane DOWN" +curl -fsS localhost:8233 >/dev/null && echo "temporal UI up" || echo "temporal DOWN" +docker ps --format '{{.Names}}' | grep -E 'mm-aep-postgres-dev|mm-aep-temporal-dev' +``` + +Bring-up order (each in its own shell; details + env in the reference): +1. `yarn dev:postgres` (docker `postgres:16-alpine`, `mm-aep-postgres-dev`, port 5432) +2. `yarn dev:temporal` (temporal dev server; UI on 8233) +3. `yarn db:migrate` +4. **worker** — `yarn dev:worker` on **Node ≥ 24.13**, env `ANTHROPIC_API_KEY=host-subscription`, `CLAUDE_CODE_EXECUTABLE=~/.local/bin/claude`, `GITHUB_TOKEN="$(gh auth token)"`, `SANDBOX_PROVIDER=local` (needs JFrog `npm login` first; relies on uncommitted local patches) +5. `yarn dev:control-plane` (`localhost:3000`) + +If any of the local patches (`local-sandbox-adapter.ts` timeout, `claude-agent-runner.ts` auth, the `perf-validation/` graph) are missing from the working tree, the reference says how to restore them — `git status` in the AEP repo should show them modified/untracked. +## Run mechanics — submit, poll, fetch + +The control-plane is a thin REST shell. Submit a PR-validation task, poll the run, pull artifacts from the evidence bundle. + +```bash +CP=localhost:3000 +PR="https://github.com/MetaMask/metamask-extension/pull/<n>" + +# Submit (publishEvidence:false ALWAYS for local runs — the platform otherwise +# writes to the public PR body even on failure, leaking local paths/usernames) +RUN_ID=$(curl -fsS -X POST "$CP/v1/tasks" -H 'content-type: application/json' -d '{ + "repo": "MetaMask/metamask-extension", + "title": "Visual validation — PR #<n>", + "taskClass": "visual_validation", + "externalRef": "'"$PR"'", + "payload": { "prUrl": "'"$PR"'", "description": "<targeting hint>", "publishEvidence": false } +}' | node -e 'process.stdin.on("data",d=>console.log(JSON.parse(d).runId||JSON.parse(d).id))') + +# Poll +curl -fsS "$CP/v1/runs/$RUN_ID" | node -e 'const r=JSON.parse(require("fs").readFileSync(0));console.log(r.status); (r.evidenceBundle?.artifactRefs||[]).forEach(a=>console.log(a.name,a.mediaType))' + +# Fetch an artifact +curl -fsS "$CP/v1/runs/$RUN_ID/artifacts/<artifactName>" -o /tmp/<artifactName> +``` + +- `taskClass`: `visual_validation` or `perf_validation`. The worker auto-enriches the payload from `prUrl` (pulls headSha, base, diff, files, linked issues via the GitHub app) — you only supply `prUrl` + a `description` targeting hint. +- The **targeting hint** (`payload.description`) is how you steer the agent to the surface under test. Be specific: which screen, which control, what to toggle. For hard-to-reach surfaces, name the reachable fallback (e.g. the Shield entry modal stands in for the Perps tutorial modal, which is gated in the default fixture). +- Artifact regex allows **png/jpg/log/txt only** — no video. Screen recordings need the side-channel recipe (catalog + publishing reference). + +### Concurrent runs (multiple agents / parallel lanes) + +Five shared resources need per-run isolation on one machine — collisions cross-contaminate evidence *silently* (wrong session's logs attributed to a run), which is an integrity failure, not flakiness: **(1)** CDP debug ports — derive per run, never hardcode; **(2)** e2e harness service ports (anvil/proxy/fixture/mocha) — one e2e run at a time per worktree, one worktree per agent (`wt new`), and never rebuild `dist/` in a worktree with an active run; **(3)** artifact dirs — per-run namespaces; `test-artifacts/` is per-worktree shared state, harvest failure artifacts before the next run overwrites the same test-title dir; **(4)** evidence-repo uploads — run-scoped paths (`pr-<n>/<run-id>/`), retry-with-fresh-sha on 409, never overwrite another run's published files; **(5)** commit-pinning — pin only after your own final upload lands, verifying your files exist at that sha. Safe to share: JFrog login, a read-only `dist/`, the AEP stack itself. + +### Trust the evidence (anti-reward-hacking) + +A green result is not proof. The vacuous-pass trap is the floor: if `promptCrafter` errors, the chain "passes" via skip with **zero artifacts** — a pass is only real if `evidenceBundle.artifactRefs` is non-empty with the expected media. Beyond that, every lane must clear a trustworthiness gate before you believe or publish it: **does the artifact show the *claimed* surface** (not a spinner/wrong screen), **does the test exercise the *changed* code** (fails on `main`), **does the signal exceed noise**, **could the assertion have failed**? The Claim Card's Falsifier is the anchor. Full gate + per-lane traps: **[references/evidence-trustworthiness.md](references/evidence-trustworthiness.md).** + +### perf_validation caveat + +The `perf-validation/` graph is **uncommitted local AEP work** (added 2026-06-11). It writes falsifiable network/static/smoke assertions and gives the tester deterministic `.aep/` helpers (CDP netlog, phase segmentation, source-map chunk membership). It requires a `yarn webpack --test` build first (the browserify `build:test` has no code splitting, so `import()` never hits the network there). Temporal caps activity results at ~2MB — artifact refs must be content-free; only `evidenceBundle` carries base64. If the graph isn't in the working tree, perf runs won't register — fall back to manual DevTools/CDP capture (catalog). +## Teardown — always, on every exit path + +The stack is the heaviest thing this skill starts — postgres + temporal + a Node worker + control-plane — and the worker holds a live Claude session while the autonomous run itself spends tokens. It is **on-demand, not resident**: bring it up for the validation window, **tear it down when the run(s) finish**. Left up, it's the single largest reclaimable footprint on a shared host and quietly keeps a Claude seat warm. + +- **On a host managed by `aep-stack` (systemd):** `aep-stack up` to preflight, **`aep-stack down` when done** — stops the services; the `--rm` postgres/temporal containers are removed, so state resets on the next `up` (fine — each run is fresh anyway). +- **Otherwise:** stop the `yarn dev:*` processes and `docker rm -f mm-aep-postgres-dev mm-aep-temporal-dev`. +- **Tear down on every exit path** — pass, refutation, *or* abort. A failed or abandoned run leaves the stack up exactly as much as a passing one; the usual leak is walking away after a refutation without stopping it. diff --git a/domains/pr-workflow/skills/pr-validate/skill.md b/domains/pr-workflow/skills/pr-validate/skill.md index 660ef602..b2ae4b83 100644 --- a/domains/pr-workflow/skills/pr-validate/skill.md +++ b/domains/pr-workflow/skills/pr-validate/skill.md @@ -1,6 +1,6 @@ --- name: pr-validate -description: Validate a MetaMask PR with objective evidence — primarily the Autonomous Engineering Platform (AEP) harness (visual_validation for visible UI behavior, perf_validation for non-visible perf behavior), backed by complementary evidence (Sentry query links, screenshots, screen recordings→GIF, DevTools/CDP output, bundle-size, web-vitals, test/CI results). Drives the AEP local stack end to end: preflight (postgres + temporal + worker + control-plane) → submit POST /v1/tasks → poll GET /v1/runs/:id → fetch artifacts → assemble an evidence bundle → publish to the PR body (re-hosting images to a public repo, scrubbing local paths). Match evidence to the PR's specific falsifiable claim, not a fixed checklist. Triggers on /pr-validate, /pr-validate visual, /pr-validate perf, /pr-validate preflight, /pr-validate status, /pr-validate evidence, /pr-validate plan, or when the user mentions validating/proving a PR, AEP / visual validation / perf validation, capturing evidence for a PR, before/after screenshots, a screen recording or GIF for a PR, attaching Sentry links or DevTools output as proof, or publishing an evidence bundle to a PR body. +description: Validate a MetaMask PR with objective evidence — match the evidence to the PR's specific falsifiable claim rather than running a fixed checklist. Covers the full catalog: before/after screenshots, falsifying regression tests, perf and render proofs, bundle and LavaMoat diffs, Sentry and Tempo links, state-migration and vault checks, plus the Autonomous Engineering Platform (AEP) harness for autonomous visual and perf capture. Assembles an evidence bundle and publishes it to the PR body, images re-hosted and local paths scrubbed. Triggers on /pr-validate and its subcommands (visual, perf, preflight, status, evidence, plan, lane, compare), or when the user mentions validating or proving a PR, AEP or visual/perf validation, capturing evidence, before/after screenshots, a screen recording for a PR, attaching Sentry or DevTools output as proof, or publishing an evidence bundle. --- # /pr-validate @@ -93,7 +93,7 @@ Not for code-correctness review (use `/review`, `/code-review`) or span-quota re | Invocation | Behavior | |---|---| -| `/pr-validate <pr>` | **Flagship.** Read the PR → state the claim → pick lanes → [preflight](#preflight) → run AEP lane(s) + gather complementary evidence → assemble bundle → **propose** the PR-body section and confirm before publishing. | +| `/pr-validate <pr>` | **Flagship.** Read the PR → state the claim → pick lanes → [preflight](references/aep-local-run.md) → run AEP lane(s) + gather complementary evidence → assemble bundle → **propose** the PR-body section and confirm before publishing. | | `/pr-validate plan <pr>` | Dry run: read the PR, state the claim, recommend lanes + targeting hints. No stack, no run. Cheap first step when unsure. | | `/pr-validate visual <pr>` | AEP `visual_validation` only. | | `/pr-validate perf <pr>` | AEP `perf_validation` only (local/uncommitted graph — see [caveat](#perf_validation-caveat)). | @@ -105,76 +105,15 @@ Not for code-correctness review (use `/review`, `/code-review`) or span-quota re `<pr>` is a number or URL on `MetaMask/metamask-extension` unless another repo is given. Every variant runs Step 1 (extract the Claim Card) first — the claim decides the lane, even when you named one. -## Preflight +## Running AEP -The hosted AEP doesn't resolve (`aep.dev.web3factory.consensys.net` is dead as of 2026-06). Everything runs locally. Health-check, then bring up only what's down. **Full procedure + every gotcha: [references/aep-local-run.md](references/aep-local-run.md).** Skim it before a first run in a session — each bullet there cost a failed run. +The hosted instance is dead; everything runs locally. Bring-up, submit/poll/fetch, and +teardown are in **[references/aep-local-run.md](references/aep-local-run.md)** — read it +once you have decided an AEP run is warranted, not before. -Fast checks: - -```bash -AEP=~/Code/metamask/metamask-autonomous-engineering-platform -curl -fsS localhost:3000/health >/dev/null && echo "control-plane up" || echo "control-plane DOWN" -curl -fsS localhost:8233 >/dev/null && echo "temporal UI up" || echo "temporal DOWN" -docker ps --format '{{.Names}}' | grep -E 'mm-aep-postgres-dev|mm-aep-temporal-dev' -``` - -Bring-up order (each in its own shell; details + env in the reference): -1. `yarn dev:postgres` (docker `postgres:16-alpine`, `mm-aep-postgres-dev`, port 5432) -2. `yarn dev:temporal` (temporal dev server; UI on 8233) -3. `yarn db:migrate` -4. **worker** — `yarn dev:worker` on **Node ≥ 24.13**, env `ANTHROPIC_API_KEY=host-subscription`, `CLAUDE_CODE_EXECUTABLE=~/.local/bin/claude`, `GITHUB_TOKEN="$(gh auth token)"`, `SANDBOX_PROVIDER=local` (needs JFrog `npm login` first; relies on uncommitted local patches) -5. `yarn dev:control-plane` (`localhost:3000`) - -If any of the local patches (`local-sandbox-adapter.ts` timeout, `claude-agent-runner.ts` auth, the `perf-validation/` graph) are missing from the working tree, the reference says how to restore them — `git status` in the AEP repo should show them modified/untracked. - -## Teardown - -The stack is the heaviest thing this skill starts — postgres + temporal + a Node worker + control-plane — and the worker holds a live Claude session while the autonomous run itself spends tokens. It is **on-demand, not resident**: bring it up for the validation window, **tear it down when the run(s) finish**. Left up, it's the single largest reclaimable footprint on a shared host and quietly keeps a Claude seat warm. - -- **On a host managed by `aep-stack` (systemd):** `aep-stack up` to preflight, **`aep-stack down` when done** — stops the services; the `--rm` postgres/temporal containers are removed, so state resets on the next `up` (fine — each run is fresh anyway). -- **Otherwise:** stop the `yarn dev:*` processes and `docker rm -f mm-aep-postgres-dev mm-aep-temporal-dev`. -- **Tear down on every exit path** — pass, refutation, *or* abort. A failed or abandoned run leaves the stack up exactly as much as a passing one; the usual leak is walking away after a refutation without stopping it. - -## Run mechanics (submit → poll → fetch) - -The control-plane is a thin REST shell. Submit a PR-validation task, poll the run, pull artifacts from the evidence bundle. - -```bash -CP=localhost:3000 -PR="https://github.com/MetaMask/metamask-extension/pull/<n>" - -# Submit (publishEvidence:false ALWAYS for local runs — the platform otherwise -# writes to the public PR body even on failure, leaking local paths/usernames) -RUN_ID=$(curl -fsS -X POST "$CP/v1/tasks" -H 'content-type: application/json' -d '{ - "repo": "MetaMask/metamask-extension", - "title": "Visual validation — PR #<n>", - "taskClass": "visual_validation", - "externalRef": "'"$PR"'", - "payload": { "prUrl": "'"$PR"'", "description": "<targeting hint>", "publishEvidence": false } -}' | node -e 'process.stdin.on("data",d=>console.log(JSON.parse(d).runId||JSON.parse(d).id))') - -# Poll -curl -fsS "$CP/v1/runs/$RUN_ID" | node -e 'const r=JSON.parse(require("fs").readFileSync(0));console.log(r.status); (r.evidenceBundle?.artifactRefs||[]).forEach(a=>console.log(a.name,a.mediaType))' - -# Fetch an artifact -curl -fsS "$CP/v1/runs/$RUN_ID/artifacts/<artifactName>" -o /tmp/<artifactName> -``` - -- `taskClass`: `visual_validation` or `perf_validation`. The worker auto-enriches the payload from `prUrl` (pulls headSha, base, diff, files, linked issues via the GitHub app) — you only supply `prUrl` + a `description` targeting hint. -- The **targeting hint** (`payload.description`) is how you steer the agent to the surface under test. Be specific: which screen, which control, what to toggle. For hard-to-reach surfaces, name the reachable fallback (e.g. the Shield entry modal stands in for the Perps tutorial modal, which is gated in the default fixture). -- Artifact regex allows **png/jpg/log/txt only** — no video. Screen recordings need the side-channel recipe (catalog + publishing reference). - -### Concurrent runs (multiple agents / parallel lanes) - -Five shared resources need per-run isolation on one machine — collisions cross-contaminate evidence *silently* (wrong session's logs attributed to a run), which is an integrity failure, not flakiness: **(1)** CDP debug ports — derive per run, never hardcode; **(2)** e2e harness service ports (anvil/proxy/fixture/mocha) — one e2e run at a time per worktree, one worktree per agent (`wt new`), and never rebuild `dist/` in a worktree with an active run; **(3)** artifact dirs — per-run namespaces; `test-artifacts/` is per-worktree shared state, harvest failure artifacts before the next run overwrites the same test-title dir; **(4)** evidence-repo uploads — run-scoped paths (`pr-<n>/<run-id>/`), retry-with-fresh-sha on 409, never overwrite another run's published files; **(5)** commit-pinning — pin only after your own final upload lands, verifying your files exist at that sha. Safe to share: JFrog login, a read-only `dist/`, the AEP stack itself. - -### Trust the evidence (anti-reward-hacking) - -A green result is not proof. The vacuous-pass trap is the floor: if `promptCrafter` errors, the chain "passes" via skip with **zero artifacts** — a pass is only real if `evidenceBundle.artifactRefs` is non-empty with the expected media. Beyond that, every lane must clear a trustworthiness gate before you believe or publish it: **does the artifact show the *claimed* surface** (not a spinner/wrong screen), **does the test exercise the *changed* code** (fails on `main`), **does the signal exceed noise**, **could the assertion have failed**? The Claim Card's Falsifier is the anchor. Full gate + per-lane traps: **[references/evidence-trustworthiness.md](references/evidence-trustworthiness.md).** - -### perf_validation caveat - -The `perf-validation/` graph is **uncommitted local AEP work** (added 2026-06-11). It writes falsifiable network/static/smoke assertions and gives the tester deterministic `.aep/` helpers (CDP netlog, phase segmentation, source-map chunk membership). It requires a `yarn webpack --test` build first (the browserify `build:test` has no code splitting, so `import()` never hits the network there). Temporal caps activity results at ~2MB — artifact refs must be content-free; only `evidenceBundle` carries base64. If the graph isn't in the working tree, perf runs won't register — fall back to manual DevTools/CDP capture (catalog). +It is the heaviest lane here: the full stack plus autonomous-agent tokens. Weigh that +against a lighter lane that closes the same falsifier (see [Sufficiency](#sufficiency--how-much-is-enough)), +and **tear the stack down on every exit path** — pass, refutation, or abort. ## Complementary evidence @@ -195,7 +134,7 @@ Screen recordings (motion a still can't prove): `mm` + a Playwright `recordVideo Match the bar to the claim; stop when the claim's falsifier is closed. Don't over-instrument a copy fix; don't under-prove a high-stakes claim. - **One lead lane that closes the falsifier** is enough for low-risk, single-claim PRs (a copy fix → one screenshot; a bug fix → the falsifying test). -- **Weigh AEP's cost before reaching for it.** A `visual_validation`/`perf_validation` run spins the full stack *and* burns autonomous-agent tokens — by far the most expensive lane. Use it when the claim genuinely needs autonomous capture of a reachable surface; when a lighter lane closes the same falsifier (a single `mm` screenshot, a falsifying test, a CDP capture, an artifact CI already produced), prefer it and skip the stack. Whenever you do start it, tear it down after (see [Teardown](#teardown)). +- **Weigh AEP's cost before reaching for it.** A `visual_validation`/`perf_validation` run spins the full stack *and* burns autonomous-agent tokens — by far the most expensive lane. Use it when the claim genuinely needs autonomous capture of a reachable surface; when a lighter lane closes the same falsifier (a single `mm` screenshot, a falsifying test, a CDP capture, an artifact CI already produced), prefer it and skip the stack. Whenever you do start it, tear it down after (see [references/aep-local-run.md](references/aep-local-run.md)). - **Lead + one corroborator** for perf/telemetry (a number *and* its source) and for anything user-facing that also moves a metric. **For a perf-targeting PR the lead lane is the measured impact itself** — a paired A/B benchmark at the current head (C5) or equivalent — never mechanism evidence alone (chunk membership, netlog exclusion prove the improvement is *possible*, not that it *happened*). A perf PR also always carries correctness + non-regression lanes: changed-surface tests green at head, affected flows exercised, neutral profile within noise. (2026-07-22, #42795 lesson.) - **Lead + integrity lane** for high-stakes surfaces regardless of size: persisted-state (migration + vault), money (tx simulation), permissions (LavaMoat + manifest), security/keyring. Size-S doesn't lower the bar here. - **Per-claim** for mixed PRs — each Claim Card needs its own closed falsifier; a strong UI proof doesn't cover the metric it also shifts. @@ -205,16 +144,23 @@ Stop when each claim has one trustworthy artifact that would have shown its fals ## Publishing the evidence bundle -**Public, outward-facing action — always confirm the rendered section with the user before writing the PR body.** Match AEP's own format so the section is idempotent and reviewer-familiar. Full recipe (markers, image re-hosting, recordings, the `### After` injection, privacy scrub): **[references/evidence-publishing.md](references/evidence-publishing.md).** Essentials: - -- **Canonical header — every validation output leads with the exact literal `## 🧪 Validation Run`.** Same string in a PR comment and in the PR-body section, never reworded or demoted — the constancy is what makes it scannable/Ctrl-F-able, like Copilot's fixed `## Pull request overview`. Line 2 is the meta line: `**Verdict:** ✅ proven — **Claim:** <one-liner>` then `head \`<sha>\` · <date> · lanes: <list>`. Enforced mechanically by `hooks/pr-evidence-gate.py` (a validation/verification/evidence heading or AEP marker without the literal blocks the `gh` write). -- **Post complete, once — and know which regime the surface is in.** Comments are **push** (audience notified once at post time; edits are silent): hold until every planned lane is present or consciously dropped, and put substantive additions or changed verdicts in a **new comment referencing the original**, never a silent edit. The PR **body** is **pull** (consulted at review time): the idempotent marker upsert on re-validation at a new head is correct there. Typo-level comment edits are fine. (Decision: `exogram-core/decisions/2026-07-23-publish-complete-bundles.md`; framework: Reprise `push-pull-artifact-edit-regimes`.) -- **Falsifier-forward.** After the meta line, foreground **what would have falsified the claim and how each falsifier is closed** — the falsifier is the load-bearing content, not a footnote. Structure the body as "what would make this false → the evidence that rules it out," not a lane inventory with a `falsifiers closed` line buried at the bottom. The reviewer should see the disproof attempt first. -- **Don't restate CI results.** Lint/build/typecheck/test/changelog outcomes are already on the PR's Checks tab — the reviewer sees them. Cite a CI result in the comment only to **highlight something specific** they'd otherwise miss; otherwise reference "green in Checks" or omit it. Restating "423 pass / 0 fail" is bundle noise (the display-side counterpart to the catalog's *rely on CI* collection rule). -- **Re-host images first.** Control-plane artifact URLs are `localhost` and won't render on GitHub. Push to the public `MajorLift/metamask-extension-skills` repo, branch `aep-evidence`, via the contents API; link the `raw.githubusercontent.com` URLs. -- **Use idempotency markers** so a re-run replaces in place: wrap the whole section in `<!-- VALIDATION_RUN_START -->` … `<!-- VALIDATION_RUN_END -->`; inside it, AEP's own `<!-- AEP_VISUAL_VALIDATION_START/END -->` for the status block and `<!-- AEP_SCREENSHOTS_START/END -->` for images, injected into the PR template's `### **After**` section (replacing the `<!-- [screenshots/recordings] -->` placeholder) when present. -- **Verdict-first, lanes nested:** under the canonical header, hand-assembled AEP blocks demote to `### AEP Visual Validation` (leave AEP's own service-published `##` blocks untouched) with `**✅ Passed**` / `**❌ Failed**` / `ℹ️`, the long narrative in `<details><summary>Validation details</summary>`, a meta line `Run \`<id>\` · [LangSmith trace](…)`. -- **Scrub** local paths and your username from any narrative before publishing — failure summaries leak them. +**Public, outward-facing — always confirm the rendered section with the user before writing +a PR body.** Full recipe, markers, image re-hosting, recordings, and the privacy scrub: +**[references/evidence-publishing.md](references/evidence-publishing.md).** + +The parts that decide *whether* to publish, rather than how: + +- **Surface follows ownership** — the PR body when you authored it, a comment when validating + someone else's. Never publish a failure to another author's PR unprompted. +- **Post complete, once.** A comment is push: its audience is notified at post time and edits + are silent, so hold until every planned lane is present, and put changed verdicts in a new + comment referencing the original. The PR body is pull, so an idempotent marker upsert is + correct there. +- **Lead with the canonical header** `## 🧪 Validation Run`, then verdict and claim. +- **Falsifier-forward** — what would have made this false, and what rules it out, before any + lane inventory. +- **Don't restate CI.** Lint, build, and test results are already on the Checks tab. +- **Scrub** local paths and usernames; failure summaries leak them. ## Validation output format From 2af1db1e11586ee2f0c85a67380382d4edc76852 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 03:25:01 -0400 Subject: [PATCH 06/63] Restore the full `pr-validate` description MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Trimmed to 885 characters to fit a 1,024 budget that turned out to be unverified — no operator observed rejects or truncates a longer description, and several over 1,024 load today. Back to 1,147, under the 1,536 budget. The description is the discovery surface, so the 262 characters were trigger cues: the subcommand list and the phrasings that route a request here rather than to another skill. Cutting them made the skill harder to select, which is a functional loss and not a cosmetic one. The body restructure is unaffected — that removed duplication behind a reference, which costs nothing at selection time. From 0b4bb26445ddabd54d8215c428921f113c5629d1 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 04:28:32 -0400 Subject: [PATCH 07/63] Actually restore the `pr-validate` description MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 2af1db1 carries the reasoning but not the change: its `git add` was chained into a command the commit guard rejected, so the commit was created from an already staged tree and landed empty — same 885-character description as its parent. This applies it. Back to 1,147 characters, under the 1,536 budget. The 262 characters are trigger cues — the subcommand list and the phrasings that route a request here rather than to a sibling skill — so losing them made the skill harder to select, which is a functional loss rather than a cosmetic one. --- domains/pr-workflow/skills/pr-validate/skill.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/pr-validate/skill.md b/domains/pr-workflow/skills/pr-validate/skill.md index b2ae4b83..351667db 100644 --- a/domains/pr-workflow/skills/pr-validate/skill.md +++ b/domains/pr-workflow/skills/pr-validate/skill.md @@ -1,6 +1,6 @@ --- name: pr-validate -description: Validate a MetaMask PR with objective evidence — match the evidence to the PR's specific falsifiable claim rather than running a fixed checklist. Covers the full catalog: before/after screenshots, falsifying regression tests, perf and render proofs, bundle and LavaMoat diffs, Sentry and Tempo links, state-migration and vault checks, plus the Autonomous Engineering Platform (AEP) harness for autonomous visual and perf capture. Assembles an evidence bundle and publishes it to the PR body, images re-hosted and local paths scrubbed. Triggers on /pr-validate and its subcommands (visual, perf, preflight, status, evidence, plan, lane, compare), or when the user mentions validating or proving a PR, AEP or visual/perf validation, capturing evidence, before/after screenshots, a screen recording for a PR, attaching Sentry or DevTools output as proof, or publishing an evidence bundle. +description: Validate a MetaMask PR with objective evidence — primarily the Autonomous Engineering Platform (AEP) harness (visual_validation for visible UI behavior, perf_validation for non-visible perf behavior), backed by complementary evidence (Sentry query links, screenshots, screen recordings→GIF, DevTools/CDP output, bundle-size, web-vitals, test/CI results). Drives the AEP local stack end to end: preflight (postgres + temporal + worker + control-plane) → submit POST /v1/tasks → poll GET /v1/runs/:id → fetch artifacts → assemble an evidence bundle → publish to the PR body (re-hosting images to a public repo, scrubbing local paths). Match evidence to the PR's specific falsifiable claim, not a fixed checklist. Triggers on /pr-validate, /pr-validate visual, /pr-validate perf, /pr-validate preflight, /pr-validate status, /pr-validate evidence, /pr-validate plan, or when the user mentions validating/proving a PR, AEP / visual validation / perf validation, capturing evidence for a PR, before/after screenshots, a screen recording or GIF for a PR, attaching Sentry links or DevTools output as proof, or publishing an evidence bundle to a PR body. --- # /pr-validate From 704128a61661a0af629beb0e3a6210a4a656f596 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 08:22:30 -0400 Subject: [PATCH 08/63] Rename `pr-validate` to `evidence` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two of the three assumptions in the old name were false. The skill runs in the inner loop against uncommitted changes, and on a symptom with no claim and no PR at all, so `pr-` narrowed it to one of three modes. `validate` named an activity where the deliverable is an artifact, and read as a checklist exercise — the posture the skill spends its opening section arguing against. `evidence` names what it produces, and covers a refutation as naturally as a confirmation. The verdicts are proven, refuted, and inconclusive; a name promising proof would make two of those read as failure. The description is rewritten rather than search-replaced. It now states all three modes, since the old one described only the PR case and so under-selected for the other two, and it names the trigger as `mms-evidence` — the form the installer actually emits. Eleven skills across the repo still promise the unprefixed `/<name>` in their descriptions while installing prefixed; this corrects the one being renamed. --- .../hooks/pr-evidence-gate.py | 0 .../references/aep-local-run.md | 0 .../references/claim-extraction.md | 2 +- .../references/evidence-catalog.md | 6 ++-- .../references/evidence-gate-setup.md | 8 ++--- .../references/evidence-publishing.md | 10 +++--- .../references/evidence-trustworthiness.md | 2 +- .../references/lane-assertions.md | 0 .../references/worked-examples.md | 0 .../skills/{pr-validate => evidence}/skill.md | 34 +++++++++---------- .../testing/skills/falsifying-test/skill.md | 4 +-- 11 files changed, 33 insertions(+), 33 deletions(-) rename domains/pr-workflow/skills/{pr-validate => evidence}/hooks/pr-evidence-gate.py (100%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/aep-local-run.md (100%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/claim-extraction.md (94%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/evidence-catalog.md (98%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/evidence-gate-setup.md (87%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/evidence-publishing.md (97%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/evidence-trustworthiness.md (90%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/lane-assertions.md (100%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/worked-examples.md (100%) rename domains/pr-workflow/skills/{pr-validate => evidence}/skill.md (84%) diff --git a/domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py b/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py similarity index 100% rename from domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py rename to domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py diff --git a/domains/pr-workflow/skills/pr-validate/references/aep-local-run.md b/domains/pr-workflow/skills/evidence/references/aep-local-run.md similarity index 100% rename from domains/pr-workflow/skills/pr-validate/references/aep-local-run.md rename to domains/pr-workflow/skills/evidence/references/aep-local-run.md diff --git a/domains/pr-workflow/skills/pr-validate/references/claim-extraction.md b/domains/pr-workflow/skills/evidence/references/claim-extraction.md similarity index 94% rename from domains/pr-workflow/skills/pr-validate/references/claim-extraction.md rename to domains/pr-workflow/skills/evidence/references/claim-extraction.md index fdd90417..9dde89d9 100644 --- a/domains/pr-workflow/skills/pr-validate/references/claim-extraction.md +++ b/domains/pr-workflow/skills/evidence/references/claim-extraction.md @@ -1,6 +1,6 @@ # Claim extraction -The linchpin of pr-validate: before choosing any lane, turn the PR into a **falsifiable, surface-specific claim**. Every lane is only as good as the claim it tests. A vague claim ("improves perf", "fixes the bug") can't be proven or refuted; a sharp claim names the precondition, action, observable outcome, and what would disprove it. +The linchpin of evidence: before choosing any lane, turn the PR into a **falsifiable, surface-specific claim**. Every lane is only as good as the claim it tests. A vague claim ("improves perf", "fixes the bug") can't be proven or refuted; a sharp claim names the precondition, action, observable outcome, and what would disprove it. ## Read these, in order diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md similarity index 98% rename from domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md rename to domains/pr-workflow/skills/evidence/references/evidence-catalog.md index 8c8b7680..d3578ca3 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md @@ -80,7 +80,7 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- - **Capture:** `ui/helpers/utils/performance-observers.ts`; `window.stateHooks.getLongTaskMetricsWithTBT()` → `{count, totalDuration, maxDuration, tbt, tbtRating}`. TBT good<200 / needs-improvement<600 / poor>600. Sampled 10% prod / 100% test. ## C4. React render & selector proof - - **Engine: the `react-render-proof` skill.** Delegate the measurement to it; it runs the source/delivery/metric gates, derives the needle from real build output, repeats the capture, and returns a band (or "not resolvable at this n" with an MDE). pr-validate packages the result. + - **Engine: the `react-render-proof` skill.** Delegate the measurement to it; it runs the source/delivery/metric gates, derives the needle from real build output, repeats the capture, and returns a band (or "not resolvable at this n" with an MDE). evidence packages the result. - **Proves:** a component/selector stopped over-rendering (cascade-amplification before/after — exogram `react-redux-performance`). - **Capture:** WDYR via `ENABLE_WHY_DID_YOU_RENDER` (`.metamaskrc` or env) — wired in `app/scripts/development/wdyr.ts` (`trackAllPureComponents`); console logs each unnecessary re-render. `yarn devtools:react` for the Profiler flame graph. Selectors use `reselect`'s `createSelector`, which **does expose a real `.recomputations()` counter** — read it (sample on an interval if the count should visibly climb) rather than injecting a log into the selector body; an injected log is an authored claim, a library API is an observation. *(This entry previously said there was no built-in counter. There is.)* - **Bar:** the delivery check comes before the number. An arm whose manipulation cannot be observed in the built bundle produces a null indistinguishable from "small effect" — and reports as the second. @@ -137,7 +137,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ # D. Build output ## C9. Retention-path analysis — memory leak from code ⭐ *(static; lead for leak claims)* -- **Engine: the `memory-leak-hunt` skill.** For a memory-leak claim, delegate the analysis to `memory-leak-hunt` — it runs Phase-1 static pairing (and Phase-2 heap investigation if a primitive can't be paired) and returns the paired/unpaired sites + verdict. pr-validate keeps **memory leak** as the evidence category: it invokes the skill on the diff and packages the result (in-situ scan capture, plus the lifecycle test / retainer graph if Phase 2 ran) as the category's evidence. The lane spec below is the method that skill implements. +- **Engine: the `memory-leak-hunt` skill.** For a memory-leak claim, delegate the analysis to `memory-leak-hunt` — it runs Phase-1 static pairing (and Phase-2 heap investigation if a primitive can't be paired) and returns the paired/unpaired sites + verdict. evidence keeps **memory leak** as the evidence category: it invokes the skill on the diff and packages the result (in-situ scan capture, plus the lifecycle test / retainer graph if Phase 2 ran) as the category's evidence. The lane spec below is the method that skill implements. - **Proves:** "X is retained past its lifecycle boundary" / "collection Y grows unboundedly" — argued from code, no runtime needed. This is the lane that works at **review time** (does this PR *introduce* retention?) and leads fix-side validation (does the fix *break* the retention path?). C7 is the runtime corroborator, not the lead — leaks need many cycles to exceed noise. Full category: `exogram-daemon/artifacts/evidence-taxonomy/category-memory-retention-from-code.md`. - **Capture — the holder → held → boundary triple, per suspect:** (1) the **holder** (listener, closure, module singleton, accumulating collection, timer); (2) the **held set** — the *specific* objects pinned (list the closure's captures; note when a closure links two objects' GC); (3) the **outlived boundary** (`destroy()`, stream close, instance replacement, request completion). Method: **pair every acquire with its release site** (`on`↔`removeListener`, push↔drain, assign↔null) — the absence of the pair, cited at the acquire site, IS the finding. Four canonical shapes: unbounded accumulator (defeated guard, no drain) · stale-instance listeners on replacement · unremoved listener + capture set · retention past `destroy()`. - **Scope to the diff, or you invent findings.** Classify every flagged primitive as *introduced by this PR* (in the added lines) vs *pre-existing* (already in the file). Charge only the introduced ones to the PR; report pre-existing un-paired primitives separately and uncharged. On extension#40684 the two new stream listeners each had a `removeListener` on `onStreamClosed` (the exact fix a reviewer suggested) and the new pending-request Map had its `.delete` — no leak introduced — while three pre-existing un-torn-down listeners were surfaced but left uncharged, matching how the human/bot reviewers treated them in-thread. This lane *is* the retention review automated; a heap snapshot (C7) is warranted only for an introduced primitive it cannot pair. @@ -151,7 +151,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **Proves:** a module moved to the intended (lazy) chunk and no longer ships on the critical path. Requires the webpack build. Mirrors AEP `perf-chunks`. ## D3. LavaMoat policy / supply-chain capability diff - - **Engine: the `supply-chain-audit` skill** (umbrella — lockfile/manifest diff, advisories, Socket Security, install scripts), which delegates capability grants to **`lavamoat-policy-diligence`** (per-grant call-site justification). Delegate the dependency change to the umbrella; it returns a disposition per lane. pr-validate keeps **supply-chain capability diff** as the evidence category and packages the output. Note the lanes are independent: a clean policy diff does not mean a safe dependency, and a known CVE never appears as a new grant. + - **Engine: the `supply-chain-audit` skill** (umbrella — lockfile/manifest diff, advisories, Socket Security, install scripts), which delegates capability grants to **`lavamoat-policy-diligence`** (per-grant call-site justification). Delegate the dependency change to the umbrella; it returns a disposition per lane. evidence keeps **supply-chain capability diff** as the evidence category and packages the output. Note the lanes are independent: a clean policy diff does not mean a safe dependency, and a known CVE never appears as a new grant. - **Proves:** a dependency change (bump/add/lockfile) grants **no *unjustified* new capability** — the supply-chain-risk lane. Note the bar: for a bump the policy *will* change, so "empty diff" is the WRONG test; the right test is **every new grant is justified by the dep's function**. Full category (trust-boundary framing, generalizes past LavaMoat to any capability-containment mechanism): `exogram-daemon/artifacts/evidence-taxonomy/category-supply-chain-capability-diff.md`. - **Capture:** **Prefer the CI-generated policy whenever one is available.** `@metamaskbot update-policies` regenerates the policy files from a real run of the code and `validate-lavamoat-policies` fails the build on drift, so the committed policy on a bot-run PR *is* the authoritative artifact — diff that. Regenerating locally when a current CI policy exists only re-does a machine that is already trusted, and a local run's provenance is weaker (your node/OS/lockfile resolution, not CI's). **Local regen is the fallback**, for when the bot hasn't run yet, the branch is unpushed, or you need a variant CI didn't cover: `yarn webpack:lavamoat:policy:build` (`:mv2` / `:mv3` for variants) over `lavamoat/webpack/build/policy.json` (+ `policy-override.json`). Either way, `git diff` the policy across **all 8 variants** (mv{2,3}/{main,beta,flask,experimental}) — a grant can appear in one and not others. Then audit **grant-by-grant**: new **globals** (`fetch`, `importScripts`, `WebAssembly`) / **builtins** (`fs`, `child_process`) on a dep that shouldn't need them, new **packages** edges to powerful APIs, or an identifier substitution (`pkgC>name` replacing `pkgB>pkgA>name` = possible dep swap). Falsifier = a surprising grant ("I wonder what it's using this for"). Guide: lavamoat.github.io/guides/policy-diff/. `allowScripts` in `package.json` gates install scripts. diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-gate-setup.md b/domains/pr-workflow/skills/evidence/references/evidence-gate-setup.md similarity index 87% rename from domains/pr-workflow/skills/pr-validate/references/evidence-gate-setup.md rename to domains/pr-workflow/skills/evidence/references/evidence-gate-setup.md index f1df5b9c..21a2c998 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-gate-setup.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-gate-setup.md @@ -23,7 +23,7 @@ Add a `PreToolUse` hook with matcher `Bash` that runs the script with `python3`. "hooks": [ { "type": "command", - "command": "python3 /absolute/path/to/pr-validate/hooks/pr-evidence-gate.py" + "command": "python3 /absolute/path/to/evidence/hooks/pr-evidence-gate.py" } ] } @@ -32,10 +32,10 @@ Add a `PreToolUse` hook with matcher `Bash` that runs the script with `python3`. } ``` -Resolve the path to wherever `pr-validate` lives on disk. Note that `tools/install` copies only the `references`/`scripts`/`assets`/`adapters` bundles into `~/.claude/skills/mms-pr-validate/` — the `hooks/` directory is **not** part of the installed bundle. Point the `command` at your checked-out skills repo instead: +Resolve the path to wherever `evidence` lives on disk. Note that `tools/install` copies only the `references`/`scripts`/`assets`/`adapters` bundles into `~/.claude/skills/mms-evidence/` — the `hooks/` directory is **not** part of the installed bundle. Point the `command` at your checked-out skills repo instead: ``` -<skills-repo>/domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py +<skills-repo>/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py ``` **When it blocks:** the hook exits `2` and prints the reason (which claim, what artifact/tracker it needs) to stderr. Claude Code surfaces that to the model, which self-corrects — attaches the missing artifact/tracker or downgrades the verdict — and re-posts. No manual intervention needed. @@ -44,7 +44,7 @@ Resolve the path to wherever `pr-validate` lives on disk. Note that `tools/insta These are independent of the hook; the skill needs them whether or not you install the gate. -1. **`gh pr comment` must be permitted — pick a grant model.** pr-validate posts its evidence bundle as a PR review comment (`gh pr edit` if publishing into your own PR body). Four options, in descending order of standing safety: +1. **`gh pr comment` must be permitted — pick a grant model.** evidence posts its evidence bundle as a PR review comment (`gh pr edit` if publishing into your own PR body). Four options, in descending order of standing safety: | Model | How | Tradeoff | |---|---|---| diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md similarity index 97% rename from domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md rename to domains/pr-workflow/skills/evidence/references/evidence-publishing.md index 6f031e99..4316c52f 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md @@ -117,7 +117,7 @@ Screenshots block (injected into `### After`, or appended under `### Screenshots ## Step 3 — Choose the surface by ownership, then publish **Publish surface depends on my relationship to the PR** (see exogram -`pr-validate-publish-surface-by-ownership`). Determine it FIRST: +`evidence-publish-surface-by-ownership`). Determine it FIRST: ```bash PR=<n>; REPO=MetaMask/metamask-extension @@ -223,7 +223,7 @@ The platform can't collect video (artifact regex = png/jpg/log/txt). Capture out ## Re-validation runs: delta-first presentation, every verdict re-earned (2026-07-21) -The common loop — a run refutes a claim, the author pushes a fix, `/pr-validate` re-runs at the new head — gets a **delta report**, not a second full bundle: +The common loop — a run refutes a claim, the author pushes a fix, `/evidence` re-runs at the new head — gets a **delta report**, not a second full bundle: - **Presentation is delta-only.** Full exhibits only for lanes whose outcome changed (flipped verdict / new lane / new residual). Unchanged lanes collapse to a `Prior run | This run` ledger, each row with a fresh run-log link from the new head plus one link to the prior run's comment for the full exhibits — and say so ("unchanged rows re-run at `<head>`; full exhibits in the prior run"). - **Evidence is never delta.** Evidence is head-pinned: re-run every automated lane at the new head and re-earn every verdict with a fresh artifact. "Unchanged" is a conclusion from the re-run, never a carried-over assumption (the stale-baseline trap at report level). Re-running is cheap — the falsifier harness already exists from the first run. @@ -231,7 +231,7 @@ The common loop — a run refutes a claim, the author pushes a fix, `/pr-validat - New head → **new hosted artifact directory keyed to the fix commit** (`pr-<n>/fix-<sha>/`), commit-pinned raw URLs; never overwrite a prior run's published files. - Residuals the fix intentionally leaves get their own row/section — don't round a fixed-with-residual claim up to fully proven. -Source of truth: `exogram-core/memory/pr-validate-revalidation-delta-reports.md`. +Source of truth: `exogram-core/memory/evidence-revalidation-delta-reports.md`. ## Lead with a lane-status ledger (no silent absence) @@ -293,11 +293,11 @@ contract. `hooks/pr-evidence-gate.py` enforces the canonical literal only on bod trip it, which is the tell that the two are different artifacts rather than one with a different skin. -**Per-scenario presentation (2026-07-21):** the same applies one level down — when the evidence spans multiple test scenarios (flag-on vs flag-off, control vs treatment in an A/B falsifier, numbered manual-testing steps), give each scenario its **own sub-section**: a heading naming the scenario in observation terms, one line on what it tests plus its verdict, and that scenario's artifacts co-located under it. Never bunch all scenarios' artifacts into one large evidence dump — the reviewer verifies "under condition X, artifact shows Y" one condition at a time, and a merged block destroys that mapping even when every artifact is real. For long artifact sets use a `<details>` block *per scenario*, not a merge. (Preference: exogram-core `memory/pr-validate-present-scenarios-separately.md`; instance #44610.) +**Per-scenario presentation (2026-07-21):** the same applies one level down — when the evidence spans multiple test scenarios (flag-on vs flag-off, control vs treatment in an A/B falsifier, numbered manual-testing steps), give each scenario its **own sub-section**: a heading naming the scenario in observation terms, one line on what it tests plus its verdict, and that scenario's artifacts co-located under it. Never bunch all scenarios' artifacts into one large evidence dump — the reviewer verifies "under condition X, artifact shows Y" one condition at a time, and a merged block destroys that mapping even when every artifact is real. For long artifact sets use a `<details>` block *per scenario*, not a merge. (Preference: exogram-core `memory/evidence-present-scenarios-separately.md`; instance #44610.) ## Artifact contract (ADR-0058 alignment) -To stay interoperable with the recipe-based verification system (MetaMask/decisions#173), shape the bundle like its reviewer-visible contract where practical: a `summary.json` (claim → verdict → evidence refs), a `trace.json` (the run/assertion log), and an artifact manifest (names + media types), with screenshots/video as the confidence layer. Publishing then becomes "render `summary.json` into the PR section." This keeps pr-validate's output and a recipe's output the same shape — see [lane-assertions.md](lane-assertions.md). Don't hand-roll a divergent format. +To stay interoperable with the recipe-based verification system (MetaMask/decisions#173), shape the bundle like its reviewer-visible contract where practical: a `summary.json` (claim → verdict → evidence refs), a `trace.json` (the run/assertion log), and an artifact manifest (names + media types), with screenshots/video as the confidence layer. Publishing then becomes "render `summary.json` into the PR section." This keeps evidence's output and a recipe's output the same shape — see [lane-assertions.md](lane-assertions.md). Don't hand-roll a divergent format. ## Checklist before you publish diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md b/domains/pr-workflow/skills/evidence/references/evidence-trustworthiness.md similarity index 90% rename from domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md rename to domains/pr-workflow/skills/evidence/references/evidence-trustworthiness.md index 06478a68..f398b2ef 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-trustworthiness.md @@ -19,7 +19,7 @@ A green result is not proof. An agent — or an eager run — can produce eviden 11. **Lanes derive from the Manual testing steps — a CI-green row is not a lane** — the Validation Run's rows are generated top-down from the claim and the PR's own **Manual testing steps**, never bottom-up from whatever links already exist. For each step the claim depends on, the lane's payload is the **captured output of executing that step** (step "in Discover, group by `trace`" → a Discover permalink / **linked** trace-id table showing N rounds → N distinct `trace_id`s, per item 12), or an honest ⏳ naming the missing capture with a tracker. A row restating CI ("tests green at head `<sha>` in [CI run]") duplicates the Checks tab and is deleted — and a validation surface carries **zero** CI references, full stop: no `actions/runs` links, no "green at head" clauses, no "as context (only)" retention. The earlier carve-out here ("a CI link is admissible as context on a beyond-CI row") was itself the next costume: within a day all four sibling bodies (extension#43928–#43931) shipped restatements phrased as the exception — rows *leading* with "green at head … in [Unit tests CI]", the same link repeated 3× per body, the remediated row keeping it re-labeled "as context only" — while the gate's excuse regex matched the mere word "revert", so vocabulary, not evidence, discharged the class. The revert lane cites the revert **outcome** (which blocks failed, at which commit); its green-at-head half is the Checks tab's information and is omitted. A carve-out in an emit-time gate is an instruction to generation to phrase every violation as the exception — deliberate exceptions route through the human, never through an excuse predicate. Borrowed evidence — a sibling PR's capture, a unit falsifier standing in for the named live surface — never upgrades an uncaptured lane to ✅: "mechanism live-proven" co-located with "was not exercised" is an inflated verdict; downgrade it. Emit-time trigger: `pr-evidence-gate.py` classes `ci-restatement` (unconditional since 2026-07-21: any CI link / CI-green phrase in validation scope fires — no verdict co-location required, no beyond-CI excuse) and `inflated-verdict`, with the shipped extension#43928 rows and the carve-out-blessed "as context" shape as regression cases (2026-07-21). 12. **Identifiers resolve in one click — a bare id is a digging assignment** — trace ids, event ids, run ids, SHAs are *pointers into a system*, not evidence. Publishing a bunch of raw trace ids hands the reviewer the job of reconstructing project/environment/time window and querying Sentry themselves — it fails item 9's ~30-second test by construction (item 9 makes the signal *findable*; this item makes it *checkable*). Every identifier published as evidence is either hyperlinked to its resolving surface (the Sentry trace/event permalink, or an absolute-windowed Discover query pre-filtered to exactly those ids) or accompanied by the re-hosted captured output (query-result rows / envelope excerpt showing the discriminating fields) — ideally both. Special case that produced the rule: ids captured **locally** (mockttp forwarder, envelope intercept) never reached Sentry, so no permalink can exist — the re-hosted capture is the *only* admissible form, and pasting the id fragments plus a re-run recipe is the "spec necessary / output sufficient" violation wearing ids as decoration (extension#43931 Validation row, 2026-07-21). Rule of construction: when any item in this gate blesses an evidence class by name ("trace-id table", "envelope log"), it means the class's *resolvable instance*, never its bare tokens — a blessed class name is otherwise the next costume. Emit-time trigger: `pr-evidence-gate.py` class `bare-identifier`; converse-of-gate note: the prior gate *whitelisted* `trace_ids?` as beyond-CI payload and its own fix-message recommended "trace-id table" unqualified — second occurrence of "audit the gate for whitelists of the violating shape." 13. **Terminal exhibits are reader-native — a live link or a visual; a dump behind a link is still an opaque reference** — item 12 makes every pointer resolve in one click; this item constrains what it may resolve *to*. A positive verdict's terminal artifact is one of the two media a reviewer natively consumes: a **live link into the resolving system** (Sentry trace/event permalink, absolute-windowed Discover query pre-filtered to the claim) or a **visual capture** (screenshot/recording, annotated or cropped to the discriminating region). Raw files (`.log`/`.json`/`.har`, MB-scale dumps) are **appendix-only** — linked once for auditability, never the exhibit a claim rests on: a link whose target is a raw dump passes item 12 and fails item 9 one click later; the digging moved a hop away, it did not disappear (extension#43931 *second* remediation, 2026-07-21: the `bare-identifier` fix shipped a ✅ row whose sole resolver was a re-hosted ~70KB run log). Two corollaries: (a) **the gate items are conjunctive** — a fix for the newest item must re-pass all prior items; satisfying resolvability with an artifact that fails legibility is the generator's next costume; (b) **ascertain the terminal medium at step zero and pick the capture lane that can produce it** — a local intercept (mockttp envelope forwarder) can never yield a live Sentry permalink, so for Sentry-observable claims it is the supplementary falsifier lane and live ingest (dev build → `SENTRY_DSN_DEV`/test-metamask) is primary, precisely because it terminates in permalinks + screenshots; choosing a lane that cannot produce the terminal medium silently displaces it. Emit-time trigger: `pr-evidence-gate.py` class `dump-resolver`, with the remediated extension#43931 row as the regression case (2026-07-21). -14. **Manual testing steps are the validation contract — steps present ⇒ live evidence definitionally required; an impossibility waiver contradicting an executable step is invalid** — the PR's own **Manual testing steps** are the author's assertion that the claimed behavior *is* live-observable, and how: each numbered step is an executability proof (a step a human reviewer can run, `/pr-validate` can run) and its text is the capture spec. Item 11 derives the lanes from the steps top-down; this item closes the other side of the hatch — a lane derived from a step may not then be *waived by argument*. Emit-time procedure: build the per-step coverage map (step → executed-output artifact); for any step without one, the only admissible state is a **per-step** ⏳ + tracker whose blocker is that step's *own* unmet precondition, checked against the step text. Three waiver-inflation patterns from the producing instance (extension#43228/#42869/#44538, 2026-07-21 — all three shipped articulate impossibility rationales in the same body whose Manual testing steps asserted the opposite): (a) **borrowed impossibility** — the excuse imported from a different mechanism or sibling PR (#43228 waived its live lane citing the async remote-flag read race, which is #44538's mechanism; #43228's overrides are build-time env vars, and its steps 1–4 are directly executable in a dev build); (b) **lane-limitation universalized** — one harness's gap stated as global impossibility (#42869: "the e2e harness emits no error `event` envelope, so … not capturable pre-merge" — the step says *dev build with Sentry enabled, trigger an error*, which ingests error events into test-metamask without the e2e harness); (c) **blocked-scope inflation** — a genuinely blocked precondition of one half of the claim expanded to waive the whole lane (#44538: LaunchDarkly provisioning [#7482](https://github.com/MetaMask/MetaMask-planning/issues/7482) blocks only the *prod-flag* half; the step's own text names the dev-injectable alternative — "or inject it into persisted `RemoteFeatureFlagController` state"). If a step is *truly* non-executable, the waiver is still inadmissible alone: the Manual testing steps are then wrong and are corrected in the same edit — a document may not simultaneously instruct a reviewer to observe X and declare X unobservable. The honest-⏳ lane blessed throughout this gate is for *not yet done*, never for *argued away*: an eloquent impossibility rationale is the cheapest token sequence that satisfies every prior item (no fake capture, no CI link, no bare id, no dump) — the generator's costume for the coverage axis. Emit-time trigger: `pr-evidence-gate.py` class `step-waiver` ("not demonstrable / capturable / observable", "not separately captured", "not attached", "rests entirely/solely on the falsifiers/unit/revert" — unconditional in validation scope; no tracker or artifact excuses it), with the three shipped waiver paragraphs as regression cases and the gate verified-blocking on all three live bodies (2026-07-21). Detection gap: the per-step consistency check (does a deferral's blocker match the step's own precondition?) stays procedural — the gate sees vocabulary, not step semantics. +14. **Manual testing steps are the validation contract — steps present ⇒ live evidence definitionally required; an impossibility waiver contradicting an executable step is invalid** — the PR's own **Manual testing steps** are the author's assertion that the claimed behavior *is* live-observable, and how: each numbered step is an executability proof (a step a human reviewer can run, `/evidence` can run) and its text is the capture spec. Item 11 derives the lanes from the steps top-down; this item closes the other side of the hatch — a lane derived from a step may not then be *waived by argument*. Emit-time procedure: build the per-step coverage map (step → executed-output artifact); for any step without one, the only admissible state is a **per-step** ⏳ + tracker whose blocker is that step's *own* unmet precondition, checked against the step text. Three waiver-inflation patterns from the producing instance (extension#43228/#42869/#44538, 2026-07-21 — all three shipped articulate impossibility rationales in the same body whose Manual testing steps asserted the opposite): (a) **borrowed impossibility** — the excuse imported from a different mechanism or sibling PR (#43228 waived its live lane citing the async remote-flag read race, which is #44538's mechanism; #43228's overrides are build-time env vars, and its steps 1–4 are directly executable in a dev build); (b) **lane-limitation universalized** — one harness's gap stated as global impossibility (#42869: "the e2e harness emits no error `event` envelope, so … not capturable pre-merge" — the step says *dev build with Sentry enabled, trigger an error*, which ingests error events into test-metamask without the e2e harness); (c) **blocked-scope inflation** — a genuinely blocked precondition of one half of the claim expanded to waive the whole lane (#44538: LaunchDarkly provisioning [#7482](https://github.com/MetaMask/MetaMask-planning/issues/7482) blocks only the *prod-flag* half; the step's own text names the dev-injectable alternative — "or inject it into persisted `RemoteFeatureFlagController` state"). If a step is *truly* non-executable, the waiver is still inadmissible alone: the Manual testing steps are then wrong and are corrected in the same edit — a document may not simultaneously instruct a reviewer to observe X and declare X unobservable. The honest-⏳ lane blessed throughout this gate is for *not yet done*, never for *argued away*: an eloquent impossibility rationale is the cheapest token sequence that satisfies every prior item (no fake capture, no CI link, no bare id, no dump) — the generator's costume for the coverage axis. Emit-time trigger: `pr-evidence-gate.py` class `step-waiver` ("not demonstrable / capturable / observable", "not separately captured", "not attached", "rests entirely/solely on the falsifiers/unit/revert" — unconditional in validation scope; no tracker or artifact excuses it), with the three shipped waiver paragraphs as regression cases and the gate verified-blocking on all three live bodies (2026-07-21). Detection gap: the per-step consistency check (does a deferral's blocker match the step's own precondition?) stays procedural — the gate sees vocabulary, not step semantics. 15. **The exhibit lives in the body — link AND visual; a live link alone is the verification path, not the exhibit** — item 13 blessed the terminal media as a *disjunction* (live link OR visual), and generation took the cheaper disjunct: a Discover permalink is producible from the API token alone, a screenshot needs a browser session — so extension#44540's live-ingestion exhibit shipped as a permalink + prose counts, with nothing in the PR body a reader could look at (2026-07-21: "only sentry link and not screenshot that makes it immediately obvious how evidence validates pr"). A live link defers validation behind **click + auth + query rendering + column interpretation** — the dump-resolver displacement one hop further, with the mountain now behind a login: it fails item 9's ~30-second test at the moment of the click, and for any reader *without* Sentry org access (most PR reviewers) a link-only exhibit degrades to a bare identifier (item 12) behind an auth wall. The repaired rule is a **conjunction**: a positive verdict's headline exhibit is an **embedded visual** — screenshot/recording of the linked resolving view (Discover result rows, trace waterfall), cropped/annotated to the discriminating region, captioned with what it should show — **and** the co-located live permalink (absolute-windowed) as the independent-verification path. Neither substitutes for the other: link-only hides the exhibit; visual-only is independently unverifiable. The 2026-07-16 clause "screenshots ride along when a browser session is available; the API token alone yields links + JSON, which is the automatable minimum" was the self-authored escape hatch of this axis (family: the "as context" carve-out, the honest-⏳ waiver): the *automatable minimum* got promoted to the shipped standard because it was the cheapest compliant artifact. A capture lane that cannot screenshot its resolving view is a lane gap to fix before publish (drive a browser session to the Discover URL), never a licensed downgrade — deliberate exceptions route through the human. Emit-time trigger: `pr-evidence-gate.py` class `link-only-exhibit` (non-negated verdict + `sentry.io` link + no image/recording embed in the unit), with the shipped #44540 paragraph as the regression case and the prior suite's permalink-only ALLOW cases flipped/augmented — third occurrence of "an ALLOW case containing the violating tokens is a specification of the next costume." Detection gaps: verdict co-location is required, so a no-verdict link-only paragraph evades mechanically; the visual-without-link converse stays procedural under item 12. 16. **The audit chain is mechanical — quote, don't transcribe; pin, don't point** — an exhibit's inline data must be a **verbatim, greppable excerpt** of the artifact (full-length identifiers, raw capture lines quoted exactly), and every repo-hosted artifact link must be a **commit-pinned, line-anchored permalink** (`/blob/<sha>/…#Lx-Ly`) to the discriminating lines. Producing real artifacts and then hand-transcribing digests severs the claim→artifact bridge at every link: an ellipsized id (`24b1e2da…`) cannot be grepped against any artifact even when the log is linked in the same block; a reformatted data block cannot be distinguished from confabulation without redoing the dig, so it reads as *claims in the form of data*; a branch-ref `/blob/<branch>/` link is a mutable pointer whose target can be rewritten after review (not tamper-evident); a file-level link without line anchors lands the reader at the top of a 1,000-line dump. The producing instance (extension#43929 validation comment, 2026-07-21) had every number substantiated by four re-hosted run logs — real, included, resolvable — and still drew "still no immediately auditable evidence just claims in the form of data": each prior item individually near-passed while the exhibit↔artifact binding stayed **editorial** (transcription + file link) instead of **mechanical** (quotation + pinned line anchor). Emit-time procedure: for every inline datum, quote the raw capture line it comes from (fenced, verbatim, full ids) and anchor it (`#L<n>`); pin every evidence link to the SHA (press `y` on the GitHub file view). Emit-time trigger: `pr-evidence-gate.py` classes `truncated-identifier` (a co-located resolver does NOT excuse it — the resolver resolves the full id, not the fragment the reader holds; hash-equality prose exempt) and `mutable-ref`, plus the **surface hole** — the shipped comment was published via `gh api` PATCH, which the porcelain-only matcher (`gh pr|issue edit|create|comment`) never saw; fifth occurrence of the converse-of-gate rule, one level down: audit the gate for *spellings of the write it cannot see*, not just tokens it excuses. **CORRECTION 2026-07-30 — this hole was recorded as closed and is not.** Verified against the deployed `hooks/pr-evidence-gate.py` (259 lines): line 47 is the only command matcher, `\bgh\s+(?:pr|issue)\s+(?:edit|create|comment)\b`, so `gh api` body writes are still invisible; and the file implements essentially one check (verdict-needs-artifact), **not** the ~9 classes named across items 11–18 (`ci-restatement`, `bare-identifier`, `dump-resolver`, `link-only-exhibit`, `data-only-exhibit`, `step-waiver`, `truncated-identifier`, `mutable-ref`, `inflated-verdict`). Treat every "Emit-time trigger: `pr-evidence-gate.py` class …" line in this document as **specified, not implemented**, until re-verified in the code — a doc asserting a class the code lacks retires the vigilance it claims to replace, which is the failure this very item warns about. Consequence observed the same day: 14 unlinked `path:line` references shipped across 12 review comments via `gh api`, with the gate both classless for that shape and unwired in `settings.json`. Detection gaps: the line-anchor half of pinning and the paraphrase-vs-quotation judgment stay procedural — the gate sees ellipses and branch refs, not editorial fidelity. 17. **Evidence is captured in its environment — data alone is insufficient even when correct** — item 16 makes the data trustworthy as *transcription* (verbatim, greppable, pinned); this item polices what transcription can never carry: **liveness provenance**. A quoted `EVIDENCE trace_id=…` line, a re-hosted gist, a hand-assembled id table can all be correct and still show nothing about *where they came from* — extracted data is indistinguishable from data typed by hand, so it cannot make it immediately apparent that the evidence was captured **live** from a **functioning** system. The exhibit for a system-of-record-observable claim therefore includes an **in-environment capture**: a screenshot/recording of the resolving system's own UI (the Sentry Discover/trace view with the query, project/environment selectors, absolute time window, and result rows all in-frame) — the environmental chrome is not decoration, it *is* the provenance: it shows the query really ran, in the real dashboard, over the real window, and returned these rows. Correctness was never the failing dimension (2026-07-21: "just the data is insufficient even if correct — it needs to be immediately apparent that evidence was captured live and is functional"). Relation to prior items: item 15's link+visual conjunction fired only when a `sentry.io` link was present, and item 13's `NATIVE_MEDIUM` blessed an inline fenced excerpt as a terminal medium — so a no-link, quoted-data exhibit (the fidelity-remediated shape: full ids, verbatim excerpts, pinned line anchors, zero environment captures) passed the whole regime while carrying zero liveness provenance. The joint rule after this item: a telemetry-observable positive verdict always carries the in-environment visual (plus the live permalink per item 15); quoted excerpts, gists, and data files are appendix beside it, never the exhibit. Emit-time trigger: `pr-evidence-gate.py` class `data-only-exhibit` (non-negated verdict + telemetry-observation vocabulary + no image/recording embed + no sentry link — with a sentry link, `link-only-exhibit` already fires), with the re-hosted-gist ALLOW case flipped (fifth occurrence of "the ALLOW case was the next costume's spec") and the #43929 quoted-excerpt shape as a regression case. Detection gaps: vocabulary-scoped (telemetry-observation terms, not bare code tokens like `trace.test.ts`), so a claim phrased entirely without them evades mechanically; and the gate cannot see whether an embedded image actually shows the environment's chrome — screenshot content stays procedural (item 2's "eyeball it" applies: the capture must show the *resolving UI*, not a cropped data region indistinguishable from a spreadsheet). diff --git a/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md b/domains/pr-workflow/skills/evidence/references/lane-assertions.md similarity index 100% rename from domains/pr-workflow/skills/pr-validate/references/lane-assertions.md rename to domains/pr-workflow/skills/evidence/references/lane-assertions.md diff --git a/domains/pr-workflow/skills/pr-validate/references/worked-examples.md b/domains/pr-workflow/skills/evidence/references/worked-examples.md similarity index 100% rename from domains/pr-workflow/skills/pr-validate/references/worked-examples.md rename to domains/pr-workflow/skills/evidence/references/worked-examples.md diff --git a/domains/pr-workflow/skills/pr-validate/skill.md b/domains/pr-workflow/skills/evidence/skill.md similarity index 84% rename from domains/pr-workflow/skills/pr-validate/skill.md rename to domains/pr-workflow/skills/evidence/skill.md index 351667db..53c3c87e 100644 --- a/domains/pr-workflow/skills/pr-validate/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -1,9 +1,9 @@ --- -name: pr-validate -description: Validate a MetaMask PR with objective evidence — primarily the Autonomous Engineering Platform (AEP) harness (visual_validation for visible UI behavior, perf_validation for non-visible perf behavior), backed by complementary evidence (Sentry query links, screenshots, screen recordings→GIF, DevTools/CDP output, bundle-size, web-vitals, test/CI results). Drives the AEP local stack end to end: preflight (postgres + temporal + worker + control-plane) → submit POST /v1/tasks → poll GET /v1/runs/:id → fetch artifacts → assemble an evidence bundle → publish to the PR body (re-hosting images to a public repo, scrubbing local paths). Match evidence to the PR's specific falsifiable claim, not a fixed checklist. Triggers on /pr-validate, /pr-validate visual, /pr-validate perf, /pr-validate preflight, /pr-validate status, /pr-validate evidence, /pr-validate plan, or when the user mentions validating/proving a PR, AEP / visual validation / perf validation, capturing evidence for a PR, before/after screenshots, a screen recording or GIF for a PR, attaching Sentry links or DevTools output as proof, or publishing an evidence bundle to a PR body. +name: evidence +description: Produce reviewer-grade evidence that a claim is true — or that it is not. Matches the evidence to the specific falsifiable claim rather than running a fixed checklist, across a catalog of 41 lanes: before/after screenshots, falsifying regression tests, render and selector proofs, bundle and LavaMoat diffs, Sentry and Tempo links, state-migration and vault checks, plus the Autonomous Engineering Platform (AEP) harness for autonomous visual and perf capture. Assembles an evidence bundle and publishes it, images re-hosted and local paths scrubbed. Runs three ways: on a PR whose claim someone else made, in the inner loop against uncommitted changes before a reviewer sees them, and on a symptom with no claim yet, where the hypothesis to kill is your own. Triggers on the evidence command and its subcommands (visual, perf, preflight, status, plan, lane, compare) — installed as mms-evidence — or when the user mentions validating or proving a PR, capturing evidence, before/after screenshots, a screen recording for a PR, attaching Sentry or DevTools output as proof, AEP or visual/perf validation, or publishing an evidence bundle. --- -# /pr-validate +# /evidence Prove a PR does what it claims with **objective, reviewer-grade evidence**. The primary engine is the **Autonomous Engineering Platform (AEP)** harness run locally — `visual_validation` for visible UI behavior, `perf_validation` for non-visible perf behavior — augmented by whatever complementary evidence the claim demands (Sentry query links, screenshots, screen recordings, DevTools/CDP output, bundle/web-vitals/test results). @@ -93,15 +93,15 @@ Not for code-correctness review (use `/review`, `/code-review`) or span-quota re | Invocation | Behavior | |---|---| -| `/pr-validate <pr>` | **Flagship.** Read the PR → state the claim → pick lanes → [preflight](references/aep-local-run.md) → run AEP lane(s) + gather complementary evidence → assemble bundle → **propose** the PR-body section and confirm before publishing. | -| `/pr-validate plan <pr>` | Dry run: read the PR, state the claim, recommend lanes + targeting hints. No stack, no run. Cheap first step when unsure. | -| `/pr-validate visual <pr>` | AEP `visual_validation` only. | -| `/pr-validate perf <pr>` | AEP `perf_validation` only (local/uncommitted graph — see [caveat](#perf_validation-caveat)). | -| `/pr-validate preflight` | Health-check the local stack; bring up what's down. No run. | -| `/pr-validate status <run-id>` | Poll `GET /v1/runs/:id`; print stage timeline + `evidenceBundle.artifactRefs`. | -| `/pr-validate evidence <pr> [--run <id>]` | Assemble + publish a bundle from an existing run and/or complementary sources (Sentry/screens/devtools). No new AEP run. | -| `/pr-validate lane <id> <pr>` | Run a single [catalog](references/evidence-catalog.md) lane by id (e.g. `lane F1`, `lane C3`, `lane D3`) — for the non-AEP lanes where you know the claim type. | -| `/pr-validate compare <pr>` | Paired A/B for a perf or refactor claim. Two arm kinds — pick by what the claim varies: **`ref`** (default) builds base + head, captures the lane on both, diffs; avoids the stale-baseline trap (catalog C5). **`substitution`** holds a **fixed head** and varies one *artifact* instead of the ref — replace the PR's hand-written type/schema/constant/policy with the authoritative equivalent and diff a checker's output (catalog D6); no build, no rebase, no merge boundary. **Per-arm checks first, one per kind:** for `ref`, verify the mechanism under test is actually active in each arm (chunk split present, span emitted, flag evaluated) — a null arm without delivered treatment is a no-op, not a control (2026-07-22, #42795 bisect lesson). For `substitution`, verify the unmodified arm is **silent** and that each diagnostic fires for the reason claimed — a noisy Arm A destroys attribution, and a diagnostic tripping one property early scores as a confirmation it isn't (trustworthiness gate item 19; 2026-07-30, #44397). | +| `/evidence <pr>` | **Flagship.** Read the PR → state the claim → pick lanes → [preflight](references/aep-local-run.md) → run AEP lane(s) + gather complementary evidence → assemble bundle → **propose** the PR-body section and confirm before publishing. | +| `/evidence plan <pr>` | Dry run: read the PR, state the claim, recommend lanes + targeting hints. No stack, no run. Cheap first step when unsure. | +| `/evidence visual <pr>` | AEP `visual_validation` only. | +| `/evidence perf <pr>` | AEP `perf_validation` only (local/uncommitted graph — see [caveat](#perf_validation-caveat)). | +| `/evidence preflight` | Health-check the local stack; bring up what's down. No run. | +| `/evidence status <run-id>` | Poll `GET /v1/runs/:id`; print stage timeline + `evidenceBundle.artifactRefs`. | +| `/evidence evidence <pr> [--run <id>]` | Assemble + publish a bundle from an existing run and/or complementary sources (Sentry/screens/devtools). No new AEP run. | +| `/evidence lane <id> <pr>` | Run a single [catalog](references/evidence-catalog.md) lane by id (e.g. `lane F1`, `lane C3`, `lane D3`) — for the non-AEP lanes where you know the claim type. | +| `/evidence compare <pr>` | Paired A/B for a perf or refactor claim. Two arm kinds — pick by what the claim varies: **`ref`** (default) builds base + head, captures the lane on both, diffs; avoids the stale-baseline trap (catalog C5). **`substitution`** holds a **fixed head** and varies one *artifact* instead of the ref — replace the PR's hand-written type/schema/constant/policy with the authoritative equivalent and diff a checker's output (catalog D6); no build, no rebase, no merge boundary. **Per-arm checks first, one per kind:** for `ref`, verify the mechanism under test is actually active in each arm (chunk split present, span emitted, flag evaluated) — a null arm without delivered treatment is a no-op, not a control (2026-07-22, #42795 bisect lesson). For `substitution`, verify the unmodified arm is **silent** and that each diagnostic fires for the reason claimed — a noisy Arm A destroys attribution, and a diagnostic tripping one property early scores as a confirmation it isn't (trustworthiness gate item 19; 2026-07-30, #44397). | `<pr>` is a number or URL on `MetaMask/metamask-extension` unless another repo is given. Every variant runs Step 1 (extract the Claim Card) first — the claim decides the lane, even when you named one. @@ -210,23 +210,23 @@ PR claims privacy mode now hides the Perps balance (the demo bug #42683): End-to-end examples for **non-visual** claims (perf, migration, flag-gated, refactor/no-op): **[references/worked-examples.md](references/worked-examples.md).** -## Positioning: AEP vs recipes vs pr-validate +## Positioning: AEP vs recipes vs evidence Three adjacent things; keep the boundary clear so they compose instead of collide: - **AEP** — governed *fleet orchestration*: sandboxes, Temporal, autonomous runs at scale. The heavy engine. - **ADR-0058 recipes** ([decisions#173](https://github.com/MetaMask/decisions/pull/173)) — a *dev-machine inner-loop* proof artifact: a declarative per-PR recipe run against the live app over CDP, emitting `summary.json`/`trace.json`/manifest. -- **pr-validate** (this skill) — the *claim→evidence methodology + taxonomy* both draw on. The Claim Card is the bridge from a PR's claim to the right proof target; the [evidence catalog](references/evidence-catalog.md) is the lane vocabulary; [lane-assertions.md](references/lane-assertions.md) maps each lane to a recipe assertion (and flags the out-of-band, non-UI lanes — the gap MajorLift's #173 review raised). +- **evidence** (this skill) — the *claim→evidence methodology + taxonomy* both draw on. The Claim Card is the bridge from a PR's claim to the right proof target; the [evidence catalog](references/evidence-catalog.md) is the lane vocabulary; [lane-assertions.md](references/lane-assertions.md) maps each lane to a recipe assertion (and flags the out-of-band, non-UI lanes — the gap MajorLift's #173 review raised). -pr-validate is the one a human drives; it can dispatch an AEP run or author a recipe as its capture step. +evidence is the one a human drives; it can dispatch an AEP run or author a recipe as its capture step. ## Workflow integration -Where pr-validate sits in the PR lifecycle (see the public `pr-workflow` siblings): +Where evidence sits in the PR lifecycle (see the public `pr-workflow` siblings): - **After `create-pr`, before `pr-review-queue`:** validate the claim, attach the bundle, *then* request review — reviewers get the before/after up front. - **On force-push / requested-change:** re-run the affected lane(s); re-validation keeps a stale evidence section honest. -- **`/triage` push items:** a `push`-state PR isn't done until its claim is proven; pr-validate produces the evidence that lets it move. +- **`/triage` push items:** a `push`-state PR isn't done until its claim is proven; evidence produces the evidence that lets it move. - **Not a CI gate** (same scope line as ADR-0058) — it's the author's inner loop, complementing unit/e2e, not replacing them. ## Boundaries diff --git a/domains/testing/skills/falsifying-test/skill.md b/domains/testing/skills/falsifying-test/skill.md index 922991f5..e0a2f367 100644 --- a/domains/testing/skills/falsifying-test/skill.md +++ b/domains/testing/skills/falsifying-test/skill.md @@ -1,6 +1,6 @@ --- name: falsifying-test -description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /falsifying-test, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by pr-validate as the engine behind its falsifying regression test evidence category. +description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /falsifying-test, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by evidence as the engine behind its falsifying regression test evidence category. maturity: experimental --- @@ -80,7 +80,7 @@ Falsifying test — <test name> (Fixes #N) ## Related -- `pr-validate` — packages this skill's output as its B3 evidence category; B7 (deterministic +- `evidence` — packages this skill's output as its B3 evidence category; B7 (deterministic interleaving) is the sibling for concurrency and temporal-ordering bugs. - `react-render-proof` — the same before/after discipline applied to a measured quantity rather than a boolean. From e7e924814a791c172499a8d3c3bfd4cf6ae78c65 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 10:26:40 -0400 Subject: [PATCH 09/63] Repair unresolvable references in `evidence` and `falsifying-test` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The engine table still routed the memory-leak category to `/memory-leak-hunt`, which was renamed to `/memory-leak` — the rename missed the PR that performed it. Two `[[snake_case]]` entries were wiki links to a private authoring vault; they render as literal brackets here and resolve for no reader. One had a real counterpart in `references/` and now links it; the other pointed at a file that does not exist and is dropped rather than left dangling. `falsifying-test` named its evidence categories as `B3` and `B7`. Those are addresses into `evidence-catalog.md`, not names, so both now use the category name and link the catalog. --- domains/pr-workflow/skills/evidence/skill.md | 5 ++--- domains/testing/skills/falsifying-test/skill.md | 5 +++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 53c3c87e..46dc7efe 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -256,12 +256,11 @@ Where evidence sits in the PR lifecycle (see the public `pr-workflow` siblings): | B3 falsifying regression test | `/falsifying-test` | | B7 deterministic interleaving (concurrency / ordering) | `/race-condition-proof` | | C4 React render & selector proof | `/react-render-proof` | - | C9 memory leak | `/memory-leak-hunt` | + | C9 memory leak | `/memory-leak` | | D supply-chain / dependency change | `/supply-chain-audit` → delegates capability grants to `/lavamoat-policy-diligence` | **An engine that defines its own output contract publishes in it.** `lavamoat-policy-diligence` is the live case: read-level triage, no verdict, its own header and marker pair. Do not re-frame it as a Validation Run — see *One comment per evidence kind* in [references/evidence-publishing.md](references/evidence-publishing.md). -- [[reference_aep_local_run]] — the source memory this skill encodes. -- [[reference_sentry_project_topology]] — Sentry project mapping for the telemetry-evidence lane. +- [references/aep-local-run.md](references/aep-local-run.md) — the local-run procedure this skill encodes. diff --git a/domains/testing/skills/falsifying-test/skill.md b/domains/testing/skills/falsifying-test/skill.md index e0a2f367..97219111 100644 --- a/domains/testing/skills/falsifying-test/skill.md +++ b/domains/testing/skills/falsifying-test/skill.md @@ -80,7 +80,8 @@ Falsifying test — <test name> (Fixes #N) ## Related -- `evidence` — packages this skill's output as its B3 evidence category; B7 (deterministic - interleaving) is the sibling for concurrency and temporal-ordering bugs. +- `evidence` — packages this skill's output as its [falsifying regression test category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md). + The deterministic-interleaving category is the sibling for concurrency and temporal-ordering + bugs; `race-condition-proof` drives it. - `react-render-proof` — the same before/after discipline applied to a measured quantity rather than a boolean. From 1e927d5833af17a3b93a5fe3ec751fe0faf40855 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 11:53:39 -0400 Subject: [PATCH 10/63] Follow the engine renames in the catalog and `falsifying-test` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `race-condition-proof` is now `race-condition-repro` and `react-render-proof` is now `react-render-delta`. Both are named here as engines, in the catalog, the engine table, and the sibling reference — none of which the renaming branches can reach. --- .../skills/evidence/references/evidence-catalog.md | 4 ++-- domains/pr-workflow/skills/evidence/skill.md | 4 ++-- domains/testing/skills/falsifying-test/skill.md | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md index d3578ca3..ae90d30f 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md @@ -44,7 +44,7 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- - **Reach for it:** every bug-fix PR. If you can't write a test that fails on main, question whether the fix addresses the reported bug. ## B7. Deterministic interleaving test (concurrency / temporal-ordering) ⭐ -- **Engine:** `race-condition-proof` — run it rather than hand-rolling the harness. +- **Engine:** `race-condition-repro` — run it rather than hand-rolling the harness. - **Proves:** an ordering guarantee under interleaving — retry, cancellation, supersession, debounce, locks, queues, async state machines — where the correctness *is* the ordering under races, not a value. Full category: `exogram-daemon/artifacts/evidence-taxonomy/category-concurrency-temporal-ordering.md`. - **Capture:** force each race deterministically — `jest.useFakeTimers()` + `advanceTimersByTimeAsync(DELAY)` to fire the delayed action at a known point; `Promise.all([opA, opB])` to overlap operations; `advanceTimersByTimeAsync(0)` to step to a precise interleaving point; then assert the ordering/cancellation outcome for **each** guarantee, including asymmetric ones (one path canceled → its recovery event `.not.toHaveBeenCalled()`; another must complete → `.toHaveBeenCalledWith(...)`). Corroborate with transition telemetry; for the integration path, a live forced-race capture (C8/CDP, the #44610 technique). - **Trust-gate:** the test must **actually interleave** — time advanced into the pending window, the superseding op injected *during* it. A sequential run exercises no race and is a vacuous green. Verify the interleaving, not just the assertion. @@ -80,7 +80,7 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- - **Capture:** `ui/helpers/utils/performance-observers.ts`; `window.stateHooks.getLongTaskMetricsWithTBT()` → `{count, totalDuration, maxDuration, tbt, tbtRating}`. TBT good<200 / needs-improvement<600 / poor>600. Sampled 10% prod / 100% test. ## C4. React render & selector proof - - **Engine: the `react-render-proof` skill.** Delegate the measurement to it; it runs the source/delivery/metric gates, derives the needle from real build output, repeats the capture, and returns a band (or "not resolvable at this n" with an MDE). evidence packages the result. + - **Engine: the `react-render-delta` skill.** Delegate the measurement to it; it runs the source/delivery/metric gates, derives the needle from real build output, repeats the capture, and returns a band (or "not resolvable at this n" with an MDE). evidence packages the result. - **Proves:** a component/selector stopped over-rendering (cascade-amplification before/after — exogram `react-redux-performance`). - **Capture:** WDYR via `ENABLE_WHY_DID_YOU_RENDER` (`.metamaskrc` or env) — wired in `app/scripts/development/wdyr.ts` (`trackAllPureComponents`); console logs each unnecessary re-render. `yarn devtools:react` for the Profiler flame graph. Selectors use `reselect`'s `createSelector`, which **does expose a real `.recomputations()` counter** — read it (sample on an interval if the count should visibly climb) rather than injecting a log into the selector body; an injected log is an authored claim, a library API is an observation. *(This entry previously said there was no built-in counter. There is.)* - **Bar:** the delivery check comes before the number. An arm whose manipulation cannot be observed in the built bundle produces a null indistinguishable from "small effect" — and reports as the second. diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 46dc7efe..55a585ed 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -254,8 +254,8 @@ Where evidence sits in the PR lifecycle (see the public `pr-workflow` siblings): | category | engine | |---|---| | B3 falsifying regression test | `/falsifying-test` | - | B7 deterministic interleaving (concurrency / ordering) | `/race-condition-proof` | - | C4 React render & selector proof | `/react-render-proof` | + | B7 deterministic interleaving (concurrency / ordering) | `/race-condition-repro` | + | C4 React render & selector proof | `/react-render-delta` | | C9 memory leak | `/memory-leak` | | D supply-chain / dependency change | `/supply-chain-audit` → delegates capability grants to `/lavamoat-policy-diligence` | diff --git a/domains/testing/skills/falsifying-test/skill.md b/domains/testing/skills/falsifying-test/skill.md index 97219111..7cfeb160 100644 --- a/domains/testing/skills/falsifying-test/skill.md +++ b/domains/testing/skills/falsifying-test/skill.md @@ -82,6 +82,6 @@ Falsifying test — <test name> (Fixes #N) - `evidence` — packages this skill's output as its [falsifying regression test category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md). The deterministic-interleaving category is the sibling for concurrency and temporal-ordering - bugs; `race-condition-proof` drives it. -- `react-render-proof` — the same before/after discipline applied to a measured quantity + bugs; `race-condition-repro` drives it. +- `react-render-delta` — the same before/after discipline applied to a measured quantity rather than a boolean. From 14e670a02b73c3b39c8519de27146e14e7a3342a Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 12:52:45 -0400 Subject: [PATCH 11/63] Add a lane index to the evidence catalog, and fix lane placement MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The catalog is 41 full lane specs with no summary, so a reader arriving from a link lands mid-document with no way to see the shape of it. Adds a generated "Lanes at a glance" table: family, count, and every lane id with its title. Two placement bugs: - `C9` sat inside the `# D. Build output` section, so scanning family C missed the lane backing `memory-leak`, and scanning D found a stranger. - `B7` sat between `B3` and `B4`. All 41 lanes now read in order. Also removes six pointers into a private authoring vault — four inline `exogram` references and two full `exogram-daemon/...` paths. They resolve for no reader of a public repository. Every substantive claim they were attached to is kept; only the dangling pointer is dropped. `memory-leak-hunt` updated to `memory-leak`. --- .../evidence/references/evidence-catalog.md | 48 ++++++++++++------- domains/pr-workflow/skills/evidence/skill.md | 2 +- 2 files changed, 33 insertions(+), 17 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md index ae90d30f..01fa1665 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md @@ -8,6 +8,22 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- --- +## Lanes at a glance + +41 lanes in 7 families. Each lane below has a full spec in its own section — what it proves, how to capture it, and its trust gate. Family G is written as one-liners rather than full sections, because those lanes are links and counts rather than captures. + +| Family | Lanes | | +|---|---|---| +| **A. AEP harness (primary, autonomous)** | 3 | `A1` visual_validation · `A2` perf_validation · `A3` AEP bundle byproducts | +| **B. Behavior & flow proof** | 7 | `B1` Visual before/after via the mm CLI · `B2` E2E trace + video · `B3` Falsifying regression test · `B4` Component / Storybook visual · `B5` Accessibility · `B6` Flaky-stability rerun · `B7` Deterministic interleaving test | +| **C. Performance & render** | 9 | `C1` Startup / custom traces + phase segmentation · `C2` Web vitals · `C3` Long-task / TBT · `C4` React render & selector proof · `C5` Benchmark A/B · `C6` DevTools / CDP profiling · `C7` Memory stability over a flow · `C8` Same-window app + DevTools capture · `C9` Retention-path analysis | +| **D. Build output** | 6 | `D1` Bundle-size diff · `D2` Chunk membership / source-map · `D3` LavaMoat policy / supply-chain capability diff · `D4` Manifest permissions diff · `D5` Build-variant matrix · `D6` Authored-vs-authoritative substitution A/B | +| **E. Production telemetry** | 3 | `E1` Sentry query links · `E2` Tempo distributed traces · `E3` Sentry error-event / breadcrumb shape | +| **F. Extension integrity (high-stakes, extension-specific)** | 8 | `F1` State migration / upgrade · `F2` Vault / keyring round-trip · `F3` Transaction simulation / gas · `F4` Provider / dapp connectivity · `F5` Feature-flag matrix · `F6` Snaps / multichain execution · `F7` i18n usage · `F8` SES lockdown / runtime containment | +| **G. CI, review & process** | 5 | `G1` CI check links · `G2` Coverage delta · `G3` Automated-reviewer output · `G4` Manual reproduction steps · `G5` CI-workflow change, run on a test fork | + +--- + # A. AEP harness (primary, autonomous) ## A1. visual_validation — before/after screenshots @@ -43,12 +59,6 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- - **Capture:** add the test, run it on the PR branch (pass) and on the PR's **merge-base** (fail) — pin the base, don't use whatever `main` points at today. Pair with the PR's `Fixes #N`. **Read the base failure's message, not its exit code:** it must fail on the assertion that encodes the bug. A `ModuleNotFoundError`, a missing fixture, or an unrelated pre-existing red produces an identical non-zero exit and falsifies nothing. - **Reach for it:** every bug-fix PR. If you can't write a test that fails on main, question whether the fix addresses the reported bug. -## B7. Deterministic interleaving test (concurrency / temporal-ordering) ⭐ -- **Engine:** `race-condition-repro` — run it rather than hand-rolling the harness. -- **Proves:** an ordering guarantee under interleaving — retry, cancellation, supersession, debounce, locks, queues, async state machines — where the correctness *is* the ordering under races, not a value. Full category: `exogram-daemon/artifacts/evidence-taxonomy/category-concurrency-temporal-ordering.md`. -- **Capture:** force each race deterministically — `jest.useFakeTimers()` + `advanceTimersByTimeAsync(DELAY)` to fire the delayed action at a known point; `Promise.all([opA, opB])` to overlap operations; `advanceTimersByTimeAsync(0)` to step to a precise interleaving point; then assert the ordering/cancellation outcome for **each** guarantee, including asymmetric ones (one path canceled → its recovery event `.not.toHaveBeenCalled()`; another must complete → `.toHaveBeenCalledWith(...)`). Corroborate with transition telemetry; for the integration path, a live forced-race capture (C8/CDP, the #44610 technique). -- **Trust-gate:** the test must **actually interleave** — time advanced into the pending window, the superseding op injected *during* it. A sequential run exercises no race and is a vacuous green. Verify the interleaving, not just the assertion. - ## B4. Component / Storybook visual - **Proves:** a component renders across states/props in isolation. - **Capture:** `.storybook/` present; `yarn storybook` (port 6006), `yarn storybook:build`, `yarn test-storybook` (visual + a11y via `@storybook/addon-a11y`). Jest snapshot diffs for serialized output. @@ -64,6 +74,12 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- --- +## B7. Deterministic interleaving test (concurrency / temporal-ordering) ⭐ +- **Engine:** `race-condition-repro` — run it rather than hand-rolling the harness. +- **Proves:** an ordering guarantee under interleaving — retry, cancellation, supersession, debounce, locks, queues, async state machines — where the correctness *is* the ordering under races, not a value. +- **Capture:** force each race deterministically — `jest.useFakeTimers()` + `advanceTimersByTimeAsync(DELAY)` to fire the delayed action at a known point; `Promise.all([opA, opB])` to overlap operations; `advanceTimersByTimeAsync(0)` to step to a precise interleaving point; then assert the ordering/cancellation outcome for **each** guarantee, including asymmetric ones (one path canceled → its recovery event `.not.toHaveBeenCalled()`; another must complete → `.toHaveBeenCalledWith(...)`). Corroborate with transition telemetry; for the integration path, a live forced-race capture (C8/CDP, the #44610 technique). +- **Trust-gate:** the test must **actually interleave** — time advanced into the pending window, the superseding op injected *during* it. A sequential run exercises no race and is a vacuous green. Verify the interleaving, not just the assertion. + # C. Performance & render ## C1. Startup / custom traces + phase segmentation @@ -73,7 +89,7 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- ## C2. Web vitals — INP / FCP / LCP / CLS - **Proves:** a user-centric metric moved. `ui/helpers/utils/web-vitals.ts` via `web-vitals/attribution` (attribution names the causing element). - **Capture:** `window.stateHooks.getWebVitalsMetrics()` (test/debug) → `{inp, fcp, lcp, cls, *Rating}`. Thresholds: INP good<200/poor>500, FCP<1800/3000, LCP<2500/4000, CLS<0.1/0.25. -- **Caveat:** **INP fires on all pages; FCP/LCP/CLS do not fire on popup pages** (sidepanel/E2E only). For extensions, INP is the high-value runtime metric (per exogram `web-vitals-runtime-metrics`). +- **Caveat:** **INP fires on all pages; FCP/LCP/CLS do not fire on popup pages** (sidepanel/E2E only). For extensions, INP is the high-value runtime metric. ## C3. Long-task / TBT - **Proves:** main-thread blocking during an interaction dropped. This is where **TBT** lives (the web-vitals lib lane does *not* collect TBT). @@ -81,16 +97,16 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- ## C4. React render & selector proof - **Engine: the `react-render-delta` skill.** Delegate the measurement to it; it runs the source/delivery/metric gates, derives the needle from real build output, repeats the capture, and returns a band (or "not resolvable at this n" with an MDE). evidence packages the result. -- **Proves:** a component/selector stopped over-rendering (cascade-amplification before/after — exogram `react-redux-performance`). +- **Proves:** a component/selector stopped over-rendering (cascade-amplification before/after). - **Capture:** WDYR via `ENABLE_WHY_DID_YOU_RENDER` (`.metamaskrc` or env) — wired in `app/scripts/development/wdyr.ts` (`trackAllPureComponents`); console logs each unnecessary re-render. `yarn devtools:react` for the Profiler flame graph. Selectors use `reselect`'s `createSelector`, which **does expose a real `.recomputations()` counter** — read it (sample on an interval if the count should visibly climb) rather than injecting a log into the selector body; an injected log is an authored claim, a library API is an observation. *(This entry previously said there was no built-in counter. There is.)* - **Bar:** the delivery check comes before the number. An arm whose manipulation cannot be observed in the built bundle produces a null indistinguishable from "small effect" — and reports as the second. ## C5. Benchmark A/B - **Proves:** a startup/journey/interaction timing moved, with a distribution not one sample. - **Capture:** `yarn test:e2e:benchmark` (`test/e2e/benchmarks/run-benchmark.ts`); presets in `shared/constants/benchmarks.ts` (`startupStandardHome`, `sendTransactions`, `swap`, `dappPageLoad`, …). -- **Caveat:** the rolling baseline (`MetaMask/extension_benchmark_stats`) can **silently freeze** behind a green check (the `store-benchmark-stats` step is `continue-on-error`; happened 2026-04-02, PR #42947). Prefer a **paired A/B** (build both refs now, compare directly) over the stored baseline. See exogram `benchmark-baseline-staleness-paired-ab`. +- **Caveat:** the rolling baseline (`MetaMask/extension_benchmark_stats`) can **silently freeze** behind a green check (the `store-benchmark-stats` step is `continue-on-error`; happened 2026-04-02, PR #42947). Prefer a **paired A/B** (build both refs now, compare directly) over the stored baseline. - **Treatment check first** — before trusting any delta, confirm the mechanism under test is actually active in each arm (split chunk present in head and absent in base; the span emitted; the flag evaluated). An arm without the treatment delivered is a no-op, not a control (2026-07-22, #42795). -- **A null needs its power stated** — "no change" and "underpowered" print the same result. When the run-to-run spread exceeds the effect under test, report **not resolvable at this n** and name the smallest detectable effect; never let it read as "no effect". Correcting a known bias (discarding a warm-up, alternating the starting arm) removes *that* bias and nothing more — it is not a trust gate, and the confounds you did not enumerate (thermal drift, background load, ordering within a round) stay live. See exogram `removing-a-bias-is-not-establishing-validity` (2026-07-24). +- **A null needs its power stated** — "no change" and "underpowered" print the same result. When the run-to-run spread exceeds the effect under test, report **not resolvable at this n** and name the smallest detectable effect; never let it read as "no effect". Correcting a known bias (discarding a warm-up, alternating the starting arm) removes *that* bias and nothing more — it is not a trust gate, and the confounds you did not enumerate (thermal drift, background load, ordering within a round) stay live. ### Capturing an authenticated view (the in-situ requirement) @@ -134,16 +150,16 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ --- -# D. Build output - ## C9. Retention-path analysis — memory leak from code ⭐ *(static; lead for leak claims)* -- **Engine: the `memory-leak-hunt` skill.** For a memory-leak claim, delegate the analysis to `memory-leak-hunt` — it runs Phase-1 static pairing (and Phase-2 heap investigation if a primitive can't be paired) and returns the paired/unpaired sites + verdict. evidence keeps **memory leak** as the evidence category: it invokes the skill on the diff and packages the result (in-situ scan capture, plus the lifecycle test / retainer graph if Phase 2 ran) as the category's evidence. The lane spec below is the method that skill implements. -- **Proves:** "X is retained past its lifecycle boundary" / "collection Y grows unboundedly" — argued from code, no runtime needed. This is the lane that works at **review time** (does this PR *introduce* retention?) and leads fix-side validation (does the fix *break* the retention path?). C7 is the runtime corroborator, not the lead — leaks need many cycles to exceed noise. Full category: `exogram-daemon/artifacts/evidence-taxonomy/category-memory-retention-from-code.md`. +- **Engine: the `memory-leak` skill.** For a memory-leak claim, delegate the analysis to `memory-leak` — it runs Phase-1 static pairing (and Phase-2 heap investigation if a primitive can't be paired) and returns the paired/unpaired sites + verdict. evidence keeps **memory leak** as the evidence category: it invokes the skill on the diff and packages the result (in-situ scan capture, plus the lifecycle test / retainer graph if Phase 2 ran) as the category's evidence. The lane spec below is the method that skill implements. +- **Proves:** "X is retained past its lifecycle boundary" / "collection Y grows unboundedly" — argued from code, no runtime needed. This is the lane that works at **review time** (does this PR *introduce* retention?) and leads fix-side validation (does the fix *break* the retention path?). C7 is the runtime corroborator, not the lead — leaks need many cycles to exceed noise. - **Capture — the holder → held → boundary triple, per suspect:** (1) the **holder** (listener, closure, module singleton, accumulating collection, timer); (2) the **held set** — the *specific* objects pinned (list the closure's captures; note when a closure links two objects' GC); (3) the **outlived boundary** (`destroy()`, stream close, instance replacement, request completion). Method: **pair every acquire with its release site** (`on`↔`removeListener`, push↔drain, assign↔null) — the absence of the pair, cited at the acquire site, IS the finding. Four canonical shapes: unbounded accumulator (defeated guard, no drain) · stale-instance listeners on replacement · unremoved listener + capture set · retention past `destroy()`. - **Scope to the diff, or you invent findings.** Classify every flagged primitive as *introduced by this PR* (in the added lines) vs *pre-existing* (already in the file). Charge only the introduced ones to the PR; report pre-existing un-paired primitives separately and uncharged. On extension#40684 the two new stream listeners each had a `removeListener` on `onStreamClosed` (the exact fix a reviewer suggested) and the new pending-request Map had its `.delete` — no leak introduced — while three pre-existing un-torn-down listeners were surfaced but left uncharged, matching how the human/bot reviewers treated them in-thread. This lane *is* the retention review automated; a heap snapshot (C7) is warranted only for an introduced primitive it cannot pair. - **Corroborate:** a falsifying lifecycle test (force the boundary, assert release — listener count zero, singleton nulled, collection drained); C7 heap-over-flow with the **retainer graph naming the same path** the static argument named. - **Trust-gate:** the triple must be specific ("this listener holds `patchStore` after `patchStore.destroy()`", not "might leak"); distinguish **bounded staleness vs unbounded growth** (severity differs); attribute **introduced vs pre-existing** honestly. +# D. Build output + ## D1. Bundle-size diff - **Proves:** the build grew/shrank by a measured amount. Use the bundle-size CI output or a local build size comparison. @@ -152,7 +168,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ ## D3. LavaMoat policy / supply-chain capability diff - **Engine: the `supply-chain-audit` skill** (umbrella — lockfile/manifest diff, advisories, Socket Security, install scripts), which delegates capability grants to **`lavamoat-policy-diligence`** (per-grant call-site justification). Delegate the dependency change to the umbrella; it returns a disposition per lane. evidence keeps **supply-chain capability diff** as the evidence category and packages the output. Note the lanes are independent: a clean policy diff does not mean a safe dependency, and a known CVE never appears as a new grant. -- **Proves:** a dependency change (bump/add/lockfile) grants **no *unjustified* new capability** — the supply-chain-risk lane. Note the bar: for a bump the policy *will* change, so "empty diff" is the WRONG test; the right test is **every new grant is justified by the dep's function**. Full category (trust-boundary framing, generalizes past LavaMoat to any capability-containment mechanism): `exogram-daemon/artifacts/evidence-taxonomy/category-supply-chain-capability-diff.md`. +- **Proves:** a dependency change (bump/add/lockfile) grants **no *unjustified* new capability** — the supply-chain-risk lane. Note the bar: for a bump the policy *will* change, so "empty diff" is the WRONG test; the right test is **every new grant is justified by the dep's function**.. - **Capture:** **Prefer the CI-generated policy whenever one is available.** `@metamaskbot update-policies` regenerates the policy files from a real run of the code and `validate-lavamoat-policies` fails the build on drift, so the committed policy on a bot-run PR *is* the authoritative artifact — diff that. Regenerating locally when a current CI policy exists only re-does a machine that is already trusted, and a local run's provenance is weaker (your node/OS/lockfile resolution, not CI's). **Local regen is the fallback**, for when the bot hasn't run yet, the branch is unpushed, or you need a variant CI didn't cover: `yarn webpack:lavamoat:policy:build` (`:mv2` / `:mv3` for variants) over `lavamoat/webpack/build/policy.json` (+ `policy-override.json`). Either way, `git diff` the policy across **all 8 variants** (mv{2,3}/{main,beta,flask,experimental}) — a grant can appear in one and not others. Then audit **grant-by-grant**: new **globals** (`fetch`, `importScripts`, `WebAssembly`) / **builtins** (`fs`, `child_process`) on a dep that shouldn't need them, new **packages** edges to powerful APIs, or an identifier substitution (`pkgC>name` replacing `pkgB>pkgA>name` = possible dep swap). Falsifier = a surprising grant ("I wonder what it's using this for"). Guide: lavamoat.github.io/guides/policy-diff/. `allowScripts` in `package.json` gates install scripts. ## D4. Manifest permissions diff @@ -244,7 +260,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **G2. Coverage delta** — `yarn test:unit:coverage` → `coverage/unit/` (and `yarn test:unit:webpack:coverage`); `codecov.yml`. Proves the new code is exercised. - **G3. Automated-reviewer output** — independent bot (e.g. cursor[bot]) found nothing blocking. Complements, never replaces, behavior evidence. - **G4. Manual reproduction steps** — human-followable steps that reproduce the fixed behavior; populates the PR template's Manual testing steps. -- **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-<X>=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: push to a branch literally named **`main`** (or `stable`) on `consensys-test/metamask-extension-test-majorlift` — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets on the fork (`INFURA_PROJECT_ID`, `TEST_SRP_*` for benchmarks; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo. Detail: `exogram-daemon/memory/ci-workflow-pr-self-validation-gap.md`. +- **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-<X>=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: push to a branch literally named **`main`** (or `stable`) on `consensys-test/metamask-extension-test-majorlift` — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets on the fork (`INFURA_PROJECT_ID`, `TEST_SRP_*` for benchmarks; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo.. --- diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 55a585ed..916fb6e6 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -234,7 +234,7 @@ Where evidence sits in the PR lifecycle (see the public `pr-workflow` siblings): - **Executes, with a confirmation gate on publish.** It runs the harness and captures evidence autonomously; it does not write to the public PR body without showing you the section first. - **Local-only AEP.** No hosted instance. The skill drives the local stack. - **Proves behavior, not code.** Pair with `/review` / `/code-review` for correctness and `/sentry-quota` for span-volume risk. -- **No persisted state.** Each run is fresh. To keep a validation record, ask — it can go to `exogram-daemon/`, but nothing writes by default. +- **No persisted state.** Each run is fresh. To keep a validation record, ask — nothing writes by default. ## Related From eceaf37d81b128981cda563f651c43201b316571 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 14:11:43 -0400 Subject: [PATCH 12/63] Add build-duration lanes `D7` and `G6` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The catalog had 41 lanes and none for how long a build takes. Family D covered build *output* — size, chunks, policy, permissions, variants — and `C5` covers runtime, so a toolchain change had no category to publish into even though a skill for measuring one is specified in #102. `D7` is the dev-loop half: paired A/B, cold and warm as separate numbers, with the four confounds that each return a favourable result when uncontrolled — warm cache leaking into the cold arm, worker-pool startup amortised away, core count that does not transfer off the measuring machine, and watch rebuilds presented as cold builds. `G6` is the CI half, in family G because its dominant confound is a process one: `get-requirements.yml` skips jobs when build output matches base, so a measured speedup is often a skipped job. Family D is retitled from "Build output" to "Build", since it now covers both. The at-a-glance index is regenerated rather than hand-patched — it is derived from the headings, and hand-editing it is how it drifts. --- .../evidence/references/evidence-catalog.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md index 01fa1665..728aae6f 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md @@ -10,18 +10,17 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- ## Lanes at a glance -41 lanes in 7 families. Each lane below has a full spec in its own section — what it proves, how to capture it, and its trust gate. Family G is written as one-liners rather than full sections, because those lanes are links and counts rather than captures. +43 lanes in 7 families. Each lane below has a full spec in its own section — what it proves, how to capture it, and its trust gate. Family G is written as one-liners rather than full sections, because those lanes are links and counts rather than captures. | Family | Lanes | | |---|---|---| | **A. AEP harness (primary, autonomous)** | 3 | `A1` visual_validation · `A2` perf_validation · `A3` AEP bundle byproducts | | **B. Behavior & flow proof** | 7 | `B1` Visual before/after via the mm CLI · `B2` E2E trace + video · `B3` Falsifying regression test · `B4` Component / Storybook visual · `B5` Accessibility · `B6` Flaky-stability rerun · `B7` Deterministic interleaving test | | **C. Performance & render** | 9 | `C1` Startup / custom traces + phase segmentation · `C2` Web vitals · `C3` Long-task / TBT · `C4` React render & selector proof · `C5` Benchmark A/B · `C6` DevTools / CDP profiling · `C7` Memory stability over a flow · `C8` Same-window app + DevTools capture · `C9` Retention-path analysis | -| **D. Build output** | 6 | `D1` Bundle-size diff · `D2` Chunk membership / source-map · `D3` LavaMoat policy / supply-chain capability diff · `D4` Manifest permissions diff · `D5` Build-variant matrix · `D6` Authored-vs-authoritative substitution A/B | +| **D. Build** | 7 | `D1` Bundle-size diff · `D2` Chunk membership / source-map · `D3` LavaMoat policy / supply-chain capability diff · `D4` Manifest permissions diff · `D5` Build-variant matrix · `D6` Authored-vs-authoritative substitution A/B · `D7` Build & rebuild duration A/B | | **E. Production telemetry** | 3 | `E1` Sentry query links · `E2` Tempo distributed traces · `E3` Sentry error-event / breadcrumb shape | | **F. Extension integrity (high-stakes, extension-specific)** | 8 | `F1` State migration / upgrade · `F2` Vault / keyring round-trip · `F3` Transaction simulation / gas · `F4` Provider / dapp connectivity · `F5` Feature-flag matrix · `F6` Snaps / multichain execution · `F7` i18n usage · `F8` SES lockdown / runtime containment | -| **G. CI, review & process** | 5 | `G1` CI check links · `G2` Coverage delta · `G3` Automated-reviewer output · `G4` Manual reproduction steps · `G5` CI-workflow change, run on a test fork | - +| **G. CI, review & process** | 6 | `G1` CI check links · `G2` Coverage delta · `G3` Automated-reviewer output · `G4` Manual reproduction steps · `G5` CI-workflow change, run on a test fork · `G6` CI job-duration delta | --- # A. AEP harness (primary, autonomous) @@ -158,7 +157,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **Corroborate:** a falsifying lifecycle test (force the boundary, assert release — listener count zero, singleton nulled, collection drained); C7 heap-over-flow with the **retainer graph naming the same path** the static argument named. - **Trust-gate:** the triple must be specific ("this listener holds `patchStore` after `patchStore.destroy()`", not "might leak"); distinguish **bounded staleness vs unbounded growth** (severity differs); attribute **introduced vs pre-existing** honestly. -# D. Build output +# D. Build ## D1. Bundle-size diff - **Proves:** the build grew/shrank by a measured amount. Use the bundle-size CI output or a local build size comparison. @@ -200,6 +199,14 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ --- +## D7. Build & rebuild duration A/B *(paired; lead for toolchain-change claims)* +- **Proves:** what a toolchain change costs or saves in the **dev loop** — a loader, transform, linter, or bundler swap. Distinct from `C5`, which times the shipped app at runtime; this times the build that produces it. The two move independently and in opposite directions often enough that measuring one and inferring the other is the failure this lane exists to prevent (`React Compiler` builds slower and runs faster; `thread-loader` builds faster and runs identically). +- **Shape:** paired A/B, both arms built now, on one machine, alternating order. **Cold and warm are separate questions and get separate numbers** — never one figure labelled "build time". +- **Capture:** N ≥ 5 per arm per mode, alternating. Cold: clear the cache explicitly between arms (`node_modules/.cache`, webpack `cache.cacheDirectory`) and state what was cleared. Warm: touch one source file, rebuild, discard the first result as pool warmup. Report median **and spread**; a median without spread hides a bimodal cache effect. +- **Falsifiers — each returns a favourable number when uncontrolled:** warm cache leaking into the "cold" arm (the largest confound, and the easiest to introduce by running arms in sequence); worker-pool startup counted once and amortised across rebuilds; core count, since parallel loaders scale with the runner and a laptop result does not transfer; watch-rebuild numbers presented as cold-build numbers. +- **Trust-gate:** state machine, core count, N, and cache handling per arm, or the number is unreproducible. A null result states the smallest effect the sample could have detected — "no difference" from N=3 is not a finding. Renders **no ship verdict**: a change that costs build time and buys runtime is a trade, and pricing it is not the same as taking it. +- **Corroborate:** `G6` for the CI half (different machine, different confounds), `C5` for the runtime half. A toolchain claim is not closed by one surface. + # E. Production telemetry ## E1. Sentry query links (before/after) @@ -261,6 +268,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **G3. Automated-reviewer output** — independent bot (e.g. cursor[bot]) found nothing blocking. Complements, never replaces, behavior evidence. - **G4. Manual reproduction steps** — human-followable steps that reproduce the fixed behavior; populates the PR template's Manual testing steps. - **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-<X>=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: push to a branch literally named **`main`** (or `stable`) on `consensys-test/metamask-extension-test-majorlift` — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets on the fork (`INFURA_PROJECT_ID`, `TEST_SRP_*` for benchmarks; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo.. +- **G6. CI job-duration delta** — compare job wall-clock across arms in the Actions UI or `gh run view`. **Falsifier: build reuse.** `get-requirements.yml` skips jobs when build output matches base, so a measured "speedup" is often a skipped job — confirm each arm actually ran the work before comparing. Runner class and queue time vary independently of the change; report job time, not wall-clock from push. Pairs with `D7`, which measures the same change on a machine you control. --- From 90d70821aa9be06603742624de1b691aafd692c4 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 21:32:58 -0400 Subject: [PATCH 13/63] Inline the publishing non-negotiables, which a real run ignored entirely MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An 18-comment trial run met none of this skill's output requirements. The cause was structural, not behavioural: `VALIDATION_RUN_START` and the in-situ capture rule occurred zero times in skill.md and only in a reference costing ~5x the body to open, described there as "image re-hosting and the privacy scrub". The publish gate checks none of them either. All three layers failed open. Moves six non-negotiables and the canonical output shape into the body, where they load with the work: 1. Ship an artifact the reader can check without trusting you. Pasted terminal text is indistinguishable from invented terminal text — running the check justifies your belief, not the reader's. 2. `proven` requires execution; reading gives shape, never power. Run arm B against your own probe: one that passes with the mechanism deleted is measuring something else. 3. No "what would close it" section — that is an unfinished run formatted to look finished. Imperative-mood prose means the artifact does not exist. 4. Write to the reviewer who arrives, not whoever commissioned the run. 5. Delete findings whose entire content is test quality, unless critical. 6. Route privacy and security findings to the private tracker. Derived from eight postmortems in exogram-core; the reference keeps the full recipe. --- domains/pr-workflow/skills/evidence/skill.md | 53 +++++++++++++++++++- 1 file changed, 52 insertions(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 916fb6e6..d88a6c98 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -145,7 +145,58 @@ Stop when each claim has one trustworthy artifact that would have shown its fals ## Publishing the evidence bundle **Public, outward-facing — always confirm the rendered section with the user before writing -a PR body.** Full recipe, markers, image re-hosting, recordings, and the privacy scrub: +a PR body.** + +### Non-negotiables — these are here, not in a reference, because a requirement you have to fetch is advisory + +**1. Ship an artifact the reader can check without trusting you.** Terminal text you pasted is +indistinguishable from terminal text you invented; it carries the weight of your assertion, not +of a measurement. Running the check justifies *your* belief. It becomes *evidence* only when the +reader can confirm it independently: a committed test CI executes, a link to a run, a capture with +visual provenance, an artifact at a URL. **If every character of the output is one you typed, you +have published an assertion.** + +**2. `proven` requires execution; reading yields `unverified`.** Reading a test establishes its +shape, never its power. A test is evidence when it *fails* on the base arm — so run arm B, including +against your own probe. A probe that passes with the mechanism deleted is measuring something else. + +**3. There is no "what would close it" section.** If you know what would close the falsifier, close +it. Three legal endings: proven with artifact attached · unproven, stated flatly and nothing +prescribed · an open question that is genuinely a human's product decision. Imperative-mood prose +(*run*, *switch*, *assert*) means the artifact does not exist. + +**4. Write to the reviewer who arrives, not whoever commissioned the run.** They have a stake in +this PR and none in your tooling. Cut calibration rationale, prior hypotheses, and corrections to +drafts they never saw. One line of disclosure that the output is automated and needs no action is +for them; everything explaining why you are running this is not. + +**5. Delete findings whose entire content is test quality** — code correct, test weak — unless the +untested path touches funds, keys, persisted state, user-visible wrongness, or silent corruption. + +**6. Privacy and security findings are routed, never published here.** File them in the private +planning tracker. Subject matter triggers this, not severity: the code cannot distinguish a missing +gate from a deliberate one. + +### Canonical output shape + +Every validation-run output — PR comment *or* PR-body section — uses exactly this, so re-runs +replace idempotently instead of accumulating: + +```markdown +<!-- VALIDATION_RUN_START --> +## 🧪 Validation Run + +**Verdict:** ✅ proven — **Claim:** <one-line falsifiable behavior under test> +head `<sha>` · <YYYY-MM-DD> · lanes: <lane ids> + +<claim → artifact table; every claim binds its artifact> +<!-- VALIDATION_RUN_END --> +``` + +Verdict icons: `✅` proven · `❌` failed · `ℹ️` otherwise. Never `❌` for a gap in *evidence* — that +reads as a verdict on the author's work. + +Full recipe — image re-hosting, recordings, AEP mirroring, the privacy scrub: **[references/evidence-publishing.md](references/evidence-publishing.md).** The parts that decide *whether* to publish, rather than how: From d412caf5a4169203feebff19f55f9ff6193930f8 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 21:39:23 -0400 Subject: [PATCH 14/63] =?UTF-8?q?Add=20`falsify-probe.sh`=20=E2=80=94=20th?= =?UTF-8?q?e=20runner=20that=20makes=20a=20lane=20reproducible?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A recipe returns as many answers as it has operators, and any output an operator retypes carries the operator's provenance rather than the measurement's. This ships the mechanism instead. Runs arm A, mutates one line, runs arm B, restores the source, and writes `falsify-<label>.{json,md}` plus both raw logs itself — nothing is transcribed. The exit code is the verdict, so CI gates on it directly: 0 falsifying, 1 vacuous, 2 arm A already failing, 3 usage error. Every artifact pins HEAD, node version, yarn.lock hash, and the tracked-change count, so two operators either produce comparable results or visibly do not. Verified against both outcomes on metamask-extension at 796685ce7b7: the perps coalescing suite reports `falsifying` (10 passed, 2 failed under mutation), and the token-search suite reports `vacuous` (3 passed both arms, so it does not test the abort it appears to test). --- .../skills/evidence/scripts/falsify-probe.sh | 134 ++++++++++++++++++ domains/pr-workflow/skills/evidence/skill.md | 23 +++ 2 files changed, 157 insertions(+) create mode 100755 domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh new file mode 100755 index 00000000..8edd34c7 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -0,0 +1,134 @@ +#!/usr/bin/env bash +# +# falsify-probe — prove a test is falsifying, by mutation rather than by reading. +# +# A test is evidence only if it FAILS when the mechanism it guards is removed. +# Reading the test establishes its shape; only this establishes its power. +# +# Runs two arms against the same tree: +# Arm A baseline — the suite as committed +# Arm B mutant — one line replaced, suite re-run, source restored +# +# Emits a captured artifact (JSON + markdown) written by this script, not +# transcribed by an operator. Exit code IS the verdict, so CI can gate on it. +# +# 0 falsifying arm A passed, arm B failed → the test has power +# 1 vacuous arm A passed, arm B ALSO passed → the test proves nothing +# 2 broken arm A failed → nothing to conclude +# 3 usage/env error +# +# Usage: +# falsify-probe.sh --test <path> --source <path> --line <n> --replace <text> +# [--label <slug>] [--out <dir>] [--runner "<cmd>"] +# +# Example: +# falsify-probe.sh \ +# --test ui/hooks/perps/coalesceBackgroundRequest.test.ts \ +# --source ui/hooks/perps/coalesceBackgroundRequest.ts \ +# --line 54 --replace ' const existing = undefined as Promise<TResult> | undefined;' \ +# --label coalesce-inflight +set -uo pipefail + +RUNNER="yarn jest" +OUT_DIR="evidence-artifacts" +LABEL="" +TEST="" SOURCE="" LINE="" REPLACE="" + +die() { printf 'falsify-probe: %s\n' "$1" >&2; exit 3; } + +while [ $# -gt 0 ]; do + case "$1" in + --test) TEST="${2:-}"; shift 2 ;; + --source) SOURCE="${2:-}"; shift 2 ;; + --line) LINE="${2:-}"; shift 2 ;; + --replace) REPLACE="${2:-}"; shift 2 ;; + --label) LABEL="${2:-}"; shift 2 ;; + --out) OUT_DIR="${2:-}"; shift 2 ;; + --runner) RUNNER="${2:-}"; shift 2 ;; + -h|--help) sed -n '2,32p' "$0"; exit 0 ;; + *) die "unknown argument: $1" ;; + esac +done + +[ -n "$TEST" ] || die "--test is required" +[ -n "$SOURCE" ] || die "--source is required" +[ -n "$LINE" ] || die "--line is required" +[ -n "$REPLACE" ] || die "--replace is required (use '' only if deleting the line)" +[ -f "$TEST" ] || die "test not found: $TEST" +[ -f "$SOURCE" ] || die "source not found: $SOURCE" +case "$LINE" in ''|*[!0-9]*) die "--line must be numeric: $LINE" ;; esac +[ "$LINE" -le "$(wc -l < "$SOURCE")" ] || die "--line $LINE is past the end of $SOURCE" + +LABEL="${LABEL:-$(basename "$SOURCE" | sed 's/\.[^.]*$//')-L$LINE}" +mkdir -p "$OUT_DIR" || die "cannot create $OUT_DIR" +STAMP="$OUT_DIR/falsify-$LABEL" + +# --- environment pin: two operators on different machines must be comparable --- +HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" +DIRTY="$(git status --porcelain 2>/dev/null | grep -v '^??' | wc -l | tr -d ' ')" +NODE_V="$(node -v 2>/dev/null || echo unknown)" +LOCK_SHA="$( { sha256sum yarn.lock 2>/dev/null || shasum -a 256 yarn.lock 2>/dev/null; } | cut -c1-16)" +ORIGINAL_LINE="$(sed -n "${LINE}p" "$SOURCE")" + +BACKUP="$(mktemp)" || die "mktemp failed" +cp "$SOURCE" "$BACKUP" +restore() { cp "$BACKUP" "$SOURCE"; rm -f "$BACKUP"; } +trap restore EXIT INT TERM + +run_arm() { # $1=logfile ; prints "passed|failed" + if $RUNNER "$TEST" > "$1" 2>&1; then echo passed; else echo failed; fi +} + +ARM_A="$(run_arm "$STAMP-armA.log")" + +if [ "$ARM_A" != "passed" ]; then + VERDICT="broken"; CODE=2; ARM_B="not-run" + : > "$STAMP-armB.log" +else + # Mutate exactly one line. `.bak` form keeps this portable across GNU/BSD sed. + awk -v n="$LINE" -v r="$REPLACE" 'NR==n{print r; next}{print}' "$SOURCE" > "$SOURCE.tmp" \ + && mv "$SOURCE.tmp" "$SOURCE" || die "mutation failed" + ARM_B="$(run_arm "$STAMP-armB.log")" + restore; trap - EXIT INT TERM + if [ "$ARM_B" = "failed" ]; then VERDICT="falsifying"; CODE=0; else VERDICT="vacuous"; CODE=1; fi +fi + +summarise() { grep -E '^(Tests|Test Suites):' "$1" 2>/dev/null | tr '\n' ' ' | sed 's/ */ /g'; } +A_SUM="$(summarise "$STAMP-armA.log")" +B_SUM="$(summarise "$STAMP-armB.log")" +FAILED_NAMES="$(grep -E '^\s+●[^›]*›' "$STAMP-armB.log" 2>/dev/null | sed 's/^ *//' | head -10)" + +cat > "$STAMP.json" <<JSON +{ + "verdict": "$VERDICT", + "exit": $CODE, + "test": "$TEST", + "mutation": { "source": "$SOURCE", "line": $LINE, + "from": $(printf '%s' "$ORIGINAL_LINE" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))'), + "to": $(printf '%s' "$REPLACE" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))') }, + "armA": { "result": "$ARM_A", "summary": "$A_SUM", "log": "$STAMP-armA.log" }, + "armB": { "result": "$ARM_B", "summary": "$B_SUM", "log": "$STAMP-armB.log" }, + "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, "node": "$NODE_V", "yarn_lock_sha256_16": "$LOCK_SHA" } +} +JSON + +{ + echo "### Falsification probe — \`$VERDICT\`" + echo + echo "| Arm | Mutation | Result |" + echo "|---|---|---|" + echo "| A — baseline | none | \`$A_SUM\` |" + echo "| B — mutant | \`$SOURCE:$LINE\` replaced | \`$B_SUM\` |" + echo + case "$VERDICT" in + falsifying) echo "The suite **fails when the mechanism is removed** and passes when restored. The test has power." ;; + vacuous) echo "The suite **passes with the mechanism removed**. It does not test what it appears to test." ;; + broken) echo "Arm A did not pass, so arm B was not run. No conclusion." ;; + esac + [ -n "$FAILED_NAMES" ] && { echo; echo "Failing under mutation:"; echo; printf '%s\n' "$FAILED_NAMES" | sed 's/^/- /'; } + echo + echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" +} > "$STAMP.md" + +printf 'falsify-probe: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 +exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index d88a6c98..1877eb77 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -177,6 +177,29 @@ untested path touches funds, keys, persisted state, user-visible wrongness, or s planning tracker. Subject matter triggers this, not severity: the code cannot distinguish a missing gate from a deliberate one. +### The runner, not the recipe + +`scripts/falsify-probe.sh` proves a test is falsifying by mutation rather than by reading, and +**writes the artifact itself** — the operator never transcribes output: + +```bash +scripts/falsify-probe.sh \ + --test ui/hooks/perps/coalesceBackgroundRequest.test.ts \ + --source ui/hooks/perps/coalesceBackgroundRequest.ts \ + --line 54 --replace ' const existing = undefined as Promise<TResult> | undefined;' +``` + +Runs arm A, mutates one line, runs arm B, restores the source, and emits +`evidence-artifacts/falsify-<label>.{json,md}` plus both raw logs. **The exit code is the +verdict**, so CI can gate on it: `0` falsifying · `1` vacuous · `2` arm A already failing · `3` +usage error. + +Every artifact pins `HEAD`, node version, `yarn.lock` hash, and the tracked-change count, so two +operators on different machines produce comparable results or visibly do not. + +Prefer this over a hand-run test in every case. A hand-run test yields a number you then retype, +which returns the provenance to you and reintroduces exactly the problem the probe solves. + ### Canonical output shape Every validation-run output — PR comment *or* PR-body section — uses exactly this, so re-runs From 559b4c074a401fb52dd7421f779f9ce6ef7ec2d4 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 21:48:31 -0400 Subject: [PATCH 15/63] Add `capture.sh` so the C9 and D3 analyses stop needing an operator to retype them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `retention-scan.py` and `policy-audit.py` already do the analysis well; both print to stdout, which makes the operator the capture device and returns provenance to whoever pasted the output. This wraps any command so the tool writes the artifact. Emits <label>.log verbatim, plus .json and an attachable .md that quotes the log rather than summarising it, with HEAD, tracked-change count, node, python, and yarn.lock hash pinned in each. The wrapped exit code passes through for CI. `--verdict` is stated by the caller, never inferred from the exit code. The first run of this script proved why: it labelled a policy audit "pass" while the output listed sixteen newly granted capabilities, because policy-audit.py exits 0 regardless. With no --verdict it now says "ran to completion, no verdict asserted". Verified on both scripts against real PRs, including a deliberately wrong invocation — which produces an artifact containing the traceback rather than a fabricated finding. --- .../skills/evidence/scripts/capture.sh | 112 ++++++++++++++++++ domains/pr-workflow/skills/evidence/skill.md | 23 ++++ 2 files changed, 135 insertions(+) create mode 100755 domains/pr-workflow/skills/evidence/scripts/capture.sh diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh new file mode 100755 index 00000000..064ed404 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -0,0 +1,112 @@ +#!/usr/bin/env bash +# +# capture — turn any analysis command into a contract-compliant evidence artifact. +# +# The analysis scripts in this repo (retention-scan.py, policy-audit.py, a jest +# probe, a selector recomputation counter) all print to stdout. Printing to stdout +# means the operator is the capture device: they read it, retype some of it into a +# comment, and the result carries their provenance rather than the measurement's. +# +# This wraps any command so the ARTIFACT is written by the tool. Nothing is retyped. +# +# capture.sh --label <slug> --lane <id> --claim "<under test>" [--verdict <word>] -- <cmd...> +# +# --verdict is stated by the caller, never inferred from the exit code: a wrapped +# tool's exit convention is its own, and guessing prints "pass" over real findings. +# +# Emits, under --out (default evidence-artifacts/): +# <label>.log raw stdout+stderr of the command, unmodified +# <label>.json machine-readable: verdict, exit code, env pin, claim +# <label>.md the block to attach, quoting the log rather than summarising it +# +# Exit code is the wrapped command's own, so CI gates on it unchanged. +# +# Example: +# capture.sh --label defi-retention --lane C9 \ +# --claim "every retention primitive this diff introduces is released" \ +# -- python3 retention-scan.py ui/store/background-connection.ts pr.patch +set -uo pipefail + +OUT_DIR="evidence-artifacts"; LABEL=""; LANE=""; CLAIM=""; MAXLOG=120; VERDICT="" +die() { printf 'capture: %s\n' "$1" >&2; exit 3; } + +while [ $# -gt 0 ]; do + case "$1" in + --label) LABEL="${2:-}"; shift 2 ;; + --lane) LANE="${2:-}"; shift 2 ;; + --claim) CLAIM="${2:-}"; shift 2 ;; + --verdict) VERDICT="${2:-}"; shift 2 ;; + --out) OUT_DIR="${2:-}"; shift 2 ;; + --max-log-lines) MAXLOG="${2:-}"; shift 2 ;; + -h|--help) sed -n '2,26p' "$0"; exit 0 ;; + --) shift; break ;; + *) die "unknown argument: $1 (did you forget -- before the command?)" ;; + esac +done + +[ -n "$LABEL" ] || die "--label is required" +[ -n "$CLAIM" ] || die "--claim is required: name the falsifiable thing under test" +[ $# -gt 0 ] || die "no command given after --" + +mkdir -p "$OUT_DIR" || die "cannot create $OUT_DIR" +STAMP="$OUT_DIR/$LABEL" + +HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" +DIRTY="$(git status --porcelain 2>/dev/null | grep -v '^??' | wc -l | tr -d ' ')" +NODE_V="$(node -v 2>/dev/null || echo n/a)" +PY_V="$(python3 -V 2>&1 || echo n/a)" +LOCK_SHA="$( { sha256sum yarn.lock 2>/dev/null || shasum -a 256 yarn.lock 2>/dev/null; } | cut -c1-16)" +[ -n "$LOCK_SHA" ] || LOCK_SHA="n/a" +CMD_STR="$*" + +# Run it. Never interpret the output — capture it verbatim. +"$@" > "$STAMP.log" 2>&1 +CODE=$? + +LINES="$(wc -l < "$STAMP.log" | tr -d ' ')" +# No verdict is inferred from the exit code. A wrapped tool's convention is its own — +# policy-audit.py exits 0 while listing sixteen new capability grants, so guessing here +# would print "pass" over a page of findings. The caller states the verdict or none is claimed. +[ -n "$VERDICT" ] || VERDICT="completed" + +jstr() { printf '%s' "${1-}" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))'; } + +cat > "$STAMP.json" <<JSON +{ + "label": $(jstr "$LABEL"), + "lane": $(jstr "$LANE"), + "claim": $(jstr "$CLAIM"), + "command": $(jstr "$CMD_STR"), + "verdict": "$VERDICT", + "exit": $CODE, + "log": "$STAMP.log", + "log_lines": $LINES, + "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, + "node": "$NODE_V", "python": "$PY_V", "yarn_lock_sha256_16": "$LOCK_SHA" } +} +JSON + +{ + if [ "$VERDICT" = "completed" ]; then + echo "### ${LANE:+$LANE — }ran to completion (exit $CODE) — read the output, no verdict asserted" + else + echo "### ${LANE:+$LANE — }\`$VERDICT\` (exit $CODE)" + fi + echo + echo "**Claim under test:** $CLAIM" + echo + echo '```console' + echo "\$ $CMD_STR" + if [ "$LINES" -gt "$MAXLOG" ]; then + head -n "$MAXLOG" "$STAMP.log" + echo "… $((LINES - MAXLOG)) further lines in $STAMP.log" + else + cat "$STAMP.log" + fi + echo '```' + echo + echo "<sub>Captured by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. Raw log: \`$STAMP.log\`.</sub>" +} > "$STAMP.md" + +printf 'capture: %s (exit %s)\n %s\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.log" "$STAMP.json" "$STAMP.md" >&2 +exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 1877eb77..aba3c70f 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -200,6 +200,29 @@ operators on different machines produce comparable results or visibly do not. Prefer this over a hand-run test in every case. A hand-run test yields a number you then retype, which returns the provenance to you and reintroduces exactly the problem the probe solves. +### `capture.sh` — for every lane that already has an analysis script + +`retention-scan.py` (C9), `policy-audit.py` (D3), and any jest or selector probe all print to +stdout, which makes the operator the capture device. Wrap them instead: + +```bash +scripts/capture.sh --label bgconn-retention --lane "C9 retention-path analysis" \ + --claim "every retention primitive this diff introduces is paired with a release" \ + -- python3 retention-scan.py "ui/store/background-connection.ts:pr.patch" +``` + +Writes `<label>.log` (verbatim), `<label>.json`, and `<label>.md` — the attachable block, quoting +the log rather than summarising it — with `HEAD`, tracked-change count, node, python, and +`yarn.lock` hash pinned in each. The wrapped command's exit code passes through unchanged. + +**`--verdict` is stated by the caller, never inferred from the exit code.** A wrapped tool's exit +convention is its own: `policy-audit.py` exits `0` while listing sixteen newly granted +capabilities, so inferring would print "pass" over a page of findings. With no `--verdict`, the +artifact says *ran to completion — read the output, no verdict asserted*, which is the honest +default. + +A crashing command produces an artifact containing the traceback, not a fabricated result. + ### Canonical output shape Every validation-run output — PR comment *or* PR-body section — uses exactly this, so re-runs From 85a575c3a7fbe591aa2a8f65536e8951a2b8d60f Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 22:08:46 -0400 Subject: [PATCH 16/63] =?UTF-8?q?Add=20`selector-recompute.sh`=20=E2=80=94?= =?UTF-8?q?=20lane=20C4=20gets=20a=20runner?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A memoization claim is a claim about a count, and reselect publishes the count. This generates a probe, runs it, deletes it, and writes the artifact, so the number never passes through an operator's hands. Three conditions; the middle one discriminates. A selector built on narrowed input selectors is unmoved by a write it does not read, while one taking `state.metamask` wholesale recomputes on every unrelated write in the app. Verified against both shapes on main, so the runner is shown to distinguish them rather than only to report success: getWalletsWithAccounts 1 / 1 / 6 narrowed selectRampsControllerState 1 / 6 / 11 recomputes on unrelated writes That completes runner coverage for the catalog lanes with engine skills: B3 and B7 via falsify-probe, C4 here, C9 and D3 via capture around their existing analysis scripts. --- .../evidence/scripts/selector-recompute.sh | 151 ++++++++++++++++++ domains/pr-workflow/skills/evidence/skill.md | 24 +++ 2 files changed, 175 insertions(+) create mode 100755 domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh new file mode 100755 index 00000000..5abb05b9 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -0,0 +1,151 @@ +#!/usr/bin/env bash +# +# selector-recompute — measure how often a reselect selector actually recomputes. +# +# Lane C4. A memoization claim ("avoids recomputation", "stops deep traversal", +# "prevents re-renders") is a claim about a COUNT. `reselect` exposes that count +# natively via `.recomputations()`, so no instrumentation and no profiler is +# needed — and no operator judgement either. +# +# Generates a throwaway probe test, runs it, captures the counter under three +# conditions, removes the probe, and writes the artifact itself. +# +# A identical state reference, repeated → memoized floor (expect 1) +# B fresh enclosing slice, unrelated field changed → does an unrelated write cost a recompute? +# C a real input key perturbed → does a relevant write cost one? (expect +1 each) +# +# B is the discriminating condition. A selector taking narrowed inputs is +# unmoved by B; one reading a whole slice recomputes on every unrelated write. +# +# Usage: +# selector-recompute.sh --module <import path> --export <name> \ +# --fixture <json path> --slice <key> --perturb <key> [--n 5] [--label <slug>] +# +# Example: +# selector-recompute.sh \ +# --module ui/selectors/multichain-accounts/account-tree \ +# --export getWalletsWithAccounts \ +# --fixture test/data/mock-state.json --slice metamask --perturb pinnedAccountList +set -uo pipefail + +N=5; OUT_DIR="evidence-artifacts"; LABEL=""; MODULE=""; EXPORT=""; FIXTURE=""; SLICE="metamask"; PERTURB="" +die() { printf 'selector-recompute: %s\n' "$1" >&2; exit 3; } + +while [ $# -gt 0 ]; do + case "$1" in + --module) MODULE="${2:-}"; shift 2 ;; + --export) EXPORT="${2:-}"; shift 2 ;; + --fixture) FIXTURE="${2:-}"; shift 2 ;; + --slice) SLICE="${2:-}"; shift 2 ;; + --perturb) PERTURB="${2:-}"; shift 2 ;; + --n) N="${2:-}"; shift 2 ;; + --label) LABEL="${2:-}"; shift 2 ;; + --out) OUT_DIR="${2:-}"; shift 2 ;; + -h|--help) sed -n '2,27p' "$0"; exit 0 ;; + *) die "unknown argument: $1" ;; + esac +done + +[ -n "$MODULE" ] || die "--module is required (import path, no extension)" +[ -n "$EXPORT" ] || die "--export is required (the selector's exported name)" +[ -n "$FIXTURE" ] || die "--fixture is required (a JSON state fixture)" +[ -n "$PERTURB" ] || die "--perturb is required (an input key the selector genuinely reads)" +[ -f "$FIXTURE" ] || die "fixture not found: $FIXTURE" +[ -f "$MODULE.ts" ] || [ -f "$MODULE.js" ] || die "module not found: $MODULE.{ts,js}" + +LABEL="${LABEL:-recompute-$EXPORT}" +mkdir -p "$OUT_DIR" || die "cannot create $OUT_DIR" +STAMP="$OUT_DIR/$LABEL" +PROBE="$(dirname "$MODULE")/__recompute_probe__.test.ts" + +# Relative import from the probe back to the module, and up to the fixture. +MOD_BASE="./$(basename "$MODULE")" +DEPTH="$(dirname "$MODULE" | tr -cd '/' | wc -c | tr -d ' ')" +UP=""; i=0; while [ "$i" -le "$DEPTH" ]; do UP="../$UP"; i=$((i+1)); done + +cleanup() { rm -f "$PROBE"; } +trap cleanup EXIT INT TERM + +cat > "$PROBE" <<PROBEEOF +import { $EXPORT } from '$MOD_BASE'; +import fixture from '$UP$FIXTURE'; + +describe('$EXPORT recomputation probe', () => { + it('counts recomputations across three conditions', () => { + const base = fixture as never as { $SLICE: Record<string, unknown> }; + const call = (s: unknown) => ($EXPORT as (x: never) => unknown)(s as never); + + ($EXPORT as unknown as { resetRecomputations: () => void }).resetRecomputations(); + const count = () => ($EXPORT as unknown as { recomputations: () => number }).recomputations(); + + for (let i = 0; i < $N; i++) call(base); + const a = count(); + + for (let i = 0; i < $N; i++) { + call({ ...base, $SLICE: { ...base.$SLICE, __unrelated__: i } }); + } + const b = count(); + + for (let i = 0; i < $N; i++) { + call({ ...base, $SLICE: { ...base.$SLICE, $PERTURB: [\`0x\${i}\`] } }); + } + const c = count(); + + // eslint-disable-next-line no-console + console.log(\`RECOMPUTE_PROBE identical=\${a} unrelated=\${b} inputChanged=\${c} n=$N\`); + expect(c).toBeGreaterThanOrEqual(b); + }); +}); +PROBEEOF + +yarn jest "$PROBE" > "$STAMP.log" 2>&1 +CODE=$? +cleanup; trap - EXIT INT TERM + +LINE="$(grep -o 'RECOMPUTE_PROBE .*' "$STAMP.log" | head -1)" +A="$(printf '%s' "$LINE" | sed -n 's/.*identical=\([0-9]*\).*/\1/p')" +B="$(printf '%s' "$LINE" | sed -n 's/.*unrelated=\([0-9]*\).*/\1/p')" +C="$(printf '%s' "$LINE" | sed -n 's/.*inputChanged=\([0-9]*\).*/\1/p')" + +if [ -z "$A" ]; then + VERDICT="probe-failed" +elif [ "$B" -gt "$A" ]; then + VERDICT="recomputes on unrelated writes" +else + VERDICT="narrowed — unrelated writes cost nothing" +fi + +HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" +DIRTY="$(git status --porcelain 2>/dev/null | grep -v '^??' | wc -l | tr -d ' ')" +NODE_V="$(node -v 2>/dev/null || echo unknown)" + +cat > "$STAMP.json" <<JSON +{ "selector": "$EXPORT", "module": "$MODULE", "verdict": "$VERDICT", "exit": $CODE, + "n_calls_per_condition": $N, + "recomputations": { "identical": ${A:-null}, "unrelated_write": ${B:-null}, "input_changed": ${C:-null} }, + "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, "node": "$NODE_V" }, + "log": "$STAMP.log" } +JSON + +{ + echo "### C4 — \`$EXPORT\` recomputation count" + echo + echo "**Verdict:** $VERDICT" + echo + echo "| Condition | Calls | Recomputations |" + echo "|---|---|---|" + echo "| Identical state reference | $N | ${A:-?} |" + echo "| Fresh \`$SLICE\` slice, unrelated field | $N | ${B:-?} |" + echo "| \`$PERTURB\` changed (a real input) | $N | ${C:-?} |" + echo + echo '```console' + echo "\$ yarn jest <generated probe>" + echo "$LINE" + echo '```' + echo + echo "<sub>Measured by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Log: \`$STAMP.log\`.</sub>" +} > "$STAMP.md" + +printf 'selector-recompute: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 +[ -n "$A" ] || exit 2 +exit 0 diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index aba3c70f..e384f4a8 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -223,6 +223,30 @@ default. A crashing command produces an artifact containing the traceback, not a fabricated result. +### `selector-recompute.sh` — lane C4 + +A memoization claim is a claim about a count, and `reselect` publishes the count. Generates a +probe, runs it, deletes it, writes the artifact: + +```bash +scripts/selector-recompute.sh --module ui/selectors/multichain-accounts/account-tree \ + --export getWalletsWithAccounts --fixture test/data/mock-state.json \ + --slice metamask --perturb pinnedAccountList +``` + +Three conditions, of which the middle one discriminates: + +| Condition | narrowed inputs | whole-slice input | +|---|---|---| +| identical state reference | 1 | 1 | +| fresh slice, **unrelated** field | **1** | **6** | +| a real input changed | 6 | 11 | + +A selector taking narrowed input selectors is unmoved by an unrelated write; one reading +`state.metamask` wholesale recomputes on every unrelated write in the app. Both rows above are +measured, not illustrative — `getWalletsWithAccounts` and `selectRampsControllerState` on +`main`. + ### Canonical output shape Every validation-run output — PR comment *or* PR-body section — uses exactly this, so re-runs From 94a764079f9bb037328e8951b32e1e8177c8cb87 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 22:16:33 -0400 Subject: [PATCH 17/63] =?UTF-8?q?Add=20`tsc-substitution.sh`=20=E2=80=94?= =?UTF-8?q?=20a=20runner=20for=20the=20tsc-blindspots=20lane?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Whether a hand-written type agrees with the source it restates is a question only the compiler can settle. Arm A typechecks the baseline, arm B applies the substitution, and the finding is the error diff. Source is restored on exit, including on interrupt. Carries the warning the lane most needs: a silent arm B is not proof of agreement. Indexing and `.match()` compile against `string` and `string[]` alike, so without `--probe` injecting a deliberately-typed sink, the lane reports false clean on exactly the divergence it exists to find. When arm A already fails it stops and says nothing was established, alongside the module/export error count — but it does not classify from that ratio. The first real run had 124 of 280 errors as install artifacts while tripping no majority rule, because other codes are downstream of the same missing types. A threshold there would be a number I could not justify, so it reports the breakdown and leaves the judgement with the operator. --- .../evidence/scripts/tsc-substitution.sh | 140 ++++++++++++++++++ domains/pr-workflow/skills/evidence/skill.md | 25 ++++ 2 files changed, 165 insertions(+) create mode 100755 domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh new file mode 100755 index 00000000..23e84c2d --- /dev/null +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -0,0 +1,140 @@ +#!/usr/bin/env bash +# +# tsc-substitution — arm A/B against the type checker. +# +# A hand-written type that restates an authoritative source either agrees with it +# or does not, and `tsc` is the only thing that can settle which. Reading the two +# declarations side by side does not: TypeScript's assignability rules are not +# obvious by inspection, which is the entire reason the lane exists. +# +# Arm A baseline typecheck, errors recorded +# Arm B the substitution applied — the hand-written type replaced by the derived +# one, or a cast removed — typecheck re-run, errors diffed +# +# The finding is the DIFF: error codes present in B and absent in A are what the +# hand-written type or the cast was concealing. +# +# 0 divergence surfaced new errors in arm B → the local type disagrees +# 1 no divergence identical error sets → substitution is silent +# 2 arm A already failing → nothing to conclude +# 3 usage/env error +# +# A silent result is NOT proof of agreement. Existing call sites may type-check +# against both shapes (indexing a `string` and a `string[]` both compile), so use +# --probe to inject a deliberately-typed sink that only one shape satisfies. +# +# Usage: +# tsc-substitution.sh --file <path> --line <n> --replace <text> +# [--probe-line <n> --probe <text>] [--label <slug>] +# [--tsc "<command>"] +set -uo pipefail + +TSC="yarn lint:tsc"; OUT_DIR="evidence-artifacts"; LABEL="" +FILE=""; LINE=""; REPLACE=""; PROBE_LINE=""; PROBE="" +die() { printf 'tsc-substitution: %s\n' "$1" >&2; exit 3; } + +while [ $# -gt 0 ]; do + case "$1" in + --file) FILE="${2:-}"; shift 2 ;; + --line) LINE="${2:-}"; shift 2 ;; + --replace) REPLACE="${2:-}"; shift 2 ;; + --probe-line) PROBE_LINE="${2:-}"; shift 2 ;; + --probe) PROBE="${2:-}"; shift 2 ;; + --label) LABEL="${2:-}"; shift 2 ;; + --out) OUT_DIR="${2:-}"; shift 2 ;; + --tsc) TSC="${2:-}"; shift 2 ;; + -h|--help) sed -n '2,30p' "$0"; exit 0 ;; + *) die "unknown argument: $1" ;; + esac +done + +[ -n "$FILE" ] || die "--file is required" +[ -f "$FILE" ] || die "file not found: $FILE" +[ -n "$LINE" ] || [ -n "$PROBE" ] || die "give --line/--replace, or --probe-line/--probe, or both" +LABEL="${LABEL:-tsc-$(basename "$FILE" | sed 's/\.[^.]*$//')}" +mkdir -p "$OUT_DIR" || die "cannot create $OUT_DIR" +STAMP="$OUT_DIR/$LABEL" + +BACKUP="$(mktemp)" || die "mktemp failed" +cp "$FILE" "$BACKUP" +restore() { cp "$BACKUP" "$FILE"; rm -f "$BACKUP"; } +trap restore EXIT INT TERM + +# tsc exits non-zero on any error, so the error SET is the signal, not the exit code. +errors_of() { grep -oE "error TS[0-9]+" "$1" 2>/dev/null | sort | uniq -c | sed 's/^ *//'; } + +$TSC > "$STAMP-armA.log" 2>&1 +A_ERRS="$(errors_of "$STAMP-armA.log")" +A_COUNT="$(grep -c "error TS" "$STAMP-armA.log" 2>/dev/null || echo 0)" + +if [ "$A_COUNT" -gt 0 ]; then + # Distinguish a genuinely failing repo from an incomplete local install. A baseline + # dominated by TS2305/TS2724/TS2307 ("has no exported member" / "cannot find module") + # means dependency types were never generated — `yarn install --mode=skip-build` does + # exactly this — and says nothing about the code. Reporting both as "baseline failing" + # would send the operator hunting a repo defect that is not there. + # Report the module/export share; do not classify from it. A threshold here would be + # a number I cannot justify — 124/280 on this repo is plainly an install artifact, yet + # trips no majority rule, because TS2339 and TS7006 are themselves downstream of the + # missing types. Surface the signal, leave the judgement with the operator. + ENVISH="$(grep -coE "error TS(2305|2307|2724)" "$STAMP-armA.log" || echo 0)" + VERDICT="baseline failing — no conclusion available" + CODE=2; B_COUNT="not-run"; NEW_ERRS="" + : > "$STAMP-armB.log" +else + # Apply substitution and/or probe, highest line first so numbering holds. + apply() { awk -v n="$1" -v r="$2" 'NR==n{print r; next}{print}' "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"; } + insert() { awk -v n="$1" -v r="$2" 'NR==n{print; print r; next}{print}' "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"; } + if [ -n "$PROBE" ] && [ -n "$PROBE_LINE" ] && [ -n "$LINE" ] && [ "$PROBE_LINE" -gt "$LINE" ]; then + insert "$PROBE_LINE" "$PROBE"; apply "$LINE" "$REPLACE" + else + [ -n "$LINE" ] && apply "$LINE" "$REPLACE" + [ -n "$PROBE" ] && [ -n "$PROBE_LINE" ] && insert "$PROBE_LINE" "$PROBE" + fi + + $TSC > "$STAMP-armB.log" 2>&1 + B_COUNT="$(grep -c "error TS" "$STAMP-armB.log" 2>/dev/null || echo 0)" + NEW_ERRS="$(grep -oE "error TS[0-9]+.*" "$STAMP-armB.log" 2>/dev/null | sort -u | head -12)" + restore; trap - EXIT INT TERM + if [ "$B_COUNT" -gt 0 ]; then VERDICT="divergence surfaced"; CODE=0; else VERDICT="substitution silent"; CODE=1; fi +fi + +HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" +DIRTY="$(git status --porcelain 2>/dev/null | grep -v '^??' | wc -l | tr -d ' ')" +TS_V="$(yarn tsc --version 2>/dev/null | tail -1 || echo unknown)" + +cat > "$STAMP.json" <<JSON +{ "verdict": "$VERDICT", "exit": $CODE, "file": "$FILE", + "arm_a_errors": $A_COUNT, "arm_b_errors": ${B_COUNT:-null}, + "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, "typescript": "$TS_V" }, + "logs": ["$STAMP-armA.log", "$STAMP-armB.log"] } +JSON + +{ + echo "### D6 — authored-vs-authoritative substitution · \`$VERDICT\`" + echo + echo "| Arm | Change | \`tsc\` errors |" + echo "|---|---|---|" + echo "| A — baseline | none | $A_COUNT |" + echo "| B — substituted | \`$FILE\`${LINE:+:$LINE}${PROBE:+ + typed sink} | ${B_COUNT} |" + echo + if [ "$CODE" = "2" ] && [ "${ENVISH:-0}" -gt 0 ]; then + echo "Arm A did not pass, so arm B was not run and **nothing about the types is established**." + echo + echo "\`$ENVISH\` of \`$A_COUNT\` baseline errors are module/export resolution" + echo "(TS2305/TS2307/TS2724). Those usually mean dependency types were never generated —" + echo "a skipped install step — rather than a defect in this repo, and other codes can be" + echo "downstream of the same cause. Confirm the toolchain is complete before reading" + echo "anything into this lane." + elif [ -n "$NEW_ERRS" ]; then + echo "Errors surfaced only under substitution:"; echo; echo '```'; printf '%s\n' "$NEW_ERRS"; echo '```' + elif [ "$CODE" = "1" ]; then + echo "**Silent — this is not proof of agreement.** Existing call sites may satisfy both shapes." + echo "Re-run with \`--probe\` to inject a sink only the authoritative type accepts." + fi + echo + echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" +} > "$STAMP.md" + +printf 'tsc-substitution: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 +exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index e384f4a8..6a258b30 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -247,6 +247,31 @@ A selector taking narrowed input selectors is unmoved by an unrelated write; one measured, not illustrative — `getWalletsWithAccounts` and `selectRampsControllerState` on `main`. +### `tsc-substitution.sh` — lane D6 (`tsc-blindspots`) + +Whether a hand-written type agrees with the source it restates is a question only the compiler +can settle; assignability is not obvious by inspection, which is the reason the lane exists. + +```bash +scripts/tsc-substitution.sh --file shared/lib/transactions-controller-utils.ts \ + --line 146 --replace ' topics?: string;' \ + --probe-line 150 --probe ' const _probe: string[] = txReceiptLogs[0].topics;' +``` + +Arm A typechecks the baseline, arm B applies the substitution, and the finding is the **error +diff**. Source is restored on exit including on interrupt. + +**A silent arm B is not proof of agreement.** Existing call sites often satisfy both shapes — +indexing and `.match()` compile against `string` and `string[]` alike — so use `--probe` to +inject a deliberately-typed sink that only the authoritative shape accepts. Without one this +lane reports false clean. + +If arm A already fails, the run stops and states that nothing was established, alongside the +count of module/export errors (TS2305/TS2307/TS2724), which usually indicate an incomplete +install rather than a repo defect. It does not classify from that ratio — on a real run 124 of +280 errors were install artifacts while tripping no majority rule, because other codes are +downstream of the same cause. + ### Canonical output shape Every validation-run output — PR comment *or* PR-body section — uses exactly this, so re-runs From 6e38d6da9e4b5a706bb6fd20c7f11417b0aa3650 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 22:31:28 -0400 Subject: [PATCH 18/63] Diff the error sets rather than requiring a clean baseline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Gating on arm A being error-free was wrong: the finding was always the diff, so a single unrelated pre-existing error — a local work-in-progress file, in the run that exposed this — vetoed the whole lane. Baseline errors are now subtracted and only errors new under substitution count. Verified end to end on shared/lib/transactions-controller-utils.ts, where the local `LogWithTopicsArray` declares `topics?: string[]` against an upstream `topics?: string`: substitution alone 1 -> 1 errors, 0 new silent substitution + typed sink 1 -> 2 errors, 1 new TS2322 at the sink That is the lane's central caution demonstrated rather than asserted. Indexing and `.match()` compile against both shapes, so the obvious probe reports a false clean; only a deliberately-typed sink surfaces the divergence. A silent arm B means the probe was too weak, not that the types agree. Also fixes a `grep -c ... || echo 0` double-fire that produced "0\n0" and an integer-comparison error — the same shape already fixed once in this script's tracked-change count. --- .../evidence/scripts/tsc-substitution.sh | 88 +++++++++---------- 1 file changed, 41 insertions(+), 47 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh index 23e84c2d..4dc4ea81 100755 --- a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -16,7 +16,7 @@ # # 0 divergence surfaced new errors in arm B → the local type disagrees # 1 no divergence identical error sets → substitution is silent -# 2 arm A already failing → nothing to conclude +# 2 usage/env error (baseline errors are subtracted, not disqualifying) # 3 usage/env error # # A silent result is NOT proof of agreement. Existing call sites may type-check @@ -64,39 +64,34 @@ trap restore EXIT INT TERM errors_of() { grep -oE "error TS[0-9]+" "$1" 2>/dev/null | sort | uniq -c | sed 's/^ *//'; } $TSC > "$STAMP-armA.log" 2>&1 -A_ERRS="$(errors_of "$STAMP-armA.log")" -A_COUNT="$(grep -c "error TS" "$STAMP-armA.log" 2>/dev/null || echo 0)" +# Baseline errors are subtracted, not disqualifying. A local WIP file or an +# unrelated pre-existing error must not veto the lane — the finding was always the +# DIFF, so compare error SETS and let anything already present fall out. +grep -oE "^[^ ]+\([0-9]+,[0-9]+\): error TS[0-9]+" "$STAMP-armA.log" 2>/dev/null | sort -u > "$STAMP-armA.set" +A_COUNT="$(wc -l < "$STAMP-armA.set" | tr -d ' ')" +ENVISH="$(grep -cE "error TS(2305|2307|2724)" "$STAMP-armA.log" 2>/dev/null)"; ENVISH="${ENVISH:-0}" -if [ "$A_COUNT" -gt 0 ]; then - # Distinguish a genuinely failing repo from an incomplete local install. A baseline - # dominated by TS2305/TS2724/TS2307 ("has no exported member" / "cannot find module") - # means dependency types were never generated — `yarn install --mode=skip-build` does - # exactly this — and says nothing about the code. Reporting both as "baseline failing" - # would send the operator hunting a repo defect that is not there. - # Report the module/export share; do not classify from it. A threshold here would be - # a number I cannot justify — 124/280 on this repo is plainly an install artifact, yet - # trips no majority rule, because TS2339 and TS7006 are themselves downstream of the - # missing types. Surface the signal, leave the judgement with the operator. - ENVISH="$(grep -coE "error TS(2305|2307|2724)" "$STAMP-armA.log" || echo 0)" - VERDICT="baseline failing — no conclusion available" - CODE=2; B_COUNT="not-run"; NEW_ERRS="" - : > "$STAMP-armB.log" +apply() { awk -v n="$1" -v r="$2" 'NR==n{print r; next}{print}' "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"; } +insert() { awk -v n="$1" -v r="$2" 'NR==n{print; print r; next}{print}' "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"; } +if [ -n "$PROBE" ] && [ -n "$PROBE_LINE" ] && [ -n "$LINE" ] && [ "$PROBE_LINE" -gt "$LINE" ]; then + insert "$PROBE_LINE" "$PROBE"; apply "$LINE" "$REPLACE" else - # Apply substitution and/or probe, highest line first so numbering holds. - apply() { awk -v n="$1" -v r="$2" 'NR==n{print r; next}{print}' "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"; } - insert() { awk -v n="$1" -v r="$2" 'NR==n{print; print r; next}{print}' "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"; } - if [ -n "$PROBE" ] && [ -n "$PROBE_LINE" ] && [ -n "$LINE" ] && [ "$PROBE_LINE" -gt "$LINE" ]; then - insert "$PROBE_LINE" "$PROBE"; apply "$LINE" "$REPLACE" - else - [ -n "$LINE" ] && apply "$LINE" "$REPLACE" - [ -n "$PROBE" ] && [ -n "$PROBE_LINE" ] && insert "$PROBE_LINE" "$PROBE" - fi + [ -n "$LINE" ] && apply "$LINE" "$REPLACE" + [ -n "$PROBE" ] && [ -n "$PROBE_LINE" ] && insert "$PROBE_LINE" "$PROBE" +fi + +$TSC > "$STAMP-armB.log" 2>&1 +grep -oE "^[^ ]+\([0-9]+,[0-9]+\): error TS[0-9]+" "$STAMP-armB.log" 2>/dev/null | sort -u > "$STAMP-armB.set" +B_COUNT="$(wc -l < "$STAMP-armB.set" | tr -d ' ')" +restore; trap - EXIT INT TERM + +NEW_ERRS="$(comm -13 "$STAMP-armA.set" "$STAMP-armB.set" | head -12)" +NEW_COUNT="$(comm -13 "$STAMP-armA.set" "$STAMP-armB.set" | wc -l | tr -d ' ')" - $TSC > "$STAMP-armB.log" 2>&1 - B_COUNT="$(grep -c "error TS" "$STAMP-armB.log" 2>/dev/null || echo 0)" - NEW_ERRS="$(grep -oE "error TS[0-9]+.*" "$STAMP-armB.log" 2>/dev/null | sort -u | head -12)" - restore; trap - EXIT INT TERM - if [ "$B_COUNT" -gt 0 ]; then VERDICT="divergence surfaced"; CODE=0; else VERDICT="substitution silent"; CODE=1; fi +if [ "$NEW_COUNT" -gt 0 ]; then + VERDICT="divergence surfaced"; CODE=0 +else + VERDICT="substitution silent"; CODE=1 fi HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" @@ -105,7 +100,7 @@ TS_V="$(yarn tsc --version 2>/dev/null | tail -1 || echo unknown)" cat > "$STAMP.json" <<JSON { "verdict": "$VERDICT", "exit": $CODE, "file": "$FILE", - "arm_a_errors": $A_COUNT, "arm_b_errors": ${B_COUNT:-null}, + "arm_a_errors": $A_COUNT, "arm_b_errors": $B_COUNT, "new_under_substitution": $NEW_COUNT, "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, "typescript": "$TS_V" }, "logs": ["$STAMP-armA.log", "$STAMP-armB.log"] } JSON @@ -113,27 +108,26 @@ JSON { echo "### D6 — authored-vs-authoritative substitution · \`$VERDICT\`" echo - echo "| Arm | Change | \`tsc\` errors |" + echo "| Arm | Change | distinct \`tsc\` errors |" echo "|---|---|---|" echo "| A — baseline | none | $A_COUNT |" - echo "| B — substituted | \`$FILE\`${LINE:+:$LINE}${PROBE:+ + typed sink} | ${B_COUNT} |" + echo "| B — substituted | \`$FILE\`${LINE:+:$LINE}${PROBE:+ + typed sink} | $B_COUNT |" + echo "| **new under substitution** | | **$NEW_COUNT** |" echo - if [ "$CODE" = "2" ] && [ "${ENVISH:-0}" -gt 0 ]; then - echo "Arm A did not pass, so arm B was not run and **nothing about the types is established**." - echo - echo "\`$ENVISH\` of \`$A_COUNT\` baseline errors are module/export resolution" - echo "(TS2305/TS2307/TS2724). Those usually mean dependency types were never generated —" - echo "a skipped install step — rather than a defect in this repo, and other codes can be" - echo "downstream of the same cause. Confirm the toolchain is complete before reading" - echo "anything into this lane." - elif [ -n "$NEW_ERRS" ]; then - echo "Errors surfaced only under substitution:"; echo; echo '```'; printf '%s\n' "$NEW_ERRS"; echo '```' - elif [ "$CODE" = "1" ]; then - echo "**Silent — this is not proof of agreement.** Existing call sites may satisfy both shapes." + if [ "$NEW_COUNT" -gt 0 ]; then + echo "Errors present in B and absent in A — what the local type was concealing:" + echo; echo '```'; printf '%s\n' "$NEW_ERRS"; echo '```' + else + echo "**Silent — this is not proof of agreement.** Existing call sites may satisfy both" + echo "shapes; indexing and \`.match()\` compile against \`string\` and \`string[]\` alike." echo "Re-run with \`--probe\` to inject a sink only the authoritative type accepts." fi + if [ "$A_COUNT" -gt 0 ]; then + echo + echo "<sub>Baseline carried $A_COUNT pre-existing error(s) (${ENVISH} module/export). These are" + echo "subtracted, not disqualifying — only errors new under substitution are the finding.</sub>" + fi echo - echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" } > "$STAMP.md" printf 'tsc-substitution: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 From 3f861c716fe4bc2e814aadf200db1abb88a6ee71 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 07:25:25 -0400 Subject: [PATCH 19/63] =?UTF-8?q?Add=20`attest-gate.sh`=20=E2=80=94=20eigh?= =?UTF-8?q?t=20mechanical=20checks=20before=20anything=20is=20published?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 0 of the /attest command. Everything greppable is checked before a model is asked for judgement, because a model asked "is this good evidence?" answers from inside the frame that produced the text. Marker pair, canonical header, verdict line, environment pin, a captured artifact, no "what would close it", no first-person process narration, and `proven` only where an execution artifact exists. Check 5 carries the weight: if every character of the output is one the operator typed, the run published an assertion. `--reference` compares capture density against a known-good artifact. Verified in both directions. A retracted run-1 comment is BLOCKED on three checks — no captured artifact, a "what would close it" section, and an unearned `proven`. A runner-produced artifact passes all eight. Building it reproduced two bugs it exists to catch: `hasre -i '<pat>'` passed `-i` as the pattern, so three checks silently grepped for the literal string and returned false passes; and an over-escaped backtick made the environment-pin check never match. Both found by running the gate against a file whose expected verdict was already known. --- .../skills/evidence/scripts/attest-gate.sh | 88 +++++++++++++++++++ domains/pr-workflow/skills/evidence/skill.md | 22 +++++ 2 files changed, 110 insertions(+) create mode 100755 domains/pr-workflow/skills/evidence/scripts/attest-gate.sh diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh new file mode 100755 index 00000000..1fe9e1d4 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# +# attest-gate — phase 0 of /attest. Mechanical, no model, hard fails only. +# +# Everything checkable is checked before anything is asked of a model, because a +# model asked "is this good evidence?" answers from inside the frame that produced +# the text. These eight are greppable, so they are not a matter of judgement. +# +# Usage: attest-gate.sh <artifact.md> [--reference <showcase.html>] +# +# 0 all checks pass → proceed to the dispatched passes +# 1 one or more failed → BLOCKED, do not publish +# 2 usage error +set -uo pipefail + +FILE="${1:-}"; REF="" +[ $# -ge 2 ] && [ "${2:-}" = "--reference" ] && REF="${3:-}" +[ -n "$FILE" ] || { echo "usage: attest-gate.sh <artifact.md> [--reference <file>]" >&2; exit 2; } +[ -f "$FILE" ] || { echo "attest-gate: not found: $FILE" >&2; exit 2; } + +FAILED=0 +pass() { printf ' PASS %s\n' "$1"; } +fail() { printf ' FAIL %s\n %s\n' "$1" "$2"; FAILED=$((FAILED+1)); } +has() { grep -qF "$1" "$FILE"; } +hasre(){ grep -qE "$1" "$FILE"; } +hasi() { grep -qiE "$1" "$FILE"; } # case-insensitive; a separate function because + # `hasre -i '<pat>'` silently greps for "-i". + +echo "attest-gate: $FILE" +echo + +has 'VALIDATION_RUN_START' && has 'VALIDATION_RUN_END' \ + && pass "1 marker pair" \ + || fail "1 marker pair" "no VALIDATION_RUN_START/_END — a re-run appends a duplicate instead of replacing" + +has '## 🧪 Validation Run' \ + && pass "2 canonical header" \ + || fail "2 canonical header" "missing '## 🧪 Validation Run'" + +hasre '^\*\*Verdict:\*\*.*\*\*Claim:\*\*' \ + && pass "3 verdict line" \ + || fail "3 verdict line" "no '**Verdict:** … — **Claim:** …' — valence is not legible at a glance" + +hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `' \ + && pass "4 environment pinned" \ + || fail "4 environment pinned" "no head SHA, toolchain version, or lockfile hash" + +# 5 — the one that matters. A tool-written log, a run link, or an image; not typed prose. +if hasre '!\[|<img|data:image|actions/runs|/gist\.|evidence-artifacts/|Captured by|Produced by \`'; then + pass "5 captured artifact" +else + fail "5 captured artifact" "every block appears operator-typed; no tool-written log, run link, or image referenced" +fi + +if hasi 'what would close it|what would prove it|closing it requires'; then + fail "6 no prescriptions" "contains a 'what would close it' section — that is an unfinished run, formatted to look finished" +elif hasre '^\s*(Run|Switch|Assert|Scroll|Compare) '; then + fail "6 no prescriptions" "imperative-mood instructions to the reader — the artifact does not exist" +else + pass "6 no prescriptions" +fi + +if hasi "I originally|correction to my earlier|filed by me|hard to calibrate|I withdraw|my earlier comment"; then + fail "7 no process narration" "contains first-person process commentary — the reader did not see the earlier draft, and the byline may not be yours" +else + pass "7 no process narration" +fi + +if hasi '\*\*Verdict:\*\*.*proven' && ! hasre 'Captured by|Produced by \`|actions/runs|evidence-artifacts/'; then + fail "8 verdict is earned" "claims 'proven' with no execution artifact — reading yields 'unverified'" +else + pass "8 verdict is earned" +fi + +if [ -n "$REF" ] && [ -f "$REF" ]; then + r=$(grep -coE '!\[|<img|data:image' "$REF"); c=$(grep -coE '!\[|<img|data:image|evidence-artifacts/|Captured by' "$FILE") + echo + printf ' ratio reference captures: %s | this artifact: %s\n' "$r" "$c" + [ "$c" -eq 0 ] && [ "$r" -gt 0 ] && printf ' reference is capture-led and this is prose-only — see check 5\n' +fi + +echo +if [ "$FAILED" -eq 0 ]; then + echo "attest-gate: phase 0 clean — proceed to /outframe ‖ /missing ‖ /press" + exit 0 +fi +echo "attest-gate: BLOCKED — $FAILED check(s) failed. Do not publish." +exit 1 diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 6a258b30..55a53d38 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -272,6 +272,28 @@ install rather than a repo defect. It does not classify from that ratio — on a 280 errors were install artifacts while tripping no majority rule, because other codes are downstream of the same cause. +### `attest-gate.sh` — run this before publishing anything + +Eight mechanical checks over the artifact as it will ship. No model is asked anything until +these pass, because a model asked "is this good evidence?" answers from inside the frame that +produced the text. + +```bash +scripts/attest-gate.sh comment.md # exit 0 = proceed, 1 = BLOCKED +``` + +Marker pair · canonical header · verdict line · environment pinned · **a captured artifact** · +no "what would close it" · no first-person process narration · `proven` only with an execution +artifact. + +Check 5 is the one that matters and the easiest to slip past: if every character of the output +is one the operator typed, the run published an assertion. Pass `--reference <showcase>` to +compare capture density against a known-good artifact. + +This is phase 0 of [`/attest`](https://github.com/MajorLift/Reprise); phases 1 and 2 dispatch +`/outframe ‖ /missing ‖ /press` then `/trim` to fresh instances, because those passes cannot be +self-run — the author is positionally the wrong reader. + ### Canonical output shape Every validation-run output — PR comment *or* PR-body section — uses exactly this, so re-runs From 2c0f2869071aab16edeac66c6cef79b85bbea8c5 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 09:38:18 -0400 Subject: [PATCH 20/63] Standardise the provenance marker across every runner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three runners wrote three different provenance verbs — "Produced by", "Captured by", "Measured by" — so the gate had to know each synonym and silently blocked the one it did not. That is a false BLOCK on valid evidence, the most expensive direction for a gate to fail in. All four now emit `Produced by <script>`, and the gate matches one marker. `tsc-substitution.sh` emitted none at all: an earlier rewrite of its markdown body dropped the footer, so its artifacts failed the captured-artifact check despite being fully machine-produced. Also removes the last escaped backticks from the gate. In ERE a backtick is not special, so `Produced by \`` matched literal-backslash-backtick and never fired — the same defect already fixed once in the environment-pin check and not generalised then. Fifth appearance of this class today; now eliminated rather than patched per-check. Verified by regating five wrapped comments through every fix: three blocked on the synonym mismatch, all six pass once the marker is uniform. --- domains/pr-workflow/skills/evidence/scripts/attest-gate.sh | 6 +++--- domains/pr-workflow/skills/evidence/scripts/capture.sh | 2 +- .../skills/evidence/scripts/selector-recompute.sh | 2 +- .../pr-workflow/skills/evidence/scripts/tsc-substitution.sh | 2 ++ 4 files changed, 7 insertions(+), 5 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 1fe9e1d4..f377697b 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -46,7 +46,7 @@ hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `' \ || fail "4 environment pinned" "no head SHA, toolchain version, or lockfile hash" # 5 — the one that matters. A tool-written log, a run link, or an image; not typed prose. -if hasre '!\[|<img|data:image|actions/runs|/gist\.|evidence-artifacts/|Captured by|Produced by \`'; then +if hasre '!\[|<img|data:image|actions/runs|/gist\.|evidence-artifacts/|Produced by '; then pass "5 captured artifact" else fail "5 captured artifact" "every block appears operator-typed; no tool-written log, run link, or image referenced" @@ -66,14 +66,14 @@ else pass "7 no process narration" fi -if hasi '\*\*Verdict:\*\*.*proven' && ! hasre 'Captured by|Produced by \`|actions/runs|evidence-artifacts/'; then +if hasi '\*\*Verdict:\*\*.*proven' && ! hasre 'Produced by |actions/runs|evidence-artifacts/'; then fail "8 verdict is earned" "claims 'proven' with no execution artifact — reading yields 'unverified'" else pass "8 verdict is earned" fi if [ -n "$REF" ] && [ -f "$REF" ]; then - r=$(grep -coE '!\[|<img|data:image' "$REF"); c=$(grep -coE '!\[|<img|data:image|evidence-artifacts/|Captured by' "$FILE") + r=$(grep -coE '!\[|<img|data:image' "$REF"); c=$(grep -coE '!\[|<img|data:image|evidence-artifacts/|Produced by' "$FILE") echo printf ' ratio reference captures: %s | this artifact: %s\n' "$r" "$c" [ "$c" -eq 0 ] && [ "$r" -gt 0 ] && printf ' reference is capture-led and this is prose-only — see check 5\n' diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh index 064ed404..fc18c4fd 100755 --- a/domains/pr-workflow/skills/evidence/scripts/capture.sh +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -105,7 +105,7 @@ JSON fi echo '```' echo - echo "<sub>Captured by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. Raw log: \`$STAMP.log\`.</sub>" + echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. Raw log: \`$STAMP.log\`.</sub>" } > "$STAMP.md" printf 'capture: %s (exit %s)\n %s\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.log" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index 5abb05b9..54f68bcd 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -143,7 +143,7 @@ JSON echo "$LINE" echo '```' echo - echo "<sub>Measured by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Log: \`$STAMP.log\`.</sub>" + echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Log: \`$STAMP.log\`.</sub>" } > "$STAMP.md" printf 'selector-recompute: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh index 4dc4ea81..7bb81c81 100755 --- a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -128,6 +128,8 @@ JSON echo "subtracted, not disqualifying — only errors new under substitution are the finding.</sub>" fi echo + echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" + echo } > "$STAMP.md" printf 'tsc-substitution: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 From 51d6f947dedcb3b0cf5a8fef6748bf79fc17077c Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 10:21:01 -0400 Subject: [PATCH 21/63] =?UTF-8?q?Add=20`render-count.sh`=20=E2=80=94=20the?= =?UTF-8?q?=20component=20half=20of=20lane=20C4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `selector-recompute` answers how often a selector recomputes. This answers the other C4 question: how many times a named consumer actually renders. A memoisation claim about context or props is a claim about that count, and a count of call sites is not it — 149 consumers can mean 149 avoided renders or none. Runs the probe, defeats the memo at a given line, re-runs, reverts. Verified on a synthetic provider: 1 consumer render across 6 parent updates, rising to 6 with the memo defeated. The probe is supplied rather than generated, deliberately. A provider's mount requirements are specific to the component, and a generated probe would either be wrong or need every prop on the command line. That distinction also bounds what this can currently prove about extension#39310: a probe mounting its own memo demonstrates the mechanism but does not measure that PR's provider, which needs the app's store and router. No run was posted there — a synthetic stand-in presented against a real claim is the substitution this lane exists to catch. --- .../skills/evidence/scripts/render-count.sh | 116 ++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100755 domains/pr-workflow/skills/evidence/scripts/render-count.sh diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh new file mode 100755 index 00000000..fff2c0d1 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# +# render-count — lane C4, the component half. +# +# `selector-recompute` answers "how often does this selector recompute". This +# answers the other C4 question: "how many times does a consumer actually +# render". A memoization claim about context or props is a claim about that +# count, and a count of call sites is not it — 149 consumers can mean 149 +# avoided renders or none. +# +# Generates a probe that mounts a provider with a counting consumer, forces the +# parent to re-render N times with the memoised value unchanged, and reports the +# consumer's render count. Arm B re-runs with the memo defeated, so the delta is +# attributable rather than assumed. +# +# Usage: +# render-count.sh --probe <probe.test.tsx> [--defeat <file> --defeat-line <n> --defeat-with <text>] +# [--label <slug>] [--out <dir>] +# +# The probe is supplied rather than generated: a provider's mount requirements +# are specific to the component, and a generated one would either be wrong or +# would need every prop passed on the command line. Write it once, keep it. +# It must print a line of the form: +# +# RENDER_COUNT consumer=<n> parentRenders=<m> +# +# 0 measured counts captured for both arms (or arm A alone if no --defeat) +# 1 no delta arm B identical to arm A — the memo is not doing what is claimed +# 2 probe did not emit RENDER_COUNT +# 3 usage error +set -uo pipefail + +OUT_DIR="evidence-artifacts"; LABEL=""; PROBE=""; DEFEAT=""; DEFEAT_LINE=""; DEFEAT_WITH="" +die() { printf 'render-count: %s\n' "$1" >&2; exit 3; } + +while [ $# -gt 0 ]; do + case "$1" in + --probe) PROBE="${2:-}"; shift 2 ;; + --defeat) DEFEAT="${2:-}"; shift 2 ;; + --defeat-line) DEFEAT_LINE="${2:-}"; shift 2 ;; + --defeat-with) DEFEAT_WITH="${2:-}"; shift 2 ;; + --label) LABEL="${2:-}"; shift 2 ;; + --out) OUT_DIR="${2:-}"; shift 2 ;; + -h|--help) sed -n '2,30p' "$0"; exit 0 ;; + *) die "unknown argument: $1" ;; + esac +done + +[ -n "$PROBE" ] || die "--probe is required" +[ -f "$PROBE" ] || die "probe not found: $PROBE" +LABEL="${LABEL:-render-$(basename "$PROBE" | sed 's/\..*$//')}" +mkdir -p "$OUT_DIR" || die "cannot create $OUT_DIR" +STAMP="$OUT_DIR/$LABEL" + +counts_from() { grep -o 'RENDER_COUNT .*' "$1" | head -1; } +consumer_of() { printf '%s' "$1" | sed -n 's/.*consumer=\([0-9]*\).*/\1/p'; } + +yarn jest "$PROBE" > "$STAMP-armA.log" 2>&1 +A_LINE="$(counts_from "$STAMP-armA.log")" +A="$(consumer_of "$A_LINE")" +[ -n "$A" ] || { printf 'render-count: probe emitted no RENDER_COUNT line\n' >&2; exit 2; } + +B=""; B_LINE="" +if [ -n "$DEFEAT" ] && [ -n "$DEFEAT_LINE" ]; then + [ -f "$DEFEAT" ] || die "defeat target not found: $DEFEAT" + BACKUP="$(mktemp)"; cp "$DEFEAT" "$BACKUP" + restore() { cp "$BACKUP" "$DEFEAT"; rm -f "$BACKUP"; } + trap restore EXIT INT TERM + awk -v n="$DEFEAT_LINE" -v r="$DEFEAT_WITH" 'NR==n{print r; next}{print}' "$DEFEAT" > "$DEFEAT.tmp" && mv "$DEFEAT.tmp" "$DEFEAT" + yarn jest "$PROBE" > "$STAMP-armB.log" 2>&1 + B_LINE="$(counts_from "$STAMP-armB.log")" + B="$(consumer_of "$B_LINE")" + restore; trap - EXIT INT TERM +else + : > "$STAMP-armB.log" +fi + +if [ -n "$B" ] && [ "$B" = "$A" ]; then VERDICT="no delta — memo not attributable"; CODE=1 +elif [ -n "$B" ]; then VERDICT="delta measured: $A → $B renders with the memo defeated"; CODE=0 +else VERDICT="baseline only: $A consumer renders"; CODE=0; fi + +HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" +DIRTY="$(git status --porcelain 2>/dev/null | grep -v '^??' | wc -l | tr -d ' ')" +NODE_V="$(node -v 2>/dev/null || echo unknown)" + +cat > "$STAMP.json" <<JSON +{ "probe": "$PROBE", "verdict": "$VERDICT", "exit": $CODE, + "consumer_renders": { "armA": ${A:-null}, "armB": ${B:-null} }, + "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, "node": "$NODE_V" }, + "logs": ["$STAMP-armA.log", "$STAMP-armB.log"] } +JSON + +{ + echo "### C4 — consumer render count" + echo + echo "**Verdict:** $VERDICT" + echo + echo "| Arm | Change | consumer renders |" + echo "|---|---|---|" + echo "| A — as committed | none | ${A:-?} |" + [ -n "$B" ] && echo "| B — memo defeated | \`$DEFEAT:$DEFEAT_LINE\` | $B |" + echo + echo '```console' + echo "\$ yarn jest $PROBE" + echo "$A_LINE" + [ -n "$B_LINE" ] && { echo "\$ yarn jest $PROBE # memo defeated"; echo "$B_LINE"; } + echo '```' + echo + echo "This counts renders of one named consumer across a defined interaction. It is not a count" + echo "of consumers, and a larger consumer count does not imply a larger effect." + echo + echo "<sub>Produced by \`render-count.sh\`; the defeat edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" +} > "$STAMP.md" + +printf 'render-count: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 +exit "$CODE" From e5b6ce1e097e7a80845944911ab5164cadf25d22 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:06:03 -0400 Subject: [PATCH 22/63] Guard the runners against the failures that masquerade as findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit falsify-probe reported `falsifying, exit 0` for a mutation that broke syntax. Arm B "failed" because the module would not load and zero tests ran — identical to a real falsification by exit code, and it would have published "the suite fails when the mechanism is removed" about a suite that never executed. Arm B must now run the same test count as arm A and fail on assertions; a load error or a dropped count reports "mutation broke the module, nothing falsified". selector-recompute now gates on correctness before reporting counts. The probe captures the selector's value across all three conditions, and a value that moves under a write the selector does not declare as an input fails the run outright. A memoisation change that alters output is a breaking change the count would never reveal. attest-gate gains check 9: the wrapper's verdict must not contradict the artifact it embeds. Comments are assembled by hand around machine output, and the hand-written header is exactly where a "vacuous" result acquires a "proven" label. Verified by relabelling a real comment — blocked. All three verified in both directions: the poisoned mutation is refused and the genuine one still reports falsifying; the mislabelled comment is blocked and the correct one passes. --- .../skills/evidence/scripts/attest-gate.sh | 11 ++++++ .../skills/evidence/scripts/falsify-probe.sh | 35 +++++++++++++++---- .../evidence/scripts/selector-recompute.sh | 34 ++++++++++++++++-- 3 files changed, 70 insertions(+), 10 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index f377697b..13d0bc39 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -72,6 +72,17 @@ else pass "8 verdict is earned" fi +# 9 — the wrapper's verdict must not contradict the artifact it embeds. A comment is +# assembled by hand around machine output, and the hand-written header is exactly where +# a "vacuous" result acquires a "proven" label. +HDR="$(grep -m1 '^\*\*Verdict:\*\*' "$FILE" | tr 'A-Z' 'a-z')" +BODY="$(grep -ioE 'vacuous|value unstable|no delta|nothing falsified|broke the module|substitution silent|probe-failed' "$FILE" | head -1 | tr 'A-Z' 'a-z')" +if printf '%s' "$HDR" | grep -q 'proven' && [ -n "$BODY" ]; then + fail "9 verdict matches artifact" "header claims 'proven' while the embedded artifact reports '$BODY'" +else + pass "9 verdict matches artifact" +fi + if [ -n "$REF" ] && [ -f "$REF" ]; then r=$(grep -coE '!\[|<img|data:image' "$REF"); c=$(grep -coE '!\[|<img|data:image|evidence-artifacts/|Produced by' "$FILE") echo diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh index 8edd34c7..29616922 100755 --- a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -12,11 +12,16 @@ # Emits a captured artifact (JSON + markdown) written by this script, not # transcribed by an operator. Exit code IS the verdict, so CI can gate on it. # -# 0 falsifying arm A passed, arm B failed → the test has power -# 1 vacuous arm A passed, arm B ALSO passed → the test proves nothing -# 2 broken arm A failed → nothing to conclude +# 0 falsifying arm A passed, arm B failed ON ASSERTIONS → the test has power +# 1 vacuous arm A passed, arm B ALSO passed → the test proves nothing +# 2 broken arm A failed, or arm B did not run → nothing to conclude # 3 usage/env error # +# Arm B failing is NOT sufficient. A mutation that breaks syntax fails every test in +# the file, which looks identical to a falsification and is worth nothing: the suite +# never executed. So arm B must run the SAME number of tests as arm A and fail some of +# them. A dropped test count means the mutation broke the module, not the mechanism. +# # Usage: # falsify-probe.sh --test <path> --source <path> --line <n> --replace <text> # [--label <slug>] [--out <dir>] [--runner "<cmd>"] @@ -79,10 +84,13 @@ run_arm() { # $1=logfile ; prints "passed|failed" if $RUNNER "$TEST" > "$1" 2>&1; then echo passed; else echo failed; fi } +total_tests() { sed -n 's/.*Tests:.*[^0-9]\([0-9][0-9]*\) total.*/\1/p' "$1" | head -1; } +load_failed() { grep -qiE "SyntaxError|Cannot find module|Unexpected token|Transform failed" "$1"; } + ARM_A="$(run_arm "$STAMP-armA.log")" if [ "$ARM_A" != "passed" ]; then - VERDICT="broken"; CODE=2; ARM_B="not-run" + VERDICT="baseline-already-failing"; CODE=2; ARM_B="not-run" : > "$STAMP-armB.log" else # Mutate exactly one line. `.bak` form keeps this portable across GNU/BSD sed. @@ -90,7 +98,19 @@ else && mv "$SOURCE.tmp" "$SOURCE" || die "mutation failed" ARM_B="$(run_arm "$STAMP-armB.log")" restore; trap - EXIT INT TERM - if [ "$ARM_B" = "failed" ]; then VERDICT="falsifying"; CODE=0; else VERDICT="vacuous"; CODE=1; fi + A_TOTAL="$(total_tests "$STAMP-armA.log")"; A_TOTAL="${A_TOTAL:-0}" + B_TOTAL="$(total_tests "$STAMP-armB.log")"; B_TOTAL="${B_TOTAL:-0}" + if [ "$ARM_B" != "failed" ]; then + VERDICT="vacuous"; CODE=1 + elif load_failed "$STAMP-armB.log" || [ "$B_TOTAL" -lt "$A_TOTAL" ]; then + # The suite did not execute under mutation, so nothing was falsified. Reported as + # broken rather than falsifying: a module that will not load fails every test, which + # is indistinguishable from a real failure by exit code alone. + VERDICT="mutation broke the module — suite ran $B_TOTAL of $A_TOTAL tests, nothing falsified" + CODE=2 + else + VERDICT="falsifying"; CODE=0 + fi fi summarise() { grep -E '^(Tests|Test Suites):' "$1" 2>/dev/null | tr '\n' ' ' | sed 's/ */ /g'; } @@ -121,9 +141,10 @@ JSON echo "| B — mutant | \`$SOURCE:$LINE\` replaced | \`$B_SUM\` |" echo case "$VERDICT" in - falsifying) echo "The suite **fails when the mechanism is removed** and passes when restored. The test has power." ;; + falsifying) echo "The suite **fails when the mechanism is removed** and passes when restored, running the same $A_TOTAL tests in both arms. The test has power." ;; vacuous) echo "The suite **passes with the mechanism removed**. It does not test what it appears to test." ;; - broken) echo "Arm A did not pass, so arm B was not run. No conclusion." ;; + baseline-already-failing) echo "Arm A did not pass, so arm B was not run. No conclusion." ;; + *) echo "**No conclusion.** $VERDICT — a module that will not load fails every test, which an exit code cannot tell apart from a real falsification." ;; esac [ -n "$FAILED_NAMES" ] && { echo; echo "Failing under mutation:"; echo; printf '%s\n' "$FAILED_NAMES" | sed 's/^/- /'; } echo diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index 54f68bcd..c8cd109f 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -78,13 +78,21 @@ describe('$EXPORT recomputation probe', () => { ($EXPORT as unknown as { resetRecomputations: () => void }).resetRecomputations(); const count = () => ($EXPORT as unknown as { recomputations: () => number }).recomputations(); - for (let i = 0; i < $N; i++) call(base); + const seen: string[] = []; + const snap = (v: unknown) => { try { return JSON.stringify(v); } catch { return '<unserialisable>'; } }; + + for (let i = 0; i < $N; i++) seen.push(snap(call(base))); const a = count(); + const stableIdentical = new Set(seen).size === 1; + const unrelatedSeen: string[] = []; for (let i = 0; i < $N; i++) { - call({ ...base, $SLICE: { ...base.$SLICE, __unrelated__: i } }); + unrelatedSeen.push(snap(call({ ...base, $SLICE: { ...base.$SLICE, __unrelated__: i } }))); } const b = count(); + // A write the selector does not read must not change what it returns. If it does, + // the memoisation is not the story — the selector has an input it does not declare. + const stableUnrelated = new Set(unrelatedSeen).size === 1 && unrelatedSeen[0] === seen[0]; for (let i = 0; i < $N; i++) { call({ ...base, $SLICE: { ...base.$SLICE, $PERTURB: [\`0x\${i}\`] } }); @@ -92,8 +100,14 @@ describe('$EXPORT recomputation probe', () => { const c = count(); // eslint-disable-next-line no-console - console.log(\`RECOMPUTE_PROBE identical=\${a} unrelated=\${b} inputChanged=\${c} n=$N\`); + console.log( + \`RECOMPUTE_PROBE identical=\${a} unrelated=\${b} inputChanged=\${c} n=$N\` + + \` valueStable=\${stableIdentical && stableUnrelated}\`, + ); expect(c).toBeGreaterThanOrEqual(b); + // Correctness gates the measurement: an unstable value makes the count meaningless. + expect(stableIdentical).toBe(true); + expect(stableUnrelated).toBe(true); }); }); PROBEEOF @@ -103,12 +117,17 @@ CODE=$? cleanup; trap - EXIT INT TERM LINE="$(grep -o 'RECOMPUTE_PROBE .*' "$STAMP.log" | head -1)" +STABLE="$(printf '%s' "$LINE" | sed -n 's/.*valueStable=\([a-z]*\).*/\1/p')" A="$(printf '%s' "$LINE" | sed -n 's/.*identical=\([0-9]*\).*/\1/p')" B="$(printf '%s' "$LINE" | sed -n 's/.*unrelated=\([0-9]*\).*/\1/p')" C="$(printf '%s' "$LINE" | sed -n 's/.*inputChanged=\([0-9]*\).*/\1/p')" if [ -z "$A" ]; then VERDICT="probe-failed" +elif [ "$STABLE" = "false" ]; then + # Correctness first. A selector whose value moves under a write it does not read has an + # undeclared input, and no recomputation count means anything until that is resolved. + VERDICT="VALUE UNSTABLE — breaking behaviour, count not meaningful" elif [ "$B" -gt "$A" ]; then VERDICT="recomputes on unrelated writes" else @@ -138,6 +157,15 @@ JSON echo "| Fresh \`$SLICE\` slice, unrelated field | $N | ${B:-?} |" echo "| \`$PERTURB\` changed (a real input) | $N | ${C:-?} |" echo + if [ "$STABLE" = "true" ]; then + echo "**Correctness:** the returned value is identical across all calls above, so the count" + echo "measures memoisation rather than a change in behaviour." + else + echo "**Correctness: FAILED.** The returned value changed under a write the selector does not" + echo "declare as an input. That is a behavioural difference, not a performance one, and it" + echo "makes the recomputation count meaningless — resolve it before reading the numbers." + fi + echo echo '```console' echo "\$ yarn jest <generated probe>" echo "$LINE" From d84f3048a450852fe162750cdb9e866ab7c41395 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:15:48 -0400 Subject: [PATCH 23/63] Give the orchestrator a runner registry, with limits and synthesis rules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seven runners existed with nothing telling the orchestrator which serves which claim, so routing was left to judgement at the exact point where judgement is what the tooling replaces. The registry pairs every runner with what it CANNOT establish, because that column is where a run stops being evidence: falsify-probe shows a test has power and says nothing about whether the fix is correct; a silent tsc-substitution is not agreement; retention-scan pairs by name and cannot show the release site is reachable; policy-audit and egress-delta cannot rule on acceptability at all. Synthesis rules, in priority order: a lead lane is required and a corroborator never substitutes for one; correctness gates measurement, since a performance number over changed behaviour is a missed regression rather than a result; an exit-2 from any runner caps the whole run at unproven and is reported on its own line rather than averaged away; security and privacy findings route privately whatever the other lanes say; and the uncovered part of the claim is named in the artifact so the covered part cannot imply coverage. Records a known gap rather than papering over it. No runner checks a diff against an architectural decision record. A merged PR added deeplinks accepting unsigned parameters — justified as "read-only screens, so unsigned routing params are safe" — and was reverted. Signed links skip the warning interstitial, so an unsigned parameter inherits the signature's trust without being covered by it, which is exploitable whether or not the destination writes anything. Nothing in the table would have caught that: it is a rule in a document, violated by code that looks unremarkable. ADR-governed surfaces route to a human until a conformance runner exists, and the artifact must say so. --- domains/pr-workflow/skills/evidence/skill.md | 50 ++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 55a53d38..9380b946 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -294,6 +294,56 @@ This is phase 0 of [`/attest`](https://github.com/MajorLift/Reprise); phases 1 a `/outframe ‖ /missing ‖ /press` then `/trim` to fresh instances, because those passes cannot be self-run — the author is positionally the wrong reader. +### Runner registry — what each establishes, and what it cannot + +Route a claim to a runner by what the claim asserts. **Read the limit column before quoting a +result**: every runner has a shape of claim it cannot reach, and reporting past that line is how +a run stops being evidence. + +| Runner | Establishes | Cannot establish | +|---|---|---| +| `falsify-probe.sh` | a test fails when its mechanism is removed | that the fix is *correct* — only that the test has power | +| `selector-recompute.sh` | recomputation counts across three input conditions | component render counts; a selector without `.recomputations()` | +| `render-count.sh` | renders of one named consumer over one interaction | that other consumers behave the same; needs a hand-written probe | +| `tsc-substitution.sh` | a hand-written type disagrees with its source | agreement — **a silent arm B means the probe was too weak** | +| `retention-scan.py` | acquire/release pairing within one file | that the release site is *reachable* from the acquire | +| `policy-audit.py` | capability delta and override scope | whether a grant is acceptable — intent is not in the files | +| `egress-delta.py` | egress added, protections removed | whether a flow is acceptable, or what happens off-diff | +| `capture.sh` | a verbatim artifact for any command | any verdict — the caller states it or none is claimed | +| `attest-gate.sh` | eight mechanical publication checks | whether the claim under test was the right one to test | + +Exit codes are uniform: `0` the checked property holds · `1` it does not · `2` no conclusion +available · `3` usage error. **A `2` from any runner caps the whole run at unproven** — one +inconclusive arm is not offset by another lane passing. + +### Synthesising a run from several runners + +1. **Lead lane first.** Pick the runner whose output *is* the claim. A corroborator strengthens + a lead; it never substitutes for one. +2. **Correctness gates measurement.** `selector-recompute` fails outright on an unstable value, + and that ordering generalises: a performance number over changed behaviour is not a + performance result, it is a missed regression. +3. **A `2` is load-bearing.** Report it as its own line. Averaging it away, or quoting the lanes + that passed, converts "we could not tell" into "it is fine". +4. **Security and privacy findings route privately** regardless of what the other lanes say. A + green performance lane does not make an egress finding publishable here. +5. **State the residue.** Name the part of the claim no runner reached, in the artifact, rather + than letting the covered part imply coverage. + +### Known gap: conformance to a written decision + +No runner here checks a diff against an architectural decision record. That class is real and +expensive — a merged PR added deeplinks accepting unsigned parameters, justified as "read-only +screens, so unsigned routing params are safe", and was reverted after review. The justification +misreads the model: signed links skip the warning interstitial, so an unsigned parameter inherits +the signature's trust without being covered by it, which is exploitable regardless of whether the +destination writes anything. + +Nothing in the table above would have found that. It is not a measurement, a count, or a diff +delta — it is a rule stated in a document, violated by code that looks unremarkable. Until a +conformance runner exists, **route ADR-governed surfaces to a human reviewer and say in the +artifact that you did**. + ### Canonical output shape Every validation-run output — PR comment *or* PR-body section — uses exactly this, so re-runs From 0d8935f3e4ac22308635cb6abb171413a1f2ade8 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:30:23 -0400 Subject: [PATCH 24/63] State the bar: float concerns, do not close them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A run succeeds when it puts concerns, falsifiers, and avenues of deeper inquiry in front of a reviewer. Not every concern caught, none required to reach a conclusion. Lower than being right; much higher than staying silent unless certain. Replaces the "known gap" framing, which had the bar wrong. That no runner checks ADR conformance is not a deficiency to apologise for — a run naming the governed surface and its falsifier has done its job while resolving nothing. Held to correctness-to-conclusion the tooling is useless precisely where review matters most, because those cases turn on intent, threat model, or a written decision that is not in the diff, and a tool held to that bar either goes silent or guesses. Guards the obvious failure mode: floating a concern still requires naming what would settle it. "This might be unsafe" is noise; "unsigned parameters on a signed link skip the interstitial, check whether the signature covers them" is actionable. The difference is whether the next step is stated. Keeps the deeplink revert as the worked case, now as an illustration of a sufficient run rather than a missed one. --- domains/pr-workflow/skills/evidence/skill.md | 47 ++++++++++++++------ 1 file changed, 34 insertions(+), 13 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 9380b946..7c7d1a72 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -330,19 +330,40 @@ inconclusive arm is not offset by another lane passing. 5. **State the residue.** Name the part of the claim no runner reached, in the artifact, rather than letting the covered part imply coverage. -### Known gap: conformance to a written decision - -No runner here checks a diff against an architectural decision record. That class is real and -expensive — a merged PR added deeplinks accepting unsigned parameters, justified as "read-only -screens, so unsigned routing params are safe", and was reverted after review. The justification -misreads the model: signed links skip the warning interstitial, so an unsigned parameter inherits -the signature's trust without being covered by it, which is exploitable regardless of whether the -destination writes anything. - -Nothing in the table above would have found that. It is not a measurement, a count, or a diff -delta — it is a rule stated in a document, violated by code that looks unremarkable. Until a -conformance runner exists, **route ADR-governed surfaces to a human reviewer and say in the -artifact that you did**. +### The bar: float concerns, do not close them + +A run succeeds when it puts **concerns, falsifiers, and avenues of deeper inquiry** in front of a +reviewer. It is not required to catch every concern, resolve each correctly, or carry any to a +conclusion. That is a lower bar than being right, and a much higher one than staying silent +unless certain. + +Three consequences worth being explicit about: + +**A coverage gap is not a failed run.** No runner here checks a diff against an architectural +decision record. A run that says *"this touches deeplinks, which are ADR-governed; the falsifier +is whether these parameters are covered by the signature"* has done its job while resolving +nothing. + +**Incomplete analysis is reportable, not suppressible.** Withholding anything short of fully +established throws away the run's actual product. An unresolved concern with a named falsifier is +the deliverable. + +**This does not license speculation.** Floating a concern still requires naming what would settle +it. *"This might be unsafe"* is noise. *"Unsigned parameters on a signed link skip the +interstitial — check whether the signature covers them"* is actionable. The difference is whether +the next step is stated. + +The runners raise questions with evidence attached; they are not oracles. Their limits are +publishable content, which is why the table above lists what each cannot establish. + +**Worked case.** A merged PR added deeplinks accepting unsigned parameters, justified as +"read-only screens, so unsigned routing params are safe". It was reverted after review. The +justification misreads the model: signed links skip the warning interstitial, so an unsigned +parameter inherits the signature's trust without being covered by it — exploitable whether or not +the destination writes anything, via navigation hijacking, request forgery, phishing through +trusted chrome, or attribution poisoning. No runner would have caught it. A run that merely +flagged *"deeplink surface, ADR-0011 governs parameter signing, is this param in the signed +set?"* would have been enough. ### Canonical output shape From 6bdffd13afc2c2e588ab97399b81f10572f12a89 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:35:19 -0400 Subject: [PATCH 25/63] Report what each measurement runner did not cover A runner's job is to put concerns in front of a reviewer, not to close them. The four measurement runners exited 0 or 1 and said nothing about the surface they left unmeasured, which reads as a clean bill of health for the whole mechanism rather than for the one property tested. --- .../pr-workflow/skills/evidence/scripts/falsify-probe.sh | 8 ++++++++ .../pr-workflow/skills/evidence/scripts/render-count.sh | 5 +++++ .../skills/evidence/scripts/selector-recompute.sh | 5 +++++ .../skills/evidence/scripts/tsc-substitution.sh | 5 +++++ 4 files changed, 23 insertions(+) diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh index 29616922..cd46cf9c 100755 --- a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -148,6 +148,14 @@ JSON esac [ -n "$FAILED_NAMES" ] && { echo; echo "Failing under mutation:"; echo; printf '%s\n' "$FAILED_NAMES" | sed 's/^/- /'; } echo + echo "**Open for review** — this run mutated one line of one file. It says nothing about" + echo "other paths into the same mechanism, whether the mechanism is reachable in production," + echo "or whether the behaviour it guards is the right behaviour. A falsifying test proves the" + echo "test has power, not that the fix is correct." + case "$VERDICT" in + vacuous) echo "Worth a look: the mechanism is unguarded by this suite — what else depends on it?" ;; + esac + echo echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" } > "$STAMP.md" diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh index fff2c0d1..a0784724 100755 --- a/domains/pr-workflow/skills/evidence/scripts/render-count.sh +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -109,6 +109,11 @@ JSON echo "This counts renders of one named consumer across a defined interaction. It is not a count" echo "of consumers, and a larger consumer count does not imply a larger effect." echo + echo "**Open for review** — one named consumer, one interaction. Other consumers of the same" + echo "provider are unmeasured, and a consumer that renders once here may render freely under" + echo "an interaction this probe does not perform. The probe also does not check that the" + echo "consumer renders the same OUTPUT, only that it renders fewer times." + echo echo "<sub>Produced by \`render-count.sh\`; the defeat edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" } > "$STAMP.md" diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index c8cd109f..e08788bb 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -171,6 +171,11 @@ JSON echo "$LINE" echo '```' echo + echo "**Open for review** — measured against one fixture with one perturbed key. A selector" + echo "unmoved here can still recompute under state this fixture does not reach, and the count" + echo "says nothing about the cost of each recomputation. Worth a look if the fixture is thin" + echo "relative to the shapes this selector sees in production." + echo echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Log: \`$STAMP.log\`.</sub>" } > "$STAMP.md" diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh index 7bb81c81..6fe4e987 100755 --- a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -128,6 +128,11 @@ JSON echo "subtracted, not disqualifying — only errors new under substitution are the finding.</sub>" fi echo + echo "**Open for review** — the compiler answers only what the probe asks. A silent arm B means" + echo "no call site in this tree distinguishes the two shapes, which is not agreement: a" + echo "divergence reachable only at runtime, or only from a caller outside this repo, will not" + echo "appear here. Worth a look at whether the authoritative type is itself correct." + echo echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" echo } > "$STAMP.md" From 30a03b6ae3b2c078900567ca5dd4c0b260e57140 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:36:43 -0400 Subject: [PATCH 26/63] Require a validation run to float something for review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `capture.sh` gains `--open`, stated by the caller like `--verdict` and never inferred; when omitted the artifact says so rather than reading as full coverage. `attest-gate.sh` gains check 10, the positive counterpart to check 6 — check 6 rejects handing the reader the run's own unfinished work, check 10 rejects an artifact that names no limit at all. --- .../skills/evidence/scripts/attest-gate.sh | 13 ++++++++++++- .../skills/evidence/scripts/capture.sh | 18 ++++++++++++++++-- 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 13d0bc39..4ceeed83 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -4,7 +4,7 @@ # # Everything checkable is checked before anything is asked of a model, because a # model asked "is this good evidence?" answers from inside the frame that produced -# the text. These eight are greppable, so they are not a matter of judgement. +# the text. These are greppable, so they are not a matter of judgement. # # Usage: attest-gate.sh <artifact.md> [--reference <showcase.html>] # @@ -83,6 +83,17 @@ else pass "9 verdict matches artifact" fi +# 10 — the positive counterpart to check 6. A run succeeds by putting concerns in front +# of a reviewer, so an artifact that floats nothing has reported only what it happened to +# measure and called that the whole picture. This is NOT satisfied by a "what would close +# it" section, which check 6 rejects: that hands the reader the run's own unfinished work, +# whereas this names a limit or a question the run is right to leave open. +if hasi 'open for review|raise with a human|falsifier|worth a look|left unmeasured|not covered by this run|no verdict offered'; then + pass "10 floats something for review" +else + fail "10 floats something for review" "no limit, open question, or falsifier named — an artifact that floats nothing implies its measurement was the whole surface" +fi + if [ -n "$REF" ] && [ -f "$REF" ]; then r=$(grep -coE '!\[|<img|data:image' "$REF"); c=$(grep -coE '!\[|<img|data:image|evidence-artifacts/|Produced by' "$FILE") echo diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh index fc18c4fd..e40e2ef5 100755 --- a/domains/pr-workflow/skills/evidence/scripts/capture.sh +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -9,11 +9,16 @@ # # This wraps any command so the ARTIFACT is written by the tool. Nothing is retyped. # -# capture.sh --label <slug> --lane <id> --claim "<under test>" [--verdict <word>] -- <cmd...> +# capture.sh --label <slug> --lane <id> --claim "<under test>" [--verdict <word>] +# [--open "<what this run leaves open>"] -- <cmd...> # # --verdict is stated by the caller, never inferred from the exit code: a wrapped # tool's exit convention is its own, and guessing prints "pass" over real findings. # +# --open is the same discipline pointed the other way. A run succeeds by putting +# concerns in front of a reviewer, not by closing them, so what the wrapped tool +# could not reach is publishable content. Omitting it is recorded, not hidden. +# # Emits, under --out (default evidence-artifacts/): # <label>.log raw stdout+stderr of the command, unmodified # <label>.json machine-readable: verdict, exit code, env pin, claim @@ -27,7 +32,7 @@ # -- python3 retention-scan.py ui/store/background-connection.ts pr.patch set -uo pipefail -OUT_DIR="evidence-artifacts"; LABEL=""; LANE=""; CLAIM=""; MAXLOG=120; VERDICT="" +OUT_DIR="evidence-artifacts"; LABEL=""; LANE=""; CLAIM=""; MAXLOG=120; VERDICT=""; OPEN="" die() { printf 'capture: %s\n' "$1" >&2; exit 3; } while [ $# -gt 0 ]; do @@ -36,6 +41,7 @@ while [ $# -gt 0 ]; do --lane) LANE="${2:-}"; shift 2 ;; --claim) CLAIM="${2:-}"; shift 2 ;; --verdict) VERDICT="${2:-}"; shift 2 ;; + --open) OPEN="${2:-}"; shift 2 ;; --out) OUT_DIR="${2:-}"; shift 2 ;; --max-log-lines) MAXLOG="${2:-}"; shift 2 ;; -h|--help) sed -n '2,26p' "$0"; exit 0 ;; @@ -78,6 +84,7 @@ cat > "$STAMP.json" <<JSON "claim": $(jstr "$CLAIM"), "command": $(jstr "$CMD_STR"), "verdict": "$VERDICT", + "open_for_review": $(jstr "$OPEN"), "exit": $CODE, "log": "$STAMP.log", "log_lines": $LINES, @@ -105,6 +112,13 @@ JSON fi echo '```' echo + if [ -n "$OPEN" ]; then + echo "**Open for review** — $OPEN" + else + echo "**Open for review** — none stated. This tool answered one question; what it does" + echo "not cover was not recorded, which is not the same as it covering everything." + fi + echo echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. Raw log: \`$STAMP.log\`.</sub>" } > "$STAMP.md" From d5c6a61d0db01e65e3ba2617853813ab33e710de Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:53:20 -0400 Subject: [PATCH 27/63] Stop the runners from cutting the part a reader needed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `capture.sh` elided the tail when output exceeded the line budget, which drops `policy-audit.py`'s RAISE WITH A HUMAN section and leaves a wall of checkboxes in its place — a tool that escalates does it last. It now elides the middle. `render-count.sh` hard-coded arm B as "memo defeated"; when the PR under test is the suspect rather than the fix, that prints the reading backwards, so the label is caller-stated via `--arm-b`. --- .../skills/evidence/scripts/capture.sh | 11 ++++++++-- .../skills/evidence/scripts/render-count.sh | 21 ++++++++++++------- 2 files changed, 23 insertions(+), 9 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh index e40e2ef5..954e01a8 100755 --- a/domains/pr-workflow/skills/evidence/scripts/capture.sh +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -105,8 +105,15 @@ JSON echo '```console' echo "\$ $CMD_STR" if [ "$LINES" -gt "$MAXLOG" ]; then - head -n "$MAXLOG" "$STAMP.log" - echo "… $((LINES - MAXLOG)) further lines in $STAMP.log" + # Elide the middle, never the end. A tool that escalates does it last: + # policy-audit.py prints a per-grant worklist first and its RAISE WITH A HUMAN + # section at the bottom, so head-truncation cuts exactly the rows that needed a + # reader and leaves a wall of checkboxes in their place. + H=$(( MAXLOG * 2 / 3 )); T=$(( MAXLOG - H )) + head -n "$H" "$STAMP.log" + printf '\n… %s lines elided from the middle — full output in %s\n\n' \ + "$((LINES - MAXLOG))" "$STAMP.log" + tail -n "$T" "$STAMP.log" else cat "$STAMP.log" fi diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh index a0784724..d2a550c5 100755 --- a/domains/pr-workflow/skills/evidence/scripts/render-count.sh +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -10,12 +10,17 @@ # # Generates a probe that mounts a provider with a counting consumer, forces the # parent to re-render N times with the memoised value unchanged, and reports the -# consumer's render count. Arm B re-runs with the memo defeated, so the delta is +# consumer's render count. Arm B re-runs with one line changed, so the delta is # attributable rather than assumed. # # Usage: # render-count.sh --probe <probe.test.tsx> [--defeat <file> --defeat-line <n> --defeat-with <text>] -# [--label <slug>] [--out <dir>] +# [--arm-b <label>] [--label <slug>] [--out <dir>] +# +# Arm B is "the memo defeated" by default, which is the shape when a PR ADDS +# memoisation. When a PR is the one under suspicion the arms invert — arm B applies +# the candidate fix — and calling that "defeated" prints the reading backwards. So +# the label is caller-stated, like every other verdict word in this suite. # # The probe is supplied rather than generated: a provider's mount requirements # are specific to the component, and a generated one would either be wrong or @@ -31,6 +36,7 @@ set -uo pipefail OUT_DIR="evidence-artifacts"; LABEL=""; PROBE=""; DEFEAT=""; DEFEAT_LINE=""; DEFEAT_WITH="" +ARM_B="memo defeated" die() { printf 'render-count: %s\n' "$1" >&2; exit 3; } while [ $# -gt 0 ]; do @@ -39,6 +45,7 @@ while [ $# -gt 0 ]; do --defeat) DEFEAT="${2:-}"; shift 2 ;; --defeat-line) DEFEAT_LINE="${2:-}"; shift 2 ;; --defeat-with) DEFEAT_WITH="${2:-}"; shift 2 ;; + --arm-b) ARM_B="${2:-}"; shift 2 ;; --label) LABEL="${2:-}"; shift 2 ;; --out) OUT_DIR="${2:-}"; shift 2 ;; -h|--help) sed -n '2,30p' "$0"; exit 0 ;; @@ -75,8 +82,8 @@ else : > "$STAMP-armB.log" fi -if [ -n "$B" ] && [ "$B" = "$A" ]; then VERDICT="no delta — memo not attributable"; CODE=1 -elif [ -n "$B" ]; then VERDICT="delta measured: $A → $B renders with the memo defeated"; CODE=0 +if [ -n "$B" ] && [ "$B" = "$A" ]; then VERDICT="no delta — arm B changed nothing measurable"; CODE=1 +elif [ -n "$B" ]; then VERDICT="delta measured: $A → $B renders with $ARM_B"; CODE=0 else VERDICT="baseline only: $A consumer renders"; CODE=0; fi HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" @@ -98,12 +105,12 @@ JSON echo "| Arm | Change | consumer renders |" echo "|---|---|---|" echo "| A — as committed | none | ${A:-?} |" - [ -n "$B" ] && echo "| B — memo defeated | \`$DEFEAT:$DEFEAT_LINE\` | $B |" + [ -n "$B" ] && echo "| B — $ARM_B | \`$DEFEAT:$DEFEAT_LINE\` | $B |" echo echo '```console' echo "\$ yarn jest $PROBE" echo "$A_LINE" - [ -n "$B_LINE" ] && { echo "\$ yarn jest $PROBE # memo defeated"; echo "$B_LINE"; } + [ -n "$B_LINE" ] && { echo "\$ yarn jest $PROBE # $ARM_B"; echo "$B_LINE"; } echo '```' echo echo "This counts renders of one named consumer across a defined interaction. It is not a count" @@ -114,7 +121,7 @@ JSON echo "an interaction this probe does not perform. The probe also does not check that the" echo "consumer renders the same OUTPUT, only that it renders fewer times." echo - echo "<sub>Produced by \`render-count.sh\`; the defeat edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" + echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" } > "$STAMP.md" printf 'render-count: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 From a3f15900c6d03c0e19ff6e07446e09a09e84729b Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:55:55 -0400 Subject: [PATCH 28/63] Reference artifacts by name in the publishable block An absolute `--out` path put the operator's home directory into the `<sub>` line of every artifact, which is the one part of the run that gets pasted into a public comment. The `.json` keeps full paths for machine use. --- domains/pr-workflow/skills/evidence/scripts/capture.sh | 2 +- domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh | 2 +- domains/pr-workflow/skills/evidence/scripts/render-count.sh | 2 +- .../pr-workflow/skills/evidence/scripts/selector-recompute.sh | 2 +- domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh index 954e01a8..027f9eff 100755 --- a/domains/pr-workflow/skills/evidence/scripts/capture.sh +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -126,7 +126,7 @@ JSON echo "not cover was not recorded, which is not the same as it covering everything." fi echo - echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. Raw log: \`$STAMP.log\`.</sub>" + echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. Raw log: \`${STAMP##*/}.log\`.</sub>" } > "$STAMP.md" printf 'capture: %s (exit %s)\n %s\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.log" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh index cd46cf9c..5b1531da 100755 --- a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -156,7 +156,7 @@ JSON vacuous) echo "Worth a look: the mechanism is unguarded by this suite — what else depends on it?" ;; esac echo - echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" + echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes. Logs: \`${STAMP##*/}-armA.log\`, \`${STAMP##*/}-armB.log\`.</sub>" } > "$STAMP.md" printf 'falsify-probe: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh index d2a550c5..511a0e5a 100755 --- a/domains/pr-workflow/skills/evidence/scripts/render-count.sh +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -121,7 +121,7 @@ JSON echo "an interaction this probe does not perform. The probe also does not check that the" echo "consumer renders the same OUTPUT, only that it renders fewer times." echo - echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" + echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Logs: \`${STAMP##*/}-armA.log\`, \`${STAMP##*/}-armB.log\`.</sub>" } > "$STAMP.md" printf 'render-count: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index e08788bb..d0f4c7e5 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -176,7 +176,7 @@ JSON echo "says nothing about the cost of each recomputation. Worth a look if the fixture is thin" echo "relative to the shapes this selector sees in production." echo - echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Log: \`$STAMP.log\`.</sub>" + echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Log: \`${STAMP##*/}.log\`.</sub>" } > "$STAMP.md" printf 'selector-recompute: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh index 6fe4e987..38ba7fa0 100755 --- a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -133,7 +133,7 @@ JSON echo "divergence reachable only at runtime, or only from a caller outside this repo, will not" echo "appear here. Worth a look at whether the authoritative type is itself correct." echo - echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" + echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. Logs: \`${STAMP##*/}-armA.log\`, \`${STAMP##*/}-armB.log\`.</sub>" echo } > "$STAMP.md" From d55b32430f2fb98814fb4ab0d1dc22e5f374061a Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 12:02:36 -0400 Subject: [PATCH 29/63] Stop the gate failing runs whose environment is not the repo's MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Check 4 recognised only repo-toolchain pins — head SHA, lockfile hash, node version — so a browser-memory run pinned to `Firefox 153.0 headless` was told its pinned environment was unpinned. Check 10's vocabulary missed a run that stated its limit as "what it does not establish". Both were false negatives against real published runs, not missing content. --- .../skills/evidence/scripts/attest-gate.sh | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 4ceeed83..a3dcf266 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -41,9 +41,13 @@ hasre '^\*\*Verdict:\*\*.*\*\*Claim:\*\*' \ && pass "3 verdict line" \ || fail "3 verdict line" "no '**Verdict:** … — **Claim:** …' — valence is not legible at a glance" -hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `' \ +# A run outside the repo's toolchain pins a different thing. A browser-memory lane +# names "Firefox 153.0"; a repo lane names a head SHA and a lockfile hash. Both are +# pins, and a check that only knows the second one fails every run of the first — +# telling an author their pinned environment is unpinned. +hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `|[Ff]irefox [0-9]+\.[0-9]|[Cc]hrom(e|ium) [0-9]+\.|[Ss]afari [0-9]+\.|[Nn]ode v?[0-9]+\.[0-9]' \ && pass "4 environment pinned" \ - || fail "4 environment pinned" "no head SHA, toolchain version, or lockfile hash" + || fail "4 environment pinned" "no head SHA, lockfile hash, or pinned toolchain/browser version" # 5 — the one that matters. A tool-written log, a run link, or an image; not typed prose. if hasre '!\[|<img|data:image|actions/runs|/gist\.|evidence-artifacts/|Produced by '; then @@ -88,7 +92,12 @@ fi # measure and called that the whole picture. This is NOT satisfied by a "what would close # it" section, which check 6 rejects: that hands the reader the run's own unfinished work, # whereas this names a limit or a question the run is right to leave open. -if hasi 'open for review|raise with a human|falsifier|worth a look|left unmeasured|not covered by this run|no verdict offered'; then +# +# The vocabulary is a fixed list because this phase asks no model anything. That makes +# it blind to a limit phrased outside the list — a real run stated its limit as "what it +# does not establish" and the check called it absent. Add phrases when that happens; +# judging whether the stated limit is substantive is the dispatched passes' job. +if hasi 'open for review|raise with a human|falsifier|worth a look|left unmeasured|not covered by this run|no verdict offered|does not establish|what it does not|cannot attribute'; then pass "10 floats something for review" else fail "10 floats something for review" "no limit, open question, or falsifier named — an artifact that floats nothing implies its measurement was the whole surface" From 4ae53736d32197f09fbc288dcfc850796528511e Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 12:14:54 -0400 Subject: [PATCH 30/63] Let the caller say where its tool puts the finding Two thirds head, one third tail is a guess. `policy-audit.py` prints 1200 lines of worklist before its escalation section, so the default budget spent itself on checkboxes; `--head-lines 10 --tail-lines 36` cut the embedded block from 9K to 5K with more of the part a reader needs. The elision notice also printed an absolute path, same leak the `<sub>` line had. --- .../skills/evidence/scripts/capture.sh | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh index 027f9eff..f51d8391 100755 --- a/domains/pr-workflow/skills/evidence/scripts/capture.sh +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -10,7 +10,8 @@ # This wraps any command so the ARTIFACT is written by the tool. Nothing is retyped. # # capture.sh --label <slug> --lane <id> --claim "<under test>" [--verdict <word>] -# [--open "<what this run leaves open>"] -- <cmd...> +# [--open "<what this run leaves open>"] +# [--max-log-lines N | --head-lines N --tail-lines N] -- <cmd...> # # --verdict is stated by the caller, never inferred from the exit code: a wrapped # tool's exit convention is its own, and guessing prints "pass" over real findings. @@ -33,6 +34,7 @@ set -uo pipefail OUT_DIR="evidence-artifacts"; LABEL=""; LANE=""; CLAIM=""; MAXLOG=120; VERDICT=""; OPEN="" +HEADL=""; TAILL="" die() { printf 'capture: %s\n' "$1" >&2; exit 3; } while [ $# -gt 0 ]; do @@ -44,6 +46,11 @@ while [ $# -gt 0 ]; do --open) OPEN="${2:-}"; shift 2 ;; --out) OUT_DIR="${2:-}"; shift 2 ;; --max-log-lines) MAXLOG="${2:-}"; shift 2 ;; + # Two thirds head / one third tail is a guess about where the finding is. For a + # tool that escalates at the bottom the useful split is the other way round, and + # the caller knows which shape its tool has. + --head-lines) HEADL="${2:-}"; shift 2 ;; + --tail-lines) TAILL="${2:-}"; shift 2 ;; -h|--help) sed -n '2,26p' "$0"; exit 0 ;; --) shift; break ;; *) die "unknown argument: $1 (did you forget -- before the command?)" ;; @@ -104,15 +111,16 @@ JSON echo echo '```console' echo "\$ $CMD_STR" - if [ "$LINES" -gt "$MAXLOG" ]; then + BUDGET=$(( ${HEADL:-0} + ${TAILL:-0} )); [ "$BUDGET" -gt 0 ] || BUDGET="$MAXLOG" + if [ "$LINES" -gt "$BUDGET" ]; then # Elide the middle, never the end. A tool that escalates does it last: # policy-audit.py prints a per-grant worklist first and its RAISE WITH A HUMAN # section at the bottom, so head-truncation cuts exactly the rows that needed a # reader and leaves a wall of checkboxes in their place. - H=$(( MAXLOG * 2 / 3 )); T=$(( MAXLOG - H )) + H="${HEADL:-$(( MAXLOG * 2 / 3 ))}"; T="${TAILL:-$(( MAXLOG - H ))}" head -n "$H" "$STAMP.log" printf '\n… %s lines elided from the middle — full output in %s\n\n' \ - "$((LINES - MAXLOG))" "$STAMP.log" + "$((LINES - BUDGET))" "${STAMP##*/}.log" tail -n "$T" "$STAMP.log" else cat "$STAMP.log" From d32bdf703525234fc05ce5bb505c4b610aa15388 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 12:25:08 -0400 Subject: [PATCH 31/63] Write for the reviewer, not for a user of this skill The published runs led with a summary table and buried the captured blocks, which inverts what a reviewer opens the comment for, and each exhibit carried a lane id, a paragraph of generic instrument limits, and two log filenames pointing at files nobody can open. Three consecutive blocks meant the same boilerplate three times. The exhibits now lead and should outweigh the prose. Generic limits go to stderr and the `.json` so the orchestrator can read them and synthesise one question about the diff in front of it. Restating an artifact's number in prose is called out as its own defect: a figure that appears only in a typed sentence is a figure on the author's word. --- .../skills/evidence/scripts/capture.sh | 21 ++++----- .../skills/evidence/scripts/falsify-probe.sh | 18 ++++---- .../skills/evidence/scripts/render-count.sh | 16 ++++--- .../evidence/scripts/selector-recompute.sh | 16 ++++--- .../evidence/scripts/tsc-substitution.sh | 16 ++++--- domains/pr-workflow/skills/evidence/skill.md | 43 ++++++++++++++++++- 6 files changed, 89 insertions(+), 41 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh index f51d8391..60666fa6 100755 --- a/domains/pr-workflow/skills/evidence/scripts/capture.sh +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -102,9 +102,9 @@ JSON { if [ "$VERDICT" = "completed" ]; then - echo "### ${LANE:+$LANE — }ran to completion (exit $CODE) — read the output, no verdict asserted" + echo "### Ran to completion (exit $CODE) — read the output, no verdict asserted" else - echo "### ${LANE:+$LANE — }\`$VERDICT\` (exit $CODE)" + echo "### \`$VERDICT\` (exit $CODE)" fi echo echo "**Claim under test:** $CLAIM" @@ -127,15 +127,16 @@ JSON fi echo '```' echo - if [ -n "$OPEN" ]; then - echo "**Open for review** — $OPEN" - else - echo "**Open for review** — none stated. This tool answered one question; what it does" - echo "not cover was not recorded, which is not the same as it covering everything." - fi - echo - echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. Raw log: \`${STAMP##*/}.log\`.</sub>" + echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`.</sub>" } > "$STAMP.md" printf 'capture: %s (exit %s)\n %s\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.log" "$STAMP.json" "$STAMP.md" >&2 +# Stated limits reach the orchestrator, not the pasted exhibit: one open question per +# comment, about this diff, beats the same sentence repeated under every block. +if [ -n "$OPEN" ]; then + printf 'limits: %s\n' "$OPEN" >&2 +else + printf 'limits: none stated. This tool answered one question; what it does not cover was +not recorded, which is not the same as it covering everything.\n' >&2 +fi exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh index 5b1531da..2677e199 100755 --- a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -148,16 +148,16 @@ JSON esac [ -n "$FAILED_NAMES" ] && { echo; echo "Failing under mutation:"; echo; printf '%s\n' "$FAILED_NAMES" | sed 's/^/- /'; } echo - echo "**Open for review** — this run mutated one line of one file. It says nothing about" - echo "other paths into the same mechanism, whether the mechanism is reachable in production," - echo "or whether the behaviour it guards is the right behaviour. A falsifying test proves the" - echo "test has power, not that the fix is correct." - case "$VERDICT" in - vacuous) echo "Worth a look: the mechanism is unguarded by this suite — what else depends on it?" ;; - esac - echo - echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes. Logs: \`${STAMP##*/}-armA.log\`, \`${STAMP##*/}-armB.log\`.</sub>" + echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes.</sub>" } > "$STAMP.md" printf 'falsify-probe: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 +# The limits below are identical on every run: they describe the instrument, not the +# change under review. Pasted into a PR comment they read as boilerplate to a reviewer +# who has no stake in this tooling, so they go to stderr and to the .json instead. The +# orchestrator reads them and writes ONE open question about THIS diff. +printf 'limits: one line of one file was mutated. Says nothing about other paths into the +same mechanism, whether it is reachable in production, or whether the guarded behaviour is +correct. A falsifying test proves the test has power, not that the fix is right.%s\n' \ + "$([ "$VERDICT" = vacuous ] && printf '\n vacuous: the mechanism is unguarded by this suite — what else depends on it?')" >&2 exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh index 511a0e5a..19c5d8fc 100755 --- a/domains/pr-workflow/skills/evidence/scripts/render-count.sh +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -98,7 +98,7 @@ cat > "$STAMP.json" <<JSON JSON { - echo "### C4 — consumer render count" + echo "### Consumer render count" echo echo "**Verdict:** $VERDICT" echo @@ -116,13 +116,15 @@ JSON echo "This counts renders of one named consumer across a defined interaction. It is not a count" echo "of consumers, and a larger consumer count does not imply a larger effect." echo - echo "**Open for review** — one named consumer, one interaction. Other consumers of the same" - echo "provider are unmeasured, and a consumer that renders once here may render freely under" - echo "an interaction this probe does not perform. The probe also does not check that the" - echo "consumer renders the same OUTPUT, only that it renders fewer times." - echo - echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Logs: \`${STAMP##*/}-armA.log\`, \`${STAMP##*/}-armB.log\`.</sub>" + echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`.</sub>" } > "$STAMP.md" printf 'render-count: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 +# The limits below are identical on every run: they describe the instrument, not the +# change under review. Pasted into a PR comment they read as boilerplate to a reviewer +# who has no stake in this tooling, so they go to stderr and to the .json instead. The +# orchestrator reads them and writes ONE open question about THIS diff. +printf 'limits: one named consumer, one interaction. Other consumers are unmeasured, and one +that renders once here may render freely under an interaction this probe does not perform. +Counts renders, not whether the output is equivalent.\n' >&2 exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index d0f4c7e5..8bd8e69d 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -147,7 +147,7 @@ cat > "$STAMP.json" <<JSON JSON { - echo "### C4 — \`$EXPORT\` recomputation count" + echo "### \`$EXPORT\` recomputation count" echo echo "**Verdict:** $VERDICT" echo @@ -171,14 +171,16 @@ JSON echo "$LINE" echo '```' echo - echo "**Open for review** — measured against one fixture with one perturbed key. A selector" - echo "unmoved here can still recompute under state this fixture does not reach, and the count" - echo "says nothing about the cost of each recomputation. Worth a look if the fixture is thin" - echo "relative to the shapes this selector sees in production." - echo - echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Log: \`${STAMP##*/}.log\`.</sub>" + echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`.</sub>" } > "$STAMP.md" printf 'selector-recompute: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 +# The limits below are identical on every run: they describe the instrument, not the +# change under review. Pasted into a PR comment they read as boilerplate to a reviewer +# who has no stake in this tooling, so they go to stderr and to the .json instead. The +# orchestrator reads them and writes ONE open question about THIS diff. +printf 'limits: one fixture, one perturbed key. A selector unmoved here can still recompute +under state this fixture does not reach, and the count says nothing about the cost of each +recomputation.\n' >&2 [ -n "$A" ] || exit 2 exit 0 diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh index 38ba7fa0..0c397eae 100755 --- a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -106,7 +106,7 @@ cat > "$STAMP.json" <<JSON JSON { - echo "### D6 — authored-vs-authoritative substitution · \`$VERDICT\`" + echo "### Authored type vs authoritative source · \`$VERDICT\`" echo echo "| Arm | Change | distinct \`tsc\` errors |" echo "|---|---|---|" @@ -128,14 +128,16 @@ JSON echo "subtracted, not disqualifying — only errors new under substitution are the finding.</sub>" fi echo - echo "**Open for review** — the compiler answers only what the probe asks. A silent arm B means" - echo "no call site in this tree distinguishes the two shapes, which is not agreement: a" - echo "divergence reachable only at runtime, or only from a caller outside this repo, will not" - echo "appear here. Worth a look at whether the authoritative type is itself correct." - echo - echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. Logs: \`${STAMP##*/}-armA.log\`, \`${STAMP##*/}-armB.log\`.</sub>" + echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V.</sub>" echo } > "$STAMP.md" printf 'tsc-substitution: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 +# The limits below are identical on every run: they describe the instrument, not the +# change under review. Pasted into a PR comment they read as boilerplate to a reviewer +# who has no stake in this tooling, so they go to stderr and to the .json instead. The +# orchestrator reads them and writes ONE open question about THIS diff. +printf 'limits: the compiler answers only what the probe asks. A silent arm B means no call +site in this tree distinguishes the two shapes, which is not agreement — a divergence +reachable only at runtime, or from a caller outside this repo, will not appear here.\n' >&2 exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 7c7d1a72..81119fde 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -377,10 +377,30 @@ replace idempotently instead of accumulating: **Verdict:** ✅ proven — **Claim:** <one-line falsifiable behavior under test> head `<sha>` · <YYYY-MM-DD> · lanes: <lane ids> -<claim → artifact table; every claim binds its artifact> +<one sentence: what kind of evidence follows> + +<the captured artifacts, unfolded> + +**Follows from the above** +<terse bullets — each one a consequence of a number in an artifact above> + +**Open for review:** <the single question this run hands to a human, about THIS diff> <!-- VALIDATION_RUN_END --> ``` +**The exhibits are the comment.** A reviewer opens this to see a measurement, so the +captured blocks go in the body, not behind a `<details>`, and they should outweigh your +prose — 70% exhibit is a reasonable floor. Everything you write around them is a caption. + +**Never restate an artifact's number in your own prose.** A figure that appears only in a +sentence you typed is a figure on your word, which is the one thing this whole skill exists +to avoid. Cite by pointing at the block; a summary table above the exhibits duplicates the +artifact's own table and downgrades it. + +**Prose is the failure mode.** Lead with the conclusion, then the exhibits, then bullets. +Paragraphs of explanation read as an infodump and bury the finding; if a bullet needs three +sentences the exhibit is not doing its job. + Verdict icons: `✅` proven · `❌` failed · `ℹ️` otherwise. Never `❌` for a gap in *evidence* — that reads as a verdict on the author's work. @@ -401,6 +421,27 @@ The parts that decide *whether* to publish, rather than how: - **Don't restate CI.** Lint, build, and test results are already on the Checks tab. - **Scrub** local paths and usernames; failure summaries leak them. +### The reader is a reviewer on this PR, not a user of this skill + +They have a stake in the change and none in the tooling. Everything internal to how the +evidence was produced is noise to them, and several of these leaked into a published run +before anyone noticed: + +| Leaks | Publish instead | +|---|---| +| Lane ids — `B3`, `C4`, `D3` | The category in words: *falsifying test*, *render count* | +| A runner's generic limits, identical on every run | One open question about **this** diff | +| The runner's own name as though it means something | `Produced by <tool>` provenance, and nothing more | +| Your process — drafts, retractions, what you tried first | The measurement as it stands now | +| Anything calibrating the skill rather than the change | Nothing; delete it | + +The runners cooperate with this: their generic limits go to stderr and to the `.json`, not +into the `.md` exhibit, precisely so a reviewer never reads the same paragraph about the +instrument under three consecutive blocks. Read them there and synthesise **one** question. + +The test: would this sentence still be worth reading if the skill did not exist? If it is +only interesting to someone who knows how the tool works, cut it. + ## Validation output format When reporting back (before publishing), lead with the verdict and the claim it tests: From 7dee1f937da381757000dc285bed19ef7902f9ae Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 12:28:47 -0400 Subject: [PATCH 32/63] Move the output template into the skill, where corrections can land Three rounds of format corrections were applied to three draft comments and to nothing else, because the template they were assembled from lived in a scratch directory. A generator outside the repo makes fixing the instance and fixing the target two separate acts, and only the first one is visible, so the second gets skipped and the correction arrives again on the next run. `references/output-templates.md` is now the generator and says so. --- .../evidence/references/output-templates.md | 72 +++++++++++++++++++ domains/pr-workflow/skills/evidence/skill.md | 5 ++ 2 files changed, 77 insertions(+) create mode 100644 domains/pr-workflow/skills/evidence/references/output-templates.md diff --git a/domains/pr-workflow/skills/evidence/references/output-templates.md b/domains/pr-workflow/skills/evidence/references/output-templates.md new file mode 100644 index 00000000..607ef807 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/references/output-templates.md @@ -0,0 +1,72 @@ +# Output templates + +The shape a validation run ships in. **This file is the generator.** A correction to how a +run reads is a defect here, not in the comment it was noticed on — fix it here and regenerate, +or the same correction arrives again on the next run. + +Drafting a template in a scratch directory is how that goes wrong: the comment gets better and +nothing else does. + +## The template + +```markdown +<!-- VALIDATION_RUN_START --> +## 🧪 Validation Run + +**Verdict:** <icon> <the conclusion, in words a reviewer can act on> — **Claim:** <the +falsifiable thing under test> head `<sha>` · <YYYY-MM-DD> · <check name in words> + +<one sentence: what kind of evidence follows, and how it was arranged> + +<captured artifact> + +<one sentence, only if a second exhibit needs a transition> + +<captured artifact> + +**Follows from the above** + +- <a consequence of a number in an exhibit above> +- <another> + +**Open for review:** <the single question this run hands to a human, about THIS change> + +<sub>Trial run of the <a href="...">MetaMask evidence skills</a> — feedback welcome. Not a +review verdict; nothing here blocks the PR.</sub> +<!-- VALIDATION_RUN_END --> +``` + +## What each slot is for + +**Verdict line.** The conclusion, not the topic. *"one of the two conjuncts is tested"* and +*"six renders where one would do"* are conclusions; *"tested the hash predicate"* is a topic. +Icons: `✅` proven · `⚠️` partial or scoped · `📋` measured, no verdict asserted · `❌` failed. +Never `❌` for a gap in *evidence* — that reads as a verdict on the author's work. + +**Check name, in words.** *falsifying-test check*, *render-count check*, +*dependency-containment check*. Never the lane id: `B3` is an address into +[evidence-catalog.md](evidence-catalog.md), which the reviewer cannot open. + +**The exhibits.** Whatever the runner wrote, pasted whole and unfolded. They should outweigh +everything else in the comment; 70% is a reasonable floor. Do not summarise them above +themselves — a table of your own restating theirs turns a measurement into your word for it. + +**Follows from the above.** Bullets, each traceable to a number in an exhibit. If a bullet +needs three sentences, the exhibit is not carrying its weight. + +**Open for review.** One question, about this change. The runners' generic limits go to stderr +and the `.json` precisely so they do not end up here three times over; read them, and write +the thing a human should actually look at. + +## Assembly + +Templates carry `@@TOKEN@@` placeholders, one per exhibit, substituted with the runner's `.md` +verbatim. Substitution — never retyping — is what keeps the provenance line attached to the +numbers it vouches for. + +Before posting, `scripts/attest-gate.sh <file>` must exit 0. + +## Worked instantiations + +Three runs assembled from this template, with the reasoning behind each choice, are in +[worked-examples.md](worked-examples.md). diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 81119fde..65a907f2 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -404,6 +404,11 @@ sentences the exhibit is not doing its job. Verdict icons: `✅` proven · `❌` failed · `ℹ️` otherwise. Never `❌` for a gap in *evidence* — that reads as a verdict on the author's work. +The template itself, slot by slot, and how a run is assembled from it: +**[references/output-templates.md](references/output-templates.md)** — that file is the +generator, so a correction to how a run reads belongs there rather than in the comment it was +noticed on. + Full recipe — image re-hosting, recordings, AEP mirroring, the privacy scrub: **[references/evidence-publishing.md](references/evidence-publishing.md).** From 2c72486606b0d2090b409495ec8a931a312d5936 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 13:15:02 -0400 Subject: [PATCH 33/63] Require a finding, not a printout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A run was published reading "measured, no verdict asserted" over a policy delta and a 24-row escalation list, leaving the reader to work out whether any of it was good news. Withholding the conclusion feels like the rigorous move under this skill's own standards, but the numbers came from an instrument the reader does not have, and the run is the only party holding the context to read them. Records the usual cause — a number with no baseline to hold it against — and the fix, which is to find the comparison rather than to hedge. --- domains/pr-workflow/skills/evidence/skill.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 65a907f2..eef4b9ed 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -401,6 +401,24 @@ artifact's own table and downgrades it. Paragraphs of explanation read as an infodump and bury the finding; if a bullet needs three sentences the exhibit is not doing its job. +**There must be a finding.** A run ends with something the reader can agree or disagree with — +this holds, or it does not. Running the instruments and publishing what they printed is not +that. `📋 measured, no verdict asserted` is not a verdict, and neither is a headline that +reports a delta: *"494 → 651 packages, 24 rows escalated"* is a measurement in a verdict's +clothes, leaving the reader to work out whether it is good news. + +The tell is a verdict line you cannot restate as a sentence with a subject and a verb. +Withholding the conclusion feels like the rigorous move under this skill's standards, but the +numbers came from an instrument the reader does not have; the run is the only party holding the +context to interpret them, and declining to is offloading rather than restraint. State the +conclusion **and** its limits — floating a concern is the residue of a finding, not a +substitute for having one. + +When no finding presents itself, the usual cause is a missing comparison rather than a +genuinely inconclusive result: a number with nothing to hold it against. Find the baseline that +turns it into a claim — a sibling artifact, the other build target, the previous release, the +arm the change did not touch. + Verdict icons: `✅` proven · `❌` failed · `ℹ️` otherwise. Never `❌` for a gap in *evidence* — that reads as a verdict on the author's work. From 41329516f660c47f2dd02fbff0dfb972733b243f Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 14:53:41 -0400 Subject: [PATCH 34/63] Check where the disclaimer is, not just that it is there Three rounds of editing for density demoted the trial-run disclaimer from a callout under the verdict to `<sub>` at the foot of the page. It survived in every comment and did nothing in any of them: it is the frame a reviewer needs before reading a verdict on their own PR, and from the bottom it arrives after the reaction it exists to shape. A fifth comment had none at all. A disclaimer is not content. Content earns its place by density and a frame earns it by arriving first, so a uniform compression pass will always demote it. Check 11 compares its line against the first exhibit's; existence alone passed in all four demoted comments. --- .../evidence/references/output-templates.md | 13 +++++++++++-- .../skills/evidence/scripts/attest-gate.sh | 16 ++++++++++++++++ 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/output-templates.md b/domains/pr-workflow/skills/evidence/references/output-templates.md index 607ef807..0aae9992 100644 --- a/domains/pr-workflow/skills/evidence/references/output-templates.md +++ b/domains/pr-workflow/skills/evidence/references/output-templates.md @@ -16,6 +16,11 @@ nothing else does. **Verdict:** <icon> <the conclusion, in words a reviewer can act on> — **Claim:** <the falsifiable thing under test> head `<sha>` · <YYYY-MM-DD> · <check name in words> +> [!NOTE] +> Trial run of the [MetaMask evidence skills](<link>) — feedback welcome, on the finding or +> on whether this format is useful to a reviewer. Not a review verdict; nothing here blocks +> the PR. + <one sentence: what kind of evidence follows, and how it was arranged> <captured artifact> @@ -31,8 +36,6 @@ falsifiable thing under test> head `<sha>` · <YYYY-MM-DD> · <check name in wor **Open for review:** <the single question this run hands to a human, about THIS change> -<sub>Trial run of the <a href="...">MetaMask evidence skills</a> — feedback welcome. Not a -review verdict; nothing here blocks the PR.</sub> <!-- VALIDATION_RUN_END --> ``` @@ -54,6 +57,12 @@ themselves — a table of your own restating theirs turns a measurement into you **Follows from the above.** Bullets, each traceable to a number in an exhibit. If a bullet needs three sentences, the exhibit is not carrying its weight. +**The disclaimer sits directly under the verdict, and stays a callout.** It is the frame a +reviewer needs *before* they read a verdict on their own PR from a source they have not seen +before — where feedback goes, and that nothing here blocks them. Edited by the same rules as +prose it drifts to the foot of the page in `<sub>`, where it arrives after the reaction it +exists to shape. Check 11 tests its position, not just its presence. + **Open for review.** One question, about this change. The runners' generic limits go to stderr and the `.json` precisely so they do not end up here three times over; read them, and write the thing a human should actually look at. diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index a3dcf266..2cb0d395 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -103,6 +103,22 @@ else fail "10 floats something for review" "no limit, open question, or falsifier named — an artifact that floats nothing implies its measurement was the whole surface" fi +# 11 — the trial-run disclaimer, and its POSITION. This is not content, it is the frame +# the reader needs before they read a verdict on their own PR from an unfamiliar source. +# Compressed and moved to the foot of the page — which is what happens when it is edited +# by the same rules as prose — it arrives after the reaction it exists to shape. +DISC="$(grep -n -i 'trial run' "$FILE" | head -1 | cut -d: -f1)" +FIRST_EXHIBIT="$(grep -n '^```' "$FILE" | head -1 | cut -d: -f1)" +if [ -z "$DISC" ]; then + fail "11 disclaimer present and early" "no trial-run disclaimer — a reviewer cannot tell what this is or where to send feedback" +elif ! grep -qi 'trial run' "$FILE" || ! grep -q 'skills/pull/\|MetaMask/skills' "$FILE"; then + fail "11 disclaimer present and early" "disclaimer does not link the skills PR, so feedback has nowhere to go" +elif [ -n "$FIRST_EXHIBIT" ] && [ "$DISC" -gt "$FIRST_EXHIBIT" ]; then + fail "11 disclaimer present and early" "disclaimer is at line $DISC, after the first exhibit at line $FIRST_EXHIBIT — it frames nothing from there" +else + pass "11 disclaimer present and early" +fi + if [ -n "$REF" ] && [ -f "$REF" ]; then r=$(grep -coE '!\[|<img|data:image' "$REF"); c=$(grep -coE '!\[|<img|data:image|evidence-artifacts/|Produced by' "$FILE") echo From e58d1c0e1c5100e8bc245269d6163cd08f527e49 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 14:56:34 -0400 Subject: [PATCH 35/63] Stop a script's summary of itself passing as a capture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A run shipped a table the script composed, one grepped line, and the command `yarn jest <generated probe>` — a path that had already been deleted. The tool's real output was in the `.log` beside it and never shown. The gate passed it because `Produced by` was present, and a marker attests who wrote a block rather than what the block contains: it is equally true of a paraphrase. That is the failure this suite exists to prevent — an operator retyping output — displaced one layer down, where the retyping is done by the script and carries a machine's byline. `selector-recompute` now prints the real command and passes jest's own stdout through, and keeps the probe beside the artifact instead of deleting the file its command line names. Check 5 fails a `$` line carrying a placeholder: nothing else in a console block advertises "composed, not captured" so plainly, and it greps. --- .../skills/evidence/scripts/attest-gate.sh | 14 +++++++++++--- .../skills/evidence/scripts/selector-recompute.sh | 15 +++++++++++---- 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 2cb0d395..1cbcb72b 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -50,10 +50,18 @@ hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `|[Ff]irefox [0-9]+\.[0-9]|[ || fail "4 environment pinned" "no head SHA, lockfile hash, or pinned toolchain/browser version" # 5 — the one that matters. A tool-written log, a run link, or an image; not typed prose. -if hasre '!\[|<img|data:image|actions/runs|/gist\.|evidence-artifacts/|Produced by '; then - pass "5 captured artifact" -else +# +# `Produced by` attests who WROTE the block, not that the block is the tool's own output. +# A script that composes a summary table and stamps itself passes on the marker alone — +# which is how a run shipped with a table the script had written, one grepped line, and a +# command reading `yarn jest <generated probe>`. A `$` line carrying a placeholder is the +# tell: it looks reproducible and cannot be run. +if ! hasre '!\[|<img|data:image|actions/runs|/gist\.|evidence-artifacts/|Produced by '; then fail "5 captured artifact" "every block appears operator-typed; no tool-written log, run link, or image referenced" +elif grep -qE '^\$ .*<[a-z][a-z ._-]*>' "$FILE"; then + fail "5 captured artifact" "a console command contains a placeholder — $(grep -m1 -oE '^\$ .*' "$FILE") is not a command a reader can run" +else + pass "5 captured artifact" fi if hasi 'what would close it|what would prove it|closing it requires'; then diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index 8bd8e69d..46b756d2 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -63,7 +63,12 @@ MOD_BASE="./$(basename "$MODULE")" DEPTH="$(dirname "$MODULE" | tr -cd '/' | wc -c | tr -d ' ')" UP=""; i=0; while [ "$i" -le "$DEPTH" ]; do UP="../$UP"; i=$((i+1)); done -cleanup() { rm -f "$PROBE"; } +# The probe used to be deleted on exit, which left the exhibit quoting +# `yarn jest <generated probe>` — a command line that cannot be run, printed where a +# reader expects a reproducible one. It is kept alongside the artifact instead, and +# removed from the working tree so the repo is left clean. +KEEP_PROBE="$STAMP.probe.test.ts" +cleanup() { [ -f "$PROBE" ] && cp "$PROBE" "$KEEP_PROBE"; rm -f "$PROBE"; } trap cleanup EXIT INT TERM cat > "$PROBE" <<PROBEEOF @@ -167,11 +172,13 @@ JSON fi echo echo '```console' - echo "\$ yarn jest <generated probe>" - echo "$LINE" + echo "\$ yarn jest $PROBE" + # The tool's own output, not a line this script composed. A summary a script writes + # about its own run carries the script's word; the runner's stdout carries the run's. + grep -E "RECOMPUTE_PROBE |^Test Suites:|^Tests: |^Time: " "$STAMP.log" | head -8 echo '```' echo - echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`.</sub>" + echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and kept beside this artifact. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`.</sub>" } > "$STAMP.md" printf 'selector-recompute: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 From 3c86466852aa5cc79a53c7b6360f28a7776a509a Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 19:20:26 -0400 Subject: [PATCH 36/63] Ask for a medium, not for better text MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Check 5 has been rewritten four times today and each time it tested a property of the plaintext — a provenance marker, a placeholder in the command, a local path. Each caught one defect and missed the next, because every property of text is forgeable by whatever emits the text. Four runs shipped through it. It now requires a capture the reader verifies without going through the author: an image of the tool's surface, a link that re-executes, or a hosted artifact. A fenced block sits beside one of those and is never the evidence itself. --- .../skills/evidence/scripts/attest-gate.sh | 32 +++++++++++++++++-- domains/pr-workflow/skills/evidence/skill.md | 20 ++++++++++++ 2 files changed, 49 insertions(+), 3 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 1cbcb72b..cc454e32 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -49,17 +49,43 @@ hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `|[Ff]irefox [0-9]+\.[0-9]|[ && pass "4 environment pinned" \ || fail "4 environment pinned" "no head SHA, lockfile hash, or pinned toolchain/browser version" -# 5 — the one that matters. A tool-written log, a run link, or an image; not typed prose. +# 5 — the one that matters, and it asks for a MEDIUM, not for better text. +# +# Every earlier version of this check tested a property of the plaintext: does it carry a +# provenance marker, does the command contain a placeholder, is the path local. Each caught +# one defect and missed the next, because every property of plaintext is forgeable by +# whatever emits the plaintext. Four runs shipped that way. +# +# So the block below is necessary but is no longer the evidence. The evidence is an image +# of the tool's own surface, a link that re-executes, or a hosted artifact the reader +# fetches without going through the author. If the artifact is small, nothing was attached. # # `Produced by` attests who WROTE the block, not that the block is the tool's own output. # A script that composes a summary table and stamps itself passes on the marker alone — # which is how a run shipped with a table the script had written, one grepped line, and a # command reading `yarn jest <generated probe>`. A `$` line carrying a placeholder is the # tell: it looks reproducible and cannot be run. -if ! hasre '!\[|<img|data:image|actions/runs|/gist\.|evidence-artifacts/|Produced by '; then - fail "5 captured artifact" "every block appears operator-typed; no tool-written log, run link, or image referenced" +# An image, a re-executing link, or a hosted artifact — verification that does not route +# through the author. `Produced by` and `evidence-artifacts/` are provenance, not this. +if ! hasre '!\[[^]]*\]\(https?://|<img [^>]*src="https?://|actions/runs/[0-9]|/gist\.|https?://[^ )]+\.(png|jpg|jpeg|gif|svg|txt|log|json)\b'; then + fail "5 captured artifact" "no reader-verifiable capture — an image of the tool surface, a run link, or a hosted artifact. A fenced block is the author\'s transcription, whatever produced it" + # No separate attribution test: a hosted artifact the reader fetches is its own + # attribution, and requiring `Produced by` on top of it only fails runs whose + # evidence is stronger than a stamped fenced block. elif grep -qE '^\$ .*<[a-z][a-z ._-]*>' "$FILE"; then fail "5 captured artifact" "a console command contains a placeholder — $(grep -m1 -oE '^\$ .*' "$FILE") is not a command a reader can run" +elif grep -qE '^\$ .*(/tmp/|/home/|/Users/)' "$FILE"; then + # A helper script in /tmp, or any absolute local path, is unreproducible by + # construction. `capture.sh` records the command honestly — but honestly + # recording `bash /tmp/dup.sh` still publishes a recipe nobody else can follow. + # Inline the commands, or ship the helper where the reader can reach it. + fail "5 captured artifact" "a console command references a local-only path — $(grep -m1 -oE '^\$ .*(/tmp/|/home/|/Users/)[^ ]*' "$FILE") cannot be run by a reader" +elif [ "$(grep -cE '^\$ ' "$FILE")" -gt 1 ] && \ + [ "$(grep -E '^\$ ' "$FILE" | sed 's/ *#.*$//' | sort -u | wc -l)" -lt "$(grep -cE '^\$ ' "$FILE")" ]; then + # Two identical commands shown as producing different outputs. The difference + # came from an edit made between runs, so the block misstates its own cause: + # running it twice reproduces the first number twice. + fail "5 captured artifact" "two console commands are identical but shown with different output — the block does not say what actually differed between them" else pass "5 captured artifact" fi diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index eef4b9ed..f400bfb7 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -392,6 +392,26 @@ head `<sha>` · <YYYY-MM-DD> · lanes: <lane ids> captured blocks go in the body, not behind a `<details>`, and they should outweigh your prose — 70% exhibit is a reasonable floor. Everything you write around them is a caption. +**A fenced block is not the evidence.** Nothing in it distinguishes real stdout from invented +stdout, or from real stdout that has since drifted from its source — and whatever would +fabricate it is what formats it. Where fabrication, hallucination, or drift is a concern at +all, and that is nearly all plaintext, the medium is wrong. + +The qualifying media are the ones where verification does not route through you: + +- an **image of the tool's own surface** — the run page, the Discover view, the waterfall; +- a **link that re-executes or re-renders** — a CI run, a query permalink, a dashboard; +- a **hosted artifact the reader fetches** — the log at a URL, not a quotation of it. + +Paste the fenced block *beside* one of those, never instead of one. This is also why a +plaintext check can never close the gap: every property of text is forgeable by whatever emits +the text, so the gate asks for a different medium rather than for better text. + +**If the artifact is small, nothing was attached.** The reference showcase runs to 2 MB because +it carries 31 embedded captures. Uploading costs a step and a decision about what may be +published; that cost is the price of the reader not having to trust you, and a pipeline with no +upload step has no evidence step. + **Never restate an artifact's number in your own prose.** A figure that appears only in a sentence you typed is a figure on your word, which is the one thing this whole skill exists to avoid. Cite by pointing at the block; a summary table above the exhibits duplicates the From dbc275429d92d00daf4c9138edabe93f526fc26b Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 20:26:16 -0400 Subject: [PATCH 37/63] Move the measurement to CI, where the run URL is the capture A local run's only witness is its author, so it cannot satisfy the requirement that a reader verify without going through them. Every defect class this suite has shipped was a local-environment one: a helper in `/tmp`, a probe deleted after the run, an absolute path, a drifted toolchain, a contended host whose numbers were published and retracted. None is expressible in CI. Two controls ride along because both were learned the hard way. A `baseline` input: twice a run reported no finding when what it lacked was a comparison. A determinism check: the head arm runs twice and the numbers are not endorsed if they move. Inputs reach the shell through `env`, never spliced into the script text. --- .../skills/evidence/assets/evidence-run.yml | 169 ++++++++++++++++++ domains/pr-workflow/skills/evidence/skill.md | 12 ++ 2 files changed, 181 insertions(+) create mode 100644 domains/pr-workflow/skills/evidence/assets/evidence-run.yml diff --git a/domains/pr-workflow/skills/evidence/assets/evidence-run.yml b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml new file mode 100644 index 00000000..cdbdbab9 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml @@ -0,0 +1,169 @@ +# Evidence runner — install into the consumer repo as +# `.github/workflows/evidence-run.yml`. +# +# Why this exists rather than running the measurement locally: a validation run is a +# claim that a command produced an output, and the reader has to be able to check that +# without going through the author. A local run's only witness is the author. Every +# failure class this suite has shipped was a local-environment failure — a helper script +# in /tmp, a probe deleted after the run, an absolute path, a toolchain that drifted, a +# contended host producing numbers that had to be retracted. +# +# In CI none of those is expressible. The workflow file is the recipe, the workspace is +# the repo, the run records its own ref, and the run URL is itself the capture — +# `actions/runs/<id>` is what the publish gate accepts. +# +# Trigger from the CLI: +# gh workflow run evidence-run.yml \ +# -f runner=falsify-probe \ +# -f ref=<sha> \ +# -f args='--test path/to.test.ts --source path/to.ts --line 9 --replace " return x;"' +# +# Then cite the run URL in the comment. The artifacts are attached to the run; the +# orchestrator reads them to write the finding. +name: Evidence run + +'on': + workflow_dispatch: + inputs: + runner: + description: Runner to execute + required: true + type: choice + options: + - falsify-probe + - selector-recompute + - render-count + - tsc-substitution + - capture + ref: + description: Commit SHA to measure. Pin it — a branch name makes the run unrepeatable. + required: true + type: string + args: + description: Arguments passed to the runner, verbatim + required: true + type: string + baseline: + description: >- + Second SHA to measure identically. Twice now a run reported "no finding" when it + had no comparison, so the baseline arm is offered here rather than left to memory. + required: false + type: string + +permissions: + contents: read + +jobs: + measure: + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - name: Reject a moving ref + env: + REF: ${{ inputs.ref }} + BASE: ${{ inputs.baseline }} + run: | + # A branch name makes the artifact unrepeatable, which is the property this + # workflow exists to provide. Checked before anything is fetched. + for r in "$REF" ${BASE:+"$BASE"}; do + case "$r" in + *[!0-9a-f]* | "") echo "::error::'$r' is not a commit SHA"; exit 1 ;; + esac + [ ${#r} -eq 40 ] || { echo "::error::'$r' must be the full 40-char SHA"; exit 1; } + done + + - name: Checkout at the measured ref + uses: actions/checkout@v6 + with: + ref: ${{ inputs.ref }} + fetch-depth: 2 # the runners diff against the parent + + - uses: actions/setup-node@v4 + with: + node-version-file: .nvmrc + cache: yarn + + - name: Install + run: yarn --immutable + + - name: Fetch the runners at a pinned version + uses: actions/checkout@v6 + with: + repository: MetaMask/skills + ref: ${{ vars.EVIDENCE_SKILLS_REF || 'main' }} + path: .evidence-skills + sparse-checkout: domains/pr-workflow/skills/evidence/scripts + + - name: Run + id: run + continue-on-error: true # the exit code IS the verdict; a finding is not a failure + env: + RUNNER: ${{ inputs.runner }} + ARGS: ${{ inputs.args }} + run: | + RUNNERS=.evidence-skills/domains/pr-workflow/skills/evidence/scripts + mkdir -p evidence-artifacts + set +e + eval "bash \"$RUNNERS/$RUNNER.sh\" $ARGS --label \"$RUNNER-head\" --out evidence-artifacts" + code=$? + set -e + echo "head_exit=$code" >> "$GITHUB_OUTPUT" + echo "runner exited $code — the exit code is the verdict, not a build failure" \ + >> "$GITHUB_STEP_SUMMARY" + + - name: Baseline arm + if: inputs.baseline != '' + continue-on-error: true + env: + RUNNER: ${{ inputs.runner }} + ARGS: ${{ inputs.args }} + BASE: ${{ inputs.baseline }} + run: | + git checkout --detach "$BASE" + yarn --immutable + RUNNERS=.evidence-skills/domains/pr-workflow/skills/evidence/scripts + eval "bash \"$RUNNERS/$RUNNER.sh\" $ARGS --label \"$RUNNER-base\" --out evidence-artifacts" + + - name: Determinism check + # Contention produced numbers that were published and then retracted. Running the + # head arm twice and diffing costs one repeat and turns that into a pre-publish + # signal rather than a correction. + continue-on-error: true + env: + RUNNER: ${{ inputs.runner }} + ARGS: ${{ inputs.args }} + REF: ${{ inputs.ref }} + run: | + RUNNERS=.evidence-skills/domains/pr-workflow/skills/evidence/scripts + git checkout --detach "$REF" + eval "bash \"$RUNNERS/$RUNNER.sh\" $ARGS --label \"$RUNNER-repeat\" --out evidence-artifacts" || true + A="evidence-artifacts/$RUNNER-head.json" + B="evidence-artifacts/$RUNNER-repeat.json" + if [ -f "$A" ] && [ -f "$B" ]; then + # env block differs by design (timing); compare the measurement only + if diff <(jq -S 'del(.env)' "$A") <(jq -S 'del(.env)' "$B") > determinism.diff; then + echo "deterministic across two runs" | tee -a "$GITHUB_STEP_SUMMARY" + else + echo "::warning::runner is NOT deterministic at this ref — do not publish these numbers" + cat determinism.diff >> "$GITHUB_STEP_SUMMARY" + fi + fi + + - name: Publish the run summary + if: always() + run: | + for f in evidence-artifacts/*.md; do + [ -f "$f" ] || continue + { echo; cat "$f"; } >> "$GITHUB_STEP_SUMMARY" + done + + - name: Upload artifacts + if: always() + uses: actions/upload-artifact@v4 + with: + name: evidence-${{ inputs.runner }}-${{ inputs.ref }} + path: | + evidence-artifacts/** + determinism.diff + retention-days: 90 + if-no-files-found: error diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index f400bfb7..20ffa882 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -412,6 +412,18 @@ it carries 31 embedded captures. Uploading costs a step and a decision about wha published; that cost is the price of the reader not having to trust you, and a pipeline with no upload step has no evidence step. +**Run the measurement in CI, not locally.** `assets/evidence-run.yml` installs into the consumer +repo and dispatches any runner at a pinned SHA. This is not about convenience: a local run's +only witness is you, so it cannot meet the requirement above, and every defect class this suite +has shipped was a local-environment one — a helper in `/tmp`, a probe deleted after the run, an +absolute path, a drifted toolchain, a contended host whose numbers had to be retracted. None of +those is expressible in CI, where the workflow file is the recipe, the workspace is the repo, +and the run URL is itself the capture. + +It also carries two controls worth having by default: a `baseline` input, because twice a run +reported "no finding" when what it lacked was a comparison; and a determinism check that runs +the head arm twice and refuses to endorse numbers that move. + **Never restate an artifact's number in your own prose.** A figure that appears only in a sentence you typed is a figure on your word, which is the one thing this whole skill exists to avoid. Cite by pointing at the block; a summary table above the exhibits duplicates the From bdf5f39ec61a8e99858bb0d1526bfffd20c1f504 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 20:28:25 -0400 Subject: [PATCH 38/63] Pin the runner source, and say so when it is missing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The workflow fetched the runners from the skills repo's `main`, where they do not exist — they are on this branch. Every dispatch would have failed at that step, and a sparse checkout of an absent path succeeds with an empty directory, so the failure would have surfaced two steps later as "No such file" with no hint that the ref was the cause. Pinned to a commit for the same reason the measured ref must be, with an explicit check that names EVIDENCE_SKILLS_REF as the thing to change. --- .../skills/evidence/assets/evidence-run.yml | 22 ++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/evidence/assets/evidence-run.yml b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml index cdbdbab9..1ce37316 100644 --- a/domains/pr-workflow/skills/evidence/assets/evidence-run.yml +++ b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml @@ -86,14 +86,34 @@ jobs: - name: Install run: yarn --immutable + # Pinned to a commit, for the same reason the measured ref must be: a branch name + # makes the run unrepeatable, and "which version of the runner produced this" is + # exactly the question a reader asks. Override per-repo with the EVIDENCE_SKILLS_REF + # variable; bump the default when the runners land on the skills repo's main. - name: Fetch the runners at a pinned version uses: actions/checkout@v6 with: repository: MetaMask/skills - ref: ${{ vars.EVIDENCE_SKILLS_REF || 'main' }} + ref: ${{ vars.EVIDENCE_SKILLS_REF || 'dbc275429d92d00daf4c9138edabe93f526fc26b' }} path: .evidence-skills sparse-checkout: domains/pr-workflow/skills/evidence/scripts + - name: Verify the runners arrived + env: + RUNNER: ${{ inputs.runner }} + run: | + # A sparse checkout of a path that does not exist on the chosen ref succeeds and + # produces an empty directory, so the next step would fail with "No such file" + # and no indication that the REF was the problem. + F=".evidence-skills/domains/pr-workflow/skills/evidence/scripts/$RUNNER.sh" + [ -f "$F" ] || { + echo "::error::$RUNNER.sh not present at the pinned skills ref." + echo "::error::Set the EVIDENCE_SKILLS_REF repository variable to a commit that has it." + exit 1 + } + echo "runner $RUNNER.sh sourced from skills @ ${{ vars.EVIDENCE_SKILLS_REF || 'pinned default' }}" \ + >> "$GITHUB_STEP_SUMMARY" + - name: Run id: run continue-on-error: true # the exit code IS the verdict; a finding is not a failure From 5c41bfbf8ab23901075e074194c33e9381b19238 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sun, 2 Aug 2026 08:06:45 -0400 Subject: [PATCH 39/63] Require the target repo, make install opt-out, drop the consumer-install framing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three corrections to the runner workflow, all found by pointing it somewhere. `target_repo` is required with no default: its stated purpose is to be proved somewhere harmless first, and a default pointed the first dispatch at the repo under review. `needs_install` skips a ten-minute yarn install for runners that only read files. And the header still described installing this into the repo under review, which stopped being true once the target became an input — that stale sentence is what made a fork of the extension look necessary. --- .../skills/evidence/assets/evidence-run.yml | 43 +++++++++++++++---- domains/pr-workflow/skills/evidence/skill.md | 5 ++- 2 files changed, 37 insertions(+), 11 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/assets/evidence-run.yml b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml index 1ce37316..b3db840f 100644 --- a/domains/pr-workflow/skills/evidence/assets/evidence-run.yml +++ b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml @@ -1,5 +1,8 @@ -# Evidence runner — install into the consumer repo as -# `.github/workflows/evidence-run.yml`. +# Evidence runner — lives in ONE repo and measures any other. +# +# It does not need to be installed in the repo under review, and there is no reason to +# fork that repo either: `target_repo` is an input and the job checks it out read-only. +# Put this in whatever repo you want the runs and artifacts to belong to. # # Why this exists rather than running the measurement locally: a validation run is a # claim that a command produced an output, and the reader has to be able to check that @@ -35,6 +38,13 @@ name: Evidence run - render-count - tsc-substitution - capture + target_repo: + description: >- + Repository to measure. No default on purpose: a default here is a standing + decision about what every unthinking dispatch touches, and this workflow's whole + argument is that it should be proved somewhere harmless first. + required: true + type: string ref: description: Commit SHA to measure. Pin it — a branch name makes the run unrepeatable. required: true @@ -43,6 +53,13 @@ name: Evidence run description: Arguments passed to the runner, verbatim required: true type: string + needs_install: + description: >- + Install the target's dependencies. Required for the jest and tsc runners; a waste + of ten minutes for `capture` wrapping git or a policy audit, which read files only. + required: false + default: true + type: boolean baseline: description: >- Second SHA to measure identically. Twice now a run reported "no finding" when it @@ -72,18 +89,21 @@ jobs: [ ${#r} -eq 40 ] || { echo "::error::'$r' must be the full 40-char SHA"; exit 1; } done - - name: Checkout at the measured ref + - name: Checkout the target at the measured ref uses: actions/checkout@v6 with: + repository: ${{ inputs.target_repo }} ref: ${{ inputs.ref }} fetch-depth: 2 # the runners diff against the parent - uses: actions/setup-node@v4 + if: inputs.needs_install with: node-version-file: .nvmrc cache: yarn - name: Install + if: inputs.needs_install run: yarn --immutable # Pinned to a commit, for the same reason the measured ref must be: a branch name @@ -124,7 +144,7 @@ jobs: RUNNERS=.evidence-skills/domains/pr-workflow/skills/evidence/scripts mkdir -p evidence-artifacts set +e - eval "bash \"$RUNNERS/$RUNNER.sh\" $ARGS --label \"$RUNNER-head\" --out evidence-artifacts" + eval "bash \"$RUNNERS/$RUNNER.sh\" --label \"$RUNNER-head\" --out evidence-artifacts $ARGS" code=$? set -e echo "head_exit=$code" >> "$GITHUB_OUTPUT" @@ -140,9 +160,9 @@ jobs: BASE: ${{ inputs.baseline }} run: | git checkout --detach "$BASE" - yarn --immutable + if [ "${{ inputs.needs_install }}" = "true" ]; then yarn --immutable; fi RUNNERS=.evidence-skills/domains/pr-workflow/skills/evidence/scripts - eval "bash \"$RUNNERS/$RUNNER.sh\" $ARGS --label \"$RUNNER-base\" --out evidence-artifacts" + eval "bash \"$RUNNERS/$RUNNER.sh\" --label \"$RUNNER-base\" --out evidence-artifacts $ARGS" - name: Determinism check # Contention produced numbers that were published and then retracted. Running the @@ -156,12 +176,15 @@ jobs: run: | RUNNERS=.evidence-skills/domains/pr-workflow/skills/evidence/scripts git checkout --detach "$REF" - eval "bash \"$RUNNERS/$RUNNER.sh\" $ARGS --label \"$RUNNER-repeat\" --out evidence-artifacts" || true + eval "bash \"$RUNNERS/$RUNNER.sh\" --label \"$RUNNER-repeat\" --out evidence-artifacts $ARGS" || true A="evidence-artifacts/$RUNNER-head.json" B="evidence-artifacts/$RUNNER-repeat.json" if [ -f "$A" ] && [ -f "$B" ]; then - # env block differs by design (timing); compare the measurement only - if diff <(jq -S 'del(.env)' "$A") <(jq -S 'del(.env)' "$B") > determinism.diff; then + # `label` and `log` name the arm, and `env` carries timing — all three differ + # between the two runs by construction. Comparing them makes the check fire on + # every run, which is the same as not having it. + if diff <(jq -S 'del(.env, .label, .log)' "$A") \ + <(jq -S 'del(.env, .label, .log)' "$B") > determinism.diff; then echo "deterministic across two runs" | tee -a "$GITHUB_STEP_SUMMARY" else echo "::warning::runner is NOT deterministic at this ref — do not publish these numbers" @@ -182,6 +205,8 @@ jobs: uses: actions/upload-artifact@v4 with: name: evidence-${{ inputs.runner }}-${{ inputs.ref }} + # the artifact name carries what was measured, so a downloaded zip is + # self-describing rather than needing the run page to interpret path: | evidence-artifacts/** determinism.diff diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 20ffa882..664f747a 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -412,8 +412,9 @@ it carries 31 embedded captures. Uploading costs a step and a decision about wha published; that cost is the price of the reader not having to trust you, and a pipeline with no upload step has no evidence step. -**Run the measurement in CI, not locally.** `assets/evidence-run.yml` installs into the consumer -repo and dispatches any runner at a pinned SHA. This is not about convenience: a local run's +**Run the measurement in CI, not locally.** `assets/evidence-run.yml` lives in one repo and +measures any other — `target_repo` is an input and the checkout is read-only, so the repo under +review needs no workflow, no fork, and no change of any kind. This is not about convenience: a local run's only witness is you, so it cannot meet the requirement above, and every defect class this suite has shipped was a local-environment one — a helper in `/tmp`, a probe deleted after the run, an absolute path, a drifted toolchain, a contended host whose numbers had to be retracted. None of From 8d1ec29542945da113ebfec9dbde2b89b07a75b6 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sun, 2 Aug 2026 08:07:58 -0400 Subject: [PATCH 40/63] Make every runner say whether a reader can verify it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Putting the showcase runs and the trial runs in one table shows the causation running the wrong way: eight of eleven hand-built runs attach a capture a reader can open, against one of twelve built by these runners. Nothing was neglected. A runner emits clean stdout, clean stdout formats beautifully into a fenced block, and a fenced block looks like evidence — so automating the measurement automated away the part that made it checkable. The hand-built runs had no such thing to reach for and went and got a real capture. Each runner's provenance line now prints the run URL under CI, and under a local run prints that there is no reader-verifiable capture and the workflow should be used before publishing. The confession belongs in the exhibit, not in a gate that has to remember to look for it. --- .../pr-workflow/skills/evidence/scripts/capture.sh | 14 +++++++++++++- .../skills/evidence/scripts/falsify-probe.sh | 14 +++++++++++++- .../skills/evidence/scripts/render-count.sh | 14 +++++++++++++- .../skills/evidence/scripts/selector-recompute.sh | 14 +++++++++++++- .../skills/evidence/scripts/tsc-substitution.sh | 14 +++++++++++++- domains/pr-workflow/skills/evidence/skill.md | 11 +++++++++++ 6 files changed, 76 insertions(+), 5 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh index 60666fa6..298cd619 100755 --- a/domains/pr-workflow/skills/evidence/scripts/capture.sh +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -33,6 +33,18 @@ # -- python3 retention-scan.py ui/store/background-connection.ts pr.patch set -uo pipefail +# A run's artifact has to say whether a reader can verify it. In CI the run URL is that +# verification; locally there is none, and the artifact says so rather than leaving the +# omission for a gate to catch later. +capture_provenance() { + if [ -n "${GITHUB_RUN_ID:-}" ]; then + printf 'Run: %s/%s/actions/runs/%s — logs and artifacts attached there.' \ + "${GITHUB_SERVER_URL:-https://github.com}" "${GITHUB_REPOSITORY:-}" "$GITHUB_RUN_ID" + else + printf 'Produced on a local machine: no reader-verifiable capture. Re-run through the evidence workflow before publishing.' + fi +} + OUT_DIR="evidence-artifacts"; LABEL=""; LANE=""; CLAIM=""; MAXLOG=120; VERDICT=""; OPEN="" HEADL=""; TAILL="" die() { printf 'capture: %s\n' "$1" >&2; exit 3; } @@ -127,7 +139,7 @@ JSON fi echo '```' echo - echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`.</sub>" + echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. $(capture_provenance)</sub>" } > "$STAMP.md" printf 'capture: %s (exit %s)\n %s\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.log" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh index 2677e199..afec5d71 100755 --- a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -34,6 +34,18 @@ # --label coalesce-inflight set -uo pipefail +# A run's artifact has to say whether a reader can verify it. In CI the run URL is that +# verification; locally there is none, and the artifact says so rather than leaving the +# omission for a gate to catch later. +capture_provenance() { + if [ -n "${GITHUB_RUN_ID:-}" ]; then + printf 'Run: %s/%s/actions/runs/%s — logs and artifacts attached there.' \ + "${GITHUB_SERVER_URL:-https://github.com}" "${GITHUB_REPOSITORY:-}" "$GITHUB_RUN_ID" + else + printf 'Produced on a local machine: no reader-verifiable capture. Re-run through the evidence workflow before publishing.' + fi +} + RUNNER="yarn jest" OUT_DIR="evidence-artifacts" LABEL="" @@ -148,7 +160,7 @@ JSON esac [ -n "$FAILED_NAMES" ] && { echo; echo "Failing under mutation:"; echo; printf '%s\n' "$FAILED_NAMES" | sed 's/^/- /'; } echo - echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes.</sub>" + echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes. $(capture_provenance)</sub>" } > "$STAMP.md" printf 'falsify-probe: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh index 19c5d8fc..eaf0ee3d 100755 --- a/domains/pr-workflow/skills/evidence/scripts/render-count.sh +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -35,6 +35,18 @@ # 3 usage error set -uo pipefail +# A run's artifact has to say whether a reader can verify it. In CI the run URL is that +# verification; locally there is none, and the artifact says so rather than leaving the +# omission for a gate to catch later. +capture_provenance() { + if [ -n "${GITHUB_RUN_ID:-}" ]; then + printf 'Run: %s/%s/actions/runs/%s — logs and artifacts attached there.' \ + "${GITHUB_SERVER_URL:-https://github.com}" "${GITHUB_REPOSITORY:-}" "$GITHUB_RUN_ID" + else + printf 'Produced on a local machine: no reader-verifiable capture. Re-run through the evidence workflow before publishing.' + fi +} + OUT_DIR="evidence-artifacts"; LABEL=""; PROBE=""; DEFEAT=""; DEFEAT_LINE=""; DEFEAT_WITH="" ARM_B="memo defeated" die() { printf 'render-count: %s\n' "$1" >&2; exit 3; } @@ -116,7 +128,7 @@ JSON echo "This counts renders of one named consumer across a defined interaction. It is not a count" echo "of consumers, and a larger consumer count does not imply a larger effect." echo - echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`.</sub>" + echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. $(capture_provenance)</sub>" } > "$STAMP.md" printf 'render-count: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index 46b756d2..5cf6e757 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -28,6 +28,18 @@ # --fixture test/data/mock-state.json --slice metamask --perturb pinnedAccountList set -uo pipefail +# A run's artifact has to say whether a reader can verify it. In CI the run URL is that +# verification; locally there is none, and the artifact says so rather than leaving the +# omission for a gate to catch later. +capture_provenance() { + if [ -n "${GITHUB_RUN_ID:-}" ]; then + printf 'Run: %s/%s/actions/runs/%s — logs and artifacts attached there.' \ + "${GITHUB_SERVER_URL:-https://github.com}" "${GITHUB_REPOSITORY:-}" "$GITHUB_RUN_ID" + else + printf 'Produced on a local machine: no reader-verifiable capture. Re-run through the evidence workflow before publishing.' + fi +} + N=5; OUT_DIR="evidence-artifacts"; LABEL=""; MODULE=""; EXPORT=""; FIXTURE=""; SLICE="metamask"; PERTURB="" die() { printf 'selector-recompute: %s\n' "$1" >&2; exit 3; } @@ -178,7 +190,7 @@ JSON grep -E "RECOMPUTE_PROBE |^Test Suites:|^Tests: |^Time: " "$STAMP.log" | head -8 echo '```' echo - echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and kept beside this artifact. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`.</sub>" + echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and kept beside this artifact. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. $(capture_provenance)</sub>" } > "$STAMP.md" printf 'selector-recompute: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh index 0c397eae..cd8e4e03 100755 --- a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -29,6 +29,18 @@ # [--tsc "<command>"] set -uo pipefail +# A run's artifact has to say whether a reader can verify it. In CI the run URL is that +# verification; locally there is none, and the artifact says so rather than leaving the +# omission for a gate to catch later. +capture_provenance() { + if [ -n "${GITHUB_RUN_ID:-}" ]; then + printf 'Run: %s/%s/actions/runs/%s — logs and artifacts attached there.' \ + "${GITHUB_SERVER_URL:-https://github.com}" "${GITHUB_REPOSITORY:-}" "$GITHUB_RUN_ID" + else + printf 'Produced on a local machine: no reader-verifiable capture. Re-run through the evidence workflow before publishing.' + fi +} + TSC="yarn lint:tsc"; OUT_DIR="evidence-artifacts"; LABEL="" FILE=""; LINE=""; REPLACE=""; PROBE_LINE=""; PROBE="" die() { printf 'tsc-substitution: %s\n' "$1" >&2; exit 3; } @@ -128,7 +140,7 @@ JSON echo "subtracted, not disqualifying — only errors new under substitution are the finding.</sub>" fi echo - echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V.</sub>" + echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. $(capture_provenance)</sub>" echo } > "$STAMP.md" diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 664f747a..81c1e391 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -407,6 +407,17 @@ Paste the fenced block *beside* one of those, never instead of one. This is also plaintext check can never close the gap: every property of text is forgeable by whatever emits the text, so the gate asks for a different medium rather than for better text. +**Automation is what removes the capture — watch for it.** The runs built by hand, before these +runners existed, attached images and hosted logs: eight of eleven carry a capture a reader can +open. The runs built by the runners attached one in twelve. Nothing was neglected; the causation +runs the other way. A runner emits clean stdout, clean stdout formats beautifully into a fenced +block, and a fenced block looks like evidence. The hand-built runs had no such thing to reach +for, so they went and got a real one. + +Every runner now states this in its own artifact: in CI it prints the run URL, and on a local +machine it prints *"no reader-verifiable capture — re-run through the evidence workflow before +publishing."* The confession is in the exhibit rather than left for a gate to catch. + **If the artifact is small, nothing was attached.** The reference showcase runs to 2 MB because it carries 31 embedded captures. Uploading costs a step and a decision about what may be published; that cost is the price of the reader not having to trust you, and a pipeline with no From 56578cee0f679881e6f928177ef3cf6d45a5bfec Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sun, 2 Aug 2026 08:11:50 -0400 Subject: [PATCH 41/63] Cite what exists; capture what you ran MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Plaintext is the wrong medium" was too broad. The rule is about where verification routes, and a line-level permalink routes it away from the author exactly as an image does — the reader clicks and sees what you saw. That is the normal case for the audit lanes, whose findings are facts about code that exists rather than results of running something. A screenshot of a policy diff is less checkable than a permalink to it, not more. The bar there is comprehensive linking: a call site per grant, a re-runnable search per claimed absence, a file and line per version claim. A row naming a package and a capability with no link is a claim on the author's word in a technical register. --- domains/pr-workflow/skills/evidence/skill.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 81c1e391..26fc0bbc 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -407,6 +407,26 @@ Paste the fenced block *beside* one of those, never instead of one. This is also plaintext check can never close the gap: every property of text is forgeable by whatever emits the text, so the gate asks for a different medium rather than for better text. +**The exception — plaintext where every claim is a citation.** The rule is about where +verification routes, not about pixels. Line-level links are externally verifiable: the reader +clicks and sees exactly what you saw. That is the *normal* case for the audit lanes — +`supply-chain-audit`, `lavamoat-policy-diligence`, `privacy-egress-diligence` — whose findings +are facts about code that exists rather than results of running something. There an image would +be worse: a screenshot of a policy diff is less checkable than a permalink to it. + +The bar for those lanes is comprehensive linking, not a link somewhere nearby: + +- every capability grant → a permalink to its **call site** in the dependency's source, at the + installed version, with the line; +- every *"no call site uses this"* → the **search that establishes the absence**, re-runnable; +- every version, advisory, or policy claim → the file and line it came from. + +An audit row naming a package and a capability with no link is the same defect as a bare +console block — a claim on your word, wearing a technical register. + +The general form: ask what the reader must do to check a claim. *Trust the transcription* means +the medium is wrong whatever it looks like. **Cite what exists; capture what you ran.** + **Automation is what removes the capture — watch for it.** The runs built by hand, before these runners existed, attached images and hosted logs: eight of eleven carry a capture a reader can open. The runs built by the runners attached one in twelve. Nothing was neglected; the causation From f6d76e87980444d2ee5c4f81957b29f21665da45 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sun, 2 Aug 2026 09:55:22 -0400 Subject: [PATCH 42/63] Fold the runner-workflow fixes back from where they were found MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four corrections, each from a run that failed rather than from review. `corepack enable` before `setup-node`: a target pinning its package manager via `packageManager` makes setup-node's `cache: yarn` probe run under the runner's global yarn 1.22, which refuses — every jest runner died at setup with nothing measured. `logs` added to the determinism exclusions, since `render-count` is the one runner writing the plural key and so failed that check on every run while reporting identical counts; a warning always wrong for one runner teaches its reader to publish through it. A `probe_path` input, because `render-count` takes a hand-written probe and a probe living only on the author's disk is the exact defect a run URL exists to remove. And `skills_repo`/`skills_ref`, because a runner fix and the run that needs it cannot both wait on a review. The sparse checkout now also pulls `domains/security/skills`, so the analysis scripts wrapped by `capture` are on disk without being runners themselves. --- .../skills/evidence/assets/evidence-run.yml | 75 +++++++++++++++++-- 1 file changed, 67 insertions(+), 8 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/assets/evidence-run.yml b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml index b3db840f..1a65b24d 100644 --- a/domains/pr-workflow/skills/evidence/assets/evidence-run.yml +++ b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml @@ -66,6 +66,30 @@ name: Evidence run had no comparison, so the baseline arm is offered here rather than left to memory. required: false type: string + skills_repo: + description: >- + Where to source the runners. Defaults to upstream. Overridable because a runner + fix and the run that needs it cannot both wait on a review: point this at a fork + branch, and say in the artifact that you did. + required: false + default: MetaMask/skills + type: string + skills_ref: + description: >- + Ref within skills_repo. Overrides the EVIDENCE_SKILLS_REF variable. + required: false + type: string + probe_path: + description: >- + Path, within skills_repo, of a probe file to copy into the target tree before the + runner executes. `render-count` takes a hand-written probe, and a probe that lives + only on the author's disk is the exact defect the run URL exists to remove. + required: false + type: string + probe_dest: + description: Where in the target tree to place probe_path. Required with probe_path. + required: false + type: string permissions: contents: read @@ -96,6 +120,15 @@ jobs: ref: ${{ inputs.ref }} fetch-depth: 2 # the runners diff against the parent + # Before setup-node, not after. A target that pins its package manager through + # `packageManager` in package.json makes setup-node's `cache: yarn` probe run + # `yarn cache dir` under the runner's global yarn 1.22, which refuses and fails + # the step — so every jest runner died at setup with nothing measured. The target + # repo's own workflows order it exactly this way. + - name: Enable corepack + if: inputs.needs_install + run: corepack enable + - uses: actions/setup-node@v4 if: inputs.needs_install with: @@ -113,10 +146,18 @@ jobs: - name: Fetch the runners at a pinned version uses: actions/checkout@v6 with: - repository: MetaMask/skills - ref: ${{ vars.EVIDENCE_SKILLS_REF || 'dbc275429d92d00daf4c9138edabe93f526fc26b' }} + repository: ${{ inputs.skills_repo || 'MetaMask/skills' }} + ref: ${{ inputs.skills_ref || vars.EVIDENCE_SKILLS_REF || '56578cee0f679881e6f928177ef3cf6d45a5bfec' }} path: .evidence-skills - sparse-checkout: domains/pr-workflow/skills/evidence/scripts + # The security-domain analysis scripts (policy-audit.py and its siblings) are + # wrapped by the `capture` runner rather than being runners themselves, so they + # need no entry in the `runner` choice list — but they do need to be on disk. + # A sparse path absent from the chosen ref is silently empty, so listing it here + # costs nothing when it is not there. + sparse-checkout: | + domains/pr-workflow/skills/evidence/scripts + domains/pr-workflow/skills/evidence/probes + domains/security/skills - name: Verify the runners arrived env: @@ -131,9 +172,24 @@ jobs: echo "::error::Set the EVIDENCE_SKILLS_REF repository variable to a commit that has it." exit 1 } - echo "runner $RUNNER.sh sourced from skills @ ${{ vars.EVIDENCE_SKILLS_REF || 'pinned default' }}" \ + echo "runner $RUNNER.sh sourced from ${{ inputs.skills_repo || 'MetaMask/skills' }} @ ${{ inputs.skills_ref || vars.EVIDENCE_SKILLS_REF || 'pinned default' }}" \ >> "$GITHUB_STEP_SUMMARY" + - name: Place the probe + if: inputs.probe_path != '' + env: + SRC: .evidence-skills/${{ inputs.probe_path }} + DEST: ${{ inputs.probe_dest }} + run: | + # Copied from the runners checkout, so the probe has a permalink of its own and + # the reader can see the file that produced the count rather than taking the + # count on the author's word. + [ -n "$DEST" ] || { echo "::error::probe_dest is required with probe_path"; exit 1; } + [ -f "$SRC" ] || { echo "::error::probe not found at $SRC on the chosen skills ref"; exit 1; } + mkdir -p "$(dirname "$DEST")" + cp "$SRC" "$DEST" + echo "probe $SRC -> $DEST" >> "$GITHUB_STEP_SUMMARY" + - name: Run id: run continue-on-error: true # the exit code IS the verdict; a finding is not a failure @@ -180,11 +236,14 @@ jobs: A="evidence-artifacts/$RUNNER-head.json" B="evidence-artifacts/$RUNNER-repeat.json" if [ -f "$A" ] && [ -f "$B" ]; then - # `label` and `log` name the arm, and `env` carries timing — all three differ + # `label`, `log` and `logs` name the arm, and `env` carries timing — all differ # between the two runs by construction. Comparing them makes the check fire on - # every run, which is the same as not having it. - if diff <(jq -S 'del(.env, .label, .log)' "$A") \ - <(jq -S 'del(.env, .label, .log)' "$B") > determinism.diff; then + # every run, which is the same as not having it. `logs` was missing from this + # list, so `render-count` — the only runner that writes the plural key — failed + # the check on every run while reporting identical counts. A warning that is + # always wrong for one runner teaches the operator to publish through it. + if diff <(jq -S 'del(.env, .label, .log, .logs)' "$A") \ + <(jq -S 'del(.env, .label, .log, .logs)' "$B") > determinism.diff; then echo "deterministic across two runs" | tee -a "$GITHUB_STEP_SUMMARY" else echo "::warning::runner is NOT deterministic at this ref — do not publish these numbers" From 0b874cf1e3681a1c0580cb5512617a453cd3abdb Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sun, 2 Aug 2026 16:10:08 -0400 Subject: [PATCH 43/63] Name the commit in the command line, so an A/B pair is not one line twice A base arm and a head arm both printed `yarn jest <probe>`, leaving the reader no way to tell which commit produced which number. On a clean result, where the two arms agree, the block reads as a single measurement printed twice. The publish gate fails a pair of identical `$` lines for exactly this reason and caught it on a real comment. Fixing the check's input rather than the check. --- domains/pr-workflow/skills/evidence/scripts/render-count.sh | 2 +- .../pr-workflow/skills/evidence/scripts/selector-recompute.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh index eaf0ee3d..05de89a1 100755 --- a/domains/pr-workflow/skills/evidence/scripts/render-count.sh +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -120,7 +120,7 @@ JSON [ -n "$B" ] && echo "| B — $ARM_B | \`$DEFEAT:$DEFEAT_LINE\` | $B |" echo echo '```console' - echo "\$ yarn jest $PROBE" + echo "\$ git checkout --detach $HEAD_SHA && yarn jest $PROBE" echo "$A_LINE" [ -n "$B_LINE" ] && { echo "\$ yarn jest $PROBE # $ARM_B"; echo "$B_LINE"; } echo '```' diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index 5cf6e757..029d5409 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -184,7 +184,7 @@ JSON fi echo echo '```console' - echo "\$ yarn jest $PROBE" + echo "\$ git checkout --detach $HEAD_SHA && yarn jest $PROBE" # The tool's own output, not a line this script composed. A summary a script writes # about its own run carries the script's word; the runner's stdout carries the run's. grep -E "RECOMPUTE_PROBE |^Test Suites:|^Tests: |^Time: " "$STAMP.log" | head -8 From a64ddb83d3de55066c99a81a18046d6a15ff4419 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sun, 2 Aug 2026 16:38:55 -0400 Subject: [PATCH 44/63] Stop a broken substitution reading as a divergence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First CI run of this runner, and it reported `divergence surfaced` with six new errors — all TS1109/TS1011/TS1128, the syntactic family. The substitution had landed one line above the type declaration and broken parsing, so arm B never type-checked at all. By error count that is indistinguishable from the local type genuinely disagreeing with its authoritative source. `falsify-probe` has carried the equivalent guard since a syntax-breaking mutation looked like a falsification. This runner shipped without one and had never executed in CI, which is where the gap surfaced. --- .../skills/evidence/scripts/tsc-substitution.sh | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh index cd8e4e03..48308f98 100755 --- a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -100,7 +100,16 @@ restore; trap - EXIT INT TERM NEW_ERRS="$(comm -13 "$STAMP-armA.set" "$STAMP-armB.set" | head -12)" NEW_COUNT="$(comm -13 "$STAMP-armA.set" "$STAMP-armB.set" | wc -l | tr -d ' ')" -if [ "$NEW_COUNT" -gt 0 ]; then +# TS1xxx is the syntactic family — "expression expected", "declaration expected". A +# substitution that lands on the wrong line breaks parsing and produces a pile of them, +# which reads as a large divergence and is worth nothing: the file never type-checked. +# `falsify-probe` has carried this guard since a broken mutation looked like a +# falsification; this runner shipped without it and reported six syntax errors as a +# divergence on its first CI run. +SYNTAX="$(comm -13 "$STAMP-armA.set" "$STAMP-armB.set" | grep -cE 'error TS1[0-9]{3}')" +if [ "$NEW_COUNT" -gt 0 ] && [ "$SYNTAX" -eq "$NEW_COUNT" ]; then + VERDICT="substitution broke parsing — $NEW_COUNT syntax error(s), nothing type-checked"; CODE=2 +elif [ "$NEW_COUNT" -gt 0 ]; then VERDICT="divergence surfaced"; CODE=0 else VERDICT="substitution silent"; CODE=1 @@ -127,8 +136,14 @@ JSON echo "| **new under substitution** | | **$NEW_COUNT** |" echo if [ "$NEW_COUNT" -gt 0 ]; then + if [ "$CODE" -eq 2 ]; then + echo "**No conclusion.** Every new error is syntactic, so arm B never type-checked —" + echo "the substitution landed on the wrong line or produced invalid TypeScript. This is" + echo "indistinguishable from a real divergence by error count alone." + else echo "Errors present in B and absent in A — what the local type was concealing:" echo; echo '```'; printf '%s\n' "$NEW_ERRS"; echo '```' + fi else echo "**Silent — this is not proof of agreement.** Existing call sites may satisfy both" echo "shapes; indexing and \`.match()\` compile against \`string\` and \`string[]\` alike." From bfdf6b4a37d9eabe510daa02ebf9c2409977716b Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sun, 2 Aug 2026 17:50:06 -0400 Subject: [PATCH 45/63] Require a run to measure the PR's range, and to say where its reach ends MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four runs measured `$SHA^..$SHA` on branches of five to twenty-nine commits and produced clean artifacts for a fraction of each change — same runner, same green run, nothing in the output distinguishing it from a finished measurement. So the range is now a non-negotiable, with the compare endpoint's `merge_base_commit.sha` named because `.base.sha` is the base branch tip and moves. Two rules alongside it, from the same batch: what a run could not see is a finding to state rather than a gap to omit, and the label on a number is caller-stated for the same reason the verdict is — a probe counting distinct context values published under a fixed "renders" heading passes every check while naming the wrong quantity. --- domains/pr-workflow/skills/evidence/skill.md | 26 ++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 26fc0bbc..584342f5 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -177,6 +177,32 @@ untested path touches funds, keys, persisted state, user-visible wrongness, or s planning tracker. Subject matter triggers this, not severity: the code cannot distinguish a missing gate from a deliberate one. +**7. Measure the pull request, which is a range, not a commit.** `$SHA^..$SHA` is one commit's +diff. On a twenty-six-commit branch it is a twenty-sixth of the change, and it looks exactly like a +finished measurement — same runner, same green run, same artifact. Take the head from +`.head.sha` and the base from `merge_base_commit.sha` on the compare endpoint, and say the range in +the comment so a reader can see what was covered. `.base.sha` is the base branch's tip, which moves +under you and is not where the branch left. + +**8. The label on a number is part of the number.** A runner reads a field out of a line its probe +printed; it knows the field's name and not what was counted. When a probe for a claim about value +identity counts distinct values, publishing that under a fixed heading of "renders" ships a correct +measurement of the wrong quantity, and every check passes. Whatever names the number is caller- +stated, like the verdict — and the comment points at the probe, which is the definition. + +### The claim is scoped to what the run could see + +A run measures a diff, a file, a probe. What sits behind an interface it calls is not in the +measurement, and a comment that speaks past that boundary is asserting rather than reporting. + +The instrumentation lanes make this concrete: a diff can show that a span is created and that a +flag gates it, and cannot show how often the surrounding package invokes the callback. That is not +a gap to apologise for — it is the finding. *"Cost scales with a call frequency decided in another +package, so nothing here bounds it"* is a real conclusion, and the reviewer is the person who knows +the number. + +Say where the edge is, in the comment, in the reviewer's terms. + ### The runner, not the recipe `scripts/falsify-probe.sh` proves a test is falsifying by mutation rather than by reading, and From faff610d0f642ba8de515a07a3b1b43f045a9f27 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 03:30:13 -0400 Subject: [PATCH 46/63] Require an instrument to publish the effect it had, not the instruction it took A mutation runner echoed its `--replace` argument into the artifact, so the two could never disagree. They did: an `awk -v` assignment escape-processed the value and wrote a different line than the one requested, narrowing a regex meant to be widened. The suite ran the same test count in both arms, a different test failed than the one targeted, and the run reported power over a mechanism it never touched. Nothing in the artifact could have shown it. --- domains/pr-workflow/skills/evidence/skill.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 584342f5..334133f6 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -190,6 +190,16 @@ identity counts distinct values, publishing that under a fixed heading of "rende measurement of the wrong quantity, and every check passes. Whatever names the number is caller- stated, like the verdict — and the comment points at the probe, which is the definition. +**9. An instrument reports what it did, never what it was asked to do.** A mutation runner that +echoes its `--replace` argument into the artifact cannot detect its own misfire, because the two +are the same string by construction. They came apart once: `awk -v r="$REPLACE"` escape-processes +the assignment, so a replacement of `/^[\s\S]{1,4096}$/u` was written to the file as +`/^[sS]{1,4096}$/u` — narrowing the regex it was meant to widen. Arm B ran the same test count as +arm A, so every guard was satisfied, a different test failed than the one targeted, and the run +reported the suite as having power over a mechanism it never touched. Read the mutated line back +off disk and publish that; keep the requested text beside it. The rule generalises past mutation: +wherever a runner takes an instruction and performs an effect, the artifact carries the effect. + ### The claim is scoped to what the run could see A run measures a diff, a file, a probe. What sits behind an interface it calls is not in the From 46372afee7ef855474a93b9554fc9328a0e95b78 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 06:05:55 -0400 Subject: [PATCH 47/63] Point `falsifying-test` at the harness, and separate it from its sibling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 6 asked for both runs shown and left the operator to produce them, which is the shape that yields a retyped paste indistinguishable from output nobody ran. The run workflow already takes `ref` and `baseline`, executes the same command at both commits, and attaches the artifacts to a URL a reader can open — so the step names it. The larger risk was confusion with `falsify-probe.sh`, which shares the two-arm shape and answers a different question: base-against-branch asks whether a test is connected to the reported bug, one-commit-with-a-mutated-line asks whether it notices the mechanism going away. A test can pass either and fail the other. The skill now states the distinction in a table rather than leaving two similarly named things to be conflated, and notes that the runner's guards mechanise this skill's own falsifier — a red arm that ran fewer tests, or failed to load, is refused rather than counted. --- .../testing/skills/falsifying-test/skill.md | 43 ++++++++++++++++--- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/domains/testing/skills/falsifying-test/skill.md b/domains/testing/skills/falsifying-test/skill.md index 7cfeb160..65f36e42 100644 --- a/domains/testing/skills/falsifying-test/skill.md +++ b/domains/testing/skills/falsifying-test/skill.md @@ -44,10 +44,16 @@ this skill lives. file or suite, scope the run to the new test (by name/path) so the red is attributable. A suite that was already red proves nothing about your assertion. -6. **Show both runs.** Base: the assertion failure, verbatim. Branch: the pass. Same command, - same filter, both commits identified. Captured terminal output beats a transcription — - retyped output is a self-report, and a real capture has caught errors that careful prose - missed. +6. **Show both runs, and let a tool write them down.** Base: the assertion failure, verbatim. + Branch: the pass. Same command, same filter, both commits identified. Retyped output is a + self-report — indistinguishable from output that was never produced — so the two arms want + to come out of a runner rather than a paste buffer. + + `evidence` ships the mechanism: its run workflow takes `ref` and `baseline`, checks out both + commits, executes the same command at each, and attaches the artifacts to a run URL a reader + can open without going through you. Wrapping the test command in `capture.sh` gets the same + property locally, minus the reader-verifiable half — and that runner's footer says so, in + the artifact, rather than leaving the gap for a reviewer to notice. 7. **Pair it with the issue.** The PR's `Fixes #N` plus a test named for the behaviour makes the causal chain checkable by a reader who runs nothing. @@ -78,10 +84,33 @@ Falsifying test — <test name> (Fixes #N) scoped: <how the run was limited to this test> ``` +## The sibling experiment, and why it is not this one + +`evidence` also ships `falsify-probe.sh`, which has the same two-arm shape and answers a +different question. The distinction is worth holding, because conflating them produces a proof +of the wrong thing: + +| | arms | question | +|---|---|---| +| **this skill** | base commit, branch commit — same test | is the test causally connected to the reported bug? | +| **`falsify-probe.sh`** | one commit, one line mutated | does the test fail when the mechanism it guards is removed? | + +A test can pass this skill and fail that one: it fails on base because the fix was not there, +and passes under mutation because it asserts something adjacent to the mechanism. The reverse +also happens. On a bug-fix PR you usually want both — the first proves the test is about *this +bug*, the second proves it will keep noticing. + +What the runner does mechanise is step 2. Its guards refuse to call a red arm a falsification +when the suite ran fewer tests than the baseline, or failed to load at all — which is this +skill's falsifier, enforced rather than remembered. It also takes the names of the tests you +expect to fail, so a red run in the wrong place is reported as such instead of passing as a +falsification. + ## Related -- `evidence` — packages this skill's output as its [falsifying regression test category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md). - The deterministic-interleaving category is the sibling for concurrency and temporal-ordering - bugs; `race-condition-repro` drives it. +- `evidence` — packages this skill's output as its [falsifying regression test category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md), + and supplies the two-commit run harness step 6 asks for. The deterministic-interleaving + category is the sibling for concurrency and temporal-ordering bugs; `race-condition-repro` + drives it. - `react-render-delta` — the same before/after discipline applied to a measured quantity rather than a boolean. From 1e7ef54804ab65a1680f4ec258daa62aa4f69c2d Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 06:09:26 -0400 Subject: [PATCH 48/63] Rename `falsifying-test` to `red-on-base` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things in this PR shared a root and a two-arm shape while answering different questions: the skill compares a base commit against a branch to show a test is connected to the reported bug, and `falsify-probe.sh` mutates one line at one commit to show a suite notices the mechanism going away. A test can satisfy either and fail the other, so the names had to stop rhyming. `red-on-base` names the skill's own discipline — the test must be red on the base commit — and shares no root with the runner. The evidence category keeps its name: a falsifying regression test is the artifact, and this is the procedure that produces one. The runner's stderr described its own result as "a falsifying test", which was the same collision inside the output of the thing causing it; it now says what it proves and points at the other experiment by name. --- .../skills/evidence/references/claim-extraction.md | 2 +- .../skills/evidence/references/evidence-catalog.md | 2 +- .../skills/evidence/references/output-templates.md | 2 +- .../pr-workflow/skills/evidence/scripts/falsify-probe.sh | 4 +++- domains/pr-workflow/skills/evidence/skill.md | 2 +- .../skills/{falsifying-test => red-on-base}/skill.md | 6 +++--- 6 files changed, 10 insertions(+), 8 deletions(-) rename domains/testing/skills/{falsifying-test => red-on-base}/skill.md (92%) diff --git a/domains/pr-workflow/skills/evidence/references/claim-extraction.md b/domains/pr-workflow/skills/evidence/references/claim-extraction.md index 9dde89d9..e5654608 100644 --- a/domains/pr-workflow/skills/evidence/references/claim-extraction.md +++ b/domains/pr-workflow/skills/evidence/references/claim-extraction.md @@ -42,7 +42,7 @@ A good claim is **falsifiable** (observable outcome + clear falsifier), **surfac |---|---| | "Improves performance" | "Opening the Activity tab: TBT drops below 200ms (was >600ms)" — name the interaction, metric, threshold | | "Fixes the bug" | "With privacy mode on, the Perps tab balance is masked" — observable behavior + precondition + surface | -| "Refactor, no behavior change" | Negation claim: "behavior of `<surface>` is unchanged" → prove via falsifying-test-stays-green / snapshot / identical output, **not** a screenshot | +| "Refactor, no behavior change" | Negation claim: "behavior of `<surface>` is unchanged" → prove via a red-on-base test that stays green / snapshot / identical output, **not** a screenshot | | "Adds a null check" (restates the diff) | "No crash when `<field>` is null on `<surface>`" — the behavior, not the code | | Body promises X, diff does Y | Not a claim — **flag the drift** to the author | diff --git a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md index 728aae6f..d7f82692 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md @@ -54,7 +54,7 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- ## B3. Falsifying regression test ⭐ - **Proves — strongest single proof a fix targets the bug:** a new test that **fails on `main` and passes on the branch**. Show both runs. - - **Engine: the `falsifying-test` skill.** + - **Engine: the `red-on-base` skill.** - **Capture:** add the test, run it on the PR branch (pass) and on the PR's **merge-base** (fail) — pin the base, don't use whatever `main` points at today. Pair with the PR's `Fixes #N`. **Read the base failure's message, not its exit code:** it must fail on the assertion that encodes the bug. A `ModuleNotFoundError`, a missing fixture, or an unrelated pre-existing red produces an identical non-zero exit and falsifies nothing. - **Reach for it:** every bug-fix PR. If you can't write a test that fails on main, question whether the fix addresses the reported bug. diff --git a/domains/pr-workflow/skills/evidence/references/output-templates.md b/domains/pr-workflow/skills/evidence/references/output-templates.md index 0aae9992..752ac449 100644 --- a/domains/pr-workflow/skills/evidence/references/output-templates.md +++ b/domains/pr-workflow/skills/evidence/references/output-templates.md @@ -46,7 +46,7 @@ falsifiable thing under test> head `<sha>` · <YYYY-MM-DD> · <check name in wor Icons: `✅` proven · `⚠️` partial or scoped · `📋` measured, no verdict asserted · `❌` failed. Never `❌` for a gap in *evidence* — that reads as a verdict on the author's work. -**Check name, in words.** *falsifying-test check*, *render-count check*, +**Check name, in words.** *red-on-base check*, *render-count check*, *dependency-containment check*. Never the lane id: `B3` is an address into [evidence-catalog.md](evidence-catalog.md), which the reviewer cannot open. diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh index afec5d71..d7e8753e 100755 --- a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -170,6 +170,8 @@ printf 'falsify-probe: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.js # orchestrator reads them and writes ONE open question about THIS diff. printf 'limits: one line of one file was mutated. Says nothing about other paths into the same mechanism, whether it is reachable in production, or whether the guarded behaviour is -correct. A falsifying test proves the test has power, not that the fix is right.%s\n' \ +correct. This probe proves the suite notices one mutated line, which is not the same as the +base-against-branch proof that a test is connected to the reported bug -- see the red-on-base +skill for that experiment.%s\n' \ "$([ "$VERDICT" = vacuous ] && printf '\n vacuous: the mechanism is unguarded by this suite — what else depends on it?')" >&2 exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 334133f6..9ac32af0 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -656,7 +656,7 @@ Where evidence sits in the PR lifecycle (see the public `pr-workflow` siblings): | category | engine | |---|---| - | B3 falsifying regression test | `/falsifying-test` | + | B3 falsifying regression test | `/red-on-base` | | B7 deterministic interleaving (concurrency / ordering) | `/race-condition-repro` | | C4 React render & selector proof | `/react-render-delta` | | C9 memory leak | `/memory-leak` | diff --git a/domains/testing/skills/falsifying-test/skill.md b/domains/testing/skills/red-on-base/skill.md similarity index 92% rename from domains/testing/skills/falsifying-test/skill.md rename to domains/testing/skills/red-on-base/skill.md index 65f36e42..0c7066af 100644 --- a/domains/testing/skills/falsifying-test/skill.md +++ b/domains/testing/skills/red-on-base/skill.md @@ -1,10 +1,10 @@ --- -name: falsifying-test -description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /falsifying-test, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by evidence as the engine behind its falsifying regression test evidence category. +name: red-on-base +description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /red-on-base, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by evidence as the engine behind its falsifying regression test evidence category. Named for its own discipline rather than for that category, so it is not mistaken for `falsify-probe.sh`, which shares the two-arm shape and answers a different question. maturity: experimental --- -# /falsifying-test +# /red-on-base Reach for this on **every bug-fix PR**. A test that passes on the branch proves the branch is green. A test that **fails on base and passes on the branch** proves the change is causally From 50b52dc90485e1086aaefa6c376e1cfe6df2c3d7 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 06:13:01 -0400 Subject: [PATCH 49/63] Move `red-on-base` out; this PR is the instrument MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every other engine `evidence` names ships in its own pull request — `memory-leak`, `race-condition-repro`, `supply-chain-audit`, `react-render-delta`, `agent-run-cost`. This one was the sole exception, and the reason was authoring order rather than design: it was written alongside the orchestrator before the split between the instrument and the reasoning that points it existed as a concept. It lands in #106 with the other reasoning skills, whose substance is the same kind — what counts as proof, and how a proof can look right while testing the wrong thing. What stays here is the machinery: the runners, the run workflow, the gate, the hooks. The B3 engine cell now names a skill that arrives in #106, which is a dangling name in a table rather than a broken link, and resolves whichever order the two merge. --- domains/testing/skills/red-on-base/skill.md | 116 -------------------- 1 file changed, 116 deletions(-) delete mode 100644 domains/testing/skills/red-on-base/skill.md diff --git a/domains/testing/skills/red-on-base/skill.md b/domains/testing/skills/red-on-base/skill.md deleted file mode 100644 index 0c7066af..00000000 --- a/domains/testing/skills/red-on-base/skill.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -name: red-on-base -description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /red-on-base, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by evidence as the engine behind its falsifying regression test evidence category. Named for its own discipline rather than for that category, so it is not mistaken for `falsify-probe.sh`, which shares the two-arm shape and answers a different question. -maturity: experimental ---- - -# /red-on-base - -Reach for this on **every bug-fix PR**. A test that passes on the branch proves the branch is -green. A test that **fails on base and passes on the branch** proves the change is causally -connected to the reported bug. Only the second is evidence, and the gap between them is where -this skill lives. - -> **Falsifier.** A test that fails on base for a reason unrelated to the bug. A missing import, -> a fixture the base commit doesn't have, a helper introduced by the branch, an unrelated -> pre-existing failure — every one produces a red run and a non-zero exit code that looks -> exactly like a correct falsification. **The exit code is not the evidence; the assertion -> message is.** - -## Method - -1. **Write the test against the reported behaviour, not the diff.** Start from the issue's - reproduction. A test derived from reading the fix tends to assert the fix's mechanism and - will pass on base the moment the mechanism is reachable by other means — or fail on base - for structural reasons rather than behavioural ones. - -2. **Run it on base FIRST, and read the failure output.** Not the exit code — the message. It - must fail on the **assertion that encodes the bug**: an expected value that differs, a state - that wasn't reached, an event that didn't fire. If base fails with a - `ModuleNotFoundError`, a syntax error, or a helper that doesn't exist yet, you have not - falsified anything; you have discovered that the test can't run there. - -3. **Pin the base explicitly.** Use the PR's actual merge-base, not whatever `main` points at - today. `main` moves; a re-run weeks later against a drifted `main` is a different - experiment and may fail for reasons that have nothing to do with the fix. - -4. **Make the test runnable on base.** When the test needs a helper or fixture the branch - introduces, split it: land the scaffolding in a form that exists on both sides, or inline - the setup so the test file is self-contained. If that's impossible, say so and downgrade the - claim — a test that *cannot* run on base gives a branch-only pass, which is a weaker piece - of evidence and should not be presented as a falsifying one. - -5. **Confirm it fails for one reason, not several.** If base has unrelated failures in the same - file or suite, scope the run to the new test (by name/path) so the red is attributable. A - suite that was already red proves nothing about your assertion. - -6. **Show both runs, and let a tool write them down.** Base: the assertion failure, verbatim. - Branch: the pass. Same command, same filter, both commits identified. Retyped output is a - self-report — indistinguishable from output that was never produced — so the two arms want - to come out of a runner rather than a paste buffer. - - `evidence` ships the mechanism: its run workflow takes `ref` and `baseline`, checks out both - commits, executes the same command at each, and attaches the artifacts to a run URL a reader - can open without going through you. Wrapping the test command in `capture.sh` gets the same - property locally, minus the reader-verifiable half — and that runner's footer says so, in - the artifact, rather than leaving the gap for a reviewer to notice. - -7. **Pair it with the issue.** The PR's `Fixes #N` plus a test named for the behaviour makes - the causal chain checkable by a reader who runs nothing. - -## When you can't write one - -This is a finding, not a gap to paper over. If no test fails on base, one of these is true: - -- **The bug isn't where the fix is.** The most common case, and the reason to run this check - before review rather than after. -- **The reported behaviour isn't reproducible in the harness** — timing, environment, or a - real-device dependency. Say which, and reach for a different evidence category (a - deterministic interleaving test for ordering bugs, an e2e trace for environment-dependent - ones). -- **The fix is a refactor or hardening change, not a bug fix.** Fine — then the PR's claim - should say that, and this category doesn't apply. - -State which one. "No test added" with no explanation reads as an omission; the diagnosis is -useful information about the change. - -## Output - -``` -Falsifying test — <test name> (Fixes #N) - base <sha> FAIL <the assertion line, verbatim> - branch <sha> PASS - command: <exact command, same on both> - scoped: <how the run was limited to this test> -``` - -## The sibling experiment, and why it is not this one - -`evidence` also ships `falsify-probe.sh`, which has the same two-arm shape and answers a -different question. The distinction is worth holding, because conflating them produces a proof -of the wrong thing: - -| | arms | question | -|---|---|---| -| **this skill** | base commit, branch commit — same test | is the test causally connected to the reported bug? | -| **`falsify-probe.sh`** | one commit, one line mutated | does the test fail when the mechanism it guards is removed? | - -A test can pass this skill and fail that one: it fails on base because the fix was not there, -and passes under mutation because it asserts something adjacent to the mechanism. The reverse -also happens. On a bug-fix PR you usually want both — the first proves the test is about *this -bug*, the second proves it will keep noticing. - -What the runner does mechanise is step 2. Its guards refuse to call a red arm a falsification -when the suite ran fewer tests than the baseline, or failed to load at all — which is this -skill's falsifier, enforced rather than remembered. It also takes the names of the tests you -expect to fail, so a red run in the wrong place is reported as such instead of passing as a -falsification. - -## Related - -- `evidence` — packages this skill's output as its [falsifying regression test category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md), - and supplies the two-commit run harness step 6 asks for. The deterministic-interleaving - category is the sibling for concurrency and temporal-ordering bugs; `race-condition-repro` - drives it. -- `react-render-delta` — the same before/after discipline applied to a measured quantity - rather than a boolean. From 594a1a7f03a238a0ca4711285f43f118a95af303 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 08:46:51 -0400 Subject: [PATCH 50/63] Check where the run is going, not only what it says MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eleven checks tested properties of the comment text. Text can be perfect and land somewhere nobody will read it, and that is what happened: across one register of published runs, 22 of 27 comments went onto pull requests that had already merged when they were posted — median 22 days after the merge, one 178 days after. The gate passed every one, because no property of a comment reveals the state of its destination. Check 12 takes `--target owner/repo#N` and blocks on anything that is not open. Omitting the target fails rather than passes: an unchecked destination is the condition that produced all 22. Without `gh` on PATH it reports UNVERIFIED and still refuses, since the point is that silence here is indistinguishable from success. --- .../skills/evidence/scripts/attest-gate.sh | 37 +++++++++++++++++-- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index cc454e32..3f058d5f 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -10,12 +10,22 @@ # # 0 all checks pass → proceed to the dispatched passes # 1 one or more failed → BLOCKED, do not publish +# +# --target owner/repo#N is how check 12 learns where this is going. Without it the gate +# cannot tell a live review from a merged one, and the difference is the whole point. # 2 usage error set -uo pipefail -FILE="${1:-}"; REF="" -[ $# -ge 2 ] && [ "${2:-}" = "--reference" ] && REF="${3:-}" -[ -n "$FILE" ] || { echo "usage: attest-gate.sh <artifact.md> [--reference <file>]" >&2; exit 2; } +FILE="${1:-}"; REF=""; TARGET="" +shift || true +while [ $# -gt 0 ]; do + case "$1" in + --reference) REF="${2:-}"; shift 2 ;; + --target) TARGET="${2:-}"; shift 2 ;; + *) shift ;; + esac +done +[ -n "$FILE" ] || { echo "usage: attest-gate.sh <artifact.md> [--reference <file>] [--target <owner/repo#N>]" >&2; exit 2; } [ -f "$FILE" ] || { echo "attest-gate: not found: $FILE" >&2; exit 2; } FAILED=0 @@ -160,6 +170,27 @@ if [ -n "$REF" ] && [ -f "$REF" ]; then [ "$c" -eq 0 ] && [ "$r" -gt 0 ] && printf ' reference is capture-led and this is prose-only — see check 5\n' fi +# 12 — the destination. Every check above tests a property of the text, and text can be +# perfect while landing somewhere nobody will read it. Measured across one register of +# published runs: 22 of 27 comments went onto pull requests that had ALREADY merged when +# they were posted, median 22 days after the merge, one of them 178 days after. The gate +# was clean on every one. A finding delivered to a closed pull request changes nothing, +# and no property of the comment can reveal that. +echo +if [ -z "$TARGET" ]; then + fail "12 destination is open" "no --target given, so nobody checked whether the pull request is still open. Pass --target owner/repo#N." +elif ! command -v gh >/dev/null 2>&1; then + printf ' ???? %s\n %s\n' "12 destination is open" "gh not on PATH — the destination is UNVERIFIED, not passing. Check it by hand before publishing." +else + t_repo="${TARGET%%#*}"; t_num="${TARGET##*#}" + t_state="$(gh api "repos/$t_repo/pulls/$t_num" --jq 'if .merged_at then "merged" else .state end' 2>/dev/null || echo unknown)" + case "$t_state" in + open) pass "12 destination is open" ;; + unknown) fail "12 destination is open" "could not read $TARGET — do not publish to a destination you could not check" ;; + *) fail "12 destination is open" "$TARGET is $t_state. A run published to a closed pull request reaches no reviewer and changes no decision." ;; + esac +fi + echo if [ "$FAILED" -eq 0 ]; then echo "attest-gate: phase 0 clean — proceed to /outframe ‖ /missing ‖ /press" From 0dee42cd2b7006838df9a0c4d7b9b32894d0c1bf Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 17:07:38 -0400 Subject: [PATCH 51/63] Bring the runner fixes back from the branch CI was actually running MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CI workflow sources runners by `skills_ref`, and every run this week pointed at a fork branch. Four fixes were made there and never reached this one: the mutation now travels through `ENVIRON` instead of an escape-processed `awk -v` assignment, the artifact reports the line read back off disk beside the line requested, `--expect-fail` turns a red arm in the wrong place into its own outcome, and `--metric` lets the caller name what a probe counted. So the defect this PR's own description cites as the reason instruments must report their effect was, until now, still live in the instrument this PR ships. The probe comes across too, with the import fix that made it resolve at the destination the workflow copies it to. Two copies of the same scripts on two branches, edited in both directions — `attest-gate.sh` had a check the fork lacked, so it stays as it is here. --- .../probes/metametrics-context.test.tsx | 68 +++++++++++++++++++ .../skills/evidence/scripts/falsify-probe.sh | 51 ++++++++++++-- .../skills/evidence/scripts/render-count.sh | 33 ++++++--- 3 files changed, 136 insertions(+), 16 deletions(-) create mode 100644 domains/pr-workflow/skills/evidence/probes/metametrics-context.test.tsx diff --git a/domains/pr-workflow/skills/evidence/probes/metametrics-context.test.tsx b/domains/pr-workflow/skills/evidence/probes/metametrics-context.test.tsx new file mode 100644 index 00000000..ae61df61 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/probes/metametrics-context.test.tsx @@ -0,0 +1,68 @@ +// Probe — MetaMetrics context value identity. +// +// PLACEMENT: copy to `ui/contexts/__render_probe__.test.tsx` in a metamask-extension tree. +// The imports below are relative to `ui/contexts/`, so a different destination resolves +// nothing and the suite fails to run with "Cannot find module" — which is a failed probe, +// not a measurement. `probe_dest` in the evidence workflow must match this path. +// +// The claim under test is about breadth: "all N consumers avoid unnecessary re-renders". +// `useContext` re-renders a consumer when the value's IDENTITY changes, and that is not a +// per-consumer property — so one distinct value across N parent renders means every +// consumer is spared, and N distinct values means none is. Counting distinct values is +// therefore the measurement the claim actually rests on; counting one consumer's renders +// would only ever describe that consumer. +// +// Resolves against both `metametrics.js` and `metametrics.tsx`, so the same file measures a +// base commit and a head commit that renamed it — the comparison is the point. +import React, { useContext, useRef, useState } from 'react'; +import { act } from '@testing-library/react'; +import configureStore from '../store/store'; +import { renderWithProvider } from '../../test/lib/render-helpers-navigate'; +import mockState from '../../test/data/mock-state.json'; +import { MetaMetricsContext, MetaMetricsProvider } from './metametrics'; + +let consumerRenders = 0; +let distinctValues = 0; +let bump: (() => void) | undefined; + +function Consumer() { + const value = useContext(MetaMetricsContext); + const last = useRef<unknown>(null); + if (last.current !== value) { + last.current = value; + distinctValues += 1; + } + consumerRenders += 1; + return null; +} + +function Parent() { + const [, setN] = useState(0); + bump = () => setN((n) => n + 1); + return ( + <MetaMetricsProvider> + <Consumer /> + </MetaMetricsProvider> + ); +} + +describe('MetaMetrics context value identity', () => { + it('counts distinct context values across parent re-renders', () => { + const PARENT_RENDERS = 5; + consumerRenders = 0; + distinctValues = 0; + + renderWithProvider(<Parent />, configureStore(mockState)); + for (let i = 0; i < PARENT_RENDERS; i++) { + act(() => { + bump?.(); + }); + } + + // eslint-disable-next-line no-console + console.log( + `RENDER_COUNT consumer=${distinctValues} parentRenders=${PARENT_RENDERS + 1} consumerRenders=${consumerRenders}`, + ); + expect(consumerRenders).toBeGreaterThan(0); + }); +}); diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh index d7e8753e..46fed7f0 100755 --- a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -24,6 +24,7 @@ # # Usage: # falsify-probe.sh --test <path> --source <path> --line <n> --replace <text> +# [--expect-fail <test name substring>]... # [--label <slug>] [--out <dir>] [--runner "<cmd>"] # # Example: @@ -50,6 +51,12 @@ RUNNER="yarn jest" OUT_DIR="evidence-artifacts" LABEL="" TEST="" SOURCE="" LINE="" REPLACE="" +# Which test names the caller predicts will fail. Caller-stated, like every other judgement +# word here, and checked rather than trusted: the guards ask whether arm B failed and whether +# it ran the same tests, never whether the RIGHT ones failed. A mutation silently corrupted +# before it reached the file failed a different case than it aimed at, ran the full suite, and +# was reported `falsifying` — a green verdict for a mechanism the run never touched. +EXPECT="" die() { printf 'falsify-probe: %s\n' "$1" >&2; exit 3; } @@ -62,6 +69,7 @@ while [ $# -gt 0 ]; do --label) LABEL="${2:-}"; shift 2 ;; --out) OUT_DIR="${2:-}"; shift 2 ;; --runner) RUNNER="${2:-}"; shift 2 ;; + --expect-fail) EXPECT="$EXPECT${EXPECT:+\n}${2:-}"; shift 2 ;; -h|--help) sed -n '2,32p' "$0"; exit 0 ;; *) die "unknown argument: $1" ;; esac @@ -105,9 +113,21 @@ if [ "$ARM_A" != "passed" ]; then VERDICT="baseline-already-failing"; CODE=2; ARM_B="not-run" : > "$STAMP-armB.log" else - # Mutate exactly one line. `.bak` form keeps this portable across GNU/BSD sed. - awk -v n="$LINE" -v r="$REPLACE" 'NR==n{print r; next}{print}' "$SOURCE" > "$SOURCE.tmp" \ - && mv "$SOURCE.tmp" "$SOURCE" || die "mutation failed" + # Mutate exactly one line. The replacement travels through the environment, not + # through `awk -v`: awk runs escape processing on a `-v` assignment, so `[\s\S]` + # arrived as `[sS]` and the mutation written to the file was not the mutation asked + # for — it narrowed the regex it was meant to widen, failed a different test, and the + # runner reported `falsifying` for a mechanism it never touched. `ENVIRON` does no + # such processing. + MUTANT_LINE="$REPLACE" awk -v n="$LINE" 'NR==n{print ENVIRON["MUTANT_LINE"]; next}{print}' \ + "$SOURCE" > "$SOURCE.tmp" && mv "$SOURCE.tmp" "$SOURCE" || die "mutation failed" + # What the artifact reports as the mutation is read back off disk, never taken from the + # argument. The two differed once and nothing in the output said so. + APPLIED_LINE="$(sed -n "${LINE}p" "$SOURCE")" + if [ "$APPLIED_LINE" != "$REPLACE" ]; then + printf 'falsify-probe: the line written differs from --replace\n asked: %s\n written: %s\n' \ + "$REPLACE" "$APPLIED_LINE" >&2 + fi ARM_B="$(run_arm "$STAMP-armB.log")" restore; trap - EXIT INT TERM A_TOTAL="$(total_tests "$STAMP-armA.log")"; A_TOTAL="${A_TOTAL:-0}" @@ -125,6 +145,23 @@ else fi fi +# Runs last, on the verdict the guards already reached: a mutation can only fail the wrong +# case if it failed something, so this narrows `falsifying` and never widens it. +MISSED="" +if [ "$CODE" -eq 0 ] && [ -n "$EXPECT" ]; then + FAILED_SO_FAR="$(grep -E "^[[:space:]]+.[^\u203a]*\u203a" "$STAMP-armB.log" 2>/dev/null)" + printf '%b\n' "$EXPECT" | while IFS= read -r want; do + [ -n "$want" ] || continue + printf '%s' "$FAILED_SO_FAR" | grep -qF "$want" || printf '%s\n' "$want" + done > "$STAMP.missed" + MISSED="$(tr '\n' '|' < "$STAMP.missed" | sed 's/|$//;s/|/, /g')" + rm -f "$STAMP.missed" + if [ -n "$MISSED" ]; then + VERDICT="falsified a different case — predicted failure absent: $MISSED" + CODE=2 + fi +fi + summarise() { grep -E '^(Tests|Test Suites):' "$1" 2>/dev/null | tr '\n' ' ' | sed 's/ */ /g'; } A_SUM="$(summarise "$STAMP-armA.log")" B_SUM="$(summarise "$STAMP-armB.log")" @@ -137,7 +174,9 @@ cat > "$STAMP.json" <<JSON "test": "$TEST", "mutation": { "source": "$SOURCE", "line": $LINE, "from": $(printf '%s' "$ORIGINAL_LINE" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))'), - "to": $(printf '%s' "$REPLACE" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))') }, + "to": $(printf '%s' "${APPLIED_LINE-$REPLACE}" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))'), + "to_requested": $(printf '%s' "$REPLACE" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))') }, + "predicted_failures_absent": $(printf '%s' "$MISSED" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))'), "armA": { "result": "$ARM_A", "summary": "$A_SUM", "log": "$STAMP-armA.log" }, "armB": { "result": "$ARM_B", "summary": "$B_SUM", "log": "$STAMP-armB.log" }, "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, "node": "$NODE_V", "yarn_lock_sha256_16": "$LOCK_SHA" } @@ -170,8 +209,6 @@ printf 'falsify-probe: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.js # orchestrator reads them and writes ONE open question about THIS diff. printf 'limits: one line of one file was mutated. Says nothing about other paths into the same mechanism, whether it is reachable in production, or whether the guarded behaviour is -correct. This probe proves the suite notices one mutated line, which is not the same as the -base-against-branch proof that a test is connected to the reported bug -- see the red-on-base -skill for that experiment.%s\n' \ +correct. A falsifying test proves the test has power, not that the fix is right.%s\n' \ "$([ "$VERDICT" = vacuous ] && printf '\n vacuous: the mechanism is unguarded by this suite — what else depends on it?')" >&2 exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh index 05de89a1..ab3dcca2 100755 --- a/domains/pr-workflow/skills/evidence/scripts/render-count.sh +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -15,7 +15,7 @@ # # Usage: # render-count.sh --probe <probe.test.tsx> [--defeat <file> --defeat-line <n> --defeat-with <text>] -# [--arm-b <label>] [--label <slug>] [--out <dir>] +# [--arm-b <label>] [--metric <words>] [--label <slug>] [--out <dir>] # # Arm B is "the memo defeated" by default, which is the shape when a PR ADDS # memoisation. When a PR is the one under suspicion the arms invert — arm B applies @@ -29,6 +29,10 @@ # # RENDER_COUNT consumer=<n> parentRenders=<m> # +# `consumer=` is the field name, not a promise about what was counted — a probe for a claim +# about context value identity counts distinct values there, and calling that "consumer +# renders" prints a different quantity than the one measured. Pass --metric to name it. +# # 0 measured counts captured for both arms (or arm A alone if no --defeat) # 1 no delta arm B identical to arm A — the memo is not doing what is claimed # 2 probe did not emit RENDER_COUNT @@ -49,6 +53,13 @@ capture_provenance() { OUT_DIR="evidence-artifacts"; LABEL=""; PROBE=""; DEFEAT=""; DEFEAT_LINE=""; DEFEAT_WITH="" ARM_B="memo defeated" +# What the probe's `consumer=` field counts, in the caller's words. Caller-stated for the +# same reason the verdict is: this script reads a number out of a line the probe printed and +# has no way to know what the probe counted. A probe that counts distinct context values — +# the right measurement when the claim is about value identity rather than one component's +# renders — was published under the fixed heading "consumer renders", which is a different +# quantity and was wrong. A wrong label on a correct number is still a wrong number. +METRIC="consumer renders" die() { printf 'render-count: %s\n' "$1" >&2; exit 3; } while [ $# -gt 0 ]; do @@ -58,6 +69,7 @@ while [ $# -gt 0 ]; do --defeat-line) DEFEAT_LINE="${2:-}"; shift 2 ;; --defeat-with) DEFEAT_WITH="${2:-}"; shift 2 ;; --arm-b) ARM_B="${2:-}"; shift 2 ;; + --metric) METRIC="${2:-}"; shift 2 ;; --label) LABEL="${2:-}"; shift 2 ;; --out) OUT_DIR="${2:-}"; shift 2 ;; -h|--help) sed -n '2,30p' "$0"; exit 0 ;; @@ -85,7 +97,9 @@ if [ -n "$DEFEAT" ] && [ -n "$DEFEAT_LINE" ]; then BACKUP="$(mktemp)"; cp "$DEFEAT" "$BACKUP" restore() { cp "$BACKUP" "$DEFEAT"; rm -f "$BACKUP"; } trap restore EXIT INT TERM - awk -v n="$DEFEAT_LINE" -v r="$DEFEAT_WITH" 'NR==n{print r; next}{print}' "$DEFEAT" > "$DEFEAT.tmp" && mv "$DEFEAT.tmp" "$DEFEAT" + # Through the environment, not `awk -v`: a `-v` assignment is escape-processed, so a + # replacement containing a backslash reaches the file altered. See falsify-probe.sh. + DEFEAT_LINE_TEXT="$DEFEAT_WITH" awk -v n="$DEFEAT_LINE" 'NR==n{print ENVIRON["DEFEAT_LINE_TEXT"]; next}{print}' "$DEFEAT" > "$DEFEAT.tmp" && mv "$DEFEAT.tmp" "$DEFEAT" yarn jest "$PROBE" > "$STAMP-armB.log" 2>&1 B_LINE="$(counts_from "$STAMP-armB.log")" B="$(consumer_of "$B_LINE")" @@ -95,8 +109,8 @@ else fi if [ -n "$B" ] && [ "$B" = "$A" ]; then VERDICT="no delta — arm B changed nothing measurable"; CODE=1 -elif [ -n "$B" ]; then VERDICT="delta measured: $A → $B renders with $ARM_B"; CODE=0 -else VERDICT="baseline only: $A consumer renders"; CODE=0; fi +elif [ -n "$B" ]; then VERDICT="delta measured: $A → $B $METRIC with $ARM_B"; CODE=0 +else VERDICT="baseline only: $A $METRIC"; CODE=0; fi HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" DIRTY="$(git status --porcelain 2>/dev/null | grep -v '^??' | wc -l | tr -d ' ')" @@ -104,17 +118,18 @@ NODE_V="$(node -v 2>/dev/null || echo unknown)" cat > "$STAMP.json" <<JSON { "probe": "$PROBE", "verdict": "$VERDICT", "exit": $CODE, - "consumer_renders": { "armA": ${A:-null}, "armB": ${B:-null} }, + "metric": "$METRIC", + "counts": { "armA": ${A:-null}, "armB": ${B:-null} }, "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, "node": "$NODE_V" }, "logs": ["$STAMP-armA.log", "$STAMP-armB.log"] } JSON { - echo "### Consumer render count" + echo "### Render probe — $METRIC" echo echo "**Verdict:** $VERDICT" echo - echo "| Arm | Change | consumer renders |" + echo "| Arm | Change | $METRIC |" echo "|---|---|---|" echo "| A — as committed | none | ${A:-?} |" [ -n "$B" ] && echo "| B — $ARM_B | \`$DEFEAT:$DEFEAT_LINE\` | $B |" @@ -125,8 +140,8 @@ JSON [ -n "$B_LINE" ] && { echo "\$ yarn jest $PROBE # $ARM_B"; echo "$B_LINE"; } echo '```' echo - echo "This counts renders of one named consumer across a defined interaction. It is not a count" - echo "of consumers, and a larger consumer count does not imply a larger effect." + echo "The number is \`$METRIC\` as printed by \`$PROBE\` — that file is what defines the" + echo "quantity. It is one probe under one interaction, not a property of the application." echo echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. $(capture_provenance)</sub>" } > "$STAMP.md" From cae0c082c0f61ca44d5825f1121c9b5259b52572 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 02:58:46 -0400 Subject: [PATCH 52/63] Check that a figure in the prose traces to an exhibit Check 9 is called "verdict matches artifact" and compares verdict words. Nothing compared the numbers, and prose drifting from the exhibit beside it is the most common way one of these goes wrong. Found by building a demonstration artifact to test this gate: the prose read "0 errors over 48 skills" directly above an exhibit reading "47 skill(s) checked", and named a warning class with zero instances in the output it was describing. Every other check passed. Two independent readers caught both, which is the argument for moving it into the layer that always runs rather than the one that costs money and sometimes never reports. Narrow on purpose, because a noisy check is an ignored one. Two-plus digits only, and only those absent from every fenced block; whole URLs, issue refs, versions, dates, SHAs, file:line citations, hyphenated identifiers and regex quantifiers are excluded as references rather than measurements. Each exclusion was added after a control run flagged something that was not a figure. Across eight real artifacts it flags one, correctly: a verdict line quoting an author's "730 tests" beside an exhibit measuring 731, where nothing distinguishes the cited figure from the measured one. --- .../skills/evidence/scripts/attest-gate.sh | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 3f058d5f..fd7d8935 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -191,6 +191,34 @@ else esac fi +# 13 — a number in the prose that appears in no exhibit. Check 9 compares verdict WORDS; +# nothing compared the figures. Measured on a demonstration artifact built to test this +# gate: the prose said "0 errors over 48 skills" directly above an exhibit reading +# "47 skill(s) checked", and named a warning class with zero instances in the output it +# was describing. Both survived every other check. Prose drifts from the exhibit it sits +# beside, and it is the most common way one of these goes wrong. +# +# Deliberately narrow, because a noisy check is an ignored check: integers of two or more +# digits only, and only those absent from every fenced block. Excluded as references +# rather than measurements — whole URLs, issue refs, version strings, dates, SHAs, +# file:line citations, hyphenated identifiers like P-256, and regex quantifiers. Every +# one of those was added after a control run flagged something that was not a figure. +echo +NUM_ORPHANS="$( + awk '/^```/{f=!f; next} f{print}' "$FILE" > "$FILE.exh" 2>/dev/null + awk '/^```/{f=!f; next} !f{print}' "$FILE" \ + | sed -E 's#https?://[^ )]*##g' \ + | sed -E 's/#[0-9]+//g; s/\bv?[0-9]+\.[0-9]+(\.[0-9]+)?\b//g; s/\b[0-9]{4}-[0-9]{2}-[0-9]{2}\b//g; s/\b[0-9a-f]{7,}\b//g; s/:[0-9]+\b//g; s/[A-Za-z]+-[0-9]+//g; s/\{[0-9,]+\}//g' \ + | grep -oE '\b[0-9]{2,}\b' | sort -u \ + | while read -r n; do grep -qF "$n" "$FILE.exh" || printf '%s ' "$n"; done + rm -f "$FILE.exh" +)" +if [ -n "$NUM_ORPHANS" ]; then + fail "13 figures trace to an exhibit" "these appear in the prose and in no exhibit: $NUM_ORPHANS — either they came from somewhere the reader cannot see, or they disagree with what is shown" +else + pass "13 figures trace to an exhibit" +fi + echo if [ "$FAILED" -eq 0 ]; then echo "attest-gate: phase 0 clean — proceed to /outframe ‖ /missing ‖ /press" From a5bfb9872065529a6399cc2838cc32f1c363e71d Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 03:15:29 -0400 Subject: [PATCH 53/63] Stop check 12 passing when it could not run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With `gh` absent the check printed UNVERIFIED and exited 0, so on a machine without it — running locally — the destination check announced that it had not run and the gate reported clean. A control that cannot run is indistinguishable from one that passed unless the exit code says otherwise. --- domains/pr-workflow/skills/evidence/scripts/attest-gate.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index fd7d8935..5fbed0ef 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -180,7 +180,10 @@ echo if [ -z "$TARGET" ]; then fail "12 destination is open" "no --target given, so nobody checked whether the pull request is still open. Pass --target owner/repo#N." elif ! command -v gh >/dev/null 2>&1; then - printf ' ???? %s\n %s\n' "12 destination is open" "gh not on PATH — the destination is UNVERIFIED, not passing. Check it by hand before publishing." + # Blocks rather than warns. An earlier version printed UNVERIFIED and exited 0, so on a + # machine without `gh` — which is to say, running locally — this check announced that it + # had not run and passed anyway. That is the shape it exists to catch, one level up. + fail "12 destination is open" "gh not on PATH, so the destination was not checked. Unverified is not passing: install gh, or confirm the target is open and re-run." else t_repo="${TARGET%%#*}"; t_num="${TARGET##*#}" t_state="$(gh api "repos/$t_repo/pulls/$t_num" --jq 'if .merged_at then "merged" else .state end' 2>/dev/null || echo unknown)" From 7da6ee12d876698dc1b499825bf7378de330c075 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 05:40:02 -0400 Subject: [PATCH 54/63] Point references at the renamed `lavamoat-policy` skill Renamed on the security-domain branch; installs as `mms-lavamoat-policy`. --- .../skills/evidence/references/evidence-catalog.md | 2 +- .../skills/evidence/references/evidence-publishing.md | 2 +- domains/pr-workflow/skills/evidence/skill.md | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md index d7f82692..103ab9b3 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md @@ -166,7 +166,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **Proves:** a module moved to the intended (lazy) chunk and no longer ships on the critical path. Requires the webpack build. Mirrors AEP `perf-chunks`. ## D3. LavaMoat policy / supply-chain capability diff - - **Engine: the `supply-chain-audit` skill** (umbrella — lockfile/manifest diff, advisories, Socket Security, install scripts), which delegates capability grants to **`lavamoat-policy-diligence`** (per-grant call-site justification). Delegate the dependency change to the umbrella; it returns a disposition per lane. evidence keeps **supply-chain capability diff** as the evidence category and packages the output. Note the lanes are independent: a clean policy diff does not mean a safe dependency, and a known CVE never appears as a new grant. + - **Engine: the `supply-chain-audit` skill** (umbrella — lockfile/manifest diff, advisories, Socket Security, install scripts), which delegates capability grants to **`lavamoat-policy`** (per-grant call-site justification). Delegate the dependency change to the umbrella; it returns a disposition per lane. evidence keeps **supply-chain capability diff** as the evidence category and packages the output. Note the lanes are independent: a clean policy diff does not mean a safe dependency, and a known CVE never appears as a new grant. - **Proves:** a dependency change (bump/add/lockfile) grants **no *unjustified* new capability** — the supply-chain-risk lane. Note the bar: for a bump the policy *will* change, so "empty diff" is the WRONG test; the right test is **every new grant is justified by the dep's function**.. - **Capture:** **Prefer the CI-generated policy whenever one is available.** `@metamaskbot update-policies` regenerates the policy files from a real run of the code and `validate-lavamoat-policies` fails the build on drift, so the committed policy on a bot-run PR *is* the authoritative artifact — diff that. Regenerating locally when a current CI policy exists only re-does a machine that is already trusted, and a local run's provenance is weaker (your node/OS/lockfile resolution, not CI's). **Local regen is the fallback**, for when the bot hasn't run yet, the branch is unpushed, or you need a variant CI didn't cover: `yarn webpack:lavamoat:policy:build` (`:mv2` / `:mv3` for variants) over `lavamoat/webpack/build/policy.json` (+ `policy-override.json`). Either way, `git diff` the policy across **all 8 variants** (mv{2,3}/{main,beta,flask,experimental}) — a grant can appear in one and not others. Then audit **grant-by-grant**: new **globals** (`fetch`, `importScripts`, `WebAssembly`) / **builtins** (`fs`, `child_process`) on a dep that shouldn't need them, new **packages** edges to powerful APIs, or an identifier substitution (`pkgC>name` replacing `pkgB>pkgA>name` = possible dep swap). Falsifier = a surprising grant ("I wonder what it's using this for"). Guide: lavamoat.github.io/guides/policy-diff/. `allowScripts` in `package.json` gates install scripts. diff --git a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md index 4316c52f..72ce8da4 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md @@ -287,7 +287,7 @@ needs a tracker it does not have. The marker pairs also collide — a re-run rep **So: choose the format from the evidence kind, not from this document's default.** The canonical `## 🧪 Validation Run` header applies when a run produced artifacts. An engine -skill that defines its own output contract (`lavamoat-policy-diligence`) publishes in that +skill that defines its own output contract (`lavamoat-policy`) publishes in that contract. `hooks/pr-evidence-gate.py` enforces the canonical literal only on bodies that *claim* validation/evidence framing — a diligence comment that renders no verdict does not trip it, which is the tell that the two are different artifacts rather than one with a diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 9ac32af0..4f3f0b77 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -446,7 +446,7 @@ the text, so the gate asks for a different medium rather than for better text. **The exception — plaintext where every claim is a citation.** The rule is about where verification routes, not about pixels. Line-level links are externally verifiable: the reader clicks and sees exactly what you saw. That is the *normal* case for the audit lanes — -`supply-chain-audit`, `lavamoat-policy-diligence`, `privacy-egress-diligence` — whose findings +`supply-chain-audit`, `lavamoat-policy`, `privacy-egress-diligence` — whose findings are facts about code that exists rather than results of running something. There an image would be worse: a screenshot of a policy diff is less checkable than a permalink to it. @@ -660,9 +660,9 @@ Where evidence sits in the PR lifecycle (see the public `pr-workflow` siblings): | B7 deterministic interleaving (concurrency / ordering) | `/race-condition-repro` | | C4 React render & selector proof | `/react-render-delta` | | C9 memory leak | `/memory-leak` | - | D supply-chain / dependency change | `/supply-chain-audit` → delegates capability grants to `/lavamoat-policy-diligence` | + | D supply-chain / dependency change | `/supply-chain-audit` → delegates capability grants to `/lavamoat-policy` | - **An engine that defines its own output contract publishes in it.** `lavamoat-policy-diligence` + **An engine that defines its own output contract publishes in it.** `lavamoat-policy` is the live case: read-level triage, no verdict, its own header and marker pair. Do not re-frame it as a Validation Run — see *One comment per evidence kind* in [references/evidence-publishing.md](references/evidence-publishing.md). From 38316eee4d3ab28ba8e749fc1381546296708179 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 05:50:22 -0400 Subject: [PATCH 55/63] Give the diligence format a gate with `attest-gate.sh --diligence` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A diligence comment renders no verdict and deliberately does not use the Validation Run envelope. That exemption meant it was checked by nothing: this gate only knew the Validation Run shape, and `pr-evidence-gate.py` by design does not trip on a body claiming no verdict. So every rule the diligence skills state about their own output — including "runtime claims need a runtime artifact" — had no execution path. It showed. A lavamoat comment shipped with no marker pair, an `npm pack` specifier set no reader could fetch, and two bare integers traceable to nothing. `--diligence` swaps the four envelope checks for that contract's own — its marker pair, its header, permalinks pinned to a tag or SHA rather than a branch head, and a runtime claim check asking for the thing a `/blob/` link cannot witness. 3, 8 and 9 report SKIP with the reason rather than passing silently, since a check that cannot fail should not read as a check that passed. Everything downstream of the envelope is shared, because those defects are shared. Run against the comment that prompted this, it fails 1, 5 and 13 and passes the rest. --- .../skills/evidence/scripts/attest-gate.sh | 57 ++++++++++++++++++- 1 file changed, 54 insertions(+), 3 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 5fbed0ef..61d3728a 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -16,16 +16,17 @@ # 2 usage error set -uo pipefail -FILE="${1:-}"; REF=""; TARGET="" +FILE="${1:-}"; REF=""; TARGET=""; MODE="run" shift || true while [ $# -gt 0 ]; do case "$1" in --reference) REF="${2:-}"; shift 2 ;; --target) TARGET="${2:-}"; shift 2 ;; + --diligence) MODE="diligence"; shift ;; *) shift ;; esac done -[ -n "$FILE" ] || { echo "usage: attest-gate.sh <artifact.md> [--reference <file>] [--target <owner/repo#N>]" >&2; exit 2; } +[ -n "$FILE" ] || { echo "usage: attest-gate.sh <artifact.md> [--reference <file>] [--target <owner/repo#N>] [--diligence]" >&2; exit 2; } [ -f "$FILE" ] || { echo "attest-gate: not found: $FILE" >&2; exit 2; } FAILED=0 @@ -39,6 +40,25 @@ hasi() { grep -qiE "$1" "$FILE"; } # case-insensitive; a separate function bec echo "attest-gate: $FILE" echo +# A diligence comment (lavamoat-policy and its siblings) renders no verdict and deliberately +# does not use the Validation Run envelope — see "One comment per evidence kind" in +# references/evidence-publishing.md. That exemption used to mean it was checked by nothing at +# all: attest-gate only knew the Validation Run shape, and pr-evidence-gate.py by design does +# not trip on a body claiming no verdict. So every rule the diligence skills state about their +# own output had no execution path, and a comment shipped with an unwitnessed local `npm pack` +# result and untraceable integers. --diligence swaps the envelope checks for that contract's +# own; everything downstream of the envelope is shared, because those defects are shared. +if [ "$MODE" = diligence ]; then + has 'LAVAMOAT_DILIGENCE_START' && has 'LAVAMOAT_DILIGENCE_END' \ + && pass "1 marker pair" \ + || fail "1 marker pair" "no LAVAMOAT_DILIGENCE_START/_END — a re-run appends a duplicate instead of replacing, and the pair must not be VALIDATION_RUN_* or an evidence re-run would eat this region" + + hasre '^\*\*LavaMoat grants|^LavaMoat grants' \ + && pass "2 canonical header" \ + || fail "2 canonical header" "missing the 'LavaMoat grants — <package> <old> -> <new>' opener" + + printf ' SKIP %s\n' "3 verdict line — a diligence comment renders none, by contract" +else has 'VALIDATION_RUN_START' && has 'VALIDATION_RUN_END' \ && pass "1 marker pair" \ || fail "1 marker pair" "no VALIDATION_RUN_START/_END — a re-run appends a duplicate instead of replacing" @@ -50,14 +70,27 @@ has '## 🧪 Validation Run' \ hasre '^\*\*Verdict:\*\*.*\*\*Claim:\*\*' \ && pass "3 verdict line" \ || fail "3 verdict line" "no '**Verdict:** … — **Claim:** …' — valence is not legible at a glance" +fi # A run outside the repo's toolchain pins a different thing. A browser-memory lane # names "Firefox 153.0"; a repo lane names a head SHA and a lockfile hash. Both are # pins, and a check that only knows the second one fails every run of the first — # telling an author their pinned environment is unpinned. +# A diligence comment pins a read, not a run: its citations are permalinks, and the thing +# that can rot is a branch-head link drifting out from under the line it names. +if [ "$MODE" = diligence ]; then + if grep -qE 'https://github\.com/[^ )]+/blob/(main|master|develop|HEAD)/' "$FILE"; then + fail "4 citations pinned" "a permalink points at a branch head; it will drift off the line it cites. Pin a tag or a 40-char SHA" + elif grep -qE 'https://github\.com/[^ )]+/blob/[^/]+/' "$FILE"; then + pass "4 citations pinned" + else + fail "4 citations pinned" "no source permalink at all — the permalink IS the evidence here; a retyped 'it needs X' proves nothing about provenance" + fi +else hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `|[Ff]irefox [0-9]+\.[0-9]|[Cc]hrom(e|ium) [0-9]+\.|[Ss]afari [0-9]+\.|[Nn]ode v?[0-9]+\.[0-9]' \ && pass "4 environment pinned" \ || fail "4 environment pinned" "no head SHA, lockfile hash, or pinned toolchain/browser version" +fi # 5 — the one that matters, and it asks for a MEDIUM, not for better text. # @@ -77,7 +110,20 @@ hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `|[Ff]irefox [0-9]+\.[0-9]|[ # tell: it looks reproducible and cannot be run. # An image, a re-executing link, or a hosted artifact — verification that does not route # through the author. `Produced by` and `evidence-artifacts/` are provenance, not this. -if ! hasre '!\[[^]]*\]\(https?://|<img [^>]*src="https?://|actions/runs/[0-9]|/gist\.|https?://[^ )]+\.(png|jpg|jpeg|gif|svg|txt|log|json)\b'; then +# In diligence mode the medium is the permalink, already required by check 4 — a reader +# clicks it and lands on the line. What a permalink cannot witness is what the AUTHOR RAN, +# and that is the defect this variant catches: an `npm pack` unpacked locally, a grep over a +# tarball, a byte-comparison across policy files. Those read as properties of the package +# and are actually properties of an unwitnessed local run. State them as the search +# ("searched N files, no match") or publish the output; do not assert them as fact. +if [ "$MODE" = diligence ]; then + if hasre "(complete|full) (specifier|import|require) set|byte-identical|identical across all|^Searched: .*tarball|npm pack" \ + && ! hasre 'actions/runs/[0-9]|/gist\.|https?://[^ )]+\.(txt|log|json)\b'; then + fail "5 runtime claims witnessed" "asserts a result only a local run could produce ($(grep -m1 -oiE '(complete|full) (specifier|import|require) set|byte-identical|identical across all|npm pack' "$FILE")) with nothing a reader can fetch. A /blob/ permalink witnesses a line, not your shell" + else + pass "5 runtime claims witnessed" + fi +elif ! hasre '!\[[^]]*\]\(https?://|<img [^>]*src="https?://|actions/runs/[0-9]|/gist\.|https?://[^ )]+\.(png|jpg|jpeg|gif|svg|txt|log|json)\b'; then fail "5 captured artifact" "no reader-verifiable capture — an image of the tool surface, a run link, or a hosted artifact. A fenced block is the author\'s transcription, whatever produced it" # No separate attribution test: a hosted artifact the reader fetches is its own # attribution, and requiring `Produced by` on top of it only fails runs whose @@ -114,6 +160,10 @@ else pass "7 no process narration" fi +if [ "$MODE" = diligence ]; then + printf ' SKIP %s\n' "8 verdict is earned — no verdict rendered" + printf ' SKIP %s\n' "9 verdict matches artifact — no verdict rendered" +else if hasi '\*\*Verdict:\*\*.*proven' && ! hasre 'Produced by |actions/runs|evidence-artifacts/'; then fail "8 verdict is earned" "claims 'proven' with no execution artifact — reading yields 'unverified'" else @@ -130,6 +180,7 @@ if printf '%s' "$HDR" | grep -q 'proven' && [ -n "$BODY" ]; then else pass "9 verdict matches artifact" fi +fi # 10 — the positive counterpart to check 6. A run succeeds by putting concerns in front # of a reviewer, so an artifact that floats nothing has reported only what it happened to From 76ecd0a9c0268f0aa1d731c7783688796f303d75 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 06:02:59 -0400 Subject: [PATCH 56/63] Remove private-repo and personal references from a public skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This repository is public. Naming a private repository here discloses its existence, its owner and roughly its contents to every reader — and a prohibition naming it ("do not re-host to X, it is private") discloses exactly as much as a recommendation would. Four such references were doing that, and the guidance survives without them: the rule is audience-reachability, which is stated directly rather than by example. Two memory-file citations offered as "source of truth" pointed into a private repo, so a reader was told to follow a rule whose justification they cannot open. The reasoning is inlined; the pointer is gone. The publish-surface snippet hardcoded a GitHub username, which decided the destination for whoever ran it. Now derived from `gh api user --jq .login`, and the surrounding prose is second-person rather than first — a shared skill has no "my PRs". `/attest` is no longer linked to a personal repository. That leaves it named but not resolvable, which is honest and is the smaller problem; the workflow depending on a command nobody else has is tracked separately. --- .../references/evidence-publishing.md | 34 +++++++++---------- .../evidence/references/lane-assertions.md | 2 +- domains/pr-workflow/skills/evidence/skill.md | 4 +-- 3 files changed, 20 insertions(+), 20 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md index 72ce8da4..65b88b4d 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md @@ -20,14 +20,15 @@ https://majorlift-artifacts-share.s3.us-west-1.amazonaws.com/public/metamask/pr- Anonymous `GetObject` is allowed under `public/*`; bucket listing is not, so the prefix is not browsable — link individual files, and don't promise readers an index. -**Do NOT re-host to `MajorLift/metamask-extension-skills`.** It is a **personal private** repo: -every raw link to it returns 404 for every reader but its owner. That was the previous target -here, and this file simultaneously said links to it were unreachable — guidance that instructed -you to publish dead links. Verified live in a published artifact. +**Do not re-host to a personal repo.** A personal private repo returns 404 for every reader but +its owner, so every artifact link published from one is dead on arrival. That was the previous +target here, and this file simultaneously said such links were unreachable — guidance that +instructed you to publish dead links. Verified live in a published artifact. -The test is **audience-reachability, not public-vs-private.** A `MetaMask/*` org repo is private -but readable by colleagues, so an internal-audience link to one is fine. A `MajorLift/*` personal -repo is unreachable by colleagues *and* by the public, so it fails for every audience. +The test is **audience-reachability, not public-vs-private.** An org repo may be private and still +readable by colleagues, so an internal-audience link to one is fine. A personal repo is unreachable +by colleagues *and* by the public, so it fails for every audience. Re-host to an org-owned +destination, or to the bucket above. - Path convention: `pr-<n>/<run-id>/<artifact-name>` keeps runs from colliding. - **Verify unauthenticated before shipping**: `curl -s -o /dev/null -w "%{http_code}"` on each @@ -116,20 +117,20 @@ Screenshots block (injected into `### After`, or appended under `### Screenshots ## Step 3 — Choose the surface by ownership, then publish -**Publish surface depends on my relationship to the PR** (see exogram -`evidence-publish-surface-by-ownership`). Determine it FIRST: +**Publish surface depends on your relationship to the PR.** Determine it FIRST: ```bash PR=<n>; REPO=MetaMask/metamask-extension -SURFACE=$(gh pr view "$PR" --repo "$REPO" --json author,commits --jq ' - if .author.login=="MajorLift" then "body" - elif ([.commits[] | select(.authors[].login=="MajorLift") - | select([.authors[].login] | map(select(.!="MajorLift" and .!="Copilot" and (test("claude|anthropic")|not))) | length == 0)] | length) > 0 +ME=$(gh api user --jq .login) +SURFACE=$(gh pr view "$PR" --repo "$REPO" --json author,commits --jq --arg me "$ME" ' + if .author.login==$me then "body" + elif ([.commits[] | select(.authors[].login==$me) + | select([.authors[].login] | map(select(.!=$me and .!="Copilot" and (test("claude|anthropic")|not))) | length == 0)] | length) > 0 then "comment" else "skip" end') ``` -- `body` — I authored the PR → upsert into the PR body (below). Validation is - part of my own claim. +- `body` — you authored the PR → upsert into the PR body (below). Validation is + part of your own claim. - `comment` — not author but I have a solo commit (no HUMAN co-author) → post a `gh pr comment` under the canonical `## 🧪 Validation Run` header. Never edit someone else's PR body. @@ -231,7 +232,6 @@ The common loop — a run refutes a claim, the author pushes a fix, `/evidence` - New head → **new hosted artifact directory keyed to the fix commit** (`pr-<n>/fix-<sha>/`), commit-pinned raw URLs; never overwrite a prior run's published files. - Residuals the fix intentionally leaves get their own row/section — don't round a fixed-with-residual claim up to fully proven. -Source of truth: `exogram-core/memory/evidence-revalidation-delta-reports.md`. ## Lead with a lane-status ledger (no silent absence) @@ -293,7 +293,7 @@ contract. `hooks/pr-evidence-gate.py` enforces the canonical literal only on bod trip it, which is the tell that the two are different artifacts rather than one with a different skin. -**Per-scenario presentation (2026-07-21):** the same applies one level down — when the evidence spans multiple test scenarios (flag-on vs flag-off, control vs treatment in an A/B falsifier, numbered manual-testing steps), give each scenario its **own sub-section**: a heading naming the scenario in observation terms, one line on what it tests plus its verdict, and that scenario's artifacts co-located under it. Never bunch all scenarios' artifacts into one large evidence dump — the reviewer verifies "under condition X, artifact shows Y" one condition at a time, and a merged block destroys that mapping even when every artifact is real. For long artifact sets use a `<details>` block *per scenario*, not a merge. (Preference: exogram-core `memory/evidence-present-scenarios-separately.md`; instance #44610.) +**Per-scenario presentation (2026-07-21):** the same applies one level down — when the evidence spans multiple test scenarios (flag-on vs flag-off, control vs treatment in an A/B falsifier, numbered manual-testing steps), give each scenario its **own sub-section**: a heading naming the scenario in observation terms, one line on what it tests plus its verdict, and that scenario's artifacts co-located under it. Never bunch all scenarios' artifacts into one large evidence dump — the reviewer verifies "under condition X, artifact shows Y" one condition at a time, and a merged block destroys that mapping even when every artifact is real. For long artifact sets use a `<details>` block *per scenario*, not a merge. (Instance: #44610.) ## Artifact contract (ADR-0058 alignment) diff --git a/domains/pr-workflow/skills/evidence/references/lane-assertions.md b/domains/pr-workflow/skills/evidence/references/lane-assertions.md index 89622327..7ba6dca7 100644 --- a/domains/pr-workflow/skills/evidence/references/lane-assertions.md +++ b/domains/pr-workflow/skills/evidence/references/lane-assertions.md @@ -23,4 +23,4 @@ Maps each evidence-catalog lane to a declarative assertion form, so a Claim Card | F7 i18n | static: `verify-locales` exit 0 | out-of-band | | F8 runtime containment | `Object.isFrozen(Object.prototype)`; scuttled global throws + exception resolves; `typeof SNOW` | **yes** — `Runtime.evaluate`, but only against the SHIPPED build variant (dev is unscuttled, test's exception list is wider) | -**Takeaway.** UI-state and runtime-metric lanes (A/B-visual, C1–C3/C6, F3/F5) map cleanly to CDP recipe assertions — ADR-0058's sweet spot. Static (D, F7), test-runner (B3, C5, F1), and dashboard (E) lanes are **out-of-band**: the recipe should *reference* them as proof targets without executing them. That out-of-band reference is precisely the **non-UI scaling gap** MajorLift's review of #173 flagged — a recipe schema that admits out-of-band assertion references (not only CDP actions) closes it. This table is the proposed taxonomy to contribute back (ITERATION item 10). +**Takeaway.** UI-state and runtime-metric lanes (A/B-visual, C1–C3/C6, F3/F5) map cleanly to CDP recipe assertions — ADR-0058's sweet spot. Static (D, F7), test-runner (B3, C5, F1), and dashboard (E) lanes are **out-of-band**: the recipe should *reference* them as proof targets without executing them. That out-of-band reference is precisely the **non-UI scaling gap** flagged in review of decisions#173 — a recipe schema that admits out-of-band assertion references (not only CDP actions) closes it. This table is the proposed taxonomy to contribute back (ITERATION item 10). diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 4f3f0b77..b879f483 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -326,7 +326,7 @@ Check 5 is the one that matters and the easiest to slip past: if every character is one the operator typed, the run published an assertion. Pass `--reference <showcase>` to compare capture density against a known-good artifact. -This is phase 0 of [`/attest`](https://github.com/MajorLift/Reprise); phases 1 and 2 dispatch +This is phase 0 of `/attest`; phases 1 and 2 dispatch `/outframe ‖ /missing ‖ /press` then `/trim` to fresh instances, because those passes cannot be self-run — the author is positionally the wrong reader. @@ -619,7 +619,7 @@ Three adjacent things; keep the boundary clear so they compose instead of collid - **AEP** — governed *fleet orchestration*: sandboxes, Temporal, autonomous runs at scale. The heavy engine. - **ADR-0058 recipes** ([decisions#173](https://github.com/MetaMask/decisions/pull/173)) — a *dev-machine inner-loop* proof artifact: a declarative per-PR recipe run against the live app over CDP, emitting `summary.json`/`trace.json`/manifest. -- **evidence** (this skill) — the *claim→evidence methodology + taxonomy* both draw on. The Claim Card is the bridge from a PR's claim to the right proof target; the [evidence catalog](references/evidence-catalog.md) is the lane vocabulary; [lane-assertions.md](references/lane-assertions.md) maps each lane to a recipe assertion (and flags the out-of-band, non-UI lanes — the gap MajorLift's #173 review raised). +- **evidence** (this skill) — the *claim→evidence methodology + taxonomy* both draw on. The Claim Card is the bridge from a PR's claim to the right proof target; the [evidence catalog](references/evidence-catalog.md) is the lane vocabulary; [lane-assertions.md](references/lane-assertions.md) maps each lane to a recipe assertion (and flags the out-of-band, non-UI lanes — the gap raised in review of decisions#173). evidence is the one a human drives; it can dispatch an AEP run or author a recipe as its capture step. From fbf89095488cdcd73708ff65633303d7c91e4ba3 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 06:13:39 -0400 Subject: [PATCH 57/63] Take the artifact bucket and test fork out of the published text MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The re-hosting section named a personal S3 bucket, its region, its prefix layout, and stated that anonymous `GetObject` is enabled under `public/*`. That is a live unauthenticated endpoint advertised, with its structure, to every reader of a public repository — a larger disclosure than the repository names removed alongside it, and one that reads as configuration rather than as a secret, which is why it survived two passes over this file. Now `EVIDENCE_BUCKET` and `EVIDENCE_REGION` from the environment. The requirements the bucket must satisfy — anonymous GetObject under `public/*`, listing disabled — are stated, because those are the load-bearing part; the name never was. The G5 lane likewise named a private test fork, which carried both the org and a personal handle. Now "your own test fork". --- .../evidence/references/evidence-catalog.md | 2 +- .../references/evidence-publishing.md | 20 ++++++++++--------- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md index 103ab9b3..adca4077 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md @@ -267,7 +267,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **G2. Coverage delta** — `yarn test:unit:coverage` → `coverage/unit/` (and `yarn test:unit:webpack:coverage`); `codecov.yml`. Proves the new code is exercised. - **G3. Automated-reviewer output** — independent bot (e.g. cursor[bot]) found nothing blocking. Complements, never replaces, behavior evidence. - **G4. Manual reproduction steps** — human-followable steps that reproduce the fixed behavior; populates the PR template's Manual testing steps. -- **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-<X>=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: push to a branch literally named **`main`** (or `stable`) on `consensys-test/metamask-extension-test-majorlift` — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets on the fork (`INFURA_PROJECT_ID`, `TEST_SRP_*` for benchmarks; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo.. +- **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-<X>=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: push to a branch literally named **`main`** (or `stable`) on your own test fork of the repo — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets on the fork (`INFURA_PROJECT_ID`, `TEST_SRP_*` for benchmarks; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo.. - **G6. CI job-duration delta** — compare job wall-clock across arms in the Actions UI or `gh run view`. **Falsifier: build reuse.** `get-requirements.yml` skips jobs when build output matches base, so a measured "speedup" is often a skipped job — confirm each arm actually ran the work before comparing. Runner class and queue time vary independently of the change; report job time, not wall-clock from push. Pairs with `D7`, which measures the same change on a machine you control. --- diff --git a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md index 65b88b4d..2b452e7d 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md @@ -10,15 +10,17 @@ Canonical source for the format: `~/Code/metamask/metamask-autonomous-engineerin Control-plane artifact URLs (`localhost:3000/v1/runs/:id/artifacts/:name`) won't render on GitHub. Re-host each artifact and link the hosted URL. -**Host: the S3 bucket `majorlift-artifacts-share`, prefix `public/`.** +**Host: an S3 bucket you configure, prefix `public/`.** Set `EVIDENCE_BUCKET` and +`EVIDENCE_REGION` in your environment; this file does not name a bucket, because a bucket name +published here is an anonymously-readable endpoint advertised to everyone who reads it. ``` -s3://majorlift-artifacts-share/public/metamask/pr-<n>/<run-id>/<artifact-name> -https://majorlift-artifacts-share.s3.us-west-1.amazonaws.com/public/metamask/pr-<n>/<run-id>/<artifact-name> +s3://$EVIDENCE_BUCKET/public/metamask/pr-<n>/<run-id>/<artifact-name> +https://$EVIDENCE_BUCKET.s3.$EVIDENCE_REGION.amazonaws.com/public/metamask/pr-<n>/<run-id>/<artifact-name> ``` -Anonymous `GetObject` is allowed under `public/*`; bucket listing is not, so the prefix is not -browsable — link individual files, and don't promise readers an index. +The bucket must allow anonymous `GetObject` under `public/*` and must **not** allow listing, so +the prefix is not browsable — link individual files, and don't promise readers an index. **Do not re-host to a personal repo.** A personal private repo returns 404 for every reader but its owner, so every artifact link published from one is dead on arrival. That was the previous @@ -28,7 +30,7 @@ instructed you to publish dead links. Verified live in a published artifact. The test is **audience-reachability, not public-vs-private.** An org repo may be private and still readable by colleagues, so an internal-audience link to one is fine. A personal repo is unreachable by colleagues *and* by the public, so it fails for every audience. Re-host to an org-owned -destination, or to the bucket above. +destination, or to the configured bucket. - Path convention: `pr-<n>/<run-id>/<artifact-name>` keeps runs from colliding. - **Verify unauthenticated before shipping**: `curl -s -o /dev/null -w "%{http_code}"` on each @@ -36,8 +38,8 @@ destination, or to the bucket above. ```bash RUN_ID=<id>; PR=<n>; CP=localhost:3000 -BUCKET=majorlift-artifacts-share -BASE="https://$BUCKET.s3.us-west-1.amazonaws.com" +BUCKET="$EVIDENCE_BUCKET" +BASE="https://$BUCKET.s3.$EVIDENCE_REGION.amazonaws.com" for name in <artifactName1> <artifactName2>; do curl -fsS "$CP/v1/runs/$RUN_ID/artifacts/$name" -o "/tmp/$name" key="public/metamask/pr-$PR/$RUN_ID/$name" @@ -306,7 +308,7 @@ To stay interoperable with the recipe-based verification system (MetaMask/decisi - [ ] Each lane passed the [trustworthiness gate](evidence-trustworthiness.md) (shows the claimed surface, signal > noise, could-have-failed) - [ ] Multi-scenario evidence rendered **per scenario** (own heading + verdict + co-located artifacts), not bunched into one block - [ ] **Automated-process voice, no first person** — published validation output never says "I ran/captured/verified"; attribute to the process ("Automated validation ran…", "the harness captured…") so readers know the evidence is machine-generated, not a manual account under the author's name -- [ ] Every image/GIF re-hosted to `majorlift-artifacts-share/public/…`; no localhost/local-path URLs in the body +- [ ] Every image/GIF re-hosted to the configured bucket under `public/…`; no localhost/local-path URLs in the body - [ ] **Every published link curl'd unauthenticated and returning 200** — never a personal private repo - [ ] Work cited by **PR link** rather than tracking-ticket id, unless the ticket's own content (an RCA, a spec) is the referent - [ ] Narrative scrubbed of username/paths/internal hosts From 663420347b866075de35be2a3e43fe7cb01ad063 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 06:25:30 -0400 Subject: [PATCH 58/63] Restore what the privacy scrub broke: a working jq filter and bucket setup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removing the hardcoded username left `gh pr view --jq --arg me "$ME"`, which is not a thing gh supports — its built-in filter takes no --arg and the command dies with "accepts at most 1 arg(s)". Piped to real jq instead, and checked against both branches of the logic: a PR authored by someone else resolves to "skip", one authored by the caller to "body". Replacing the named bucket with `EVIDENCE_BUCKET` removed a working default and put nothing in its place, so the section told you to configure a bucket without saying what "conforming" meant. The policy is now stated: anonymous `s3:GetObject` under `public/*`, public-access blocks off for that bucket, `s3:ListBucket` to nobody. With a note that an org-owned bucket beats a personal one, since artifact links outlive their publisher. --- .../references/evidence-publishing.md | 23 +++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md index 2b452e7d..d86337b3 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md @@ -20,7 +20,24 @@ https://$EVIDENCE_BUCKET.s3.$EVIDENCE_REGION.amazonaws.com/public/metamask/pr-<n ``` The bucket must allow anonymous `GetObject` under `public/*` and must **not** allow listing, so -the prefix is not browsable — link individual files, and don't promise readers an index. +the prefix is not browsable — link individual files, and don't promise readers an index. If you +do not have one, that is the whole policy: + +```json +{ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Principal": "*", + "Action": "s3:GetObject", + "Resource": "arn:aws:s3:::YOUR-BUCKET/public/*" + }] +} +``` + +with `BlockPublicPolicy` and `RestrictPublicBuckets` disabled on that bucket and +`s3:ListBucket` granted to nobody. An org-owned bucket is preferable to a personal one: artifact +links outlive the person who published them. **Do not re-host to a personal repo.** A personal private repo returns 404 for every reader but its owner, so every artifact link published from one is dead on arrival. That was the previous @@ -124,7 +141,9 @@ Screenshots block (injected into `### After`, or appended under `### Screenshots ```bash PR=<n>; REPO=MetaMask/metamask-extension ME=$(gh api user --jq .login) -SURFACE=$(gh pr view "$PR" --repo "$REPO" --json author,commits --jq --arg me "$ME" ' +# Piped to jq rather than `gh --jq`: gh's built-in filter takes no --arg, and passing one +# fails with "accepts at most 1 arg(s)". +SURFACE=$(gh pr view "$PR" --repo "$REPO" --json author,commits | jq -r --arg me "$ME" ' if .author.login==$me then "body" elif ([.commits[] | select(.authors[].login==$me) | select([.authors[].login] | map(select(.!=$me and .!="Copilot" and (test("claude|anthropic")|not))) | length == 0)] | length) > 0 From d5fa3a549b7670d9d90cde1e91cb3afd3ef527e7 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 06:45:47 -0400 Subject: [PATCH 59/63] Enforce the evidence rules where the model cannot route around them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A results section reached a public PR whose entire content was hand-typed to look like terminal output. Three independent things had to hold for that, and all three did. The gate is model-invoked, so it can be skipped: the publish and the gate ran as two statements rather than one chain, and the verdict was read after the write. The hook that fires on the publish call carried a SECOND, narrower copy of the rules — keyed on verdict tokens — so a comment rendering no verdict satisfied neither copy. Two rule sets means the weaker one governs whatever falls between them. The hook now delegates to `attest-gate.sh`: one rule set, invoked by construction rather than by choice, in the mode the body's markers imply. It fails CLOSED once it has identified a body it is about to publish — an enforcement point that waves things through when it cannot find its rules is not one. And check 5 in `--diligence` had been rewritten as a phrase denylist ("npm pack", "complete specifier set"), which is precisely the regression its own comment records as having shipped four times: every property of plaintext is forgeable by whatever emits the plaintext. It is a medium test again — if the artifact shows a command or a run result, it owes the reader something fetchable. `/blob/` links are excluded, because a permalink to a `.json` file satisfied a naive extension test and was the specific reason the hand-typed section passed. Comment-update URLs carry the comment id, not the issue's, so check 12 was asking whether pull #5177261620 was open. Resolved through the API instead. Four-arm verified: blocks the exact command and body that shipped; ignores `ls`; ignores a `gh` read with no body write; refuses when the gate is unreachable. --- .../skills/evidence/hooks/pr-evidence-gate.py | 95 ++++++++++++++++++- .../skills/evidence/scripts/attest-gate.sh | 21 +++- 2 files changed, 110 insertions(+), 6 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py b/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py index 2bcd4651..03ac744e 100755 --- a/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py +++ b/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py @@ -13,13 +13,17 @@ Each class below implements a numbered item of `references/evidence-trustworthiness.md`. Contract: reads PreToolUse JSON on stdin. Exit 0 = allow. Exit 2 = block -(stderr shown to the model). Fails OPEN on anything it cannot parse, so it -never bricks unrelated Bash commands. +(stderr shown to the model). Fails OPEN on anything it cannot parse, so it never +bricks unrelated Bash commands — but once it has identified a body it is going to +publish, it fails CLOSED: if attest-gate.sh cannot be found or run, the write is +refused rather than waved through. """ import json import os import re +import subprocess import sys +import tempfile def _out_allow(): @@ -63,6 +67,7 @@ def main(): _out_allow() # can't read it -> don't block; nothing to scan violations = _scan(body) + violations += _run_attest_gate(body, cmd) if not violations: _out_allow() @@ -90,6 +95,9 @@ def main(): NEEDS = { + "attest-gate": "the check named above to pass — run scripts/attest-gate.sh yourself to iterate", + "gate-missing": "attest-gate.sh on disk; refusing to publish a body nothing verified", + "gate-error": "attest-gate.sh to run successfully; refusing to publish unverified", "verdict": "an inspectable ARTIFACT (https:// permalink, /blob/<sha>/, or a *.test.ts ref)", "observation": "an OBSERVATION artifact (screenshot/recording/log/JSON/permalink) — " "a /blob/ code link witnesses code, not runtime behavior", @@ -240,6 +248,89 @@ def _extract_body(cmd): ) + +# ── attest-gate delegation ──────────────────────────────────────────────────── +# The rules live in attest-gate.sh. This hook used to carry a second, narrower copy +# of them — keyed on verdict tokens — and a diligence comment that renders no verdict +# satisfied neither the copy here nor the copy there. Two rule sets means the weaker +# one governs whatever falls between them, which is how a results section of hand-typed +# terminal output reached a public PR under both gates. +# +# So: one rule set, invoked at the one point the model cannot route around. The model +# runs the gate by choice; this runs it by construction. +def _gv(kind, token, snippet): + """attest-gate findings, in the shape the reporter already renders.""" + return {"kind": kind, "token": token, "snippet": snippet} + + +def _repo_pr_from_cmd(cmd): + """owner/repo#N for check 12. A comment-update URL carries the COMMENT id, not the + issue's — reading it as a PR number asks the gate whether pull #5177261620 is open, + which 404s and reports as 'destination unknown'. So resolve it.""" + m = re.search(r"(?:--repo\s+|github\.com/|repos/)([\w.-]+/[\w.-]+)", cmd) + repo = m.group(1) if m else "" + if not repo: + return "" + c = re.search(r"issues/comments/(\d+)", cmd) + if c: + try: + out = subprocess.run( + ["gh", "api", f"repos/{repo}/issues/comments/{c.group(1)}", + "--jq", ".issue_url"], + capture_output=True, text=True, timeout=30) + n = re.search(r"/issues/(\d+)\s*$", out.stdout.strip()) + return f"{repo}#{n.group(1)}" if n else "" + except Exception: # noqa: BLE001 + return "" + n = re.search(r"(?:issues|pulls?)/(\d+)|\bpr\s+(?:comment|edit|create)\s+(\d+)", cmd) + num = next((g for g in (n.groups() if n else ()) if g), "") + return f"{repo}#{num}" if num else "" + + +def _find_gate(): + here = os.path.dirname(os.path.abspath(__file__)) + for cand in ( + os.path.join(here, "..", "scripts", "attest-gate.sh"), + os.path.join(here, "attest-gate.sh"), + os.path.expanduser("~/.claude/skills/mms-evidence/scripts/attest-gate.sh"), + os.environ.get("ATTEST_GATE", ""), + ): + if cand and os.path.isfile(cand): + return os.path.abspath(cand) + return "" + + +def _run_attest_gate(body, cmd): + gate = _find_gate() + if not gate: + # Fails CLOSED. An enforcement point that waves things through when it cannot + # find its rules is not an enforcement point; the whole reason this exists is + # that the model-invoked path was skippable. + return [_gv("gate-missing", "attest-gate.sh not found", + "Set ATTEST_GATE to its path, or install mms-evidence.")] + mode = ["--diligence"] if "LAVAMOAT_DILIGENCE_START" in body else [] + target = _repo_pr_from_cmd(cmd) + with tempfile.NamedTemporaryFile("w", suffix=".md", delete=False) as fh: + fh.write(body) + path = fh.name + try: + argv = ["bash", gate, path] + mode + (["--target", target] if target else []) + proc = subprocess.run(argv, capture_output=True, text=True, timeout=120) + except Exception as exc: # noqa: BLE001 - any failure to run it is a failure to verify + os.unlink(path) + return [_gv("gate-error", str(exc), "attest-gate.sh could not be run.")] + os.unlink(path) + if proc.returncode == 0: + return [] + out = proc.stdout.splitlines() + fails = [] + for i, ln in enumerate(out): + if ln.strip().startswith("FAIL"): + detail = out[i + 1].strip() if i + 1 < len(out) else "" + fails.append(_gv("attest-gate", ln.strip()[6:].strip(), detail)) + return fails or [_gv("attest-gate", f"exit {proc.returncode}", proc.stdout[-200:])] + + def _scan(body): # Strip bot-generated summary block — not our claim. body = re.sub(r"<!--\s*CURSOR_SUMMARY\s*-->.*?<!--\s*/CURSOR_SUMMARY\s*-->", diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 61d3728a..019274da 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -117,11 +117,24 @@ fi # and are actually properties of an unwitnessed local run. State them as the search # ("searched N files, no match") or publish the output; do not assert them as fact. if [ "$MODE" = diligence ]; then - if hasre "(complete|full) (specifier|import|require) set|byte-identical|identical across all|^Searched: .*tarball|npm pack" \ - && ! hasre 'actions/runs/[0-9]|/gist\.|https?://[^ )]+\.(txt|log|json)\b'; then - fail "5 runtime claims witnessed" "asserts a result only a local run could produce ($(grep -m1 -oiE '(complete|full) (specifier|import|require) set|byte-identical|identical across all|npm pack' "$FILE")) with nothing a reader can fetch. A /blob/ permalink witnesses a line, not your shell" + # A permalink is the medium for a CITATION — a reader clicks it and lands on the line. It is + # not the medium for anything you RAN. The first version of this branch tested for phrases + # ("npm pack", "complete specifier set") and passed an artifact whose entire results section + # was hand-typed to look like terminal output, because none of those words appeared in it. + # That is the regression the block below already documents as having shipped four times: + # every property of plaintext is forgeable by whatever emits the plaintext. So the test is + # the same one, on the same terms — if the artifact shows a command or a run result, it owes + # the reader something fetchable. + # `/blob/` is a CITATION, never a capture — it witnesses a line in a file, not a run. + # Excluding it matters: a permalink to `policy-override.json` ends in `.json` and satisfied + # a naive extension test, so an artifact whose entire results section was hand-typed passed + # on the strength of a source link. + if hasre '^\$ |^ *\$ |\bexit [0-9]|\bexit=[0-9]' \ + && ! grep -qE 'actions/runs/[0-9]|/gist\.|!\[[^]]*\]\(https?://' "$FILE" \ + && ! grep -E 'https?://[^ )]+\.(txt|log|json)\b' "$FILE" | grep -qv '/blob/'; then + fail "5 captured artifact" "shows a command or a run result with nothing a reader can fetch — a fenced block is your transcription, whatever produced it. Publish the log/gist/run and link it" else - pass "5 runtime claims witnessed" + pass "5 captured artifact" fi elif ! hasre '!\[[^]]*\]\(https?://|<img [^>]*src="https?://|actions/runs/[0-9]|/gist\.|https?://[^ )]+\.(png|jpg|jpeg|gif|svg|txt|log|json)\b'; then fail "5 captured artifact" "no reader-verifiable capture — an image of the tool surface, a run link, or a hosted artifact. A fenced block is the author\'s transcription, whatever produced it" From 27be7aeeb0a02f25df84eadfc51116432ad172d3 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 07:59:16 -0400 Subject: [PATCH 60/63] =?UTF-8?q?Add=20`attest`=20=E2=80=94=20the=20publis?= =?UTF-8?q?h=20gate,=20ported=20from=20a=20personal=20command?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `mms-evidence` named `/attest` as the gate its output passes and shipped no such thing. Anyone installing the skill got phase 0 as a script and a reference to a command only its author had, which is half a publish path for everyone else. Phase 0 is `attest-gate.sh`, already here. Phase 1 is three briefs sent to fresh instances — frame, coverage, and how it reads to a stranger — written out in `references/dispatched-passes.md` so dispatch does not depend on commands that live elsewhere. The check table in `references/phase-0-checks.md` is generated from the gate rather than retyped, so it cannot drift from what runs. Two things the port makes explicit that the original left to discipline. The gate must be the same shell chain as the publish, because running both and reading the verdict after the write is how a blocked artifact reached a public PR. And softening a check to fit the case in hand is called out as an anti-pattern: if the new version could be satisfied by better prose alone, it is no longer the check. --- .../attest/references/dispatched-passes.md | 41 +++++++ .../attest/references/phase-0-checks.md | 35 ++++++ domains/pr-workflow/skills/attest/skill.md | 103 ++++++++++++++++++ .../skills/evidence/scripts/attest-gate.sh | 7 +- domains/pr-workflow/skills/evidence/skill.md | 2 +- 5 files changed, 186 insertions(+), 2 deletions(-) create mode 100644 domains/pr-workflow/skills/attest/references/dispatched-passes.md create mode 100644 domains/pr-workflow/skills/attest/references/phase-0-checks.md create mode 100644 domains/pr-workflow/skills/attest/skill.md diff --git a/domains/pr-workflow/skills/attest/references/dispatched-passes.md b/domains/pr-workflow/skills/attest/references/dispatched-passes.md new file mode 100644 index 00000000..0583920e --- /dev/null +++ b/domains/pr-workflow/skills/attest/references/dispatched-passes.md @@ -0,0 +1,41 @@ +# Phase 1 — the three dispatched briefs + +Send each to a **fresh instance** with the artifact and nothing else: not the transcript, not +your reasoning, not what you expect it to find. Context is what you are testing for. An instance +that knows what you meant will read what you meant. + +Run them concurrently — they are independent, and sequencing lets the first one's findings frame +the others. + +## outframe — contest the frame + +> You are reading a finished set of findings you did not produce. Do not check whether the +> findings are correct. Ask what claim was chosen and what a different framing makes visible: +> what question would a reader with different priorities have asked of the same material, what +> does the chosen frame make it impossible to notice, and which of the findings only look +> significant because of how the problem was cut. Return findings the framing hid, not a +> critique of the writing. + +## missing — contest the coverage + +> You are auditing a completed run for what it did not do. Enumerate: a modality that was not +> run, a claim asserted but not verified, a source cited but not read, a case the method +> structurally cannot reach. For each, say what running it would cost and what it could change. +> Do not restate what the run found. Absence is the deliverable. + +## press — read it as the stranger + +> You are the reviewer this lands in front of, with no context and a decision to make. Read only +> the artifact. Say what you would have to take on trust, which number you could not check if you +> wanted to, what reads as a measurement but is a sentence, and anything that assumes you were +> present for work you were not. Flag register slips: hedging that reads as concealment, +> confidence that outruns the evidence, and any place the author's process shows through. + +## Reading the returns + +A finding from any pass that invalidates the claim is `BLOCKED`. A finding that qualifies it is +`ATTESTED WITH` — and the caveat goes **into the published artifact**, not just into the verdict, +or the reader never sees it. + +Disagreement between passes is signal, not noise: `press` clearing something `outframe` flagged +usually means the artifact reads well and is framed wrong, which is the more dangerous state. diff --git a/domains/pr-workflow/skills/attest/references/phase-0-checks.md b/domains/pr-workflow/skills/attest/references/phase-0-checks.md new file mode 100644 index 00000000..994ad33c --- /dev/null +++ b/domains/pr-workflow/skills/attest/references/phase-0-checks.md @@ -0,0 +1,35 @@ +# Phase 0 — what each check catches + +Generated from the checks in `mms-evidence/scripts/attest-gate.sh`; that script is the +authority. Each entry exists because a run shipped without it. + +| # | check | run mode | diligence mode | +|---|---|---|---| +| 1 | marker pair | ✓ | ✓ | +| 2 | canonical header | ✓ | ✓ | +| 3 | verdict line | ✓ | ✓ | +| 4 | citations pinned | ✓ | ✓ | +| 5 | captured artifact | ✓ | ✓ | +| 6 | no prescriptions | ✓ | ✓ | +| 7 | no process narration | ✓ | ✓ | +| 8 | verdict is earned | ✓ | ✓ | +| 9 | verdict matches artifact | ✓ | ✓ | +| 10 | floats something for review | ✓ | ✓ | +| 11 | disclaimer present and early | ✓ | ✓ | +| 12 | destination is open | ✓ | ✓ | +| 13 | figures trace to an exhibit | ✓ | ✓ | + +Checks 1–4 differ by mode: in `--diligence` they test that contract's own marker pair, its +header, and that citations are pinned to a tag or SHA rather than a branch head, and the +verdict-line check reports SKIP because a diligence artifact renders none. Checks 8 and 9 SKIP +for the same reason. Everything from 5 down is shared, because those defects are shared. + +**Check 5 is the one that matters, and it asks for a medium.** Every earlier version tested a +property of the plaintext — does it carry a marker, does the command contain a placeholder — and +each caught one defect and missed the next, because every property of plaintext is forgeable by +whatever emits the plaintext. Four runs shipped that way. A `/blob/` permalink is a citation and +does not satisfy it: it witnesses a line in a file, never a run. + +**Check 12 tests the destination**, which no property of the text reveals. Across one register of +published runs, 22 of 27 comments went to pull requests that had already merged — median 22 days +after the merge, gate-clean every time. diff --git a/domains/pr-workflow/skills/attest/skill.md b/domains/pr-workflow/skills/attest/skill.md new file mode 100644 index 00000000..e5af6d89 --- /dev/null +++ b/domains/pr-workflow/skills/attest/skill.md @@ -0,0 +1,103 @@ +--- +name: attest +description: The gate an evidence artifact passes before it is published to a pull request, issue or shared tracker. Two halves that do not substitute for each other — a mechanical pass that greps for the properties a reader needs (marker pair, pinned environment, a captured artifact rather than typed prose, a destination that is still open) and a dispatched pass sent to fresh instances that contest the framing, the coverage, and how it reads to a stranger. The author is the wrong checker: they remember running the check, and the memory supplies the provenance the text lacks. Verdicts are attested, attested with named caveats, blocked, or not a run — the last being common and legitimate, because a run that could not execute has produced nothing to publish. Triggers on mms-attest, or before posting any evidence, validation or diligence output to a public surface. +maturity: experimental +--- + +# /mms-attest + +The gate an evidence artifact passes before it leaves your hands. Use before posting any +`/mms-evidence` or diligence output to a pull request, issue, or shared tracker. + +## The author is the wrong reader, and the wrong checker + +A validation run claims something was measured. Its characteristic failure is not a wrong number +— it is **prose that reads like a measurement**. An operator who ran the check cannot see this, +because they remember running it; the memory supplies the provenance the text lacks, before the +eye registers that it was missing. + +This is not hypothetical. A run in this workflow shipped a results section whose commands, exit +codes and "reached 100%" were typed by hand, while the real logs sat unpublished on disk. The +author had the skill installed that forbids exactly that. + +So the gate has two halves, and neither substitutes for the other. + +**The mechanical half is not advisory.** Marker presence, a pinned environment, whether any +fenced block is a tool's output rather than the author's transcription, whether the destination +is still open — all greppable. Anything checkable is checked before a model is asked for +judgement, because a model asked "is this good evidence?" answers from inside the frame that +produced it. + +**The dispatched half is positional.** Contesting the frame, the coverage, and the reading cannot +be self-run, for the same reason an author cannot proofread their own sentence for a word their +eye supplies. + +## Phase 0 — mechanical, no model + +``` +scripts/attest-gate.sh <artifact.md> --target <owner/repo#N> +scripts/attest-gate.sh <artifact.md> --target <owner/repo#N> --diligence +``` + +Thirteen checks; every one a hard fail. `--diligence` swaps the four Validation-Run envelope +checks for a no-verdict contract's own and shares everything downstream. See +[references/phase-0-checks.md](references/phase-0-checks.md) for what each check exists to catch +and the run that caused it to be written. + +**Run it as the same command that publishes, or it is a log line.** The gate and the write must +be one chain — `gate && publish`. Running both and reading the verdict afterwards is how a +blocked artifact reaches a public PR. The `hooks/pr-evidence-gate.py` PreToolUse hook enforces +this independently of your discipline, and fails closed; phase 0 is what you run to iterate +before it does. + +## Phase 1 — dispatched, three lenses + +| pass | reads for | returns | +|---|---|---| +| **outframe** | the frame — what claim was chosen, and what a different framing makes visible | findings the framing hid | +| **missing** | coverage — modality not run, claim unverified, source unread | the gap list | +| **press** | the text as it ships, as the stranger who has to act on it | leak and register findings | + +Dispatch to fresh instances is the mechanism, not an optimisation: a self-run frame check is +composed inside the frame it is meant to test. Briefs in +[references/dispatched-passes.md](references/dispatched-passes.md). + +Skipping a pass is allowed. Silently skipping it is not — name it as skipped in the verdict. + +## Phase 2 — shape + +Front-load the verdict, cut anything that does not change what the reader does, keep every +artifact and move only its placement. Shape only, after content is settled — a shape pass that +reaches content is how a capability table gets dissolved into paragraphs and the comment's +payload disappears. + +## Verdict + +``` +ATTESTED phase 0 clean, no blocking finding from phase 1 +ATTESTED WITH publishable, with named caveats carried INTO the artifact +BLOCKED phase 0 failure, or a phase 1 finding that invalidates the claim +NOT A RUN nothing was measured; there is no artifact to publish +``` + +`NOT A RUN` is legitimate and common. A run that could not execute its check produced no +evidence, and publishing the attempt with a disclaimer is worse than publishing nothing — the +disclaimer reads as hedging and the figure is kept anyway. + +## Anti-patterns + +| Bad | Good | +|---|---| +| Running phase 1 to decide phase 0 | Mechanical checks first; cheap and unarguable | +| Self-running the dispatched passes | Dispatch, or skip and say it was skipped | +| Attesting your own run | The gate is positional; an author attesting themselves attests nothing | +| Treating phase 0 items as advisory | Every one is a hard fail | +| `ATTESTED WITH` as a soft pass | The caveat goes *into the published artifact*, not just the verdict | +| Softening a check to fit the case in hand | If the new version could be satisfied by better prose alone, it is no longer the check | + +## Related + +- `mms-evidence` — produces the artifact this gates +- `mms-instrument-check` — prove the instrument fires before its output counts +- `mms-unmeasured-join` — audit the inference between the facts +- `mms-scope-of-search` — what a negative result is a fact about diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 019274da..4168e740 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -167,7 +167,12 @@ else pass "6 no prescriptions" fi -if hasi "I originally|correction to my earlier|filed by me|hard to calibrate|I withdraw|my earlier comment"; then +# Drafting history is the author's, not the reader's: a reader who never saw the earlier +# version learns nothing from being told it existed, and the byline may not be yours. +# The list grew after a comment shipped a '### Correction:' section retracting its own +# previous revision in place — right instinct, wrong surface. Retract by restating the +# finding correctly; the account of how it changed belongs in a postmortem. +if hasi "I originally|correction to my earlier|filed by me|hard to calibrate|I withdraw|my earlier comment|earlier revision|previous revision|an earlier version of this|is withdrawn|that claim was wrong|^#{1,4} *Correction[: ]|corrected below|see the correction"; then fail "7 no process narration" "contains first-person process commentary — the reader did not see the earlier draft, and the byline may not be yours" else pass "7 no process narration" diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index b879f483..78ae160a 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -326,7 +326,7 @@ Check 5 is the one that matters and the easiest to slip past: if every character is one the operator typed, the run published an assertion. Pass `--reference <showcase>` to compare capture density against a known-good artifact. -This is phase 0 of `/attest`; phases 1 and 2 dispatch +This is phase 0 of `mms-attest`; phases 1 and 2 dispatch `/outframe ‖ /missing ‖ /press` then `/trim` to fresh instances, because those passes cannot be self-run — the author is positionally the wrong reader. From e06906154f23f5a6f111965eab9fdc1d2c048bea Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 08:52:47 -0400 Subject: [PATCH 61/63] Add `gate-controls.sh`, and scope the gate to evidence artifacts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three copies of this hook were on one machine and the oldest was the one wired into settings — no `gh api` matcher, no delegation to `attest-gate.sh`. Every publish through that path went ungated, and nothing noticed, because a gate that blocks nothing looks exactly like a gate with nothing to block. `gate-controls.sh` is the thing that would have noticed. Six arms: three publish routes that must block, three inputs that must pass. It copies the hook somewhere with no sibling `scripts/` so `_find_gate()` resolves the way it does in production rather than the way it does in a checkout — the difference matters, and testing the checkout copy is how the deployed one stayed broken. Both halves earned their place immediately. The negative arm caught that delegating to attest-gate ran it over EVERY published body, so an ordinary reply was judged as a failed validation run; wiring that would have blocked every normal comment. The gate now applies only to bodies carrying an artifact marker or a verdict line. And the positive arm caught itself: the enrichment probe was a single sentence, which that rule correctly ignores, so the arm had been passing because attest-gate blocked the body for an unrelated reason. A probe that fires for the wrong reason reports a working rule. The `enrichment` class is ported forward from the older copy, with the constant it depends on — it existed in the deployed version and in neither newer one. --- .../skills/evidence/hooks/gate-controls.sh | 56 +++++++++++++++++++ .../skills/evidence/hooks/pr-evidence-gate.py | 54 ++++++++++++++++++ 2 files changed, 110 insertions(+) create mode 100755 domains/pr-workflow/skills/evidence/hooks/gate-controls.sh diff --git a/domains/pr-workflow/skills/evidence/hooks/gate-controls.sh b/domains/pr-workflow/skills/evidence/hooks/gate-controls.sh new file mode 100755 index 00000000..ce857791 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/hooks/gate-controls.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# +# Control matrix for the emit-time gate. Run it from anywhere; it copies the hook to a +# directory with no sibling scripts/ so `_find_gate()` resolves the way it does in +# production rather than the way it does in a checkout. +# +# It exists because three copies of this hook were on one machine, the oldest was the one +# wired into settings, and it had no `gh api` matcher — so every publish through that path +# went ungated for weeks while two newer copies sat unused. Nothing noticed, because a gate +# that blocks nothing is indistinguishable from a gate with nothing to block. +# +# Positives must block (exit 2). Negatives must pass (exit 0). Both halves matter: a gate +# that blocks everything is as broken as one that blocks nothing, and only the negative +# arm catches it. +set -uo pipefail +HOOK="${1:-$(cd "$(dirname "$0")" && pwd)/pr-evidence-gate.py}" +[ -f "$HOOK" ] || { echo "usage: gate-controls.sh [path/to/pr-evidence-gate.py]" >&2; exit 2; } + +tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' EXIT +cp "$HOOK" "$tmp/hook.py" + +printf '<!-- LAVAMOAT_DILIGENCE_START -->\n**LavaMoat grants — x**\n\n```\n$ yarn build\n exit 0\n```\n<!-- LAVAMOAT_DILIGENCE_END -->\n' > "$tmp/bad.md" +printf 'Addressed: see the linked run.\n' > "$tmp/reply.md" +# The enrichment rule needs the REPORT shape, not report vocabulary: three or more +# paragraphs, a cited link, and no reply-template opener. A one-line probe passes it for +# the wrong reason, which is how a mis-specified positive arm reads as a working rule. +cat > "$tmp/finding.md" <<'BODY' +The migration path is ground-truthed against the fixture set and rules out the ordering hazard. + +Two of the three cases resolve through the same upstream guard, so the remaining exposure is +the un-guarded third: https://github.com/o/r/blob/abc123/src/migrate.ts#L40 + +That leaves the rollback lane unaccounted for, which is worth its own pass before this lands. +BODY + +probe() { printf '{"tool_name":"Bash","tool_input":{"command":%s}}' "$(python3 -c 'import json,sys;print(json.dumps(sys.argv[1]))' "$1")"; } + +fails=0 +check() { # name expected command + local name="$1" want="$2" cmd="$3" got + probe "$cmd" | python3 "$tmp/hook.py" >/dev/null 2>&1; got=$? + if [ "$got" = "$want" ]; then printf ' ok %-34s exit=%s\n' "$name" "$got" + else printf ' FAIL %-34s exit=%s want=%s\n' "$name" "$got" "$want"; fails=$((fails+1)); fi +} + +echo "gate-controls: $HOOK" +check "positive: gh api body write" 2 "gh api repos/o/r/issues/comments/1 -X PATCH -F body=@$tmp/bad.md" +check "positive: gh pr comment" 2 "gh pr comment 1 --repo o/r --body-file $tmp/bad.md" +check "positive: finding via comment" 2 "gh issue comment 1 --repo o/r --body-file $tmp/finding.md" +check "negative: unrelated command" 0 "ls -la" +check "negative: gh read, no body" 0 "gh pr view 1 --repo o/r" +check "negative: a reply is a reply" 0 "gh issue comment 1 --repo o/r --body-file $tmp/reply.md" + +echo +[ "$fails" -eq 0 ] && { echo "gate-controls: all arms behave"; exit 0; } +echo "gate-controls: $fails arm(s) wrong — the gate is not doing what it claims"; exit 1 diff --git a/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py b/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py index 03ac744e..aae8ba6f 100755 --- a/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py +++ b/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py @@ -67,6 +67,8 @@ def main(): _out_allow() # can't read it -> don't block; nothing to scan violations = _scan(body) + if re.search(r"\bgh\s+issue\s+comment\b", cmd): + violations += _scan_enrichment_via_comment(body) violations += _run_attest_gate(body, cmd) if not violations: _out_allow() @@ -95,6 +97,7 @@ def main(): NEEDS = { + "enrichment": "a BODY EDIT instead (`gh issue edit --body-file`) — this reads like a resolved finding, not a reply", "attest-gate": "the check named above to pass — run scripts/attest-gate.sh yourself to iterate", "gate-missing": "attest-gate.sh on disk; refusing to publish a body nothing verified", "gate-error": "attest-gate.sh to run successfully; refusing to publish unverified", @@ -300,7 +303,27 @@ def _find_gate(): return "" +# Only evidence artifacts are held to the evidence contract. An ordinary reply is not a +# failed validation run, and running the gate over every published body blocks every normal +# comment on checks 1-4 — caught by the negative arm of gate-controls.sh before this was +# wired, which is the entire reason that arm exists. A gate that blocks everything is as +# broken as one that blocks nothing, and only the negative control tells them apart. +ARTIFACT_MARKERS = ( + "VALIDATION_RUN_START", + "LAVAMOAT_DILIGENCE_START", + "## 🧪 Validation Run", +) + + +def _is_evidence_artifact(body): + if any(m in body for m in ARTIFACT_MARKERS): + return True + return bool(re.search(r"^\*\*Verdict:\*\*", body, re.M)) + + def _run_attest_gate(body, cmd): + if not _is_evidence_artifact(body): + return [] gate = _find_gate() if not gate: # Fails CLOSED. An enforcement point that waves things through when it cannot @@ -331,6 +354,37 @@ def _run_attest_gate(body, cmd): return fails or [_gv("attest-gate", f"exit {proc.returncode}", proc.stdout[-200:])] +REPLY_TEMPLATE_OPENER = re.compile( + r"(?i)^\s*(?:addressed|resolved|reverted)\s*:\s*\S" +) + + +def _scan_enrichment_via_comment(body): + """`gh issue comment` posting a standalone finding — should be a body edit. + + Structural signal, not a vocabulary one: the real instance this is modeled + on (planning#7508) used none of VERDICT's literal words ("ground-truthed", + "rules out", "de-risks" — not "confirmed"/"proven"/etc), so reusing that + regex as the discriminator missed it entirely on the first attempt (caught + by testing against the real text, not by reasoning about it). What actually + distinguishes a standalone report from a reply, regardless of vocabulary: + several paragraphs, at least one cited link, and no reply-template opener. + A properly-templated reply (Addressed:/Resolved:/Reverted: <fact>.) is + excused unconditionally — that template is itself the correct convention + for a comment (exogram-core: ghostwrite-review-reply-register), so + following it is the signal of doing this right, not a loophole. + """ + if REPLY_TEMPLATE_OPENER.search(body.strip()): + return [] + paras = [p.strip() for p in re.split(r"\n\s*\n", body) if p.strip()] + if len(paras) < 3: + return [] # short reply, even with a link, isn't a standalone report + if not re.search(r"https?://\S+", body): + return [] # no cited evidence — not the report shape either + snip = re.sub(r"\s+", " ", paras[0])[:120] + return [{"token": "standalone finding", "snippet": snip, "kind": "enrichment"}] + + def _scan(body): # Strip bot-generated summary block — not our claim. body = re.sub(r"<!--\s*CURSOR_SUMMARY\s*-->.*?<!--\s*/CURSOR_SUMMARY\s*-->", From 2bb5a29a7137c2b3dd33249a25f57f7127129fa3 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 11:01:29 -0400 Subject: [PATCH 62/63] Check that the gate is wired, not only that it works MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The six arms prove the script blocks what it should. They say nothing about whether anything calls it, and those are different questions — a hook that is unwired, or wired to a path that no longer exists, is indistinguishable from a hook with nothing to block. One session ran start to finish with every PreToolUse hook inert: 306 certification markers written, none enforcing anything, and the ritual read as compliance. The check enumerates config roots rather than trusting `$HOME`. Its first version did trust it, found one settings file, reported it as "the" wiring and never looked at the second — because `$HOME` here points at a per-account directory rather than the login home. That is the same defect one level up, caught only because two configs were known to exist and one was missing from the output. It stops short of claiming liveness, and says so: a settings file naming an existing file is not proof the running session loaded it. Only a command the gate must block, issued in a session and observed to be blocked, shows that. --- .../skills/evidence/hooks/gate-controls.sh | 59 ++++++++++++++++++- 1 file changed, 58 insertions(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/evidence/hooks/gate-controls.sh b/domains/pr-workflow/skills/evidence/hooks/gate-controls.sh index ce857791..44fdf913 100755 --- a/domains/pr-workflow/skills/evidence/hooks/gate-controls.sh +++ b/domains/pr-workflow/skills/evidence/hooks/gate-controls.sh @@ -43,7 +43,57 @@ check() { # name expected command else printf ' FAIL %-34s exit=%s want=%s\n' "$name" "$got" "$want"; fails=$((fails+1)); fi } +# ── wiring ─────────────────────────────────────────────────────────────────────── +# The arms below prove the SCRIPT works. They say nothing about whether anything calls +# it, and those are different questions: a hook that is not wired, or wired to a path +# that no longer exists, is indistinguishable from a hook with nothing to block. One +# session ran to completion with every PreToolUse hook inert — 306 certification markers +# written, none of them enforcing anything — because nobody asked this question. +wiring() { + local found=0 + # $HOME is not necessarily the login home — an account-switching setup points it at a + # per-account directory, which is exactly the case this was first run in. Enumerating + # from $HOME alone found one config, reported it as "the" wiring, and never looked at + # the other. Derive the roots instead, and de-duplicate by realpath so a symlinked + # config is not counted twice or missed once. + local roots=() seen=() r + for r in "$HOME" "$(getent passwd "$(id -un)" | cut -d: -f6)" /home/*/ ; do + [ -d "$r" ] || continue + roots+=("$r/.claude/settings.json") + for a in "$r"/.claude-accts/*/.claude/settings.json; do [ -f "$a" ] && roots+=("$a"); done + done + for cfg in "${roots[@]}"; do + [ -f "$cfg" ] || continue + local rp; rp=$(readlink -f "$cfg") + case " ${seen[*]} " in *" $rp "*) continue ;; esac + seen+=("$rp") + local cmd + cmd=$(python3 -c ' +import json,sys +try: d=json.load(open(sys.argv[1])) +except Exception: sys.exit(0) +out=[] +def w(o): + if isinstance(o,dict): + for k,v in o.items(): + if k=="command" and isinstance(v,str) and "pr-evidence-gate" in v: out.append(v) + else: w(v) + elif isinstance(o,list): + [w(x) for x in o] +w(d.get("hooks",{})) +print(out[0] if out else "")' "$cfg") + [ -n "$cmd" ] || continue + found=1 + local path; path=$(printf '%s' "$cmd" | grep -oE '[^ "]*pr-evidence-gate\.py') + path="${path/\$HOME/$HOME}" + if [ -f "$path" ]; then printf ' ok wired: %s\n' "${cfg/#$HOME/~}" + else printf ' FAIL wired to a missing file: %s → %s\n' "${cfg/#$HOME/~}" "$path"; fails=$((fails+1)); fi + done + [ "$found" = 1 ] || { printf ' FAIL no settings file registers the gate as a PreToolUse hook\n'; fails=$((fails+1)); } +} + echo "gate-controls: $HOOK" +wiring check "positive: gh api body write" 2 "gh api repos/o/r/issues/comments/1 -X PATCH -F body=@$tmp/bad.md" check "positive: gh pr comment" 2 "gh pr comment 1 --repo o/r --body-file $tmp/bad.md" check "positive: finding via comment" 2 "gh issue comment 1 --repo o/r --body-file $tmp/finding.md" @@ -52,5 +102,12 @@ check "negative: gh read, no body" 0 "gh pr view 1 --repo o/r" check "negative: a reply is a reply" 0 "gh issue comment 1 --repo o/r --body-file $tmp/reply.md" echo -[ "$fails" -eq 0 ] && { echo "gate-controls: all arms behave"; exit 0; } +if [ "$fails" -eq 0 ]; then + echo "gate-controls: all arms behave, and the gate is wired" + echo + echo "Wiring is not liveness. This proves a settings file names an existing file; it" + echo "cannot prove the running session loaded it. For that, run a command the gate must" + echo "block and confirm it is blocked — in a session, not here." + exit 0 +fi echo "gate-controls: $fails arm(s) wrong — the gate is not doing what it claims"; exit 1 From d3d153d0ad918f8c53f129616f9367dcf980f3f2 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Wed, 5 Aug 2026 08:28:35 -0400 Subject: [PATCH 63/63] Fail closed when the body cannot be read MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This is the bypass. The hook reads the command as text, so `--body-file $DIR/comment.md` resolves to nothing — and the code called that "can't read it -> nothing to scan" and allowed the write. Every publish in one long session used a shell variable for the path, so every one of them went ungated, including a comment whose entire results section was hand-typed to look like terminal output. That same body, passed by literal path, is blocked. An unreadable body is not an absent risk. By that point the command is already identified as an outward-facing write; not knowing what it carries is the reason to stop. Also rejects a body argument carrying `$` or a backtick — `--body "$(cat f)"` extracts the literal characters, scans clean, and publishes whatever the shell substitutes afterwards. The check reads the command ARGUMENT, not the body. A first version scanned body text for shell metacharacters and blocked every evidence comment ever written, because markdown inline code is backticks and these artifacts are full of them. Caught by the negative arm, which is the half of a control matrix that earns its place on days like this. Three arms added, verified end to end: the command that slipped through minutes earlier is now stopped by the deployed hook. --- .../skills/evidence/hooks/gate-controls.sh | 7 +++ .../skills/evidence/hooks/pr-evidence-gate.py | 45 ++++++++++++++++++- 2 files changed, 51 insertions(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/evidence/hooks/gate-controls.sh b/domains/pr-workflow/skills/evidence/hooks/gate-controls.sh index 44fdf913..286d067f 100755 --- a/domains/pr-workflow/skills/evidence/hooks/gate-controls.sh +++ b/domains/pr-workflow/skills/evidence/hooks/gate-controls.sh @@ -101,6 +101,13 @@ check "negative: unrelated command" 0 "ls -la" check "negative: gh read, no body" 0 "gh pr view 1 --repo o/r" check "negative: a reply is a reply" 0 "gh issue comment 1 --repo o/r --body-file $tmp/reply.md" +# The gate reads the command as text, so a body it cannot resolve is a body it cannot +# check. These three are how an entire session of publishes went ungated while every +# other arm above was green: the path was assembled from a shell variable each time. +check "positive: body path via \$VAR" 2 'gh pr comment 1 --repo o/r --body-file $D/c.md' +check "positive: body via \$(cat ...)" 2 'gh pr comment 1 --repo o/r --body "$(cat c.md)"' +check "positive: gh api body via \$VAR" 2 'gh api repos/o/r/issues/1/comments -F body=@$D/c.md' + echo if [ "$fails" -eq 0 ]; then echo "gate-controls: all arms behave, and the gate is wired" diff --git a/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py b/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py index aae8ba6f..b79d0e20 100755 --- a/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py +++ b/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py @@ -63,8 +63,38 @@ def main(): _out_allow() body = _extract_body(cmd) + # An unexpanded shell construct is not a body. `--body "$(cat f)"` extracts the literal + # characters `$(cat f)`, which scans clean and publishes whatever the shell substitutes + # later — the gate would be inspecting a string the reader never sees. + # + # Look at the ARGUMENT, not the body text. A first attempt scanned the body for `$` and + # backticks and rejected every evidence comment ever written, because markdown inline + # code is backticks and these artifacts are full of them. The shell metacharacters that + # matter are in the command; the body is just prose. + if _body_arg_is_unresolvable(cmd): + body = "" if not body: - _out_allow() # can't read it -> don't block; nothing to scan + # FAIL CLOSED. The previous reasoning here was "can't read it -> nothing to scan", + # which inverts the situation: by this point the command has already been identified + # as an outward-facing write, so an unreadable body is not an absent risk, it is an + # unverifiable one. + # + # This was not theoretical. The extraction is textual, so a path assembled from a + # shell variable — `--body-file $S/comment.md` — or a body spliced in with + # `--body "$(cat f)"` yields nothing, and every such publish sailed through while + # the gate reported itself healthy. An entire session of publishes went ungated this + # way, including one the gate blocks when handed the same body by literal path. + _block( + "EVIDENCE GATE (PreToolUse) — blocked an outward-facing write whose body " + "could not be read.\n\n" + "The body path could not be resolved from the command. This hook reads the " + "command as text and cannot expand shell variables, command substitution, or " + "heredocs, so a body assembled that way is unverifiable rather than safe.\n\n" + "Pass a literal path:\n" + " gh pr comment <n> --repo <owner/repo> --body-file /abs/path/to/comment.md\n\n" + "If the body genuinely has no file, write it to one first. The gate has to see " + "what you are about to publish.\n" + ) violations = _scan(body) if re.search(r"\bgh\s+issue\s+comment\b", cmd): @@ -385,6 +415,19 @@ def _scan_enrichment_via_comment(body): return [{"token": "standalone finding", "snippet": snip, "kind": "enrichment"}] +# The text following --body/--body-file, up to the next argument. If it carries a variable, +# a command substitution, or a backtick, this hook cannot know what will actually be sent. +BODY_ARG = re.compile(r"(?:--body-file|--body|-F\s+body|--field\s+body|--raw-field\s+body)[=\s]+(\S+)") + + +def _body_arg_is_unresolvable(cmd): + m = BODY_ARG.search(cmd) + if not m: + return False + arg = m.group(1) + return bool(re.search(r"\$|`", arg)) + + def _scan(body): # Strip bot-generated summary block — not our claim. body = re.sub(r"<!--\s*CURSOR_SUMMARY\s*-->.*?<!--\s*/CURSOR_SUMMARY\s*-->",