From 4a4cd854e55508d1605d9ff35e7eca17b7cbb0d4 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Fri, 5 Jun 2026 14:52:31 -0400 Subject: [PATCH 001/135] Add analytics domain: Sentry quota, MCP queries, instrumentation --- CHANGELOG.md | 4 + .../knowledge/metrametrics-identity.md | 41 +++++++ .../analytics/knowledge/segment-governance.md | 40 +++++++ .../analytics/knowledge/span-sub-sampling.md | 72 ++++++++++++ .../repos/metamask-extension.md | 61 ++++++++++ .../skills/analytics-instrumentation/skill.md | 89 +++++++++++++++ .../repos/metamask-extension.md | 60 ++++++++++ .../skills/sentry-mcp-queries/skill.md | 107 ++++++++++++++++++ .../sentry-quota/repos/metamask-extension.md | 50 ++++++++ .../analytics/skills/sentry-quota/skill.md | 78 +++++++++++++ 10 files changed, 602 insertions(+) create mode 100644 domains/analytics/knowledge/metrametrics-identity.md create mode 100644 domains/analytics/knowledge/segment-governance.md create mode 100644 domains/analytics/knowledge/span-sub-sampling.md create mode 100644 domains/analytics/skills/analytics-instrumentation/repos/metamask-extension.md create mode 100644 domains/analytics/skills/analytics-instrumentation/skill.md create mode 100644 domains/analytics/skills/sentry-mcp-queries/repos/metamask-extension.md create mode 100644 domains/analytics/skills/sentry-mcp-queries/skill.md create mode 100644 domains/analytics/skills/sentry-quota/repos/metamask-extension.md create mode 100644 domains/analytics/skills/sentry-quota/skill.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 383d49c7..89ff9e2b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- Add `analytics` domain: Sentry span-quota guard, Sentry MCP query workflows, instrumentation methodology, and supporting knowledge + ## [0.1.0] ### Added diff --git a/domains/analytics/knowledge/metrametrics-identity.md b/domains/analytics/knowledge/metrametrics-identity.md new file mode 100644 index 00000000..1faed3ba --- /dev/null +++ b/domains/analytics/knowledge/metrametrics-identity.md @@ -0,0 +1,41 @@ +--- +name: metrametrics-identity +domain: analytics +description: isOptIn:true unconditionally strips user identity in MetaMetricsController — always sends as anonymous ID +--- + +# MetaMetrics Identity Stripping + +## The Mechanism + +In `MetaMetricsController` (`app/scripts/controllers/metametrics-controller.ts`): + +```typescript +if (excludeMetaMetricsId || (isOptIn && !metaMetricsIdOverride)) { + idType = 'anonymousId'; + idValue = METAMETRICS_ANONYMOUS_ID; // 0x0000000000000000 +} +``` + +When `isOptIn: true` with no `metaMetricsIdOverride`: +- The user's real `metaMetricsId` is discarded +- ALL such events share a single anonymous ID (`0x0000000000000000`) in Segment +- User-level attribution is completely lost + +This is **unconditional** — it applies to fully opted-in users with valid IDs, not just anonymous users. + +## Intended Use + +The onboarding opt-in flow (`creation-successful.tsx`) — where the user hasn't committed to MetaMetrics yet and no `metaMetricsId` has been persisted. The event must fire regardless of opt-in state. + +## The Misuse Pattern + +Post-opt-in `trackEvent` calls with `{ isOptIn: true }` without `metaMetricsIdOverride`. Defeats the purpose of Segment user-level dimensions (account types, feature flags). + +## Detection + +```bash +grep -r "isOptIn: true" app/scripts/ ui/ --include="*.ts" --include="*.tsx" +``` + +Any occurrence outside `creation-successful.tsx` (or the onboarding flow) is suspect. diff --git a/domains/analytics/knowledge/segment-governance.md b/domains/analytics/knowledge/segment-governance.md new file mode 100644 index 00000000..2ebe1e07 --- /dev/null +++ b/domains/analytics/knowledge/segment-governance.md @@ -0,0 +1,40 @@ +--- +name: segment-governance +domain: analytics +description: Segment event governance via segment-schema is advisory — no CI enforcement prevents unregistered events from shipping +--- + +# Segment Event Governance + +## Architecture + +| Component | Location | +|-----------|----------| +| Tracking plan | `Consensys/segment-schema` → `tracking-plans/metamask-extension.yaml` | +| Event registry | `shared/constants/metametrics.ts` → `MetaMetricsEventName` enum (300+ entries) | +| Review process | `CONTRIBUTING.md` in segment-schema; Data Council review | +| Governance channel | `#metamask-metametrics`, `@consensys/data-council` | + +## The Gap + +There is **no CI enforcement** in the extension repo. A developer can: + +1. Add entry to `MetaMetricsEventName` enum +2. Call `trackEvent` with it +3. Merge and ship to production + +...without registering in segment-schema or going through Data Council review. + +## Implications + +- Schema drift between tracking plan and production events +- No property schema validation for unregistered events +- Billing impact goes unreviewed +- Data Council review is bypassable by omission + +## Recommended Fix + +CI check that: +1. Parses `MetaMetricsEventName` entries +2. Validates each against `tracking-plans/metamask-extension.yaml` +3. Fails build if event is missing from the plan diff --git a/domains/analytics/knowledge/span-sub-sampling.md b/domains/analytics/knowledge/span-sub-sampling.md new file mode 100644 index 00000000..fa90f08e --- /dev/null +++ b/domains/analytics/knowledge/span-sub-sampling.md @@ -0,0 +1,72 @@ +--- +name: span-sub-sampling +domain: analytics +description: Deterministic per-trace sub-sampling for high-frequency custom spans — global tracesSampleRate × span sub-rate, traceId-hash bucketed +--- + +# Span Sub-Sampling + +Durable fix for a custom span that fans out and eats the span budget. Layer a per-trace sub-rate **under** the global `tracesSampleRate`, keyed on the trace id so every span in a trace is kept-or-dropped together. Source: [PR #39891](https://github.com/MetaMask/metamask-extension/pull/39891) (`shared/lib/wrapper-sampling.ts`). + +## Rate Math + +``` +effective rate = global tracesSampleRate × span sub-rate +``` + +- Global `tracesSampleRate` is already small (extension prod: 0.75%). +- The sub-rate cuts the custom span on top: `0.75% × 1% = 0.0075%`. +- PR #39891 ships a sub-rate of 0.5% (`WRAPPER_SAMPLE_RATE = 0.005`) — a conservative pilot — and names 5% as the step-up once the denylist is confirmed effective in production. + +Pick the sub-rate from how many sampled traces the metric needs to stay useful — not from the quota alone. Too low and the metric goes dark. + +## Pattern + +```ts +const WRAPPER_SAMPLE_RATE = 0.005; + +// Deterministic: same answer for the same traceId, so all spans in a trace +// are kept or dropped together — clean waterfalls, no partial gaps. +export function shouldSampleWrappers(traceId: string | undefined): boolean { + if (!traceId || traceId.length < 8) { + return false; + } + const hashBucket = parseInt(traceId.slice(0, 8), 16) % 10000; + return hashBucket < WRAPPER_SAMPLE_RATE * 10000; +} +``` + +**Why deterministic, not `Math.random()` per call:** independent per-span sampling shreds a trace into partial waterfalls (some spans present, siblings missing) — useless for attribution. Hashing the trace id makes keep/drop a property of the whole trace. + +## Gate Order (cheapest check first) + +```ts +const traceId = sentryGetActiveSpan()?.spanContext().traceId; +if (!traceId || isReadOnlyAction(action) || !shouldSampleWrappers(traceId)) { + return doWorkWithoutSpan(); +} +return trace({ name, op, data }, doWorkWithSpan); +``` + +1. No active trace → no span. +2. Denylist → skip noise (below). +3. Sub-sample miss → skip this trace's spans. + +## Denylist: cut before you sample + +Drop spans with no timing/attribution signal before sub-sampling. In PR #39891, read-only verbs are ~90% of `messenger.call` volume: + +```ts +const READ_ONLY_VERB = /^(?:get|has|find|is|peek)(?:[A-Z]|$)/u; +``` + +Removing ~90% of volume before the sample multiplies headroom — a higher sub-rate then yields the same span budget, so kept traces are denser and more useful. + +## Where the Gate Goes + +- **Consumer (extension):** spans go through `trace()`. Gate at the call site, or for a whole span family inside the wrapper. `traceId` from `sentryGetActiveSpan()?.spanContext().traceId`. +- **Controller package (core):** controllers call an injected `trace` callback. Gate in the package's trace util or the callback so every consumer inherits the cap. Pull the trace id from the controller's tracing context, not a fresh Sentry import. + +## Kill Switch + +Ship every always-on span family with an env disable flag (PR #39891: `SENTRY_DISTRIBUTED_TRACING_DISABLED` returns the messenger un-wrapped). It turns a future emergency cut into a config flip instead of a cherry-pick. diff --git a/domains/analytics/skills/analytics-instrumentation/repos/metamask-extension.md b/domains/analytics/skills/analytics-instrumentation/repos/metamask-extension.md new file mode 100644 index 00000000..5175f96e --- /dev/null +++ b/domains/analytics/skills/analytics-instrumentation/repos/metamask-extension.md @@ -0,0 +1,61 @@ +--- +repo: metamask-extension +parent: analytics-instrumentation +--- + +## Key Files + +| Content | Path | +|---------|------| +| Sentry trace wrapper | `shared/lib/trace.ts` | +| Trace name enum | `shared/lib/trace.ts` → `TraceName` | +| MetaMetrics controller | `app/scripts/controllers/metametrics-controller.ts` | +| Event enum | `shared/constants/metametrics.ts` → `MetaMetricsEventName` | +| Sentry setup + sample rate | `app/scripts/lib/setupSentry.js` → `getTracesSampleRate()` | +| Segment tracking plan | `Consensys/segment-schema` → `tracking-plans/metamask-extension.yaml` | + +## Cross-Process Context (UI → Background) + +The extension has two Sentry hubs — one in the UI process and one in the background service worker. A trace starting in UI and continuing in background requires explicit context propagation across the RPC boundary: + +```typescript +// Serialize at UI call site +const context: SerializedTraceContext = { + _name: TraceName.MyOperation, + _traceId: span.spanContext().traceId, + _spanId: span.spanContext().spanId, +} + +// Background receives context, creates child span +trace({ name: TraceName.MyOperation, parentContext: context }, async () => { ... }) +``` + +Without propagation: Sentry shows two disconnected operations. With propagation: complete tree from user action to RPC call. + +## Sentry Sample Rate + +```bash +grep -n "tracesSampleRate" app/scripts/lib/setupSentry.js +# Verify current value before calculating — it has changed between releases +``` + +## Sentry Traces Explorer Query (Volume Estimation) + +``` +Environment: production | Time range: 30 days | Mode: aggregate +Query: span.op:http.client span.description:*{endpoint}* +Group by: span.description, transaction +Sort: -count(span.duration) +``` + +## Detect `isOptIn` Misuse + +```bash +grep -rn "isOptIn: true" app/scripts/ ui/ --include="*.ts" --include="*.tsx" +# Any occurrence outside the onboarding opt-in flow is suspect +``` + +## Data Council Contact + +- Slack: `#metamask-metametrics` +- Team: `@consensys/data-council` diff --git a/domains/analytics/skills/analytics-instrumentation/skill.md b/domains/analytics/skills/analytics-instrumentation/skill.md new file mode 100644 index 00000000..8819dcee --- /dev/null +++ b/domains/analytics/skills/analytics-instrumentation/skill.md @@ -0,0 +1,89 @@ +--- +maturity: experimental +name: analytics-instrumentation +description: Create and update Sentry spans, MetaMetrics events, and Segment events — methodology, policies, common pitfalls +--- + +# Analytics Instrumentation + +## When To Use + +- Adding or modifying a MetaMetrics (Segment) event +- Adding or modifying a Sentry performance span +- Estimating event or span volume from production data +- Auditing existing instrumentation for correctness + +--- + +## Do Not Use When + +- Adding local debug logging with no telemetry destination +- Investigating an existing Sentry error report (use `sentry-mcp-queries`) +- Internal feature flag evaluation not surfaced as an analytics event + +--- + +## Sentry Spans + +### Creating a Span + +1. **Register a named trace entry** in the repo's trace name enum before writing any span code. Unnamed spans are invisible in Sentry filters. +2. **Use the repo's `trace()` wrapper**, not raw `Sentry.startSpan()`. Wrappers handle cross-process context propagation, active-span inheritance, and consistent tag injection. +3. **Inherit parent automatically** — when no `parentContext` is provided, the wrapper inherits from `Sentry.getActiveSpan()`, making the new span a child of the active parent (e.g., a `pageload` span). + +### Updating a Span + +- Adding a tag: no governance required +- Renaming a trace name enum entry: grep all callsites; update enum and references atomically +- Changing an `op` value: breaks saved queries and dashboards — coordinate with whoever owns them + +--- + +## MetaMetrics / Segment Events + +### Creating an Event + +1. **Check the event name enum** — event may already exist under a different phrasing. +2. **Check the segment tracking plan** — event may be registered under a different name than the enum key. +3. **Add to the enum**, then implement the `trackEvent` call. +4. **Do NOT use `isOptIn: true` outside the onboarding opt-in flow.** It strips user identity unconditionally for all users, not just non-opted-in ones (see Reference Knowledge: metrametrics-identity). +5. **Open a data governance review** before merging. There is usually no CI enforcement on schema registration — this step is easy to skip (see Reference Knowledge: segment-governance). +6. **Register in the team's segment tracking plan** before shipping. + +### Updating an Event + +- Adding a property: requires governance review and schema update +- Renaming an event: deprecate old + add new in tracking plan; coordinate on migration window +- Removing an event: confirm no active dashboards depend on it before removing + +--- + +## Volume Estimation via Sentry + +When direct Segment access is unavailable, estimate from Sentry production span data: + +1. **Find a correlated HTTP endpoint** — one that fires 1:1 with the event. +2. **Query Sentry Traces Explorer** (aggregate mode): + ``` + span.op:http.client span.description:*{endpoint}* + ``` +3. **Extrapolate:** + ``` + estimated_actual = sampled_count × (1 / tracesSampleRate) + ``` +4. **Interpret as upper bound** — endpoint may have callers outside the event path. + +Caveats: sample population is MetaMetrics opted-in users only; verify the current `tracesSampleRate` before calculating (it changes between releases). + +--- + +## Common Pitfalls + +| Mistake | Correct Approach | +|---------|-----------------| +| `isOptIn: true` on post-onboarding events | Strips user identity for all users; only valid in onboarding flow | +| Ship event without tracking-plan registration | No CI gate — add governance review explicitly to PR checklist | +| Raw `Sentry.startSpan()` instead of the repo's `trace()` wrapper | Use the wrapper — handles cross-process context and active-span inheritance | +| New span with no trace name enum entry | Register enum entry first; unnamed spans are invisible in Sentry filters | +| Multiply sampled count by `tracesSampleRate` | Multiply by inverse: `sampled × (1 / rate)` | +| Treat Sentry estimates as exact counts | Probabilistic sample — state sample size and confidence | diff --git a/domains/analytics/skills/sentry-mcp-queries/repos/metamask-extension.md b/domains/analytics/skills/sentry-mcp-queries/repos/metamask-extension.md new file mode 100644 index 00000000..efe70acd --- /dev/null +++ b/domains/analytics/skills/sentry-mcp-queries/repos/metamask-extension.md @@ -0,0 +1,60 @@ +--- +repo: metamask-extension +parent: sentry-mcp-queries +--- + +## Organization and Projects + +``` +mcp__sentry__find_organizations → confirm org slug +mcp__sentry__find_projects → metamask-extension (Chrome/MV3 + Firefox/MV2) +``` + +## Standard Filter Set for Extension Errors + +``` +environment:production +installType:normal +``` + +Then add `dist:mv3` or `dist:mv2` to isolate by manifest. + +## Sample Rate + +Production `tracesSampleRate` = `0.0075` (0.75%) → multiplier ≈ 133× + +```bash +# Verify current value before using +grep "tracesSampleRate" app/scripts/lib/setupSentry.js +``` + +## Volume Estimation — Worked Example + +`AssetsFirstInitFetchCompleted` correlates 1:1 with `accounts.api.cx.metamask.io/v1/supportedNetworks` (fires once per init) — **not** `/v4/multiaccount/balances` (fires per account): + +``` +/v1/supportedNetworks: 2.6M sampled (30d) × 133 ≈ 346M event fires / month +/v4/multiaccount/balances: ~26M sampled × 133 ≈ 3.5B balance API calls / month (per-account — NOT the event rate) +``` + +Lesson: pick the once-per-event endpoint or you over-count by the fan-out factor. + +## Common Issue Searches + +| What you're looking for | Query | +|---|---| +| Background connection errors | `is:unresolved background connection` | +| MV3-only errors | `is:unresolved dist:mv3` | +| Errors spiking in recent release | `is:unresolved times_seen:>100` | +| Performance issues | `issue.category:performance` | + +## Tag: `dist` Values + +| Value | Meaning | +|-------|---------| +| `mv3` | Chrome (Manifest V3 — service worker) | +| `mv2` | Firefox (Manifest V2 — background page) | + +## Seer Analysis Notes + +Seer has access to the Sentry issue, stack traces, and recent events. It does not have access to the codebase. Validate its hypothesis against the actual handler chain in the source — especially for keepalive, lifecycle, and concurrency conclusions. diff --git a/domains/analytics/skills/sentry-mcp-queries/skill.md b/domains/analytics/skills/sentry-mcp-queries/skill.md new file mode 100644 index 00000000..0ddddbff --- /dev/null +++ b/domains/analytics/skills/sentry-mcp-queries/skill.md @@ -0,0 +1,107 @@ +--- +maturity: experimental +name: sentry-mcp-queries +description: Query Sentry via MCP — error triage, tag distribution, volume estimation, replay retrieval +--- + +# Sentry MCP Queries + +## When To Use + +- Investigating a production error before attributing root cause +- Checking dist (MV3 vs MV2) error distribution +- Estimating event or span volume from production data +- Comparing error rates release-over-release for regression detection +- Retrieving session replay or profiling data + +## Do Not Use When + +- The error reproduces locally with a full stack trace +- Reading product analytics (Segment events, not Sentry errors/spans) +- Pre-merge investigation — Sentry data is post-merge only + +## Setup + +Run once per session: + +``` +mcp__sentry__whoami → confirm auth +mcp__sentry__find_organizations → org slug +mcp__sentry__find_projects → project slug(s) +``` + +All subsequent tools require `organization_slug` and usually `project_slug`. Slug mismatch causes silent empty results. + +## Workflow: Error Triage + +1. `mcp__sentry__search_issues` — find by title, fingerprint, or keyword +2. `mcp__sentry__get_issue_tag_values` — check `dist` distribution **before** attributing root cause +3. If 99%+ one dist → platform lifecycle root cause (see `extension-errors-debugging`) +4. `mcp__sentry__search_issue_events` — individual events for stack trace detail +5. `mcp__sentry__analyze_issue_with_seer` — AI-assisted hypothesis (validate against code) + +## Workflow: Volume Estimation + +Segment event volume is invisible from Sentry, but a correlated `http.client` span is not. Anchor estimation on an HTTP endpoint the event's controller calls **1:1** with the event firing. + +1. Identify the correlated endpoint — the one that fires **once per event**, not per sub-call (e.g. a per-init call, not a per-account call). Picking a per-sub-call endpoint over-counts. +2. `mcp__sentry__search_events` aggregate mode, filter `span.op:http.client` + endpoint +3. Read sampled span count +4. Extrapolate: `estimated = sampled × (1 / tracesSampleRate)` +5. Treat as an **upper bound** — the endpoint may have callers beyond the event path. Sample population = MetaMetrics-opted-in users only (Sentry opt-in is tied to MetaMetrics). Sample rate changes — verify the current value. + +## Workflow: Release Comparison + +Compare error rates or metrics across releases for regression detection: + +1. `mcp__sentry__find_releases` — list releases sorted by date +2. **Filter out unreliable releases** (see below) before comparing +3. `mcp__sentry__search_events` with `release:12.5.0` for baseline +4. `mcp__sentry__search_events` with `release:12.6.0` for comparison +5. **Normalize by sessions or users** — raw counts conflate traffic changes with error rate changes: + ``` + rate = events / sessions_for_that_release + ``` +6. Report delta against baseline with sample-size caveat + +## Filtering Unreliable Releases + +Patch releases have uneven adoption — comparing raw counts against them produces false signal. Skip a release before comparing if: + +| Filter | Threshold | Reason | +|---|---|---| +| Age since publish | < 48–72h | Browser auto-update rollout still ramping (Chrome/Firefox/Edge) | +| Session count | < ~50% of previous stable release | Sample too small for meaningful rates | +| Release stage | `dev`, `canary`, `nightly` | Non-production build — different error profile | +| Environment | not `production` | Development / staging noise | +| Manifest split | compare only within same `dist` | MV3 and MV2 populations have different error distributions | + +**Rule of thumb:** use the newest release that has ≥ 3 days of production adoption **and** session volume comparable to the previous stable release. Everything in between is hotfix noise — skip it for regression comparisons unless investigating that specific patch. + +## Workflow: Replay and Profile + +1. `mcp__sentry__search_issue_events` — find an event ID with replay/profile +2. `mcp__sentry__get_replay_details` / `mcp__sentry__get_profile_details` for that event ID + +## Tag Filters + +| Tag | Values | Use | +|-----|--------|-----| +| `dist` | `mv3`, `mv2` | Isolate by manifest version | +| `environment` | `production`, `staging` | Exclude non-prod noise | +| `installType` | `normal`, `development`, `sideload`, `admin` | Exclude developer-loaded builds | + +**Do not conflate `environment` and `installType`** — a production build can have `installType:development` if loaded unpacked. + +## Common Pitfalls + +| Mistake | Correct Approach | +|---------|-----------------| +| Attribute root cause before checking `dist` distribution | Check tag values first — 99%+ MV3 → lifecycle, not app logic | +| Use raw sampled count as event volume | Multiply by `1 / tracesSampleRate` | +| Filter `environment:development` for dev builds | Filter `installType:normal` — environment ≠ install method | +| Skip `whoami` and guess org slug | Slug mismatch causes silent empty results | +| Treat Seer analysis as ground truth | Use as hypothesis to validate against code/traces | +| Compare raw event counts across releases | Normalize by sessions — traffic changes masquerade as regressions | +| Include a <48h-old release in a regression comparison | Wait for rollout; auto-update adoption takes 2–7 days | +| Treat every patch release as a comparison point | Most patches have low adoption — compare to the last *widely-adopted* release | diff --git a/domains/analytics/skills/sentry-quota/repos/metamask-extension.md b/domains/analytics/skills/sentry-quota/repos/metamask-extension.md new file mode 100644 index 00000000..d3d2b571 --- /dev/null +++ b/domains/analytics/skills/sentry-quota/repos/metamask-extension.md @@ -0,0 +1,50 @@ +--- +repo: metamask-extension +parent: sentry-quota +--- + +## File Paths + +| Path | Role | +|---|---| +| `shared/lib/trace.ts` | `TraceName` / `TraceOperation` enums = the custom-span registry; `trace({ name, op, data }, cb)` API | +| `shared/lib/wrapper-sampling.ts` | `shouldSampleWrappers(traceId)` — the Tier-2 deterministic sub-sample gate | +| `shared/lib/messenger-tracing.ts` | `wrapMessengerWithTracing` + `isReadOnlyAction` read-only denylist (~90% volume cut before sampling) | +| `app/scripts/lib/createMetaRPCHandler.ts` | `rpc.handler` span — gated behind `shouldSampleWrappers` | +| `app/scripts/lib/setupSentry.js` | global `tracesSampleRate` (`0.0075` = 0.75%) | + +Core controller instrumentation lives in the **`MetaMask/core`** monorepo: per-package `TraceName` in `packages//src/**/{constants/traces,utils/trace}.ts` (e.g. `bridge-controller/src/constants/traces.ts`). Controllers don't import Sentry — they call an injected `trace` callback (`traceAsControllerCallback` in the extension). + +## Commands + +```bash +EXT= +CORE= + +# Span registries (the inventory) +rg -n 'enum TraceName' "$EXT/shared/lib/trace.ts" +rg -n -g '**/{traces,trace}.ts' 'enum TraceName' "$CORE/packages" + +# Locate a culprit's emit site +rg -n '|TraceName.' "$EXT" "$CORE/packages" + +# All span creation sites — then read each enclosing scope for loop/poller (fan-out) +rg -n 'trace\(' "$EXT/app" "$EXT/shared" "$CORE/packages//src" + +# Gate present before the span? (absence = always-on) +rg -n 'shouldSample|tracesSampleRate|hashBucket|Math.random' + +# Kill-switch present? +rg -n 'SENTRY_[A-Z_]*DISABLED' "$EXT" "$CORE/packages//src" + +# PR review — added instrumentation lines only +gh pr diff --repo MetaMask/metamask-extension \ + | rg '^\+' | rg 'TraceName|trace\(|shouldSampleWrappers|SENTRY_.*DISABLED|op:' +``` + +## Architectural Notes + +- **Gate location differs by repo.** Extension spans go through `trace()` — gate at the call site or in the wrapper. Core controller spans go through the injected callback — gate in the package's trace util or the callback so every consumer (extension, mobile) inherits the cap. +- **`BackgroundRpc` / `MessengerCall`** (the `TraceName` tail) are the already-gated wrapper spans from [PR #39891](https://github.com/MetaMask/metamask-extension/pull/39891) — the reference implementation of the Tier-2 sub-sample pattern and the `SENTRY_DISTRIBUTED_TRACING_DISABLED` kill-switch. +- **Tier-0 fix path is a core PR + a patch on the extension release branch.** Controller instrumentation originates in `MetaMask/core`; the release branch is where the cherry-pick lands. The sev-1 blocker goes on the in-flight release milestone — e.g. [issue #43211](https://github.com/MetaMask/metamask-extension/issues/43211) ("Assets Controller Sentry Instrumentation exceeding quota"). +- **Spotting the culprit first:** `sentry-mcp-queries` → Volume Estimation (`span.op` aggregate × `1 / tracesSampleRate`) ranks span contributors; this skill takes over once you have the offending span name. diff --git a/domains/analytics/skills/sentry-quota/skill.md b/domains/analytics/skills/sentry-quota/skill.md new file mode 100644 index 00000000..1602ca55 --- /dev/null +++ b/domains/analytics/skills/sentry-quota/skill.md @@ -0,0 +1,78 @@ +--- +maturity: experimental +name: sentry-quota +description: Catch quota-risky Sentry span instrumentation in code and PRs — fan-out × ungated × no-kill-switch — before it blows the span budget +--- + +# Sentry Span Quota Guard + +Find and fix custom Sentry span instrumentation that blows the project span budget. Operates on **code and PRs**, not Sentry dashboards — you spot the culprit in Sentry (`sentry-mcp-queries`), this skill fixes it in code. + +## When To Use + +- A PR adds custom Sentry spans (`trace()` calls / `TraceName` entries) — review it before merge. +- A custom span/transaction dominates span volume in Sentry — locate where it's emitted and fix it. +- Auditing controllers/UI for always-on, fan-out-prone instrumentation. +- A custom span is the top span-count contributor and must be cut fast (release blocker). + +## Do Not Use When + +- Reading the live span counts themselves — that's `sentry-mcp-queries` (Volume Estimation). +- Product-analytics events (Segment / `trackEvent`) — that's `analytics-instrumentation` + `segment-governance`. +- The span is already behind a per-trace sample gate **and** a kill-switch — already mitigated. + +## Breach Triad + +A custom span is a quota risk when these stack. The first three together are the breach profile. + +| Signal | Static signature | Why it blows quota | +|---|---|---| +| **Fan-out** | span created in a loop / `.map` / `.forEach` / per-asset / per-account / per-chain / poller | N spans per trace, not 1 | +| **Always-on** | no `tracesSampleRate` sub-rate, no hash gate before the span | every qualifying call emits | +| **No kill-switch** | not guarded by an env flag | disabling needs a release, not a config flip | +| Hot path | data-source / update-pipeline / network callback, not a discrete user action | high call frequency | + +Low fan-out + discrete user action + already gated = fine. Don't flag healthy spans. + +## Workflow + +### PR review (pre-merge gate) +1. `gh pr diff ` — scan **added** lines for new `TraceName` entries and `trace(` call sites. +2. Score each new span against the breach triad: is the enclosing scope a loop/poller? is there a gate? a kill-switch? +3. Block if a new always-on span has no gate — require a sub-sample gate (`span-sub-sampling`) before merge. Cheaper than a post-ship cherry-pick. +4. If the diff adds no `trace(` sites and no `TraceName` entries → "no new instrumentation, no quota risk", stop. + +### Locate (incident) +1. Grep the span name / `TraceName.X` across the consuming repo **and** the controller package source. +2. Open the call site; read the enclosing scope for the fan-out verdict (loop/poller?). +3. **No grep hits ≠ safe** — the culprit may be on a release ref not checked out. Verify the package version / `gh pr checkout` the shipping ref before concluding clean. + +### Audit +1. Sweep the span registries (`TraceName` enums) + `trace(` call sites. +2. Rank by breach triad — surface ungated × hot-path × fan-out first. + +### Mitigate +Pick the lowest tier that stops the bleed. + +## Mitigation Ladder + +| Tier | When | Action | +|---|---|---| +| **0 — Immediate** | a span fans out and is actively breaching on the live release | disable the `trace()` call at source (or env-guard it) + **cherry-pick to the release branch** + file a sev-1 release blocker on the in-flight release milestone | +| **1 — Release containment** | spike concentrated in an old, already-patched release with lingering users | Sentry **inbound filter** dropping `release:` spans + force-update. The only dashboard action. Filters target a whole release, not one span — don't filter a release you still want data from | +| **2 — Durable** | the span is justified long-term but ungated | deterministic `traceId`-hash sub-sample gate before the span (`span-sub-sampling`) | +| **3 — Wrong tool** | the metric needs full fidelity; sampling loses the signal | move the metric off trace spans — they are the wrong substrate for always-on high-cardinality metrics. Segment is the usual target, but it has its own ungoverned billing gap (`segment-governance`), so it is not a free lunch | + +Tier 0 + 1 stop the bleed now; Tier 2 is the follow-up so the metric returns. + +## Common Pitfalls + +| Mistake | Correct approach | +|---|---| +| Per-call random sampling (`Math.random()` per span) | Deterministic `traceId`-hash bucket — all spans in a trace kept-or-dropped together, clean waterfalls | +| Gate the span in Sentry config | Gate at the call site; for an injected-callback controller span, gate in the callback so every consumer inherits the cap | +| Inbound-filter a release you still need data from | Filters drop the whole release — fix in code (Tier 0/2) instead | +| "No grep hits, so it's safe" | The culprit may be on a release ref not checked out — verify the version/ref | +| Disable the span on `main` only | Cherry-pick to the active release branch — `main` alone leaves the live release breaching | +| Treat "move to Segment" as free | Segment events ship without CI governance or billing review (`segment-governance`) | +| Ship new always-on instrumentation with no kill-switch | Add an env disable flag on day one — turns a future cut into a config flip, not a cherry-pick | From 78d8342b4d0a3cec0c4a15223c36634150793536 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Fri, 5 Jun 2026 14:53:40 -0400 Subject: [PATCH 002/135] Add performance skills: measurement + React/Redux anti-pattern reviews --- CHANGELOG.md | 4 + .../knowledge/effect-anti-patterns.md | 101 +++++++++++ .../knowledge/metrics-pipeline-design.md | 67 +++++++ .../performance/knowledge/render-cascade.md | 68 ++++++++ .../knowledge/selector-anti-patterns.md | 115 ++++++++++++ .../web-vitals-attribution-import.md | 19 ++ .../web-vitals-production-vs-benchmarks.md | 21 +++ .../knowledge/web-vitals-runtime-metrics.md | 22 +++ .../performance/skills/data-analysis/skill.md | 164 ++++++++++++++++++ .../repos/metamask-extension.md | 27 +++ .../repos/metamask-mobile.md | 31 ++++ .../effect-anti-pattern-review/skill.md | 54 ++++++ .../repos/metamask-extension.md | 43 +++++ .../repos/metamask-mobile.md | 35 ++++ .../selector-anti-pattern-review/skill.md | 120 +++++++++++++ 15 files changed, 891 insertions(+) create mode 100644 domains/performance/knowledge/effect-anti-patterns.md create mode 100644 domains/performance/knowledge/metrics-pipeline-design.md create mode 100644 domains/performance/knowledge/render-cascade.md create mode 100644 domains/performance/knowledge/selector-anti-patterns.md create mode 100644 domains/performance/knowledge/web-vitals-attribution-import.md create mode 100644 domains/performance/knowledge/web-vitals-production-vs-benchmarks.md create mode 100644 domains/performance/knowledge/web-vitals-runtime-metrics.md create mode 100644 domains/performance/skills/data-analysis/skill.md create mode 100644 domains/performance/skills/effect-anti-pattern-review/repos/metamask-extension.md create mode 100644 domains/performance/skills/effect-anti-pattern-review/repos/metamask-mobile.md create mode 100644 domains/performance/skills/effect-anti-pattern-review/skill.md create mode 100644 domains/performance/skills/selector-anti-pattern-review/repos/metamask-extension.md create mode 100644 domains/performance/skills/selector-anti-pattern-review/repos/metamask-mobile.md create mode 100644 domains/performance/skills/selector-anti-pattern-review/skill.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 383d49c7..614ec51c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- Add `performance` skills: `data-analysis`, web-vitals + metrics-pipeline knowledge, and `effect-anti-pattern-review` / `selector-anti-pattern-review` (review-time complements to the `perf-*` optimization skills) with `render-cascade` / anti-pattern knowledge + ## [0.1.0] ### Added diff --git a/domains/performance/knowledge/effect-anti-patterns.md b/domains/performance/knowledge/effect-anti-patterns.md new file mode 100644 index 00000000..00a5d9b1 --- /dev/null +++ b/domains/performance/knowledge/effect-anti-patterns.md @@ -0,0 +1,101 @@ +--- +name: effect-anti-patterns +domain: performance +description: Four React `useEffect` patterns that cause unnecessary renders, memory leaks, or race conditions +--- + +# Effect Anti-Patterns + +Four `useEffect` patterns that are systemically broken in React codebases. Each pattern has a broken example, a fixed example, and a detection recipe. + +## 1. `JSON.stringify` in Dependency Array + +`JSON.stringify` produces a new string on every render when the input is an object. React compares dependency arrays by reference for primitives and by identity for objects. A stringified object is a new primitive every render, so the effect fires every render. + +```typescript +// ❌ BROKEN: effect runs on every render +useEffect(() => { + doSomething(config) +}, [JSON.stringify(config)]) + +// ✅ FIXED: destructure and depend on primitives +const { a, b } = config +useEffect(() => { + doSomething({ a, b }) +}, [a, b]) + +// ✅ ALSO FIXED: stabilize via useMemo +const stableConfig = useMemo(() => config, [config.a, config.b]) +useEffect(() => { + doSomething(stableConfig) +}, [stableConfig]) +``` + +Detection: `grep -rnE 'useEffect.*\[.*JSON\.stringify' ` + +## 2. `useEffect` + `setState` (State Mirror Pattern) + +Using an effect to mirror one piece of state into another is almost always wrong. The computed value should be derived inline or via `useMemo`. Mirror-effects trigger an extra render and create synchronization bugs. + +```typescript +// ❌ BROKEN: two renders, possible stale state +const [fullName, setFullName] = useState('') +useEffect(() => { + setFullName(`${first} ${last}`) +}, [first, last]) + +// ✅ FIXED: derived inline, one render +const fullName = `${first} ${last}` + +// ✅ ALSO FIXED: memoized if expensive +const fullName = useMemo(() => expensiveJoin(first, last), [first, last]) +``` + +The React docs explicitly call this out: [You Might Not Need an Effect](https://react.dev/learn/you-might-not-need-an-effect). + +Detection: `grep -rnB1 -A3 'useEffect' | grep -B2 -A1 'set[A-Z]'` (review hits manually) + +## 3. Missing Interval/Timer Cleanup + +Every `setInterval` and `setTimeout` inside an effect must be cleared in the cleanup function. Otherwise the timer survives component unmount and fires on dead state, leaking memory and causing "setState on unmounted component" warnings. + +```typescript +// ❌ BROKEN: timer leaks after unmount +useEffect(() => { + setInterval(poll, 1000) +}, []) + +// ✅ FIXED +useEffect(() => { + const id = setInterval(poll, 1000) + return () => clearInterval(id) +}, []) +``` + +Detection: `grep -rnB2 -A10 'setInterval\|setTimeout' | grep -B5 'useEffect' | grep -v 'clearInterval\|clearTimeout'` + +## 4. Missing `AbortController` in Async Effects + +Async work inside an effect should be cancellable. Without `AbortController`, a request initiated before unmount can resolve after unmount, triggering `setState` on a dead component and masking memory issues. + +```typescript +// ❌ BROKEN: fetch races unmount +useEffect(() => { + fetch(url).then((r) => setData(r)) +}, [url]) + +// ✅ FIXED +useEffect(() => { + const ctrl = new AbortController() + fetch(url, { signal: ctrl.signal }) + .then((r) => setData(r)) + .catch((e) => { if (e.name !== 'AbortError') throw e }) + return () => ctrl.abort() +}, [url]) +``` + +## Why These Matter + +- **Renders.** State-mirror effects double every render in the affected component tree. +- **Memory.** Uncleared intervals and timers leak proportional to how often the component mounts. +- **Correctness.** Async effects without cancellation cause `"Can't perform a React state update on an unmounted component"` warnings and, worse, data races where an older response overwrites a newer one. diff --git a/domains/performance/knowledge/metrics-pipeline-design.md b/domains/performance/knowledge/metrics-pipeline-design.md new file mode 100644 index 00000000..03821ef5 --- /dev/null +++ b/domains/performance/knowledge/metrics-pipeline-design.md @@ -0,0 +1,67 @@ +--- +name: metrics-pipeline-design +domain: performance +description: Four-layer metric pipeline architecture for E2E benchmarks, with domain-specific statistical bounds and split reporting paths. +--- + +# Metrics Pipeline Design + +Architecture for adding metric types to an E2E benchmark suite. Separates collection, running, statistics, and reporting into independent layers. + +## Architecture + +``` +Collector → Runner → Statistics → Reporter +``` + +| Layer | Responsibility | +|-------|----------------| +| **Collector** | Extract raw metric from browser/extension per iteration | +| **Runner** | Per-iteration capture + aggregation orchestration | +| **Statistics** | Domain-specific filtering, outlier detection, percentiles | +| **Reporter** | Per-run spans (for quality gate comparison) + aggregated structured logs (for dashboards) | + +Flow files call the collector and return snapshots alongside timers. No flow file does statistics or reporting. + +## Adding a New Metric Type + +1. **Create collector** — function returning typed snapshot with nullable fields for unobserved metrics +2. **Define types** — per-run snapshot, aggregated (reuse `TimerStatistics` for numeric fields), summary +3. **Add domain-specific bounds** — each numeric field gets `{ min, max, allowZero }` +4. **Wire into runner** — collect alongside timers, call aggregation +5. **Add reporter** — per-run spans with `setMeasurement`, aggregated summary as structured log + +## Domain-Specific Statistical Bounds + +Generic timer bounds (1ms–120s, zero=invalid) silently discard valid data from other domains. + +```typescript +// WRONG: CLS values (0–1) all rejected by min=1ms floor +const result = filterBySanityChecks(clsValues); // → empty array + +// RIGHT: per-metric bounds +const BOUNDS = { + inp: { min: 1, max: 30_000, allowZero: false }, // ms + lcp: { min: 1, max: 60_000, allowZero: false }, // ms + cls: { min: 0, max: 10, allowZero: true }, // unitless ratio +}; +``` + +**Rule:** When adding a new metric type, verify whether existing `filterBySanityChecks` assumptions (ms units, zero=invalid) hold. If not, define metric-specific bounds. + +`allowZero` is the critical distinction: CLS=0 means perfect stability (valid); timer=0ms means measurement error (invalid). + +## Split Reporting Path + +| Data | Mechanism | Rationale | +|------|-----------|-----------| +| Aggregated statistics (mean, p75, p95) | Structured log | Low cardinality, dashboard-friendly | +| Per-run snapshots | Sentry spans + `setMeasurement` | Preserves granularity, enables quality gate comparison via Mann-Whitney U | + +`tracesSampleRate: 1.0` required in CI so all per-run spans are captured. + +## SDK Isolation Pattern + +When CI benchmark scripts run in Node but the extension uses a browser SDK (e.g. `@sentry/node` vs `@sentry/browser`): these never share a process. The package manager resolves separate versions per dependency tree. No compatibility issue — they are fully isolated under different lockfile entries. + +Risk: a shared module accidentally importing from the wrong SDK at bundle time. Mitigation: keep the CI SDK as a devDependency excluded from extension builds. diff --git a/domains/performance/knowledge/render-cascade.md b/domains/performance/knowledge/render-cascade.md new file mode 100644 index 00000000..e872081f --- /dev/null +++ b/domains/performance/knowledge/render-cascade.md @@ -0,0 +1,68 @@ +--- +name: render-cascade +domain: performance +description: React+Redux render cascade failure mode — single state change triggers multiple re-render cycles +--- + +# Render Cascade + +Single state change → broken selector returns new reference → `useSelector` detects "change" → parent re-renders all children → children trigger more selectors → cycle repeats 5+ times before stabilizing. + +## Cost Scaling + +| Factor | Impact | +|--------|--------| +| Component tree depth | Each level multiplies re-renders | +| User data size | O(n) selectors × n items = O(n²) operations | +| State update frequency | Background polling compounds the problem | + +Power users (large datasets, many accounts/tokens/transactions) are disproportionately affected. + +## Root Causes + +| Cause | Pattern | Fix | +|-------|---------|-----| +| Plain function selector | `export function get...` | Wrap in `createSelector` | +| Identity function selector | Transform in input, identity in result | Move transform to result function | +| Unnecessary deep equality | `createDeepEqualSelector` on stable Immer inputs | Use `createSelector` | +| O(n) lookup | `.find()` in selector | Normalize state to map; use direct access | +| Chained transforms (unmemoized) | Multiple `.map`/`.filter` in plain function | Single `createSelector` with all transforms | +| Context provider instability | `` inline | `useMemo` the value | +| Props recreation | `useParams()` passed directly as prop | `useMemo` the props object | + +## Selector Creator Decision Tree + +``` +Is INPUT unstable (not from Immer/Redux)? +├── YES → createDeepEqualSelector +└── NO → Is OUTPUT unstable (new array/object from transform)? + ├── YES → createResultEqualSelector (or createShallowResultSelector) + └── NO → createSelector +``` + +## Fix Order — Root Selectors First + +Selectors form a dependency graph. When a root selector returns an unstable reference, the cost cascades: + +- recomputations: **O(m)** — all m dependent selectors recompute +- cascade depth: **O(log m)** — propagates through the tree +- re-renders: **O(m × k)** — each selector triggers k subscribers + +**Fixing downstream selectors is ineffective until the upstream root is stable** — a fixed `getActiveAccount` still receives a new input every render if `getAccounts` is broken. + +``` +getAccountsObject (stable) + └─ getAccounts (broken: returns new array) + ├─ getActiveAccount ├─ getAccountCount └─ getAccountNames … +``` + +Triage the dependency graph top-down; fix roots first. + +## Why Cascade Breaks All Other Optimizations + +| Optimization | Without Cascade Fix | With Cascade Fix | +|---|---|---| +| Virtualization | Parent still re-renders all | Works as intended | +| `React.memo` | Parent defeats it | Works as intended | +| React Compiler | Can't cross file boundaries | Complements selectors | +| `useMemo`/`useCallback` | Recreated on parent render | Stable references | diff --git a/domains/performance/knowledge/selector-anti-patterns.md b/domains/performance/knowledge/selector-anti-patterns.md new file mode 100644 index 00000000..60bf1473 --- /dev/null +++ b/domains/performance/knowledge/selector-anti-patterns.md @@ -0,0 +1,115 @@ +--- +name: selector-anti-patterns +domain: performance +description: Five Redux selector patterns that that break selector memoization and cause render cascades +--- + +# Selector Anti-Patterns + +Each pattern causes `useSelector` to return a new reference on every call, triggering unnecessary re-renders. + +## The Five Patterns + +### 1. Plain Function Selector + +No memoization. Returns new reference every call. + +```typescript +// ❌ BROKEN +export function getPendingApprovals(state) { + return Object.values(state.metamask.pendingApprovals ?? {}); +} + +// ✅ FIXED +const getPendingApprovalsObject = (state) => state.metamask.pendingApprovals ?? {}; +export const getPendingApprovals = createSelector( + getPendingApprovalsObject, + (approvals) => Object.values(approvals), +); +``` + +Detection: `grep -r "export function get" ui/selectors/` + +### 2. Identity Function Selector + +Transform in input, identity in result → memoization is broken. + +```typescript +// ❌ BROKEN: Object.values() in INPUT creates new array +export const getAccounts = createSelector( + (state) => Object.values(state.accounts), + (accounts) => accounts, // identity — cache never hits +); + +// ✅ FIXED: Stable input, transform in OUTPUT +export const getAccounts = createSelector( + (state) => state.accounts, // stable Immer reference + (accounts) => Object.values(accounts), +); +``` + +Detection: Jest warning `"result function returned its own inputs"` + +### 3. Unnecessary Deep Equality + +`createDeepEqualSelector` adds O(n) overhead when Immer already provides stable references. + +```typescript +// ❌ UNNECESSARY: state.accounts is already stable +const getAccounts = createDeepEqualSelector( + (state) => state.metamask.accounts, + (accounts) => transformAccounts(accounts), +); + +// ✅ CORRECT +const getAccounts = createSelector( + (state) => state.metamask.accounts, + (accounts) => transformAccounts(accounts), +); +``` + +Use `createDeepEqualSelector` only when inputs are genuinely not from Immer/Redux state. + +### 4. O(n) Lookups + +`.find()` on Object.values is O(n). With n items × m selectors per state change = O(n×m). + +```typescript +// ❌ BROKEN +export const getAccountByAddress = (state, address) => + Object.values(state.accounts).find((a) => a.address === address); + +// ✅ FIXED: normalized state, O(1) access +export const getAccountByAddress = (state, address) => state.accounts[address]; +``` + +### 5. Chained Transforms (Unmemoized) + +Each transform creates a new array. Multiple transforms = multiple new references per call. + +```typescript +// ❌ BROKEN: 3 new arrays per call +export function getSortedItems(state) { + const items = Object.values(state.items); // array 1 + const filtered = items.filter(isVisible); // array 2 + return filtered.sort(byDate); // array 3 +} + +// ✅ FIXED: single memoized output +export const getSortedItems = createSelector( + (state) => state.items, + getFilterCriteria, + (items, criteria) => + Object.values(items).filter((i) => matchesCriteria(i, criteria)).sort(byDate), +); +``` + +## Selector Creator Decision Tree + +``` +Is INPUT unstable (not from Immer/Redux)? +├── YES → createDeepEqualSelector +└── NO → Is OUTPUT unstable (new array/object from transform)? + ├── YES → createResultEqualSelector (or createShallowResultSelector) + └── NO → createSelector +``` diff --git a/domains/performance/knowledge/web-vitals-attribution-import.md b/domains/performance/knowledge/web-vitals-attribution-import.md new file mode 100644 index 00000000..98c0067b --- /dev/null +++ b/domains/performance/knowledge/web-vitals-attribution-import.md @@ -0,0 +1,19 @@ +--- +name: web-vitals-attribution-import +domain: performance +description: web-vitals/attribution is a module import path, not a separate package — no meaningful bundle cost, gives the symptom→cause link +--- + +# Web Vitals Attribution Import + +`web-vitals/attribution` is a **module import path**, not a separate package. The attribution build: +- Provides which script/element caused each metric +- Does **not** meaningfully increase production bundle size (tree-shaking applies) + +Don't skip it for "bundle size" reasons — that's a misread. + +## Why it matters +Attribution is the symptom→cause link: +- INP spike of 500ms +- Attribution: `eventTarget: '#confirm-swap-button'`, `eventType: 'click'` +- Combined with tracing → identifies the controller that blocked diff --git a/domains/performance/knowledge/web-vitals-production-vs-benchmarks.md b/domains/performance/knowledge/web-vitals-production-vs-benchmarks.md new file mode 100644 index 00000000..b6bfdf63 --- /dev/null +++ b/domains/performance/knowledge/web-vitals-production-vs-benchmarks.md @@ -0,0 +1,21 @@ +--- +name: web-vitals-production-vs-benchmarks +domain: performance +description: Web Vitals need different collection in production (web-vitals lib) vs benchmarks (PerformanceObserver); no TBT in prod +--- + +# Web Vitals — Production vs Benchmarks + +Collection differs by environment due to timing constraints. + +## Production — `web-vitals` library +- Reports on `visibilitychange` / `pagehide` +- Handles browser quirks, bfcache, session windowing +- Attribution build shows which element/script caused the metric +- Metrics: **INP, LCP, CLS** (not TBT) + +**Why no TBT in production:** TBT is cumulative and unbounded — it grows indefinitely over an open-ended session. INP is per-interaction → meaningful for real users. TBT fits bounded flows (benchmarks), not sessions. + +## Benchmarks — direct `PerformanceObserver` +- Query on demand (not dependent on page hide) +- Fits an existing `collectMetrics()` pattern diff --git a/domains/performance/knowledge/web-vitals-runtime-metrics.md b/domains/performance/knowledge/web-vitals-runtime-metrics.md new file mode 100644 index 00000000..e05d7c55 --- /dev/null +++ b/domains/performance/knowledge/web-vitals-runtime-metrics.md @@ -0,0 +1,22 @@ +--- +name: web-vitals-runtime-metrics +domain: performance +description: Core Web Vitals (INP, TBT) are runtime responsiveness metrics, not just page-load — high-value for extension UX gates +--- + +# Web Vitals as Runtime Metrics + +Core Web Vitals (INP, TBT) measure **runtime responsiveness**, not just page load. For a browser extension this distinction is critical. + +- **Page load is less relevant** — the popup opens fast; there's no traditional navigation. +- **Runtime interactions matter** — every button click, form submit, confirmation. INP and TBT measure responsiveness during interactions → high-value for extension UX quality gates. + +## Orthogonal to distributed tracing + +| | Web Vitals | Distributed Tracing | +|---|---|---| +| Question | "How did the user perceive it?" | "Which controller caused it?" | +| Scope | user perception | operation attribution | +| Granularity | per-interaction aggregate | per-operation breakdown | + +Use both — perception (web vitals) + attribution (tracing) — not one instead of the other. diff --git a/domains/performance/skills/data-analysis/skill.md b/domains/performance/skills/data-analysis/skill.md new file mode 100644 index 00000000..4566045d --- /dev/null +++ b/domains/performance/skills/data-analysis/skill.md @@ -0,0 +1,164 @@ +--- +maturity: experimental +name: data-analysis +description: Structured approach for analyzing metrics, attributing changes, and communicating findings — five phases (collection → filtering → curation → questioning → synthesis), confidence assignment, audience-appropriate artifacts +--- + +# Data Analysis Skill + +Structured approach for analyzing metrics, attributing changes, and communicating findings. + +--- + +## When to Use + +- Performance analysis from production metrics +- Attribution of improvements/regressions to code changes +- Creating executive summaries or stakeholder communications +- Any analysis requiring correlation of changes to measured outcomes + +--- + +## Quick Reference + +### Five Phases + +``` +Collection → Filtering → Curation → Questioning → Synthesis +``` + +| Phase | Key Question | Output | +| ----------- | --------------------------- | ----------------------------------- | +| Collection | What are we measuring? | Baseline, scope, change list | +| Filtering | What's signal vs. noise? | Categorized changes with confidence | +| Curation | What correlates with what? | Attribution table | +| Questioning | Do we KNOW or BELIEVE this? | Validated claims with caveats | +| Synthesis | Who needs to know what? | Audience-appropriate artifacts | + +### Confidence Assignment + +| Level | Use When | +| ---------- | ---------------------------------------------------------------- | +| **High** | Clear mechanism + timing alignment + targets measured population | +| **Medium** | Plausible mechanism but confounded by other changes | +| **Low** | Speculative or enabling-only | + +### Attribution Table Template + +| Change | Evidence | Release | Metric | Confidence | Notes | +| ------------- | ----------- | --------- | ----------------- | ------------ | --------------------- | +| [Description] | [PR/commit] | [version] | [affected metric] | High/Med/Low | [mechanism or caveat] | + +--- + +## Process + +### 1. Collection + +```markdown +**Metrics:** [What are you measuring?] +**Population:** [Who? All users, p75, specific cohort?] +**Period:** [Measurement window - release tags or dates] +**Source:** [APM, logs, synthetic benchmarks?] +**Baseline:** [Starting values with methodology] +``` + +Enumerate ALL changes in scope: + +- Code changes (PRs, commits) +- Config changes +- External factors (traffic, user growth, infrastructure) + +### 2. Filtering + +Categorize each change: + +- **Direct:** Clear causal path to measured metric +- **Indirect:** Enabling infrastructure (value materializes later) +- **Unknown:** In scope but mechanism unclear +- **Noise:** Unlikely to affect measured metrics + +### 3. Curation + +Build attribution table: + +1. Map changes to metric movements by release +2. Note co-landed changes (shared attribution) +3. Flag anomalies (improvement without cause, unexplained regression) +4. Separate measured vs. post-cutoff work + +### 4. Questioning + +Challenge every attribution: + +- [ ] "Do we KNOW this, or do we BELIEVE this?" +- [ ] "What would need to be true for this to be wrong?" +- [ ] "Are there alternative explanations?" + +Document what's missing: + +- [ ] Unexplained improvements +- [ ] Unexplained regressions +- [ ] Work that SHOULD have helped but didn't +- [ ] Metrics you wish you had + +### 5. Synthesis + +Create audience-appropriate artifacts: + +| Artifact | Audience | Focus | +| --------------------- | --------------- | ------------------------------------- | +| Executive Summary | Leadership | Hard data, key wins, team recognition | +| Attribution Catalogue | Engineering | Detailed per-change analysis | +| Methodology Doc | Future analysts | Process, assumptions, data sources | +| Communication Post | Stakeholders | Exciting but honest, caveats visible | + +--- + +## Communication Template + +```markdown +**[Metric]: [Before] → [After] ([Change %])** + +Population: [Who this measures] +Caveat: [Key limitation] +What's NOT included: [Equally interesting gaps] + +Notable contributors: + +- [Change 1] — [mechanism] +- [Change 2] — [mechanism] + +Bottom line: [One sentence impact statement] +``` + +--- + +## Anti-Patterns + +| Don't | Do Instead | +| ---------------------------------------- | ------------------------------------------------ | +| Claim causation from correlation | "Correlates with" or "plausible contributor" | +| Attribute release total to single change | Note multiple changes, unknown isolated impact | +| Bury caveats in footnotes | Caveats are part of the story | +| Use superlatives without data | Let numbers speak | +| Hide uncertainty | Use qualifiers: "likely," "plausible," "unknown" | + +--- + +## Checklist + +Before finalizing: + +- [ ] Measurement methodology documented +- [ ] Baseline values recorded with source +- [ ] All changes in scope enumerated +- [ ] Confidence levels assigned with justification +- [ ] Unexplained anomalies noted +- [ ] Limitations explicitly stated +- [ ] What's NOT included documented +- [ ] Uncertainty reflected in language +- [ ] Links/references for all claims +- [ ] Multiple artifacts for different audiences + +--- diff --git a/domains/performance/skills/effect-anti-pattern-review/repos/metamask-extension.md b/domains/performance/skills/effect-anti-pattern-review/repos/metamask-extension.md new file mode 100644 index 00000000..67d41395 --- /dev/null +++ b/domains/performance/skills/effect-anti-pattern-review/repos/metamask-extension.md @@ -0,0 +1,27 @@ +--- +repo: metamask-extension +parent: effect-anti-pattern-review +--- + +## Paths + +- Component sources: [`ui/`](https://github.com/MetaMask/metamask-extension/tree/develop/ui) +- Shared hooks: [`ui/hooks/`](https://github.com/MetaMask/metamask-extension/tree/develop/ui/hooks) + +## Commands + +```bash +# Pattern 1: JSON.stringify in deps +grep -rnE 'useEffect\([^)]*\[.*JSON\.stringify' ui/ --include="*.ts" --include="*.tsx" + +# Pattern 3: setInterval / setTimeout +grep -rnE 'setInterval|setTimeout' ui/ --include="*.ts" --include="*.tsx" + +# Pattern 4: fetch inside useEffect (manual review required for context) +grep -rn 'fetch(' ui/ --include="*.ts" --include="*.tsx" +``` + +## Reference Docs + +- [Frontend Performance Optimization Guidelines](https://github.com/MetaMask/contributor-docs/pull/159) (contributor-docs PR #159) +- [You Might Not Need an Effect](https://react.dev/learn/you-might-not-need-an-effect) diff --git a/domains/performance/skills/effect-anti-pattern-review/repos/metamask-mobile.md b/domains/performance/skills/effect-anti-pattern-review/repos/metamask-mobile.md new file mode 100644 index 00000000..dde98072 --- /dev/null +++ b/domains/performance/skills/effect-anti-pattern-review/repos/metamask-mobile.md @@ -0,0 +1,31 @@ +--- +repo: metamask-mobile +parent: effect-anti-pattern-review +--- + +## Paths + +- Component sources: [`app/`](https://github.com/MetaMask/metamask-mobile/tree/main/app) +- Shared hooks: [`app/component-library/hooks/`](https://github.com/MetaMask/metamask-mobile/tree/main/app/component-library/hooks) + +## Commands + +```bash +# Pattern 1: JSON.stringify in deps +grep -rnE 'useEffect\([^)]*\[.*JSON\.stringify' app/ --include="*.ts" --include="*.tsx" + +# Pattern 3: setInterval / setTimeout +grep -rnE 'setInterval|setTimeout' app/ --include="*.ts" --include="*.tsx" + +# Pattern 4: fetch inside useEffect (manual review required for context) +grep -rn 'fetch(' app/ --include="*.ts" --include="*.tsx" +``` + +## Differences from Extension + +- Prefer `AbortController` for all new async effects. No shared `useIsMounted` hook exists. +- React Native's `fetch` behaves identically to browser `fetch` for cancellation purposes. + +## Reference Docs + +- [You Might Not Need an Effect](https://react.dev/learn/you-might-not-need-an-effect) diff --git a/domains/performance/skills/effect-anti-pattern-review/skill.md b/domains/performance/skills/effect-anti-pattern-review/skill.md new file mode 100644 index 00000000..97283ccb --- /dev/null +++ b/domains/performance/skills/effect-anti-pattern-review/skill.md @@ -0,0 +1,54 @@ +--- +maturity: experimental +name: effect-anti-pattern-review +description: Review PR diffs that add or modify `useEffect` for the four systemic React effect anti-patterns +--- + +# Effect Anti-Pattern Review + +**Scope:** Pre-merge review of PRs that add or modify `useEffect` calls. The workflow is a grep-driven checklist against the four patterns catalogued in [`effect-anti-patterns`](../../knowledge/effect-anti-patterns.md). + +Applies to both `metamask-extension` and `metamask-mobile`. See overlays for repo-specific paths. + +## When To Use + +- Reviewing a PR that adds or modifies a `useEffect` call +- Reviewing a PR that adds `setInterval`, `setTimeout`, `fetch`, or `addEventListener` inside a component +- Investigating a "Can't perform a React state update on an unmounted component" warning + +## Do Not Use When + +- Reviewing selector or render-cascade issues (use [`selector-anti-pattern-review`](../selector-anti-pattern-review/skill.md)) +- Reviewing non-React code (background scripts, workers, test utilities) +- Reviewing an effect that is intentionally one-shot with no async work or timers (check patterns below anyway, but most do not apply) + +## Workflow + +1. **List changed files with `useEffect`.** `git diff --name-only origin/main...HEAD | xargs grep -l 'useEffect'` +2. **Run the [grep checklist](#grep-checklist)** against the changed files. +3. **For each hit, map to a pattern** in [`effect-anti-patterns`](../../knowledge/effect-anti-patterns.md) and apply the fix from the knowledge file. +4. **Block on pattern 1.** `JSON.stringify` in a dependency array is always broken. Do not merge. +5. **Block on pattern 3 without cleanup.** Any `setInterval` / `setTimeout` without a matching `clearInterval` / `clearTimeout` in the cleanup function is blocking. +6. **Require cancellation for async effects.** Any `fetch` / network call inside `useEffect` must use `AbortController`. + +## Grep Checklist + +| Pattern | Detection | Knowledge ref | +|---|---|---| +| 1. `JSON.stringify` in deps | `grep -rnE 'useEffect.*\[.*JSON\.stringify' ` | [§1](../../knowledge/effect-anti-patterns.md#1-jsonstringify-in-dependency-array) | +| 2. State-mirror effect | Hand review — look for `useEffect` that calls `setX` based on other state/props | [§2](../../knowledge/effect-anti-patterns.md#2-useeffect--setstate-state-mirror-pattern) | +| 3. Missing interval/timer cleanup | `grep -rnE 'setInterval\|setTimeout' ` then check each effect returns a cleanup | [§3](../../knowledge/effect-anti-patterns.md#3-missing-intervaltimer-cleanup) | +| 4. Missing `AbortController` | `grep -rnB2 -A10 'fetch\(' ` within `useEffect` blocks | [§4](../../knowledge/effect-anti-patterns.md#4-missing-abortcontroller-in-async-effects) | + +See the repo overlay for the concrete `` path. + +## Common Pitfalls + +| Mistake | Correct approach | +|---|---| +| Accept `JSON.stringify` in deps because "the effect needs to rerun when X changes" | Destructure to primitives or `useMemo` the object — never stringify | +| Accept a state-mirror effect because "the computation is expensive" | Use `useMemo` for expensive derivations. Effects are for side effects, not state derivation | +| Let `setInterval` ship without cleanup because "the component rarely unmounts" | Cleanup is non-negotiable — unmount frequency doesn't matter, correctness does | +| Treat "can't perform state update on unmounted component" as a cosmetic warning | It is a data race. An old response can overwrite a new one | +| Add a lint rule disable on `react-hooks/exhaustive-deps` | Almost always wrong. Destructure or memoize instead | +| Refactor toward `useEffect` + `setState` because it "feels like state" | You probably do not need an effect. See [You Might Not Need an Effect](https://react.dev/learn/you-might-not-need-an-effect) | diff --git a/domains/performance/skills/selector-anti-pattern-review/repos/metamask-extension.md b/domains/performance/skills/selector-anti-pattern-review/repos/metamask-extension.md new file mode 100644 index 00000000..aafb8e9d --- /dev/null +++ b/domains/performance/skills/selector-anti-pattern-review/repos/metamask-extension.md @@ -0,0 +1,43 @@ +--- +repo: metamask-extension +parent: selector-anti-pattern-review +--- + +## Paths + +- Selector definitions: [`ui/selectors/`](https://github.com/MetaMask/metamask-extension/tree/develop/ui/selectors) +- Selector creators: [`shared/lib/selectors/selector-creators.ts`](https://github.com/MetaMask/metamask-extension/blob/develop/shared/lib/selectors/selector-creators.ts) — source of truth for `createSelector`, `createDeepEqualSelector`, `createResultEqualSelector`, `createShallowResultSelector` +- Controller state shape: [`app/scripts/metamask-controller.js`](https://github.com/MetaMask/metamask-extension/blob/develop/app/scripts/metamask-controller.js) +- Component consumption sites: anywhere under [`ui/`](https://github.com/MetaMask/metamask-extension/tree/develop/ui) that calls `useSelector` + +## Commands + +```bash +# Enable WDYR for post-merge diagnosis +ENABLE_WHY_DID_YOU_RENDER=true yarn start + +# Pre-merge grep checklist +grep -rE 'export function get' ui/selectors/ --include="*.ts" +grep -rn createDeepEqualSelector ui/ --include="*.ts" +grep -rnE 'useSelector\([^,]+,\s*(isEqual|shallowEqual)' ui/ --include="*.ts" --include="*.tsx" +grep -rnE '\.find\(' ui/selectors/ +``` + +## Selector Creators + +`shared/lib/selectors/selector-creators.ts` + +| Creator | Use Case | +|---------|----------| +| `createSelector` | Standard memoization (default) | +| `createDeepEqualSelector` | Genuinely unstable inputs (rare — see [narrow exception](../skill.md#overuse-of-createdeepequalselector)) | +| `createResultEqualSelector` | Unstable outputs requiring deep comparison | +| `createShallowResultSelector` | Unstable outputs, shallow comparison sufficient | + +## Example Fix Methodology + +[PR #37147](https://github.com/MetaMask/metamask-extension/pull/37147) fixed `getInternalAccounts` as the canonical example. Before: `createSelector(selectInternalAccounts, (accounts) => accounts)` (identity function, defeats memoization). After: `createSelector(getInternalAccountsObject, (accounts) => Object.values(accounts))`. Impact: 50+ component re-renders eliminated per state update. + +## Reference + +- [Frontend Performance Optimization Guidelines](https://github.com/MetaMask/contributor-docs/pull/159) (contributor-docs PR #159) diff --git a/domains/performance/skills/selector-anti-pattern-review/repos/metamask-mobile.md b/domains/performance/skills/selector-anti-pattern-review/repos/metamask-mobile.md new file mode 100644 index 00000000..cb28ddab --- /dev/null +++ b/domains/performance/skills/selector-anti-pattern-review/repos/metamask-mobile.md @@ -0,0 +1,35 @@ +--- +repo: metamask-mobile +parent: selector-anti-pattern-review +--- + +## Paths + +- Selector definitions: [`app/selectors/`](https://github.com/MetaMask/metamask-mobile/tree/main/app/selectors) +- Redux store: [`app/store/`](https://github.com/MetaMask/metamask-mobile/tree/main/app/store) +- Engine / state shape: [`app/core/Engine/Engine.ts`](https://github.com/MetaMask/metamask-mobile/blob/main/app/core/Engine/Engine.ts) +- WDYR setup: [`wdyr.js`](https://github.com/MetaMask/metamask-mobile/blob/main/wdyr.js) at repo root +- Component consumption sites: anywhere under [`app/`](https://github.com/MetaMask/metamask-mobile/tree/main/app) that calls `useSelector` + +## Commands + +```bash +# Enable WDYR (env var gate, same as extension) +ENABLE_WHY_DID_YOU_RENDER=true yarn start + +# Pre-merge grep checklist +grep -rE 'export function get' app/selectors/ --include="*.ts" +grep -rn createDeepEqualSelector app/ --include="*.ts" +grep -rnE 'useSelector\([^,]+,\s*(isEqual|shallowEqual)' app/ --include="*.ts" --include="*.tsx" +grep -rnE '\.find\(' app/selectors/ +``` + +## WDYR + +Mobile has `wdyr.js` at the repo root. It is gated on `__DEV__ && process.env.ENABLE_WHY_DID_YOU_RENDER === 'true'` and imported from the entry file. No manual setup required — flip the env var and restart Metro. + +Current configuration (at time of authoring): `trackAllPureComponents: true`, `onlyLogs: true` (Metro/Hermes console doesn't group well). + +## Differences from Extension + +- React Compiler adoption and `"use no memo"` opt-outs are extension-only at this time. diff --git a/domains/performance/skills/selector-anti-pattern-review/skill.md b/domains/performance/skills/selector-anti-pattern-review/skill.md new file mode 100644 index 00000000..31b30be8 --- /dev/null +++ b/domains/performance/skills/selector-anti-pattern-review/skill.md @@ -0,0 +1,120 @@ +--- +maturity: experimental +name: selector-anti-pattern-review +description: Review and diagnose Redux selector anti-patterns that cause render cascades, pre-merge and post-merge +--- + +# Selector Anti-Pattern Review + +**Scope:** Redux selector anti-patterns are the dominant cause of React render cascades in the MetaMask UI. This skill covers both review phases: pre-merge PR review (grep-driven checklist) and post-merge diagnosis (WDYR-driven workflow). Both modes resolve to the same root cause and the same fix set, catalogued in [`selector-anti-patterns`](../../knowledge/selector-anti-patterns.md) and [`render-cascade`](../../knowledge/render-cascade.md). + +Both `metamask-extension` and `metamask-mobile` share the same React + Redux architecture; this skill applies to both (see overlays for repo-specific paths). + +## When To Use + +- **Pre-merge.** Reviewing a PR that touches a `selectors/` directory, adds a `useSelector` call, or modifies a `createSelector` / `createDeepEqualSelector` definition +- **Post-merge.** Re-renders are disproportionate to state change size, performance degrades non-linearly with user data size, or components re-render during idle +- **Triage.** A WDYR counter jumps 5+ times per action, or a React render counter shows unexpected re-renders + +## Do Not Use When + +- Non-selector performance concerns (effects → use `effect-anti-pattern-review`, context providers, virtualization) +- Network-bound slowness (use the Network panel, not WDYR) +- Startup or initial-mount perf (use startup profiling) +- Non-React trees (worker messaging, background script perf) + +## Mode A: Pre-Merge Review (grep-driven) + +1. **List changed selector/consumer files.** `git diff --name-only origin/main...HEAD | grep -E '(selectors|useSelector)'` +2. **Run the [grep checklist](#grep-checklist)** against the changed files. +3. **Match each hit to a pattern** in [`selector-anti-patterns`](../../knowledge/selector-anti-patterns.md) (numbered 1–5) or to one of the [team-specific workarounds](#team-specific-workarounds) below. +4. **Block on Jest warning.** If the PR's test run surfaces `"result function returned its own inputs"`, the PR introduces [Pattern 2](../../knowledge/selector-anti-patterns.md#2-identity-function-selector). Do not merge. +5. **Require a fix, not a justification.** None of the five patterns have a valid use case. See [Pitfalls](#common-pitfalls) for the narrow `createDeepEqualSelector` exception. + +## Mode B: Post-Merge Diagnosis (WDYR-driven) + +1. **Confirm cascade.** Add a render counter to a high-level component. If count jumps 5+ per action, cascade is confirmed. + ```tsx + const [count, increment] = useReducer((n) => n + 1, 0) + useEffect(() => { increment() }) + console.log('Render:', count) + ``` +2. **Enable WDYR.** `ENABLE_WHY_DID_YOU_RENDER=true yarn start` (same env var on extension and mobile). +3. **Identify root component.** The first WDYR log is the cascade origin. Do not fix downstream symptoms first. +4. **Classify via the [WDYR message table](#wdyr-message-interpretation).** If the root cause is a selector, return to [Mode A](#mode-a-pre-merge-review-grep-driven) and apply the fix set. If it is a context value or prop identity issue, see [`render-cascade`](../../knowledge/render-cascade.md). +5. **Verify.** Repeat the action. Confirm the counter stabilizes (e.g. 0→2, not 0→25). Divide raw counts by 2 under React Strict Mode. + +## Grep Checklist + +| Pattern | Detection | Knowledge ref | +|---|---|---| +| 1. Plain function selector | `grep -rE 'export function get' /` | [§1](../../knowledge/selector-anti-patterns.md#1-plain-function-selector) | +| 2. Identity function selector | Jest warning `result function returned its own inputs` | [§2](../../knowledge/selector-anti-patterns.md#2-identity-function-selector) | +| 3. Unnecessary `createDeepEqualSelector` | `grep -rn 'createDeepEqualSelector' /` then verify each input is not from Immer state | [§3](../../knowledge/selector-anti-patterns.md#3-unnecessary-deep-equality) | +| 4. O(n) lookup | `grep -rnE '\.find\(.*=>.*address' /` | [§4](../../knowledge/selector-anti-patterns.md#4-on-lookups) | +| 5. Chained unmemoized transforms | `grep -rnE 'export function get.*\{' / -A5` and check for multiple `.filter/.map/.sort` without memoization | [§5](../../knowledge/selector-anti-patterns.md#5-chained-transforms-unmemoized) | + +See the repo overlay for the concrete `` path. + +## Team-Specific Workarounds + +Two patterns show up beyond the five in the knowledge file. Both are workarounds for broken selectors downstream. The fix is always to fix the selector, never to propagate the workaround. + +### `useSelector(selector, isEqual)` from `react-redux` + +```typescript +// Workaround that hides the real problem +const accounts = useSelector(getAccounts, isEqual) +``` + +- **Detection:** `grep -rnE 'useSelector\([^,]+,\s*(isEqual|shallowEqual)'` +- **Review action:** Find `getAccounts` (or whichever selector). Fix it to return a stable reference. Remove the `isEqual` argument in the same PR. +- **Why it's wrong:** Deep equality at the consumption site adds O(n) per render and leaves every other consumer of the same selector broken. + +### Overuse of `createDeepEqualSelector` + +```typescript +// Unnecessary when input is from Immer-managed Redux state +const getTokens = createDeepEqualSelector( + (state) => state.metamask.tokens, + (tokens) => transformTokens(tokens), +) +``` + +- **Detection:** `grep -rn createDeepEqualSelector /` +- **Review action:** For each instance, check if the inputs come from Redux state. If yes, swap to `createSelector`. Immer already gives stable references. +- **The narrow exception:** Inputs that are genuinely not from Immer/Redux state (e.g. derived from a non-Redux source, or passed in as props). These stay. + +## WDYR Message Interpretation + +For post-merge diagnosis, map the WDYR log message to the root cause: + +| Message | Root Cause | Fix | +|---------|------------|-----| +| `different objects that are equal by value` | Object recreated | `useMemo` (or fix selector that produced it) | +| `different functions with the same name` | Callback recreated | `useCallback` with stable deps | +| `different React elements` | JSX passed as prop | Extract to constant | +| `props object itself changed but values equal` | Parent cascade | Fix parent, not child | +| `[hook useContext result]` | Context value unstable | `useMemo` provider value | + +## Diagnostic Signals + +| Red | Green | +|-----|-------| +| Same component 5+ times in WDYR | Re-render count ≤ expected per action | +| Counter jumps 5+ per action | No WDYR logs during idle | +| Render count scales with data size | Render count stable regardless of data | +| Re-renders during idle | — | + +## Common Pitfalls + +| Mistake | Correct approach | +|---|---| +| Accept `useSelector(sel, isEqual)` because "it works" | The underlying selector is broken; fix it and remove the workaround | +| Approve `createDeepEqualSelector` without checking input source | Trace every input to verify it's not already Immer-stable | +| Treat the five patterns as preferences | They are measurably broken — each generates CI warnings | +| Ask the author to justify rather than fix | None of the patterns have a valid use case except the narrow exception above | +| Review only the selector definition, not consumption sites | Pattern 1 (plain function) hides at the call site | +| Fix downstream components first during post-merge diagnosis | Fix the root-cause selector; downstream fixes become wasted work | +| Add `React.memo` to symptom component | Requires stable parent. Fix the parent (usually a selector) first | +| Divide WDYR counts by 1 | React Strict Mode double-renders. Divide raw counts by 2 | From 98e896390e74f5fadf8d5a2a978b97ae79a83d30 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Fri, 5 Jun 2026 14:54:15 -0400 Subject: [PATCH 003/135] Add platform domain plus testing and utility skills --- CHANGELOG.md | 4 + .../skills/resilient-api-collection/skill.md | 188 ++++++++++++++++++ .../browser-extension-profiling/skill.md | 65 ++++++ .../knowledge/extension-architecture.md | 89 +++++++++ .../platform/knowledge/mv3-service-worker.md | 94 +++++++++ .../repos/metamask-extension.md | 46 +++++ .../extension-errors-debugging/skill.md | 59 ++++++ .../extension-lifecycle-decoupling/skill.md | 55 +++++ .../benchmark-statistical-hygiene.md | 45 +++++ .../testing/skills/benchmark-design/skill.md | 71 +++++++ 10 files changed, 716 insertions(+) create mode 100644 domains/coding/skills/resilient-api-collection/skill.md create mode 100644 domains/performance/skills/browser-extension-profiling/skill.md create mode 100644 domains/platform/knowledge/extension-architecture.md create mode 100644 domains/platform/knowledge/mv3-service-worker.md create mode 100644 domains/platform/skills/extension-errors-debugging/repos/metamask-extension.md create mode 100644 domains/platform/skills/extension-errors-debugging/skill.md create mode 100644 domains/platform/skills/extension-lifecycle-decoupling/skill.md create mode 100644 domains/testing/knowledge/benchmark-statistical-hygiene.md create mode 100644 domains/testing/skills/benchmark-design/skill.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 383d49c7..a4a265e2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- Add `platform` domain (extension errors/lifecycle debugging + architecture knowledge), `testing/benchmark-design`, `performance/browser-extension-profiling`, and `coding/resilient-api-collection` + ## [0.1.0] ### Added diff --git a/domains/coding/skills/resilient-api-collection/skill.md b/domains/coding/skills/resilient-api-collection/skill.md new file mode 100644 index 00000000..6d00ca51 --- /dev/null +++ b/domains/coding/skills/resilient-api-collection/skill.md @@ -0,0 +1,188 @@ +--- +name: resilient-api-collection +description: Build resilient data collection scripts that paginate APIs, handle rate limits, and retry transient errors. Use when writing scrapers, API collectors, data pipelines, or any script that fetches paginated data from external APIs (GitHub GraphQL, REST APIs, etc.). +--- + +# Resilient API Collection Scripts + +## Core Architecture + +Every collection script needs these layers: + +``` +run_query() → single request with retry + error classification +fetch_all_pages() → pagination loop with adaptive page sizing +main() → orchestration, dedup, persistence +``` + +## 1. Error Classification + +Classify errors **before** choosing a recovery strategy. Different errors need different fixes. + +| Error Type | Signal | Recovery | +|---|---|---| +| **Resource/complexity limit** | Query too expensive for server | Reduce page size | +| **Rate limit** (primary) | 429, `X-RateLimit-Remaining: 0` | Wait until reset time | +| **Rate limit** (secondary) | 403 + "secondary rate limit" | Exponential backoff (start 60s) | +| **Transient server error** | 502, 503, 504, stream reset | Retry with exponential backoff | +| **Client error** | 400, 401, 404 | Don't retry — fix the request | + +### CLI tools hide error details + +Tools like `gh`, `curl`, `httpie` surface errors differently than raw HTTP responses: + +- **`gh api graphql`**: "Resource limits exceeded" appears in `stderr` with non-zero exit code, NOT in the JSON response `errors` array. Always check `stderr` first, before checking `returncode`. +- Rate limit info may be in response headers (not visible via CLI) or in error messages. + +```python +# Check stderr BEFORE returncode — some errors are in stderr even on exit 0 +stderr = result.stderr.strip() + +if "Resource limits" in stderr or "resource limit" in stderr.lower(): + return RESOURCE_LIMIT_SIGNAL # caller reduces page size + +if result.returncode == 0: + data = json.loads(result.stdout) + # Also check JSON errors (some APIs put limits here) + if "errors" in data: + msg = data["errors"][0].get("message", "") + if "Resource limits" in msg or "timeout" in msg.lower(): + return RESOURCE_LIMIT_SIGNAL + return data + +# Classify non-zero exit +is_transient = any(s in stderr for s in [ + "502", "503", "504", "429", "rate limit", + "secondary", "stream error", "CANCEL" +]) +``` + +## 2. Retry with Exponential Backoff + +```python +MAX_RETRIES = 5 +INITIAL_BACKOFF = 5 # seconds + +for attempt in range(1, MAX_RETRIES + 1): + result = execute_request(...) + + if success: + return result + if is_resource_limit(error): + return RESOURCE_LIMIT_SIGNAL # don't retry, reduce page size + if not is_transient(error): + return None # permanent failure + if attempt == MAX_RETRIES: + return None # exhausted + + wait = INITIAL_BACKOFF * (2 ** (attempt - 1)) + log(f"Transient error (attempt {attempt}/{MAX_RETRIES}), retrying in {wait}s") + time.sleep(wait) +``` + +Key: resource-limit errors should NOT be retried — the same query will fail identically. Signal the caller to reduce page size instead. + +## 3. Adaptive Page Sizing + +Start conservatively. Halve on resource-limit errors. Set a floor. + +```python +MIN_PAGE_SIZE = 5 +MAX_REDUCTIONS = 4 +page_size = 50 # not 100 — nested sub-selections multiply complexity + +while has_more_pages: + data = run_query(..., page_size=page_size) + + if data == RESOURCE_LIMIT_SIGNAL: + reductions += 1 + if reductions > MAX_REDUCTIONS or page_size <= MIN_PAGE_SIZE: + break # can't go smaller + page_size = max(MIN_PAGE_SIZE, page_size // 2) + time.sleep(10) # cool down before retry + continue # retry same page with smaller size + + # process nodes, advance cursor... + time.sleep(2) # inter-page delay to avoid secondary rate limits +``` + +### Why 50, not 100? + +GraphQL query cost = `nodes × sub-selections`. A query fetching 100 PRs with `reviews(first:50)`, `participants(first:30)`, `commits(first:1)` easily exceeds GitHub's 500K node limit. Starting at 50 avoids most resource-limit errors. + +## 4. Deduplication and Incremental Collection + +Always dedup by natural key before writing. This lets re-runs extend existing data. + +```python +def dedup(existing, new, key_fn): + by_key = {} + for item in existing: + by_key[key_fn(item)] = item + for item in new: + by_key[key_fn(item)] = item # new overwrites old + return list(by_key.values()) + +# On write: +existing = load_json(path) if os.path.exists(path) else [] +final = dedup(existing, new_items, key_fn=lambda x: (x["repo"], x["number"])) +save_json(path, final) +``` + +## 5. Observability + +### Force unbuffered output + +Python buffers stdout when output is captured (subprocess, pipe, file redirect). Progress lines never appear. + +```python +import sys +sys.stdout.reconfigure(line_buffering=True) +# OR run with: python3 -u script.py +``` + +### Log structure for monitoring + +``` +=== repo-name (query-type) === + Page 1: 50 nodes, hasNext=True (size=50) + Page 2: 50 nodes, hasNext=True (size=50) + Resource limit exceeded (page_size=50), signaling page-size reduction + Reducing page size to 25 and retrying page 3 (reduction 1/4) + Page 3: 25 nodes, hasNext=True (size=25) + ... + Total: 430 items, collected 430, 3169 sub-items +``` + +Every log line should include: page number, items returned, whether there are more pages, and current page size. + +## 6. Inter-Page Delays + +GitHub's secondary rate limit triggers on sustained request volume, not individual request cost. Add 2-3s between pages. + +```python +PAGE_DELAY = 2 # seconds + +# After each successful page: +time.sleep(PAGE_DELAY) + +# After a resource-limit reduction: +time.sleep(INITIAL_BACKOFF * 2) # longer cooldown +``` + +## Checklist + +When writing a collection script, verify: + +- [ ] Error classification distinguishes resource-limit from rate-limit from transient +- [ ] Resource-limit errors reduce page size (not retry same query) +- [ ] Transient errors retry with exponential backoff +- [ ] Non-retryable errors fail fast +- [ ] Page size starts at 50 or lower for nested queries +- [ ] Page size has a floor (5-10) and max-reduction cap +- [ ] Inter-page delay prevents secondary rate limits +- [ ] Output is unbuffered (`-u` flag or `reconfigure`) +- [ ] Each log line includes page number, count, hasNext, page size +- [ ] Data is deduped by natural key before writing +- [ ] Re-runs merge with existing data (incremental collection) +- [ ] Collection log records run metadata (timestamps, repos, filters) diff --git a/domains/performance/skills/browser-extension-profiling/skill.md b/domains/performance/skills/browser-extension-profiling/skill.md new file mode 100644 index 00000000..d956dea8 --- /dev/null +++ b/domains/performance/skills/browser-extension-profiling/skill.md @@ -0,0 +1,65 @@ +--- +maturity: experimental +name: browser-extension-profiling +description: Compare browser extension performance between branches using WDYR, React DevTools Profiler, and E2E benchmarks with statistical rigor. +--- + +# Browser Extension Profiling + +Methodology for profiling and comparing extension performance across branches or commits. + +## When To Use + +- Validating that a refactor reduces unnecessary re-renders (needs before/after comparison) +- Establishing baseline metrics for a performance initiative +- Investigating a reported UI slowdown in the extension + +## Do Not Use When + +- Single-run comparisons — statistical significance requires ≥10 runs per scenario +- The change touches only non-render paths (background scripts, network with no UI impact) +- Target behavior is server-side latency, not UI rendering + +## Workflow + +1. **Build both branches** with `yarn build:test` on the same machine and Chrome version + +2. **WDYR profiling** (unnecessary re-render counts) + ```bash + ENABLE_WHY_DID_YOU_RENDER=true yarn start + ``` + Flags to watch: + - `different objects that are equal by value` → object recreation + - `different functions with the same name` → callback recreation + - `props object itself changed but values equal` → parent cascade + +3. **React DevTools Profiler** for flame graphs and commit timings + ```bash + yarn devtools:react + ``` + +4. **E2E benchmarks** for scenario durations + ```bash + yarn test:e2e:benchmark + ``` + +5. **Collect ≥10 runs** per scenario. Discard top/bottom 10%. Report mean, median, stddev, p75, p95. + +6. **Statistical threshold:** Cohen's d > 0.5 for a meaningful difference. + +## Common Pitfalls + +| Mistake | Correct Approach | +|---------|-----------------| +| Running branches on different machines or Chrome versions | Same machine, same Chrome, no other apps running | +| Pooling all runs including noisy late-session ones | Compute per-round stats first; report cleanest signal with explicit round attribution | +| Reporting absolute re-render counts without scenario context | Normalize per-action; cascade fixes show multiplied impact at root | +| Skipping cache and state reset between runs | Clear browser cache, reset extension state for each run | + +## Pre-Profiling Checklist + +- [ ] Both branches built with `yarn build:test` +- [ ] Same machine, same Chrome version +- [ ] No other tabs or applications running +- [ ] WDYR enabled: `ENABLE_WHY_DID_YOU_RENDER=true` +- [ ] Cache and extension state cleared between runs diff --git a/domains/platform/knowledge/extension-architecture.md b/domains/platform/knowledge/extension-architecture.md new file mode 100644 index 00000000..9b87656e --- /dev/null +++ b/domains/platform/knowledge/extension-architecture.md @@ -0,0 +1,89 @@ +--- +name: extension-architecture +domain: platform +description: MetaMask extension — background/UI boundary, state sync, build types, key directories +--- + +# Extension Architecture + +## Background / UI Boundary + +The extension runs two separate JavaScript contexts that cannot share memory. + +| Context | Entry | Access | +|---------|-------|--------| +| Background (Service Worker / background page) | `app/scripts/` | DOM-less; controllers, wallet logic | +| UI (popup/tab) | `ui/` | React + Redux; rendering only | +| Shared | `shared/` | Constants, utilities, type definitions | + +Communication is message-based (Chrome runtime messaging). Code in `app/scripts/` cannot `import` from `ui/` and vice versa. + +## State Sync Flow + +``` +Controller state changes (app/scripts/) + ↓ +metamask-controller.js batches via debounce (200ms) + ↓ +UI receives batched state via sendUpdate + ↓ +Redux dispatches UPDATE_METAMASK_STATE + ↓ +Immer applies patches (structural sharing — unchanged paths keep stable references) + ↓ +useSelector evaluates; components re-render if output changed +``` + +Key file: `app/scripts/metamask-controller.js` — aggregates all controller state. + +## Build Types + +| Build | Command | Background | Security Policy | +|-------|---------|------------|-----------------| +| Development | `yarn start` | Webpack, hot reload | No LavaMoat | +| Production | `yarn dist` | Browserify | LavaMoat enforced | +| Test | `yarn build:test` | Browserify | Partial LavaMoat | + +LavaMoat restricts package capabilities at runtime. After adding/updating dependencies, run `yarn lavamoat:auto` to regenerate policies. + +## Manifest Versions + +| Version | Background | Lifecycle | +|---------|------------|-----------| +| MV3 (Chrome) | Service Worker | Can terminate and restart | +| MV2 (Firefox) | Background Page | Always running | + +Errors concentrated in MV3 (99%+) → root cause is service worker lifecycle, not application logic. + +## Key Directories + +``` +app/scripts/ +├── controllers/ # Feature controllers (one per domain) +├── lib/ # Background utilities +└── metamask-controller.js # Main aggregator; 200ms debounce + +ui/ +├── components/ # Reusable React components +├── pages/ # Page-level components +│ ├── routes/ # routes.component.tsx (high selector count) +│ └── home/ # home.container.js (legacy connect()) +├── ducks/ # Redux slices +├── selectors/ # All selectors +│ ├── selectors.js # Main file (~2500 lines) +│ └── .ts # Feature-specific selectors +└── contexts/ # React Context providers + +shared/ +├── constants/ +├── lib/ +└── modules/ + └── selectors/ + └── selector-creators.ts +``` + +## React Compiler Scope + +Enabled for `ui/components`, `ui/contexts`, `ui/hooks`, `ui/layouts`, `ui/pages`. + +Does NOT cross file boundaries — selector values from `useSelector` require manual `useMemo`. diff --git a/domains/platform/knowledge/mv3-service-worker.md b/domains/platform/knowledge/mv3-service-worker.md new file mode 100644 index 00000000..5acdc7a1 --- /dev/null +++ b/domains/platform/knowledge/mv3-service-worker.md @@ -0,0 +1,94 @@ +--- +name: mv3-service-worker +domain: platform +description: MV3 service worker lifecycle — Chrome background termination model, MetaMask's idle-termination mitigation, and cold-start failure modes +--- + +# MV3 Service Worker Lifecycle + +## MV2 vs MV3 + +| Manifest | Background | Default Lifecycle | Mitigated in MetaMask? | +|----------|------------|-------------------|------------------------| +| MV3 (Chrome) | Service Worker | Idle termination after 30s, hard cap ~5 min | Yes — see Idle Termination Mitigation | +| MV2 (Firefox) | Background Page | Always running | N/A | + +## Idle Termination Mitigation + +`app/scripts/background.js:750-758` runs a 2s `browser.storage.session` write loop. `saveTimestamp` (defined at `background.js:651-655`) writes an ISO timestamp into session storage: + + function saveTimestamp() { + const timestamp = new Date().toISOString(); + browser.storage.session.set({ timestamp }); + } + ... + const SAVE_TIMESTAMP_INTERVAL_MS = 2 * 1000; + saveTimestamp(); + setInterval(saveTimestamp, SAVE_TIMESTAMP_INTERVAL_MS); + +Each `chrome.*` / `browser.*` API call resets the 30s idle timer. At 2s cadence the worker stays alive indefinitely while the extension is active. `storage.session` (not `storage.local`) is deliberate — it is MV3-only, in-memory, and does not accumulate disk writes from a heartbeat. + +| Property | Value | +|---|---| +| API | `browser.storage.session.set` (MV3-only, in-memory) | +| Interval | 2000 ms (`SAVE_TIMESTAMP_INTERVAL_MS`) | +| Gate | `PreferencesController.enableMV3TimestampSave !== false` (default true) | +| Inline comment | `background.js:752` — "This keeps the service worker alive" | +| Pattern origin | De facto community consensus, not officially endorsed by Chrome DevRel | +| Re-verify if | Chromium policy change on idle-timer API interactions | + +Ongoing idle termination is **not** a live failure mode while the extension is running. Cold starts (browser launch, extension enable/reload, crash recovery) are the actual source of MV3-concentrated failures. + +## Verification Discipline + +Before attributing an MV3-concentrated error to "idle termination pressure": + +1. Verify `background.js:750-758` keepalive loop still exists and `saveTimestamp` still calls a `chrome.*` / `browser.*` API +2. Verify `enableMV3TimestampSave` is not disabled in affected Sentry events +3. Check whether error timing correlates with cold-start events, not idle periods + +If any check out, the working hypothesis is cold-start cascade race, not ongoing termination. + +## Error Concentration Signal + +| Distribution | Conclusion | +|---|---| +| ~50/50 MV3/MV2 | Application bug (affects both contexts equally) | +| 99%+ MV3 only | MV3 service worker lifecycle — check cold-start cascade before assuming idle termination | +| 99%+ MV2 only | Firefox-specific browser behavior | + +## Sentry Tag Dimensions + +Independent — do not conflate. + +| Tag | Meaning | +|-----|---------| +| `environment` | Build configuration (production, staging, development) | +| `installType` | How extension was loaded (normal, development, sideload, admin) | +| `dist` | Manifest version (mv3, mv2) | + +A production build can have `installType: development` if loaded unpacked. Filter carefully. + +## MV3-Specific Failure Modes + +| Failure | Cause | Mitigated? | +|---------|-------|------------| +| Cold-start cascade race (`APP_INIT_ALIVE` sent before UI listener bound) | `app-init.js` → dynamic-import `background.js` → listener registration races against an open port | No | +| `Background connection unresponsive` via ongoing idle termination | Worker idle-killed mid-session | Yes — 2s keepalive loop | +| `Background connection unresponsive` via cold-start latency | Cold start on browser launch + first-flush latency before `startUiSync` | No — keepalive does not apply before worker exists | +| Silent `postMessage` failure | Port disconnected during wake/termination, try/catch swallows error | No | +| In-memory state lost on cold start | New worker instance has empty in-memory state | No (fresh persistence read required) | + +## Sentry Diagnostic Instrumentation + +| Tag | Purpose | Status | +|-----|---------|--------| +| `uiStartup.receivedAppInitPing` | Distinguishes cold-start cascade race cases; `false` + `ALIVE` received ⇒ `APP_INIT_ALIVE` lost on cold start | Missing on `Background connection unresponsive` path as of 13.26.0 — instrumentation gap, being fixed | +| Phase-specific critical error types (`BACKGROUND_INITIALIZED`, `START_UI_SYNC`) | Distinguishes which startup phase hung | Added by 3-phase startup watchdog (PR #40306) | + +## When to Investigate MV3 Separately + +- Error volume is 10× higher in Chrome than Firefox +- Error involves background connectivity, keepalive, or startup handshake +- Error disappears when running with the worker kept alive manually +- Error correlates with browser-launch or extension-reload timestamps, not idle gaps diff --git a/domains/platform/skills/extension-errors-debugging/repos/metamask-extension.md b/domains/platform/skills/extension-errors-debugging/repos/metamask-extension.md new file mode 100644 index 00000000..5020917c --- /dev/null +++ b/domains/platform/skills/extension-errors-debugging/repos/metamask-extension.md @@ -0,0 +1,46 @@ +--- +repo: metamask-extension +parent: extension-errors-debugging +--- + +## Sentry Filters + +Filter by `dist` tag to isolate manifest version: +- `dist:mv3` — Chrome builds +- `dist:mv2` — Firefox builds + +Filter by `installType` to exclude developer-loaded builds: +- `installType:normal` — store-installed +- `installType:development` — sideloaded (unpacked); includes production builds loaded via developer mode + +## Build Commands + +```bash +# MV3 development (Chrome, service worker) +yarn start + +# MV2 development (Firefox, background page) +yarn start:mv2 + +# Production build (both manifests) +yarn dist + +# After dependency changes — regenerate LavaMoat policies +yarn lavamoat:auto +``` + +## Background Keepalive + +| Property | Value | +|---|---| +| Location | `app/scripts/background.js:750-758` | +| Function | `saveTimestamp` at `background.js:651-655` calls `browser.storage.session.set({ timestamp })` | +| Cadence | 2000 ms via `setInterval` | +| Effect | Each call resets Chrome's 30s SW idle timer — prevents idle eviction during active sessions | +| Gate | `PreferencesController.enableMV3TimestampSave !== false` | + +Active-session keepalive failures are rare and should be investigated as code bugs, not platform behavior. Cold-start cascade and first-flush latency are the actual MV3-concentrated failure modes — see `mv3-service-worker` knowledge for mechanism, failure modes table, and verification discipline. + +## Controller-Messenger Pattern + +Controllers communicate via `ControllerMessenger` (`@metamask/base-controller`). A controller's public API is its registered actions and events — not direct method calls. Cross-controller calls that bypass the messenger will not work across the background/UI boundary. diff --git a/domains/platform/skills/extension-errors-debugging/skill.md b/domains/platform/skills/extension-errors-debugging/skill.md new file mode 100644 index 00000000..885f59f8 --- /dev/null +++ b/domains/platform/skills/extension-errors-debugging/skill.md @@ -0,0 +1,59 @@ +--- +maturity: experimental +name: extension-errors-debugging +description: Diagnose browser extension errors — MV3 vs MV2, background/UI context, error tagging +--- + +# Extension Errors Debugging + +## When To Use + +- Errors appear in one manifest version but not the other +- Background connection or keepalive failures +- Errors that are hard to reproduce in development (only manifest in prod) +- Diagnosing Sentry errors before attributing root cause + +## Do Not Use When + +- Local development errors with full stack traces and reliable repro +- Build/compile errors (TypeScript, ESLint, bundler) +- Test failures unrelated to extension runtime behavior + +## Workflow + +1. **Check distribution** — Filter by `dist` tag. Is the error 99%+ MV3, MV2, or split? +2. **Classify root cause** — MV3-only → service worker lifecycle (specifically cold-start cascade; ongoing idle termination is mitigated — see `mv3-service-worker` knowledge). Split → application logic. MV2-only → Firefox behavior. +3. **Identify context** — Is the error from background (`app/scripts/`) or UI (`ui/`)? Stack trace file paths reveal this. +4. **Check error tags** — Verify `environment`, `installType`, and `dist` are what you expect (these are independent dimensions). +5. **Reproduce** — Use `dist` tag filter to reproduce in the right manifest version. + +## Context Identification from Stack Traces + +| Path prefix in trace | Context | +|---------------------|---------| +| `app/scripts/controllers/` | Background controller | +| `app/scripts/metamask-controller.js` | Background aggregator | +| `ui/components/` or `ui/pages/` | UI (React) | +| `shared/` | Either — shared module | + +## Background-Specific Error Types + +| Error | MV3 Root Cause | Mitigated? | +|-------|---------------|------------| +| Background connection unresponsive (cold-start cascade) | `app-init.js` → `background.js` listener race on worker cold start | No | +| Background connection unresponsive (first-flush latency) | Cold start + background state aggregation before `startUiSync` | No | +| Background connection unresponsive (idle termination) | Worker idle-killed mid-session | Yes — 2s `browser.storage.session` keepalive | +| Port disconnected (wake/termination race) | Port closed during worker lifecycle transition; silent via try/catch | No | +| Keepalive timer missed (active session) | Would imply `browser.storage.session.set` interval failed — rare; investigate as application bug, not platform behavior | N/A | +| In-memory state lost (cold start) | New worker instance re-reads persisted state | No | + +## Common Pitfalls + +| Mistake | Correct Approach | +|---------|-----------------| +| Attribute 99% MV3 error to application code | Check if error requires running background; MV3 SW lifecycle is the likely root cause | +| Default to "SW was terminated mid-session" for MV3 errors | Ongoing idle termination is mitigated by the 2s `browser.storage.session` keepalive. The likely mechanism is cold-start cascade or first-flush latency — see `mv3-service-worker` knowledge | +| "Keepalive timer missed" ⇒ SW slept | The 2s keepalive prevents idle sleep while active. A missed keepalive during active session is a code bug, not platform behavior | +| Use `environment` to filter for dev builds | Use `installType: development` — a prod build can be sideloaded | +| Conflate `dist` and `environment` | They are independent; filter both when needed | +| Reproduce MV2-only error in Chrome | Use Firefox; `installType` doesn't replicate MV3/MV2 lifecycle difference | diff --git a/domains/platform/skills/extension-lifecycle-decoupling/skill.md b/domains/platform/skills/extension-lifecycle-decoupling/skill.md new file mode 100644 index 00000000..ef908aed --- /dev/null +++ b/domains/platform/skills/extension-lifecycle-decoupling/skill.md @@ -0,0 +1,55 @@ +--- +maturity: experimental +name: extension-lifecycle-decoupling +description: Verify platform lifecycle events before assuming they cause application-level side effects +--- + +# Extension Lifecycle Decoupling + +## When To Use + +- Estimating event frequency based on service worker eviction +- Debugging behavior that "should" trigger on lock/unlock but doesn't +- Investigating keepalive, timer, or state persistence behavior + +## Do Not Use When + +- Working on UI-only code with no background process interaction +- The behavior reproduces reliably in development without service worker eviction + +## Core Distinction + +| Layer | Examples | Characteristics | +|-------|---------|----------------| +| Platform lifecycle | SW eviction, page unload | Infrastructure-level | +| Application lifecycle | Lock, unlock, init | User-level | + +These layers are often **decoupled**. The mapping between them is an implementation detail — verify it, don't assume it. + +## Verification Checklist + +Before claiming a platform lifecycle event causes application behavior: + +1. Is there an explicit handler (`onSuspend`, `beforeunload`) that triggers the claimed effect? +2. Is there a keepalive mechanism preventing the lifecycle event? +3. Does relevant state persist across restarts (`chrome.storage.session`, IndexedDB)? +4. Are timers alarm-based (persist across SW restart) or `setTimeout`-based (don't)? +5. Is the guard/flag reset by the lifecycle event or by a separate application event? + +## MV3 MetaMask Specifics + +| Assumption | Reality | +|------------|---------| +| SW eviction triggers lock | No `onSuspend` lock handler — SW eviction does NOT trigger lock | +| Timers lost on SW restart | Auto-lock uses Chrome Alarms API — persists across SW restarts | +| State lost on SW restart | Wallet state persists in `chrome.storage.session` and IndexedDB | +| SW evicts frequently during active use | `background.js:750-758` calls `browser.storage.session.set` every 2s. Each `chrome.*`/`browser.*` call resets the 30s idle timer, so active-session eviction is effectively prevented. Cold starts (browser launch, extension reload) still happen. See `mv3-service-worker` knowledge for mechanism and verification discipline | + +## Common Pitfalls + +| Mistake | Correct Approach | +|---------|-----------------| +| "SW evicts N times/day → event fires N times/day" | Check if application code has handler for eviction | +| Assume frequency from platform behavior | Grep for actual handler chains in `background.js`, `app-state-controller.ts` | +| Conflate platform restart with application reset | Check which state is persisted vs re-initialized | +| "Keepalive uses `chrome.alarms`" | It does not — keepalive uses `browser.storage.session.set` at 2s cadence. `chrome.alarms` is used separately for auto-lock timers that must persist across SW restart | diff --git a/domains/testing/knowledge/benchmark-statistical-hygiene.md b/domains/testing/knowledge/benchmark-statistical-hygiene.md new file mode 100644 index 00000000..7015bbc5 --- /dev/null +++ b/domains/testing/knowledge/benchmark-statistical-hygiene.md @@ -0,0 +1,45 @@ +--- +name: benchmark-statistical-hygiene +domain: testing +description: Three patterns for defensible A/B benchmark results: per-round best-subset reporting, fix-vector isolation, and artifact sort-order trap. +--- + +# Benchmark Statistical Hygiene + +Three patterns that prevent the most common classes of invalid benchmark conclusions. + +## Pattern: Per-Round Best-Subset Reporting + +Later benchmark rounds accumulate system noise (background load, memory pressure, I/O contention). Pooling all rounds blindly treats noisy late-session data equally with clean early-session data. + +**Instead:** Compute per-round statistics first, then report the cleanest signal per metric with explicit round attribution. + +``` +Round 1 (clean): metric X → treatment wins, p=0.04, d=-1.7 +Round 2 (moderate): metric X → treatment wins, p=0.08, d=-0.9 +Round 3 (noisy): metric X → no effect, p=0.90, d=+0.04 + +Pooled (all): metric X → no effect, p=0.50, d=-0.2 ← signal destroyed + +Correct report: "X improved 49% (Round 1, n=5, p=0.04, d=-1.7). + Pooled n=20 loses significance due to Round 3 outliers." +``` + +A small N with large effect size (|d| > 1.5, p < 0.05) is more defensible than a large N where noise has diluted significance to nothing. + +## Pattern: Isolate the Fix Vector + +Design each benchmark flow to exercise the optimization's specific input vector as its primary signal source. Incidental coverage produces fragile results where signal-to-noise depends on how much of the measured duration is optimization-affected. + +| | Weak | Strong | +|-|------|--------| +| Design | End-to-end flow that incidentally triggers target once among many other operations | Rapid sequence of actions each triggering the target with minimal other overhead | +| Optimization signal | ~5% of measured duration | ~80% of measured duration | + +## Pattern: Artifact Sort-Order Trap + +Unpadded iteration numbers in filenames break lexicographic sorting: `iteration-1, iteration-10, iteration-2, ...` interleaves data from different rounds when processed in glob order. + +**Rule:** When processing sequentially-numbered artifacts, extract the embedded timestamp or numeric value for sorting. Never rely on string sort order when numbers cross digit boundaries. + +**Diagnosis:** If pipeline results look implausible (p-values that are too perfect, round-level stats that don't match spot checks), print the actual file ordering the pipeline used. Check for lexicographic interleaving at digit boundaries. Re-sort by extracted timestamp or zero-padded key. diff --git a/domains/testing/skills/benchmark-design/skill.md b/domains/testing/skills/benchmark-design/skill.md new file mode 100644 index 00000000..578691df --- /dev/null +++ b/domains/testing/skills/benchmark-design/skill.md @@ -0,0 +1,71 @@ +--- +maturity: experimental +name: benchmark-design +description: Design, run, and analyze E2E performance benchmarks — session hygiene, per-round reporting, artifact grouping +--- + +# Benchmark Design + +## When To Use + +- Writing a new E2E benchmark flow +- Interpreting or presenting benchmark results +- Adding new metrics to existing benchmarks +- Diagnosing unexpected benchmark results + +## Do Not Use When + +- Adding unit, integration, or correctness E2E tests +- Profiling a single user-reported slowdown (use `selector-anti-pattern-review`) +- Writing micro-benchmarks outside the E2E harness + +## Workflow + +1. **Design the flow** — target ONE optimization vector per benchmark. Maximize ratio of optimization-affected time to total measured time. +2. **Run reference benchmarks first** in any session — session state degrades over time. +3. **Compute per-round statistics** before pooling. Check each round for stability (CV < 0.3 is a reasonable threshold). +4. **Group artifacts by timestamp**, not filename sort order. +5. **Report per-metric best subset** with explicit round attribution. Show pooled data as supplementary. + +## Flow Design by Optimization Type + +| Optimization | Primary cascade vector | Recommended flow | +|---|---|---| +| Selector memoization | State mutations | Multi-confirmation queue | +| Context memoization | Any state update | Account switching cycle | +| HOC stabilization | Route changes | Rapid route cycling (8+ transitions) | +| Dead code removal | Navigation | Return-to-home timer | + +## Session Hygiene + +System state degrades over long sessions — background load and memory pressure inflate variance and can **invert** treatment effects. + +- Run reference/critical benchmarks first +- If a late round contradicts clean earlier rounds, suspect session degradation before re-running the full suite + +## Artifact Grouping + +Filenames use `{test}-iteration-{N}-{ISO-timestamp}.json`. Unpadded N produces incorrect lexicographic sort. + +```javascript +// Extract seconds-since-midnight for round assignment +const match = filename.match(/T(\d{2})-(\d{2})-(\d{2})/); +const secondsOfDay = +match[1] * 3600 + +match[2] * 60 + +match[3]; +// Group by time range — never by filename position or array index +``` + +## Adding Metrics + +Extend `collectMetrics()` in `test/e2e/webdriver/driver.js` and register the metric key in `test/e2e/benchmarks/utils/constants.ts` → `ALL_METRICS`. + +- **Performance API metrics** (paint, navigation timing): collect directly inside `collectMetrics()` via `window.performance.getEntriesByType(...)`. +- **Long Task / TBT metrics**: already wired — `collectMetrics()` reads `window.stateHooks.getLongTaskMetricsWithTBT()`. Adding new long-task-derived metrics requires extending the `stateHooks` observer, not the driver. + +## Common Pitfalls + +| Mistake | Correct Approach | +|---------|-----------------| +| Pool all rounds before checking per-round stats | Per-round first — late-session noise can invert the treatment effect | +| Sort artifacts by filename | Extract ISO timestamp; sort by numeric time value | +| Benchmark flow that exercises multiple vectors | One vector per flow — mixed flows produce ambiguous signal | +| Report pooled p-value as primary result | Report cleanest per-metric signal with round attribution; pooled is supplementary | From 4e97c4b91897826a26f27ea2e54d800627e0de42 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Mon, 22 Jun 2026 15:48:46 -0400 Subject: [PATCH 004/135] Add 30D+ query fidelity guidance and performance-attribution skill sentry-mcp-queries: document data-fidelity loss on older releases for longer-range (30D+) queries (sample-rate drift, extrapolation hiding thin samples, retention downsampling) and percentile (p75+) sample-size/quality filtering; add table rows + pitfalls for stored-span-count and superseded-patch releases. performance-attribution: new skill for attributing release-over-release p75/p95 movements to code changes via black-box diff analysis, with an extension repo file (Trace Explorer queries, key transactions, highest-sample-patch version selection, 90d-vs-30d empirics, hot-path files, core-package changelog analysis, worked v13.11->v13.15 catalogue). analytics-instrumentation: cross-link volume-estimation caveats to the new fidelity guidance. CHANGELOG updated. --- CHANGELOG.md | 2 +- .../skills/analytics-instrumentation/skill.md | 2 +- .../repos/metamask-extension.md | 96 +++++++++++++++++++ .../skills/performance-attribution/skill.md | 90 +++++++++++++++++ .../skills/sentry-mcp-queries/skill.md | 22 +++++ 5 files changed, 210 insertions(+), 2 deletions(-) create mode 100644 domains/analytics/skills/performance-attribution/repos/metamask-extension.md create mode 100644 domains/analytics/skills/performance-attribution/skill.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 89ff9e2b..ebc02533 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- Add `analytics` domain: Sentry span-quota guard, Sentry MCP query workflows, instrumentation methodology, and supporting knowledge +- Add `analytics` domain: Sentry span-quota guard, Sentry MCP query workflows (including longer-range/30D+ query fidelity and percentile-sample-size filtering), release-over-release performance attribution, instrumentation methodology, and supporting knowledge ## [0.1.0] diff --git a/domains/analytics/skills/analytics-instrumentation/skill.md b/domains/analytics/skills/analytics-instrumentation/skill.md index 8819dcee..eab3328f 100644 --- a/domains/analytics/skills/analytics-instrumentation/skill.md +++ b/domains/analytics/skills/analytics-instrumentation/skill.md @@ -73,7 +73,7 @@ When direct Segment access is unavailable, estimate from Sentry production span ``` 4. **Interpret as upper bound** — endpoint may have callers outside the event path. -Caveats: sample population is MetaMetrics opted-in users only; verify the current `tracesSampleRate` before calculating (it changes between releases). +Caveats: sample population is MetaMetrics opted-in users only; verify the current `tracesSampleRate` before calculating (it changes between releases). For longer-range (30D+) or release-over-release queries, the sampled count is **not** comparable at face value — older releases are downsampled / retention-truncated and `.0` releases are sample-thin; see `sentry-mcp-queries` (Longer-Range Queries and Percentile Fidelity) and the `performance-attribution` skill. --- diff --git a/domains/analytics/skills/performance-attribution/repos/metamask-extension.md b/domains/analytics/skills/performance-attribution/repos/metamask-extension.md new file mode 100644 index 00000000..4d0ab9ef --- /dev/null +++ b/domains/analytics/skills/performance-attribution/repos/metamask-extension.md @@ -0,0 +1,96 @@ +--- +repo: metamask-extension +parent: performance-attribution +--- + +## Source & Project + +Primary source is Sentry **Trace Explorer** (not Dashboard 219877): + +- Project `metamask` (ID `273505`), `environment:production` +- Mode `Aggregates`, **Group By** `release`, **Visualize** `p75(span.duration)` and `p95(span.duration)` +- Time `90d` (primary). Dashboard 219877 (30d) is legacy/context only. + +## Key Transactions + +| Transaction | What it measures | +|---|---| +| `UI Startup` | Extension click → interactive UI | +| `/home.html` | Home page render | +| `Asset Details` | Token/NFT detail view render | +| `/notification.html` | dApp confirmation popup (approvals/signatures) — high-frequency for power users, compounds with usage | + +## Query Template + +``` +is_transaction:true environment:production transaction:"UI Startup" (release:metamask-extension@13.11.2 OR release:metamask-extension@13.12.2 OR release:metamask-extension@13.13.1 OR release:metamask-extension@13.14.2 OR release:metamask-extension@13.15.0) +``` + +Swap the `transaction:"…"` value per metric; keep `statsPeriod=90d`. + +## Version Selection — Highest-Sample Patch Per Minor + +Anchor each minor line on its highest-sample patch, never the `.0`: + +| Minor | Patch used | Rationale | +|---|---|---| +| 13.11 | 13.11.2 | Highest sample count | +| 13.12 | 13.12.2 | Highest sample count | +| 13.13 | 13.13.1 | Highest sample count | +| 13.14 | 13.14.2 | Highest sample count | +| 13.15 | 13.15.0 | Current release | + +`.0` releases have **10–100× fewer samples** — never anchor a percentile on a `.0` when a higher patch exists in the same minor line. + +## 90d vs 30d — Empirical + +30d baselines ran **~2× higher** than 90d for the same metric (e.g. UI Startup p75 `9.39s → 3.47s` at 30d vs `4.40s → 3.34s` at 90d). Cause unconfirmed — residual-user population and/or sampling of residual traffic; **not** confirmed "power users" (no cohort segmentation). Report 90d; cite 30d only for context. Note: Sentry share links may render 30d in the UI even when the report figure is 90d — verify `statsPeriod=90d`. + +## Hot-Path Files + +| Path | Why it matters | +|---|---| +| `babel.config.js` | Build-time transforms (e.g. React Compiler) — broad scope | +| `ui/selectors/*.js` | Redux selectors — run on every state change | +| `ui/hooks/*.ts` | Hooks — component lifecycle | +| `ui/components/` | Virtualization / render patterns | +| `package.json` | Dependency runtime behavior + core-package bumps | + +## Analysis Commands + +```bash +git log v13.X.X..v13.Y.Y --oneline --no-merges | wc -l # commit count between releases +git diff v13.X.X..v13.Y.Y --stat -- ui/selectors babel.config.js # file-level change summary +git diff v13.X.X..v13.Y.Y -- # detailed diff for one file +git log v13.X.X..v13.Y.Y --oneline -- # commits touching specific paths +``` + +## Core Packages to Monitor + +App-repo diffs miss work shipped as version bumps. Diff `package.json`, then read each package CHANGELOG: + +| Package | Performance relevance | +|---|---| +| `@metamask/assets-controllers` | Token detection, balance fetching, NFT metadata | +| `@metamask/transaction-controller` | Transaction state size, history storage | +| `@metamask/network-controller` | RPC call handling, retry logic | + +```bash +git diff v13.X.X..v13.Y.Y -- package.json | grep -E "@metamask/(assets-controllers|transaction-controller|network-controller)" +``` + +Example findings: + +- `@metamask/transaction-controller` v62.8.0 — deprecated `history` / `sendFlowHistory` from `TransactionMeta` → significant state-size reduction for power users (consumed in extension [#38665](https://github.com/MetaMask/metamask-extension/pull/38665)). +- `@metamask/assets-controllers` v94.0.0 ([core #7408](https://github.com/MetaMask/core/pull/7408)) — Account API v2 → v4 for token detection → fewer RPC calls, delegated detection. + +## Worked Example: v13.11 → v13.15 (90d) + +| Metric | p75 (typical) | p95 (tail) | +|---|---|---| +| UI Startup | 4.40s → 3.34s (-24%) | 15.65s → 9.11s (**-42%**, -6.5s) | +| /home.html | 1.69s → 1.19s (-30%) | 4.96s → 3.24s (-35%) | +| Asset Details | 100ms → 47ms (**-53%**) | 287ms → 94ms (**-67%**) | +| /notification.html | 1.36s → 1.05s (-23%) | 4.30s → 4.71s (+9%, **high variance — inconclusive**) | + +Most UI Startup and /home.html gains landed in 13.12 (p95 UI Startup -40% in one release); Asset Details improved across 13.14 → 13.15. Treat the per-release header deltas as measured totals and attribute individual code changes as likely contributors only. diff --git a/domains/analytics/skills/performance-attribution/skill.md b/domains/analytics/skills/performance-attribution/skill.md new file mode 100644 index 00000000..379dac6d --- /dev/null +++ b/domains/analytics/skills/performance-attribution/skill.md @@ -0,0 +1,90 @@ +--- +maturity: experimental +name: performance-attribution +description: Attribute release-over-release p75/p95 performance movements to specific code changes via black-box diff analysis +--- + +# Performance Attribution + +Pair a **measured** percentile movement (from Sentry Trace Explorer) with **black-box code-diff analysis** to produce confidence-rated attributions: what changed across releases, how much it moved, and why. + +## When To Use + +- Explaining a confirmed p75/p95 latency change across releases +- Building a per-release attribution catalogue (change → confidence → metric) +- Auditing whether a "performance initiative" actually moved a metric +- Attributing movement that spans the app repo **and** `@metamask/*` core-package bumps + +## Do Not Use When + +- The metric movement isn't yet confirmed reliable — run query hygiene first (see `sentry-mcp-queries`: filter superseded/low-sample releases, normalize, verify stored sample size) +- You need proof of causation — this yields *likely contributors*, not isolated causes (see Limitations) +- Pre-merge perf review of a single PR — there is no production metric to attribute yet + +## Step 1 — Get the Measurement Right First + +Attribution is only as good as the metric. Lock these down before touching code: + +- **Percentile.** p75 = typical user (more stable signal). p95 = slowest 5% — *assumed* large-wallet/power users, but **not cohort-verified** (also slow hardware / poor network). Prioritize p95 when the optimization targets data size (memoization, virtualization) that disproportionately helps the tail; trust p75 as the more reliable number. +- **Time window.** Use the **longer (90d) window as primary** — it includes traffic from when older releases were actively used, so the population is representative and comparable across releases. A 30d window over-weights residual users still lingering on old versions → inflated baselines and bigger-looking deltas between *different* populations. Report 90d; cite 30d only as context. +- **Version selection.** Per minor line, anchor on the **highest-sample patch**, never the `.0`. `.0` releases have 10–100× fewer samples and are rollout-biased. See `sentry-mcp-queries` → *Filtering Unreliable Releases* and *Longer-Range (30D+) Queries and Percentile Fidelity*. + +## Step 2 — Black-Box Code Analysis + +Assess impact on **code content + execution frequency alone**. Deliberately ignore commit messages, PR titles/descriptions, claimed impact, and epic/initiative goals — they bias the read. Base it on: diff content, file location (→ execution frequency), algorithmic complexity, and memoization patterns. + +Hot-path categories — a change here can move a render metric: + +| Category | Why it matters | +|---|---| +| Build config | Build-time transforms (e.g. React Compiler) apply broadly | +| Selectors | Run on every state change — hottest path | +| Hooks | Affect component lifecycle / re-render frequency | +| Components | Virtualization & render patterns | +| Dependencies | Version bumps change runtime behavior | + +Pattern catalogue — what to grep for: + +| Pattern | Signal | Confidence | +|---|---|---| +| `createDeepEqualSelector` → `createSelector` + `EMPTY_ARRAY` sentinel | Removes per-change deep compares | HIGH if foundational selector | +| Identity-function selector `(foo => foo)` → real transform | Broken memoization fixed | HIGH if many consumers | +| In-place mutation `.sort()/.reverse()/.splice()` → spread copy | Mutation had broken all downstream memoization | HIGH | +| Build-plugin addition with broad scope | Build-time optimization | HIGH if scope = all `ui/` | +| O(n) string parse → O(1) lookup | Algorithmic reduction | MEDIUM — depends on call frequency | + +## Step 3 — Score Confidence + +1. **Mechanism** — how does this reduce work? (fewer re-renders / less allocation / better caching) +2. **Frequency** — is the path hot? (selector per state change = hot) +3. **Scope** — how many components/files does it touch? +4. **Match** — does it target what the metric measures? + +| Confidence | Criteria | +|---|---| +| HIGH | Clear mechanism + hot path + timing matches the metric move | +| MEDIUM | Mechanism clear, frequency or scope uncertain | +| LOW | Indirect or infrastructure-only | + +## Step 4 — Don't Forget Core Packages + +App-repo diffs miss work shipped as `@metamask/*` version bumps — it surfaces only as a `package.json` change. For each bump between releases, read the package CHANGELOG "Changed"/"Fixed" sections for state-size reduction, caching, fewer RPC calls, batching, or data-structure/field deprecations. (Commands and the packages to watch live in the repo file.) + +## Reading / Writing an Attribution Catalogue + +- Release-header totals = the **measured** improvement for the whole release +- Table rows = **likely contributors**, not isolated causes +- "High confidence" = mechanism + timing + population align +- Always keep an **Unattributed** section for movement no change explains +- Flag high-variance metrics (a noisy confirmation-popup p95) as inconclusive, not as wins + +## Limitations + +- **Correlation, not causation** — change + improvement in the same release does not prove the change caused it +- **Release totals, not isolated impact** — a "-44%" reflects the entire release, not one change +- **Production variance** — user hardware and network are uncontrolled +- **Code analysis, not runtime profiling** — based on structure, not measured execution paths +- **p95 cohort is assumed, not verified** — no power-user segmentation +- **Window choice changes the baseline** — always state which window a number came from + +For more precise attribution: per-optimization feature flags / A-B tests, CI synthetic benchmarks, and verified user-cohort segmentation. diff --git a/domains/analytics/skills/sentry-mcp-queries/skill.md b/domains/analytics/skills/sentry-mcp-queries/skill.md index 0ddddbff..8a134576 100644 --- a/domains/analytics/skills/sentry-mcp-queries/skill.md +++ b/domains/analytics/skills/sentry-mcp-queries/skill.md @@ -72,12 +72,31 @@ Patch releases have uneven adoption — comparing raw counts against them produc |---|---|---| | Age since publish | < 48–72h | Browser auto-update rollout still ramping (Chrome/Firefox/Edge) | | Session count | < ~50% of previous stable release | Sample too small for meaningful rates | +| Stored span count | < ~few hundred for p75, < ~few thousand for p95+ | Tail percentiles are computed over the *stored* sample — extrapolated counts hide how few events back them | +| Superseded patch | a higher patch in the same `X.Y.*` line exists **and** the active window (`first_seen`→`last_seen`) is short | Hotfixed-past releases collect few spans, biased to early-updaters during the rollout/migration window | | Release stage | `dev`, `canary`, `nightly` | Non-production build — different error profile | | Environment | not `production` | Development / staging noise | | Manifest split | compare only within same `dist` | MV3 and MV2 populations have different error distributions | **Rule of thumb:** use the newest release that has ≥ 3 days of production adoption **and** session volume comparable to the previous stable release. Everything in between is hotfix noise — skip it for regression comparisons unless investigating that specific patch. +## Longer-Range (30D+) Queries and Percentile Fidelity + +Widening the window past ~30 days to gain sample size trades it back for **fidelity loss on older releases**. Three effects compound: + +- **Sample-rate drift** — `tracesSampleRate` changes between releases, so absolute span counts across a 30D+ window mix different capture rates. Normalize each release by *its own* sample rate (or by sessions/users), never a single global rate. +- **Extrapolation hides thin samples** — span datasets report sample-rate-weighted (extrapolated) counts. A release with 40 stored spans at 0.75% extrapolates to ~5,300 — a real-looking number backed by 40 events. Always check the **stored** sample count, not the extrapolated total, before trusting a release. +- **Retention downsampling** — spans near the retention boundary are partially evicted, so an old release's count is truncated, not representative. Treat the oldest releases in a 30D+ window as lower bounds only. + +**For p75+ analysis** (any tail percentile — p75/p90/p95/p99), sample size *and* quality both matter: + +- **Size** — percentiles are computed over stored events. p50 stabilizes in the low hundreds; p75 needs more; p95/p99 need thousands of stored spans. Below that, a handful of outliers move the number — don't report a tail percentile you can't back with stored count. +- **Quality** — rollout-window spans (first-launch, cold cache, state migration) skew the tail high. A superseded patch release's spans are disproportionately these, so its p75+ reads worse than its steady state would. + +**Resolving the size-vs-fidelity tension:** when a single release lacks the sample to support p75+, **collapse the patch chain** — aggregate `release:X.Y.*` across the minor line, or compare against the last *widely-adopted* patch — rather than extending the window into aged, downsampled, sample-rate-drifted territory. Reach for sample size *across adjacent stable patches inside the retention-safe window*, not by going further back in time. Use a longer (90d) window as the **primary, comparable-across-releases** source for p75/p95 and a 30d window only as **secondary context** — 30d over-weights the users still lingering on old versions and inflates baselines. + +For attributing a confirmed p75/p95 movement to specific code changes, see the `performance-attribution` skill. + ## Workflow: Replay and Profile 1. `mcp__sentry__search_issue_events` — find an event ID with replay/profile @@ -105,3 +124,6 @@ Patch releases have uneven adoption — comparing raw counts against them produc | Compare raw event counts across releases | Normalize by sessions — traffic changes masquerade as regressions | | Include a <48h-old release in a regression comparison | Wait for rollout; auto-update adoption takes 2–7 days | | Treat every patch release as a comparison point | Most patches have low adoption — compare to the last *widely-adopted* release | +| Trust a release's p95 because its (extrapolated) span count looks large | Check the *stored* sample — p75+ needs hundreds-to-thousands of stored events to be stable | +| Compare span counts across a 30D+ window at face value | Normalize per-release sample rate; older releases are downsampled / retention-truncated | +| Anchor a percentile on a `.0` release | `.0` releases have 10–100× fewer samples — use the highest-sample patch in the minor line | From 837ba9003f9acc602d1f26a0791d590eafac2ed5 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Wed, 15 Jul 2026 14:33:28 -0400 Subject: [PATCH 005/135] =?UTF-8?q?sentry-quota:=20catch=20incidental=20in?= =?UTF-8?q?strumentation=20=E2=80=94=20memoized-selector=20fan-out,=20late?= =?UTF-8?q?nt=20trace=3F=20param,=20trace-arg=20PR-review=20scan,=20per-na?= =?UTF-8?q?me=20budget?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- domains/analytics/skills/sentry-quota/skill.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/domains/analytics/skills/sentry-quota/skill.md b/domains/analytics/skills/sentry-quota/skill.md index 1602ca55..6ae2e1b1 100644 --- a/domains/analytics/skills/sentry-quota/skill.md +++ b/domains/analytics/skills/sentry-quota/skill.md @@ -34,13 +34,17 @@ A custom span is a quota risk when these stack. The first three together are the Low fan-out + discrete user action + already gated = fine. Don't flag healthy spans. +**The subtlest fan-out has no visible loop: a memoized selector.** A `trace` passed into a memoized selector (`createSelector` / `reselect`, or any function called from `useSelector`) fires on every input change by reference. If the selector also iterates entities, it is fan-out × recompute-frequency. Its volume tracks internal state-churn, not user action, so no user-facing metric predicts it — you cannot capacity-plan it. Treat any `trace` reaching a selector as fan-out. + ## Workflow ### PR review (pre-merge gate) -1. `gh pr diff ` — scan **added** lines for new `TraceName` entries and `trace(` call sites. -2. Score each new span against the breach triad: is the enclosing scope a loop/poller? is there a gate? a kill-switch? +1. `gh pr diff ` — scan **added** lines for three things, not two: new `TraceName` entries, new `trace(` call sites, **and a `trace`/trace-callback passed as an *argument*** into a call (`fn(…, trace)`). The third is the one reviews miss — a caller wiring up a function's optional `trace?` param adds instrumentation with no `trace(` site and no `TraceName` entry. +2. Score each against the breach triad: is the enclosing scope a loop, poller, **or selector**? is there a gate? a kill-switch? 3. Block if a new always-on span has no gate — require a sub-sample gate (`span-sub-sampling`) before merge. Cheaper than a post-ship cherry-pick. -4. If the diff adds no `trace(` sites and no `TraceName` entries → "no new instrumentation, no quota risk", stop. +4. If the diff adds no `trace(` sites, no `TraceName` entries, **and no `trace` argument passed into a call** → "no new instrumentation", stop. + +> **Instrumentation is not always added by an instrumentation PR.** The costliest spans arrive incidentally — a caller passes a `trace` argument into an existing function during an unrelated change (a bug fix, a refactor), so the PR's stated purpose gives no signal to review it for quota. Do not gate this scan on the PR *looking* like instrumentation. And accept the limit: a `trace` argument buried in a bug-fix diff will slip a human reviewer, which is why the runtime backstops (per-name volume alerting, the per-name sampler budget below) exist. This skill lowers the rate; it does not eliminate the class. ### Locate (incident) 1. Grep the span name / `TraceName.X` across the consuming repo **and** the controller package source. @@ -65,6 +69,8 @@ Pick the lowest tier that stops the bleed. Tier 0 + 1 stop the bleed now; Tier 2 is the follow-up so the metric returns. +**Prevent the next one, not just this one.** Every tier above requires *naming* the offender first, so a new one runs unbounded until someone catches it. A per-transaction-name budget in the sampler — sample the first N of a name per session, then decay — bounds *any* name with no advance knowledge of which will misbehave. It is the only control that acts before the offender is named, and the only one that catches instrumentation added incidentally rather than deliberately. + ## Common Pitfalls | Mistake | Correct approach | @@ -76,3 +82,6 @@ Tier 0 + 1 stop the bleed now; Tier 2 is the follow-up so the metric returns. | Disable the span on `main` only | Cherry-pick to the active release branch — `main` alone leaves the live release breaching | | Treat "move to Segment" as free | Segment events ship without CI governance or billing review (`segment-governance`) | | Ship new always-on instrumentation with no kill-switch | Add an env disable flag on day one — turns a future cut into a config flip, not a cherry-pick | +| An optional `trace?` param passes review because it emits nothing | It is a dormant fan-out — it detonates when any caller supplies the argument. Remove the *param*, not just the argument, so one line can't re-arm it. | +| Disable one entry point of a multi-path change | One change can reach the backend by more than one path (a controller callback *and* a selector param). Audit every entry point it added, not just the one that fired. | +| Filter a release before its successor is fixed | The filter redirects users onto the next build; if that carries the same span, volume only moves. Filter a release only once the build users update to is clean. | From 0061f40d9bab6363ad23fbab1fb6ab6fc138670c Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Wed, 15 Jul 2026 14:41:36 -0400 Subject: [PATCH 006/135] CHANGELOG: note expanded sentry-quota detection (selector fan-out, incidental instrumentation) --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ebc02533..64e8da78 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- Add `analytics` domain: Sentry span-quota guard, Sentry MCP query workflows (including longer-range/30D+ query fidelity and percentile-sample-size filtering), release-over-release performance attribution, instrumentation methodology, and supporting knowledge +- Add `analytics` domain: Sentry span-quota guard, Sentry MCP query workflows (including longer-range/30D+ query fidelity and percentile-sample-size filtering), release-over-release performance attribution, instrumentation methodology (including memoized-selector fan-out and incidentally-added instrumentation), and supporting knowledge ## [0.1.0] From 8eb0970b3f2defb8290a4f7d6c2339aafc1c88c2 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Thu, 16 Jul 2026 11:00:06 -0400 Subject: [PATCH 007/135] Add `typescript-typing` skill for `any`-handling and type derivation --- CHANGELOG.md | 4 ++ .../coding/skills/typescript-typing/skill.md | 58 +++++++++++++++++++ 2 files changed, 62 insertions(+) create mode 100644 domains/coding/skills/typescript-typing/skill.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 8557ff8f..3fe0994a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- Add `typescript-typing` skill (coding domain): the reasoning layer beneath `coding-guidelines`' TypeScript section — `any` as a type-checking-off directive with substitute-by-position guidance (assignee → `unknown`, assigned → `never`) and the generic-constraint exception, plus derive-types-from-authoritative-sources over ad-hoc declarations. Experimental. + ## [0.2.0] ### Added diff --git a/domains/coding/skills/typescript-typing/skill.md b/domains/coding/skills/typescript-typing/skill.md new file mode 100644 index 00000000..b3e9cc64 --- /dev/null +++ b/domains/coding/skills/typescript-typing/skill.md @@ -0,0 +1,58 @@ +--- +name: typescript-typing +description: TypeScript typing discipline — treat `any` as a directive that disables type checking (substitute by position: assignee → `unknown`, assigned → `never`), and derive types from authoritative sources rather than hand-writing ad-hoc ones that duplicate and drift. +maturity: experimental +--- + +# TypeScript Typing Discipline + +Deepens the TypeScript section of `mms-coding-guidelines` — which already says "avoid `any`, prefer `unknown`" and links `MetaMask/contributor-docs` `docs/typescript.md`. This skill is the reasoning layer underneath those bullets: why `any` is dangerous and how to replace it by position, and how to avoid hand-writing a type that an authoritative source already defines. Grounded in `docs/typescript.md` (§ Avoid `any`, § Prefer type inference). + +## `any` is not a type — it is a directive that disables type checking + +The mental model matters more than the ESLint rule, because `@typescript-eslint/no-explicit-any` (already `error` in extension CI) does not stop the reasoning that reaches for `any`: + +- **`any` is not "the widest type" — that is `unknown`.** `any` is a compiler directive that _disables_ type checking for the value it annotates. +- **It suppresses every error about its assignee** — the equivalent of `@ts-ignore` on every use of that variable. The errors still affect the code; `any` only makes them invisible. +- **It subsumes what it touches.** Any type in a union, intersection, or property relationship with `any` becomes `any` — an unmitigated loss of type information. +- **It infects downstream code.** One `any` at a source (e.g. a library type that resolves to `any`) propagates silently through every consumer, converting compile-time errors into **silent runtime failures** — defeating the point of a statically-typed language. + +## Substitute `any` by position — assignee vs assigned + +Identify which side of an assignment the `any` sits on: + +- **Assignee** (a variable, parameter, or return that _receives_ a value — "it could be anything"): **try `unknown` first**, then narrow. `unknown` is the true universal supertype: everything is assignable to it, but it forces a type guard before use. `any` ↔ `unknown` are interchangeable in this position, so it is almost always a safe swap. + - 🚫 `type Fn = () => any; const xs: any[]` + - ✅ `type Fn = () => unknown; const xs: unknown[]` +- **Assigned** (a value that _flows into_ a slot): **try `never` first**, then widen to a subtype of the assignee's type. `unknown` cannot substitute here (it is only assignable to `unknown`); `never` is the bottom type, assignable to everything. + +## The one acceptable exception — generic *constraints* + +`any` is acceptable in a generic **constraint**, and only there. It bounds a type parameter without being assigned to a value, so it neither pollutes nor infects. + +```typescript +class BaseController< + // eslint-disable-next-line @typescript-eslint/no-explicit-any + Messenger extends RestrictedMessenger, +> // ... +``` + +Three conditions attach: + +- **Declare it explicitly.** `no-explicit-any` is `error`, so a constraint `any` needs an inline `// eslint-disable-next-line @typescript-eslint/no-explicit-any` at the site — a deliberate, visible exception. +- **Constraints only — never a generic _argument_.** Passing `any` as an argument (`ControllerMessenger`) is 🚫 — that assigns `any` and infects. Constraint-vs-argument is the whole distinction. +- **Prefer a narrower constraint anyway.** Reach for `any` here only when the narrower bound is genuinely unavailable. + +When `any` still seems unavoidable, prefer the narrower, greppable escape hatches: `as unknown as` as a documented last resort, or `@ts-expect-error` with a TODO. Never reach for `any` to unblock feature work "to fix later." + +## Derive types from authoritative sources — don't hand-write ad-hoc ones + +When a type already exists at an authoritative source — a controller's state type, a function's return, a library's exported type, a schema/struct — **derive from it** rather than restating it. Derive with indexed access (`State['field']`), `typeof`, `ReturnType` / `Parameters`, utility types (`Pick` / `Omit` / `Partial`), and `Infer`; let inference carry the rest. This is the structural form of the contributor-docs rule *Prefer type inference over annotations and assertions*: inferred and derived types stay "responsive to changes in code," while hand-written declarations "rely on hard-coding, making them brittle against code drift." + +An ad-hoc type — one hand-defined to describe a value an authoritative type already describes — carries three dangers: + +- **Duplication.** The same shape is stated twice; every reader reconciles them and every change touches both. +- **Incorrect, usually too wide.** A hand-written type is a _guess_ at the source's shape, and the guess is almost always looser than the real type — it admits values the authoritative type would reject, so invalid data still type-checks. +- **Drift.** The source evolves; the copy does not. Because it is hand-written rather than derived, the compiler cannot flag the divergence — the bug surfaces at runtime, not at build. + +**Rule:** before writing a type, ask where the value comes from and whether that source already types it. If it does, derive. Define a fresh type only when no authoritative source exists — a genuinely new shape at a boundary you own. Before defining, exhaust deriving: search the internal `@metamask/*` packages and the consuming repo for the authoritative source first. From 9e20763e8775838730e59883de4f96be684317e7 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Thu, 16 Jul 2026 12:52:34 -0400 Subject: [PATCH 008/135] Ground the derive rule in a real `metamask-extension` #42583 counterexample --- .../coding/skills/typescript-typing/skill.md | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/domains/coding/skills/typescript-typing/skill.md b/domains/coding/skills/typescript-typing/skill.md index b3e9cc64..1b4d23d9 100644 --- a/domains/coding/skills/typescript-typing/skill.md +++ b/domains/coding/skills/typescript-typing/skill.md @@ -55,4 +55,30 @@ An ad-hoc type — one hand-defined to describe a value an authoritative type al - **Incorrect, usually too wide.** A hand-written type is a _guess_ at the source's shape, and the guess is almost always looser than the real type — it admits values the authoritative type would reject, so invalid data still type-checks. - **Drift.** The source evolves; the copy does not. Because it is hand-written rather than derived, the compiler cannot flag the divergence — the bug surfaces at runtime, not at build. +**A grounded example (`metamask-extension` #42583).** A `wallet-services` module hand-wrote a slice of `NetworkController` state instead of deriving it. + +🚫 Re-declared — every field optional (wider than the real, _required_ field), keyed by `string` not `Hex`, and unlinked from the source, so it drifts silently when the controller changes: + +```typescript +type NetworkControllerState = { + networkConfigurationsByChainId?: Record< + string, + { + defaultRpcEndpointIndex?: number; + rpcEndpoints?: { networkClientId?: string }[]; + } + >; +}; +``` + +✅ Derived — tracks the authoritative shape (`Record`), narrowed to the one field in use: + +```typescript +import type { NetworkState } from '@metamask/network-controller'; + +type NetworkConfigurations = NetworkState['networkConfigurationsByChainId']; +``` + +A too-wide copy does not save work; it moves the work downstream. The same PR typed a dependency as `getMetaMaskState: () => Record`, so every consumer then had to re-cast the shape back by hand — including a `{ metamask: getMetaMaskState() } as never` double-cast. Deriving that dependency from the authoritative state type deletes the casts. The same file _did_ derive one type correctly (`type Action = (typeof ACTIONS)[number]`), so the pattern was already in hand; the discipline is extending it to every referenced type. + **Rule:** before writing a type, ask where the value comes from and whether that source already types it. If it does, derive. Define a fresh type only when no authoritative source exists — a genuinely new shape at a boundary you own. Before defining, exhaust deriving: search the internal `@metamask/*` packages and the consuming repo for the authoritative source first. From be1dc98c5c1dfafaddde569ec0fb07181ff1a31e Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Thu, 16 Jul 2026 13:09:53 -0400 Subject: [PATCH 009/135] Split typescript-typing into a `typescript` domain: avoid-any, derive-types, decompose-large-files --- CHANGELOG.md | 2 +- .../coding/skills/typescript-typing/skill.md | 84 ------------------- domains/typescript/skills/avoid-any/skill.md | 46 ++++++++++ .../skills/decompose-large-files/skill.md | 46 ++++++++++ .../typescript/skills/derive-types/skill.md | 51 +++++++++++ 5 files changed, 144 insertions(+), 85 deletions(-) delete mode 100644 domains/coding/skills/typescript-typing/skill.md create mode 100644 domains/typescript/skills/avoid-any/skill.md create mode 100644 domains/typescript/skills/decompose-large-files/skill.md create mode 100644 domains/typescript/skills/derive-types/skill.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 3fe0994a..65fe0f20 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- Add `typescript-typing` skill (coding domain): the reasoning layer beneath `coding-guidelines`' TypeScript section — `any` as a type-checking-off directive with substitute-by-position guidance (assignee → `unknown`, assigned → `never`) and the generic-constraint exception, plus derive-types-from-authoritative-sources over ad-hoc declarations. Experimental. +- Add `typescript` domain (experimental) with three skills: `avoid-any` (`any` is a type-checking-off directive, not a type — substitute by position: assignee → `unknown`, assigned → `never`; the one exception is a generic constraint), `derive-types` (derive from authoritative sources over ad-hoc declarations that duplicate, run too wide, and drift), and `decompose-large-files` (decompose a large file by coherent, independently-mergeable units for modularity, maintainability, and reviewability — and to unblock incremental TS migration). ## [0.2.0] diff --git a/domains/coding/skills/typescript-typing/skill.md b/domains/coding/skills/typescript-typing/skill.md deleted file mode 100644 index 1b4d23d9..00000000 --- a/domains/coding/skills/typescript-typing/skill.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -name: typescript-typing -description: TypeScript typing discipline — treat `any` as a directive that disables type checking (substitute by position: assignee → `unknown`, assigned → `never`), and derive types from authoritative sources rather than hand-writing ad-hoc ones that duplicate and drift. -maturity: experimental ---- - -# TypeScript Typing Discipline - -Deepens the TypeScript section of `mms-coding-guidelines` — which already says "avoid `any`, prefer `unknown`" and links `MetaMask/contributor-docs` `docs/typescript.md`. This skill is the reasoning layer underneath those bullets: why `any` is dangerous and how to replace it by position, and how to avoid hand-writing a type that an authoritative source already defines. Grounded in `docs/typescript.md` (§ Avoid `any`, § Prefer type inference). - -## `any` is not a type — it is a directive that disables type checking - -The mental model matters more than the ESLint rule, because `@typescript-eslint/no-explicit-any` (already `error` in extension CI) does not stop the reasoning that reaches for `any`: - -- **`any` is not "the widest type" — that is `unknown`.** `any` is a compiler directive that _disables_ type checking for the value it annotates. -- **It suppresses every error about its assignee** — the equivalent of `@ts-ignore` on every use of that variable. The errors still affect the code; `any` only makes them invisible. -- **It subsumes what it touches.** Any type in a union, intersection, or property relationship with `any` becomes `any` — an unmitigated loss of type information. -- **It infects downstream code.** One `any` at a source (e.g. a library type that resolves to `any`) propagates silently through every consumer, converting compile-time errors into **silent runtime failures** — defeating the point of a statically-typed language. - -## Substitute `any` by position — assignee vs assigned - -Identify which side of an assignment the `any` sits on: - -- **Assignee** (a variable, parameter, or return that _receives_ a value — "it could be anything"): **try `unknown` first**, then narrow. `unknown` is the true universal supertype: everything is assignable to it, but it forces a type guard before use. `any` ↔ `unknown` are interchangeable in this position, so it is almost always a safe swap. - - 🚫 `type Fn = () => any; const xs: any[]` - - ✅ `type Fn = () => unknown; const xs: unknown[]` -- **Assigned** (a value that _flows into_ a slot): **try `never` first**, then widen to a subtype of the assignee's type. `unknown` cannot substitute here (it is only assignable to `unknown`); `never` is the bottom type, assignable to everything. - -## The one acceptable exception — generic *constraints* - -`any` is acceptable in a generic **constraint**, and only there. It bounds a type parameter without being assigned to a value, so it neither pollutes nor infects. - -```typescript -class BaseController< - // eslint-disable-next-line @typescript-eslint/no-explicit-any - Messenger extends RestrictedMessenger, -> // ... -``` - -Three conditions attach: - -- **Declare it explicitly.** `no-explicit-any` is `error`, so a constraint `any` needs an inline `// eslint-disable-next-line @typescript-eslint/no-explicit-any` at the site — a deliberate, visible exception. -- **Constraints only — never a generic _argument_.** Passing `any` as an argument (`ControllerMessenger`) is 🚫 — that assigns `any` and infects. Constraint-vs-argument is the whole distinction. -- **Prefer a narrower constraint anyway.** Reach for `any` here only when the narrower bound is genuinely unavailable. - -When `any` still seems unavoidable, prefer the narrower, greppable escape hatches: `as unknown as` as a documented last resort, or `@ts-expect-error` with a TODO. Never reach for `any` to unblock feature work "to fix later." - -## Derive types from authoritative sources — don't hand-write ad-hoc ones - -When a type already exists at an authoritative source — a controller's state type, a function's return, a library's exported type, a schema/struct — **derive from it** rather than restating it. Derive with indexed access (`State['field']`), `typeof`, `ReturnType` / `Parameters`, utility types (`Pick` / `Omit` / `Partial`), and `Infer`; let inference carry the rest. This is the structural form of the contributor-docs rule *Prefer type inference over annotations and assertions*: inferred and derived types stay "responsive to changes in code," while hand-written declarations "rely on hard-coding, making them brittle against code drift." - -An ad-hoc type — one hand-defined to describe a value an authoritative type already describes — carries three dangers: - -- **Duplication.** The same shape is stated twice; every reader reconciles them and every change touches both. -- **Incorrect, usually too wide.** A hand-written type is a _guess_ at the source's shape, and the guess is almost always looser than the real type — it admits values the authoritative type would reject, so invalid data still type-checks. -- **Drift.** The source evolves; the copy does not. Because it is hand-written rather than derived, the compiler cannot flag the divergence — the bug surfaces at runtime, not at build. - -**A grounded example (`metamask-extension` #42583).** A `wallet-services` module hand-wrote a slice of `NetworkController` state instead of deriving it. - -🚫 Re-declared — every field optional (wider than the real, _required_ field), keyed by `string` not `Hex`, and unlinked from the source, so it drifts silently when the controller changes: - -```typescript -type NetworkControllerState = { - networkConfigurationsByChainId?: Record< - string, - { - defaultRpcEndpointIndex?: number; - rpcEndpoints?: { networkClientId?: string }[]; - } - >; -}; -``` - -✅ Derived — tracks the authoritative shape (`Record`), narrowed to the one field in use: - -```typescript -import type { NetworkState } from '@metamask/network-controller'; - -type NetworkConfigurations = NetworkState['networkConfigurationsByChainId']; -``` - -A too-wide copy does not save work; it moves the work downstream. The same PR typed a dependency as `getMetaMaskState: () => Record`, so every consumer then had to re-cast the shape back by hand — including a `{ metamask: getMetaMaskState() } as never` double-cast. Deriving that dependency from the authoritative state type deletes the casts. The same file _did_ derive one type correctly (`type Action = (typeof ACTIONS)[number]`), so the pattern was already in hand; the discipline is extending it to every referenced type. - -**Rule:** before writing a type, ask where the value comes from and whether that source already types it. If it does, derive. Define a fresh type only when no authoritative source exists — a genuinely new shape at a boundary you own. Before defining, exhaust deriving: search the internal `@metamask/*` packages and the consuming repo for the authoritative source first. diff --git a/domains/typescript/skills/avoid-any/skill.md b/domains/typescript/skills/avoid-any/skill.md new file mode 100644 index 00000000..b7a207d9 --- /dev/null +++ b/domains/typescript/skills/avoid-any/skill.md @@ -0,0 +1,46 @@ +--- +name: avoid-any +description: Handle `any` correctly — it is not a type but a directive that disables type checking. Substitute by position (assignee → `unknown`, assigned → `never`); the one exception is a generic constraint, declared with an inline eslint-disable. +maturity: experimental +--- + +# Avoid `any` + +Deepens the TypeScript guidance in `mms-coding-guidelines` — which already says "avoid `any`, prefer `unknown`" and links `MetaMask/contributor-docs` `docs/typescript.md`. This is the reasoning layer beneath that bullet: why `any` is dangerous, and how to replace it by position. Grounded in `docs/typescript.md` (§ Avoid `any`). + +## `any` is not a type — it is a directive that disables type checking + +The mental model matters more than the ESLint rule, because `@typescript-eslint/no-explicit-any` (already `error` in extension CI) does not stop the reasoning that reaches for `any`: + +- **`any` is not "the widest type" — that is `unknown`.** `any` is a compiler directive that _disables_ type checking for the value it annotates. +- **It suppresses every error about its assignee** — the equivalent of `@ts-ignore` on every use of that variable. The errors still affect the code; `any` only makes them invisible. +- **It subsumes what it touches.** Any type in a union, intersection, or property relationship with `any` becomes `any` — an unmitigated loss of type information. +- **It infects downstream code.** One `any` at a source (e.g. a library type that resolves to `any`) propagates silently through every consumer, converting compile-time errors into **silent runtime failures** — defeating the point of a statically-typed language. + +## Substitute `any` by position — assignee vs assigned + +Identify which side of an assignment the `any` sits on: + +- **Assignee** (a variable, parameter, or return that _receives_ a value — "it could be anything"): **try `unknown` first**, then narrow. `unknown` is the true universal supertype: everything is assignable to it, but it forces a type guard before use. `any` ↔ `unknown` are interchangeable in this position, so it is almost always a safe swap. + - 🚫 `type Fn = () => any; const xs: any[]` + - ✅ `type Fn = () => unknown; const xs: unknown[]` +- **Assigned** (a value that _flows into_ a slot): **try `never` first**, then widen to a subtype of the assignee's type. `unknown` cannot substitute here (it is only assignable to `unknown`); `never` is the bottom type, assignable to everything. + +## The one acceptable exception — generic *constraints* + +`any` is acceptable in a generic **constraint**, and only there. It bounds a type parameter without being assigned to a value, so it neither pollutes nor infects. + +```typescript +class BaseController< + // eslint-disable-next-line @typescript-eslint/no-explicit-any + Messenger extends RestrictedMessenger, +> // ... +``` + +Three conditions attach: + +- **Declare it explicitly.** `no-explicit-any` is `error`, so a constraint `any` needs an inline `// eslint-disable-next-line @typescript-eslint/no-explicit-any` at the site — a deliberate, visible exception. +- **Constraints only — never a generic _argument_.** Passing `any` as an argument (`ControllerMessenger`) is 🚫 — that assigns `any` and infects. Constraint-vs-argument is the whole distinction. +- **Prefer a narrower constraint anyway.** Reach for `any` here only when the narrower bound is genuinely unavailable. + +When `any` still seems unavoidable, prefer the narrower, greppable escape hatches: `as unknown as` as a documented last resort, or `@ts-expect-error` with a TODO. Never reach for `any` to unblock feature work "to fix later." diff --git a/domains/typescript/skills/decompose-large-files/skill.md b/domains/typescript/skills/decompose-large-files/skill.md new file mode 100644 index 00000000..99a749d5 --- /dev/null +++ b/domains/typescript/skills/decompose-large-files/skill.md @@ -0,0 +1,46 @@ +--- +name: decompose-large-files +description: Decompose a large file into coherent, independently-mergeable modules to improve modularity, maintainability, reviewability, and code organization — and to unblock incremental TS migration. Extract by coherent subject/domain cluster; never fragment for its own sake. +maturity: experimental +--- + +# Decompose Large Files By Coherent Units + +A file that has grown to thousands of lines is hard to review, hard to maintain, and — if it is still `.js` — hard to migrate to TypeScript in one pass. Decompose it by moving coherent subject/domain clusters into their own modules. The goal is **modularity, maintainability, reviewability, and code organization**; unblocking an incremental JS→TS migration is a direct benefit, because each extracted module becomes a small, independently-typable unit. + +Reference application: `metamask-extension` #41735 (`MetamaskController` decomposition, 9,260 → ~3,500 lines). + +## The decision that matters: what is a coherent unit? + +Extraction is worth it only when the extracted piece is a **coherent unit that can move independently**. Apply this judgment _before_ proposing any module: + +- **Coherent subject.** The cluster is about one thing — one domain, one lifecycle, one concern (phishing detection, metrics emission, badge rendering). A reader can state its responsibility in one sentence. +- **Independently mergeable.** It can be lifted behind a defined seam — injected dependencies, or a messenger action — without dragging half the file with it. Its coupling to shared state and to the composition root is small and nameable. +- **Not fragmentation.** Extraction for its own sake — splitting a cohesive routine across files, or pulling out a 20-line helper that only one caller uses and has no independent identity — makes the code _harder_ to follow, not easier. If pulling the piece out means the two halves must still change together, leave it inline. **Refactoring is a means to modularity, not an end; a change that raises the file count without raising coherence is a regression.** + +Some code should **stay** in the original file: the thin **composition / bootstrap root** that wires the modules together. Extracting the wiring itself fragments rather than clarifies — it is the one place the whole is assembled. The tell is code that references _everything_ (the central object plus the shared mutable state every cluster reads); relocating it behind a wide "params bag" just moves the tangle. + +## How to extract one unit (self-contained, per #41735) + +Each extraction is one self-contained change — no separate "final deletion" or "integration" ticket: + +1. **Scaffold** the module (its own file/dir, TypeScript from the start). +2. **Port** the bodies in, unchanged in behavior. +3. **Define the seam** — inject the dependencies the module needs (or register its public methods as messenger actions) instead of reaching back into the file's globals. This is where the human judgment is. +4. **Rewire** the call sites to go through the seam. +5. **Delete the original** in the same change; leave no forwarding stub. +6. **Add a structural unit test** against a stub/mock of the seam — especially valuable when the original file had no tests. + +Steps 1, 2, 5, 6 are largely mechanical (codemod territory — `jscodeshift` on the source, `ts-morph` on the module). Step 3/4 is the part that needs a person. + +## Sizing and sequencing + +- **Size each unit S / M / L / XL** by body size × coupling to rewire. Ship one module (or one subject area) per PR — reviewer context window is usually the binding constraint, so a focused S/M PR merges where an XL one stalls. +- **Sequence lowest-coupling-first.** Extract the clusters with the smallest, cleanest seam first: they establish the pattern and shrink the file so later, more-entangled extractions are easier to see. Save the most coupled cluster (often the core lifecycle) for last, or leave it as the composition root. +- **Enforce the new boundary** so the file cannot silently re-absorb the module — an ESLint `import/no-restricted-paths` rule on the module directory (per #41735). + +## When NOT to decompose + +- The file is large but **already cohesive** — one subject, read top to bottom. Size alone is not a reason. +- The only available splits are **arbitrary** (by line count, or a catch-all `utils`) rather than by subject. That produces fragments, not modules. +- The extraction **cannot get a clean seam** — everything it touches is shared mutable state with the rest of the file. Fix the coupling first, or leave it inline and say so. diff --git a/domains/typescript/skills/derive-types/skill.md b/domains/typescript/skills/derive-types/skill.md new file mode 100644 index 00000000..84bdbe7c --- /dev/null +++ b/domains/typescript/skills/derive-types/skill.md @@ -0,0 +1,51 @@ +--- +name: derive-types +description: Derive types from authoritative sources (indexed access, `typeof`, `ReturnType`/`Parameters`, `Pick`/`Omit`, `Infer`) instead of hand-writing ad-hoc types that duplicate, run too wide, and drift. +maturity: experimental +--- + +# Derive Types From Authoritative Sources + +Deepens the TypeScript guidance in `mms-coding-guidelines`, and is the structural counterpart to the contributor-docs rule *Prefer type inference over annotations and assertions* (`MetaMask/contributor-docs` `docs/typescript.md`). Where inference handles values, derivation handles types that reference other types. + +## Derive, don't re-declare + +When a type already exists at an authoritative source — a controller's state type, a function's return, a library's exported type, a schema/struct — **derive from it** rather than restating it. Derive with indexed access (`State['field']`), `typeof`, `ReturnType` / `Parameters`, utility types (`Pick` / `Omit` / `Partial`), and `Infer`; let inference carry the rest. Inferred and derived types stay "responsive to changes in code," while hand-written declarations "rely on hard-coding, making them brittle against code drift." + +An ad-hoc type — one hand-defined to describe a value an authoritative type already describes — carries three dangers: + +- **Duplication.** The same shape is stated twice; every reader reconciles them and every change touches both. +- **Incorrect, usually too wide.** A hand-written type is a _guess_ at the source's shape, and the guess is almost always looser than the real type — it admits values the authoritative type would reject, so invalid data still type-checks. +- **Drift.** The source evolves; the copy does not. Because it is hand-written rather than derived, the compiler cannot flag the divergence — the bug surfaces at runtime, not at build. + +## A grounded example (`metamask-extension` #42583) + +A `wallet-services` module hand-wrote a slice of `NetworkController` state instead of deriving it. + +🚫 Re-declared — every field optional (wider than the real, _required_ field), keyed by `string` not `Hex`, and unlinked from the source, so it drifts silently when the controller changes: + +```typescript +type NetworkControllerState = { + networkConfigurationsByChainId?: Record< + string, + { + defaultRpcEndpointIndex?: number; + rpcEndpoints?: { networkClientId?: string }[]; + } + >; +}; +``` + +✅ Derived — tracks the authoritative shape (`Record`), narrowed to the one field in use: + +```typescript +import type { NetworkState } from '@metamask/network-controller'; + +type NetworkConfigurations = NetworkState['networkConfigurationsByChainId']; +``` + +A too-wide copy does not save work; it moves the work downstream. The same PR typed a dependency as `getMetaMaskState: () => Record`, so every consumer then had to re-cast the shape back by hand — including a `{ metamask: getMetaMaskState() } as never` double-cast. Deriving that dependency from the authoritative state type deletes the casts. The same file _did_ derive one type correctly (`type Action = (typeof ACTIONS)[number]`), so the pattern was already in hand; the discipline is extending it to every referenced type. + +## Rule + +Before writing a type, ask where the value comes from and whether that source already types it. If it does, derive. Define a fresh type only when no authoritative source exists — a genuinely new shape at a boundary you own. Before defining, exhaust deriving: search the internal `@metamask/*` packages and the consuming repo for the authoritative source first. From a89095d6301960892cce4e1b7a619d6c1c9a0da0 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Fri, 17 Jul 2026 05:39:05 -0400 Subject: [PATCH 010/135] Add "why decompose first" rationale to `decompose-large-files`: boundary-identification is step one of any migration --- domains/typescript/skills/decompose-large-files/skill.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/domains/typescript/skills/decompose-large-files/skill.md b/domains/typescript/skills/decompose-large-files/skill.md index 99a749d5..081192b5 100644 --- a/domains/typescript/skills/decompose-large-files/skill.md +++ b/domains/typescript/skills/decompose-large-files/skill.md @@ -10,6 +10,12 @@ A file that has grown to thousands of lines is hard to review, hard to maintain, Reference application: `metamask-extension` #41735 (`MetamaskController` decomposition, 9,260 → ~3,500 lines). +## Why decompose first — even if the file could convert in one pass + +Documenting a large file's modularizable boundaries — one coherent unit per ticket — is worth doing **even if the file could somehow be converted and reviewed in a single PR**, because identifying those boundaries is the first logical step of _any_ migration process, human or AI. The boundary map is not throwaway scaffolding; it is the migration's own plan. + +And a single-pass conversion is impractical even for a capable AI. Converting a multi-thousand-line file in one PR means holding the entire file in context **and** progressively loading every upstream file whose source types the code should derive from (see `derive-types`) **and** every downstream file that imports it and must be updated. That context fan-in (source types) and fan-out (consumers) is the real cost — not the line count of the file itself. Decomposing shrinks each unit's context to one subject plus its seam, which is what makes the conversion tractable and reviewable at all. + ## The decision that matters: what is a coherent unit? Extraction is worth it only when the extracted piece is a **coherent unit that can move independently**. Apply this judgment _before_ proposing any module: From 388dd98f3debcb003d1ac78ae27a393d8980419b Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Fri, 17 Jul 2026 05:51:52 -0400 Subject: [PATCH 011/135] Add `migration-context-cost` skill (fan-in/fan-out) + avoid-any second exception (bivariant callback `any`) --- CHANGELOG.md | 2 +- domains/typescript/skills/avoid-any/skill.md | 32 +++++++++++++++++-- .../skills/decompose-large-files/skill.md | 2 +- .../skills/migration-context-cost/skill.md | 27 ++++++++++++++++ 4 files changed, 59 insertions(+), 4 deletions(-) create mode 100644 domains/typescript/skills/migration-context-cost/skill.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 65fe0f20..8c839195 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- Add `typescript` domain (experimental) with three skills: `avoid-any` (`any` is a type-checking-off directive, not a type — substitute by position: assignee → `unknown`, assigned → `never`; the one exception is a generic constraint), `derive-types` (derive from authoritative sources over ad-hoc declarations that duplicate, run too wide, and drift), and `decompose-large-files` (decompose a large file by coherent, independently-mergeable units for modularity, maintainability, and reviewability — and to unblock incremental TS migration). +- Add `typescript` domain (experimental) with four skills: `avoid-any` (`any` is a type-checking-off directive, not a type — substitute by position: assignee → `unknown`, assigned → `never`; two narrow exceptions: generic constraints and bivariant callback parameters), `derive-types` (derive from authoritative sources over ad-hoc declarations that duplicate, run too wide, and drift), `decompose-large-files` (decompose a large file by coherent, independently-mergeable units for modularity, maintainability, and reviewability — and to unblock incremental TS migration), and `migration-context-cost` (a file's JS→TS migration cost is dominated by context fan-in + fan-out, not line count — scope and sequence tickets by it). ## [0.2.0] diff --git a/domains/typescript/skills/avoid-any/skill.md b/domains/typescript/skills/avoid-any/skill.md index b7a207d9..b9256e87 100644 --- a/domains/typescript/skills/avoid-any/skill.md +++ b/domains/typescript/skills/avoid-any/skill.md @@ -1,6 +1,6 @@ --- name: avoid-any -description: Handle `any` correctly — it is not a type but a directive that disables type checking. Substitute by position (assignee → `unknown`, assigned → `never`); the one exception is a generic constraint, declared with an inline eslint-disable. +description: Handle `any` correctly — it is not a type but a directive that disables type checking. Substitute by position (assignee → `unknown`, assigned → `never`). Two narrow exceptions: a generic constraint, and a callback parameter caught in a bivariant position between two fixed, irresolvable function-type constraints — both declared with an inline eslint-disable. maturity: experimental --- @@ -26,7 +26,7 @@ Identify which side of an assignment the `any` sits on: - ✅ `type Fn = () => unknown; const xs: unknown[]` - **Assigned** (a value that _flows into_ a slot): **try `never` first**, then widen to a subtype of the assignee's type. `unknown` cannot substitute here (it is only assignable to `unknown`); `never` is the bottom type, assignable to everything. -## The one acceptable exception — generic *constraints* +## Acceptable exception 1 — generic *constraints* `any` is acceptable in a generic **constraint**, and only there. It bounds a type parameter without being assigned to a value, so it neither pollutes nor infects. @@ -43,4 +43,32 @@ Three conditions attach: - **Constraints only — never a generic _argument_.** Passing `any` as an argument (`ControllerMessenger`) is 🚫 — that assigns `any` and infects. Constraint-vs-argument is the whole distinction. - **Prefer a narrower constraint anyway.** Reach for `any` here only when the narrower bound is genuinely unavailable. +## Acceptable exception 2 — a callback parameter between two irresolvable function-type constraints + +A callback's parameter may be `any` when all three hold: + +1. **Bivariant position** — the callback is _assignable to_ a wider function type **and** an _assignee of_ a narrower one. +2. **Irresolvable** — no concrete type satisfies both directions, because the wider function type is not a supertype of the narrower (`WideParam extends T extends NarrowParam` has no solution). +3. **Fixed** — neither constraint can be redesigned without breaking callers or losing accuracy. + +Under `--strictFunctionTypes` parameters are contravariant, so the callback's parameter must be a _supertype_ of the outer slot's (outward: `unknown` ✓, `never` ✗) **and** a _subtype_ of the incoming value's (inward: `never` ✓, `unknown` ✗). `any` is the only inhabitant of both the top and bottom of the assignability lattice, so it is the only escape. (Return types stay covariant — keep `unknown`.) + +🚫 If the constraints are **redesignable**, the contravariance error is a real design smell — fix it (usually by parametrizing with a generic), don't suppress: + +```typescript +declare const acceptGeneric: (handler: (event: E) => void) => void; +acceptGeneric(onA); // no `any` needed +``` + +✅ Only when the constraints are **fixed and irresolvable**: + +```typescript +// eslint-disable-next-line @typescript-eslint/no-explicit-any -- Bivariant position with irresolvable, fixed constraints +let bridge: (x: any) => void; +``` + +Like the generic-constraint case, this `any` is **not infectious** — it is scoped to one parameter position, and both constraint types re-impose their signatures at each use site. Annotate the `eslint-disable` with the criteria so a reviewer can check them. Caveat: the safety claim holds only if the constraint types are accurate — a fixed constraint that is itself imprecise (a library type that is `any` internally) still forces the bridge `any` but no longer preserves safety at the use sites. + +Canonical instance: a messenger `registerActionHandler` slot typed `(...args: any[]) => any` — strongly-typed handlers flow inward at registration, strongly-typed argument tuples outward at dispatch; `unknown[]` fails registration, `never[]` fails dispatch. It encodes rank-N polymorphism (`∀α. (α) => R`) that TypeScript cannot express directly. + When `any` still seems unavoidable, prefer the narrower, greppable escape hatches: `as unknown as` as a documented last resort, or `@ts-expect-error` with a TODO. Never reach for `any` to unblock feature work "to fix later." diff --git a/domains/typescript/skills/decompose-large-files/skill.md b/domains/typescript/skills/decompose-large-files/skill.md index 081192b5..5d91e44f 100644 --- a/domains/typescript/skills/decompose-large-files/skill.md +++ b/domains/typescript/skills/decompose-large-files/skill.md @@ -14,7 +14,7 @@ Reference application: `metamask-extension` #41735 (`MetamaskController` decompo Documenting a large file's modularizable boundaries — one coherent unit per ticket — is worth doing **even if the file could somehow be converted and reviewed in a single PR**, because identifying those boundaries is the first logical step of _any_ migration process, human or AI. The boundary map is not throwaway scaffolding; it is the migration's own plan. -And a single-pass conversion is impractical even for a capable AI. Converting a multi-thousand-line file in one PR means holding the entire file in context **and** progressively loading every upstream file whose source types the code should derive from (see `derive-types`) **and** every downstream file that imports it and must be updated. That context fan-in (source types) and fan-out (consumers) is the real cost — not the line count of the file itself. Decomposing shrinks each unit's context to one subject plus its seam, which is what makes the conversion tractable and reviewable at all. +And a single-pass conversion is impractical even for a capable AI — the binding constraint is the file's **context fan-in and fan-out** (upstream source types + downstream consumers), not its line count (see `migration-context-cost`). Decomposing shrinks each unit's context to one subject plus its seam, which is what makes the conversion tractable and reviewable at all. ## The decision that matters: what is a coherent unit? diff --git a/domains/typescript/skills/migration-context-cost/skill.md b/domains/typescript/skills/migration-context-cost/skill.md new file mode 100644 index 00000000..9ff9b6d6 --- /dev/null +++ b/domains/typescript/skills/migration-context-cost/skill.md @@ -0,0 +1,27 @@ +--- +name: migration-context-cost +description: A file's JS→TS migration cost is dominated by context fan-in (upstream files whose source types to derive from) + fan-out (downstream files that import it and must update), not its line count. Scope and sequence migration tickets by this cost. +maturity: experimental +--- + +# TypeScript Migration Context Cost — Fan-In and Fan-Out + +The cost of converting a file to TypeScript is not its line count. It is the **context the conversion pulls in** — everything the converter, human or AI, must load to do it correctly. + +## The two axes + +- **Fan-in (upstream source types).** To type the file's values correctly you must load every upstream module whose types the code should _derive_ from — controller state, function returns, library exports, schemas (see `derive-types`). A file that touches many upstream types has high fan-in even if it is short. +- **Fan-out (downstream consumers).** Changing the file's types ripples to every module that imports it; each may need its own update or re-type. A widely-imported file has high fan-out even if it is small. + +The real work — and the real review surface — is the sum of these, not the target file's length. A 200-line hub imported by 100 files can be a larger migration than a self-contained 2,000-line leaf. + +## Why it matters even for AI + +Single-pass conversion of a high-fan-in/fan-out file is impractical even for a capable AI: it must hold the target file in context **and** progressively load every upstream source-type file **and** every downstream consumer. That context fan-in/fan-out is the binding constraint, not the model's ability to read the file itself. + +## How to use it + +- **Scope tickets by context cost, not LOC.** Estimate fan-in (upstream types the file derives from) + fan-out (`grep -rl` importer count) before sizing a migration ticket. Size by the import-rewrite / re-type surface, not the line count. +- **Sequence low-cost first.** Convert leaf / low-fan-out files early (few downstream updates), and files whose upstream types are already TypeScript (low fan-in), so later conversions have more typed ground to derive from. +- **When the cost won't fit one PR, reduce it before converting.** High fan-in → the upstream types may need to land first. High fan-out through a hub → decompose the hub into coherent units so each unit's fan-out is bounded (see `decompose-large-files`). Decomposition is one response to high context cost — not the only place the cost applies. +- **Keep the fan-out map honest.** A barrel / re-export file inflates apparent fan-out and hides real dependencies; importing from the actual source file (not a barrel) keeps the dependency graph — and the cost estimate — accurate. From 8483d91e835f30a2f8297e7b96f319992afc1ac1 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Fri, 17 Jul 2026 06:09:45 -0400 Subject: [PATCH 012/135] Fix migration-context-cost inaccuracies, reframe decompose how-to around unit conversion, swap in the stronger derive-types example, slim CHANGELOG - migration-context-cost: line count is a factor not a non-factor; fan-in is a reading cost (not a change/review surface), fan-out is the change surface; drop the wrong "upstream types land first" and off-topic barrel bullet - decompose-large-files: the point is converting to TS in small self-contained units, not extraction - derive-types: replace the NetworkState restatement with the reinvented-messenger + hand-copied-return example (derive via `ReturnType`) - CHANGELOG: list the domain, not each skill --- CHANGELOG.md | 2 +- .../skills/decompose-large-files/skill.md | 4 +- .../typescript/skills/derive-types/skill.md | 39 ++++++++++++------- .../skills/migration-context-cost/skill.md | 21 +++++----- 4 files changed, 39 insertions(+), 27 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c839195..7f0f46d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- Add `typescript` domain (experimental) with four skills: `avoid-any` (`any` is a type-checking-off directive, not a type — substitute by position: assignee → `unknown`, assigned → `never`; two narrow exceptions: generic constraints and bivariant callback parameters), `derive-types` (derive from authoritative sources over ad-hoc declarations that duplicate, run too wide, and drift), `decompose-large-files` (decompose a large file by coherent, independently-mergeable units for modularity, maintainability, and reviewability — and to unblock incremental TS migration), and `migration-context-cost` (a file's JS→TS migration cost is dominated by context fan-in + fan-out, not line count — scope and sequence tickets by it). +- Add `typescript` domain (experimental) — TypeScript authoring and JS→TS migration guidance. ## [0.2.0] diff --git a/domains/typescript/skills/decompose-large-files/skill.md b/domains/typescript/skills/decompose-large-files/skill.md index 5d91e44f..f8a13708 100644 --- a/domains/typescript/skills/decompose-large-files/skill.md +++ b/domains/typescript/skills/decompose-large-files/skill.md @@ -26,9 +26,9 @@ Extraction is worth it only when the extracted piece is a **coherent unit that c Some code should **stay** in the original file: the thin **composition / bootstrap root** that wires the modules together. Extracting the wiring itself fragments rather than clarifies — it is the one place the whole is assembled. The tell is code that references _everything_ (the central object plus the shared mutable state every cluster reads); relocating it behind a wide "params bag" just moves the tangle. -## How to extract one unit (self-contained, per #41735) +## How to convert one unit (self-contained, per #41735) -Each extraction is one self-contained change — no separate "final deletion" or "integration" ticket: +The extraction is not the point — **converting the file to TypeScript in small, self-contained units is.** Each unit is a piece you can type and review on its own instead of holding the whole file at once; pulling it into its own module is just what makes that unit self-contained and independently convertible. Each unit is one self-contained change — no separate "final deletion" or "integration" ticket: 1. **Scaffold** the module (its own file/dir, TypeScript from the start). 2. **Port** the bodies in, unchanged in behavior. diff --git a/domains/typescript/skills/derive-types/skill.md b/domains/typescript/skills/derive-types/skill.md index 84bdbe7c..242238d3 100644 --- a/domains/typescript/skills/derive-types/skill.md +++ b/domains/typescript/skills/derive-types/skill.md @@ -20,31 +20,44 @@ An ad-hoc type — one hand-defined to describe a value an authoritative type al ## A grounded example (`metamask-extension` #42583) -A `wallet-services` module hand-wrote a slice of `NetworkController` state instead of deriving it. +A `wallet-services` module hand-rolled a messenger type, re-declaring each controller action's signature and **hand-copying its return shape** inline. -🚫 Re-declared — every field optional (wider than the real, _required_ field), keyed by `string` not `Hex`, and unlinked from the source, so it drifts silently when the controller changes: +🚫 Reinvents the controller's messenger and re-states its action returns: ```typescript -type NetworkControllerState = { - networkConfigurationsByChainId?: Record< - string, - { - defaultRpcEndpointIndex?: number; - rpcEndpoints?: { networkClientId?: string }[]; - } +type TokenResolutionMessenger = { + call( + action: 'AssetsContractController:getTokenStandardAndDetails', + address: string, + // … + ): Promise< + | { + balance?: string | number | bigint | { toString(radix?: number): string }; + decimals?: string | number | bigint | { toString(radix?: number): string }; + standard?: string; + symbol?: string; + } + | undefined >; + // the other action's return is discarded entirely: + call(action: 'AssetsContractController:getBalancesInSingleCall' /* … */): Promise; }; ``` -✅ Derived — tracks the authoritative shape (`Record`), narrowed to the one field in use: +✅ Derive each return from the controller's exported action type; don't hand-copy it: ```typescript -import type { NetworkState } from '@metamask/network-controller'; +import type { AssetsContractControllerGetTokenStandardAndDetailsAction } from '@metamask/assets-controllers'; -type NetworkConfigurations = NetworkState['networkConfigurationsByChainId']; +// the action already types its own return — derive it +type TokenDetails = ReturnType< + AssetsContractControllerGetTokenStandardAndDetailsAction['handler'] +>; ``` -A too-wide copy does not save work; it moves the work downstream. The same PR typed a dependency as `getMetaMaskState: () => Record`, so every consumer then had to re-cast the shape back by hand — including a `{ metamask: getMetaMaskState() } as never` double-cast. Deriving that dependency from the authoritative state type deletes the casts. The same file _did_ derive one type correctly (`type Action = (typeof ACTIONS)[number]`), so the pattern was already in hand; the discipline is extending it to every referenced type. +The messenger itself should extend the controller's `RestrictedMessenger` parameterized with those exported action types, so every `call` signature comes from the controller rather than a hand-rolled overload. The hand-rolled version is worse than a plain duplicate: one return is hand-copied (already looser than the controller's real type), the other (`Promise`) discards the type entirely. + +The same PR also typed a dependency `getMetaMaskState: () => Record`, which forced every consumer to re-cast the shape by hand — including a `{ metamask: getMetaMaskState() } as never` double-cast. That downstream cast tax is what a too-wide type always imposes; deriving the dependency from the authoritative state type deletes it. Notably the same file _did_ derive one type correctly (`type Action = (typeof ACTIONS)[number]`), so the pattern was already in hand — the discipline is extending it to every referenced type. ## Rule diff --git a/domains/typescript/skills/migration-context-cost/skill.md b/domains/typescript/skills/migration-context-cost/skill.md index 9ff9b6d6..ea1ca3b4 100644 --- a/domains/typescript/skills/migration-context-cost/skill.md +++ b/domains/typescript/skills/migration-context-cost/skill.md @@ -1,27 +1,26 @@ --- name: migration-context-cost -description: A file's JS→TS migration cost is dominated by context fan-in (upstream files whose source types to derive from) + fan-out (downstream files that import it and must update), not its line count. Scope and sequence migration tickets by this cost. +description: A file's JS→TS migration cost is driven mostly by context fan-in (upstream types it must read to derive from) and fan-out (downstream files that import it and must be updated), not by its line count. Scope and sequence migration tickets by it. maturity: experimental --- # TypeScript Migration Context Cost — Fan-In and Fan-Out -The cost of converting a file to TypeScript is not its line count. It is the **context the conversion pulls in** — everything the converter, human or AI, must load to do it correctly. +A file's line count is a factor in how hard it is to convert to TypeScript, but usually not the dominant one. The dominant cost is the **context the conversion pulls in** — everything the converter, human or AI, must read or touch to do it correctly. -## The two axes +## The two axes are different kinds of cost -- **Fan-in (upstream source types).** To type the file's values correctly you must load every upstream module whose types the code should _derive_ from — controller state, function returns, library exports, schemas (see `derive-types`). A file that touches many upstream types has high fan-in even if it is short. -- **Fan-out (downstream consumers).** Changing the file's types ripples to every module that imports it; each may need its own update or re-type. A widely-imported file has high fan-out even if it is small. +- **Fan-in (upstream source types) — a _reading_ cost.** To type the file's values correctly you must load every upstream module whose types the code should _derive_ from: controller state, function returns, library exports, schemas (see `derive-types`). You read these; you do not change them, and they are usually already TypeScript. A file that derives from many sources is expensive to hold in context even if it is short. +- **Fan-out (downstream consumers) — a _change_ cost.** Re-typing the file ripples to every module that imports it; each may need its own update. These are files you actually edit, so fan-out — together with the file itself — is the **change and review surface**. A widely-imported file has a large change surface even if it is small. -The real work — and the real review surface — is the sum of these, not the target file's length. A 200-line hub imported by 100 files can be a larger migration than a self-contained 2,000-line leaf. +The two are not the same kind of cost: fan-in is what you must _read_ to get the types right; fan-out is what you must _modify_. A 200-line hub imported by 100 files can be a larger migration than a self-contained 2,000-line leaf. ## Why it matters even for AI -Single-pass conversion of a high-fan-in/fan-out file is impractical even for a capable AI: it must hold the target file in context **and** progressively load every upstream source-type file **and** every downstream consumer. That context fan-in/fan-out is the binding constraint, not the model's ability to read the file itself. +Single-pass conversion of a high-fan-in/fan-out file is impractical even for a capable AI: it must hold the target file in context **and** read every upstream source-type file **and** edit every downstream consumer. That context load plus change surface — not the model's ability to read the file itself — is the binding constraint. ## How to use it -- **Scope tickets by context cost, not LOC.** Estimate fan-in (upstream types the file derives from) + fan-out (`grep -rl` importer count) before sizing a migration ticket. Size by the import-rewrite / re-type surface, not the line count. -- **Sequence low-cost first.** Convert leaf / low-fan-out files early (few downstream updates), and files whose upstream types are already TypeScript (low fan-in), so later conversions have more typed ground to derive from. -- **When the cost won't fit one PR, reduce it before converting.** High fan-in → the upstream types may need to land first. High fan-out through a hub → decompose the hub into coherent units so each unit's fan-out is bounded (see `decompose-large-files`). Decomposition is one response to high context cost — not the only place the cost applies. -- **Keep the fan-out map honest.** A barrel / re-export file inflates apparent fan-out and hides real dependencies; importing from the actual source file (not a barrel) keeps the dependency graph — and the cost estimate — accurate. +- **Scope tickets by context cost, not LOC.** Before sizing a migration ticket, estimate fan-in (how many upstream types it must derive from) and fan-out (`grep -rl` importer count). Size by the read-context plus the change surface, not the line count. +- **Sequence low-fan-out first.** Convert leaf / low-fan-out files early — few downstream edits per PR. Files whose upstream types are already TypeScript are cheaper on the fan-in side and give later conversions more typed ground to derive from. +- **When the change surface won't fit one PR, reduce it first.** A hub with high fan-out is the case to decompose: split it into coherent units so each unit's fan-out — and thus each PR — is bounded (see `decompose-large-files`). Decomposition is one response to high context cost, not the only place the cost applies. From 801758fa0f682296ac676f1e1b90a1dbd69622f0 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Fri, 17 Jul 2026 06:50:52 -0400 Subject: [PATCH 013/135] Reframe decompose how-to: identifying the boundaries is the key; extraction is optional --- domains/typescript/skills/decompose-large-files/skill.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/domains/typescript/skills/decompose-large-files/skill.md b/domains/typescript/skills/decompose-large-files/skill.md index f8a13708..dc1ccbf5 100644 --- a/domains/typescript/skills/decompose-large-files/skill.md +++ b/domains/typescript/skills/decompose-large-files/skill.md @@ -26,9 +26,13 @@ Extraction is worth it only when the extracted piece is a **coherent unit that c Some code should **stay** in the original file: the thin **composition / bootstrap root** that wires the modules together. Extracting the wiring itself fragments rather than clarifies — it is the one place the whole is assembled. The tell is code that references _everything_ (the central object plus the shared mutable state every cluster reads); relocating it behind a wide "params bag" just moves the tangle. -## How to convert one unit (self-contained, per #41735) +## Identifying the boundaries is the key — extraction is optional -The extraction is not the point — **converting the file to TypeScript in small, self-contained units is.** Each unit is a piece you can type and review on its own instead of holding the whole file at once; pulling it into its own module is just what makes that unit self-contained and independently convertible. Each unit is one self-contained change — no separate "final deletion" or "integration" ticket: +**The key step is identifying the coherent boundaries.** Once you know them, you can convert the file to TypeScript in small, self-contained units — type and review one cluster at a time — instead of holding the whole file at once. That is true whether or not you physically move anything: the boundary map is what makes incremental conversion possible, and documenting it (one unit per ticket) is the deliverable. + +**Extraction — moving a unit into its own module — is optional.** It buys modularity, reviewability, and a bounded per-unit change surface, so it is often worth doing, but you convert in units because you identified the boundaries, not because you moved the code. Extract where the move adds value; leave a cluster in place where it doesn't. + +When you *do* extract a unit, it is one self-contained change — no separate "final deletion" or "integration" ticket: 1. **Scaffold** the module (its own file/dir, TypeScript from the start). 2. **Port** the bodies in, unchanged in behavior. From 71bf451752a1b50e4fab8a5ddac05189568e3450 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Mon, 27 Jul 2026 09:58:25 -0400 Subject: [PATCH 014/135] feat(analytics): add grafana-tempo-queries skill --- .../skills/grafana-tempo-queries/skill.md | 125 ++++++++++++++++++ 1 file changed, 125 insertions(+) create mode 100644 domains/analytics/skills/grafana-tempo-queries/skill.md diff --git a/domains/analytics/skills/grafana-tempo-queries/skill.md b/domains/analytics/skills/grafana-tempo-queries/skill.md new file mode 100644 index 00000000..cc720dbf --- /dev/null +++ b/domains/analytics/skills/grafana-tempo-queries/skill.md @@ -0,0 +1,125 @@ +--- +name: grafana-tempo-queries +description: Query backend traces in Grafana Tempo with TraceQL — find traces by service or span attribute, fetch a trace by id, inspect its span tree, and enumerate tag values. Covers the datasource-proxy access path, the credential-expiry failure that returns empty results indistinguishable from "no data", the negative control that proves a filter actually applied, and the id/kind/base64 decoding quirks in the response. Use when investigating backend latency, checking what the backend recorded for a request, or establishing which infrastructure tiers a trace reaches. Triggers on Tempo, TraceQL, Grafana traces, backend span inspection, "does the backend have this trace", or tracing a request past the API boundary. +maturity: experimental +--- + +# grafana-tempo-queries + +Tempo holds **backend** spans. Client spans from the extension and mobile go to Sentry via the SDK's own transport and never appear here — so a Tempo trace normally starts at an inbound server span, and a missing root is expected rather than broken. To join the two halves, see `sentry-grafana-cross-ref`. + +## Setup + +Everything goes through Grafana's datasource proxy, so a Grafana session is the only credential needed. Keep the host, datasource uid, org id, and session in your environment — this repository is public, so never commit them. + +```bash +# Set these once per shell, from your own Grafana instance: +# GRAFANA_HOST e.g. https://grafana. +# TEMPO_UID the Tempo datasource uid (see discovery below) +# GRAFANA_ORG the numeric org id the datasource belongs to +# GRAFANA_SESSION value of the grafana_session cookie from an authenticated browser +BASE="$GRAFANA_HOST/api/datasources/proxy/uid/$TEMPO_UID" +AUTH=(-H "Cookie: grafana_session=$GRAFANA_SESSION" -H "X-Grafana-Org-Id: $GRAFANA_ORG") +``` + +Discover the datasource uid rather than guessing it: + +```bash +curl -s "$GRAFANA_HOST/api/datasources" "${AUTH[@]}" \ + | node -e 'JSON.parse(require("fs").readFileSync(0)).filter(d=>d.type==="tempo").forEach(d=>console.log(d.uid,d.name))' +``` + +## Check the instrument before believing a result + +**A stale session returns HTTP 401 with an empty body, and a naive parser reports that as zero results** — indistinguishable from "this data does not exist". This is the single most expensive failure mode here: it produces confident negative conclusions about instrumentation coverage. + +```bash +# 1. Prove you are authenticated. Do this first, every session. +curl -s -o /dev/null -w 'grafana auth: HTTP %{http_code}\n' "$GRAFANA_HOST/api/user" "${AUTH[@]}" + +# 2. Prove the filter is actually being applied, with a query that must match nothing. +curl -s -G "$BASE/api/search" "${AUTH[@]}" \ + --data-urlencode 'q={span.db.system = "not-a-real-db-xyz"}' \ + --data-urlencode "start=$START" --data-urlencode "end=$NOW" \ + | node -e 'const j=JSON.parse(require("fs").readFileSync(0));console.log("control traces:",(j.traces||[]).length,"(must be 0)")' +``` + +If several different filters all return exactly your `limit`, the filter is not being applied — treat the results as unfiltered until the negative control returns 0. + +## Core queries + +Every endpoint wants an explicit epoch-seconds window. Omitting it on a by-id lookup makes the request hunt across all blocks and hit a context deadline. + +```bash +NOW=$(date +%s); START=$((NOW-3600)) +``` + +**Search by TraceQL.** Returns trace summaries plus the spans that matched. + +```bash +curl -s -G "$BASE/api/search" "${AUTH[@]}" \ + --data-urlencode 'q={resource.service.name="my-service"}' \ + --data-urlencode "start=$START" --data-urlencode "end=$NOW" \ + --data-urlencode "limit=20" +``` + +**Fetch one trace in full** (OTLP JSON: resource batches → scope spans → spans). + +```bash +curl -s "$BASE/api/traces/$TRACE_ID?start=$START&end=$NOW" "${AUTH[@]}" +``` + +**Enumerate values for a tag** — useful for inventorying what a fleet emits. Expect a `502` on high-cardinality tags; fall back to inspecting individual traces rather than concluding the tag is unused. + +```bash +curl -s -G "$BASE/api/v2/search/tag/span.db.system/values" "${AUTH[@]}" \ + --data-urlencode "start=$START" --data-urlencode "end=$NOW" +``` + +## TraceQL patterns worth knowing + +| Goal | Query | +| --- | --- | +| One service | `{resource.service.name="svc-name"}` | +| Several services | `{resource.service.name=~"(svc-a|svc-b)-prd"}` | +| Attribute present at all | `{span.db.system != nil}` | +| Span kind | `{kind=server}`, `{kind=client}` | +| Slow spans | `{duration > 1s}` | +| **Two conditions anywhere in the same trace** | `{resource.service.name="svc-a"} && {span.db.system != nil}` | + +The last one is the important one: `&&` between two brace groups is a **trace-level** conjunction, not a single-span filter. It answers "does a request into this service reach a database at all", which is how you map how deep a trace goes without reading traces one at a time. + +## Reading the response + +- **Span and trace ids are base64**, not hex. Decode before comparing them to anything from a header or from Sentry: `Buffer.from(id,"base64").toString("hex")`. +- **`kind` is a string** (`SPAN_KIND_SERVER`, `SPAN_KIND_CLIENT`, `SPAN_KIND_INTERNAL`), not the numeric enum. Filtering on `sp.kind === 2` silently matches nothing. +- **Search results drop leading zeros from trace ids.** A 31-character id is a 32-character id with a leading zero; zero-pad before using it anywhere else, or the lookup fails for a reason that looks like absence. +- **`rootServiceName: ""`** means the trace's root is not in Tempo. For client-originated requests that is the normal case — the root is a client span living in Sentry — and it is the marker for finding them. +- Resource attributes carry deployment context (`service.name`, kubernetes pod/namespace/cluster, region); span attributes carry the request (`http.*`, `net.*`, `db.*`). + +## Deep links for sharing + +A link is more useful than a pasted id. Build a Grafana Explore URL with the query pre-filled: + +```bash +node -e ' +const left={datasource:process.env.TEMPO_UID, + queries:[{refId:"A",datasource:{type:"tempo",uid:process.env.TEMPO_UID},queryType:"traceql",query:process.argv[1]}], + range:{from:"now-6h",to:"now"}}; +console.log(`${process.env.GRAFANA_HOST}/explore?orgId=${process.env.GRAFANA_ORG}&left=${encodeURIComponent(JSON.stringify(left))}`); +' '' +``` + +Prefer an absolute `from`/`to` when the link needs to outlive the event; a relative window slides off it and the reader opens an empty result. + +## Failure modes + +| Symptom | Cause | Response | +| --- | --- | --- | +| All queries return 0 | Session expired (401, empty body) | Check `/api/user` first | +| Every filter returns exactly `limit` | Filter not applied | Run the negative control | +| By-id lookup times out | No time window | Pass `start`/`end` | +| Tag-values returns 502 | High cardinality | Inspect traces directly | +| Id from search not found elsewhere | Leading zeros stripped | Zero-pad to 32 chars | +| Kind filter matches nothing | Comparing to a number | Compare to `SPAN_KIND_*` | +| Trace has no root | Root is a client span | Expected; see `sentry-grafana-cross-ref` | From 5590e2d661d8c82df6f3584167738ae9995971f4 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Mon, 27 Jul 2026 09:58:26 -0400 Subject: [PATCH 015/135] feat(analytics): add sentry-grafana-cross-ref skill --- .../skills/sentry-grafana-cross-ref/skill.md | 100 ++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 domains/analytics/skills/sentry-grafana-cross-ref/skill.md diff --git a/domains/analytics/skills/sentry-grafana-cross-ref/skill.md b/domains/analytics/skills/sentry-grafana-cross-ref/skill.md new file mode 100644 index 00000000..56e8c327 --- /dev/null +++ b/domains/analytics/skills/sentry-grafana-cross-ref/skill.md @@ -0,0 +1,100 @@ +--- +name: sentry-grafana-cross-ref +description: Join one trace across Sentry and Grafana Tempo by trace id to see the whole client-to-backend path, and diagnose why a half is missing. Covers the split-store model (client spans reach Sentry through the SDK and survive only head sampling; backend spans reach Tempo through tail sampling and Sentry through environment routing), the classification of both-halves / client-only / backend-only outcomes with the sampling and routing rule that causes each, and the id-padding, time-window, and query-syntax traps that make a present trace look absent. Use when a trace looks truncated, a backend span has no parent, per-hop latency needs attributing across the seam, or you need to know which store should hold a given span. Triggers on cross-stack trace, orphaned span, trace id lookup, client-backend correlation, split waterfall, or "where did the rest of the trace go". +maturity: experimental +--- + +# sentry-grafana-cross-ref + +One request produces spans in two stores, joined only by `trace_id`. Reading a trace end to end means querying both and knowing which absences are expected. + +Prerequisite: `grafana-tempo-queries` for the Tempo side, `sentry-mcp-queries` for richer Sentry work. + +## The model — what lands where, and why a half goes missing + +| Span | Reaches | Gated by | +| --- | --- | --- | +| Client (`pageload`, `navigation`, `http.client`, custom) | Sentry, via the SDK transport | the client's `tracesSampleRate` head decision | +| Backend (`http.server`, internal, db, messaging) | Tempo, via the collector | collector tail-sampling policy | +| Backend, additionally | Sentry, if the collector forwards it | an environment attribute on the span matching a routing policy | + +Three consequences drive every diagnosis below: + +- **The client's sampled flag and the client's own retention are separate decisions.** The propagated `traceparent` flag tells the backend whether to record; the client's head sampling decides whether the client span is kept. When the flag says record and head sampling drops the client span, the backend records a span whose parent was never stored anywhere — an orphan. This is the normal case at low client sample rates, not an anomaly. +- **A `-00` (not-sampled) flag can suppress the backend span entirely**, because a parent-respecting sampler delegates to "never record" for an unsampled remote parent. No backend span is created at all — different from one being dropped later. +- **Backend spans only reach Sentry if their environment attribute matches a routing policy.** A service that expresses environment under a different attribute name matches nothing and is silently absent from Sentry while still present in Tempo. + +## Setup + +Keep organisation slugs, project ids, and hosts in your environment — do not commit them. + +```bash +# SENTRY_ORG, SENTRY_PROJECT_ID (numeric), SENTRY_AUTH_TOKEN +# plus the grafana-tempo-queries variables for the Tempo side +``` + +## Query the Sentry half + +```bash +curl -fsS -G "https://sentry.io/api/0/organizations/$SENTRY_ORG/events/" \ + -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \ + --data-urlencode "dataset=spans" \ + --data-urlencode "field=span.op" --data-urlencode "field=span_id" \ + --data-urlencode "field=parent_span" --data-urlencode "field=span.description" \ + --data-urlencode "field=timestamp" \ + --data-urlencode "query=trace:$TRACE_ID" \ + --data-urlencode "project=$SENTRY_PROJECT_ID" \ + --data-urlencode "statsPeriod=24h" \ + --data-urlencode "sort=-timestamp" +``` + +Two syntax traps that produce misleading emptiness: + +- **Any field you sort on must also be selected.** Sorting by `-timestamp` without requesting `timestamp` returns `400 orderby must also be in the selected columns or groupby` — and a script that swallows errors reports it as no results. +- **`has:parent_span` is not valid**; request `parent_span` as a field and filter client-side. + +Use `project=-1` to search every project at once when you do not yet know which one should hold the span — that is how you tell "in the wrong project" apart from "absent". + +## Procedure — Tempo to Sentry + +Use when you have a backend trace and want its client context. + +1. Find client-originated backend traces: search your services in Tempo, then keep the results whose `rootServiceName` reports the root was never received. Those reference a client parent that is not in Tempo. +2. **Zero-pad each trace id to 32 characters** before querying Sentry. Tempo search strips leading zeros, and an unpadded id returns nothing for a reason that looks like absence. +3. Query Sentry for `trace:`, first in the client's project, then with `project=-1`. +4. Classify with the table below. + +## Procedure — Sentry to Tempo + +Use when a Sentry trace looks truncated at the network boundary. + +1. Take the trace id from the Sentry trace view. +2. Look for a matching `http.server` span in Sentry itself first — if the collector forwards backend spans for that environment, both halves may already be in one place and no cross-store hop is needed. +3. Otherwise fetch the trace from Tempo by id, with a time window that brackets the client span's timestamp. +4. If Tempo has nothing, the backend either never recorded it (a `-00` flag), or its trace fell outside the tail-sampling policy. + +## Classification + +| What you find | Meaning | Where to look next | +| --- | --- | --- | +| Client and backend spans, backend parented on the client's request span | Healthy join; per-hop latency is attributable | — | +| Client and backend spans, backend parented on an enclosing operation root | Propagation is attaching the wrong parent, so the backend span sits beside its caller instead of beneath it | The client's header-injection path | +| Backend spans only, client parent referenced but nowhere | Orphan: the flag instructed recording, head sampling discarded the client span | Client sample rate, or decoupling the flag from head sampling | +| Client spans only, no backend span anywhere | Either no header was propagated to that host, or the flag was `-00` so the backend never created a span | Propagation targets, then the flag | +| Backend in Tempo but not in Sentry when it should be | Environment attribute does not match a forwarding policy | The service's environment tagging | +| Nothing in either store | Head-sampled out end to end | Expected at low sample rates | + +## Checking whether a backend span nests correctly + +The parent identity, not the picture, is what determines nesting. Take the backend `http.server` span's `parentSpanId` (hex-decode it from Tempo's base64), then look that id up among the client's spans in Sentry: + +- Resolves to an `http.client` span whose description matches the same URL → correctly nested beneath the request that caused it. +- Resolves to a transaction root or custom operation span → the backend span is a sibling of its caller; hop latency cannot be read off the waterfall. +- Resolves to nothing in either store → orphan. + +## Traps + +- **Time windows differ per store.** Tempo retention is typically much shorter than Sentry's, so an older trace legitimately exists in one and not the other. Confirm the window before concluding a half is missing. +- **Verify credentials on both sides first.** An expired Grafana session and an out-of-scope Sentry token both present as empty results, which reads as a real finding about instrumentation. +- **A relative time window on a shared link expires.** Pin absolute ranges when the link needs to outlive the incident. +- **One sampling decision can be shared across a long-lived trace id.** If a client reuses a trace id across many operations, the proportion of spans marked sampled will not match the nominal client rate; do not read that ratio as an effective sample rate. From 4d0aab2c7f3b9c9c703da0a77783f0cb5f502ae2 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Thu, 30 Jul 2026 08:00:51 -0400 Subject: [PATCH 016/135] feat(coding): add memory-leak-hunt skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two-phase retention review for JavaScript/TypeScript. Phase 1 is a static read of a diff: enumerate the retention primitives the change introduces — listeners, timers, pending-request registries, subscriptions, module singletons, growing collections — and pair every acquire with its release site. A primitive with a teardown is safe; one without is the finding. Phase 2 escalates to DevTools/CDP heap snapshots only for a primitive the read cannot pair. Leading with the read rather than the instrument settles most leak claims without ever taking a snapshot. --- .../references/heap-investigation.md | 67 ++++++++ .../scripts/heap-over-cycles.example.ts | 54 ++++++ .../scripts/retention-scan.py | 62 +++++++ .../coding/skills/memory-leak-hunt/skill.md | 155 ++++++++++++++++++ 4 files changed, 338 insertions(+) create mode 100644 domains/coding/skills/memory-leak-hunt/references/heap-investigation.md create mode 100644 domains/coding/skills/memory-leak-hunt/scripts/heap-over-cycles.example.ts create mode 100644 domains/coding/skills/memory-leak-hunt/scripts/retention-scan.py create mode 100644 domains/coding/skills/memory-leak-hunt/skill.md diff --git a/domains/coding/skills/memory-leak-hunt/references/heap-investigation.md b/domains/coding/skills/memory-leak-hunt/references/heap-investigation.md new file mode 100644 index 00000000..f2a25950 --- /dev/null +++ b/domains/coding/skills/memory-leak-hunt/references/heap-investigation.md @@ -0,0 +1,67 @@ +# Phase 2 — runtime investigation + +Reach for this **only** when Phase 1 (static pairing) leaves an introduced primitive it +cannot pair, or when the claim is explicitly about magnitude ("retained heap grows across N +cycles", "detached nodes accumulate"). A snapshot confirms a *suspected* leak and shows its size and retainer chain — but a +non-leak is also worth demonstrating: a flat retained-heap curve across N cycles is positive +evidence, valid **only beside a positive control** (a known-leaking arm that grows), because a +measurement that cannot detect a leak cannot prove its absence. See `scripts/heap-over-cycles.example.ts` +for a two-arm driver (real code flat vs control grows) over the real module. + +## Order of escalation (cheapest first) + +### 1. Falsifying lifecycle test (preferred) + +Deterministic, fast, and it lives in the suite as a regression guard. Force the boundary the +primitive should release at, then assert the release directly: + +- listener: assert `emitter.listenerCount(ev)` returns to its pre-acquire value after the + boundary (stream close, `destroy()`, instance replacement). +- singleton / cache: assert the reference is nulled / the entry evicted. +- pending registry: assert the map is empty after the flow (all requests settled or rejected). +- subscription: assert the unsubscribe was called (spy) and no further dispatches land. + +The test **fails on the leaking code and passes on the fix** — that falsifiability is the +point. A test that passes on both proves nothing. + +### 2. Heap-over-a-flow (when a unit test can't reach it) + +One snapshot shows occupancy, not a leak. You need the **delta across repetition**: + +1. Drive the flow once to warm caches; take a baseline snapshot. +2. Run N cycles of the suspected flow (open/close, mount/unmount, connect/disconnect). +3. Force GC, take a second snapshot. +4. Compare **retained size**, **detached DOM nodes**, and **listener count** — a leak grows + roughly linearly in N. A flat delta refutes the leak. + +Capture (Chrome, extension context): +- DevTools Memory panel → *Allocation instrumentation on timeline* or two heap snapshots + with *Comparison* view; or +- CDP: `HeapProfiler.takeHeapSnapshot` before/after, diff the node counts. `mm cdp` drives + the extension's contexts (page, service worker) over the protocol. + +### 3. Retainer graph — must match the static argument + +Select a surviving object in the post-flow snapshot and read its **retainer chain** (why it +is still reachable). That chain must name the **same holder → held → boundary** the Phase-1 +read named. If the profiler says the object is retained by a path the static argument did not +predict, the static argument is incomplete — reconcile before concluding. Agreement between +the two independently-derived paths is what makes the finding trustworthy; either alone is +weaker. + +## Trust gate + +- **A single snapshot is not evidence of a leak** — it is occupancy. Only the delta across N + cycles is. +- **GC must be forced** before the comparison snapshot, or you measure collection lag, not + retention. +- **The retainer chain is the discriminator** — "retained size went up" without a chain + naming the culprit is a symptom, not a diagnosis. +- **Warm the caches first** — the first cycle populates legitimate one-time caches that would + otherwise read as a leak. + +## Scrub before sharing + +Heap snapshots and retainer graphs can contain live application state (URLs, account +identifiers, in-flight request payloads). Scrub or crop before any snapshot leaves the +machine; never attach a raw `.heapsnapshot` to a public surface. diff --git a/domains/coding/skills/memory-leak-hunt/scripts/heap-over-cycles.example.ts b/domains/coding/skills/memory-leak-hunt/scripts/heap-over-cycles.example.ts new file mode 100644 index 00000000..f302ae67 --- /dev/null +++ b/domains/coding/skills/memory-leak-hunt/scripts/heap-over-cycles.example.ts @@ -0,0 +1,54 @@ +// Phase-2 runtime evidence — PR #40684 introduced pending-request Map does not leak. +// Drives the REAL PatchStoreSubstreamConnection over N cycles, measures retained V8 +// heap. A no-leak result is meaningful only beside a control that grows: ARM B drains +// the request stream but withholds responses, so entries accumulate. +import v8 from 'node:v8'; +import ObjectMultiplex from '@metamask/object-multiplex'; +import { PATCH_STORE_SUBSTREAM_METHODS } from '../../shared/constants/patch-store-substream-methods'; +import { PatchStoreSubstreamConnection } from './patch-store-substream-connection'; + +function pair() { + const uiMux = new ObjectMultiplex(); const bgMux = new ObjectMultiplex(); + uiMux.pipe(bgMux).pipe(uiMux); + return { uiStream: uiMux.createStream('patch-store'), bgStream: bgMux.createStream('patch-store') }; +} +const flush = () => new Promise((r) => setImmediate(r)); +function usedMB() { global.gc!(); global.gc!(); return v8.getHeapStatistics().used_heap_size / 1048576; } +const N = 100000; + +async function main() { + console.log(`PR #40684 · PatchStoreSubstreamConnection · pending-request Map · ${N} request cycles`); + console.log('='.repeat(74)); + + // ARM A — head code: every request answered → entry .delete on response + { const { uiStream, bgStream } = pair(); + bgStream.on('data', (m: any) => { if (m?.method === PATCH_STORE_SUBSTREAM_METHODS.GetStatePatches) bgStream.write({ id: m.id, jsonrpc: '2.0', result: [] }); }); + const conn = new PatchStoreSubstreamConnection(uiStream, { handleSendUpdate: () => undefined }); + let got = 0; await conn.getStatePatches(); + const before = usedMB(); + for (let i = 0; i < N; i++) { const r = await conn.getStatePatches(); got += r.length === 0 ? 1 : 0; } + await flush(); + const after = usedMB(); + console.log(`\nARM A head code — all ${N} requests answered (${got} responses consumed)`); + console.log(` retained heap ${before.toFixed(1)} -> ${after.toFixed(1)} MB Δ ${(after - before >= 0 ? '+' : '') + (after - before).toFixed(1)} MB ── FLAT`); + console.log(` every .set(id) on request is matched by .delete(id) on response; the Map returns to empty`); + } + + // ARM B — control: requests consumed but never answered → Map accumulates N entries + { const { uiStream, bgStream } = pair(); + bgStream.on('data', () => { /* consume the request, send no response */ }); + const conn = new PatchStoreSubstreamConnection(uiStream, { handleSendUpdate: () => undefined }); + const held: Promise[] = []; + const before = usedMB(); + for (let i = 0; i < N; i++) held.push(conn.getStatePatches().catch(() => {})); + await flush(); + const after = usedMB(); + console.log(`\nARM B control — same code, ${N} requests, none answered (${held.length} promises pending)`); + console.log(` retained heap ${before.toFixed(1)} -> ${after.toFixed(1)} MB Δ +${(after - before).toFixed(1)} MB ── GROWS`); + console.log(` the .set(id) has no matching .delete; entries pile up — proving the measurement catches a leak`); + } + + console.log(`\nVERDICT: the pending-request Map introduced by #40684 does not retain across ${N} cycles —`); + console.log(` confirmed at runtime, beside a control that does. The static pairing is corroborated, not merely asserted.`); +} +main().then(() => process.exit(0)); diff --git a/domains/coding/skills/memory-leak-hunt/scripts/retention-scan.py b/domains/coding/skills/memory-leak-hunt/scripts/retention-scan.py new file mode 100644 index 00000000..aad49597 --- /dev/null +++ b/domains/coding/skills/memory-leak-hunt/scripts/retention-scan.py @@ -0,0 +1,62 @@ +#!/usr/bin/env python3 +"""Retention review, scoped to a diff. For each file+patch: find every retention +primitive, pair each acquire with its release IN THE SAME FILE, and mark each +NEW (line appears in the patch's added lines) or pre-existing. Charge only NEW +un-paired primitives; report pre-existing separately. Re-runnable: inputs are +the head files and the PR patch, both fetched from the repo by ref.""" +import re, sys + +def added_lines(patch_path): + out=set() + try: + for l in open(patch_path): + if l.startswith('+') and not l.startswith('+++'): + out.add(l[1:].strip()) + except FileNotFoundError: + pass + return out + +def scan(src_path, patch_path): + src=open(src_path).read(); lines=src.split('\n'); added=added_lines(patch_path) + rows=[] + # listeners: pair .on(ev,handler) with .removeListener(ev,handler) + for m in re.finditer(r'(\w+)\.(?:on|addListener)\(\s*[\'"](\w+)[\'"]\s*,\s*(\w+)', src): + emitter,ev,handler=m.groups(); ln=src[:m.start()].count('\n')+1 + acquire_line=lines[ln-1].strip() + new = acquire_line in added + rem=re.search(r'\.(?:removeListener|off)\(\s*[\'"]'+ev+r'[\'"]\s*,\s*'+handler, src) + if rem: + rln=src[:rem.start()].count('\n')+1 + ctx=src[max(0,rem.start()-140):rem.start()] + onclose='Closed' in ctx or 'close' in ctx + rows.append((new,'ok',f"{emitter}.on('{ev}', {handler})",ln, + f"removeListener L{rln}"+(" on stream close" if onclose else ""))) + else: + rows.append((new,'OPEN',f"{emitter}.on('{ev}', {handler})",ln,"no removeListener in file")) + # pending registries: Map with set paired with delete + for m in re.finditer(r'(#?\w*[Pp]ending\w*|#?\w*[Rr]equests?\w*)\s*[=:][^\n]*new Map', src): + name=m.group(1); ln=src[:m.start()].count('\n')+1 + new=lines[ln-1].strip() in added + setm=re.search(re.escape(name)+r'\.set\(', src); delm=re.search(re.escape(name)+r'\.delete\(', src) + if setm: + sln=src[:setm.start()].count('\n')+1 + if delm: + rows.append((new,'ok',f"{name} (.set L{sln})",ln,f".delete L{src[:delm.start()].count(chr(10))+1}")) + else: + rows.append((new,'OPEN',f"{name} (.set L{sln})",ln,"no .delete — entries accumulate")) + return rows + +print("RETENTION REVIEW — PR #40684, scoped to the diff (re-run: retention-scoped.py )") +print("="*74) +new_open=0 +for pair in sys.argv[1:]: + f,patch=pair.split(':') + print(f"\n{f.split('/')[-1]}") + for new,mark,what,ln,status in sorted(scan(f,patch), key=lambda r:(not r[0], r[3])): + tag='NEW' if new else 'pre-exist' + if new and mark=='OPEN': new_open+=1 + print(f" [{tag:9}] {mark:4} L{ln}: {what}") + print(f" -> {status}") +print() +print("VERDICT:", "no retention path INTRODUCED — every NEW primitive is torn down; no heap snapshot warranted" + if new_open==0 else f"{new_open} NEW un-paired primitive(s) → escalate to a heap snapshot (Phase 2)") diff --git a/domains/coding/skills/memory-leak-hunt/skill.md b/domains/coding/skills/memory-leak-hunt/skill.md new file mode 100644 index 00000000..bdd9c312 --- /dev/null +++ b/domains/coding/skills/memory-leak-hunt/skill.md @@ -0,0 +1,155 @@ +--- +name: memory-leak-hunt +description: Find and investigate memory leaks / retention issues in JavaScript/TypeScript. Two phases. (1) Static identification from a diff — enumerate the retention primitives the change introduces (event listeners, timers, pending-request registries, subscriptions, module singletons, growing collections), pair every acquire with its release site, and scope findings to what the diff adds versus what pre-exists. (2) Runtime investigation, only for a primitive that cannot be paired statically — DevTools/CDP heap snapshots over N cycles, the retainer graph, detached-node count, and a falsifying lifecycle test. Leads with the cheap static read (the retention review a reviewer already performs) and escalates to a heap snapshot only where the read is inconclusive. Triggers on /memory-leak-hunt, or when asked to find or investigate a memory leak, check listener/subscription/timer cleanup, review a diff for retention, or take and read a heap snapshot. Callable by pr-validate as the engine behind its memory-leak evidence category. +maturity: experimental +--- + +# /memory-leak-hunt + +Find where an object outlives its purpose — and prove it, or prove it doesn't. A memory +leak is a **retention path**: something acquires a reference (a listener, a timer, a map +entry, a subscription) and never releases it at the boundary where it should +(`destroy()`, stream close, instance replacement, request completion). The object, and +everything its closure pins, survives past its lifecycle. + +**The core move — pair every acquire with its release.** For each retention primitive the +code introduces, find the matching teardown in the same scope. A primitive *with* a +teardown is safe. A primitive *without* one is the finding — and the only place a heap +snapshot could earn its cost. + +> **Lead with the read, not the instrument.** A heap snapshot is the *last* step, not the +> first. The decisive, cheap step is the read a reviewer already does: enumerate the +> primitives, pair each against its release. Escalate to the profiler only for a primitive +> the read cannot pair. Most leak claims are settled without ever taking a snapshot. + +## Phase 1 — Identification (static, from the diff) — the lead + +Enumerate the **retention primitives** the change introduces, and for each, name the +**holder → held set → outlived boundary** triple, then pair the acquire with its release. + +**The primitives to hunt** (each is an acquire that needs a matching release): + +| Primitive | Acquire | Release to pair it with | +|---|---|---| +| Event listener | `.on(ev, h)` · `addListener` · `addEventListener` | `removeListener(ev, h)` · `off` · `removeEventListener` — **same handler reference** | +| Timer | `setInterval` · recurring `setTimeout` | `clearInterval` · `clearTimeout` | +| Pending registry | `map.set(id, {resolve})` | `map.delete(id)` on **every** completion/close/error path | +| Subscription | `.subscribe()` · `messenger.subscribe` · store `subscribe` | the returned unsubscribe, called at teardown | +| Module singleton / cache | assignment to module/`this` scope | reset to `null` / eviction on replacement | +| Growing collection | `push` / `set` / `add` | a `drain` / `delete` / bounded eviction policy | + +**The three things to state per suspect:** +1. **Holder** — the primitive above. +2. **Held set** — the *specific* objects pinned. For a listener, list the closure's + captures (`outStream`, `api`, `messengerSubscription`…). Note when a closure links two + otherwise-independent objects' GC. +3. **Outlived boundary** — the moment release *should* happen but doesn't. + +**The pairing check is the finding.** The absence of the release, cited at the acquire +site, *is* the evidence. Cite it as `acquire L` with `no release in scope`, or as +`acquire L → release L (on )` when it is paired. + +**Four canonical leak shapes** (what an unpaired primitive usually is): +- **Unbounded accumulator** — a collection with a defeated or missing eviction, no drain. +- **Stale-instance listener** — on singleton replacement, the old instance's listeners + are never removed; both instances now receive dispatches. +- **Unremoved listener + capture set** — a listener whose handler closure pins a large set, + never removed, retained for the emitter's life. +- **Retention past `destroy()`** — teardown runs but misses one primitive. + +### Scope to the diff, or you invent findings + +Classify every flagged primitive as **introduced by this change** (in the added lines) vs +**pre-existing** (already in the file). Charge only the introduced ones. Report pre-existing +un-paired primitives **separately and uncharged** — flagging them is useful, but attributing +a pre-existing leak to the change under review is a false positive. (On MetaMask +extension#40684 the two new stream listeners each had a `removeListener` on +`onStreamClosed` and the new pending Map had its `.delete` — no leak introduced — while +three pre-existing un-torn-down listeners were surfaced and left uncharged, matching how the +reviewers treated them.) + +## Phase 2 — Investigation (runtime) — only for an unpaired primitive + +A snapshot is warranted **only** when Phase 1 finds an introduced primitive it cannot pair, +or when the claim is specifically about *magnitude* ("retained heap grows across N cycles"). +Full runtime procedure: **[references/heap-investigation.md](references/heap-investigation.md).** +In brief: + +- **Falsifying lifecycle test first** (cheaper than a snapshot, and deterministic): force the + boundary in a test, assert release — listener count returns to zero, singleton nulled, + collection drained. Fails on the leaking code, passes on the fix. +- **Heap-over-a-flow** when a test can't reach it: DevTools/CDP heap snapshots before and + after N cycles of the flow; compare **retained size** and **detached-node / listener + count**, not a single snapshot (one snapshot shows occupancy, not growth). +- **The retainer graph must name the same path** the static argument named. If the profiler's + retainer chain does not match the Phase-1 holder→held→boundary, one of them is wrong — + reconcile before concluding. +- **The intervention test carries causation.** Change *only* the one thing the retainer graph + named — the accessor, the missing teardown, the line — and re-measure. If the slope + flattens, the graph found the *cause*; if it persists, it found a correlate. A before/after + snapshot of unchanged code shows retention but never that *this* is what creates it. The fix + itself is the strongest form of this test. + +## Output + +Report the verdict scoped to the change, with each primitive shown paired or not: + +``` +Retention review — +NEW (introduced here): + ok → (on ) + OPEN → no release in scope ← heap-snapshot candidate +PRE-EXISTING (surfaced, not charged): + -- → no release (pre-existing) +Verdict: no retention path introduced | OPEN candidate warrants a snapshot (Phase 2) +``` + +Every figure resolves to a line number a reader can open. Present it in situ where possible +(the scan output, the failing lifecycle test, the retainer graph) rather than as prose. + +## Worked example — extension#40684 (extract patch-store substream) + +Phase 1 on the diff found three introduced primitives: +`outStream.on('data', handleIncomingMessage)` (L6881), `this.on('update', handleUpdate)` +(L6883), and a `#pendingGetStatePatchesRequests` Map (L49). Each paired: `removeListener` +at L6886/L6887 inside `onStreamClosed`, and `.delete` at L187 against the `.set` at L107. +**Verdict: no leak introduced — no snapshot taken.** The teardown at L6886 was the exact fix +a reviewer had suggested in-thread; the static read reproduced the review's conclusion. Three +pre-existing un-paired listeners were surfaced and left uncharged. + +## Worked example — extension#44352 (Firefox detached-window leak, a real leak) + +Phase 1 finds nothing to pair: the leak is not a listener, timer, or map the diff adds — it is +a *native object's* lifecycle. Snow's (pre-existing) picture-in-picture hook reads +`win.documentPictureInPicture.requestWindow` on every window it wraps; that property read +lazily instantiates a per-window `DocumentPictureInPicture`, and Firefox's cycle collector +cannot break its preserved-wrapper cycle — so every closed popup's document is retained. There +is no acquire/release in the changed lines to match, so the evidence is Phase 2 run forward: + +- **Magnitude, not a snapshot** — retained heap climbs ~105 MB (~70 detached windows) per popup + open/close, *linearly*; 30 cycles → 3.56 GB, and the detached documents survive a forced GC. + One snapshot shows occupancy; the slope across cycles is the leak. +- **Retainer graph** — names the holder (the per-window `documentPictureInPicture` instance) + and the boundary (window close, where the collector should reclaim it but can't). +- **Intervention test** — the fix reads the constructor prototype + `win.DocumentPictureInPicture.prototype.requestWindow` instead of the instance getter. No + per-window instance is created, the cycle never forms, the slope flattens. Changing *only* + the accessor the graph named — instance to prototype — and watching the growth vanish is what + proves the graph found the cause, not a correlate. A three-line patch to `@lavamoat/snow`; + linked issue #42891. + +**The lesson for the hunt:** a native-lifecycle leak — a property read that instantiates an +object the engine can't collect — is invisible to Phase 1 pairing, because there is no +acquire/release in the diff. When the claim is about *magnitude* and no diff primitive explains +it, go straight to Phase 2, and let the intervention test carry the causal claim. #44352 is the +Phase-2 counterpart to #40684: the same discipline that *proves the absence* of a leak +(#40684, the read settles it) *proves the presence and cause* of one here. + +## Called by pr-validate + +pr-validate keeps **memory leak** as an evidence category and delegates the analysis here: +it invokes this skill on the PR's diff, takes the verdict + the paired/unpaired sites, and +packages them as the category's evidence (an in-situ capture of the scan, plus the lifecycle +test or retainer graph if Phase 2 ran). This skill is the engine; pr-validate is the +orchestrator that publishes the result. Usable standalone for any leak hunt, in review or in +an incident, PR or not. From cfd6a37944c511ce3033d8cbbc81b41dcd8e0e1f Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Thu, 30 Jul 2026 08:03:07 -0400 Subject: [PATCH 017/135] feat(security): add security domain with supply-chain-audit and lavamoat-policy-diligence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two composing skills for "is this dependency change safe to take". `supply-chain-audit` is the breadth pass: Socket findings, `yarn npm audit` advisories, lockfile and manifest diffs, and the fronts no upstream scanner sees because they are things the repo does to its dependencies afterwards — yarn patches that modify dependency source at install, `resolutions` that force or stub versions, `npmAuditIgnoreAdvisories` suppression lists, CI actions riding mutable tags, and yarn plugins that execute at install. `lavamoat-policy-diligence` is the depth pass it delegates capability containment to. Because a LavaMoat policy is generated from a real run, every grant has a call site by construction — so "each addition is justified" is a tautology, not a finding. It instead reads each grant's use at the installed version to find its gate, and sorts into removable / removable-at-a-cost / load-bearing. Neither renders an accept/reject verdict; disposition belongs to the people who own the dependency. Adds a CODEOWNERS entry for the new domain, defaulted to the platform teams. --- .github/CODEOWNERS | 1 + .../scripts/policy-audit.py | 60 +++++ .../skills/lavamoat-policy-diligence/skill.md | 231 ++++++++++++++++++ .../skills/supply-chain-audit/skill.md | 158 ++++++++++++ 4 files changed, 450 insertions(+) create mode 100644 domains/security/skills/lavamoat-policy-diligence/scripts/policy-audit.py create mode 100644 domains/security/skills/lavamoat-policy-diligence/skill.md create mode 100644 domains/security/skills/supply-chain-audit/skill.md diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index f156c522..cfd79699 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -18,6 +18,7 @@ /domains/performance/ @MetaMask/extension-platform @MetaMask/mobile-platform /domains/perps/ @MetaMask/perps /domains/pr-workflow/ @MetaMask/extension-platform @MetaMask/mobile-platform +/domains/security/ @MetaMask/extension-platform @MetaMask/mobile-platform /domains/swaps/ @MetaMask/swaps-engineers /domains/testing/ @MetaMask/qa /domains/ui/ @MetaMask/design-system-engineers diff --git a/domains/security/skills/lavamoat-policy-diligence/scripts/policy-audit.py b/domains/security/skills/lavamoat-policy-diligence/scripts/policy-audit.py new file mode 100644 index 00000000..6c7331e8 --- /dev/null +++ b/domains/security/skills/lavamoat-policy-diligence/scripts/policy-audit.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python3 +"""Turn a LavaMoat policy base/head pair into a per-grant justification worklist. + +Detection is LavaMoat's job: `@metamaskbot update-policies` regenerates the policy from a +real run of the code and CI fails on drift. This script does NOT re-derive or classify that +diff — it enumerates every capability newly granted so each can be JUSTIFIED with a permalink +to the dependency's own source (accept), or REJECTED where no call site uses it. + +Usage: policy-audit.py +Falsifier: a listed grant for which no upstream call site can be found. +""" +import json +import sys + + +def resources(path): + with open(path) as f: + return json.load(f).get("resources", {}) + + +def newly_granted(head, base): + """Every (pkg, kind, capability) that is true in head and absent/false in base.""" + out = [] + for pkg, cfg in head.items(): + for kind in ("globals", "builtins", "packages"): + for cap, val in (cfg.get(kind) or {}).items(): + if val and not ((base.get(pkg, {}).get(kind) or {}).get(cap)): + out.append((pkg, kind, cap)) + return out + + +def main(): + if len(sys.argv) != 3: + sys.exit("usage: policy-audit.py ") + base = resources(sys.argv[1]) + head = resources(sys.argv[2]) + grants = sorted(newly_granted(head, base)) + + print("PER-GRANT JUSTIFICATION WORKLIST") + print("=" * 74) + print("Detection is LavaMoat's; each row below needs a REASON, not a category.") + print("Justify with a permalink to the dependency's source at the installed version") + print("(accept), or reject where no call site uses the capability.\n") + + if not grants: + print(" (no new grants between base and head — nothing to justify)") + return + + for pkg, kind, cap in grants: + print( + f" [ ] {pkg[:38]:40s} {kind[:3]}:{cap:22s}" + " reason: verdict: accept|REJECT" + ) + + print(f"\n {len(grants)} grant(s) to justify.") + print(" A grant with no locatable call site is the finding — reject it.") + + +if __name__ == "__main__": + main() diff --git a/domains/security/skills/lavamoat-policy-diligence/skill.md b/domains/security/skills/lavamoat-policy-diligence/skill.md new file mode 100644 index 00000000..a1e3491f --- /dev/null +++ b/domains/security/skills/lavamoat-policy-diligence/skill.md @@ -0,0 +1,231 @@ +--- +name: lavamoat-policy-diligence +description: Triage a LavaMoat policy change for least privilege — which newly granted capabilities can be dropped without breaking anything. Detection is delegated to `@metamaskbot update-policies` plus CI, and because the policy is generated from a real run, every grant has a call site by construction, so "each addition is justified" is a tautology and not the deliverable. Instead read each grant's use at the installed version to find its gate — a config flag nobody sets, an API nobody calls, a branch our payloads never take, an error-only path — and sort into removable / removable-at-a-cost / load-bearing, with the removal test (drop it, rebuild, run e2e) proposed for the policy owners to run. Lead with removal candidates and anything the reading turned up that bears on security; never render an accept/reject verdict, that call is the reviewer's. Hand it over untagged while the workflow is in trial. Triggers on /lavamoat-policy-diligence, or when asked about a LavaMoat policy grant, policy.json diff, capability containment, scuttling, allowScripts, or why a package needs a global/builtin. The specialized engine behind `supply-chain-audit`'s capability-containment lane. +maturity: experimental +--- + +# /lavamoat-policy-diligence + +Detection is not the job. LavaMoat already tells you which capabilities a dependency change +grants: `@metamaskbot update-policies` regenerates the `policy.json` files from a real run of +the code, and CI's `validate-lavamoat-policies` fails the build if the committed policy drifts +from that regeneration. Re-deriving the diff by hand, or sorting the grants into +network/DOM/red-flag buckets, only re-does a machine that is already trusted. + +**Finding the call site is not the job either — that search cannot fail.** The policy is +*generated from a real run*, so every grant in it corresponds to something the bundled code +did. "Each addition has a call site, therefore each is justified" is a tautology dressed as an +audit; it will report 11 of 11 justified every time, and a check that cannot come back negative +carries no information. + +**The job is least privilege: which of these grants can be dropped without breaking +anything?** A grant exists because an identifier appears in bundled source. That is *not* the +same as a reachable path needing it — a capability read behind a config flag nobody sets, or in +a branch our usage never takes, is removable. So for each grant, ask what executes it under +*our* usage, and sort: + +| | | +|---|---| +| **removable** | nothing on our path executes the read → candidate; propose the test | +| **removable at a cost** | only a convenience or error-detail path executes it → name the cost | +| **load-bearing** | our usage genuinely needs it → say so briefly and move on | + +The lead is the first two rows plus anything the reading turned up that bears on security. +Load-bearing grants still each get a row in the capability → call-site table (step 5) — they just +don't get paragraphs. + +> **Falsifier.** A grant you called load-bearing that a build with it removed still passes. +> The test is cheap and it is the only thing that settles the question: drop the grant from the +> resource, rebuild, run the relevant e2e. Propose it; the policy owners run it. + +**Corollary — the reading is where the real findings come from.** Locating each call site means +reading the code that uses the capability, and that is when genuine issues surface: unbounded work +on remote input, a decode path with no size cap, a feature-detection fallback that makes a grant +droppable. Those observations are worth more than the grant inventory. Lead with them. + +## Method + +1. **Take the diff from LavaMoat; don't re-derive it.** The bot's `update-policies` run + produces the authoritative delta and CI enforces it. Your input is the list of + newly-`true` grants per package, not a hand-rolled scan. (`scripts/policy-audit.py` turns a + base/head policy pair into that list as a worklist — it enumerates, it does not classify.) + + **When no current CI policy exists, regenerate locally — that is the fallback, not a + competing method.** The bot hasn't run, the branch is unpushed, or a variant CI didn't + cover: `yarn webpack:lavamoat:policy:build` (`:mv2` / `:mv3`) produces the same base/head + pair to feed step 2. Note what changes and what doesn't: the *worklist* is equally valid, + but its **provenance is weaker** — it reflects your node version, OS, and lockfile + resolution rather than CI's. Say which source the diff came from when handing the + justification over, and re-check against the bot's policy once it runs. Never regenerate + locally *in preference to* an available CI policy; that re-does a trusted machine and + substitutes a less reproducible artifact for a more reproducible one. + +2. **Know what denial actually does before you reason about it.** An ungranted global is **absent + from the package's endowments and reads as `undefined`** — it does not throw. + `getEndowmentsForConfig` collects `whitelistedReads`, `makeMinimalViewOfRef` builds an object + holding only those, and a `false` value simply keeps the path out of that list + ([endowmentsToolkit.js](https://github.com/LavaMoat/LavaMoat/blob/f5e52ab457c16c3aea72cc8a9dd0833547dd7d2c/packages/core/src/endowmentsToolkit.js#L101-L162)). + This is the whole basis of the analysis, so get it right: **do not confuse per-package + `globals` policy with scuttling**, which is a separate root-realm mechanism. Asserting the + wrong one to the LavaMoat maintainer got the reply "This is jibberish" (#45024, 2026-07-30). + + The consequence is what makes denial testable: a package that reads a global behind a + `globalThis.X || ` guard **keeps working when denied**, because the read yields + `undefined` and the fallback engages. A feature-detection shim is therefore evidence *for* + removability, not against it. + +3. **Read each grant's call site — then ask what executes it under our usage.** Read the + dependency's code *at the version being installed*. Locating the use is the start, not the + answer; the question the reading has to settle is whether anything on our path runs it. Look + for the gate: a config flag (`BigNumber.set({CRYPTO:true})`), an API we never call + (`.random()`), a feature-detection fallback, a branch keyed on a payload type we never send + (`data instanceof Blob` — note this one *does* break when denied, since `instanceof undefined` + throws), an error-only path. A gated read whose gate we never open is a removal candidate. + + Check reachability from *our* side too, not just the dependency's: does our code subscribe to + the feed, import the subpath, take that option? A capability behind a feature we don't use is + the cleanest removal there is — and a capability behind one we *do* use is load-bearing, which + is worth one line and no more. + +4. **Cite it at a pinned tag, not a branch head.** A permalink to `…/blob//#Ln` is + immutable; a branch-head link drifts out from under the citation. The permalink *is* the + evidence — a reader clicks it and lands on the code, convinced without re-running anything. + "It needs X" retyped into a table proves nothing about provenance. + +5. **Lead with removal candidates and anything security-relevant — then give the full table.** + Open on what can be dropped and what the reading turned up, not on an inventory. But every + grant still gets its own row in the capability → call-site table, load-bearing ones included: + that mapping is what a reviewer came for, and a load-bearing row is one short row, not a + reason to merge it into prose with its neighbours. **Removed grants get their names only** + (`WebSocket` and `CustomEvent` are removed by this bump) — no table, no justification column, + since a removal reduces capability and needs no defence. The exception is a removal that is + itself interesting: one that was load-bearing implies a behaviour change worth a sentence. + + Target a few hundred words of *prose*; the table does not count against that and must not be + compressed to hit it. (Violated on extension#45024, 2026-07-30 — a trim pass dissolved the + table into paragraphs and destroyed the comment's key content.) + + **The accept/reject call belongs to the human reviewer; never write it.** No `accept` + column, no `REJECT`, no "Verdict: safe to take", no ✅/❌. Those words do the reviewer's + deciding for them and anchor the judgment before they have read the evidence — and if the + call is wrong, it is wrong in a document that looks authoritative. Describing *risk* is in + scope where it is a fact about the capability ("this reads the global on every exception + path", "these are decode and timer primitives, no filesystem or subprocess reach"); the + disposition is not. State findings and open questions, and let the reviewer conclude. + (Violated on extension#45024, 2026-07-30 — 11 `accept` cells and a "Verdict" section.) + + **A grant with no locatable call site is a real finding, and rare.** On a generated policy it + usually means the identifier is present but the generator saw it in a path you haven't found + — say what you searched rather than implying nothing uses it. + +6. **Put it where the policy is reviewed — but do not tag anyone.** Post the justification as a + comment on the PR carrying the `policy.json` change, so it lands in front of the people who + own the policy rather than standing as a unilateral assertion elsewhere. + + **Show the full comment body in the response before running `gh pr comment`.** The permission + prompt renders the command, not the `--body-file` contents, so approving it blind is approving + unseen text published under the user's name. Paste the table and prose inline first, then post. + Naming the scratchpad path instead of showing the text is the same failure. + + **Do not add `cc @MetaMask/policy-reviewers` — or any `@`-mention — unless the user asks for + it in this session.** Authorization to post a comment is not authorization to notify a team, + and this step being written into the skill does not supply that authorization; editing the + comment afterwards does not un-send the ping. Draft without the tag, post, then offer the cc + line as a separate ready-to-paste suggestion. + + **Status as of 2026-07-30: hold the cc — this workflow is in a trial phase.** The tag is + expected to become standard once the output has proven itself; it is being withheld for now, + not forbidden on principle. So the rule is about *timing being the user's call*, not about + tagging being wrong. Re-confirm before assuming trial phase still applies — and even after it + ends, the tag goes in because the user says so, not because this line stops saying "hold". + (Violated on extension#45024, 2026-07-30.) + +7. **One reason covers the variants.** Extension builds carry several policy files + (`lavamoat/webpack/{mv2,mv3}/{beta,experimental,flask,main}/policy.json`). When the grant + delta is identical across them, a single justification covers all — confirm the identity + once. A grant that appears in one variant and not others is itself a question. + +## Output + +**The capability → call-site table is the deliverable. Never dissolve it into prose.** +One row per grant, every grant, with its permalink and its removability in the row. A reviewer +scans the column, not paragraphs — 11 rows is denser and faster to read than three paragraphs +carrying the same 11 facts, so the table *is* the trimmed form. Prose around it is what gets cut. + +``` +LavaMoat grants — -> + + + +| capability | package | call site | can it go? | +|---|---|---|---| +| | | | yes — | +| | | · | at a cost — | +| | | | no | +…every grant gets a row… + +Test for the "yes" rows: drop the grant, rebuild, run . +Note on : +Removed: , . ← names only, no table, no justification column +Loose ends: + +``` + +Order is the point: the lead frames the question, the table answers it, and a reviewer hits the +removable rows immediately. Post it on the PR carrying the `policy.json` change, untagged. + +**Runtime claims need a runtime artifact.** "Byte-identical across all 8 policy files" is an +observation, not something a `/blob/` link witnesses — publish the check output (JSON) and link +it. The `pr-evidence-gate` hook enforces this and will block the post otherwise; it is right to. + +## Worked example — extension#42867 (@sentry/browser 8.33.1 → 10.38.0) + +The bump added grants across the `@sentry/*` subtree; the bot produced the diff and CI +enforced it — detection was never in question. On `mv2/main/policy.json` a reviewer questioned +two grants: *"I wonder what it's using this for. Likewise for `importScripts`."* Each was +answered with the upstream line, pinned to `10.38.0`: + +- **`WebAssembly`** → the event builder's `isWebAssemblyException` check, which runs on *every* + exception (defined L165, called on the exception path L186 and L203): + `https://github.com/getsentry/sentry-javascript/blob/10.38.0/packages/browser/src/eventbuilder.ts#L163-L168` +- **`importScripts`** → the profiling utils' main-thread detection at module scope + (`typeof importScripts === 'undefined'`): + `https://github.com/getsentry/sentry-javascript/blob/10.38.0/packages/browser/src/profiling/utils.ts#L33-L34` + +Both run unconditionally — the exception path and module scope — and under scuttling the read +itself throws unless excepted, so both are load-bearing with no gate to close. That is the +useful conclusion: *not* "each grant has a reason" (it always will) but "neither is removable, +and here is the unconditional path that makes it so." + +**Counter-example from extension#45024, which the first pass got wrong.** That comment reported +"11 additions, 11 reasons, each resolving to a line" as its headline. Tautological — the policy +is generated from a run, so the count was guaranteed. Reading for *gates* instead surfaced the +actual findings: `crypto` on `bignumber.js` is reachable only via `BigNumber.set({CRYPTO:true})` +or `.random()`, neither of which the consumer calls, so it is a removal candidate; and the +`DecompressionStream` grant sits on a `fastAssetCtxs` decode path that inflates remote input +with no size cap. Same reading, same permalinks — the first framing hid both. + +## Scope — what this skill is NOT + +This covers **capability containment only**: what a dependency is *permitted to reach* under +LavaMoat, and whether each new permission has a reason. It says nothing about whether the +dependency is *known-vulnerable* or *behaving maliciously* — those are different questions with +different detectors and different falsifiers, and they live in `supply-chain-audit`: + +| question | detector | skill | +|---|---|---| +| does this dep now reach a capability it didn't? | LavaMoat policy diff | **this skill** | +| is this dep version known-vulnerable? | `yarn npm audit`, advisories | `supply-chain-audit` | +| is this package behaving maliciously / newly-authored / install-scripted? | Socket Security | `supply-chain-audit` | + +A clean policy diff does not mean a safe dependency, and a known CVE does not show up as a new +grant. Run the umbrella skill when the question is "is this bump safe"; run this one when the +question is "why does it need that". + +## Called by supply-chain-audit and pr-validate + +`supply-chain-audit` delegates its capability-containment lane here. pr-validate keeps +**supply-chain** as an evidence category and packages the per-grant justification (accept / +reject, each with its permalink) posted where the policy is reviewed. Engine helper: +`scripts/policy-audit.py`. Usable standalone whenever a policy grant needs a reason. diff --git a/domains/security/skills/supply-chain-audit/skill.md b/domains/security/skills/supply-chain-audit/skill.md new file mode 100644 index 00000000..32dbffd2 --- /dev/null +++ b/domains/security/skills/supply-chain-audit/skill.md @@ -0,0 +1,158 @@ +--- +name: supply-chain-audit +description: Assess whether a dependency change is safe to take, across every detector that answers a different part of that question — Socket Security (malicious/anomalous package behavior, install scripts, new maintainers), `yarn npm audit` and advisories (known vulnerabilities), lockfile and manifest diffs (what actually changed, including transitive and resolution swaps), and LavaMoat policy grants (new capabilities, delegated to `lavamoat-policy-diligence`). Also covers the fronts no upstream scanner sees because they are things your own repo does to dependencies afterwards: yarn patches that modify dependency source at install, `resolutions` that force or stub versions, `npmAuditIgnoreAdvisories` suppression lists, CI actions riding mutable tags instead of pinned SHAs, and yarn plugins that execute at install. The falsifier is a lane whose finding is unaccounted for — a flagged package, an unresolved advisory, or a grant with no call site. Detection belongs to the tools; the job is disposition, and handing it to the humans who own the dependency. Triggers on /supply-chain-audit, or when asked whether a dependency bump is safe, to review a lockfile or package.json change, to triage a Socket or audit finding, or to assess supply-chain risk of a change. Callable by pr-validate as the engine behind its supply-chain evidence category. +maturity: experimental +--- + +# /supply-chain-audit + +**"Is this bump safe?" is not one question.** A dependency can be free of known CVEs and still +reach `child_process` for the first time. It can have a clean policy diff and ship a +newly-added install script from a maintainer who joined last week. Each detector answers a +different question and is blind to the others, so a single green check is never the answer. + +> **Falsifier.** Any lane's finding left unaccounted for: a Socket alert nobody dispositioned, +> an advisory with no upgrade path or accepted-risk note, a capability grant with no call site. +> An unexplained finding is the output, not a nit to wave through. + +## Lanes + +| question | detector | disposition | +|---|---|---| +| what actually changed? | lockfile / `package.json` diff | direct vs transitive; resolution swaps; version range widening | +| known-vulnerable? | `yarn npm audit`, GitHub advisories, Dependabot | fixed-in version, or an explicit accepted-risk with reachability | +| behaving maliciously or anomalously? | **Socket Security** | per-alert disposition — see below | +| new capability reached? | **LavaMoat** policy diff | **delegate to `lavamoat-policy-diligence`** | +| install-time code execution? | `allowScripts` in `package.json` (`@lavamoat/allow-scripts`) | a newly-`true` entry is a finding in its own right | +| **is dependency source modified in-repo?** | **`.yarn/patches/*.patch`** | read the diff — see below | +| **is a version being forced?** | **`resolutions`** in `package.json` | pinned below a fix? stubbed out? | +| **are findings being suppressed?** | **`npmAuditIgnoreAdvisories`** in `.yarnrc.yml` | every entry needs a reason and a re-check date | +| **does untrusted code run in CI?** | **`uses:` pinning** in `.github/workflows` | third-party actions pinned to a full SHA, not a mutable tag | +| **does untrusted code run at install?** | **`.yarn/plugins/*.cjs`** + their `spec:` URLs | committed bundle reviewed; spec pinned, not `main` | + +## The fronts that no scanner covers + +Socket, `audit`, and LavaMoat all examine the dependency **as published**. The last five lanes +above are things *your own repo, or your CI,* does around dependencies afterwards, so no +upstream scanner sees them. Measured on `metamask-extension` today, to show these aren't hypothetical: + +- **Yarn patches — 53 of them.** A patch is arbitrary modification of a dependency's source, + applied at install, living in your repo. It is the single most direct injection point in the + list and the least watched: the package can be clean at every scanner and still execute your + patch. **Read every patch diff on change**, the same way you'd read a diff to `app/`. A patch + that grows beyond its stated purpose, or touches a file unrelated to the bug it works around, + is the finding. Record why each patch exists and what removes it (upstream fix, version bump) + — an unattributed patch is technical debt with a security surface. + +- **`resolutions` — 149 entries.** Forcing a version across the tree. Two failure modes: a pin + that holds a transitive *below* the version that fixed an advisory (audit may not flag it, + because the range resolves), and outright substitution — this repo maps several packages to + `npm:npm-empty-package@1.0.0` to neutralize them. Substitution is legitimate and deliberate, + but it means **"same version range" does not imply "same code"**, so treat a resolution + change as a dependency change and re-run the lanes on it. + +- **`npmAuditIgnoreAdvisories` — a suppression list.** Entries here are accepted risks by + definition, and this repo's numeric IDs carry no inline reason (its deprecation entries do). + This directly contradicts this skill's own falsifier: an unaccounted finding is the output. + Each entry wants a reason, an owner, and a condition that retires it. An ignore list nobody + revisits converts a finding into silence. + +- **CI action pinning — 7 of 47 third-party `uses:` are SHA-pinned.** The rest ride mutable + tags (`actions/checkout@v6`, `actions/github-script@v9`). A tag can be repointed by its owner + or by anyone who compromises that account, and CI holds secrets — this is the + `tj-actions/changed-files` failure mode. Pin third-party actions to a full 40-char commit + SHA. First-party (`MetaMask/*`, 25 here) is lower risk but the same mechanism. + +- **Yarn plugins execute at install with full privilege.** Three `.cjs` bundles are committed + (good — the committed bytes are what runs), but their `spec:` URLs point at + `raw.githubusercontent.com/.../main/...`, a moving branch. Re-importing pulls whatever `main` + holds that day. Pin the spec to a tag or SHA, and review the bundle diff when it changes. + +**Also consider `enableHardenedMode`** (Yarn 4) — not currently set here. It validates +resolutions and checksums against the registry, and is designed for exactly the untrusted-PR +case. And leave `checksumBehavior` at its default (`throw`): a checksum mismatch means the +registry served different bytes for a version you already resolved, which is a signal, not a +nuisance. + +Run the lanes the change actually touches. A lockfile-only bump of a build-time dev dependency +does not need the same treatment as a new runtime dependency in the wallet's hot path — but +say which lanes you ran and which you skipped, and why. + +## Method + +1. **Establish what changed before assessing it.** Direct bump, transitive pull-through, or a + *resolution swap* (same range, different resolved package)? The last is the easiest to miss + and the most interesting: an identifier substitution in a policy or lockfile + (`pkgC>name` replacing `pkgB>pkgA>name`) can mean the dependency was replaced rather than + updated. + +2. **Take each tool's findings as the worklist; don't re-derive them.** Socket, audit, and + LavaMoat all run in CI and are trusted machines. Re-implementing their detection by hand + re-does work and produces a less reproducible artifact. Your input is their output. + +3. **Prefer the CI-generated artifact over a local regeneration.** Where a bot regenerates + something (policies especially) and CI enforces drift, that committed artifact is + authoritative. Regenerate locally only as a **fallback** — bot hasn't run, branch unpushed, + variant not covered — and note that the provenance is weaker (your node version, OS, and + lockfile resolution, not CI's). Re-check against the bot's artifact once it runs. + +4. **Disposition every finding; the reason is the deliverable.** For each alert or advisory: + what the tool flagged, whether it is reachable from how *this* project uses the package, and + the outcome — fixed by upgrading to X, accepted with a stated reason, or blocking. Socket's + common alert classes need different reasoning: `install scripts` (what does it run, at whose + trust level), `new author` / `low download count` (typosquat and takeover surface), + `network access` / `filesystem access` in a package with no business doing either, + `obfuscated code`, `protestware`. A tool's severity is an input to that judgment, not a + substitute for it. + +5. **Cite at a pinned version, not a branch head.** Every claim about what a dependency does + resolves to a permalink at the version being installed. A branch-head link drifts out from + under the citation; the permalink *is* the evidence, because a reader clicks it and is + convinced without re-running anything. + +6. **Hand it to the owners.** Post the disposition where the people who own the dependency + read it, not as a unilateral assertion. This skill produces a justification for humans to + act on; it does not approve anything. + +## Capability containment → `lavamoat-policy-diligence` + +LavaMoat policy grants are a specialized lane with their own method and tooling. **Delegate to +`lavamoat-policy-diligence`** and fold its result in as this audit's capability-containment lane. + +Do not restate that lane's question as "does each new capability have a call site" — the policy +is generated from a real run, so it always does, and that check cannot fail. The lane's actual +output is a least-privilege triage: which grants are **removable** (their gate is never opened by +our usage), which are removable at a stated cost, which are load-bearing, plus anything the +reading turned up bearing on security. Carry those findings through; do not compress them to a +pass/fail. + +Keep the boundary straight in the writeup: **a clean policy diff does not mean a safe +dependency, and a known CVE does not appear as a new grant.** They are independent. + +## Output + +``` +Supply-chain assessment — -> () + lockfile/manifest + advisories → fixed in | accepted: | none + Socket → | no alerts + install scripts | unchanged + patches <.yarn/patches touched> → diff read: | unchanged + resolutions | unchanged + audit ignores → reason + retire-when | unchanged + ci actions → SHA-pinned? | unchanged + capabilities → lavamoat-policy-diligence: | no policy change + lanes skipped — +Unresolved: — | none +``` + +Lead with whatever is actionable — an unresolved finding, a removable capability, a patch whose +scope exceeded its purpose. Lanes that came back clean are a compact line each, not sections. +**No overall accept/reject verdict and no `@`-mentions**: the disposition belongs to the people +who own the dependency, and tagging them is the user's call, not this skill's. Close on what is +unresolved and what would settle it. + +## Related + +- `lavamoat-policy-diligence` — the capability-containment engine this skill delegates to. +- `pr-validate` — packages this skill's output as its supply-chain evidence category. From c635aa8823a5260ac082b3a1a734a9241e4a44b3 Mon Sep 17 00:00:00 2001 From: Jongsun Suh Date: Thu, 30 Jul 2026 08:04:06 -0400 Subject: [PATCH 018/135] feat(pr-workflow): add pr-validate and falsifying-test skills MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces #56, whose skill.md shipped frontmatter with no body at all — the installer's bodyAfterFrontmatter() returned empty, so an agent loading it got a description and six references it had no instruction to read. pr-validate: for a PR's specific falsifiable claim, name the observation that would prove the claim false, gather it, and publish it into the PR body. Drives the AEP harness (visual_validation, perf_validation) as the primary engine, backed by a catalog of complementary lanes, a trustworthiness gate that rejects vacuous passes, and a publishing flow with an audience-reachability rule for re-hosted artifacts. falsifying-test: the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. Its own falsifier is a base-commit failure for the wrong reason (import error, missing fixture, unrelated red), which looks identical in an exit code and proves nothing. pr-validate calls it as the engine behind lane B3. hooks/pr-evidence-gate.py enforces the trustworthiness gate at emit time, blocking an outward-facing write whose body carries an unbacked verdict, an untracked deferral, a CI restatement, a bare or truncated identifier, a mutable ref, a dump-as-resolver, a link-only or data-only exhibit, or a step waiver. It polices `gh api` body writes as well as the porcelain, since a PATCH to a comment is the same publish with a different spelling. --- .../skills/falsifying-test/skill.md | 86 ++++ .../pr-validate/hooks/pr-evidence-gate.py | 390 ++++++++++++++++++ .../references/claim-extraction.md | 63 +++ .../references/evidence-catalog.md | 254 ++++++++++++ .../references/evidence-gate-setup.md | 58 +++ .../references/evidence-publishing.md | 291 +++++++++++++ .../references/evidence-trustworthiness.md | 43 ++ .../pr-validate/references/lane-assertions.md | 26 ++ .../pr-validate/references/worked-examples.md | 30 ++ .../pr-workflow/skills/pr-validate/skill.md | 305 ++++++++++++++ 10 files changed, 1546 insertions(+) create mode 100644 domains/pr-workflow/skills/falsifying-test/skill.md create mode 100755 domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py create mode 100644 domains/pr-workflow/skills/pr-validate/references/claim-extraction.md create mode 100644 domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md create mode 100644 domains/pr-workflow/skills/pr-validate/references/evidence-gate-setup.md create mode 100644 domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md create mode 100644 domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md create mode 100644 domains/pr-workflow/skills/pr-validate/references/lane-assertions.md create mode 100644 domains/pr-workflow/skills/pr-validate/references/worked-examples.md create mode 100644 domains/pr-workflow/skills/pr-validate/skill.md diff --git a/domains/pr-workflow/skills/falsifying-test/skill.md b/domains/pr-workflow/skills/falsifying-test/skill.md new file mode 100644 index 00000000..922991f5 --- /dev/null +++ b/domains/pr-workflow/skills/falsifying-test/skill.md @@ -0,0 +1,86 @@ +--- +name: falsifying-test +description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /falsifying-test, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by pr-validate as the engine behind its falsifying regression test evidence category. +maturity: experimental +--- + +# /falsifying-test + +Reach for this on **every bug-fix PR**. A test that passes on the branch proves the branch is +green. A test that **fails on base and passes on the branch** proves the change is causally +connected to the reported bug. Only the second is evidence, and the gap between them is where +this skill lives. + +> **Falsifier.** A test that fails on base for a reason unrelated to the bug. A missing import, +> a fixture the base commit doesn't have, a helper introduced by the branch, an unrelated +> pre-existing failure — every one produces a red run and a non-zero exit code that looks +> exactly like a correct falsification. **The exit code is not the evidence; the assertion +> message is.** + +## Method + +1. **Write the test against the reported behaviour, not the diff.** Start from the issue's + reproduction. A test derived from reading the fix tends to assert the fix's mechanism and + will pass on base the moment the mechanism is reachable by other means — or fail on base + for structural reasons rather than behavioural ones. + +2. **Run it on base FIRST, and read the failure output.** Not the exit code — the message. It + must fail on the **assertion that encodes the bug**: an expected value that differs, a state + that wasn't reached, an event that didn't fire. If base fails with a + `ModuleNotFoundError`, a syntax error, or a helper that doesn't exist yet, you have not + falsified anything; you have discovered that the test can't run there. + +3. **Pin the base explicitly.** Use the PR's actual merge-base, not whatever `main` points at + today. `main` moves; a re-run weeks later against a drifted `main` is a different + experiment and may fail for reasons that have nothing to do with the fix. + +4. **Make the test runnable on base.** When the test needs a helper or fixture the branch + introduces, split it: land the scaffolding in a form that exists on both sides, or inline + the setup so the test file is self-contained. If that's impossible, say so and downgrade the + claim — a test that *cannot* run on base gives a branch-only pass, which is a weaker piece + of evidence and should not be presented as a falsifying one. + +5. **Confirm it fails for one reason, not several.** If base has unrelated failures in the same + file or suite, scope the run to the new test (by name/path) so the red is attributable. A + suite that was already red proves nothing about your assertion. + +6. **Show both runs.** Base: the assertion failure, verbatim. Branch: the pass. Same command, + same filter, both commits identified. Captured terminal output beats a transcription — + retyped output is a self-report, and a real capture has caught errors that careful prose + missed. + +7. **Pair it with the issue.** The PR's `Fixes #N` plus a test named for the behaviour makes + the causal chain checkable by a reader who runs nothing. + +## When you can't write one + +This is a finding, not a gap to paper over. If no test fails on base, one of these is true: + +- **The bug isn't where the fix is.** The most common case, and the reason to run this check + before review rather than after. +- **The reported behaviour isn't reproducible in the harness** — timing, environment, or a + real-device dependency. Say which, and reach for a different evidence category (a + deterministic interleaving test for ordering bugs, an e2e trace for environment-dependent + ones). +- **The fix is a refactor or hardening change, not a bug fix.** Fine — then the PR's claim + should say that, and this category doesn't apply. + +State which one. "No test added" with no explanation reads as an omission; the diagnosis is +useful information about the change. + +## Output + +``` +Falsifying test — (Fixes #N) + base FAIL + branch PASS + command: + scoped: +``` + +## Related + +- `pr-validate` — packages this skill's output as its B3 evidence category; B7 (deterministic + interleaving) is the sibling for concurrency and temporal-ordering bugs. +- `react-render-proof` — the same before/after discipline applied to a measured quantity + rather than a boolean. diff --git a/domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py b/domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py new file mode 100755 index 00000000..2bcd4651 --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py @@ -0,0 +1,390 @@ +#!/usr/bin/env python3 +""" +Emit-time evidence gate (PreToolUse:Bash). + +Blocks outward-facing `gh pr|issue edit|create|comment` — and `gh api` body +writes, which bypass the porcelain — whose body contains, in a validation-scoped +paragraph, a claim that the trustworthiness gate would reject. Rationale: an +unbacked "confirmed / verified / proven / observed / ingested / ✅" launders an +unverified assertion as fact under the author's name, and an untracked "remains +pending" decays to never. + +The trustworthiness gate is the checklist; THIS is the trigger that runs it. +Each class below implements a numbered item of `references/evidence-trustworthiness.md`. + +Contract: reads PreToolUse JSON on stdin. Exit 0 = allow. Exit 2 = block +(stderr shown to the model). Fails OPEN on anything it cannot parse, so it +never bricks unrelated Bash commands. +""" +import json +import os +import re +import sys + + +def _out_allow(): + sys.exit(0) + + +def _block(msg): + sys.stderr.write(msg) + sys.exit(2) + + +# Outward-facing gh write surfaces. The porcelain set is wider than +# `gh pr edit|create` because the same unbacked verdict launders identically +# through a PR comment or an issue body. `gh api` is included because a PATCH +# to .../comments/ is the same publish with a different spelling — a gate +# that cannot see the write it is meant to police is not a gate. +GH_PORCELAIN = re.compile(r"\bgh\s+(?:pr|issue)\s+(?:edit|create|comment)\b") +GH_API = re.compile(r"\bgh\s+api\b") + + +def main(): + try: + payload = json.load(sys.stdin) + except Exception: + _out_allow() + + if payload.get("tool_name") != "Bash": + _out_allow() + + cmd = (payload.get("tool_input") or {}).get("command", "") + + is_porcelain = bool(GH_PORCELAIN.search(cmd)) + is_api = bool(GH_API.search(cmd)) and re.search(r"(?:-F|-f|--field|--raw-field)\s+body=|--input\b", cmd) + if not (is_porcelain or is_api): + _out_allow() + if is_porcelain and "--body" not in cmd: # covers --body and --body-file + _out_allow() + + body = _extract_body(cmd) + if not body: + _out_allow() # can't read it -> don't block; nothing to scan + + violations = _scan(body) + if not violations: + _out_allow() + + lines = [ + "EVIDENCE GATE (PreToolUse) — blocked outward-facing GitHub write.", + "", + "Each finding names the trustworthiness-gate item it violates. Fix by", + "attaching the missing artifact in the SAME block, or by downgrading the", + "claim (⚠️ inconclusive / remove it). Do not rephrase around the check.", + "", + ] + for v in violations[:12]: + need = NEEDS.get(v.get("kind", "verdict"), "ARTIFACT") + lines.append(f' • [{v["kind"]}] "{v["token"]}"') + lines.append(f' needs: {need}') + lines.append(f' in: {v["snippet"]}') + if len(violations) > 12: + lines.append(f" … and {len(violations) - 12} more.") + lines += [ + "", + "If the evidence exists on disk, BIND it: every collected artifact the", + "claim rests on gets referenced or re-hosted before the write.", + ] + _block("\n".join(lines) + "\n") + + +NEEDS = { + "verdict": "an inspectable ARTIFACT (https:// permalink, /blob//, or a *.test.ts ref)", + "observation": "an OBSERVATION artifact (screenshot/recording/log/JSON/permalink) — " + "a /blob/ code link witnesses code, not runtime behavior", + "deferral": "a co-located TRACKER (#issue, issues/pull URL, 'triage', 'tracked in')", + "ci-restatement": "removal — a validation surface carries zero CI references. " + "The Checks tab already shows them; cite CI only as the revert " + "lane's outcome, never as 'green at head'", + "inflated-verdict": "a downgraded verdict — 'live-proven' co-located with " + "'not exercised' is inflated; borrowed evidence never " + "upgrades an uncaptured lane", + "bare-identifier": "a resolving link for the id (permalink or absolute-windowed " + "query) OR the re-hosted capture showing it", + "truncated-identifier": "the FULL identifier, quoted verbatim — an ellipsized id " + "cannot be grepped against any artifact, and a co-located " + "resolver does not excuse it", + "mutable-ref": "a commit-pinned permalink (/blob//…#Lx-Ly) — a branch ref " + "can be rewritten after review", + "dump-resolver": "a reader-native exhibit — a live link or a visual. A raw " + "log/JSON/HAR dump is appendix-only, never the exhibit a claim rests on", + "link-only-exhibit": "an embedded visual of the linked view ALONGSIDE the permalink — " + "link-only defers validation behind click + auth + query rendering", + "data-only-exhibit": "an in-environment capture (the resolving UI with its query, " + "project/environment selectors and time window in-frame) — " + "quoted data alone carries no liveness provenance", + "step-waiver": "a per-step ⏳ + tracker whose blocker is that step's OWN unmet " + "precondition — an impossibility argument is not a discharge", +} + + +def _extract_body(cmd): + # 1) --body-file / --input + m = re.search(r"--(?:body-file|input)[=\s]+(?:'([^']+)'|\"([^\"]+)\"|(\S+))", cmd) + if m: + path = m.group(1) or m.group(2) or m.group(3) + try: + with open(os.path.expanduser(path), "r", encoding="utf-8") as fh: + raw = fh.read() + except Exception: + return "" + # `gh api --input` takes a JSON file; pull .body out of it. + try: + obj = json.loads(raw) + if isinstance(obj, dict) and isinstance(obj.get("body"), str): + return obj["body"] + except Exception: + pass + return raw + # 2) gh api -F body=@ / --field body=@ + m = re.search(r"(?:-F|--field|--raw-field)\s+body=@(?:'([^']+)'|\"([^\"]+)\"|(\S+))", cmd) + if m: + path = m.group(1) or m.group(2) or m.group(3) + try: + with open(os.path.expanduser(path), "r", encoding="utf-8") as fh: + return fh.read() + except Exception: + return "" + # 3) --body "$(cat <<'EOF' ... EOF)" heredoc + m = re.search(r"<<-?'?EOF'?\s*\n(.*?)\n\s*EOF", cmd, re.DOTALL) + if m: + return m.group(1) + # 4) --body '...' / --body "..." / gh api -f body='...' + m = re.search(r"(?:--body|(?:-f|--field|--raw-field)\s+body=)[=\s]*'((?:[^']|'\\'')*)'", cmd, re.DOTALL) + if m: + return m.group(1) + m = re.search(r'(?:--body|(?:-f|--field|--raw-field)\s+body=)[=\s]*"(.*?)"', cmd, re.DOTALL) + if m: + return m.group(1) + return "" + + +# ── item 1/5: verdict claims ──────────────────────────────────────────────── +VERDICT = re.compile( + r"(?i)(?:\bcapture[ds]?\s+confirm\w*|\bconfirm(?:s|ed)\b|\bverif(?:y|ies|ied)\b" + r"|\bproven\b|\bobserved\b|\bingested\b|\bdemonstrat(?:e|es|ed)\b" + r"|\blive-proven\b|\bsuccessful\b|\bvalidated\b" + r"|does not drop\b|✅)" +) +ARTIFACT = re.compile( + r"(?i)(?:https?://\S+|actions/runs/\d+|/blob/|\bjob/\d+" + r"|`?[\w./-]*\.(?:test|spec)\.[tj]sx?(?::\d+)?`?)" +) +# ── item 2: runtime observation claims ───────────────────────────────────── +OBSERVATION = re.compile( + r"(?i)(?:\brendered\b|byte-identical(?:ly)?|\bsnapshot\s+shows?\b" + r"|\bscreenshots?\s+show\w*|\breproduc(?:ed|es)\b" + r"|\bstill\s+(?:shown|shows|fails|failing|raises)\b" + r"|\bin\s+a\s+(?:real|live)\s+browser\b|\blive\s+test\s+build\b" + r"|\bin\s+two\s+independent\s+runs\b|\bworks\s+as\s+described\b)" +) +OBS_ARTIFACT = re.compile( + r"(?i)(?:!\[|/ instead of /blob// ───────── +MUTABLE_REF = re.compile( + r"(?i)https?://github\.com/[\w.-]+/[\w.-]+/blob/(?![0-9a-f]{7,40}[/#])[\w.-]+/" +) +# ── item 13: dump-as-resolver ────────────────────────────────────────────── +DUMP_LINK = re.compile(r"(?i)https?://\S+\.(?:log|json|har|txt)\b") +IMAGE_EMBED = re.compile( + r"(?i)(?:!\[|.*?", + "", body, flags=re.DOTALL) + violations = [] + section = "" + for block in re.split(r"(?m)^(?=\s*#{1,6}\s)", body): + hm = re.match(r"\s*#{1,6}\s*(.+)", block) + if hm: + section = hm.group(1) + section_in_scope = bool(SCOPE_HEADING.search(section)) + for para in re.split(r"\n\s*\n", block): + scan_lines = [] + for ln in para.splitlines(): + s = ln.strip() + if re.match(r"-\s*\[[ xX]\]", s): # checklist item + continue + if s.startswith(">"): # blockquote (bot NOTE) + continue + if s.startswith("_Status key"): # legend + continue + if s.startswith("#"): # heading line + continue + scan_lines.append(ln) + chunk = "\n".join(scan_lines) + if not chunk.strip(): + continue + if not (section_in_scope or SCOPE_PARA.search(chunk)): + continue + # A markdown table row is its own claim unit — scan each row so an + # artifact two rows down cannot excuse a bare row. + units = chunk.splitlines() if chunk.lstrip().startswith("|") else [chunk] + for unit in units: + _scan_unit(unit, violations) + return violations + + +def _add(violations, kind, token, unit): + violations.append({ + "kind": kind, + "token": token, + "snippet": re.sub(r"\s+", " ", unit.strip())[:120], + }) + + +def _positive_verdict(unit): + """A non-negated verdict token in this unit, or None.""" + for m in VERDICT.finditer(unit): + if not _negated(unit, m.start()): + return m.group(0) + return None + + +def _scan_unit(unit, violations): + # ── VERDICT: excused by a co-located inspectable artifact. + if not ARTIFACT.search(unit): + tok = _positive_verdict(unit) + if tok: + _add(violations, "verdict", tok, unit) + + # ── OBSERVATION: needs an observation-class artifact. A /blob/ code + # permalink does NOT excuse it. + if not OBS_ARTIFACT.search(unit): + for m in OBSERVATION.finditer(unit): + if _negated(unit, m.start()): + continue + _add(violations, "observation", m.group(0), unit) + break + + # ── DEFERRAL: excused by a co-located tracker, NOT by an artifact. + if not TRACKER.search(unit): + dm = DEFERRAL.search(unit) + if dm: + _add(violations, "deferral", dm.group(0), unit) + + # ── CI RESTATEMENT (item 11): unconditional in validation scope. No + # verdict co-location required, no "beyond-CI"/"as context" excuse — + # a carve-out here is an instruction to phrase every violation as the + # exception. + cm = CI_RESTATEMENT.search(unit) + if cm: + _add(violations, "ci-restatement", cm.group(0), unit) + + # ── INFLATED VERDICT (item 11): proof language co-located with an + # admission the surface was not exercised. + nm = NOT_EXERCISED.search(unit) + if nm and _positive_verdict(unit): + _add(violations, "inflated-verdict", nm.group(0), unit) + + # ── STEP WAIVER (item 14): an impossibility argument never discharges a + # lane derived from an executable Manual testing step. + sm = STEP_WAIVER.search(unit) + if sm: + _add(violations, "step-waiver", sm.group(0), unit) + + # ── TRUNCATED IDENTIFIER (item 16): a co-located resolver does NOT + # excuse — the resolver resolves the full id, not the fragment the + # reader holds. Hash-equality prose is exempt. + if not HASH_EQUALITY.search(unit): + tm = TRUNCATED_ID.search(unit) + if tm: + _add(violations, "truncated-identifier", tm.group(0), unit) + + # ── BARE IDENTIFIER (item 12): an id with no resolving link and no + # re-hosted capture is a digging assignment. + if not RESOLVER.search(unit) and not OBS_ARTIFACT.search(unit): + bm = BARE_ID.search(unit) + if bm: + _add(violations, "bare-identifier", bm.group(0), unit) + + # ── MUTABLE REF (item 16): pin evidence links to a SHA. + mm = MUTABLE_REF.search(unit) + if mm: + _add(violations, "mutable-ref", mm.group(0)[:60], unit) + + # ── DUMP RESOLVER (item 13): a positive verdict whose only resolver is a + # raw dump behind a link. The digging moved a hop away, it did not + # disappear. + if _positive_verdict(unit) and DUMP_LINK.search(unit) and not IMAGE_EMBED.search(unit) \ + and not LIVE_LINK.search(unit): + _add(violations, "dump-resolver", DUMP_LINK.search(unit).group(0)[:60], unit) + + # ── LINK-ONLY EXHIBIT (item 15): a live permalink defers validation + # behind click + auth + query rendering. Needs the visual too. + if _positive_verdict(unit) and LIVE_LINK.search(unit) and not IMAGE_EMBED.search(unit): + _add(violations, "link-only-exhibit", LIVE_LINK.search(unit).group(0)[:60], unit) + + # ── DATA-ONLY EXHIBIT (item 17): telemetry claim with neither a visual + # nor a live link carries no liveness provenance — extracted data is + # indistinguishable from data typed by hand. + if _positive_verdict(unit) and TELEMETRY_VOCAB.search(unit) \ + and not IMAGE_EMBED.search(unit) and not LIVE_LINK.search(unit): + _add(violations, "data-only-exhibit", TELEMETRY_VOCAB.search(unit).group(0), unit) + + +def _negated(text, pos): + """A verdict token preceded by a negator is a hedge, not a claim.""" + pre = text[max(0, pos - 16):pos].lower() + if re.search(r"\b(not|never|no|isn't|aren't|cannot|can't|without|un|yet)\s*$", pre): + return True + # 'unverified' / 'unproven' — negator fused onto the token + if pre.endswith("un"): + return True + return False + + +if __name__ == "__main__": + main() diff --git a/domains/pr-workflow/skills/pr-validate/references/claim-extraction.md b/domains/pr-workflow/skills/pr-validate/references/claim-extraction.md new file mode 100644 index 00000000..fdd90417 --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/claim-extraction.md @@ -0,0 +1,63 @@ +# Claim extraction + +The linchpin of pr-validate: before choosing any lane, turn the PR into a **falsifiable, surface-specific claim**. Every lane is only as good as the claim it tests. A vague claim ("improves perf", "fixes the bug") can't be proven or refuted; a sharp claim names the precondition, action, observable outcome, and what would disprove it. + +## Read these, in order + +1. **PR body** — Description (what/why), `Fixes #N`, Manual testing steps, the Before/After intent. +2. **Linked issue(s)** — the bug report / acceptance criteria; "Steps to reproduce" and "Expected vs actual" are the claim in the reporter's words. +3. **The diff** (`gh pr diff`) — what actually changed: which surfaces, controllers, modules. The claim must be anchored to what the code can do, not only what the body promises. +4. **Labels / type** — bug vs feat vs perf vs refactor changes the claim shape (see [special cases](#special-cases)). + +## Extraction steps + +1. **Asserted change** — what does the PR say it does? (body + issue) +2. **Anchor to the diff** — which surface/module changed? Reconcile intent with the diff. If the body promises X but the diff can't deliver X, **flag the drift** — that's a finding, not a claim. +3. **Phrase as falsifiable** — `Given , when , then .` The outcome must be observable and checkable. Replace vague verbs (improve / fix / handle / support) with the concrete observable. +4. **Pin the surface + reachability** — exact screen / API / metric. Reachable in the default fixture, or does it need state seeding, a feature flag, or a fallback surface? + - **A surface need not be a screen.** A pipeline's job graph, a build artifact, a policy file, a telemetry shape, or a harness's determinism are all legitimate surfaces with their own falsifiers. Do not force a user-visible observable onto a claim that does not have one — routing a CI or build claim through a product effect is the *wrong* bar, not a stricter one. + - **When the changed code is the automation, the PR's own run may not exercise it.** A CI-config diff commonly skips the very path it edits (build reuse, `needs-*` resolution, event-type conditions). Execute the changed workflow where its trigger conditions hold — a test fork, a branch whose name satisfies the condition — with the failure state forced. Name that substitution explicitly; a claim about *this* repo's pipeline is not proven by a run on another. +5. **Classify the type** → routes to lanes via the matching guide: visible UI · non-visible perf · telemetry · persisted-state · build-output · behavior-no-UI. +6. **Decompose mixed claims** — a PR that changes UI *and* shifts a metric is two claims; validate each. + +## Claim Card (output) + +``` +Claim: Given , when , then . +Surface: (reachable? seed / flag / fallback: …) +Type: → lanes +Falsifier: +Baseline: +``` + +One card per claim. For a refactor, the claim is a **negation** (see below). + +## Claim quality bar + +A good claim is **falsifiable** (observable outcome + clear falsifier), **surface-specific** (names the exact screen/API/metric, not "the app"), **diff-anchored** (the changed code can plausibly produce it), **bounded** (one behavior, one precondition), and **measurable** where quantitative (a number + threshold, not "faster"). + +## Anti-patterns → refinements + +| Vague claim | Refined | +|---|---| +| "Improves performance" | "Opening the Activity tab: TBT drops below 200ms (was >600ms)" — name the interaction, metric, threshold | +| "Fixes the bug" | "With privacy mode on, the Perps tab balance is masked" — observable behavior + precondition + surface | +| "Refactor, no behavior change" | Negation claim: "behavior of `` is unchanged" → prove via falsifying-test-stays-green / snapshot / identical output, **not** a screenshot | +| "Adds a null check" (restates the diff) | "No crash when `` is null on ``" — the behavior, not the code | +| Body promises X, diff does Y | Not a claim — **flag the drift** to the author | + +## Special cases + +- **Refactor / no-op:** the claim is "nothing observable changed." Falsifier = any behavior/output diff. Lanes: regression test stays green, snapshot diff empty, bundle/output identical (D1/D2), benchmark within noise. A passing screenshot proves nothing here. +- **Bug fix:** the strongest claim form ships its own falsifier — a test that fails on `main` and passes on the branch (catalog **B3**). Extract the claim straight from the issue's "Expected vs actual." +- **Perf:** always quantify — metric + interaction + threshold + baseline. Without a number it isn't falsifiable. +- **Persisted-state / migration:** claim = "upgrading from `` preserves `` and applies ``." Falsifier = corrupted/lost state. Baseline = a profile from the prior version (catalog **F1**). +- **Flag-gated:** two claims, one per flag state (catalog **F5**). + +## Worked examples + +- **Visible (#42683):** body "privacy mode doesn't hide the Perps balance"; issue: expected masked, actual visible; diff touches the Perps balance component. → **Claim:** *Given privacy mode on, when I open the Perps tab, the balance is masked.* **Surface:** Perps tab (gated → fallback: Shield entry modal). **Type:** visible → A1/B1. **Falsifier:** balance digits visible under privacy mode. **Baseline:** same flow on base reproduces the bug. +- **Perf:** body "defer Rive wasm at startup"; diff: dynamic `import()` of the Rive runtime. → **Claim:** *On cold start of the home view, the Rive wasm chunk is not requested until the animation surface mounts.* **Surface:** startup network + chunk graph. **Type:** perf → A2/C6/D2. **Falsifier:** the chunk appears in the cold-start waterfall. **Baseline:** base requests it at startup. +- **Migration:** diff adds migration NNN. → **Claim:** *Loading a profile from `` applies migration NNN; `changedKeys = {}`; all other state intact.* **Type:** state → F1. **Falsifier:** an untouched controller mutated, or migrated state malformed. **Baseline:** a prior-version profile. + +A sharp claim is also a good recipe **proof target** (ADR-0058): precondition → action → observable maps to pre-conditions → assertions → screenshot points. Extraction pays off in both lanes. diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md b/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md new file mode 100644 index 00000000..b34a5e20 --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md @@ -0,0 +1,254 @@ +# Evidence catalog + +The menu of evidence kinds for validating a MetaMask **extension** PR, with **what each proves**, **how to capture it (verified against the live repo)**, and **when to reach for it**. AEP is the primary autonomous engine; the rest are complementary. The skill's job is to **match evidence to the claim** and to **proactively suggest kinds the author didn't think of**. + +Pick the evidence that would **falsify the claim if it were false**. Prefer a lane that yields an artifact a reviewer can independently re-check (a link, an image, a number, a replayable trace) over prose. Don't run the whole menu — match, then capture. Capture commands are written against the `metamask-extension` checkout; verify script names against its `package.json` (they drift). + +Legend: **first-class lanes** are `##`-headed; closely-related variants are sub-bullets. Capture marked *(manual)* has no repo helper — it's a DevTools/CDP action. + +--- + +# A. AEP harness (primary, autonomous) + +## A1. visual_validation — before/after screenshots +- **Proves:** a visible UI change on the real surface. Deterministic state seed + agent navigation; PNG artifacts in `evidenceBundle.artifactRefs`. +- **Capture:** `taskClass: visual_validation`, `payload.prUrl` + `description` hint. See the *Preflight* and *Run mechanics* sections of [skill.md](../skill.md). +- **Reach for it:** anything a human would screenshot for the PR's `### After`. + +## A2. perf_validation — falsifiable network/static/smoke assertions +- **Proves:** non-visible behavior (hover-preload, no double-fetch, chunk membership, smoke boot). CDP netlog / phase segmentation / source-map membership. +- **Capture:** `taskClass: perf_validation` (needs a `yarn webpack --test` build). Confirm the perf-validation graph is registered in your AEP checkout; falls back to C6/D2 manually if it isn't present. + +## A3. AEP bundle byproducts (free with any run) +- Test results (`executionResult`/`checkResults`), diff stats, automated `reviewResult` findings, and the **LangSmith trace** of the run. Include the relevant subset; link the trace for auditability. + +--- + +# B. Behavior & flow proof + +## B1. Visual before/after via the `mm` CLI (`visual-testing`) +- **Proves:** UI behavior on a real headed build, with controlled state/network. Defers to the public `visual-testing` skill. +- **Capture:** `yarn build:test:webpack` → `dist/chrome`; `yarn mm launch` → `mm describe-screen` / `mm screenshot` / `mm click` / `mm type` / `mm navigate`. README: `test/e2e/playwright/llm-workflow/`. + - **Degraded-path:** `mm mock-network` to force error/slow responses (session-scoped; add after launch, before the action; can't intercept pre-launch startup). + - **a11y / DOM:** `mm accessibility-snapshot` and `mm cdp` (per the `visual-testing` skill; `a11yRef`s are ephemeral — re-describe after navigation). + +## B2. E2E trace + video (Playwright / Selenium) +- **Proves:** a full flow works, replayably. The strongest "it works end-to-end" artifact. +- **Capture (Playwright):** `yarn playwright test `; trace is `'on'` by default (`playwright.config.ts`), video is `'off'` (enable in config if needed). View: `yarn test:e2e:pw:report`. Artifacts under `public/playwright/`. +- **Capture (Selenium):** `yarn test:e2e:single --browser chrome|firefox|all [--retries n]`; screenshots auto-captured on failure to `test/test-results/e2e/`. + +## B3. Falsifying regression test ⭐ +- **Proves — strongest single proof a fix targets the bug:** a new test that **fails on `main` and passes on the branch**. Show both runs. + - **Engine: the `falsifying-test` skill.** +- **Capture:** add the test, run it on the PR branch (pass) and on the PR's **merge-base** (fail) — pin the base, don't use whatever `main` points at today. Pair with the PR's `Fixes #N`. **Read the base failure's message, not its exit code:** it must fail on the assertion that encodes the bug. A `ModuleNotFoundError`, a missing fixture, or an unrelated pre-existing red produces an identical non-zero exit and falsifies nothing. +- **Reach for it:** every bug-fix PR. If you can't write a test that fails on main, question whether the fix addresses the reported bug. + +## B7. Deterministic interleaving test (concurrency / temporal-ordering) ⭐ +- **Proves:** an ordering guarantee under interleaving — retry, cancellation, supersession, debounce, locks, queues, async state machines — where the correctness *is* the ordering under races, not a value. +- **Capture:** force each race deterministically — `jest.useFakeTimers()` + `advanceTimersByTimeAsync(DELAY)` to fire the delayed action at a known point; `Promise.all([opA, opB])` to overlap operations; `advanceTimersByTimeAsync(0)` to step to a precise interleaving point; then assert the ordering/cancellation outcome for **each** guarantee, including asymmetric ones (one path canceled → its recovery event `.not.toHaveBeenCalled()`; another must complete → `.toHaveBeenCalledWith(...)`). Corroborate with transition telemetry; for the integration path, a live forced-race capture (C8/CDP, the #44610 technique). +- **Trust-gate:** the test must **actually interleave** — time advanced into the pending window, the superseding op injected *during* it. A sequential run exercises no race and is a vacuous green. Verify the interleaving, not just the assertion. + +## B4. Component / Storybook visual +- **Proves:** a component renders across states/props in isolation. +- **Capture:** `.storybook/` present; `yarn storybook` (port 6006), `yarn storybook:build`, `yarn test-storybook` (visual + a11y via `@storybook/addon-a11y`). Jest snapshot diffs for serialized output. + +## B5. Accessibility (a11y) +- **Proves:** no a11y regression / an a11y improvement. +- **Capture:** `yarn test-storybook` (Storybook a11y addon) for components; `mm accessibility-snapshot` for live flows. (No axe-core in the e2e suite — don't claim it.) + +## B6. Flaky-stability rerun +- **Proves:** a flow/test is not flaky (or that a fix removed flakiness). +- **Capture:** Playwright retries `1` on CI / `0` local (`playwright.config.ts`); Selenium `--retries n`; benchmarks default `--retries 2`. Run N× and report the pass rate. See `e2e-flakiness-patterns`. +- Sub: jest snapshot diffs; a unit run for just the changed module (`yarn test:unit `); fuzz/property tests for parsers/encoders. + +--- + +# C. Performance & render + +## C1. Startup / custom traces + phase segmentation +- **Proves:** which startup phase moved (init → FirstRender → interactive), per named span. +- **Capture:** `shared/lib/trace.ts` `TraceName` enum (UIStartup, LoadScripts, FirstRender, …); read in test/debug via `window.stateHooks.getCustomTraces()`. LCP fallback mark: `performance.mark('mm-hero-painted')`. `driver.collectMetrics()` aggregates paint/navigation/long-task/custom traces in e2e. + +## C2. Web vitals — INP / FCP / LCP / CLS +- **Proves:** a user-centric metric moved. `ui/helpers/utils/web-vitals.ts` via `web-vitals/attribution` (attribution names the causing element). +- **Capture:** `window.stateHooks.getWebVitalsMetrics()` (test/debug) → `{inp, fcp, lcp, cls, *Rating}`. Thresholds: INP good<200/poor>500, FCP<1800/3000, LCP<2500/4000, CLS<0.1/0.25. +- **Caveat:** **INP fires on all pages; FCP/LCP/CLS do not fire on popup pages** (sidepanel/E2E only). For extensions, INP is the high-value runtime metric. + +## C3. Long-task / TBT +- **Proves:** main-thread blocking during an interaction dropped. This is where **TBT** lives (the web-vitals lib lane does *not* collect TBT). +- **Capture:** `ui/helpers/utils/performance-observers.ts`; `window.stateHooks.getLongTaskMetricsWithTBT()` → `{count, totalDuration, maxDuration, tbt, tbtRating}`. TBT good<200 / needs-improvement<600 / poor>600. Sampled 10% prod / 100% test. + +## C4. React render & selector proof + - **Engine: the `react-render-proof` skill.** Delegate the measurement to it; it runs the source/delivery/metric gates, derives the needle from real build output, repeats the capture, and returns a band (or "not resolvable at this n" with an MDE). pr-validate packages the result. +- **Proves:** a component/selector stopped over-rendering (cascade-amplification before/after). +- **Capture:** WDYR via `ENABLE_WHY_DID_YOU_RENDER` (`.metamaskrc` or env) — wired in `app/scripts/development/wdyr.ts` (`trackAllPureComponents`); console logs each unnecessary re-render. `yarn devtools:react` for the Profiler flame graph. Selectors use `reselect`'s `createSelector`, which **does expose a real `.recomputations()` counter** — read it (sample on an interval if the count should visibly climb) rather than injecting a log into the selector body; an injected log is an authored claim, a library API is an observation. *(This entry previously said there was no built-in counter. There is.)* +- **Bar:** the delivery check comes before the number. An arm whose manipulation cannot be observed in the built bundle produces a null indistinguishable from "small effect" — and reports as the second. + +## C5. Benchmark A/B +- **Proves:** a startup/journey/interaction timing moved, with a distribution not one sample. +- **Capture:** `yarn test:e2e:benchmark` (`test/e2e/benchmarks/run-benchmark.ts`); presets in `shared/constants/benchmarks.ts` (`startupStandardHome`, `sendTransactions`, `swap`, `dappPageLoad`, …). +- **Caveat:** the rolling baseline (`MetaMask/extension_benchmark_stats`) can **silently freeze** behind a green check (the `store-benchmark-stats` step is `continue-on-error`; happened 2026-04-02, PR #42947). Prefer a **paired A/B** (build both refs now, compare directly) over the stored baseline. +- **Treatment check first** — before trusting any delta, confirm the mechanism under test is actually active in each arm (split chunk present in head and absent in base; the span emitted; the flag evaluated). An arm without the treatment delivered is a no-op, not a control (2026-07-22, #42795). +- **A null needs its power stated** — "no change" and "underpowered" print the same result. When the run-to-run spread exceeds the effect under test, report **not resolvable at this n** and name the smallest detectable effect; never let it read as "no effect". Correcting a known bias (discarding a warm-up, alternating the starting arm) removes *that* bias and nothing more — it is not a trust gate, and the confounds you did not enumerate (thermal drift, background load, ordering within a round) stay live. + +### Capturing an authenticated view (the in-situ requirement) + +Headless Chrome's `--screenshot` cannot set cookies, so an authenticated dashboard +(Grafana/Tempo, Sentry Discover, an internal panel) screenshots as a login page. Drive +Chrome over CDP instead — inject the session cookie, navigate, capture: + +```bash +COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN= \ + cdp-shot "" out.png 25000 1500 2400 +``` + +- **Deep-link to the exact view** so the capture and the reader's verification path are the + same URL (Grafana: `/explore?schemaVersion=1&panes=`). +- **Wait generously** — a trace waterfall or Discover table renders well after `load`. +- **Capture tall + `captureBeyondViewport`**, then crop; the interesting span is usually + below the fold, and cropping after the fact beats guessing a viewport. +- **Crop out the chrome that identifies the operator** (profile avatar, org switcher) + before the image leaves the machine. +- **Keep the trace/query id, timestamp, and result count in frame** — that is what makes + the exhibit reproducible rather than decorative. +- Never echo the cookie value, never commit it, never pass it to a subagent. + +## C6. DevTools / CDP profiling *(manual)* +- **Proves:** a flame-chart hot path shrank, a request was removed/deferred, frame rate held, or it holds on slow hardware. +- **Capture (manual via DevTools or `mm cdp`):** performance profile / flame chart; network waterfall (HAR) + request-count delta; **CPU throttling** (CDP `Emulation.setCPUThrottlingRate` — *no repo helper*, set it in DevTools); **animation/Rive FPS / dropped frames** (DevTools rendering FPS meter — *no repo helper*); JS coverage for dead-code. + +## C7. Memory stability over a flow *(manual)* +- **Proves:** a leak is fixed across repeated interactions (not one snapshot): retained heap stays flat, detached DOM nodes / listeners don't accumulate. +- **Capture:** DevTools heap snapshots before/after N cycles of the flow; compare retained size + detached nodes. +- Sub: redux dispatch/action count per interaction; network payload bytes; forced-reflow / layout-thrash count (DevTools Performance). + +## C8. Same-window app + DevTools capture *(manual)* +- **Proves:** the UI behavior **and** its internal evidence (console log, network row, storage state) in **one frame** — cause and effect temporally correlated in a single artifact. Two separate captures can't prove they came from the same run; one frame can. Canonical use: "the toast does NOT appear *while* the console shows the silent-handling path executed". +- **Capture (macOS, OS-level — Playwright `recordVideo` sees only the page viewport, never DevTools):** + 1. Tab-target DevTools: launch Chrome with `--auto-open-devtools-for-tabs` so DevTools opens **docked in the same window** (dock side persists per profile; set once via the DevTools ⋮ menu if a fresh profile defaults to undocked). + 2. MV3 **service-worker console has no dockable host** — open its dedicated inspector (`chrome://extensions` → *Inspect views: service worker*) and tile it flush beside the app window: `osascript -e 'tell application "Google Chrome" to set bounds of front window to {x, y, w, h}'` (the SW inspector is a Chrome window too and tiles the same way; CDP `Browser.setWindowBounds` also works per `windowId`).\ + 3. Record the union region, not a single window: stills `screencapture -x -R out.png`; video `screencapture -v -V -R out.mov`, then ffmpeg two-pass palette → GIF (recipe in [evidence-publishing](evidence-publishing.md)). First use prompts for macOS Screen Recording permission for the terminal. +- **Legibility rule:** console text dies in GIF downscale. Keep the GIF ≥720px wide, and pair it with (a) a full-res PNG of the same frame and (b) a text dump of the console via CDP (`Runtime.consoleAPICalled` on the SW target, `npx mm cdp` or a 20-line ws script) so the log lines are quotable/searchable. +- **Trust note:** arrange windows *before* triggering the behavior so the recording shows trigger → console line → UI (non-)reaction as one continuous take; a post-hoc composite of separate captures is exactly what this lane exists to avoid. + +--- + +# D. Build output + +## C9. Retention-path analysis — memory leak from code ⭐ *(static; lead for leak claims)* +- **Engine: the `memory-leak-hunt` skill.** For a memory-leak claim, delegate the analysis to `memory-leak-hunt` — it runs Phase-1 static pairing (and Phase-2 heap investigation if a primitive can't be paired) and returns the paired/unpaired sites + verdict. pr-validate keeps **memory leak** as the evidence category: it invokes the skill on the diff and packages the result (in-situ scan capture, plus the lifecycle test / retainer graph if Phase 2 ran) as the category's evidence. The lane spec below is the method that skill implements. +- **Proves:** "X is retained past its lifecycle boundary" / "collection Y grows unboundedly" — argued from code, no runtime needed. This is the lane that works at **review time** (does this PR *introduce* retention?) and leads fix-side validation (does the fix *break* the retention path?). C7 is the runtime corroborator, not the lead — leaks need many cycles to exceed noise. +- **Capture — the holder → held → boundary triple, per suspect:** (1) the **holder** (listener, closure, module singleton, accumulating collection, timer); (2) the **held set** — the *specific* objects pinned (list the closure's captures; note when a closure links two objects' GC); (3) the **outlived boundary** (`destroy()`, stream close, instance replacement, request completion). Method: **pair every acquire with its release site** (`on`↔`removeListener`, push↔drain, assign↔null) — the absence of the pair, cited at the acquire site, IS the finding. Four canonical shapes: unbounded accumulator (defeated guard, no drain) · stale-instance listeners on replacement · unremoved listener + capture set · retention past `destroy()`. +- **Scope to the diff, or you invent findings.** Classify every flagged primitive as *introduced by this PR* (in the added lines) vs *pre-existing* (already in the file). Charge only the introduced ones to the PR; report pre-existing un-paired primitives separately and uncharged. On extension#40684 the two new stream listeners each had a `removeListener` on `onStreamClosed` (the exact fix a reviewer suggested) and the new pending-request Map had its `.delete` — no leak introduced — while three pre-existing un-torn-down listeners were surfaced but left uncharged, matching how the human/bot reviewers treated them in-thread. This lane *is* the retention review automated; a heap snapshot (C7) is warranted only for an introduced primitive it cannot pair. +- **Corroborate:** a falsifying lifecycle test (force the boundary, assert release — listener count zero, singleton nulled, collection drained); C7 heap-over-flow with the **retainer graph naming the same path** the static argument named. +- **Trust-gate:** the triple must be specific ("this listener holds `patchStore` after `patchStore.destroy()`", not "might leak"); distinguish **bounded staleness vs unbounded growth** (severity differs); attribute **introduced vs pre-existing** honestly. + +## D1. Bundle-size diff +- **Proves:** the build grew/shrank by a measured amount. Use the bundle-size CI output or a local build size comparison. + +## D2. Chunk membership / source-map +- **Proves:** a module moved to the intended (lazy) chunk and no longer ships on the critical path. Requires the webpack build. Mirrors AEP `perf-chunks`. + +## D3. LavaMoat policy / supply-chain capability diff + - **Engine: the `supply-chain-audit` skill** (umbrella — lockfile/manifest diff, advisories, Socket Security, install scripts), which delegates capability grants to **`lavamoat-policy-diligence`** (per-grant call-site justification). Delegate the dependency change to the umbrella; it returns a disposition per lane. pr-validate keeps **supply-chain capability diff** as the evidence category and packages the output. Note the lanes are independent: a clean policy diff does not mean a safe dependency, and a known CVE never appears as a new grant. +- **Proves:** a dependency change (bump/add/lockfile) grants **no *unjustified* new capability** — the supply-chain-risk lane. Note the bar: for a bump the policy *will* change, so "empty diff" is the WRONG test; the right test is **every new grant is justified by the dep's function**. The framing generalizes past LavaMoat to any capability-containment mechanism. +- **Capture:** **Prefer the CI-generated policy whenever one is available.** `@metamaskbot update-policies` regenerates the policy files from a real run of the code and `validate-lavamoat-policies` fails the build on drift, so the committed policy on a bot-run PR *is* the authoritative artifact — diff that. Regenerating locally when a current CI policy exists only re-does a machine that is already trusted, and a local run's provenance is weaker (your node/OS/lockfile resolution, not CI's). **Local regen is the fallback**, for when the bot hasn't run yet, the branch is unpushed, or you need a variant CI didn't cover: `yarn webpack:lavamoat:policy:build` (`:mv2` / `:mv3` for variants) over `lavamoat/webpack/build/policy.json` (+ `policy-override.json`). Either way, `git diff` the policy across **all 8 variants** (mv{2,3}/{main,beta,flask,experimental}) — a grant can appear in one and not others. Then audit **grant-by-grant**: new **globals** (`fetch`, `importScripts`, `WebAssembly`) / **builtins** (`fs`, `child_process`) on a dep that shouldn't need them, new **packages** edges to powerful APIs, or an identifier substitution (`pkgC>name` replacing `pkgB>pkgA>name` = possible dep swap). Falsifier = a surprising grant ("I wonder what it's using this for"). Guide: lavamoat.github.io/guides/policy-diff/. `allowScripts` in `package.json` gates install scripts. + +## D4. Manifest permissions diff +- **Proves:** no permission/host-permission scope creep. +- **Capture:** `git diff app/manifest/v3/_base.json app/manifest/v2/_base.json` (+ `chrome.json`/`firefox.json`). Flag new sensitive perms (webRequest, broad host patterns). + +## D5. Build-variant matrix +- **Proves:** the change works across build types, not just main. +- **Capture:** `yarn build:test:flask` / `:beta` / `:mv2` (`ENABLE_MV3=false`, Firefox). Run the relevant lane per variant when behavior is build-type-gated. + +--- + +# E. Production telemetry + +## E1. Sentry query links (before/after) +- **Proves:** error-rate / transaction count / latency moved in prod. A link a reviewer opens beats a chart screenshot. +- **Capture:** Sentry MCP (`search_events`/`search_issues`) → hand the discover/dashboard link with the before/after window, scoped to the release. Projects: `metamask` = prod, `metamask-performance` = CI. +- **Boundary:** PRs that *add/change span instrumentation* (volume/quota) → `/sentry-quota`, not this lane. +- **Perf-PR promotion (standard, not just complementary):** for a **performance-focused PR**, the main-branch Sentry **trend** for the affected metric across the PR's merge (before/after the merge commit's release) is **standalone lead evidence** — CI already sends every main/release `startupPowerUserHome` / journey benchmark to Sentry, so the trend is a real before/after on the actual metric, continuously tracked, with no local run. Prefer it over a local paired A/B when a clean merge-boundary window exists: it sidesteps the stale committed-baseline trap (`benchmark-baseline-staleness-paired-ab`). Still bound to the trust gate — a **windowed, release-scoped, one-click-resolvable** trend link with the merge boundary visible, never a prose "looks fine." A local interleaved paired A/B (C5) remains the precision complement when the merge window is noisy or the metric CI doesn't track (selector-eval count, re-render count, INP-on-typing — none of which CI captures). + +## E2. Tempo distributed traces +- **Proves:** a span/transaction now appears / is shaped correctly (e.g. background-RPC tracing). Link the trace + note the release. + +## E3. Sentry error-event / breadcrumb shape +- **Proves:** an instrumentation PR captures the intended error-event state / breadcrumbs (relevant after the Sentry-v10 error-event capture changes). Show the captured event payload. + +--- + +# F. Extension integrity (high-stakes, extension-specific) + +## F1. State migration / upgrade ⭐ +- **Proves:** a persisted-state change doesn't corrupt existing users. +- **Capture:** migrations in `app/scripts/migrations/NNN.ts`, runner `app/scripts/lib/migrator/`; scaffold with `./development/generate-migration.sh NNN`. The `NNN.test.js` asserts `meta.version` and that the `changedKeys` Set covers only mutated controllers — i.e. untouched state is preserved. Run it; show old-state-in / new-state-out. + +## F2. Vault / keyring round-trip +- **Proves:** no key/vault corruption; encrypt→decrypt is lossless. +- **Capture:** `app/scripts/lib/encryptor-factory.ts` (`@metamask/browser-passworder`, PBKDF2). E2E: `test/e2e/dist/vault-decryption-chrome.spec.ts`; `test/e2e/tests/vault-corruption/`. Storage-size via `getFileSize` on the encrypted blob. + +## F3. Transaction simulation / gas +- **Proves:** tx behavior/balance-changes/gas are correct before submit. +- **Capture:** `app/scripts/lib/transaction/containers/enforced-simulations.ts`; e2e `test/e2e/tests/simulation-details/`; mock `test/e2e/tests/confirmations/mocks/simulation.ts` (returns `gasUsed`, `callTrace`, `stateDiff`, token balance changes). TX_SENTINEL_URL in `shared/constants/transaction.ts`. + +## F4. Provider / dapp connectivity +- **Proves:** dapp integration works (injection, connect, requests). +- **Capture:** `yarn dapp` (serves `@metamask/test-dapp` on :8080); EIP-6963 `test/e2e/provider/eip-6963.spec.js`; multi-provider `test/e2e/multi-injected-provider/`; EIP-1193 reconnect tests under `test/e2e/tests/mm-connect/`. + +## F5. Feature-flag matrix (on/off) +- **Proves:** correct behavior in both remote-flag states (the Perps-gating class of bug). +- **Capture:** remote-feature-flag-controller (`app/scripts/lib/update-remote-feature-flags.ts`); flags come from `client-config.api.cx.metamask.io/v1/flags` — **not** `.metamaskrc`. In e2e, mock the response (see `test/e2e/tests/remote-feature-flag/`) to force each state; read via `uiState.metamask.remoteFeatureFlags`. + +## F6. Snaps / multichain execution +- **Proves:** snap behavior across multichain (e.g. `snap_startTrace`/`snap_endTrace`). +- **Capture:** `test/e2e/flask/snaps/preinstalled-example.spec.ts` (the snap-trace test), broader `test/e2e/snaps/`. Build flask (`yarn build:test:flask`). + +## F7. i18n usage +- **Proves:** no hardcoded strings; locales resolve. +- **Capture:** `yarn verify-locales` (`development/verify-locale-strings.js`); locales in `app/_locales/`. `yarn verify-locales:fix` to auto-fix. + +## F8. SES lockdown / runtime containment ⭐ +- **Proves:** the runtime defenses are **actually in force in the shipped artifact** — SES `lockdown()` and its taming levels, LavaMoat global scuttling, Snow's anti-escape hooks, Snaps compartments. Distinct from D3: D3 is the build-time *policy* (what a package may reach), this is whether containment *holds at runtime*. A correct policy ships alongside a lockdown that silently failed, and no policy diff would show it. +- **Capture:** `Runtime.evaluate` over CDP against the **built variant under discussion** — `Object.isFrozen(Object.prototype)`; a scuttled global throws while an exception-list global still resolves; `typeof SNOW === 'function'`; the `lockdown({…})` options as they appear *in the bundle*. Pair a positive with a negative — a check that only confirms the permitted case passes in a completely unlocked environment. +- **Bar — three divergences make this a lane, not a checkbox:** (1) the `lockdown()` call is wrapped in `try/catch` that logs to Sentry and **continues unlocked** (added for Firefox v56 contentscript injection), so it is a runtime assertion, never a guaranteed precondition; (2) **scuttling is off entirely in DEV builds** (`shouldScuttle = entryTask !== BUILD_TARGETS.DEV`); (3) **TEST builds widen the scuttling exception list** for chromedriver (`Proxy`, `ret_nodes`, `browser`, `chrome`, `indexedDB`). So **a green e2e run is evidence about a wider-open global than users get** — always state which build variant produced the evidence. +- **Reach for it:** any change touching the lockdown call site or its ordering (lockdown must precede untrusted code), the scuttling exception list, a taming level, compartment boundaries, or a `@lavamoat/snow` bump (Snow is patched in-repo — re-read the patch; see `supply-chain-audit`'s patch lane). + +--- + +# G. CI, review & process + +- **G1. CI check links** — `gh pr checks `; link the full suite (AEP's bundle is often `partial`). Always worth a one-line "all green" + link. +- **G2. Coverage delta** — `yarn test:unit:coverage` → `coverage/unit/` (and `yarn test:unit:webpack:coverage`); `codecov.yml`. Proves the new code is exercised. +- **G3. Automated-reviewer output** — independent bot (e.g. cursor[bot]) found nothing blocking. Complements, never replaces, behavior evidence. +- **G4. Manual reproduction steps** — human-followable steps that reproduce the fixed behavior; populates the PR template's Manual testing steps. +- **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: in a fork you control, push to a branch literally named **`main`** (or `stable`) — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets configured on that fork (the benchmark jobs need the Infura and test-account secrets; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo. + +--- + +# Matching guide (claim → lanes) + +| The PR claims… | Lead with | Corroborate | +|---|---|---| +| a visible UI behavior | A1 / B1 visual | B2 recording for motion; B5 a11y | +| a fixed bug (any) | **B3 falsifying test** | A1/B1 if visible; E1 if it errored | +| preload / no-double-fetch / lazy-load | A2 perf | C6 netlog, D2 chunk | +| a render/over-render fix | C4 WDYR/profiler | C1 traces | +| interaction responsiveness | C2 INP, C3 TBT | C6 profile | +| startup/load timing | C5 benchmark (paired) | C1 phase traces, C2 FCP/LCP | +| smaller/cleaner bundle | D1 size | D2 chunk | +| a memory leak fixed / introduced | **C9 retention-path from code** (holder → held → boundary) | C7 heap-over-flow + retainer graph; falsifying lifecycle test | +| an error/crash fixed | E1 Sentry rate→0 | B3 test, A1 if visible | +| a dep change is safe | D3 LavaMoat + D4 manifest | D1 size; supply-chain-audit's patch/resolutions/ignore lanes | +| runtime containment / SES / scuttling | **F8 runtime containment** (on the shipped variant) | D3 policy; E1 for `Lockdown failed` events | +| persisted-state change | **F1 migration** | F2 vault | +| tx/confirmation behavior | F3 simulation | B2 e2e | +| dapp/provider behavior | F4 connectivity | B2 e2e | +| flag-gated behavior | F5 flag matrix | A1/B1 per state | +| snap behavior | F6 snaps | E2 trace | +| copy/localization | F7 i18n | A1 visual | +| CI workflow behavior | **G5 fork run** (branch named `main`) | G1 checks, G4 repro steps | + +Run the cheapest lane that yields an independently re-checkable artifact, confirm the claim holds, then escalate. Don't over-instrument a one-line copy fix; don't under-prove a startup-latency or migration claim with a single screenshot. diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-gate-setup.md b/domains/pr-workflow/skills/pr-validate/references/evidence-gate-setup.md new file mode 100644 index 00000000..f1df5b9c --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-gate-setup.md @@ -0,0 +1,58 @@ +# Evidence gate — setup (optional, Claude Code only) + +`hooks/pr-evidence-gate.py` is an **optional** mechanical enforcement of the disciplines documented in [`evidence-trustworthiness.md`](./evidence-trustworthiness.md). It is a Claude Code `PreToolUse:Bash` hook: before an outward-facing write runs, it scans the body for a validation-scoped claim the trustworthiness gate would reject, and blocks the write if it finds one. + +**Surfaces policed:** the `gh pr|issue edit|create|comment` porcelain (`--body`, `--body-file`) *and* `gh api` body writes (`-f body=…`, `-F body=@file`, `--input file.json`) — a PATCH to a comment is the same publish with a different spelling, so a porcelain-only matcher is a hole rather than a gate. Read-only `gh api` calls pass through untouched. + +**Classes enforced:** `verdict`, `observation`, `deferral`, `ci-restatement`, `inflated-verdict`, `bare-identifier`, `truncated-identifier`, `mutable-ref`, `dump-resolver`, `link-only-exhibit`, `data-only-exhibit`, `step-waiver` — each implementing a numbered item of [`evidence-trustworthiness.md`](./evidence-trustworthiness.md). What the hook cannot see (whether a screenshot shows the resolving UI, whether a deferral's blocker matches its step, quotation fidelity) stays reader-applied. + +The hook is **Claude-Code-specific**. Other operators (Cursor, Codex, plain review) don't get the mechanical gate — for them the same disciplines apply as *documentation*, self-enforced by reading `evidence-trustworthiness.md`. The hook is not required to use the skill; it just moves the checklist from "remember to run it" to "runs automatically at emit time." + +It **fails open**: anything it cannot parse (non-`gh` command, unreadable body, malformed JSON) is allowed through, so it never bricks unrelated Bash commands. It uses the Python 3 standard library only (`json`, `re`, `sys`) — no dependencies to install. + +## Wire it up (Claude Code `settings.json`) + +Add a `PreToolUse` hook with matcher `Bash` that runs the script with `python3`. Put this in your user `~/.claude/settings.json` or a project `.claude/settings.json`: + +```json +{ + "hooks": { + "PreToolUse": [ + { + "matcher": "Bash", + "hooks": [ + { + "type": "command", + "command": "python3 /absolute/path/to/pr-validate/hooks/pr-evidence-gate.py" + } + ] + } + ] + } +} +``` + +Resolve the path to wherever `pr-validate` lives on disk. Note that `tools/install` copies only the `references`/`scripts`/`assets`/`adapters` bundles into `~/.claude/skills/mms-pr-validate/` — the `hooks/` directory is **not** part of the installed bundle. Point the `command` at your checked-out skills repo instead: + +``` +/domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py +``` + +**When it blocks:** the hook exits `2` and prints the reason (which claim, what artifact/tracker it needs) to stderr. Claude Code surfaces that to the model, which self-corrects — attaches the missing artifact/tracker or downgrades the verdict — and re-posts. No manual intervention needed. + +## Two other setup requirements the skill needs + +These are independent of the hook; the skill needs them whether or not you install the gate. + +1. **`gh pr comment` must be permitted — pick a grant model.** pr-validate posts its evidence bundle as a PR review comment (`gh pr edit` if publishing into your own PR body). Four options, in descending order of standing safety: + + | Model | How | Tradeoff | + |---|---|---| + | **`ask` (recommended)** | `"Bash(gh pr comment:*)"` in `permissions.ask` | Per-post confirmation prompt. Combined with this hook (content gate) and a draft-confirm habit, that's three independent layers. | + | **`allow` + hook** | same pattern in `permissions.allow`, hook wired | Frictionless posting; safety rests entirely on the hook and your draft discipline. Only sensible where the hook is actually installed — not for operators without hook support. | + | **Allowlisted wrapper** | keep raw `gh pr comment` denied; allowlist a small script that takes `--repo`/`--pr`/`--body-file`, checks preconditions (canonical header present), and is the only sanctioned path | Tightest scoping — the raw verb stays blocked; costs a script to maintain. | + | **No grant — manual post** | the model prepares the body file; you run `gh pr comment --repo --body-file ` yourself | Zero standing grant; you are the bottleneck. The universal fallback, and the only option on operators with no permission system. | + + Avoid a bare **deny** on the comment verbs if you use this skill: it hard-blocks the publish step with no prompt, which reads as a mysterious failure mid-run. + +2. **Image re-hosting needs your own public evidence repo.** Screenshots and recordings captured locally must be re-hosted to a public URL before a reviewer can see them (see items 8–9 in `evidence-trustworthiness.md`). This repo is **yours to provide** — set it to a public repo you control, referenced here as ``. There is no shared/default host: parameterize it in your own configuration and push captures there, then reference the resulting raw URLs in the PR comment. Do not hardcode someone else's host. diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md b/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md new file mode 100644 index 00000000..b1a46e1c --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md @@ -0,0 +1,291 @@ +# Publishing the evidence bundle to a PR body + +How to take run artifacts + complementary evidence and write a clean, idempotent, reviewer-familiar section into the PR body — **matching AEP's own format** so a re-run replaces in place instead of stacking duplicates. + +Canonical source for the format: `packages/github/src/pr-body-builder.ts` (`upsertVisualValidationSection`) in the [AEP repo](https://github.com/MetaMask/metamask-autonomous-engineering-platform). Mirror it. + +> **Publishing is public and outward-facing. Always render the section and get explicit confirmation before writing the PR body. Use `publishEvidence: false` on the run; this manual flow is the only publish path.** + +## Step 1 — Re-host images (artifacts are localhost) + +Control-plane artifact URLs (`localhost:3000/v1/runs/:id/artifacts/:name`) won't render on GitHub. Re-host each artifact and link the hosted URL. + +**Host: an object store or repo whose read access matches your audience.** Configure it once and +reuse it; the examples below assume an S3 bucket exposed through an environment variable: + +```bash +# set these to a bucket you control whose `public/` prefix allows anonymous GetObject +EVIDENCE_BUCKET= +EVIDENCE_BASE="https://$EVIDENCE_BUCKET.s3..amazonaws.com" +``` + +``` +s3://$EVIDENCE_BUCKET/public/metamask/pr-// +$EVIDENCE_BASE/public/metamask/pr-// +``` + +Allow anonymous `GetObject` under `public/*` but not bucket listing, so the prefix is not +browsable — link individual files, and don't promise readers an index. + +**Do NOT re-host to a personal repo.** A personal private repo returns 404 for every reader but +its owner, so every raw link to it is dead on arrival. + +The test is **audience-reachability, not public-vs-private.** An org repo that is private but +readable by colleagues is fine for an internal-audience link. A personal repo is unreachable by +colleagues *and* by the public, so it fails for every audience. + +- Path convention: `pr-//` keeps runs from colliding. +- **Verify unauthenticated before shipping**: `curl -s -o /dev/null -w "%{http_code}"` on each + published URL. A 200 from your own browser proves nothing — you are logged in. + +```bash +RUN_ID=; PR=; CP=localhost:3000 +BUCKET="$EVIDENCE_BUCKET" +BASE="$EVIDENCE_BASE" +for name in ; do + curl -fsS "$CP/v1/runs/$RUN_ID/artifacts/$name" -o "/tmp/$name" + key="public/metamask/pr-$PR/$RUN_ID/$name" + aws s3 cp "/tmp/$name" "s3://$BUCKET/$key" --only-show-errors + url="$BASE/$key" + # the link is not shippable until it resolves WITHOUT credentials + code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 25 "$url") + [ "$code" = "200" ] || { echo "UNREACHABLE ($code): $url" >&2; exit 1; } + echo "$url" +done +``` + +No base64 round-trip and no 1 MB contents-API ceiling — the ceiling silently truncated a +1.7 MB gif to **0 bytes** on one run, and the loop reported success. Size-check anything you +transfer by another route. + +Files >1MB exceed `ARG_MAX` for an inline `-f content=` — use `gh api -F content=@` (write the base64 to a file first). For GIFs, re-host the same way. + +## Step 2 — Build the section (canonical header + mirror AEP) + +**Canonical header (2026-07-21):** every validation-run output — a PR comment *or* the PR-body section — leads with the exact literal `## 🧪 Validation Run`. Never reworded, never demoted to `###`: the constant string is the identifiability anchor, exactly like Copilot's fixed `## Pull request overview`. `hooks/pr-evidence-gate.py` blocks any `gh` write whose body has a validation/verification/evidence heading or AEP marker without this literal. + +Marker pairs, used so re-runs replace idempotently: + +- Whole section: `` … `` +- AEP status block (nested): `` … `` +- Screenshots block: `` … `` + +AEP prefers to inject screenshots into the PR template's `### **After**` section (replacing the `` placeholder), falling back to a `### Screenshots` block inside the status block when there's no After scaffold. Do the same. + +Section shape: + +```markdown + +## 🧪 Validation Run + +**Verdict:** ✅ proven — **Claim:** +head `` · · lanes: + + + + +### AEP Visual Validation + +**✅ Passed** + + + +
Validation details + +** — . " lines> + +
+ +Run `` · [LangSmith trace]() + + +``` + +Verdict icon: `✅` Passed, `❌` Failed, `ℹ️` otherwise. For perf, retitle the nested block `### AEP Perf Validation` and put `M/M assertions proven` in the headline. When AEP's *service* publishes its own `## AEP Visual Validation` block (publishEvidence:true, not the local flow), leave that block's heading alone — the demotion to `###` applies to hand-assembled bundles under the canonical header. + +Screenshots block (injected into `### After`, or appended under `### Screenshots`): + +```markdown + +
+ +<artifact-name> + +[Open full-size image]() + +
+ +``` + +`
` so reviewers see evidence without a click. One block per image; before/after read top-to-bottom. + +## Step 3 — Choose the surface by ownership, then publish + +**Publish surface depends on your relationship to the PR.** Determine it FIRST: + +```bash +PR=; REPO=MetaMask/metamask-extension +ME=$(gh api user --jq .login) +SURFACE=$(gh pr view "$PR" --repo "$REPO" --json author,commits --jq --arg me "$ME" ' + if .author.login==$me then "body" + elif ([.commits[] | select(.authors[].login==$me) + | select([.authors[].login] | map(select(.!=$me and .!="Copilot" and (test("claude|anthropic")|not))) | length == 0)] | length) > 0 + then "comment" else "skip" end') +``` + +- `body` — I authored the PR → upsert into the PR body (below). Validation is + part of my own claim. +- `comment` — not author but I have a solo commit (no HUMAN co-author) → post a + `gh pr comment` under the canonical `## 🧪 Validation Run` header. Never edit + someone else's PR body. +- `skip` — my only commits are co-authored with a human (review/pairing) OR I + have no commits → **do not publish**. Not my PR to validate outward. + +### Publish the script that produced a computed artifact, next to the artifact + +Any number you derived rather than read off a tool — a hash comparison, a count, a delta, a +statistic — is only as trustworthy as the reader's ability to re-run it. **A prose `method:` field +is not provenance.** Reviewers discount computed figures from an agent by default, and correctly: +on extension#45024 a reviewer dismissed a policy-identity check as *"we know LLMs are really bad at +this"*. It had in fact been a deterministic `sha256`, not the model counting — but the script lived +in a throwaway `python3 - <<'PY'` heredoc, so nothing could show that. The objection was +unanswerable because of how the evidence was packaged, not because of what it said. + +So: + +- **Write the script to a file, never an inline heredoc**, when its output will be published. The + heredoc survives only in the transcript, which the reader does not have. +- **Publish the script alongside its output**, and cross-reference: the artifact carries + `provenance: { script, script_sha256, command }`; the comment links the artifact. +- **Include the exact command** with its inputs (PR ref, head SHA), so the run is reproducible + rather than merely described. +- **Verify the round trip** — fetch the published script anonymously, hash it, and confirm it + matches `script_sha256`. A link that 200s is not proof the bytes are the ones you ran. +- Prefer a script that takes arguments and is re-runnable against a different PR. A one-off that + only works on your paths is weak provenance even when published. + +State plainly what the script does and does not do (`no model judgement; not a count of '+' +characters`) — that sentence is what actually retires the reviewer's prior. + +### Before any of the commands below: show the body in the response + +Every publish path here uses `--body-file`, so the **permission prompt displays a file path, not +the content**. The user is then asked to authorize publishing something under their name that they +cannot read, and the correct answer to that is no. + +**Paste the complete body inline in the response first, then run the command.** For an edit, also +say what changed relative to what is currently live. "I've drafted it, shall I post?" with a path +instead of the text is incomplete — pointing at `/tmp/validation-run.md` is the same failure as the +prompt itself. If the body is too long to show comfortably, that is a signal to trim it. +(Three consecutive denials on extension#45024, 2026-07-30, all from this.) + +### body surface (I own the PR) +```bash +gh pr view "$PR" --json body -q .body > /tmp/pr-body.md +# Replace the region between VALIDATION_RUN markers if present, else append. +# (Legacy bodies: replace the AEP_VISUAL_VALIDATION region and re-wrap it under +# the canonical "## 🧪 Validation Run" header + VALIDATION_RUN markers.) +# Replace the region between AEP_SCREENSHOTS markers if present, else inject after +# the "### **After**" heading (replacing the [screenshots/recordings] placeholder). +# ...edit /tmp/pr-body.md... +gh pr edit "$PR" --body-file /tmp/pr-body.md +``` + +### comment surface (I contributed but don't own) +```bash +# Same canonical "## 🧪 Validation Run" header + bundle; post as a comment. +gh pr comment "$PR" --repo "$REPO" --body-file /tmp/validation-run.md +``` + +Idempotency: because both regions are marker-delimited, re-running replaces them — never append a second copy. If the markers are absent (human-authored body), append the status block at the end and inject screenshots into `### After` when that heading exists. + +## Step 4 — Privacy scrub (before writing) + +Failure summaries and agent narratives leak the dev environment. Before publishing, strip: + +- Absolute local paths (`/Users//…`, `~/Code/…`) → describe the surface, not the path. +- The username anywhere it appears. +- `localhost` / `127.0.0.1` URLs → must be re-hosted public URLs only. +- Internal hostnames, JFrog/registry URLs, tokens. + +A failed run still must not publish raw — either omit the section or publish a scrubbed `❌ Failed` summary, with confirmation. + +## Recordings → GIF (for flows/motion a still can't prove) + +The platform can't collect video (artifact regex = png/jpg/log/txt). Capture out-of-band: + +1. In a **built** PR checkout (mm's fixture infra is required — a bare `dist/chrome` won't boot), write a preload `/tmp/patch-record.mjs` that monkey-patches `playwright-core`'s `chromium.launchPersistentContext` to inject `recordVideo: { dir }`. Resolve the module via `createRequire(/package.json)` so the patch hits the same module instance the `mm` daemon uses. +2. `NODE_OPTIONS="--import /tmp/patch-record.mjs" npx mm launch --state onboarding` → drive the flow (or let it sit) → `npx mm stop` flushes the `.webm`. States: `default | onboarding | custom`. +3. Convert with `ffmpeg` two-pass palette (better color than single-pass): + ```bash + ffmpeg -i in.webm -vf "fps=12,scale=480:-1:flags=lanczos,palettegen" -y /tmp/pal.png + ffmpeg -i in.webm -i /tmp/pal.png -lavfi "fps=12,scale=480:-1:flags=lanczos[x];[x][1:v]paletteuse" -y out.gif + ``` + webm/mp4 don't render inline in GitHub PR bodies; GIF does. +4. Re-host the GIF (Step 1) and embed like a screenshot. + +**Same-window app + DevTools (lane C8):** when the claim needs UI + console/network in one frame (e.g. "no toast *while* the log shows the silent path ran"), skip `recordVideo` entirely — it can't see DevTools. Use the OS-level region capture in [evidence-catalog C8](evidence-catalog.md): dock tab DevTools with `--auto-open-devtools-for-tabs`, tile the SW inspector window via `osascript`/CDP `Browser.setWindowBounds`, then `screencapture -v -V -R` → same ffmpeg GIF recipe. Publish the GIF + one full-res PNG + the CDP console text dump (GIF downscale makes log lines illegible on their own). + +## Re-validation runs: delta-first presentation, every verdict re-earned (2026-07-21) + +The common loop — a run refutes a claim, the author pushes a fix, `/pr-validate` re-runs at the new head — gets a **delta report**, not a second full bundle: + +- **Presentation is delta-only.** Full exhibits only for lanes whose outcome changed (flipped verdict / new lane / new residual). Unchanged lanes collapse to a `Prior run | This run` ledger, each row with a fresh run-log link from the new head plus one link to the prior run's comment for the full exhibits — and say so ("unchanged rows re-run at ``; full exhibits in the prior run"). +- **Evidence is never delta.** Evidence is head-pinned: re-run every automated lane at the new head and re-earn every verdict with a fresh artifact. "Unchanged" is a conclusion from the re-run, never a carried-over assumption (the stale-baseline trap at report level). Re-running is cheap — the falsifier harness already exists from the first run. +- Same canonical header + markers; the meta line names the fix commit and links the prior run. Comments: one per run, chronological, each linking its predecessor. PR-body section: replaced in place via markers. +- New head → **new hosted artifact directory keyed to the fix commit** (`pr-/fix-/`), commit-pinned raw URLs; never overwrite a prior run's published files. +- Residuals the fix intentionally leaves get their own row/section — don't round a fixed-with-residual claim up to fully proven. + + +## Lead with a lane-status ledger (no silent absence) + +The published section must **enumerate every lane the claim type calls for and give each an explicit status** — never render only the lanes you happen to have and let the rest be silently absent. An unmarked gap is indistinguishable from a lane that ran and came back empty; the reader (and you, on the next pass) can't tell "no evidence because none needed" from "no evidence because not done." This is the vacuous-pass trap at the publish layer — carry the run's `✅/❌/⚠️` verdict into the PR body, don't leave it in the internal report-back. + +Open the evidence section with a ledger: + +```markdown +| Lane | Status | Evidence | +|---|---|---| +| B3 falsifying test | ✅ proven | 32/32 head, 3/32 reverted | +| E1 Sentry before/after | ✅ proven | [discover](…) — distinct trace ids | +| A1 visual | ➖ N/A | background change, no UI surface | +| C6 CDP netlog | ⏳ not-captured | — | +``` + +Status vocabulary: `✅ proven` (link) · `⚠️ inconclusive` (name what's missing) · `➖ N/A` (reason) · `⏳ not-captured`. Mirror the `N/A — ` convention the `### Screenshots` block already uses for no-UI PRs. Never upgrade a `⏳`/`⚠️` to a pass by omission. + +**Sibling-PR parity:** when a set of PRs shares a claim shape (same program, same author, "root the X traces"), their ledgers must match lane-for-lane. A lane present on one and absent on another is either added or explicitly marked `➖ N/A — ` — a bar that silently drifts between siblings is a finding (postmortem 2026-07-17, #43929/#43930). + +## Non-visual & multi-lane evidence + +Screenshots are only one lane. Most claims (perf, telemetry, state, build) publish as **text/links/tables**, not images. Put them in the same verdict-first section so a reviewer sees one coherent bundle, not scattered comments. + +Per-lane rendering: + +- **Sentry / Tempo (E1/E2):** a markdown link to the discover/trace query with the before/after window baked in, plus the headline numbers inline (`errors: 1.2% → 0.0% over 24h post-release`). Link, not screenshot — reviewers re-run it. +- **Benchmark / web-vitals / TBT (C2/C3/C5):** a small before/after table (metric · base · head · Δ · threshold). State it's a **paired A/B** if the stored baseline was bypassed. +- **Migration (F1):** the `changedKeys` set + a before/after state-shape snippet, and a link to the migration-test run. +- **Bundle / chunk / LavaMoat / manifest (D1–D4):** the diff or size delta in a fenced block; for policy/manifest, the actual `git diff` (or "diff empty — no new capability"). +- **Trace artifacts (B2):** link the Playwright trace-viewer report / attach the `trace.zip`; don't paste raw. + +Multi-claim PRs get one sub-block per claim under the status section, each with its own ✅/❌/⚠️ verdict — mirror the Claim Cards. Keep the visual block (markers + `### After` injection) for the image lanes; render the rest as text beneath it. + +**Per-scenario presentation (2026-07-21):** the same applies one level down — when the evidence spans multiple test scenarios (flag-on vs flag-off, control vs treatment in an A/B falsifier, numbered manual-testing steps), give each scenario its **own sub-section**: a heading naming the scenario in observation terms, one line on what it tests plus its verdict, and that scenario's artifacts co-located under it. Never bunch all scenarios' artifacts into one large evidence dump — the reviewer verifies "under condition X, artifact shows Y" one condition at a time, and a merged block destroys that mapping even when every artifact is real. For long artifact sets use a `
` block *per scenario*, not a merge. + +## Artifact contract (ADR-0058 alignment) + +To stay interoperable with the recipe-based verification system (MetaMask/decisions#173), shape the bundle like its reviewer-visible contract where practical: a `summary.json` (claim → verdict → evidence refs), a `trace.json` (the run/assertion log), and an artifact manifest (names + media types), with screenshots/video as the confidence layer. Publishing then becomes "render `summary.json` into the PR section." This keeps pr-validate's output and a recipe's output the same shape — see [lane-assertions.md](lane-assertions.md). Don't hand-roll a divergent format. + +## Checklist before you publish + +- [ ] Section opens with a **lane-status ledger** — every claim-required lane marked `✅`/`⚠️`/`➖ N/A`/`⏳`; no lane silently absent (and sibling PRs' ledgers match lane-for-lane) +- [ ] `evidenceBundle.artifactRefs` non-empty with expected media (not a vacuous pass) +- [ ] Each lane passed the [trustworthiness gate](evidence-trustworthiness.md) (shows the claimed surface, signal > noise, could-have-failed) +- [ ] Multi-scenario evidence rendered **per scenario** (own heading + verdict + co-located artifacts), not bunched into one block +- [ ] **Automated-process voice, no first person** — published validation output never says "I ran/captured/verified"; attribute to the process ("Automated validation ran…", "the harness captured…") so readers know the evidence is machine-generated, not a manual account under the author's name +- [ ] Every image/GIF re-hosted to your configured evidence host; no localhost/local-path URLs in the body +- [ ] **Every published link curl'd unauthenticated and returning 200** — never a personal private repo +- [ ] Work cited by **PR link** rather than tracking-ticket id, unless the ticket's own content (an RCA, a spec) is the referent +- [ ] Narrative scrubbed of username/paths/internal hosts +- [ ] Marker pairs present so the upsert is idempotent +- [ ] Section rendered and **confirmed by the user** diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md b/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md new file mode 100644 index 00000000..8933a8de --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md @@ -0,0 +1,43 @@ +# Evidence trustworthiness (anti-reward-hacking) + +A green result is not proof. An agent — or an eager run — can produce evidence that *looks* like it validates the claim but doesn't. Before believing or publishing any lane, run it through this gate. It extends the vacuous-pass trap to all lanes; the Claim Card's **Falsifier** is the anchor: trustworthy evidence is evidence that *could* have shown the falsifier and didn't. + +> **Which items are mechanically enforced.** Several items below close with an *"Emit-time trigger: `pr-evidence-gate.py` class …"* note. Every such class is implemented in [`hooks/pr-evidence-gate.py`](../hooks/pr-evidence-gate.py), which runs as a `PreToolUse:Bash` hook and blocks the write: `verdict`, `observation`, `deferral`, `ci-restatement`, `inflated-verdict`, `bare-identifier`, `truncated-identifier`, `mutable-ref`, `dump-resolver`, `link-only-exhibit`, `data-only-exhibit`, `step-waiver`. It polices both the `gh pr|issue edit|create|comment` porcelain and `gh api` body writes, since a PATCH to a comment is the same publish with a different spelling. +> +> **What stays procedural.** The hook sees vocabulary, not semantics. It cannot tell whether an embedded screenshot actually shows the resolving UI's chrome (item 17), whether a deferral's stated blocker matches the step's own precondition (item 14), or whether inline data is a faithful quotation rather than a transcription (item 16). Those remain reader-applied. Setup: [evidence-gate-setup](evidence-gate-setup.md). + +## The gate (per lane, before publish) + +1. **Non-empty & expected media** — the bundle has artifacts of the expected kind. Zero artifacts = not a pass (the vacuous-pass guard). +2. **Shows the claimed surface** — the screenshot/recording is the Claim Card surface in the asserted state — not a loading spinner, an error toast, the wrong screen, or a pre-action frame. Eyeball it. +3. **Exercises the changed code** — the test/flow actually hits the diff. For a test: it **fails on `main`** (catalog B3). For a flow: the changed component/route is on the path. A green test that never imports the changed module proves nothing. +4. **Signal exceeds noise — and a null states its power** — a perf delta must be beyond run-to-run variance (paired A/B, multiple iterations); a 3% move on a noisy metric is not evidence. The same bar applies in reverse: when the spread is wider than the effect being looked for, the finding is **"not resolvable at this sample size"**, never "no change" — an underpowered run and a true null print the same word, and reporting the word alone lets the reader infer the stronger claim. State the smallest effect the design could have detected. + - **Removing a bias is not establishing validity.** Correcting a flaw you found (discarding a warm-up, alternating the starting arm, pinning CPU governor) removes *that* bias and licenses no more than that. It is not a trust gate, because a trust gate names how the evidence could **still** be vacuous — residual risk, not completed work. List what remains uncontrolled (thermal drift, background load, ordering within a round); an unenumerated confound reads as a nonexistent one. + - **When correcting an overclaim, cut the certainty, not the evidence.** A falsifier that actually fired is the strongest thing on the page — downgrade the conclusion around it, don't delete it with the overclaim. +5. **Could have failed** — the assertion has a reachable failure mode. Always-true assertions (`expect(true)`, a screenshot with no assertion, a Sentry query with no time bound) can't falsify anything. +6. **Right baseline** — "before" is the actual base ref / prior version / pre-window, not a stale or mismatched comparison. +7. **Artifacts are independent & honestly labeled** — checksum every capture set (`md5 *`). Byte-identical files across supposedly independent runs/cases cannot stand as separate observations: either explain the identity in the artifact bundle (deterministic fixture rendering) with per-run provenance that *does* differ (the harness state dump, timestamps, a manifest), or re-capture at distinct moments. Labels must describe the observation, not the interpretation — a file named for the state it *should* show under the claim (`steady-state`, `no-toast`) misleads when the capture shows the refutation. +8. **The finding ships with its artifacts** — a findings comment (including a refutation shared privately) carries functional links to the re-hosted observation artifacts at *draft* time, not descriptions of artifacts that exist only on the capturing machine. "Would need re-hosting" is not a reason to omit: re-hosting is the procedure ([evidence-publishing](evidence-publishing.md) Step 1). Code permalinks + a runnable repro are corroboration, not a substitute for the observation itself. +9. **Signal is surfaced — least-effort validation** — evidence is judged at the reader's eyes, not the author's disk: signal the reader must excavate from a mountain of attached data is, for evidence purposes, no evidence. Every published exhibit leads with a one-line pointer — *what to open, where to look, what it should show*. Deltas are presented **as** deltas (annotated side-by-side, diff, before→after crop of the differing region), never two full captures for the reader to compare by eye; if the claim is "no visual change," publish one image plus the hash-equality line, never N identical-looking copies as separate exhibits. Bulk artifacts (MB-scale JSON, full logs) are excerpted inline to the discriminating lines, with the full file linked as appendix. Emit-time test, per exhibit: can a reader who did not run the session confirm the claim in ~30 seconds from what is directly visible? If not, restructure the presentation — attaching more data cannot fix it. Coverage is the converse constraint (2026-07-21): this item governs *form*, never column-set minimalism — a valid, relevant dimension is never omitted because it duplicates another's signal (redundant corroboration costs a skippable glance; an omitted column is unfalsifiable and reads as cherry-picking). Exclusion requires invalidity (metric void on this surface, e.g. TTFB on `chrome-extension://` pages) or irrelevance (different claim/different data → sibling exhibit, not a column), each stated in a one-line disposition. +10. **Parallel exhibits are format-uniform** — sibling exhibits (table rows, per-scenario blocks, the legs of an A/B pair) carry the same evidence format and quality. If one row links its artifact inline, every row does; if one scenario gets an annotated timeline, action-log provenance, and co-located full-res/raw links, every scenario does. The bar is the **best sibling**: when the presentation standard improves mid-session, re-normalize the whole document up to it before publish — never apply the improvement only to the exhibit being produced (append-only drafting). Any asymmetry carries an explicit stated reason co-located with the weaker exhibit ("close-event variant unit-uncoverable", "manual-only trigger"); an unexplained format gap reads as an evidence gap — the reader cannot tell an unlinked artifact from a missing one, and inconsistency spends credibility on *every* exhibit, including the strong ones. Emit-time test: enumerate the sibling sets, diff each against the best-formatted member, and for every deviation either normalize it or state the reason. +11. **Lanes derive from the Manual testing steps — a CI-green row is not a lane** — the Validation Run's rows are generated top-down from the claim and the PR's own **Manual testing steps**, never bottom-up from whatever links already exist. For each step the claim depends on, the lane's payload is the **captured output of executing that step** (step "in Discover, group by `trace`" → a Discover permalink / **linked** trace-id table showing N rounds → N distinct `trace_id`s, per item 12), or an honest ⏳ naming the missing capture with a tracker. A row restating CI ("tests green at head `` in [CI run]") duplicates the Checks tab and is deleted — and a validation surface carries **zero** CI references, full stop: no `actions/runs` links, no "green at head" clauses, no "as context (only)" retention. The earlier carve-out here ("a CI link is admissible as context on a beyond-CI row") was itself the next costume: within a day all four sibling bodies (extension#43928–#43931) shipped restatements phrased as the exception — rows *leading* with "green at head … in [Unit tests CI]", the same link repeated 3× per body, the remediated row keeping it re-labeled "as context only" — while the gate's excuse regex matched the mere word "revert", so vocabulary, not evidence, discharged the class. The revert lane cites the revert **outcome** (which blocks failed, at which commit); its green-at-head half is the Checks tab's information and is omitted. A carve-out in an emit-time gate is an instruction to generation to phrase every violation as the exception — deliberate exceptions route through the human, never through an excuse predicate. Borrowed evidence — a sibling PR's capture, a unit falsifier standing in for the named live surface — never upgrades an uncaptured lane to ✅: "mechanism live-proven" co-located with "was not exercised" is an inflated verdict; downgrade it. Emit-time trigger: `pr-evidence-gate.py` classes `ci-restatement` (unconditional since 2026-07-21: any CI link / CI-green phrase in validation scope fires — no verdict co-location required, no beyond-CI excuse) and `inflated-verdict`, with the shipped extension#43928 rows and the carve-out-blessed "as context" shape as regression cases (2026-07-21). +12. **Identifiers resolve in one click — a bare id is a digging assignment** — trace ids, event ids, run ids, SHAs are *pointers into a system*, not evidence. Publishing a bunch of raw trace ids hands the reviewer the job of reconstructing project/environment/time window and querying Sentry themselves — it fails item 9's ~30-second test by construction (item 9 makes the signal *findable*; this item makes it *checkable*). Every identifier published as evidence is either hyperlinked to its resolving surface (the Sentry trace/event permalink, or an absolute-windowed Discover query pre-filtered to exactly those ids) or accompanied by the re-hosted captured output (query-result rows / envelope excerpt showing the discriminating fields) — ideally both. Special case that produced the rule: ids captured **locally** (mockttp forwarder, envelope intercept) never reached Sentry, so no permalink can exist — the re-hosted capture is the *only* admissible form, and pasting the id fragments plus a re-run recipe is the "spec necessary / output sufficient" violation wearing ids as decoration (extension#43931 Validation row, 2026-07-21). Rule of construction: when any item in this gate blesses an evidence class by name ("trace-id table", "envelope log"), it means the class's *resolvable instance*, never its bare tokens — a blessed class name is otherwise the next costume. Emit-time trigger: `pr-evidence-gate.py` class `bare-identifier`; converse-of-gate note: the prior gate *whitelisted* `trace_ids?` as beyond-CI payload and its own fix-message recommended "trace-id table" unqualified — second occurrence of "audit the gate for whitelists of the violating shape." +13. **Terminal exhibits are reader-native — a live link or a visual; a dump behind a link is still an opaque reference** — item 12 makes every pointer resolve in one click; this item constrains what it may resolve *to*. A positive verdict's terminal artifact is one of the two media a reviewer natively consumes: a **live link into the resolving system** (Sentry trace/event permalink, absolute-windowed Discover query pre-filtered to the claim) or a **visual capture** (screenshot/recording, annotated or cropped to the discriminating region). Raw files (`.log`/`.json`/`.har`, MB-scale dumps) are **appendix-only** — linked once for auditability, never the exhibit a claim rests on: a link whose target is a raw dump passes item 12 and fails item 9 one click later; the digging moved a hop away, it did not disappear (extension#43931 *second* remediation, 2026-07-21: the `bare-identifier` fix shipped a ✅ row whose sole resolver was a re-hosted ~70KB run log). Two corollaries: (a) **the gate items are conjunctive** — a fix for the newest item must re-pass all prior items; satisfying resolvability with an artifact that fails legibility is the generator's next costume; (b) **ascertain the terminal medium at step zero and pick the capture lane that can produce it** — a local intercept (mockttp envelope forwarder) can never yield a live Sentry permalink, so for Sentry-observable claims it is the supplementary falsifier lane and live ingest (dev build → `SENTRY_DSN_DEV`/test-metamask) is primary, precisely because it terminates in permalinks + screenshots; choosing a lane that cannot produce the terminal medium silently displaces it. Emit-time trigger: `pr-evidence-gate.py` class `dump-resolver`, with the remediated extension#43931 row as the regression case (2026-07-21). +14. **Manual testing steps are the validation contract — steps present ⇒ live evidence definitionally required; an impossibility waiver contradicting an executable step is invalid** — the PR's own **Manual testing steps** are the author's assertion that the claimed behavior *is* live-observable, and how: each numbered step is an executability proof (a step a human reviewer can run, `/pr-validate` can run) and its text is the capture spec. Item 11 derives the lanes from the steps top-down; this item closes the other side of the hatch — a lane derived from a step may not then be *waived by argument*. Emit-time procedure: build the per-step coverage map (step → executed-output artifact); for any step without one, the only admissible state is a **per-step** ⏳ + tracker whose blocker is that step's *own* unmet precondition, checked against the step text. Three waiver-inflation patterns from the producing instance (extension#43228/#42869/#44538, 2026-07-21 — all three shipped articulate impossibility rationales in the same body whose Manual testing steps asserted the opposite): (a) **borrowed impossibility** — the excuse imported from a different mechanism or sibling PR (#43228 waived its live lane citing the async remote-flag read race, which is #44538's mechanism; #43228's overrides are build-time env vars, and its steps 1–4 are directly executable in a dev build); (b) **lane-limitation universalized** — one harness's gap stated as global impossibility (#42869: "the e2e harness emits no error `event` envelope, so … not capturable pre-merge" — the step says *dev build with Sentry enabled, trigger an error*, which ingests error events into test-metamask without the e2e harness); (c) **blocked-scope inflation** — a genuinely blocked precondition of one half of the claim expanded to waive the whole lane (#44538: LaunchDarkly provisioning (tracked internally) blocks only the *prod-flag* half; the step's own text names the dev-injectable alternative — "or inject it into persisted `RemoteFeatureFlagController` state"). If a step is *truly* non-executable, the waiver is still inadmissible alone: the Manual testing steps are then wrong and are corrected in the same edit — a document may not simultaneously instruct a reviewer to observe X and declare X unobservable. The honest-⏳ lane blessed throughout this gate is for *not yet done*, never for *argued away*: an eloquent impossibility rationale is the cheapest token sequence that satisfies every prior item (no fake capture, no CI link, no bare id, no dump) — the generator's costume for the coverage axis. Emit-time trigger: `pr-evidence-gate.py` class `step-waiver` ("not demonstrable / capturable / observable", "not separately captured", "not attached", "rests entirely/solely on the falsifiers/unit/revert" — unconditional in validation scope; no tracker or artifact excuses it), with the three shipped waiver paragraphs as regression cases and the gate verified-blocking on all three live bodies (2026-07-21). Detection gap: the per-step consistency check (does a deferral's blocker match the step's own precondition?) stays procedural — the gate sees vocabulary, not step semantics. +15. **The exhibit lives in the body — link AND visual; a live link alone is the verification path, not the exhibit** — item 13 blessed the terminal media as a *disjunction* (live link OR visual), and generation took the cheaper disjunct: a Discover permalink is producible from the API token alone, a screenshot needs a browser session — so extension#44540's live-ingestion exhibit shipped as a permalink + prose counts, with nothing in the PR body a reader could look at (2026-07-21: "only sentry link and not screenshot that makes it immediately obvious how evidence validates pr"). A live link defers validation behind **click + auth + query rendering + column interpretation** — the dump-resolver displacement one hop further, with the mountain now behind a login: it fails item 9's ~30-second test at the moment of the click, and for any reader *without* Sentry org access (most PR reviewers) a link-only exhibit degrades to a bare identifier (item 12) behind an auth wall. The repaired rule is a **conjunction**: a positive verdict's headline exhibit is an **embedded visual** — screenshot/recording of the linked resolving view (Discover result rows, trace waterfall), cropped/annotated to the discriminating region, captioned with what it should show — **and** the co-located live permalink (absolute-windowed) as the independent-verification path. Neither substitutes for the other: link-only hides the exhibit; visual-only is independently unverifiable. The 2026-07-16 clause "screenshots ride along when a browser session is available; the API token alone yields links + JSON, which is the automatable minimum" was the self-authored escape hatch of this axis (family: the "as context" carve-out, the honest-⏳ waiver): the *automatable minimum* got promoted to the shipped standard because it was the cheapest compliant artifact. A capture lane that cannot screenshot its resolving view is a lane gap to fix before publish (drive a browser session to the Discover URL), never a licensed downgrade — deliberate exceptions route through the human. Emit-time trigger: `pr-evidence-gate.py` class `link-only-exhibit` (non-negated verdict + `sentry.io` link + no image/recording embed in the unit), with the shipped #44540 paragraph as the regression case and the prior suite's permalink-only ALLOW cases flipped/augmented — third occurrence of "an ALLOW case containing the violating tokens is a specification of the next costume." Detection gaps: verdict co-location is required, so a no-verdict link-only paragraph evades mechanically; the visual-without-link converse stays procedural under item 12. +16. **The audit chain is mechanical — quote, don't transcribe; pin, don't point** — an exhibit's inline data must be a **verbatim, greppable excerpt** of the artifact (full-length identifiers, raw capture lines quoted exactly), and every repo-hosted artifact link must be a **commit-pinned, line-anchored permalink** (`/blob//…#Lx-Ly`) to the discriminating lines. Producing real artifacts and then hand-transcribing digests severs the claim→artifact bridge at every link: an ellipsized id (`24b1e2da…`) cannot be grepped against any artifact even when the log is linked in the same block; a reformatted data block cannot be distinguished from confabulation without redoing the dig, so it reads as *claims in the form of data*; a branch-ref `/blob//` link is a mutable pointer whose target can be rewritten after review (not tamper-evident); a file-level link without line anchors lands the reader at the top of a 1,000-line dump. The producing instance (extension#43929 validation comment, 2026-07-21) had every number substantiated by four re-hosted run logs — real, included, resolvable — and still drew "still no immediately auditable evidence just claims in the form of data": each prior item individually near-passed while the exhibit↔artifact binding stayed **editorial** (transcription + file link) instead of **mechanical** (quotation + pinned line anchor). Emit-time procedure: for every inline datum, quote the raw capture line it comes from (fenced, verbatim, full ids) and anchor it (`#L`); pin every evidence link to the SHA (press `y` on the GitHub file view). Emit-time trigger: `pr-evidence-gate.py` classes `truncated-identifier` (a co-located resolver does NOT excuse it — the resolver resolves the full id, not the fragment the reader holds; hash-equality prose exempt) and `mutable-ref`, plus the closed **surface hole**: the shipped comment was published via `gh api` PATCH, which the porcelain-only matcher (`gh pr|issue edit|create|comment`) never saw — the gate now scans `gh api` body writes (`-F body=@file`, `-f body=…`, `--input`); fifth occurrence of the converse-of-gate rule, one level down: audit the gate for *spellings of the write it cannot see*, not just tokens it excuses. Detection gaps: the line-anchor half of pinning and the paraphrase-vs-quotation judgment stay procedural — the gate sees ellipses and branch refs, not editorial fidelity. +17. **Evidence is captured in its environment — data alone is insufficient even when correct** — item 16 makes the data trustworthy as *transcription* (verbatim, greppable, pinned); this item polices what transcription can never carry: **liveness provenance**. A quoted `EVIDENCE trace_id=…` line, a re-hosted gist, a hand-assembled id table can all be correct and still show nothing about *where they came from* — extracted data is indistinguishable from data typed by hand, so it cannot make it immediately apparent that the evidence was captured **live** from a **functioning** system. The exhibit for a system-of-record-observable claim therefore includes an **in-environment capture**: a screenshot/recording of the resolving system's own UI (the Sentry Discover/trace view with the query, project/environment selectors, absolute time window, and result rows all in-frame) — the environmental chrome is not decoration, it *is* the provenance: it shows the query really ran, in the real dashboard, over the real window, and returned these rows. Correctness was never the failing dimension (2026-07-21: "just the data is insufficient even if correct — it needs to be immediately apparent that evidence was captured live and is functional"). Relation to prior items: item 15's link+visual conjunction fired only when a `sentry.io` link was present, and item 13's `NATIVE_MEDIUM` blessed an inline fenced excerpt as a terminal medium — so a no-link, quoted-data exhibit (the fidelity-remediated shape: full ids, verbatim excerpts, pinned line anchors, zero environment captures) passed the whole regime while carrying zero liveness provenance. The joint rule after this item: a telemetry-observable positive verdict always carries the in-environment visual (plus the live permalink per item 15); quoted excerpts, gists, and data files are appendix beside it, never the exhibit. Emit-time trigger: `pr-evidence-gate.py` class `data-only-exhibit` (non-negated verdict + telemetry-observation vocabulary + no image/recording embed + no sentry link — with a sentry link, `link-only-exhibit` already fires), with the re-hosted-gist ALLOW case flipped (fifth occurrence of "the ALLOW case was the next costume's spec") and the #43929 quoted-excerpt shape as a regression case. Detection gaps: vocabulary-scoped (telemetry-observation terms, not bare code tokens like `trace.test.ts`), so a claim phrased entirely without them evades mechanically; and the gate cannot see whether an embedded image actually shows the environment's chrome — screenshot content stays procedural (item 2's "eyeball it" applies: the capture must show the *resolving UI*, not a cropped data region indistinguishable from a spreadsheet). +18. **"Successful" is an evidence predicate, not a run status — and the default Sentry exhibit is fixed in advance** — a validation run may be scored/reported "successful"/"validated" only when its published surface already carries, for every Sentry-observable lane, the default exhibit pair: an **in-environment Sentry-UI screenshot** (item 17) **plus the co-located live permalink** (item 15). Completed runs, green falsifiers, staged drafts, and honest ⏳ lanes do not confer success — a run without the pair is at most "run-complete, evidence-owed." The default recipe needs no per-PR ascertainment: for Sentry, **generally capture actual screenshots from the Sentry UI and attach the link** — that pair is step zero's pre-computed answer for any Sentry-observable claim, never the terminus of axis-by-axis escalation. Capture-first ordering: the capture executes before any rule/gate/postmortem authoring may close a validation session — writing a new rule or gate class discharges nothing (2026-07-22: ten postmortems and 17 gate items shipped while zero Sentry-UI screenshots did; every "successful" run was claims-only, because success was assigned by run-completion and meta-work substituted for capture work). Emit-time trigger: `pr-evidence-gate.py` `VERDICT` vocabulary now includes the status spellings `successful`/`validated`/`live-proven`, so a claims-only unit scoring itself successful blocks like any bare "confirmed." Detection gap: the gate fires only on re-emit — already-shipped "successful" surfaces are audited by backward re-score, enumerated from live state, never from the ledger (the discharge-granularity rule applies to success statuses verbatim). + +## Lane-specific traps + +- **Visual:** spinner/skeleton mistaken for the loaded state; the toggle (privacy/redaction) not actually flipped; a cached screenshot from a prior run; the fallback surface shown without saying so. +- **Perf / benchmark:** stale frozen baseline (catalog C5 caveat); single sample; warm-vs-cold mismatch; measuring a different interaction than the claim. +- **Test:** snapshot regenerated to match the bug (`--updateSnapshot` masking a regression); the test mocks out the changed path; it passes on `main` too (so it's not a regression test). +- **Telemetry:** query window excludes the release; the error regrouped under a different fingerprint; sample-rate makes "0 events" meaningless. +- **Migration:** only the happy path asserted; `changedKeys` not checked against actual mutations; no real prior-version fixture. +- **Coverage:** a line covered ≠ a behavior asserted (executed but never checked). + +## When evidence fails the gate + +Don't publish it. Either re-capture correctly, **downgrade the verdict to ⚠️ inconclusive** and name what's missing, or — if the evidence shows the claim is false — switch to the [refutation path](SKILL.md). Never round a weak pass up to "proven." diff --git a/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md b/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md new file mode 100644 index 00000000..31316c66 --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md @@ -0,0 +1,26 @@ +# Lane → declarative assertion mapping (ADR-0058 bridge) + +Maps each evidence-catalog lane to a declarative assertion form, so a Claim Card can be expressed as an ADR-0058 recipe (pre-conditions → proof targets → assertions → screenshot points) where possible — and so we know which lanes are **CDP-expressible** vs **out-of-band**. State/log assertions give determinism; screenshots/video give reviewer confidence. See [[../ITERATION]] items 9–11 and MetaMask/decisions#173. + +| Lane | Assertion form | Expressible as a CDP recipe action? | +|---|---|---| +| A1 / B1 visual | screenshot at a proof point + (optional) DOM/a11y assertion | **yes** — Chrome CDP | +| B2 e2e | the spec's own assertions; trace.zip as artifact | yes — it *is* a driver | +| B3 falsifying test | test exit code: fail@`main`, pass@branch | out-of-band (test runner) | +| C1 startup traces | `stateHooks.getCustomTraces()[name] < threshold` | **yes** — `Runtime.evaluate` | +| C2 web-vitals | `stateHooks.getWebVitalsMetrics().inp < 200` | **yes** | +| C3 long-task / TBT | `stateHooks.getLongTaskMetricsWithTBT().tbt < 200` | **yes** | +| C4 render (WDYR) | console-log assertion: 0 unnecessary re-renders | partial — needs console capture | +| C5 benchmark | metric delta vs paired baseline > threshold | out-of-band (benchmark runner) | +| C6 DevTools/CDP | netlog: request absent/present; profile metric | **yes** | +| D1 / D2 bundle/chunk | static: chunk-manifest membership / size delta | out-of-band (build artifact) | +| D3 LavaMoat | static: `policy.json` diff empty / justified | out-of-band (git diff) | +| D4 manifest | static: permissions diff empty | out-of-band | +| E1 / E2 Sentry/Tempo | external query link (before/after window) | out-of-band (dashboard) | +| F1 migration | `changedKeys == expected` + state shape valid | out-of-band (migration test) | +| F3 simulation | `simulationData.{gasUsed,stateDiff}` matches | **yes** — `Runtime.evaluate` on state | +| F5 flag matrix | the same assertion repeated per `remoteFeatureFlags` state | **yes** | +| F7 i18n | static: `verify-locales` exit 0 | out-of-band | +| F8 runtime containment | `Object.isFrozen(Object.prototype)`; scuttled global throws + exception resolves; `typeof SNOW` | **yes** — `Runtime.evaluate`, but only against the SHIPPED build variant (dev is unscuttled, test's exception list is wider) | + +**Takeaway.** UI-state and runtime-metric lanes (A/B-visual, C1–C3/C6, F3/F5) map cleanly to CDP recipe assertions — ADR-0058's sweet spot. Static (D, F7), test-runner (B3, C5, F1), and dashboard (E) lanes are **out-of-band**: the recipe should *reference* them as proof targets without executing them. That out-of-band reference is precisely the **non-UI scaling gap** raised in review on decisions#173 — a recipe schema that admits out-of-band assertion references (not only CDP actions) closes it. This table is the proposed taxonomy to contribute back (ITERATION item 10). diff --git a/domains/pr-workflow/skills/pr-validate/references/worked-examples.md b/domains/pr-workflow/skills/pr-validate/references/worked-examples.md new file mode 100644 index 00000000..e98a5f1f --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/worked-examples.md @@ -0,0 +1,30 @@ +# Worked examples (end-to-end) + +Full runs: claim → lanes → capture → trust-gate → publish. The visual case is in SKILL.md; these cover the non-visual claim shapes. + +## Perf — "defer Rive wasm at startup" +- **Claim:** on cold start of the home view, the Rive wasm chunk isn't requested until the animation surface mounts. **Surface:** startup network + chunk graph. **Falsifier:** the chunk appears in the cold-start waterfall. **Baseline:** base requests it at startup. +- **Lanes:** A2 `perf_validation` (primary) → D2 chunk membership + C6 CDP netlog (corroborate). +- **Capture:** paired build of base vs head (`yarn webpack --test`); CDP netlog over cold start for each; source-map chunk membership of the Rive runtime. +- **Trust gate:** cold-vs-cold (not warm); the chunk truly absent (not deferred by a few ms); the netlog covers the whole startup window. +- **Publish:** before/after request list + a chunk-membership table in the PR body. No screenshot needed. + +## Migration — "add migration NNN" +- **Claim:** loading a profile from `` applies NNN; `changedKeys = {X, Y}`; all other state intact. **Falsifier:** an untouched controller mutated / malformed state. **Baseline:** a prior-version profile. +- **Lanes:** F1 migration test (primary) → F2 vault round-trip (if the vault is touched). +- **Capture:** run `NNN.test.js` (old-state-in → new-state-out); assert `changedKeys`; load a real prior-version profile and confirm boot. +- **Trust gate:** the test asserts more than the happy path; `changedKeys` matches the actual mutations; the fixture is a real prior profile, not synthetic. +- **Publish:** the `changedKeys` assertion + before/after state shape; link the test run. + +## Flag-gated — "Perps banner behind a remote feature flag" +- **Claim (×2):** flag on → banner shows; flag off → banner absent. **Surface:** home/Perps. **Falsifier:** banner state ≠ flag state. **Baseline:** each flag state is its own baseline. +- **Lanes:** F5 flag matrix → A1/B1 visual per state. +- **Capture:** mock the client-config response for each flag state; screenshot each. +- **Trust gate:** the flag is actually toggled (read `remoteFeatureFlags`); two distinct states are shown, not the same frame twice. +- **Publish:** a two-up before/after (flag off / flag on) in the PR body. + +## Refactor / no-op — "extract a hook, no behavior change" +- **Claim (negation):** behavior of `` is unchanged. **Falsifier:** any output/behavior diff. **Baseline:** base behavior. +- **Lanes:** B3 regression suite stays green + B4 snapshot diff empty + D1 bundle within noise. +- **Trust gate:** snapshots were *not* regenerated to hide a diff; the tests actually cover the surface; bundle delta is within noise, not "small but real". +- **Publish:** "no behavior change — regression suite green, snapshots unchanged, bundle ±0"; link CI. A passing screenshot is not evidence here. diff --git a/domains/pr-workflow/skills/pr-validate/skill.md b/domains/pr-workflow/skills/pr-validate/skill.md new file mode 100644 index 00000000..bb8e20f9 --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/skill.md @@ -0,0 +1,305 @@ +--- +name: pr-validate +description: Validate a MetaMask PR with objective evidence — primarily the Autonomous Engineering Platform (AEP) harness (visual_validation for visible UI behavior, perf_validation for non-visible perf behavior), backed by complementary evidence (Sentry query links, screenshots, screen recordings→GIF, DevTools/CDP output, bundle-size, web-vitals, test/CI results). Drives the AEP local stack end to end: preflight (postgres + temporal + worker + control-plane) → submit POST /v1/tasks → poll GET /v1/runs/:id → fetch artifacts → assemble an evidence bundle → publish to the PR body (re-hosting images to a public repo, scrubbing local paths). Match evidence to the PR's specific falsifiable claim, not a fixed checklist. Triggers on /pr-validate, /pr-validate visual, /pr-validate perf, /pr-validate preflight, /pr-validate status, /pr-validate evidence, /pr-validate plan, or when the user mentions validating/proving a PR, AEP / visual validation / perf validation, capturing evidence for a PR, before/after screenshots, a screen recording or GIF for a PR, attaching Sentry links or DevTools output as proof, or publishing an evidence bundle to a PR body. +maturity: experimental +--- + +# /pr-validate + +Prove a PR does what it claims with **objective, reviewer-grade evidence**. The primary engine is the **Autonomous Engineering Platform (AEP)** harness run locally — `visual_validation` for visible UI behavior, `perf_validation` for non-visible perf behavior — augmented by whatever complementary evidence the claim demands (Sentry query links, screenshots, screen recordings, DevTools/CDP output, bundle/web-vitals/test results). + +This skill **executes**: it brings up the local stack, runs the harness, captures artifacts, assembles the bundle, and — **only with confirmation** — publishes to the public PR body. The platform decides what passes; the model does not declare victory. See the AEP creed: *"Tests prove the code compiles. Screenshots prove the user actually sees the fix."* + +> **Hard rule — demonstrate, don't claim.** Every verdict must *demonstrate that the **ticket objective** is achieved* with an inspectable artifact (link / screenshot / screen recording / run / Sentry query / CDP capture) — never explain or claim in prose that it is achieved. Anchor to the linked issue's objective, not just the PR body's self-description. An unbacked "objective achieved" narrative is a vacuous pass → report **⚠️ inconclusive** and name what's missing; never upgrade prose to **✅ proven**. + +## Principles + +Twenty rules the rest of this skill implements. When a situation isn't covered below, decide by these. + +**What you may claim** +- **Falsifiability** — name the observation that would disprove the claim, then go looking for it. A review that cannot fail is not a review. +- **Falsifier coverage is not exhaustive — human intervention point.** There is no fixed checklist, so there is no completeness guarantee: the falsifiers found are bounded by claim-extraction quality and by what the reviewer thought to test. "No falsifier fired" is not "no falsifier exists." A human judges whether the falsifier chosen matches the claim's actual risk, and whether a mixed or high-stakes claim needed more than one — this skill closes the falsifiers it finds, it does not attest that it found all of them. +- **Diff-anchored** — the claim is what the code *can* do, not what the PR body promises. Drift between them is a finding, not a claim. +- **Surface-specific, and a surface need not be a screen** — a job graph, a build artifact, a policy file, or a telemetry shape are all legitimate surfaces with their own falsifiers. + +**What counts as evidence** +- **Demonstrate, don't claim** — a verdict shows the objective met with an inspectable artifact. Prose asserting it is a vacuous pass. +- **In situ** — present output on the tool's own surface (the run page, the Discover view, the trace waterfall, the console). Retyping output into the report launders evidence into claim: verbatim text proves nothing about provenance, and a transcription is a place to be selective without noticing you are being selective. +- **Reproducibility of assertions** — the bar is not that the reader *can* re-run it (a working link is the floor) but that they *needn't*: the exhibit is complete enough — the numbers, the window, the method, the control — that reading it makes the result near-certain. The re-issuable link/query is a backstop for the skeptic, offered second, never the headline. + +**Why believe it** +- **No vacuous passes** — green is not proof. Assert non-empty artifacts; ask whether the assertion *could* have failed and whether the test exercises the changed code. +- **Check the instrument, not just the result** — measurement design can manufacture a finding. Verify the treatment is actually delivered in each arm before interpreting any delta. +- **Removing a bias is not establishing validity** — correcting a flaw you found licenses only that correction. A trust gate names how the evidence could *still* be vacuous; if the sentence describes work you did rather than risk that remains, it is not a gate. +- **A null states its power** — when the spread exceeds the effect under test, report *not resolvable at this n* and name the smallest detectable effect. Never let it read as "no effect". +- **Premises are claims** — probe the *because* ("unavailable", "access-limited", "can't be done here") as hard as the verdict. A false premise silently justifies the wrong method, and "unavailable" is the highest-suspicion premise because it licenses weaker evidence. +- **Recompute stated counts** against the source before publishing. A number true of an earlier draft's scope is the commonest stale fact. + +**When to stop** +- **Stop at the falsifier** — match the bar to the claim's risk; evidence past the closed falsifier is noise. +- **Defer to CI** where CI already covers it, unless the coverage is itself the point. +- **State what was not covered** — steps that could not be automated are recorded as open, with the reason. A report listing only successes reads the same as one where nothing was checked. + +**How it is handled** +- **Refutation is a successful validation** — report it, localize it, hand back the repro. Don't fix, and don't publish a failure to someone else's PR unprompted. +- **Publish surface follows ownership** — the PR body when you authored it; a comment when validating someone else's. +- **Scrub before publishing, confirm before any public write** — local paths and usernames leak through failure summaries; one PR's approval does not carry to the next. +- **Isolate concurrent runs** — colliding ports, artifact dirs, or upload paths cross-contaminate evidence *silently*. That is an integrity failure, not flakiness. + +## The core move: match evidence to the claim + +A PR makes a **falsifiable claim** ("privacy mode now hides the Perps balance"; "hovering the asset row preloads the chart with no double-fetch"; "this cuts startup http.client time"). Validation = pick the evidence that would **falsify that claim if it were false**, then run it. Do not run a fixed checklist. + +**Step 1 — extract the claim.** Read the PR (`gh pr view`, `gh pr diff`) *and the linked issue*, then write a **Claim Card** — the linchpin; every lane is only as good as the claim. Full rubric, anti-patterns, and special cases (refactor/no-op, bug-fix, perf, migration, flag-gated): **[references/claim-extraction.md](references/claim-extraction.md).** + +``` +Claim: Given , when , then . +Surface: (reachable? seed / flag / fallback: …) +Type: → lanes <…> +Falsifier: +Baseline: +``` + +A claim must be falsifiable, surface-specific, **anchored to the diff** (if the body promises X but the diff can't deliver it, flag the drift — that's a finding, not a claim), bounded, and quantified where it's a perf claim. Decompose a mixed PR into one card per claim. + +**Step 2 — match each claim to lanes** from the [evidence catalog](references/evidence-catalog.md): + +| Claim shape | Primary lane | Complementary | +|---|---|---| +| Visible UI change (layout, copy, show/hide, theme) | **A1 `visual_validation`** / B1 mm-CLI — before/after screenshots | recording→GIF for motion; B5 a11y | +| A bug fix (any kind) | **⭐ B3 falsifying test** — fails on `main`, passes on the branch | A1/B1 if visible; E1 if it errored | +| Non-visible perf (preload, no-double-fetch, lazy-load, chunk) | **A2 `perf_validation`** — falsifiable assertions | C6 CDP netlog, D2 chunk membership | +| Render / over-render | **C4 WDYR + `devtools:react`** | C1 startup traces | +| Interaction responsiveness / startup timing | **C2 INP · C3 TBT · C5 benchmark (paired A/B)** | C1 phase traces, C6 profile | +| Telemetry / error-rate / latency in prod | **E1 Sentry links** (before/after) | E2 Tempo; span-volume → `/sentry-quota` | +| Bundle / build output | **D1 size · D2 chunk membership** | — | +| A dependency change is safe | **D3 LavaMoat policy + D4 manifest diff** | D1 size | +| Runtime containment still holds | **F8 SES lockdown / scuttling, on the shipped variant** | D3 policy | +| Persisted-state change | **⭐ F1 migration** (`changedKeys`, old→new state) | F2 vault round-trip | +| Tx / dapp / flag / snap / i18n behavior | **F3 sim · F4 provider · F5 flag matrix · F6 snaps · F7 i18n** | B2 e2e trace | +| Behavior with no UI | **B3 test + G4 repro** | G1 CI checks | + +Lane IDs (A1, B3, …) index [references/evidence-catalog.md](references/evidence-catalog.md) — the full menu with verified capture commands and the complete matching guide. When a PR mixes claims (a UI fix that also shifts a metric), run more than one lane and assemble them into one bundle. + +## When to use + +- **Prove a PR** before requesting review or merge — produce the before/after a reviewer expects. +- **Re-validate** after a force-push or a requested change. +- **Back a perf/telemetry claim** with numbers and links, not prose. +- **Assemble + publish** an evidence bundle from a run you already have (`evidence` subcommand). + +Not for code-correctness review (use `/review`, `/code-review`) or span-quota review (use `/sentry-quota`). This skill proves *behavior*, not code quality. + +## Subcommands + +| Invocation | Behavior | +|---|---| +| `/pr-validate ` | **Flagship.** Read the PR → state the claim → pick lanes → [preflight](#preflight) → run AEP lane(s) + gather complementary evidence → assemble bundle → **propose** the PR-body section and confirm before publishing. | +| `/pr-validate plan ` | Dry run: read the PR, state the claim, recommend lanes + targeting hints. No stack, no run. Cheap first step when unsure. | +| `/pr-validate visual ` | AEP `visual_validation` only. | +| `/pr-validate perf ` | AEP `perf_validation` only — check the graph is present first, see [caveat](#perf_validation-caveat). | +| `/pr-validate preflight` | Health-check the local stack; bring up what's down. No run. | +| `/pr-validate status ` | Poll `GET /v1/runs/:id`; print stage timeline + `evidenceBundle.artifactRefs`. | +| `/pr-validate evidence [--run ]` | Assemble + publish a bundle from an existing run and/or complementary sources (Sentry/screens/devtools). No new AEP run. | +| `/pr-validate lane ` | Run a single [catalog](references/evidence-catalog.md) lane by id (e.g. `lane F1`, `lane C3`, `lane D3`) — for the non-AEP lanes where you know the claim type. | +| `/pr-validate compare ` | Paired A/B for a perf or refactor claim: build base + head, capture the lane on both, diff. Avoids the stale-baseline trap (catalog C5). **Per-arm treatment check first:** verify the mechanism under test is actually active in each arm (chunk split present, span emitted, flag evaluated) before interpreting deltas — a null arm without delivered treatment is a no-op, not a control (2026-07-22, #42795 bisect lesson). | + +`` is a number or URL on `MetaMask/metamask-extension` unless another repo is given. Every variant runs Step 1 (extract the Claim Card) first — the claim decides the lane, even when you named one. + +## Preflight + +The AEP harness runs as a local stack: postgres, a temporal server, a worker, and a control plane. Bring-up steps, required Node version, registry auth, and environment are documented in the [AEP repository](https://github.com/MetaMask/metamask-autonomous-engineering-platform) itself — follow its README rather than a copy here, which drifts. Health-check first and bring up only what is down. + +Fast checks: + +```bash +curl -fsS localhost:3000/health >/dev/null && echo "control-plane up" || echo "control-plane DOWN" +curl -fsS localhost:8233 >/dev/null && echo "temporal UI up" || echo "temporal DOWN" +docker ps --format '{{.Names}}' | grep -E 'aep-postgres|aep-temporal' +``` + +If the control plane answers on `localhost:3000/health`, the stack is ready and you can skip to *Run mechanics*. + +## Teardown + +The stack is the heaviest thing this skill starts — postgres + temporal + a Node worker + control-plane — and the worker holds a live Claude session while the autonomous run itself spends tokens. It is **on-demand, not resident**: bring it up for the validation window, **tear it down when the run(s) finish**. Left up, it's the single largest reclaimable footprint on a shared host and quietly keeps a Claude seat warm. + +- **If your host wraps the stack in a service manager**, use its own down command — it stops the services and removes the `--rm` postgres/temporal containers, so state resets on the next bring-up (fine, each run is fresh anyway). +- **Otherwise:** stop the `yarn dev:*` processes and remove the postgres/temporal containers. +- **Tear down on every exit path** — pass, refutation, *or* abort. A failed or abandoned run leaves the stack up exactly as much as a passing one; the usual leak is walking away after a refutation without stopping it. + +## Run mechanics (submit → poll → fetch) + +The control-plane is a thin REST shell. Submit a PR-validation task, poll the run, pull artifacts from the evidence bundle. + +```bash +CP=localhost:3000 +PR="https://github.com/MetaMask/metamask-extension/pull/" + +# Submit (publishEvidence:false ALWAYS for local runs — the platform otherwise +# writes to the public PR body even on failure, leaking local paths/usernames) +RUN_ID=$(curl -fsS -X POST "$CP/v1/tasks" -H 'content-type: application/json' -d '{ + "repo": "MetaMask/metamask-extension", + "title": "Visual validation — PR #", + "taskClass": "visual_validation", + "externalRef": "'"$PR"'", + "payload": { "prUrl": "'"$PR"'", "description": "", "publishEvidence": false } +}' | node -e 'process.stdin.on("data",d=>console.log(JSON.parse(d).runId||JSON.parse(d).id))') + +# Poll +curl -fsS "$CP/v1/runs/$RUN_ID" | node -e 'const r=JSON.parse(require("fs").readFileSync(0));console.log(r.status); (r.evidenceBundle?.artifactRefs||[]).forEach(a=>console.log(a.name,a.mediaType))' + +# Fetch an artifact +curl -fsS "$CP/v1/runs/$RUN_ID/artifacts/" -o /tmp/ +``` + +- `taskClass`: `visual_validation` or `perf_validation`. The worker auto-enriches the payload from `prUrl` (pulls headSha, base, diff, files, linked issues via the GitHub app) — you only supply `prUrl` + a `description` targeting hint. +- The **targeting hint** (`payload.description`) is how you steer the agent to the surface under test. Be specific: which screen, which control, what to toggle. For hard-to-reach surfaces, name the reachable fallback (e.g. the Shield entry modal stands in for the Perps tutorial modal, which is gated in the default fixture). +- Artifact regex allows **png/jpg/log/txt only** — no video. Screen recordings need the side-channel recipe (catalog + publishing reference). + +### Concurrent runs (multiple agents / parallel lanes) + +Five shared resources need per-run isolation on one machine — collisions cross-contaminate evidence *silently* (wrong session's logs attributed to a run), which is an integrity failure, not flakiness: **(1)** CDP debug ports — derive per run, never hardcode; **(2)** e2e harness service ports (anvil/proxy/fixture/mocha) — one e2e run at a time per worktree, one worktree per agent, and never rebuild `dist/` in a worktree with an active run; **(3)** artifact dirs — per-run namespaces; `test-artifacts/` is per-worktree shared state, harvest failure artifacts before the next run overwrites the same test-title dir; **(4)** evidence-repo uploads — run-scoped paths (`pr-//`), retry-with-fresh-sha on 409, never overwrite another run's published files; **(5)** commit-pinning — pin only after your own final upload lands, verifying your files exist at that sha. Safe to share: registry auth, a read-only `dist/`, the AEP stack itself. + +### Trust the evidence (anti-reward-hacking) + +A green result is not proof. The vacuous-pass trap is the floor: if `promptCrafter` errors, the chain "passes" via skip with **zero artifacts** — a pass is only real if `evidenceBundle.artifactRefs` is non-empty with the expected media. Beyond that, every lane must clear a trustworthiness gate before you believe or publish it: **does the artifact show the *claimed* surface** (not a spinner/wrong screen), **does the test exercise the *changed* code** (fails on `main`), **does the signal exceed noise**, **could the assertion have failed**? The Claim Card's Falsifier is the anchor. Full gate + per-lane traps: **[references/evidence-trustworthiness.md](references/evidence-trustworthiness.md).** + +### perf_validation caveat + +The `perf-validation/` graph writes falsifiable network/static/smoke assertions and gives the tester deterministic `.aep/` helpers (CDP netlog, phase segmentation, source-map chunk membership). Two constraints worth knowing before a perf run: + +- It requires a `yarn webpack --test` build first — the browserify `build:test` has no code splitting, so `import()` never hits the network there. +- Temporal caps activity results at ~2MB, so artifact refs must be content-free; only `evidenceBundle` carries base64. + +**Check the graph is present in your AEP checkout before relying on it.** It is newer than the visual-validation graph and may not be in every version — if it isn't registered, perf runs silently won't dispatch, and the fallback is manual DevTools/CDP capture (see the catalog). + +## Complementary evidence + +AEP is primary but rarely sufficient alone. Pull whatever the claim needs — **and proactively suggest evidence the PR author likely didn't think of**. The catalog is grouped into 7 families; full menu with verified capture commands and "what it proves": **[references/evidence-catalog.md](references/evidence-catalog.md).** Families: + +- **A. AEP** — `visual_validation` / `perf_validation` / bundle byproducts (primary autonomous engine). +- **B. Behavior & flow** — mm-CLI visual, E2E trace+video, **⭐ falsifying regression test** (fails on main, passes on branch — the strongest bug proof), Storybook/component, a11y, flaky-stability rerun. +- **C. Performance & render** — startup/custom traces, web-vitals (**INP/FCP/LCP/CLS** via `stateHooks`), long-task **TBT** (separate observer), React render/selector (WDYR), benchmark A/B (paired), DevTools/CDP profiling, memory-over-flow, **same-window app+DevTools capture** (C8 — UI + console evidence in one frame, OS-level region recording). +- **D. Build output** — bundle-size, chunk membership, **LavaMoat policy diff**, manifest permissions diff, build-variant matrix. + (Runtime containment — SES lockdown, scuttling, Snow — is **F8**, not D: D is what the build *permits*, F8 is what the running artifact *enforces*.) +- **E. Production telemetry** — Sentry links (span-volume → `/sentry-quota`), Tempo traces, error-event shape. +- **F. Extension integrity** — **⭐ state migration**, vault/keyring, tx simulation, provider/dapp, feature-flag matrix, snaps, i18n. +- **G. CI/review/process** — check links, coverage delta, reviewer bot, manual repro. + +Screen recordings (motion a still can't prove): `mm` + a Playwright `recordVideo` preload → `ffmpeg` two-pass palette GIF (webm/mp4 don't render inline). See [references/evidence-publishing.md](references/evidence-publishing.md). + +## Sufficiency — how much is enough + +Match the bar to the claim; stop when the claim's falsifier is closed. Don't over-instrument a copy fix; don't under-prove a high-stakes claim. + +- **One lead lane that closes the falsifier** is enough for low-risk, single-claim PRs (a copy fix → one screenshot; a bug fix → the falsifying test). +- **Weigh AEP's cost before reaching for it.** A `visual_validation`/`perf_validation` run spins the full stack *and* burns autonomous-agent tokens — by far the most expensive lane. Use it when the claim genuinely needs autonomous capture of a reachable surface; when a lighter lane closes the same falsifier (a single `mm` screenshot, a falsifying test, a CDP capture, an artifact CI already produced), prefer it and skip the stack. Whenever you do start it, tear it down after (see [Teardown](#teardown)). +- **Lead + one corroborator** for perf/telemetry (a number *and* its source) and for anything user-facing that also moves a metric. **For a perf-targeting PR the lead lane is the measured impact itself** — a paired A/B benchmark at the current head (C5) or equivalent — never mechanism evidence alone (chunk membership, netlog exclusion prove the improvement is *possible*, not that it *happened*). A perf PR also always carries correctness + non-regression lanes: changed-surface tests green at head, affected flows exercised, neutral profile within noise. (2026-07-22, #42795 lesson.) +- **Lead + integrity lane** for high-stakes surfaces regardless of size: persisted-state (migration + vault), money (tx simulation), permissions (LavaMoat + manifest), runtime containment (SES lockdown / scuttling), security/keyring. Size-S doesn't lower the bar here. +- **Per-claim** for mixed PRs — each Claim Card needs its own closed falsifier; a strong UI proof doesn't cover the metric it also shifts. +- **Rely on CI for routine coverage — don't re-collect what CI already establishes.** Lint, build, typecheck, the full test suite, changelog validation: CI is the authoritative source; **cite the check result** (e.g. "423 pass / 0 fail at head") instead of re-running it locally. Spend independent evidence only on (a) the claim's load-bearing falsifier, (b) specifically important/noteworthy areas (security, money, permissions, the exact changed surface), or (c) where the trust-gate warns a green result could be vacuous/misattributed. This is the economy counterpart to *"don't trust green blindly"*: that gate polices the **claim-critical** lane; this rule spares the **routine** coverage — re-collecting what CI covers is bundle noise. (#9628: cited CI's pass matrix for build/test, ran independent evidence only for the load-bearing homogeneity + resolution lanes.) + +Stop when each claim has one trustworthy artifact that would have shown its falsifier. More evidence past that is noise. + +## Publishing the evidence bundle + +**Public, outward-facing action — always confirm the rendered section with the user before writing the PR body.** Match AEP's own format so the section is idempotent and reviewer-familiar. Full recipe (markers, image re-hosting, recordings, the `### After` injection, privacy scrub): **[references/evidence-publishing.md](references/evidence-publishing.md).** Essentials: + +- **Canonical header — every validation output leads with the exact literal `## 🧪 Validation Run`.** Same string in a PR comment and in the PR-body section, never reworded or demoted — the constancy is what makes it scannable/Ctrl-F-able, like Copilot's fixed `## Pull request overview`. Line 2 is the meta line: `**Verdict:** ✅ proven — **Claim:** ` then `head \`\` · · lanes: `. Enforced mechanically by `hooks/pr-evidence-gate.py` (a validation/verification/evidence heading or AEP marker without the literal blocks the `gh` write). +- **Post complete, once — and know which regime the surface is in.** Comments are **push** (audience notified once at post time; edits are silent): hold until every planned lane is present or consciously dropped, and put substantive additions or changed verdicts in a **new comment referencing the original**, never a silent edit. The PR **body** is **pull** (consulted at review time): the idempotent marker upsert on re-validation at a new head is correct there. Typo-level comment edits are fine. +- **Falsifier-forward.** After the meta line, foreground **what would have falsified the claim and how each falsifier is closed** — the falsifier is the load-bearing content, not a footnote. Structure the body as "what would make this false → the evidence that rules it out," not a lane inventory with a `falsifiers closed` line buried at the bottom. The reviewer should see the disproof attempt first. +- **Don't restate CI results.** Lint/build/typecheck/test/changelog outcomes are already on the PR's Checks tab — the reviewer sees them. Cite a CI result in the comment only to **highlight something specific** they'd otherwise miss; otherwise reference "green in Checks" or omit it. Restating "423 pass / 0 fail" is bundle noise (the display-side counterpart to the catalog's *rely on CI* collection rule). +- **Re-host images first.** Control-plane artifact URLs are `localhost` and won't render on GitHub. Re-host each artifact somewhere **your readers can reach unauthenticated**, then link the hosted URL — see [evidence-publishing.md](references/evidence-publishing.md) for the host choice and the mandatory unauthenticated `curl` check. A personal repo or a private bucket fails this for every reader but you. +- **Use idempotency markers** so a re-run replaces in place: wrap the whole section in `` … ``; inside it, AEP's own `` for the status block and `` for images, injected into the PR template's `### **After**` section (replacing the `` placeholder) when present. +- **Verdict-first, lanes nested:** under the canonical header, hand-assembled AEP blocks demote to `### AEP Visual Validation` (leave AEP's own service-published `##` blocks untouched) with `**✅ Passed**` / `**❌ Failed**` / `ℹ️`, the long narrative in `
Validation details`, a meta line `Run \`\` · [LangSmith trace](…)`. +- **Scrub** local paths and your username from any narrative before publishing — failure summaries leak them. + +## Validation output format + +When reporting back (before publishing), lead with the verdict and the claim it tests: + +``` +PR # — +Claim: <the falsifiable behavior under test> +Verdict: ✅ proven / ❌ refuted / ⚠️ inconclusive (vacuous pass — 0 artifacts) +Evidence: + - visual_validation run <id> — N screenshots (before/after <surface>) + - perf_validation run <id> — M/M assertions proven + - Sentry: <before/after link> +Artifacts: <local paths or re-hosted URLs> +Next: publish to PR body? (y/N) +``` + +If a lane comes back inconclusive, say so and name what's missing — never upgrade a vacuous pass to "proven". + +### When validation refutes the claim (❌) + +A refutation is a *successful* validation — the skill did its job. Report it constructively, do **not** publish a public "Failed" section to the author's PR unprompted: + +- **Lead with the falsifier you hit:** "Claim refuted — under privacy mode the Perps balance is still visible (screenshot)." Show the evidence that disproves it. +- **Localize:** which lane, which surface, the exact observation vs the expected. Tie it to the diff if you can see why. +- **Separate refuted from inconclusive:** refuted = evidence shows the claim is false; inconclusive = evidence couldn't be captured / was untrustworthy (trust-gate fail). Don't conflate. +- **Hand back, don't fix:** this skill proves behavior; fixing is the author's loop (or a `bug_fix`/`pr_feedback` run). Offer the repro, not a patch. +- Surface privately first; only post to the PR if the author asks or it's your own PR. + +## Safety & privacy + +- **`publishEvidence: false` on every local submit.** Publish manually, only after a real pass, only with confirmation. +- **Re-host before linking** — never put a `localhost` URL or a local file path in a public PR body. +- **Scrub** usernames/paths from narratives. Failure summaries are the usual leak. +- **Don't trust green blindly** — assert non-empty `artifactRefs` (vacuous-pass trap). +- **Confirm before any PR-body write.** One PR's approval doesn't carry to the next. + +## Worked example + +PR claims privacy mode now hides the Perps balance (the demo bug #42683): +1. `gh pr view` → claim = "with privacy mode on, the Perps tab balance is masked like everywhere else." +2. Lane = `visual_validation` (visible). Preflight stack. +3. Submit with `description: "Onboard, enable privacy mode in Settings, open the Perps tab, confirm the balance is masked. If the Perps tutorial modal blocks, use the Shield entry modal as the reachable surface."` + `publishEvidence:false`. +4. Poll to completion; assert `artifactRefs` has the before/after pair (not a vacuous skip). +5. Fetch the two PNGs; re-host them to your configured evidence host; assemble the `AEP_VISUAL_VALIDATION` section with the hosted URLs injected into the template's `### After`. +6. Show the rendered section; on confirm, upsert the PR body. + +End-to-end examples for **non-visual** claims (perf, migration, flag-gated, refactor/no-op): **[references/worked-examples.md](references/worked-examples.md).** + +## Positioning: AEP vs recipes vs pr-validate + +Three adjacent things; keep the boundary clear so they compose instead of collide: + +- **AEP** — governed *fleet orchestration*: sandboxes, Temporal, autonomous runs at scale. The heavy engine. +- **ADR-0058 recipes** ([decisions#173](https://github.com/MetaMask/decisions/pull/173)) — a *dev-machine inner-loop* proof artifact: a declarative per-PR recipe run against the live app over CDP, emitting `summary.json`/`trace.json`/manifest. +- **pr-validate** (this skill) — the *claim→evidence methodology + taxonomy* both draw on. The Claim Card is the bridge from a PR's claim to the right proof target; the [evidence catalog](references/evidence-catalog.md) is the lane vocabulary; [lane-assertions.md](references/lane-assertions.md) maps each lane to a recipe assertion (and flags the out-of-band, non-UI lanes — the gap raised in review on decisions#173). + +pr-validate is the one a human drives; it can dispatch an AEP run or author a recipe as its capture step. + +## Workflow integration + +Where pr-validate sits in the PR lifecycle (see the public `pr-workflow` siblings): + +- **After `create-pr`, before `pr-review-queue`:** validate the claim, attach the bundle, *then* request review — reviewers get the before/after up front. +- **On force-push / requested-change:** re-run the affected lane(s); re-validation keeps a stale evidence section honest. +- **`/triage` push items:** a `push`-state PR isn't done until its claim is proven; pr-validate produces the evidence that lets it move. +- **Not a CI gate** (same scope line as ADR-0058) — it's the author's inner loop, complementing unit/e2e, not replacing them. + +## Boundaries + +- **Executes, with a confirmation gate on publish.** It runs the harness and captures evidence autonomously; it does not write to the public PR body without showing you the section first. +- **Local-only AEP.** No hosted instance. The skill drives the local stack. +- **Proves behavior, not code.** Pair with `/review` / `/code-review` for correctness and `/sentry-quota` for span-volume risk. +- **No persisted state.** Each run is fresh. To keep a validation record, ask — nothing is written by default. + +## Related + +- [references/claim-extraction.md](references/claim-extraction.md) — Step 1: turn a PR into a falsifiable Claim Card. +- [references/evidence-catalog.md](references/evidence-catalog.md) — the full menu of evidence kinds, verified capture commands, and what each proves. +- [references/evidence-trustworthiness.md](references/evidence-trustworthiness.md) — the anti-reward-hacking gate before believing/publishing a lane. +- [references/evidence-publishing.md](references/evidence-publishing.md) — PR-body format, non-visual/multi-lane rendering, image re-hosting, recordings→GIF, privacy scrub, ADR-0058 artifact contract. +- [references/worked-examples.md](references/worked-examples.md) — end-to-end runs for perf / migration / flag-gated / refactor claims. +- [references/lane-assertions.md](references/lane-assertions.md) — lane → declarative recipe-assertion mapping (ADR-0058 bridge). +- [MetaMask/metamask-autonomous-engineering-platform](https://github.com/MetaMask/metamask-autonomous-engineering-platform) — the AEP repo: stack bring-up in its README, plus `docs/demo-runbook.md`, `packages/agent-chain/src/graphs/{visual,perf}-validation/`, and `packages/github/src/pr-body-builder.ts` (the canonical PR-body format this skill mirrors). +- `MetaMask/decisions#173` — ADR-0058 Recipe-Based Verification (the adjacent inner-loop proof system). +- `/sentry-quota` — sibling skill for span-volume PR review; `/review`, `/code-review` — code correctness. +- `/memory-leak-hunt` — the engine behind the **memory leak** evidence category (C9). pr-validate delegates retention analysis to it and packages the verdict; it also runs standalone. +- [[reference_aep_local_run]] — the source memory this skill encodes. +- [[reference_sentry_project_topology]] — Sentry project mapping for the telemetry-evidence lane. From c3c3b21d1a1b282744f58dd717741932d8e60e20 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 08:08:46 -0400 Subject: [PATCH 019/135] =?UTF-8?q?Drop=20the=20CHANGELOG=20entry=20?= =?UTF-8?q?=E2=80=94=20it=20is=20for=20the=20CLI=20package,=20not=20skills?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CHANGELOG.md tracks consumer-facing changes to the `@metamask/skills` package, per CONTRIBUTING's "CLI / tooling changes" section. No merged skill-only PR adds an entry (#80, #78, #70, #62, #61 all touch zero changelog lines). Carrying one here bought nothing and was the sole source of this branch's conflict with `main`, since every skill PR edits the same `[Unreleased]` block. Restoring the file to its merge-base state makes the branch conflict-free without a merge commit. --- CHANGELOG.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 64e8da78..383d49c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,10 +7,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -### Added - -- Add `analytics` domain: Sentry span-quota guard, Sentry MCP query workflows (including longer-range/30D+ query fidelity and percentile-sample-size filtering), release-over-release performance attribution, instrumentation methodology (including memoized-selector fan-out and incidentally-added instrumentation), and supporting knowledge - ## [0.1.0] ### Added From 712d17a406665e6357221f3cb0796ab75ef1ca88 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 08:09:53 -0400 Subject: [PATCH 020/135] =?UTF-8?q?Drop=20the=20CHANGELOG=20entry=20?= =?UTF-8?q?=E2=80=94=20it=20is=20for=20the=20CLI=20package,=20not=20skills?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CHANGELOG.md tracks consumer-facing changes to the `@metamask/skills` package, per CONTRIBUTING's "CLI / tooling changes" section. No merged skill-only PR adds an entry (#80, #78, #70, #62, #61 all touch zero changelog lines). It was also the sole source of this branch's conflict with `main`, since every skill PR edits the same `[Unreleased]` block. --- CHANGELOG.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f0f46d6..8557ff8f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,10 +7,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -### Added - -- Add `typescript` domain (experimental) — TypeScript authoring and JS→TS migration guidance. - ## [0.2.0] ### Added From 349efe54e360b3ee8dcfebfdaf026b9162f68453 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 08:10:29 -0400 Subject: [PATCH 021/135] =?UTF-8?q?Drop=20the=20CHANGELOG=20entry=20?= =?UTF-8?q?=E2=80=94=20it=20is=20for=20the=20CLI=20package,=20not=20skills?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CHANGELOG.md tracks consumer-facing changes to the `@metamask/skills` package, per CONTRIBUTING's "CLI / tooling changes" section. No merged skill-only PR adds an entry (#80, #78, #70, #62, #61 all touch zero changelog lines). It was also the sole source of this branch's conflict with `main`, since every skill PR edits the same `[Unreleased]` block. --- CHANGELOG.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a4a265e2..383d49c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,10 +7,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -### Added - -- Add `platform` domain (extension errors/lifecycle debugging + architecture knowledge), `testing/benchmark-design`, `performance/browser-extension-profiling`, and `coding/resilient-api-collection` - ## [0.1.0] ### Added From 6e6bbb7c83755d20233abac6ef6d5076af265376 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 08:22:24 -0400 Subject: [PATCH 022/135] Make knowledge/ the single source for the React perf pattern taxonomy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The selector and effect anti-pattern definitions existed in two places: these knowledge files, and the `performance` skill's own mm-* references already on main. Same patterns, same worked examples, two homes that would drift. knowledge/selector-anti-patterns.md and knowledge/effect-anti-patterns.md are now the canonical, platform-agnostic taxonomy — the union of both sides. The selector file absorbs mutation-in-result and over-broad-input from mm-selector-memoization; the effect file absorbs the dependency-side patterns from mm-hook-dependency-arrays and the lifecycle-side patterns (derived state, effect chains, uncancelled async). mm-selector-memoization.md keeps everything only it can say — the codebase's own selector creators, the verified instance table with file:line, the fix recipes, the scoped greps, the don't-over-correct caveats — and maps each generic pattern onto this codebase instead of redefining it. mm-hook-dependency-arrays.md keeps its richer JSON.stringify treatment and gains a scope note. Citations are by NAME, not by relative link. `install` copies domain knowledge/ and a skill's references/ as siblings under the installed skill directory, so `../../../knowledge/x.md` resolves in the repo and breaks once installed, and `../knowledge/x.md` does the reverse. Section anchors are dropped for the same reason — they broke the moment the taxonomy was renumbered. Also drops the CHANGELOG entry: that file tracks the @metamask/skills CLI package, no merged skill-only PR adds one, and it was this branch's sole conflict with main. --- CHANGELOG.md | 4 - .../knowledge/effect-anti-patterns.md | 146 ++++++++++++------ .../knowledge/selector-anti-patterns.md | 120 +++++++++----- .../effect-anti-pattern-review/skill.md | 23 +-- .../references/mm-hook-dependency-arrays.md | 2 + .../references/mm-selector-memoization.md | 48 ++---- .../selector-anti-pattern-review/skill.md | 27 ++-- 7 files changed, 226 insertions(+), 144 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 614ec51c..383d49c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,10 +7,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -### Added - -- Add `performance` skills: `data-analysis`, web-vitals + metrics-pipeline knowledge, and `effect-anti-pattern-review` / `selector-anti-pattern-review` (review-time complements to the `perf-*` optimization skills) with `render-cascade` / anti-pattern knowledge - ## [0.1.0] ### Added diff --git a/domains/performance/knowledge/effect-anti-patterns.md b/domains/performance/knowledge/effect-anti-patterns.md index 00a5d9b1..3b74afe0 100644 --- a/domains/performance/knowledge/effect-anti-patterns.md +++ b/domains/performance/knowledge/effect-anti-patterns.md @@ -1,69 +1,95 @@ --- name: effect-anti-patterns domain: performance -description: Four React `useEffect` patterns that cause unnecessary renders, memory leaks, or race conditions +description: The React `useEffect` patterns that cause unnecessary renders, memory leaks, or race conditions — the canonical, platform-agnostic taxonomy that per-repo effect references instantiate --- # Effect Anti-Patterns -Four `useEffect` patterns that are systemically broken in React codebases. Each pattern has a broken example, a fixed example, and a detection recipe. +**This file is the single source for the pattern taxonomy.** Per-repo references — such as +the `mm-hook-dependency-arrays` and `mm-useeffect-antipatterns` references shipped with the +`performance` skill — name these patterns rather than redefining them, and add what only +they can: verified instances with `file:line`, repo-specific lint gaps, and fix recipes. -## 1. `JSON.stringify` in Dependency Array +Two halves, and they fail differently. Patterns 1–2 are about **when an effect re-runs** +(the dependency side). Patterns 3–5 are about **what happens inside and after it** (the +lifecycle side). -`JSON.stringify` produces a new string on every render when the input is an object. React compares dependency arrays by reference for primitives and by identity for objects. A stringified object is a new primitive every render, so the effect fires every render. +## 1. Unstable dependency identity + +A dependency array is supposed to be a cheap identity check. Anything that produces a new +value every render defeats it — and usually signals an unstable reference upstream. ```typescript -// ❌ BROKEN: effect runs on every render -useEffect(() => { - doSomething(config) -}, [JSON.stringify(config)]) +// ❌ serializes on EVERY render just to build the dep key +useEffect(() => { doSomething(config) }, [JSON.stringify(config)]) -// ✅ FIXED: destructure and depend on primitives -const { a, b } = config -useEffect(() => { - doSomething({ a, b }) -}, [a, b]) +// ❌ new object every render → effect runs every render (or loops forever) +useEffect(() => { ... }, [{ id: user.id }]) -// ✅ ALSO FIXED: stabilize via useMemo -const stableConfig = useMemo(() => config, [config.a, config.b]) -useEffect(() => { - doSomething(stableConfig) -}, [stableConfig]) +// ✅ stabilize the reference upstream, then depend on it directly +const stableConfig = useMemo(() => derive(a, b), [a, b]) +useEffect(() => { doSomething(stableConfig) }, [stableConfig]) + +// ✅ or depend on the primitives +useEffect(() => { ... }, [user.id]) ``` -Detection: `grep -rnE 'useEffect.*\[.*JSON\.stringify' <source-dir>` +Stabilizing the source beats hashing it. If you genuinely cannot, a primitive key computed +**once** (`useMemo(() => xs.join(','), [xs])`) still beats a per-render `JSON.stringify`. -## 2. `useEffect` + `setState` (State Mirror Pattern) +Detection: grep for `JSON.stringify` inside a dependency array, and for inline `{`/`[` +literals in the dep position. -Using an effect to mirror one piece of state into another is almost always wrong. The computed value should be derived inline or via `useMemo`. Mirror-effects trigger an extra render and create synchronization bugs. +## 2. Wrong dependencies ```typescript -// ❌ BROKEN: two renders, possible stale state -const [fullName, setFullName] = useState('') -useEffect(() => { - setFullName(`${first} ${last}`) -}, [first, last]) +// ❌ empty deps but reads state → stale closure, value frozen at first render +const onPress = useCallback(() => doThing(count), []) -// ✅ FIXED: derived inline, one render -const fullName = `${first} ${last}` +// ❌ empty deps and reads nothing → this was never a hook, hoist it out +const config = useMemo(() => ({ a: 1, b: 2 }), []) +``` -// ✅ ALSO FIXED: memoized if expensive -const fullName = useMemo(() => expensiveJoin(first, last), [first, last]) +**Fix:** include what you read; or if there is genuinely nothing to read, move the constant +outside the component. Where `react-hooks/exhaustive-deps` is not enabled, this is not +caught automatically and must be reviewed by hand. + +## 3. Derived state via effect + setState + +If a value is computable from props/state/store, compute it during render. State plus an +effect is for *synchronizing with something external*, not for derivation. + +```typescript +// ❌ two render passes per change: render → effect → setState → render again +const [visible, setVisible] = useState([]) +useEffect(() => { setVisible(items.filter((t) => !t.hidden)) }, [items]) + +// ✅ derive during render — one pass, no state to drift out of sync +const visible = useMemo(() => items.filter((t) => !t.hidden), [items]) ``` -The React docs explicitly call this out: [You Might Not Need an Effect](https://react.dev/learn/you-might-not-need-an-effect). +The React docs call this out directly: +[You Might Not Need an Effect](https://react.dev/learn/you-might-not-need-an-effect). + +### 3a. Cascading effect chains -Detection: `grep -rnB1 -A3 'useEffect' <source-dir> | grep -B2 -A1 'set[A-Z]'` (review hits manually) +The same mistake compounded: effect A sets state, which triggers effect B, which sets +state, which triggers effect C. Each link is a full extra render pass *and* a window where +the UI shows an inconsistent intermediate combination. -## 3. Missing Interval/Timer Cleanup +**Fix:** collapse the chain into render-time derivation — one `useMemo` per step, or one +for the lot. -Every `setInterval` and `setTimeout` inside an effect must be cleared in the cleanup function. Otherwise the timer survives component unmount and fires on dead state, leaking memory and causing "setState on unmounted component" warnings. +## 4. Missing timer cleanup + +Every `setInterval` and recurring `setTimeout` started in an effect must be cleared in its +cleanup. Otherwise the timer outlives unmount, fires against dead state, and leaks in +proportion to how often the component mounts. ```typescript // ❌ BROKEN: timer leaks after unmount -useEffect(() => { - setInterval(poll, 1000) -}, []) +useEffect(() => { setInterval(poll, 1000) }, []) // ✅ FIXED useEffect(() => { @@ -72,19 +98,23 @@ useEffect(() => { }, []) ``` -Detection: `grep -rnB2 -A10 'setInterval\|setTimeout' <source-dir> | grep -B5 'useEffect' | grep -v 'clearInterval\|clearTimeout'` +## 5. Uncancelled async work -## 4. Missing `AbortController` in Async Effects - -Async work inside an effect should be cancellable. Without `AbortController`, a request initiated before unmount can resolve after unmount, triggering `setState` on a dead component and masking memory issues. +Async work started in an effect can resolve *after* unmount — or after the input changed, +letting a stale response overwrite a newer one. ```typescript -// ❌ BROKEN: fetch races unmount +// ❌ fetch races unmount; stale data can win +useEffect(() => { fetchMeta(address).then(setMeta) }, [address]) + +// ✅ cancelled flag — cheapest, works for any promise useEffect(() => { - fetch(url).then((r) => setData(r)) -}, [url]) + let cancelled = false + fetchMeta(address).then((m) => { if (!cancelled) setMeta(m) }) + return () => { cancelled = true } +}, [address]) -// ✅ FIXED +// ✅ AbortController — also cancels the request itself useEffect(() => { const ctrl = new AbortController() fetch(url, { signal: ctrl.signal }) @@ -94,8 +124,26 @@ useEffect(() => { }, [url]) ``` -## Why These Matter +Pick one and apply it consistently. + +## Why these matter + +- **Renders.** Derived-state effects double every render in the affected subtree, and + chains multiply it. +- **Memory.** Uncleared timers and subscriptions leak proportional to mount count. +- **Correctness.** Uncancelled async work produces "state update on an unmounted + component" warnings and, worse, races where an older response overwrites a newer one. + +## Don't over-correct + +- Don't add `useMemo`/`useCallback` everywhere — only where profiling shows wasted work, or + where a memoized child depends on the reference. Compilers handle many cases on opted-in + paths. +- A `JSON.stringify` on a cold path with a small object is acceptable. Prioritize hot render + paths. + +## Related -- **Renders.** State-mirror effects double every render in the affected component tree. -- **Memory.** Uncleared intervals and timers leak proportional to how often the component mounts. -- **Correctness.** Async effects without cancellation cause `"Can't perform a React state update on an unmounted component"` warnings and, worse, data races where an older response overwrites a newer one. +- `render-cascade` — how effect-driven re-renders propagate through the component graph. +- `selector-anti-patterns` — the store-side counterpart; an unstable selector result is a + common source of the unstable dependency in pattern 1. diff --git a/domains/performance/knowledge/selector-anti-patterns.md b/domains/performance/knowledge/selector-anti-patterns.md index 60bf1473..8bf32e9f 100644 --- a/domains/performance/knowledge/selector-anti-patterns.md +++ b/domains/performance/knowledge/selector-anti-patterns.md @@ -1,78 +1,114 @@ --- name: selector-anti-patterns domain: performance -description: Five Redux selector patterns that that break selector memoization and cause render cascades +description: The Redux selector patterns that break memoization and cause render cascades — the canonical, platform-agnostic taxonomy that per-repo selector references instantiate --- # Selector Anti-Patterns -Each pattern causes `useSelector` to return a new reference on every call, triggering unnecessary re-renders. +**This file is the single source for the pattern taxonomy.** Per-repo references — such as +the `mm-selector-memoization` reference shipped with the `performance` skill — name these +patterns rather than redefining them, and add what only they can: the codebase's own +selector-creator utilities, verified instances with `file:line`, and fix recipes. -## The Five Patterns +Every pattern below has the same failure shape: `useSelector` returns a **new reference** +when the underlying data did not change, so every consumer re-renders. One broken selector +near the root of the graph cascades through everything downstream, and the cost scales +superlinearly with user data. -### 1. Plain Function Selector +## 1. Unmemoized selector -No memoization. Returns new reference every call. +A plain function that allocates. No memoization at all — a new reference on every call. ```typescript // ❌ BROKEN export function getPendingApprovals(state) { - return Object.values(state.metamask.pendingApprovals ?? {}); + return Object.values(state.pendingApprovals ?? {}); } // ✅ FIXED -const getPendingApprovalsObject = (state) => state.metamask.pendingApprovals ?? {}; +const getPendingApprovalsObject = (state) => state.pendingApprovals ?? {}; export const getPendingApprovals = createSelector( getPendingApprovalsObject, (approvals) => Object.values(approvals), ); ``` -Detection: `grep -r "export function get" ui/selectors/` +Detection: grep exported `function get…` in the selectors directory. -### 2. Identity Function Selector +## 2. Identity / passthrough result -Transform in input, identity in result → memoization is broken. +The transform happens in the **input** and the result function returns its input unchanged, +so the cache can never hit. A plain `createSelector` only helps when its *inputs* are +reference-stable; controller-state slices usually are not. ```typescript -// ❌ BROKEN: Object.values() in INPUT creates new array +// ❌ BROKEN: Object.values() in the INPUT creates a new array each call export const getAccounts = createSelector( (state) => Object.values(state.accounts), (accounts) => accounts, // identity — cache never hits ); -// ✅ FIXED: Stable input, transform in OUTPUT +// ✅ FIXED: stable input, transform in the OUTPUT export const getAccounts = createSelector( - (state) => state.accounts, // stable Immer reference + (state) => state.accounts, // stable structural reference (accounts) => Object.values(accounts), ); ``` -Detection: Jest warning `"result function returned its own inputs"` +Detection: the reselect/Jest warning `"result function returned its own inputs"`. -### 3. Unnecessary Deep Equality +## 3. New collection allocated in the result function -`createDeepEqualSelector` adds O(n) overhead when Immer already provides stable references. +Even a correctly-shaped `createSelector` returns a new reference whenever it recomputes — +and if its inputs are unstable, that is every dispatch. ```typescript -// ❌ UNNECESSARY: state.accounts is already stable -const getAccounts = createDeepEqualSelector( - (state) => state.metamask.accounts, - (accounts) => transformAccounts(accounts), -); +// ❌ new array/Set/Map/object every call → always "changed" +(accounts) => Object.values(accounts).sort(...) +(transactions) => new Set(transactions.flatMap(...)) +(items) => items.filter(...) +(state) => state.swapsTransactions ?? {} // a fresh {} on every nullish hit +``` + +**Fix:** a deep-equal selector creator (returns the *cached* reference when data is +unchanged), a stable module-level constant for the empty case, or a result-equality check. + +## 4. Mutation in the result function + +```typescript +// ❌ mutates the input array AND returns a corrupting reference +createSelector([getItems], (items) => { items.sort(cmp); return items; }) +``` + +**Fix:** copy first — `[...items].sort(cmp)`. -// ✅ CORRECT -const getAccounts = createSelector( - (state) => state.metamask.accounts, +## 5. Over-broad input + +`state => state`, or a large slice, as an input selector forces recomputation on **any** +state change anywhere. Narrow the input to the smallest slice that actually feeds the +result. + +## 6. Unnecessary deep equality + +Deep-equal creators cost O(n) per comparison. Reaching for one when the input is already +reference-stable pays that cost for nothing — and deep-comparing a large slice on every +dispatch can be worse than the re-render it prevents. + +```typescript +// ❌ UNNECESSARY: this slice is already reference-stable +const getAccounts = createDeepEqualSelector( + (state) => state.accounts, (accounts) => transformAccounts(accounts), ); ``` -Use `createDeepEqualSelector` only when inputs are genuinely not from Immer/Redux state. +Prefer **narrowing the input** over deep-equalizing a giant object. -### 4. O(n) Lookups +## 7. O(n) lookups over unnormalized state -`.find()` on Object.values is O(n). With n items × m selectors per state change = O(n×m). +`.find()` over `Object.values()` is O(n). With n items × m selectors per state change that +is O(n×m) on every dispatch. ```typescript // ❌ BROKEN @@ -83,16 +119,16 @@ export const getAccountByAddress = (state, address) => export const getAccountByAddress = (state, address) => state.accounts[address]; ``` -### 5. Chained Transforms (Unmemoized) +## 8. Chained unmemoized transforms -Each transform creates a new array. Multiple transforms = multiple new references per call. +Each transform allocates. Several in sequence means several new references per call. ```typescript // ❌ BROKEN: 3 new arrays per call export function getSortedItems(state) { const items = Object.values(state.items); // array 1 const filtered = items.filter(isVisible); // array 2 - return filtered.sort(byDate); // array 3 + return filtered.sort(byDate); // array 3 } // ✅ FIXED: single memoized output @@ -104,12 +140,24 @@ export const getSortedItems = createSelector( ); ``` -## Selector Creator Decision Tree +## Selector creator decision tree ``` -Is INPUT unstable (not from Immer/Redux)? -├── YES → createDeepEqualSelector -└── NO → Is OUTPUT unstable (new array/object from transform)? - ├── YES → createResultEqualSelector (or createShallowResultSelector) - └── NO → createSelector +Is the INPUT unstable (a fresh object/array every dispatch)? +├── YES → deep-equal selector creator (but prefer narrowing the input first) +└── NO → Is the OUTPUT unstable (a new array/object from the transform)? + ├── YES → result-equality selector creator + └── NO → plain createSelector ``` + +## Don't over-correct + +- A selector returning a **primitive** is fine even if it filters internally — the consumer + memoizes on the primitive value. Wasteful allocation, not a re-render bug. +- Memoization is not free. Prefer narrowing inputs over adding comparison work. + +## Related + +- `render-cascade` — what one broken root selector does to the component graph downstream. +- Per-repo instances: the `mm-selector-memoization` reference documents a codebase's own + selector creators, its verified broken selectors, and the fix recipe for each. diff --git a/domains/performance/skills/effect-anti-pattern-review/skill.md b/domains/performance/skills/effect-anti-pattern-review/skill.md index 97283ccb..1ecc8d86 100644 --- a/domains/performance/skills/effect-anti-pattern-review/skill.md +++ b/domains/performance/skills/effect-anti-pattern-review/skill.md @@ -1,12 +1,12 @@ --- maturity: experimental name: effect-anti-pattern-review -description: Review PR diffs that add or modify `useEffect` for the four systemic React effect anti-patterns +description: Review PR diffs that add or modify `useEffect` for the systemic React effect anti-patterns --- # Effect Anti-Pattern Review -**Scope:** Pre-merge review of PRs that add or modify `useEffect` calls. The workflow is a grep-driven checklist against the four patterns catalogued in [`effect-anti-patterns`](../../knowledge/effect-anti-patterns.md). +**Scope:** Pre-merge review of PRs that add or modify `useEffect` calls. The workflow is a grep-driven checklist against the patterns catalogued in the **`effect-anti-patterns`** knowledge file, which is the single source for their definitions and fixes (installed alongside this skill under `knowledge/`). Applies to both `metamask-extension` and `metamask-mobile`. See overlays for repo-specific paths. @@ -26,19 +26,20 @@ Applies to both `metamask-extension` and `metamask-mobile`. See overlays for rep 1. **List changed files with `useEffect`.** `git diff --name-only origin/main...HEAD | xargs grep -l 'useEffect'` 2. **Run the [grep checklist](#grep-checklist)** against the changed files. -3. **For each hit, map to a pattern** in [`effect-anti-patterns`](../../knowledge/effect-anti-patterns.md) and apply the fix from the knowledge file. -4. **Block on pattern 1.** `JSON.stringify` in a dependency array is always broken. Do not merge. -5. **Block on pattern 3 without cleanup.** Any `setInterval` / `setTimeout` without a matching `clearInterval` / `clearTimeout` in the cleanup function is blocking. +3. **For each hit, map to a pattern** in `effect-anti-patterns` and apply the fix from the knowledge file. +4. **Block on unstable dependency identity.** `JSON.stringify` in a dependency array is always broken. Do not merge. +5. **Block on a timer without cleanup.** Any `setInterval` / `setTimeout` without a matching `clearInterval` / `clearTimeout` in the cleanup function is blocking. 6. **Require cancellation for async effects.** Any `fetch` / network call inside `useEffect` must use `AbortController`. ## Grep Checklist -| Pattern | Detection | Knowledge ref | -|---|---|---| -| 1. `JSON.stringify` in deps | `grep -rnE 'useEffect.*\[.*JSON\.stringify' <source-dir>` | [§1](../../knowledge/effect-anti-patterns.md#1-jsonstringify-in-dependency-array) | -| 2. State-mirror effect | Hand review — look for `useEffect` that calls `setX` based on other state/props | [§2](../../knowledge/effect-anti-patterns.md#2-useeffect--setstate-state-mirror-pattern) | -| 3. Missing interval/timer cleanup | `grep -rnE 'setInterval\|setTimeout' <source-dir>` then check each effect returns a cleanup | [§3](../../knowledge/effect-anti-patterns.md#3-missing-intervaltimer-cleanup) | -| 4. Missing `AbortController` | `grep -rnB2 -A10 'fetch\(' <source-dir>` within `useEffect` blocks | [§4](../../knowledge/effect-anti-patterns.md#4-missing-abortcontroller-in-async-effects) | +| Pattern (`effect-anti-patterns` §) | Detection | +|---|---| +| §1 Unstable dependency identity | `grep -rnE 'useEffect.*\[.*JSON\.stringify' <source-dir>`, plus inline `{`/`[` literals in the dep position | +| §2 Wrong dependencies | Hand review — empty deps that read state (stale closure), or deps that read nothing | +| §3 Derived state via effect + setState | Hand review — `useEffect` that calls `setX` from other state/props; §3a for chains of them | +| §4 Missing timer cleanup | `grep -rnE 'setInterval\|setTimeout' <source-dir>` then check each effect returns a cleanup | +| §5 Uncancelled async work | `grep -rnB2 -A10 'fetch\(' <source-dir>` within `useEffect` blocks | See the repo overlay for the concrete `<source-dir>` path. diff --git a/domains/performance/skills/performance/references/mm-hook-dependency-arrays.md b/domains/performance/skills/performance/references/mm-hook-dependency-arrays.md index 5733ff81..babcb790 100644 --- a/domains/performance/skills/performance/references/mm-hook-dependency-arrays.md +++ b/domains/performance/skills/performance/references/mm-hook-dependency-arrays.md @@ -8,6 +8,8 @@ tags: useEffect, useMemo, useCallback, dependencies, JSON.stringify Dependency arrays decide when `useEffect`/`useMemo`/`useCallback` re-run. The most common MetaMask problem is **`JSON.stringify` inside a dependency array** — it runs a synchronous serialization on every render just to compute the dependency key, which is both expensive and a sign the upstream reference is unstable. +> **Scope.** This file is the *dependency* half of effect performance, instantiated for this codebase. The platform-agnostic taxonomy — unstable dependency identity, wrong dependencies, derived state via effect, cascading effect chains, missing cleanup, uncancelled async — lives in the **`effect-anti-patterns`** knowledge file, installed alongside this skill under `knowledge/`. Read that for the general shape; read this for the verified instances and the repo's lint gaps. + ## Pattern — `JSON.stringify` in deps ```ts diff --git a/domains/performance/skills/performance/references/mm-selector-memoization.md b/domains/performance/skills/performance/references/mm-selector-memoization.md index b9e117db..6bbb7636 100644 --- a/domains/performance/skills/performance/references/mm-selector-memoization.md +++ b/domains/performance/skills/performance/references/mm-selector-memoization.md @@ -13,38 +13,22 @@ Broken or absent memoization in widely-used selectors is the single highest-impa - `createSelector` from `reselect` — reference-equality on inputs. - **`createDeepEqualSelector`** from `app/selectors/util.ts` — `createSelectorCreator(lruMemoize, deepEqual)`. Recomputes only when inputs are **deeply** equal-or-not. Use this when an input selector returns a fresh object/array on every dispatch (very common with controller state slices). -## The four deadly patterns - -### 1. Identity / passthrough in a plain `createSelector` -```ts -// ❌ Does nothing — output is the input, but the input ref changes every dispatch -export const selectX = createSelector(selectControllerState, (s) => s.things); -``` -A plain `createSelector` only helps if its **inputs** are reference-stable. Controller-state slices usually are not. Result: recomputes + new ref every dispatch. - -**Fix:** use `createDeepEqualSelector`, or narrow the input to the smallest stable slice. - -### 2. New collection in the result function -```ts -// ❌ new array/Set/Map/object every call → always "changed" -(accounts) => Object.values(accounts).sort(...) -(transactions) => new Set(transactions.flatMap(...)) -(items) => items.filter(...) -(state) => state.swapsTransactions ?? {} // new {} when nullish -``` -Even a correct `createSelector` produces a new reference whenever it recomputes; if the inputs aren't stable, that's every dispatch. - -**Fix:** `createDeepEqualSelector` (deep-compares so it returns the *cached* ref when data is unchanged), or a stable module-level constant for the empty case, or a `resultEqualityCheck`. - -### 3. Mutation in the result function -```ts -// ❌ mutates the input array AND returns a new-but-corrupting ref -createSelector([getItems], (items) => { items.sort(cmp); return items; }) -``` -**Fix:** copy first — `[...items].sort(cmp)`. - -### 4. `state => state` (or a huge slice) as an input selector -Forces recomputation on **any** state change anywhere. Narrow the input. +## The patterns, and what they look like here + +The pattern taxonomy itself lives in the **`selector-anti-patterns`** knowledge file, +installed alongside this skill under `knowledge/`. It is the single source — read it for the +full definition, the worked before/after of each, and the selector-creator decision tree. +This section maps each pattern onto *this* codebase. + +| Pattern | How it shows up in Mobile | Fix here | +|---|---|---| +| **Identity / passthrough result** | `createSelector(selectControllerState, (s) => s.things)` — controller-state slices are not reference-stable, so it recomputes and returns a new ref every dispatch | `createDeepEqualSelector`, or narrow the input to the smallest stable slice | +| **New collection in the result function** | `Object.values(...).sort(...)`, `new Set(...flatMap(...))`, `items.filter(...)`, `state.swapsTransactions ?? {}` | `createDeepEqualSelector`, a stable module-level constant for the empty case, or a `resultEqualityCheck` | +| **Mutation in the result function** | `createSelector([getItems], (items) => { items.sort(cmp); return items; })` | copy first — `[...items].sort(cmp)` | +| **Over-broad input** | `state => state`, or a whole controller slice, as an input selector | narrow the input | +| **Unnecessary deep equality** | reaching for `createDeepEqualSelector` on an already-stable slice | plain `createSelector`; see *Don't over-correct* below | + +The two that dominate the verified instances below are the first two. ## Verified MetaMask instances diff --git a/domains/performance/skills/selector-anti-pattern-review/skill.md b/domains/performance/skills/selector-anti-pattern-review/skill.md index 31b30be8..0b541d08 100644 --- a/domains/performance/skills/selector-anti-pattern-review/skill.md +++ b/domains/performance/skills/selector-anti-pattern-review/skill.md @@ -6,7 +6,7 @@ description: Review and diagnose Redux selector anti-patterns that cause render # Selector Anti-Pattern Review -**Scope:** Redux selector anti-patterns are the dominant cause of React render cascades in the MetaMask UI. This skill covers both review phases: pre-merge PR review (grep-driven checklist) and post-merge diagnosis (WDYR-driven workflow). Both modes resolve to the same root cause and the same fix set, catalogued in [`selector-anti-patterns`](../../knowledge/selector-anti-patterns.md) and [`render-cascade`](../../knowledge/render-cascade.md). +**Scope:** Redux selector anti-patterns are the dominant cause of React render cascades in the MetaMask UI. This skill covers both review phases: pre-merge PR review (grep-driven checklist) and post-merge diagnosis (WDYR-driven workflow). Both modes resolve to the same root cause and the same fix set, catalogued in the **`selector-anti-patterns`** and **`render-cascade`** knowledge files — the single source for their definitions (installed alongside this skill under `knowledge/`). Both `metamask-extension` and `metamask-mobile` share the same React + Redux architecture; this skill applies to both (see overlays for repo-specific paths). @@ -27,8 +27,8 @@ Both `metamask-extension` and `metamask-mobile` share the same React + Redux arc 1. **List changed selector/consumer files.** `git diff --name-only origin/main...HEAD | grep -E '(selectors|useSelector)'` 2. **Run the [grep checklist](#grep-checklist)** against the changed files. -3. **Match each hit to a pattern** in [`selector-anti-patterns`](../../knowledge/selector-anti-patterns.md) (numbered 1–5) or to one of the [team-specific workarounds](#team-specific-workarounds) below. -4. **Block on Jest warning.** If the PR's test run surfaces `"result function returned its own inputs"`, the PR introduces [Pattern 2](../../knowledge/selector-anti-patterns.md#2-identity-function-selector). Do not merge. +3. **Match each hit to a pattern** in `selector-anti-patterns` or to one of the [team-specific workarounds](#team-specific-workarounds) below. +4. **Block on Jest warning.** If the PR's test run surfaces `"result function returned its own inputs"`, the PR introduces an identity/passthrough result (`selector-anti-patterns` §2). Do not merge. 5. **Require a fix, not a justification.** None of the five patterns have a valid use case. See [Pitfalls](#common-pitfalls) for the narrow `createDeepEqualSelector` exception. ## Mode B: Post-Merge Diagnosis (WDYR-driven) @@ -41,24 +41,27 @@ Both `metamask-extension` and `metamask-mobile` share the same React + Redux arc ``` 2. **Enable WDYR.** `ENABLE_WHY_DID_YOU_RENDER=true yarn start` (same env var on extension and mobile). 3. **Identify root component.** The first WDYR log is the cascade origin. Do not fix downstream symptoms first. -4. **Classify via the [WDYR message table](#wdyr-message-interpretation).** If the root cause is a selector, return to [Mode A](#mode-a-pre-merge-review-grep-driven) and apply the fix set. If it is a context value or prop identity issue, see [`render-cascade`](../../knowledge/render-cascade.md). +4. **Classify via the [WDYR message table](#wdyr-message-interpretation).** If the root cause is a selector, return to [Mode A](#mode-a-pre-merge-review-grep-driven) and apply the fix set. If it is a context value or prop identity issue, see the `render-cascade` knowledge file. 5. **Verify.** Repeat the action. Confirm the counter stabilizes (e.g. 0→2, not 0→25). Divide raw counts by 2 under React Strict Mode. ## Grep Checklist -| Pattern | Detection | Knowledge ref | -|---|---|---| -| 1. Plain function selector | `grep -rE 'export function get' <selectors-dir>/` | [§1](../../knowledge/selector-anti-patterns.md#1-plain-function-selector) | -| 2. Identity function selector | Jest warning `result function returned its own inputs` | [§2](../../knowledge/selector-anti-patterns.md#2-identity-function-selector) | -| 3. Unnecessary `createDeepEqualSelector` | `grep -rn 'createDeepEqualSelector' <selectors-dir>/` then verify each input is not from Immer state | [§3](../../knowledge/selector-anti-patterns.md#3-unnecessary-deep-equality) | -| 4. O(n) lookup | `grep -rnE '\.find\(.*=>.*address' <selectors-dir>/` | [§4](../../knowledge/selector-anti-patterns.md#4-on-lookups) | -| 5. Chained unmemoized transforms | `grep -rnE 'export function get.*\{' <selectors-dir>/ -A5` and check for multiple `.filter/.map/.sort` without memoization | [§5](../../knowledge/selector-anti-patterns.md#5-chained-transforms-unmemoized) | +| Pattern (`selector-anti-patterns` §) | Detection | +|---|---| +| §1 Unmemoized selector | `grep -rE 'export function get' <selectors-dir>/` | +| §2 Identity / passthrough result | Jest warning `result function returned its own inputs` | +| §3 New collection in the result function | `grep -rnE 'new Set\|new Map\|Object\.(values\|keys\|entries)\|\?\? \{\}\|\?\? \[\]' <selectors-dir>/` | +| §4 Mutation in the result function | `grep -rnE '\.sort\(\|\.reverse\(\|\.push\(\|\.splice\(' <selectors-dir>/` | +| §5 Over-broad input | `grep -rn 'state) => state\b' <selectors-dir>/` | +| §6 Unnecessary deep equality | `grep -rn 'createDeepEqualSelector' <selectors-dir>/` then verify each input is genuinely unstable | +| §7 O(n) lookup | `grep -rnE '\.find\(.*=>.*address' <selectors-dir>/` | +| §8 Chained unmemoized transforms | `grep -rnE 'export function get.*\{' <selectors-dir>/ -A5`, then look for several `.filter/.map/.sort` without memoization | See the repo overlay for the concrete `<selectors-dir>` path. ## Team-Specific Workarounds -Two patterns show up beyond the five in the knowledge file. Both are workarounds for broken selectors downstream. The fix is always to fix the selector, never to propagate the workaround. +Two patterns show up beyond those in the knowledge file. Both are workarounds for broken selectors downstream. The fix is always to fix the selector, never to propagate the workaround. ### `useSelector(selector, isEqual)` from `react-redux` From f5aa6f9be786e27c07db6be9bf4762eb29bccb9e Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 08:38:40 -0400 Subject: [PATCH 023/135] Consolidate the shift-left performance work into one domain PR MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Folds in the react-render-proof skill (was #82) and the mobile reference-library additions (was #49). All three were the same effort seen from different ends — moving performance work earlier in the loop — and they share a substrate, so reviewing them apart meant reviewing the substrate three times. The loop this domain now covers: - catch it at review — effect/selector anti-pattern review skills, driven by the knowledge taxonomy - prove it moved — react-render-proof, with a delivery gate so an arm whose treatment never reached the bundle cannot report as a null - measure it honestly — data-analysis, benchmark hygiene, web-vitals framing - know the codebase — the mm-* reference library and its audit playbook Also neutralizes five references to private planning tickets, which do not belong on a public repository — they named internal epic and audit-ticket numbers. The surrounding guidance is unchanged; only the identifiers are gone. --- .../references/mm-audit-playbook.md | 19 ++- .../references/mm-hook-dependency-arrays.md | 1 + .../performance/references/mm-planning.md | 6 +- .../mm-react-compiler-error-triage.md | 90 +++++++++++ .../references/mm-react-compiler.md | 3 + .../references/mm-redux-antipatterns.md | 4 + .../references/mm-selector-cascade.md | 122 ++++++++++++++ .../references/mm-selector-memoization.md | 2 + .../references/mm-state-normalization.md | 136 ++++++++++++++++ .../skills/performance/references/mm-tools.md | 20 ++- .../references/mm-useeffect-antipatterns.md | 149 ++++++++++++++++++ .../performance/repos/metamask-mobile.md | 10 +- .../skills/react-render-proof/skill.md | 112 +++++++++++++ 13 files changed, 661 insertions(+), 13 deletions(-) create mode 100644 domains/performance/skills/performance/references/mm-react-compiler-error-triage.md create mode 100644 domains/performance/skills/performance/references/mm-selector-cascade.md create mode 100644 domains/performance/skills/performance/references/mm-state-normalization.md create mode 100644 domains/performance/skills/performance/references/mm-useeffect-antipatterns.md create mode 100644 domains/performance/skills/react-render-proof/skill.md diff --git a/domains/performance/skills/performance/references/mm-audit-playbook.md b/domains/performance/skills/performance/references/mm-audit-playbook.md index b3f302c7..ff756989 100644 --- a/domains/performance/skills/performance/references/mm-audit-playbook.md +++ b/domains/performance/skills/performance/references/mm-audit-playbook.md @@ -12,6 +12,7 @@ For reviewing a PR/diff or auditing a file, component, or feature. Output: findi - **Targeted** (single file / component / small diff): read the files and report concrete findings with `file:line`. - **Broad** (whole feature / repo): run the grep sweeps below and triage hits; don't read everything. +- **Audit wave / program** (scheduled audit of a surface or division): per-surface audits miss mechanism-level patterns that live in *shared* infrastructure (`app/selectors`, shared hooks, the store) — run the cross-cutting sweeps below over the shared dirs **once per wave**, not once per team, and route findings to surface owners. Attach quantified acceptance criteria up front (template in [mm-planning.md](mm-planning.md)). If the surface ships on both platforms, cross-check the sibling platform's audit findings for the same surface before fresh discovery — the React/Redux mechanism patterns recur across extension and mobile. Always: **measure before asserting impact** where feasible, and respect the guardrails at the bottom (don't over-flag). @@ -42,8 +43,10 @@ Read the call sites: is a data hook running for tabs/pages/items that aren't vis grep -rn "createSelector(" app/selectors --include="*.ts" | grep -v createDeepEqualSelector grep -rn "=> .*\.\(map\|filter\|sort\|reverse\)\|new Set\|new Map\|Object\.\(values\|keys\|entries\)\|?? {}\|?? \[\]" app/selectors --include="*.ts" grep -rn "\.sort(\|\.reverse(\|\.push(\|\.splice(" app/selectors --include="*.ts" # mutation +grep -rn "(_state\|(_," app/selectors --include="*.ts" # parameterized selectors — single-entry cache → mm-state-normalization.md +grep -rnE "export (function|const) (get|select)[A-Z][A-Za-z]* = \(state|export function (get|select)" app/selectors --include="*.ts" # plain unmemoized function selectors (no createSelector at all) ``` -Check each result function for: identity/passthrough, new collection without deep-equal, mutation, `state=>state` input. +Check each result function for: identity/passthrough, new collection without deep-equal, mutation, `state=>state` input. If one broken selector has **many consumers**, switch to the cascade playbook — map the dependency tree to closure and plan the fix order *before* fixing anything: [mm-selector-cascade.md](mm-selector-cascade.md). ### Redux / useSelector → [mm-redux-antipatterns.md](mm-redux-antipatterns.md) ```bash @@ -57,11 +60,18 @@ grep -rn "dispatch(" app --include="*.ts" --include="*.tsx" | grep -v ".test." | grep -rn "Provider value={{" app --include="*.tsx" | grep -v ".test." ``` -### Hooks → [mm-hook-dependency-arrays.md](mm-hook-dependency-arrays.md) +### Hooks → [mm-hook-dependency-arrays.md](mm-hook-dependency-arrays.md) / [mm-useeffect-antipatterns.md](mm-useeffect-antipatterns.md) ```bash grep -rn "\[JSON.stringify\|, JSON.stringify" app --include="*.ts" --include="*.tsx" | grep -v ".test." +grep -rn -A6 "useEffect(" app --include="*.ts" --include="*.tsx" | grep -E "fetch\(|\.then\(" | grep -v "signal\|cancelled\|abort" | grep -v ".test." # async effects without cancellation ``` -(`exhaustive-deps` is NOT linted in this repo — check effect deps by hand.) +(`exhaustive-deps` is NOT linted in this repo — check effect deps by hand.) For effect-body problems — derived state via useEffect+setState, effect chains, post-unmount setState — use the read pass in [mm-useeffect-antipatterns.md](mm-useeffect-antipatterns.md). + +### React Compiler coverage → [mm-react-compiler-error-triage.md](mm-react-compiler-error-triage.md) +```bash +grep -rn "use no memo" app --include="*.ts" --include="*.tsx" # opt-outs: each needs a reason + TODO +``` +For a re-render-heavy screen, confirm the components are actually **compiled** (`Memo ✨` in DevTools) before suggesting manual memoization — they may be sitting in the error/unsupported bucket. ### Animations → [mm-layout-animations.md](mm-layout-animations.md) ```bash @@ -97,6 +107,7 @@ Grep finds *syntactic* patterns. The highest-impact re-render bugs are *data-flo - **Render-phase side effects / setState** — any `setState(...)`, `dispatch(...)`, or `trackEvent(...)` in a render body (not inside `useEffect`/`useCallback`)? Triggers extra render passes. - **O(n²) reduce-with-spread** — `reduce((acc, x) => ({ ...acc, ... }), {})` rebuilt every render. - **Per-item subscription hooks** — trace each into its manager; shared subscription = fine, per-subscriber whole-dataset snapshot = bug. → [mm-streaming-realtime.md](mm-streaming-realtime.md) +- **Deep-equal selector inputs** — for every `createDeepEqualSelector`, read its *input selectors*: an input function that allocates a fresh composite per call (object spreads of controller state, other selectors' results collected into a new object) forces the deep compare to run over the whole composite on every check — and no result-function grep catches it. `grep -rn -B3 "createDeepEqualSelector(" app/selectors` lists the sites; read each first argument. → [mm-selector-cascade.md](mm-selector-cascade.md) (proactive mode) Confirm any hit with the Profiler ("why did this render?") before asserting — see [mm-tools.md](mm-tools.md). @@ -107,6 +118,8 @@ Confirm any hit with the Profiler ("why did this render?") before asserting — - [ ] No real-time / high-frequency data dispatched to Redux - [ ] `Context.Provider value` is memoized (not an inline object) - [ ] No `JSON.stringify` in a hot dependency array +- [ ] Async effects guard against post-unmount / stale setState (cancelled flag or `AbortController`) +- [ ] No new parameterized selector (single-entry cache) on a list/hot path — use a lookup-map selector instead - [ ] Layout animations use Reanimated v3, not `Animated` + `useNativeDriver:false` - [ ] Growable lists use FlashList with stable keys (+ `getItemType` if mixed) - [ ] New event listeners / timers / subscriptions have cleanup diff --git a/domains/performance/skills/performance/references/mm-hook-dependency-arrays.md b/domains/performance/skills/performance/references/mm-hook-dependency-arrays.md index babcb790..5ff27dc0 100644 --- a/domains/performance/skills/performance/references/mm-hook-dependency-arrays.md +++ b/domains/performance/skills/performance/references/mm-hook-dependency-arrays.md @@ -85,5 +85,6 @@ For each hit, ask: *does this dependency change identity every render?* If yes, ## Related +- [mm-useeffect-antipatterns.md](mm-useeffect-antipatterns.md) — the effect-body side: derived state, effect chains, unmount-safe async, cleanup - [js-react-compiler.md](js-react-compiler.md) / [mm-react-compiler.md](mm-react-compiler.md) — automatic memoization on opted-in paths - [js-concurrent-react.md](js-concurrent-react.md) — defer expensive derived work diff --git a/domains/performance/skills/performance/references/mm-planning.md b/domains/performance/skills/performance/references/mm-planning.md index ca23c644..3a610685 100644 --- a/domains/performance/skills/performance/references/mm-planning.md +++ b/domains/performance/skills/performance/references/mm-planning.md @@ -20,9 +20,9 @@ The cheapest performance fix is the one you make before writing code. Catch arch | Risk | Trigger question | Default mitigation | |---|---|---| | Real-time / WebSocket data | Updates faster than once per user action? | Never put it in Redux. Local state / shared value / direct UI update. Manage subscribe/unsubscribe by visibility + app foreground/background; avoid double-subscribe. See [mm-redux-antipatterns.md](mm-redux-antipatterns.md). | -| Unbounded data | Can the list/dataset grow without ceiling? | Paginate + virtualize from day one; plan server-side filtering. | +| Unbounded data | Can the list/dataset grow without ceiling? | Paginate + virtualize from day one; plan server-side filtering. Never persist unbounded data via redux-persist — use a dedicated storage layer. | | Large lists | >~50 items now, infinite later? | FlashList v2 with stable keys + `getItemType`; no heavy work per item. [js-lists-flatlist-flashlist.md](js-lists-flatlist-flashlist.md) | -| New selector / derived state | Adding `createSelector`? | Decide memoization + equality up front; never identity/mutation. [mm-selector-memoization.md](mm-selector-memoization.md) | +| New selector / derived state | Adding `createSelector`? | Decide memoization + equality up front; never identity/mutation. [mm-selector-memoization.md](mm-selector-memoization.md). Frequent keyed lookups? Decide the lookup shape now (keyed index vs O(n) scan) — [mm-state-normalization.md](mm-state-normalization.md) | | Heavy computation | Big transforms, sorts, regex on large input? | Server offload, or memoize, or defer with `useDeferredValue`. | | Crypto | Hashing/signing/derivation in hot path? | `react-native-quick-crypto` (already installed); keep off the JS thread. | | New npm dependency | Adds to `package.json`? | Check size (Expo Atlas / bundlephobia); avoid main-package/barrel imports; reuse existing libs (we already have dayjs, luxon, lodash). [bundle-library-size.md](bundle-library-size.md) | @@ -34,7 +34,7 @@ The cheapest performance fix is the one you make before writing code. Catch arch ## System-design checklist -- **State shape:** new Redux slice for real-time data? → flag. New selector? → memoization + equality decided now. +- **State shape:** new Redux slice for real-time data? → flag. New selector? → memoization + equality decided now. Frequent lookups by key? → plan a `byId`/`byAddress` index ([mm-state-normalization.md](mm-state-normalization.md)). - **Subscription lifecycle:** diagram subscribe/unsubscribe tied to mount/unmount + foreground/background; no double-subscribe; cleanup guaranteed. - **List strategy:** ScrollView only for <20 fixed items; FlashList for anything that can grow; no `.map()` in JSX for growable lists. - **Data flow:** minimize how many components subscribe to a frequently-updating selector. diff --git a/domains/performance/skills/performance/references/mm-react-compiler-error-triage.md b/domains/performance/skills/performance/references/mm-react-compiler-error-triage.md new file mode 100644 index 00000000..e726279d --- /dev/null +++ b/domains/performance/skills/performance/references/mm-react-compiler-error-triage.md @@ -0,0 +1,90 @@ +--- +title: React Compiler Error Triage & Coverage Accounting (MetaMask) +impact: HIGH +tags: react-compiler, panicThreshold, error-triage, coverage, babel, build +--- + +# Skill: React Compiler Error Triage & Coverage Accounting + +The React Compiler **fails open**: when it can't compile a component, it silently skips it and ships the unoptimized original. The build stays green, DevTools shows no warning — you just don't get the memoization. Once the compiler is enabled broadly (metamask-mobile#31171 enabled v1.0.0 app-wide), the question stops being "is it on?" and becomes **"what is it actually compiling, and which of its errors are worth fixing?"** This file is the triage playbook. Extension PR metamask-extension#38007 is the reference implementation. + +## The `panicThreshold` ladder + +`panicThreshold` controls when a compiler diagnostic fails the build instead of silently skipping the file: + +| Setting | Build fails on | Use for | +|---|---|---| +| `'none'` (default) | never — every failed file is **silently skipped** | production builds, always | +| `'critical_errors'` | only critical errors (compiler-internal invariant violations) | CI / debug builds, first ratchet target | +| `'all_errors'` | every diagnostic, including unsupported syntax | CI / debug builds, end-state ratchet | + +**The ratchet strategy** (extension roadmap, tracked internally): keep production at `'none'` permanently; aim for a *non-production* build that passes at `'critical_errors'`, then at `'all_errors'`. Each ratchet step turns a class of silent skips into a visible, fixable error list. Never enable a non-`'none'` threshold in a release build — one un-compilable file would block the release for an optimization that is optional by design. + +## Triage: unsupported syntax vs. legitimate errors + +Compiler diagnostics are **not one bucket**. The logger event's `category` field separates them, and the distinction decides whether you act: + +- **`category === 'Todo'` → "unsupported."** Syntax or a pattern the compiler *itself* has not implemented yet. There is **no actionable fix on our side** — rewriting working code to appease an unimplemented compiler path is wasted effort and churn. Count these separately, leave the code alone, and re-check after compiler upgrades. +- **Any other category (e.g. `InvalidReact`, `InvalidJS`) → legitimate, actionable.** A real Rules-of-React violation in our code (mutation during render, conditional hooks, side effects in render). Fixing it both unlocks compilation *and* removes a latent correctness bug. + +A healthcheck that doesn't make this split is noise: the `Todo` count swamps the actionable list and the team learns to ignore the output. The extension's verbose run at enablement (metamask-extension#38007) is the canonical illustration — of 7,308 files processed: 253 compiled, **31 actionable errors**, **7,024 unsupported** (`Todo`). Without the split that reads as ~7,000 hopeless errors; with it, the team's backlog is 31 files and the rest is the compiler's to burn down across upgrades. The extension's webpack wrapper makes the split in ~10 lines: + +```ts +// adapted from metamask-extension development/webpack/utils/loaders/reactCompilerLoaderWrapper.ts +// (mobile equivalent: pass a `logger` in babel-plugin-react-compiler options) +logger: { + logEvent(filename, event) { + switch (event.kind) { + case 'CompileSuccess': record(filename, 'compiled'); break; + case 'CompileSkip': record(filename, 'skipped'); break; + case 'CompileError': { + const category = event.detail?.options?.category ?? event.detail?.category; + // 'Todo' = not yet supported by the compiler — no actionable fix on our side + record(filename, category === 'Todo' ? 'unsupported' : 'error'); + break; + } + } + }, +} +``` + +The extension exposes this as `yarn webpack --reactCompilerVerbose` (per-file ✅/⏭️/🔍/❌ output + summary stats) and `--reactCompilerDebug={all|critical|none}` (maps to `panicThreshold: '<value>_errors'`). On mobile the same taxonomy is available through the Babel plugin's `logger` option or `eslint-plugin-react-compiler` (the lint rule runs the same analysis the compiler does). + +## Coverage accounting + +Track four buckets — **compiled / skipped / errors / unsupported** — at file and component granularity, with **worst-status-wins per file** (`error > unsupported > skipped > compiled`): a file with five compiled components and one error is an *error file*, otherwise mixed files inflate the compiled count and the number lies to you. + +What the buckets tell you: + +- **compiled** — your real optimization coverage. "The compiler is enabled" claims nothing; this number does. +- **errors** — the actionable backlog. Each is a Rules-of-React fix. +- **unsupported** — the compiler's backlog, not yours. Trend it across compiler upgrades. +- **skipped** — intentional exclusions: test/story files, `'use no memo'` directives, and **class components** (never compiled — metamask-mobile#30919 counted 53 at full enablement; migration to function components is the only way to move them into the compiled bucket). + +## Staged adoption roadmap + +The extension's sequence (internal epic) generalizes to any repo: + +1. **Lint clean:** update `eslint-plugin-react-hooks` / `eslint-plugin-react-compiler` to latest; fix violations — these are exactly what the compiler will refuse to compile. +2. **Audit opt-outs:** every `'use no memo'` carries a reason + TODO; the count only goes down. `grep -rn "use no memo" app --include="*.ts*"`. +3. **Ratchet `critical_errors`:** non-prod build passes; fix what surfaces. +4. **Ratchet `all_errors`:** remaining actionable errors fixed; what's left is the `Todo` (unsupported) set, which you wait out. + +## Verify + +- Per component: `Memo ✨` badge in React DevTools (see [js-profile-react.md](js-profile-react.md)). +- Per repo: the compiled-files count from the logger stats rises (or at least doesn't silently fall) release over release — silent coverage regressions are the failure mode this file exists to catch. +- After a compiler version bump: re-run the verbose build and diff the `unsupported` list — `Todo`s that became `compiled` are free wins; new `error`s are regressions to triage. + +## Don't over-correct + +- **Never "fix" a `Todo`.** Rewriting working code around an unimplemented compiler feature is churn with no perf evidence; the next compiler release may compile it as-is. +- Don't gate releases on compiler errors (`panicThreshold` stays `'none'` in production builds). +- Don't treat `skipped` as a problem — tests, stories, and deliberate opt-outs belong there. The smell is *unexplained* `'use no memo'` directives, not the bucket itself. +- A component without `Memo ✨` is not automatically a bug to chase — check the buckets first; it may be `unsupported`. + +## Related + +- [mm-react-compiler.md](mm-react-compiler.md) — enabling the compiler in this repo (Babel config, Metro cache, ESLint healthcheck) +- [js-react-compiler.md](js-react-compiler.md) — how the compiler transforms code; Rules-of-React background +- [mm-selector-cascade.md](mm-selector-cascade.md) — what the compiler **cannot** fix: unstable values crossing file boundaries (selectors, imported hooks) diff --git a/domains/performance/skills/performance/references/mm-react-compiler.md b/domains/performance/skills/performance/references/mm-react-compiler.md index e95d70d6..4f987cfb 100644 --- a/domains/performance/skills/performance/references/mm-react-compiler.md +++ b/domains/performance/skills/performance/references/mm-react-compiler.md @@ -51,6 +51,8 @@ React Compiler auto-memoizes components, callbacks, and computed values at build On opted-in paths you can gradually drop hand-written `useMemo`/`useCallback`/`React.memo` once the compiler is verified working — but do it deliberately and re-measure. Off opted-in paths, manual memoization still matters. +**Exception — effect dependencies.** Keep any `useMemo`/`useCallback` whose output is used as a `useEffect` dependency, here or in a consumer: the compiler's memoization is not guaranteed to match the manual strategy, and a mismatch causes over/under-firing of effects or infinite loops — a correctness change, not a perf tweak. Official guidance is to leave existing manual memoization in place and only omit it in *new* code ([reactwg/react-compiler#16](https://github.com/reactwg/react-compiler/discussions/16)). + ## What breaks compilation (it will skip the component) - Mutating props or state during render. @@ -68,4 +70,5 @@ Fix the ESLint `react-compiler` warnings on a path before/after opting it in. ## Related - [js-react-compiler.md](js-react-compiler.md) — upstream reference on how the compiler transforms code +- [mm-react-compiler-error-triage.md](mm-react-compiler-error-triage.md) — triaging compiler errors (`Todo`/unsupported vs actionable), `panicThreshold` ratcheting, and measuring real coverage - [mm-selector-memoization.md](mm-selector-memoization.md) — fix data-layer re-renders the compiler can't diff --git a/domains/performance/skills/performance/references/mm-redux-antipatterns.md b/domains/performance/skills/performance/references/mm-redux-antipatterns.md index a830bc35..965811a7 100644 --- a/domains/performance/skills/performance/references/mm-redux-antipatterns.md +++ b/domains/performance/skills/performance/references/mm-redux-antipatterns.md @@ -38,6 +38,8 @@ const browserTabs = useSelector((state: any) => state.browser.tabs); // also dro **Why it's wrong:** an inline accessor returning an array/object hands a fresh reference to the consumer whenever that slice changes (and defeats reuse/memoization across the app). For derived data it's worse — `useSelector(s => s.items.filter(...))` allocates every render. +**Perf-triage note:** an inline accessor returning a **primitive or stable field** (`s => s.settings.basicFunctionalityEnabled`) is reuse/type debt, not a re-render bug — the new arrow function per render is irrelevant; only the result's identity matters. Flag it for cleanup, not as a perf finding. + **Fix:** create a named selector in `app/selectors/`: ```ts // selectors/browser.ts @@ -86,5 +88,7 @@ grep -rn "dispatch(" app --include="*.ts" --include="*.tsx" | grep -v ".test." \ ## Related - [mm-selector-memoization.md](mm-selector-memoization.md) — the upstream fix for Pattern 1 +- [mm-selector-cascade.md](mm-selector-cascade.md) — repairing the whole dependency graph and removing accumulated `isEqual` band-aids after the root fix +- [mm-state-normalization.md](mm-state-normalization.md) — consolidating many `useSelector` calls into one view selector - [mm-context-performance.md](mm-context-performance.md) — the Context equivalent of over-broad subscriptions - [js-profile-react.md](js-profile-react.md) — confirm the re-render reduction diff --git a/domains/performance/skills/performance/references/mm-selector-cascade.md b/domains/performance/skills/performance/references/mm-selector-cascade.md new file mode 100644 index 00000000..403f9e5c --- /dev/null +++ b/domains/performance/skills/performance/references/mm-selector-cascade.md @@ -0,0 +1,122 @@ +--- +title: Selector Dependency Cascades — Blast Radius & Repair (MetaMask) +impact: CRITICAL +tags: reselect, cascade, dependency-graph, isEqual, structural-sharing, react-compiler +--- + +# Skill: Selector Dependency Cascades + +> **Scope.** What a broken root selector does to the component graph is defined generically +> in the **`render-cascade`** knowledge file, and the selector patterns that cause it in +> **`selector-anti-patterns`** — both installed alongside this skill under `knowledge/`. +> This file is the MetaMask Mobile instance: the real dependency graph, its blast radius, +> and the repair order. + +[mm-selector-memoization.md](mm-selector-memoization.md) catalogues the broken-selector *patterns*. This file is about what happens **downstream of one broken root selector** — and how to repair the whole graph instead of patching its leaves. Reference case: extension PR metamask-extension#37147, where a single identity output selector (`getInternalAccounts`) was recomputing through **15 direct + 35+ transitive consumer selectors into 50+ components on every dispatch** — every 5-second balance poll, every keystroke in the send flow. + +## Anatomy of a cascade + +```ts +// ❌ the root: identity output selector — memoizes nothing, new "result" every dispatch +export const getInternalAccounts = createSelector( + (state) => state.engine.internalAccounts.accounts, + (accounts) => accounts, // output === input: the cache can never hit meaningfully +); +``` + +Every consumer selector that takes the root as an input now sees a "changed" input on every dispatch, recomputes, and — because most result functions allocate (`.filter()`, `.map()`, `Object.values()`) — emits its *own* fresh reference, propagating the invalidation one layer further. Three layers down, nobody remembers the root; they see "my selector keeps firing" and reach for local fixes: + +```ts +// ❌ the band-aids that accumulate downstream of a broken root +const accounts = useSelector(getAccountsByScope, isEqual); // deep compare per dispatch +export const getX = createDeepEqualSelector(getInternalAccounts, …); // deep compare per dispatch +export const getMemoizedAccounts = createSelector(getInternalAccounts, (a) => a); // does nothing +``` + +Each band-aid suppresses the re-render for one consumer while *adding* an O(n) deep comparison on every dispatch — and the cascade cost scales superlinearly with power-user data (see [mm-power-user-scenario.md](mm-power-user-scenario.md)). + +A live cascade also **nullifies every optimization downstream of it**: `React.memo` children re-render anyway (their props are fresh refs), virtualized rows churn, compiler-memoized components re-render (the unstable value crosses the file boundary), and `useMemo`s recompute. Fix the cascade before evaluating any other optimization on the screen — and re-measure them after. + +## Step 1 — Traverse the dependency tree exhaustively before fixing + +The repair PR's evidence (and its review) should enumerate the graph **to closure** — every selector reachable from the suspect, not just its immediate neighborhood — the way #37147 did: + +1. **Direct consumers:** every selector that lists the suspect as an input. `grep -rn "getInternalAccounts" app/selectors --include="*.ts"` +2. **Transitive consumers:** repeat for each direct consumer until the frontier adds no new selectors. Don't stop at a fixed depth — cascades often have **more than one broken root**, and a partial map produces a wrong fix order. +3. **Component consumers:** `useSelector` call sites of anything in the graph. +4. **Recomputation count:** instrument with `selector.recomputations()` (reselect) or a `console.count` in the result function across a few dispatches (a balance poll is a convenient metronome). +5. **WDYR pass — already wired in this repo:** `wdyr.js` at the repo root tracks `useSelector` hook diffs. Run `ENABLE_WHY_DID_YOU_RENDER=true yarn start`, reproduce one dispatch, and every consumer logging *same values, different reference* is a node in the cascade — the live counterpart of the static map above. + +A before/after table — *recomputations per dispatch, re-renders per poll cycle, on the same interaction* — is what distinguishes a verified cascade fix from a speculative refactor. + +**Proactive mode — find the big trees without waiting for a symptom.** Rank roots by blast radius first (grep each selector's name across `app/` for consumer-file counts; appearances inside other selectors' input arrays give direct dependents), then for each large root verify its **input reference-stability**, not just its result function. The pattern that defeats every result-function grep: an input *function* that builds a fresh composite per call — spreading controller states and collecting other selectors' results into a new object. It looks disciplined, passes all pattern sweeps, and silently downgrades a `createDeepEqualSelector` into a whole-composite deep compare on **every check**. Verified instance: `getStateForAssetSelector` feeding `selectAssetsBySelectedAccountGroup` (`app/selectors/assets/assets-list.ts:107`) — the root of the asset-surface tree (15+ dependent selectors, including the per-row `selectAsset`), deep-comparing effectively the entire asset state per consumer per flush. + +## Step 2 — Plan the memoization fix order from the map: roots first + +Write down the fix order before writing any fix. The order is **topological** — roots, then their descendants, layer by layer: + +- A descendant fix can't be *verified* while any of its inputs is still unstable: its output identity keeps changing for upstream reasons, so the before/after numbers measure the wrong thing. +- Most descendant "problems" stop being fixes once the roots are stable — they reclassify from "add memoization here" to "remove the band-aid here" (Step 4). The plan is what tells you which is which *in advance*, instead of memoizing selectors that were only recomputing because of their inputs. +- If the map surfaced multiple roots sharing consumers, fix them together — otherwise the shared consumers keep re-rendering and the first root's win never shows up in the numbers. + +## Step 3 — Fix the root, not the 50 consumers + +Memoizing consumers one by one is whack-a-mole: each fix adds comparison cost and the graph keeps re-deriving from a poisoned root. Trace **upward** (who are my inputs? are *they* stable?) until you hit the selector whose output identity changes without its data changing — that's the root. Fix its memoization there (patterns + recipes in [mm-selector-memoization.md](mm-selector-memoization.md)). + +**Know your reference-stability contract first.** What the correct fix looks like depends on whether your store gives you stable references for unchanged data: + +- With **Immer-based reducers** (Redux Toolkit), structural sharing guarantees `state.a.b` keeps its reference **iff** nothing under that path changed. Under that contract, a plain `createSelector` over a *narrow* input is already correct, and deep-equal selectors are pure overhead. +- Where state is replaced wholesale on sync (documented for this repo's controller-state slices in [mm-selector-memoization.md](mm-selector-memoization.md)), input references break even when data didn't change, and `createDeepEqualSelector` at the *root* is the pragmatic tool. + +Establish which contract a slice actually follows (log `prev === next` for the input across two unrelated dispatches) before choosing — the answer differs per slice, and assuming the wrong contract either reintroduces the cascade or buys deep-compares you don't need. + +Then match the tool to **which side is unstable**: an unstable *input* (slice replaced wholesale on sync) calls for a deep-equal **input** compare (`createDeepEqualSelector`); a stable input with an unstable *output* (the result function allocates a fresh collection) calls for a `resultEqualityCheck`, so an unchanged result returns the cached ref. Deep-equalizing inputs to paper over an allocating result function runs the wrong comparison on every dispatch. + +Verified mechanism for this repo (`app/core/redux/slices/engine`): `UPDATE_BG_STATE` replaces only the **changed controller's key** with `Engine.state[key]`, and BaseController v2 state is Immer-produced — so an unchanged controller keeps its reference across flushes, and unchanged paths *within* a changed controller are structurally shared. Plain accessors into controller state are stable by construction; deep-equal is only warranted where a selector's *inputs* genuinely churn. And remember a deep-equal selector is output-**stable** but pays its compare per check, scaled by input size — over a power-user transaction history that is an O(n) deep compare per consumer per flush. + +## Step 4 — Sweep the graph and *remove* the band-aids + +This is the step most fixes skip. After the root is stable, every downstream `isEqual`, `createDeepEqualSelector`-wrapping-a-now-stable-input, and `getMemoized*` duplicate is dead weight: it still runs its deep comparison on every dispatch, and it **masks regressions** — if the root breaks again, the band-aids hide it until the app is slow everywhere again. + +```bash +# downstream band-aid sweep, scoped to the fixed graph +grep -rn "useSelector(.*isEqual)" app --include="*.tsx" | grep -v ".test." +grep -rn "createDeepEqualSelector" app/selectors --include="*.ts" +grep -rn "getMemoized\|selectMemoized" app/selectors --include="*.ts" +``` + +For each hit that consumes the fixed root (directly or transitively): remove the equality argument / downgrade to plain `createSelector`, and re-verify the consumer doesn't re-render on unrelated dispatches. #37147 deleted the band-aids in the same PR as the root fix — that's the model. + +## What the React Compiler can and cannot do here + +The compiler memoizes **within a file**. A `useSelector` result, an imported hook's return value, or an external context value is opaque to it — if the selector hands back a fresh reference, the compiled component still re-renders, and any derivation from it still recomputes (internal audit ticket): + +```tsx +const tokens = useSelector(selectTokens); // compiler cannot see/stabilize this +const rows = tokens.map(toRow); // ❌ recomputes every render even when compiled +const rows = useMemo(() => tokens.map(toRow), [tokens]); // ✅ still needed +``` + +Rule of thumb: values that **cross a file boundary** (Redux selectors, imported hooks/functions, external context) keep their manual `useMemo`/`useCallback`; same-file props/state derivations can lean on the compiler. Fix the selector graph first — automatic memoization downstream of an unstable root optimizes nothing. + +## Don't over-correct + +- Not every busy selector is a cascade root — a selector returning a **primitive** breaks the chain at that point regardless of recomputation (allocation waste ≠ re-render bug). +- A *global* top-level cascade (an unstable value in a root provider/HOC re-rendering the whole tree on every state change) is a pattern the extension audit found at app root — worth **ruling out** with one profiler pass ("why did this render?" on a top-level component during an unrelated dispatch), but don't assume it exists here; verify before restructuring providers. See [mm-context-performance.md](mm-context-performance.md) for the provider-value mechanics. +- Don't add `useMemo` around every `useSelector` read preemptively — only where a non-primitive result feeds a derivation or a memoized child (see guardrails in [mm-hook-dependency-arrays.md](mm-hook-dependency-arrays.md)). + +## Verify + +1. Root selector returns the **same reference** across two unrelated dispatches (the contract test from [mm-selector-memoization.md](mm-selector-memoization.md)). +2. Recomputation counts on direct + transitive consumers drop to ~0 on unrelated dispatches. +3. Profiler on a top consumer (account list, send flow): the re-render cascade is gone during a balance poll. +4. The band-aid greps above return no hits inside the repaired graph. +5. Lock the win in CI: add a Reassure `*.perf-test.tsx` on a top consumer so the cascade can't silently return. + +## Related + +- [mm-selector-memoization.md](mm-selector-memoization.md) — the root-selector patterns and fix recipes +- [mm-redux-antipatterns.md](mm-redux-antipatterns.md) — `useSelector(x, isEqual)` as symptom; per-consumer view +- [mm-unstable-hook-return.md](mm-unstable-hook-return.md) — the same cascade shape, with a hook as the root +- [mm-state-normalization.md](mm-state-normalization.md) — state shape that prevents cascade-prone selectors +- [mm-react-compiler-error-triage.md](mm-react-compiler-error-triage.md) — confirming what the compiler actually covers diff --git a/domains/performance/skills/performance/references/mm-selector-memoization.md b/domains/performance/skills/performance/references/mm-selector-memoization.md index 6bbb7636..c6731a62 100644 --- a/domains/performance/skills/performance/references/mm-selector-memoization.md +++ b/domains/performance/skills/performance/references/mm-selector-memoization.md @@ -97,4 +97,6 @@ Escalate severity by one level if the selector is imported in **10+ files**. ## Related - [mm-redux-antipatterns.md](mm-redux-antipatterns.md) — `useSelector(x, isEqual)` is the *symptom* of a broken selector; fix the selector, then remove the `isEqual`. +- [mm-selector-cascade.md](mm-selector-cascade.md) — graph-level view: blast radius of one broken root, and sweeping out downstream band-aids after the fix. +- [mm-state-normalization.md](mm-state-normalization.md) — state/selector *shape*: O(1) lookups, parameterized-selector cache thrashing, view-selector consolidation. - [js-profile-react.md](js-profile-react.md) — prove the re-render reduction. diff --git a/domains/performance/skills/performance/references/mm-state-normalization.md b/domains/performance/skills/performance/references/mm-state-normalization.md new file mode 100644 index 00000000..dc59b349 --- /dev/null +++ b/domains/performance/skills/performance/references/mm-state-normalization.md @@ -0,0 +1,136 @@ +--- +title: State Normalization & Selector Shape (MetaMask) +impact: HIGH +tags: redux, normalization, selectors, O(1)-lookups, cache-thrashing, useSelector +--- + +# Skill: State Normalization & Selector Shape + +> **Scope.** The generic form of the O(n)-lookup problem is `selector-anti-patterns` §7, +> in the knowledge file installed alongside this skill under `knowledge/`. This file is the +> MetaMask Mobile instance, plus the parameterized-selector cache-thrashing and +> view-selector consolidation work that is specific to this store's shape. + +Selector *memoization* fixes when things recompute; state and selector **shape** fixes how much each recomputation costs and how many subscriptions fire. The patterns here come from the extension performance audit (internal audit tickets), where linear scans and reshaping selectors multiplied across power-user data: with 1,000 tokens, 27 `.find()`-based lookups per render is 27,000 comparisons — per render. + +## Pattern — O(n) scans where the state shape should provide O(1) lookups + +```ts +// ❌ linear scan through all accounts on every call +export const getAccountByAddress = createSelector( + selectAccounts, + (_, address) => address, + (accounts, address) => + Object.values(accounts).find((a) => a.address.toLowerCase() === address.toLowerCase()), +); +``` + +When lookups by some key are frequent, **index the state once** instead of scanning per consumer: + +```ts +// ✅ build the index once per data change; lookups are O(1) key access +export const selectAccountsByAddress = createSelector(selectAccounts, (accounts) => + Object.fromEntries(Object.values(accounts).map((a) => [a.address.toLowerCase(), a])), +); +// consumers key into the memoized index — no scan, no per-arg selector cache to bust +const account = useSelector(selectAccountsByAddress)[address.toLowerCase()]; +``` + +Normalized shape (`byId` / `byAddress` maps + an `ids` array for order) is the same idea applied at the reducer level — the index is maintained on write instead of derived on read. + +## Pattern — parameterized selector cache thrashing + +`createSelector` has a **single-entry cache**. A parameterized selector called with different arguments from different components busts that one cache slot on every call: + +```ts +// ❌ each component's call evicts the previous component's result +const a1 = useSelector((s) => getAccountByAddress(s, addr1)); // miss +const a2 = useSelector((s) => getAccountByAddress(s, addr2)); // miss, evicts addr1 +const a3 = useSelector((s) => getAccountByAddress(s, addr3)); // miss, evicts addr2 — and so on every render cycle +``` + +In a list rendering N rows, the "memoized" selector recomputes N times per render, forever. **Check the memoizer before flagging:** this codebase already uses `weakMapMemoize` for some parameterized selectors (e.g. `selectNetworkConfigurationByChainId`), which caches per-argument and doesn't thrash — but only for *stable* arguments. A fresh **object literal** argument per call (`selectAsset(state, { address, chainId, isStaked })`) defeats `weakMapMemoize` too: every call is a new WeakMap key. Fixes, in order of preference: + +1. **Lookup-map selector** (above): select the whole memoized index once; key into it. Sidesteps per-arg caching entirely. +2. **Per-instance selector**: a factory (`makeSelectAccountByAddress()`) instantiated in the component with `useMemo`, so each call site owns its own cache slot. +3. **Bigger cache**: reselect's `lruMemoize` with `maxSize: N` — last resort; sizing is a guess that goes stale. + +```bash +# parameterized selectors: second input selector reads the argument, not state +grep -rn "(_, \|(_state" app/selectors --include="*.ts" +``` + +## Pattern — selectors that reorganize nested state + +```ts +// ❌ inverts { account → chain → tokens } into { chain → account → tokens } on every recompute +export const getTokensByChain = createSelector(selectAllTokens, (byAccount) => { + const byChain = {}; + for (const [account, chains] of Object.entries(byAccount)) + for (const [chainId, tokens] of Object.entries(chains)) + (byChain[chainId] ??= {})[account] = tokens; + return byChain; +}); +``` + +A full restructure allocates a new tree every recomputation — expensive to build, and every consumer sees a fresh reference. If two access patterns are both hot, **store both shapes** (maintain the second index in the reducer on write) or normalize so both reads are key lookups. A reshaping selector is acceptable only for cold paths. + +## Pattern — deep property access instead of composed input selectors + +```ts +// ❌ re-derives the full path; recomputes when ANY ancestor changes; nothing is reusable +export const getGroupName = (state, walletId, groupId) => + state.engine.accountTree.wallets[walletId]?.groups[groupId]?.metadata?.name; +``` + +Compose granular selectors at each level (`selectWallets` → `selectWalletById` → `selectGroupById` → …). Each layer memoizes independently, intermediate results are reusable by other selectors, and a change to one wallet no longer recomputes selectors reading a different one. This is also what keeps inputs *narrow* — the prerequisite for the memoization patterns in [mm-selector-memoization.md](mm-selector-memoization.md). + +## Pattern — many useSelector calls where one view selector should exist + +```tsx +// ❌ 11 store subscriptions; each runs on every dispatch; component re-checks 11 results +const quotes = useSelector(getQuotes); +const currency = useSelector(getCurrentCurrency); +const gasFee = useSelector(getGasFee); +// … ×8 more +``` + +Each `useSelector` is an independent store subscription with its own equality check per store notification. **Check the dispatch cadence before flagging count alone:** in this codebase, controller state changes batch into a 250ms flush (`app/core/Batcher`, `EngineService`'s `updateBatcher`) and dispatch inside `unstable_batchedUpdates`, so checks run at most a few times per second and React renders once per flush — N cheap accessor reads are *not* a problem. The actionable findings inside a high-count component are the **expensive** selectors (cost paid on every check) and the **unstable-ref** selectors (a re-render per flush) — triage and fix those individually first. + +Audit calibration (this codebase, 2026-06): a per-selector triage of the 10 highest-count components (9-19 reads each) ruled out ~90% of reads — feature-flag booleans, primitive accessors, and correctly `useMemo`'d factory selectors. The real findings were per-row parameterized selectors and deep-equal selectors over power-user-scaled data. The count was noise; the triage found what mattered. + +Consolidating related reads into **one memoized view selector** still earns its keep in two cases: a component repeated per row (per-row × per-flush multiplication of any expensive check), and derivation logic that would otherwise sit unmemoized in the component (where the React Compiler can't stabilize it — see [mm-selector-cascade.md](mm-selector-cascade.md)). One subscription, one equality check, one place where the shape is defined. + +The same consolidation applies to **duplicate derived-data implementations**: the extension audit found 4+ independent fiat-conversion code paths recomputing the same numbers in different components. One canonical selector ends both the wasted compute and the drift between implementations. + +## How to find + +```bash +# linear scans inside selectors/hooks +grep -rn "Object.values(.*)\.\(find\|filter\)\|\.find((" app/selectors app/components --include="*.ts*" | grep -v ".test." + +# reshaping selectors: nested loops/reduce building objects in a result function +grep -rn -B2 "??= {}\|reduce((acc" app/selectors --include="*.ts" + +# components with many subscriptions — triage the N selectors for cost/stability, don't flag the count itself +grep -rc "useSelector(" app/components --include="*.tsx" | awk -F: '$2>=5' | sort -t: -k2 -rn | head -20 +``` + +## Verify + +- Lookup fix: recomputation count on the index selector is ~1 per data change (not per render); list scroll/render time drops in the Profiler. +- Consolidation: the component's "why did this render" shows one subscription firing instead of N; render count per dispatch drops. +- Normalization: reducer tests confirm both shapes stay in sync on write. + +## Don't over-correct + +- Don't normalize a slice that's only ever iterated in full — indexes pay for themselves on *keyed lookups*, not on `.map()` over everything. +- Don't merge *unrelated* selectors into one mega view selector — that re-couples components to data they don't read and re-renders them for it. Consolidate related values consumed together. +- Don't flag a component for its `useSelector` **count** — with batched controller sync (250ms flush + `unstable_batchedUpdates`), N cheap subscriptions are noise. Flag the expensive or unstable selectors *among* them. +- `maxSize`/factory-selector machinery is for genuinely parameterized hot paths; for one or two call sites the lookup-map pattern is simpler and stays correct. + +## Related + +- [mm-selector-memoization.md](mm-selector-memoization.md) — memoization correctness for the selectors shaped here +- [mm-selector-cascade.md](mm-selector-cascade.md) — graph-level repair when a root selector poisons consumers +- [mm-redux-antipatterns.md](mm-redux-antipatterns.md) — inline selectors and `isEqual` band-aids diff --git a/domains/performance/skills/performance/references/mm-tools.md b/domains/performance/skills/performance/references/mm-tools.md index e7e7d2df..5de42764 100644 --- a/domains/performance/skills/performance/references/mm-tools.md +++ b/domains/performance/skills/performance/references/mm-tools.md @@ -84,6 +84,9 @@ Then read what's emitted — a load log (e.g. `source: 'cache' | 'fresh_fetch'`, "Components re-render too much" → React Native DevTools → "why did this render?" → mm-selector-memoization.md / mm-redux-antipatterns.md + → or WDYR (wired at wdyr.js, tracks useSelector diffs): ENABLE_WHY_DID_YOU_RENDER=true yarn start + — logs consumers re-rendering on same-values/new-reference; ideal for tracing a selector cascade + → mm-selector-cascade.md "Search/filter input lags while typing" → js-concurrent-react.md (useDeferredValue) — and memo() the expensive child @@ -95,8 +98,10 @@ Then read what's emitted — a load log (e.g. `source: 'cache' | 'fresh_fetch'`, "A hook/component re-renders the whole list even though children are memoized" → mm-unstable-hook-return.md (a hook returns a new array/object ref every render → defeats downstream memo) -"I can't see network calls on Android" - → Reactotron (DevTools network tab doesn't work on Android) + +"Is this slow flow data-bound or render-bound?" + → Network panel for request timings → js-network-panel.md + → Performance panel for the full React+JS+network timeline → js-performance-panel.md "Memory grows over a session / crashes after long use" → js-memory-leaks.md (JS) or native-memory-leaks.md (native) @@ -129,15 +134,17 @@ Then read what's emitted — a load log (e.g. `source: 'cache' | 'fresh_fetch'`, - **Interpret:** JS drops → expensive renders/selectors/computation. UI drops → native rendering/animation. Both → start JS-side. - **Next:** React Native DevTools. -### React Native DevTools (re-renders, timing, memory) — iOS + Android +### React Native DevTools (re-renders, timing, memory, performance) — iOS + Android - **Open:** press `j` in Metro, or shake → "Open DevTools". Hermes is on for both platforms, so this works everywhere. - **Profiler:** ⚙️ → enable "Record why each component rendered" → Start → reproduce the **exact** interaction → Stop. - **Read:** flamegraph (yellow = slow), Ranked view (slowest first), right panel "why did this render?" (props/hook/parent). - **Next:** props churn → `useCallback`/memo; selector new-ref → [mm-selector-memoization.md](mm-selector-memoization.md); parent re-render → move state down / [mm-context-performance.md](mm-context-performance.md). - **JS CPU:** the JavaScript Profiler tab → Heavy (Bottom-Up) for non-React hot functions. +- **Performance panel:** Performance tab → Record → run the flow → Stop. Shows React scheduler phases, the Components flamegraph, the JS Thread, and Network events on one timeline. Use it when the Profiler alone doesn't explain a slow flow. See [js-performance-panel.md](js-performance-panel.md). +- **Network panel:** Network tab → records `fetch()`/`XHR`/`<Image>` automatically. Timings, headers, response previews, and an Initiator call stack. Tells you if a slow screen is data-bound vs render-bound. See [js-network-panel.md](js-network-panel.md). -### Reactotron (network on Android) -- Network inspection when the DevTools network tab is unavailable on Android. +### Reactotron (network on Android — pre-0.83 fallback) +- Network inspection for **pre-0.83** setups where the DevTools Network tab is unavailable on Android. On RN 0.83+ prefer the DevTools [Network panel](js-network-panel.md) (works on Android). WebSocket events still aren't captured by the Network panel — Reactotron remains useful there. ### Flashlight (optional, external — NOT installed in this repo) - **Not in `package.json`** — it's an external Callstack tool. Don't assume it's available; for day-to-day FPS use Perf Monitor + RN DevTools. @@ -162,6 +169,7 @@ endTrace({ name: TraceName.AssetDetails }); // end const x = trace({ name: TraceName.Tokens, op: TraceOperation.UIStartup }, () => build()); ``` - New flow → add a `TraceName` (+ `TraceOperation`) to `app/util/trace.ts`, then wrap it. +- **Quota guardrail:** never start a span per list item, per row, or per poll tick — span volume multiplies by data size × user count. A high-frequency span needs a deterministic sub-sample gate (and a kill-switch) before it ships. - **Component-level: use a per-feature measurement hook, not raw `trace()`.** The repo convention is a declarative `useXMeasurement` hook (e.g. `app/components/UI/Predict/hooks/usePredictMeasurement.ts`, `usePerpsMeasurement`, `useSectionPerformance`) that starts on mount and ends when conditions are true — which structurally enforces the "end on data-loaded, not mount" rule below: ```ts usePredictMeasurement({ traceName: TraceName.PredictMarketDetailsView, conditions: [dataLoaded, !isLoading] }); @@ -217,4 +225,4 @@ A code session often has no running device, so "Measure first" isn't literally p ## Related -- [mm-power-user-scenario.md](mm-power-user-scenario.md) · [js-measure-fps.md](js-measure-fps.md) · [js-profile-react.md](js-profile-react.md) · [native-measure-tti.md](native-measure-tti.md) +- [mm-power-user-scenario.md](mm-power-user-scenario.md) · [js-measure-fps.md](js-measure-fps.md) · [js-profile-react.md](js-profile-react.md) · [js-performance-panel.md](js-performance-panel.md) · [js-network-panel.md](js-network-panel.md) · [native-measure-tti.md](native-measure-tti.md) diff --git a/domains/performance/skills/performance/references/mm-useeffect-antipatterns.md b/domains/performance/skills/performance/references/mm-useeffect-antipatterns.md new file mode 100644 index 00000000..f9eb1fd8 --- /dev/null +++ b/domains/performance/skills/performance/references/mm-useeffect-antipatterns.md @@ -0,0 +1,149 @@ +--- +title: useEffect Lifecycle Anti-Patterns (MetaMask) +impact: HIGH +tags: useEffect, setState, cleanup, AbortController, unmount, memory-leaks +--- + +# Skill: useEffect Lifecycle Anti-Patterns + +> **Scope.** The platform-agnostic taxonomy — unstable dependency identity, wrong +> dependencies, derived state via effect, cascading effect chains, missing timer cleanup, +> uncancelled async — is the single source in the **`effect-anti-patterns`** knowledge file, +> installed alongside this skill under `knowledge/`. This file is the MetaMask Mobile +> instance of its lifecycle half: the verified instances, the repo's own idioms, and the +> fix recipes. + +[mm-hook-dependency-arrays.md](mm-hook-dependency-arrays.md) covers *when* effects re-run (the deps side). This file covers what goes wrong **inside and after** the effect: state derived in effects instead of render, effects chained off each other's setState, async work that outlives the component, and missing cleanup. These patterns cause extra render passes, memory leaks, and the classic "setState on unmounted component" warnings — and they're invisible to selector/re-render sweeps. + +## Pattern — derived state via useEffect + setState ("you might not need an effect") + +```tsx +// ❌ two render passes per change: render → effect → setState → render again +const [visibleTokens, setVisibleTokens] = useState([]); +useEffect(() => { + setVisibleTokens(tokens.filter((t) => !t.hidden)); +}, [tokens]); + +// ✅ derive during render — one pass, no state to drift out of sync +const visibleTokens = useMemo(() => tokens.filter((t) => !t.hidden), [tokens]); +``` + +If a value is computable from props/state/store, compute it in render (memoize only if it's expensive or feeds a memoized child). State + effect is for *synchronizing with something external*, not for derivation. + +## Pattern — cascading effect chains + +```tsx +// ❌ effect A sets state → triggers effect B → sets state → triggers effect C… +useEffect(() => { setAccount(deriveAccount(accounts, selected)); }, [accounts, selected]); +useEffect(() => { setBalances(deriveBalances(account)); }, [account]); +useEffect(() => { setFiat(deriveFiat(balances, rate)); }, [balances, rate]); +// 4 render passes for one upstream change, and the intermediate renders show stale combinations +``` + +**Fix:** collapse the chain into render-time derivation (one `useMemo` per step, or one for the lot). Each link in a setState-chain is a full extra render pass *and* a window where the UI shows an inconsistent intermediate state. + +## Pattern — async work that outlives the component + +```tsx +// ❌ fetch resolves after unmount (or after the input changed) → setState on dead component / stale data wins +useEffect(() => { + fetchTokenMetadata(address).then((meta) => setMetadata(meta)); +}, [address]); +``` + +Two equivalent fixes — pick one and use it consistently: + +```tsx +// ✅ cancelled flag — cheapest, works for any promise +useEffect(() => { + let cancelled = false; + fetchTokenMetadata(address).then((meta) => { + if (!cancelled) setMetadata(meta); + }); + return () => { cancelled = true; }; +}, [address]); + +// ✅ AbortController — also cancels the network request itself (RN fetch supports `signal`) +useEffect(() => { + const controller = new AbortController(); + fetch(url, { signal: controller.signal }) + .then((r) => r.json()) + .then(setData) + .catch((e) => { if (e.name !== 'AbortError') setError(e); }); + return () => controller.abort(); +}, [url]); +``` + +The cancelled flag prevents the *setState*; AbortController additionally stops the request from consuming bandwidth/battery. The race-condition variant (stale response overwriting fresh data when `address` changes quickly) is fixed by the same cleanup — the old effect's closure is cancelled before the new one runs. + +**Codify, don't copy-paste** (internal extension epic): once a repo has three hand-rolled cancelled flags, extract shared hooks — `useIsMounted()`, `useAbortableEffect(fn, deps)` (effect receives a signal), `useEventListener(target, event, handler)` (auto-removes on unmount) — so cleanup is the default, not per-site diligence. + +## Pattern — missing cleanup for timers / subscriptions / listeners + +```tsx +// ❌ each mount adds another interval/listener; none are removed +useEffect(() => { + const id = setInterval(refreshGasEstimate, 15000); + emitter.on('update', onUpdate); +}, []); + +// ✅ every subscription returns its teardown +useEffect(() => { + const id = setInterval(refreshGasEstimate, 15000); + emitter.on('update', onUpdate); + return () => { clearInterval(id); emitter.off('update', onUpdate); }; +}, []); +``` + +Leaked intervals keep firing (and keep dispatching) forever; leaked listeners hold the closure — and everything it captured — out of garbage collection. See [js-memory-leaks.md](js-memory-leaks.md) for hunting these in a running app, and [mm-streaming-realtime.md](mm-streaming-realtime.md) for subscription lifecycles tied to visibility. + +## Pattern — regular variable where a ref is needed + +```tsx +// ❌ reset to false on every render — the guard never works +let hasLoggedImpression = false; +useEffect(() => { + if (!hasLoggedImpression) { logImpression(); hasLoggedImpression = true; } +}); + +// ✅ useRef persists across renders without triggering them +const hasLoggedImpression = useRef(false); +``` + +Any mutable flag/cache/previous-value that must survive re-renders but shouldn't cause them belongs in a ref, not a closure variable (and not state). + +## Pattern — large objects captured in effect closures + +An effect (or its cleanup) that closes over a large object — full token lists, raw API payloads — pins that object in memory for as long as the subscription lives. Extract the fields you need into locals *before* the closure, or read through a ref, so the big object can be collected. + +## How to find + +```bash +# setState-from-effect derivation candidates (review hits — some are legitimate syncs) +grep -rn -A3 "useEffect(" app --include="*.tsx" | grep -B1 "set[A-Z]" | grep -v ".test." + +# fetch/promises in effects with no signal/cancelled handling nearby +grep -rn -A6 "useEffect(" app --include="*.ts*" | grep -E "fetch\(|\.then\(" | grep -v "signal\|cancelled\|abort" | grep -v ".test." + +# intervals/timeouts/listeners inside effects — then eyeball for a `return () =>` teardown +grep -rn "setInterval\|setTimeout\|addEventListener\|\.on(" app --include="*.ts*" | grep -v ".test." | grep -v "clear\|remove\|off(" +``` + +## Verify + +- React DevTools highlight-updates: the derive-in-render fix removes the double render pass on the affected component. +- No "setState on unmounted component" / no stale-data flash when rapidly switching the input (account/network) that drives the effect. +- For cleanup fixes: navigate to the screen and back N times → timer/listener count stays flat (see [js-memory-leaks.md](js-memory-leaks.md)). + +## Don't over-correct + +- Effects that *synchronize with external systems* (subscriptions, navigation, imperative APIs) are the legitimate use — don't mechanically rewrite every effect as `useMemo`. +- An async effect whose component provably never unmounts mid-flight (e.g. root-level, app lifetime) doesn't need a cancelled flag — but say so in review rather than assuming. +- Don't wrap trivial derivations in `useMemo` while de-effecting — plain expressions are fine until profiling or a memoized child says otherwise. + +## Related + +- [mm-hook-dependency-arrays.md](mm-hook-dependency-arrays.md) — the deps side: JSON.stringify, inline literals, stale closures +- [js-memory-leaks.md](js-memory-leaks.md) — measuring leaks the missing cleanups cause +- [mm-streaming-realtime.md](mm-streaming-realtime.md) — subscription setup/teardown for real-time screens +- [mm-unstable-hook-return.md](mm-unstable-hook-return.md) — unstable hook returns that make effects re-run diff --git a/domains/performance/skills/performance/repos/metamask-mobile.md b/domains/performance/skills/performance/repos/metamask-mobile.md index d077f7df..e136589d 100644 --- a/domains/performance/skills/performance/repos/metamask-mobile.md +++ b/domains/performance/skills/performance/repos/metamask-mobile.md @@ -54,6 +54,9 @@ Always pair measurement with the **power-user scenario on Android** — see [ref | `useSelector` returns new refs; `useSelector(x, isEqual)` band-aids | [mm-redux-antipatterns.md](references/mm-redux-antipatterns.md) | | Whole subtree re-renders under a Context provider | [mm-context-performance.md](references/mm-context-performance.md) | | `useEffect`/`useMemo` re-runs constantly; `JSON.stringify` in deps | [mm-hook-dependency-arrays.md](references/mm-hook-dependency-arrays.md) | +| Effect chains (`setState` in effect triggers next effect); setState after unmount; missing timer/listener cleanup | [mm-useeffect-antipatterns.md](references/mm-useeffect-antipatterns.md) | +| **One selector change re-renders half the app**; `isEqual`/`createDeepEqualSelector` band-aids accumulating downstream | [mm-selector-cascade.md](references/mm-selector-cascade.md) | +| O(n) `.find()` scans per render; parameterized selector recomputes for every list row; component with 5+ `useSelector` calls | [mm-state-normalization.md](references/mm-state-normalization.md) | | Animation janky; `useNativeDriver: false` on width/height | [mm-layout-animations.md](references/mm-layout-animations.md) → [js-animations-reanimated.md](references/js-animations-reanimated.md) | | List scroll jank / unbounded list | [js-lists-flatlist-flashlist.md](references/js-lists-flatlist-flashlist.md) | | Search/filter input blocks typing | [js-concurrent-react.md](references/js-concurrent-react.md) | @@ -61,12 +64,15 @@ Always pair measurement with the **power-user scenario on Android** — see [ref | **Real-time / websocket screen slow or janky** (prices, order book, live balances); slow only on first-open / after backgrounding | [mm-streaming-realtime.md](references/mm-streaming-realtime.md) | | **List re-renders fully even though children are memoized** (a hook returns a new array/object every render) | [mm-unstable-hook-return.md](references/mm-unstable-hook-return.md) | | FPS drops; want to localize JS vs UI thread | [js-measure-fps.md](references/js-measure-fps.md) → [js-profile-react.md](references/js-profile-react.md) | +| Need a full timeline (React scheduler + JS + network) for a slow flow, not just re-renders | [js-performance-panel.md](references/js-performance-panel.md) | +| Inspect network requests / API timings; is a slow screen data-bound or render-bound? | [js-network-panel.md](references/js-network-panel.md) | | Memory grows over a session | [js-memory-leaks.md](references/js-memory-leaks.md) / [native-memory-leaks.md](references/native-memory-leaks.md) | | Slow startup (TTI) | [native-measure-tti.md](references/native-measure-tti.md) → [bundle-analyze-js.md](references/bundle-analyze-js.md) | | Bundle too big / barrel imports / heavy lib | [bundle-barrel-exports.md](references/bundle-barrel-exports.md) → [bundle-analyze-js.md](references/bundle-analyze-js.md) → [bundle-library-size.md](references/bundle-library-size.md) | | Native module / sync method blocking JS | [native-sdks-over-polyfills.md](references/native-sdks-over-polyfills.md) | | Native lib crashes on 16KB-page Android | [native-android-16kb-alignment.md](references/native-android-16kb-alignment.md) | | Enable automatic memoization | [mm-react-compiler.md](references/mm-react-compiler.md) → [js-react-compiler.md](references/js-react-compiler.md) | +| Compiler is enabled but a component shows no `Memo ✨`; compiler errors in build output — which are real? | [mm-react-compiler-error-triage.md](references/mm-react-compiler-error-triage.md) | ## Verified anti-pattern catalogue (this codebase) @@ -86,6 +92,8 @@ Ordered by impact. Each links to the guide with the fix. **The `Where` column li | High | lodash main-package imports (98 files, no tree-shaking) | 98 files | [bundle-library-size.md](references/bundle-library-size.md) | | High | FlatList missing perf props on growing lists | 65 FlatList JSX | [js-lists-flatlist-flashlist.md](references/js-lists-flatlist-flashlist.md) | | High | AppState listener without cleanup | `app/core/SDKConnectV2/services/connection-registry.ts:487` | [js-memory-leaks.md](references/js-memory-leaks.md) | +| High | Parameterized selector (single-entry cache, busted per arg) doing an O(n) `Object.values().flat().find()` scan per call | `selectSingleTokenByAddressAndChainId` `app/selectors/tokensController.ts:174`; also `app/selectors/assets/assets-list.ts`, `app/selectors/moneyAccountController/index.ts` | [mm-state-normalization.md](references/mm-state-normalization.md) | +| Medium | Async effect without cancellation; setState-chain effects; derived state via useEffect+setState | feature-specific — run the guide's greps | [mm-useeffect-antipatterns.md](references/mm-useeffect-antipatterns.md) | | Medium | Inline `useSelector(state => state.x)` bypassing named selectors | 3 files | [mm-redux-antipatterns.md](references/mm-redux-antipatterns.md) | | Medium | Lottie where Rive fits (Rive already installed) | 5 files | [js-animations-reanimated.md](references/js-animations-reanimated.md) | | Low | dayjs + luxon both present (dedup) | 4 + 6 files | [bundle-library-size.md](references/bundle-library-size.md) | @@ -101,4 +109,4 @@ Ordered by impact. Each links to the guide with the fix. **The `Where` column li ## Attribution -Generic React Native references (`js-*`, `native-*`, `bundle-*`) adapted from "The Ultimate Guide to React Native Optimization" by Callstack. MetaMask-specific guidance (`mm-*`) from the internal Performance Guide for Engineers and verified codebase audits. +Generic React Native references (`js-*`, `native-*`, `bundle-*`) adapted from "The Ultimate Guide to React Native Optimization" by Callstack. MetaMask-specific guidance (`mm-*`) from the internal Performance Guide for Engineers and verified codebase audits. Cross-platform React/Redux guidance (`mm-selector-cascade`, `mm-useeffect-antipatterns`, `mm-state-normalization`, `mm-react-compiler-error-triage`) adapted from MetaMask contributor-docs [`frontend-performance.md`](https://github.com/MetaMask/contributor-docs/blob/main/docs/frontend-performance.md) and the extension performance audit (extension PRs metamask-extension#38007, metamask-extension#37147). diff --git a/domains/performance/skills/react-render-proof/skill.md b/domains/performance/skills/react-render-proof/skill.md new file mode 100644 index 00000000..4cb4fa2c --- /dev/null +++ b/domains/performance/skills/react-render-proof/skill.md @@ -0,0 +1,112 @@ +--- +name: react-render-proof +description: Prove a React rendering or memoization change actually reduced work, with a delivery gate and a reported band. Covers re-render counts (why-did-you-render), selector recomputes (reselect's real `.recomputations()` API), and A/B arms toggled at a FIXED commit rather than across a merge boundary. The falsifier is an arm whose treatment never reached the built bundle — a null from undelivered treatment is indistinguishable from a null from a small effect and reports as the second. Triggers on /react-render-proof, or when asked to prove a component stopped over-rendering, measure selector recomputation, validate a memoization/React Compiler change, run a render-count A/B, or interpret a re-render benchmark. Callable by pr-validate as the engine behind its React render & selector proof evidence category. +maturity: experimental +--- + +# /react-render-proof + +A render-count number is worthless until two things are true: the **treatment reached the +artifact the browser executes**, and the number is reported as a **band** rather than a point. +Most of this skill is those two checks. The measurement itself is easy; the failure mode is +reporting a difference between arms that never differed. + +> **Falsifier.** An arm whose manipulation cannot be observed in the built bundle. If you +> cannot point at output that differs *in kind* between arms — a symbol present in one and +> absent in the other, a flag line in a log — the A/B is not designed yet, and any delta it +> produces is noise with a story attached. + +## Method + +1. **Name the delivery check before building arms, and verify it emits.** State how the run + itself will show the arms differ, then confirm that output exists on one real build before + scaling to N repeats. This ordering is the whole skill. A measurement launched before the + instrument is proven emits a null that reads exactly like "no effect". + +2. **Derive the needle from output at the stage you will grep, not one stage upstream.** + Compiler and bundler output are not the same text. Worked failures, both real: + - React Compiler at `target: '17'` emits `react-compiler-runtime`; at `target: '19'` it + emits `react/compiler-runtime`. Grepping for the wrong one returns 0 in *both* arms and + fails the arm that actually got the treatment. + - `_c(` is the form **babel** emits. Metro transforms it further — in a real 119 MB React + Native bundle it scored 13 hits, every one a minified vendor identifier + (`function _c(e,t){return e|t}`), while the true compiler output went uncounted. The form + that survived metro was `memo_cache_sentinel` (4681 in the treated arm vs 166 in the + control). Same needle, two bundlers, two different answers. + + Compile one real file through the project's own config and read the output. Ten minutes + here saves a whole run. + +3. **A name is not a witness — count what only exists when the module is included.** A bare + module specifier appears in bundled `package.json` dependency lists whether or not the + module was ever pulled in; a clean control arm scored exactly 1 that way and was wrongly + failed. Gate on artifacts that cannot appear otherwise (a runtime sentinel, a compiled call + site). Keep the specifier count as a diagnostic — 3081-vs-1 is informative, it just isn't a + boolean. + +4. **Use the library's real counter before injecting your own.** `reselect` exposes + **`.recomputations()`** on memoized selectors — a genuine API, not a patch. Read it (sample + on an interval if the count should visibly climb). An injected `console.log` you added to a + selector body is an authored claim, not an observation; reach for it only when no real API + exists, and say so when you do. *(Note: pr-validate's C4 entry long claimed there was "no + built-in selector-call counter". There is.)* + +5. **Toggle at a fixed commit, not across a merge boundary.** Same tree in both arms, one + thing different. A commit boundary drags in unrelated change you will then be unable to + exclude. When the real commit bundles two changes (a scope change *and* a version bump), + reproduce only the one under test — moving both reintroduces the confound the fixed-commit + design exists to remove, and can silently flip your delivery needle mid-experiment. + +6. **Repeat the capture, not the build; report the band.** Counts vary run to run — one + baseline measured 153/164/224 across three runs. The build dominates cost (~6 min vs ~90 s + per capture), so repeats are nearly free. Publish one artifact; report every repeat's count. + +7. **When the delta is under the spread, say "not resolvable at this n" and give the MDE.** + Not "no effect". State the smallest detectable effect and what n would resolve the observed + difference. A real worked result: 112–128 vs 115–133, delta 4.6%, t=1.33 — with delivery + proven (1244 compiled sites vs 0), so the null was about effect size, not plumbing. + +8. **A check that finds nothing needs a positive control.** Before believing a zero, confirm + the same check finds something it should. A search that returned "0 references" looked like + confirmation until searching for a string known to be present *also* returned 0 — the index + didn't reach that content and the zero meant nothing. + +## Gates, in order + +| gate | asserts | on failure | +|---|---|---| +| source manipulation | the intended edit applied, and *only* it | abort the arm | +| **delivery** | the change reached the built bundle | abort **before any capture** | +| metric | the instrument emitted a non-zero count on capture 1 | abort before spending repeats | + +Each catches what the previous cannot. Source changing is not delivery; delivery is not the +instrument working. Wire them as script-level aborts so a broken arm cannot report a number — +"refusing to emit a render count from an arm whose treatment is unproven" is the correct +output, and it is not a failure of the run. + +**Never relax a gate to make an arm pass.** When a gate fires, go read the artifact and find +the mechanism first. Loosening is the work-reducing direction, which is exactly where scrutiny +collapses. Demoting a needle from gate to diagnostic *after* proving it fires for an unrelated +reason is legitimate; doing it because the arm failed is not. + +## What the count does and does not mean + +WDYR counts **every** re-render in the measured window, including boot settling — not only the +cascade a given fix targeted. So an RCA predicting "→ 0 re-renders" for a specific cascade is +not refuted by a non-zero WDYR total. Say which quantity you measured, and don't let a global +counter stand in for a scoped claim. + +Global application does not imply a large effect: 1244 auto-memoized call sites moved one +interaction's re-render count under 5%. Reach for a flow the change plausibly dominates, and +treat a single flow as a lower bound on reach, not a summary of it. + +## Caveats to publish with the number + +Fixture parity (structurally matched vs byte-identical), arm ordering (randomized or not), +what window the counter covers, and how many flows were measured. State them; they are cheap +and their absence is what makes a number unfalsifiable. + +## Related + +- `pr-validate` — packages this skill's output as its C4 evidence category. +- `memory-leak-hunt`, `supply-chain-audit` — sibling engines behind other categories. From ed3d6ebea516e3d65cd95dd1f0e3e4fec01bdd82 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 08:41:34 -0400 Subject: [PATCH 024/135] Restore the internal planning-ticket references MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit These name MetaMask-org planning epics and audit tickets. The audience for this repo is the MetaMask org, for whom those identifiers are load-bearing context — they are where the guidance came from and where the follow-up lives. The scrub line is personal references, not org-internal ones. --- .../performance/references/mm-react-compiler-error-triage.md | 4 ++-- .../skills/performance/references/mm-selector-cascade.md | 2 +- .../skills/performance/references/mm-state-normalization.md | 2 +- .../performance/references/mm-useeffect-antipatterns.md | 2 +- .../performance/skills/performance/repos/metamask-mobile.md | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/domains/performance/skills/performance/references/mm-react-compiler-error-triage.md b/domains/performance/skills/performance/references/mm-react-compiler-error-triage.md index e726279d..68db17f8 100644 --- a/domains/performance/skills/performance/references/mm-react-compiler-error-triage.md +++ b/domains/performance/skills/performance/references/mm-react-compiler-error-triage.md @@ -18,7 +18,7 @@ The React Compiler **fails open**: when it can't compile a component, it silentl | `'critical_errors'` | only critical errors (compiler-internal invariant violations) | CI / debug builds, first ratchet target | | `'all_errors'` | every diagnostic, including unsupported syntax | CI / debug builds, end-state ratchet | -**The ratchet strategy** (extension roadmap, tracked internally): keep production at `'none'` permanently; aim for a *non-production* build that passes at `'critical_errors'`, then at `'all_errors'`. Each ratchet step turns a class of silent skips into a visible, fixable error list. Never enable a non-`'none'` threshold in a release build — one un-compilable file would block the release for an optimization that is optional by design. +**The ratchet strategy** (extension roadmap, MetaMask-planning#6552 → #6553): keep production at `'none'` permanently; aim for a *non-production* build that passes at `'critical_errors'`, then at `'all_errors'`. Each ratchet step turns a class of silent skips into a visible, fixable error list. Never enable a non-`'none'` threshold in a release build — one un-compilable file would block the release for an optimization that is optional by design. ## Triage: unsupported syntax vs. legitimate errors @@ -63,7 +63,7 @@ What the buckets tell you: ## Staged adoption roadmap -The extension's sequence (internal epic) generalizes to any repo: +The extension's sequence (epic MetaMask-planning#6549) generalizes to any repo: 1. **Lint clean:** update `eslint-plugin-react-hooks` / `eslint-plugin-react-compiler` to latest; fix violations — these are exactly what the compiler will refuse to compile. 2. **Audit opt-outs:** every `'use no memo'` carries a reason + TODO; the count only goes down. `grep -rn "use no memo" app --include="*.ts*"`. diff --git a/domains/performance/skills/performance/references/mm-selector-cascade.md b/domains/performance/skills/performance/references/mm-selector-cascade.md index 403f9e5c..387e9cbe 100644 --- a/domains/performance/skills/performance/references/mm-selector-cascade.md +++ b/domains/performance/skills/performance/references/mm-selector-cascade.md @@ -89,7 +89,7 @@ For each hit that consumes the fixed root (directly or transitively): remove the ## What the React Compiler can and cannot do here -The compiler memoizes **within a file**. A `useSelector` result, an imported hook's return value, or an external context value is opaque to it — if the selector hands back a fresh reference, the compiled component still re-renders, and any derivation from it still recomputes (internal audit ticket): +The compiler memoizes **within a file**. A `useSelector` result, an imported hook's return value, or an external context value is opaque to it — if the selector hands back a fresh reference, the compiled component still re-renders, and any derivation from it still recomputes (extension audit ticket MetaMask-planning#6661): ```tsx const tokens = useSelector(selectTokens); // compiler cannot see/stabilize this diff --git a/domains/performance/skills/performance/references/mm-state-normalization.md b/domains/performance/skills/performance/references/mm-state-normalization.md index dc59b349..6a8b0bc1 100644 --- a/domains/performance/skills/performance/references/mm-state-normalization.md +++ b/domains/performance/skills/performance/references/mm-state-normalization.md @@ -11,7 +11,7 @@ tags: redux, normalization, selectors, O(1)-lookups, cache-thrashing, useSelecto > MetaMask Mobile instance, plus the parameterized-selector cache-thrashing and > view-selector consolidation work that is specific to this store's shape. -Selector *memoization* fixes when things recompute; state and selector **shape** fixes how much each recomputation costs and how many subscriptions fire. The patterns here come from the extension performance audit (internal audit tickets), where linear scans and reshaping selectors multiplied across power-user data: with 1,000 tokens, 27 `.find()`-based lookups per render is 27,000 comparisons — per render. +Selector *memoization* fixes when things recompute; state and selector **shape** fixes how much each recomputation costs and how many subscriptions fire. The patterns here come from the extension performance audit (MetaMask-planning#6580, #6484), where linear scans and reshaping selectors multiplied across power-user data: with 1,000 tokens, 27 `.find()`-based lookups per render is 27,000 comparisons — per render. ## Pattern — O(n) scans where the state shape should provide O(1) lookups diff --git a/domains/performance/skills/performance/references/mm-useeffect-antipatterns.md b/domains/performance/skills/performance/references/mm-useeffect-antipatterns.md index f9eb1fd8..7b24d982 100644 --- a/domains/performance/skills/performance/references/mm-useeffect-antipatterns.md +++ b/domains/performance/skills/performance/references/mm-useeffect-antipatterns.md @@ -76,7 +76,7 @@ useEffect(() => { The cancelled flag prevents the *setState*; AbortController additionally stops the request from consuming bandwidth/battery. The race-condition variant (stale response overwriting fresh data when `address` changes quickly) is fixed by the same cleanup — the old effect's closure is cancelled before the new one runs. -**Codify, don't copy-paste** (internal extension epic): once a repo has three hand-rolled cancelled flags, extract shared hooks — `useIsMounted()`, `useAbortableEffect(fn, deps)` (effect receives a signal), `useEventListener(target, event, handler)` (auto-removes on unmount) — so cleanup is the default, not per-site diligence. +**Codify, don't copy-paste** (extension epic MetaMask-planning#6525): once a repo has three hand-rolled cancelled flags, extract shared hooks — `useIsMounted()`, `useAbortableEffect(fn, deps)` (effect receives a signal), `useEventListener(target, event, handler)` (auto-removes on unmount) — so cleanup is the default, not per-site diligence. ## Pattern — missing cleanup for timers / subscriptions / listeners diff --git a/domains/performance/skills/performance/repos/metamask-mobile.md b/domains/performance/skills/performance/repos/metamask-mobile.md index e136589d..c22f6162 100644 --- a/domains/performance/skills/performance/repos/metamask-mobile.md +++ b/domains/performance/skills/performance/repos/metamask-mobile.md @@ -109,4 +109,4 @@ Ordered by impact. Each links to the guide with the fix. **The `Where` column li ## Attribution -Generic React Native references (`js-*`, `native-*`, `bundle-*`) adapted from "The Ultimate Guide to React Native Optimization" by Callstack. MetaMask-specific guidance (`mm-*`) from the internal Performance Guide for Engineers and verified codebase audits. Cross-platform React/Redux guidance (`mm-selector-cascade`, `mm-useeffect-antipatterns`, `mm-state-normalization`, `mm-react-compiler-error-triage`) adapted from MetaMask contributor-docs [`frontend-performance.md`](https://github.com/MetaMask/contributor-docs/blob/main/docs/frontend-performance.md) and the extension performance audit (extension PRs metamask-extension#38007, metamask-extension#37147). +Generic React Native references (`js-*`, `native-*`, `bundle-*`) adapted from "The Ultimate Guide to React Native Optimization" by Callstack. MetaMask-specific guidance (`mm-*`) from the internal Performance Guide for Engineers and verified codebase audits. Cross-platform React/Redux guidance (`mm-selector-cascade`, `mm-useeffect-antipatterns`, `mm-state-normalization`, `mm-react-compiler-error-triage`) adapted from MetaMask contributor-docs [`frontend-performance.md`](https://github.com/MetaMask/contributor-docs/blob/main/docs/frontend-performance.md) and the extension performance audit (MetaMask-planning#6571; extension PRs metamask-extension#38007, metamask-extension#37147). From 26a7daf83dfa30128cd847019866d40a037746c2 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 08:41:58 -0400 Subject: [PATCH 025/135] Restore the MetaMask-planning link in the step-waiver item MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Over-scrubbed. The audience is the MetaMask org, and the ticket is the evidence for the claim the item makes — that the LaunchDarkly provisioning blocker covers only the prod-flag half of that lane. Without it the example is an assertion. The scrub line is personal references, not org-internal ones. --- .../skills/pr-validate/references/evidence-trustworthiness.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md b/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md index 8933a8de..482f01e5 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md @@ -23,7 +23,7 @@ A green result is not proof. An agent — or an eager run — can produce eviden 11. **Lanes derive from the Manual testing steps — a CI-green row is not a lane** — the Validation Run's rows are generated top-down from the claim and the PR's own **Manual testing steps**, never bottom-up from whatever links already exist. For each step the claim depends on, the lane's payload is the **captured output of executing that step** (step "in Discover, group by `trace`" → a Discover permalink / **linked** trace-id table showing N rounds → N distinct `trace_id`s, per item 12), or an honest ⏳ naming the missing capture with a tracker. A row restating CI ("tests green at head `<sha>` in [CI run]") duplicates the Checks tab and is deleted — and a validation surface carries **zero** CI references, full stop: no `actions/runs` links, no "green at head" clauses, no "as context (only)" retention. The earlier carve-out here ("a CI link is admissible as context on a beyond-CI row") was itself the next costume: within a day all four sibling bodies (extension#43928–#43931) shipped restatements phrased as the exception — rows *leading* with "green at head … in [Unit tests CI]", the same link repeated 3× per body, the remediated row keeping it re-labeled "as context only" — while the gate's excuse regex matched the mere word "revert", so vocabulary, not evidence, discharged the class. The revert lane cites the revert **outcome** (which blocks failed, at which commit); its green-at-head half is the Checks tab's information and is omitted. A carve-out in an emit-time gate is an instruction to generation to phrase every violation as the exception — deliberate exceptions route through the human, never through an excuse predicate. Borrowed evidence — a sibling PR's capture, a unit falsifier standing in for the named live surface — never upgrades an uncaptured lane to ✅: "mechanism live-proven" co-located with "was not exercised" is an inflated verdict; downgrade it. Emit-time trigger: `pr-evidence-gate.py` classes `ci-restatement` (unconditional since 2026-07-21: any CI link / CI-green phrase in validation scope fires — no verdict co-location required, no beyond-CI excuse) and `inflated-verdict`, with the shipped extension#43928 rows and the carve-out-blessed "as context" shape as regression cases (2026-07-21). 12. **Identifiers resolve in one click — a bare id is a digging assignment** — trace ids, event ids, run ids, SHAs are *pointers into a system*, not evidence. Publishing a bunch of raw trace ids hands the reviewer the job of reconstructing project/environment/time window and querying Sentry themselves — it fails item 9's ~30-second test by construction (item 9 makes the signal *findable*; this item makes it *checkable*). Every identifier published as evidence is either hyperlinked to its resolving surface (the Sentry trace/event permalink, or an absolute-windowed Discover query pre-filtered to exactly those ids) or accompanied by the re-hosted captured output (query-result rows / envelope excerpt showing the discriminating fields) — ideally both. Special case that produced the rule: ids captured **locally** (mockttp forwarder, envelope intercept) never reached Sentry, so no permalink can exist — the re-hosted capture is the *only* admissible form, and pasting the id fragments plus a re-run recipe is the "spec necessary / output sufficient" violation wearing ids as decoration (extension#43931 Validation row, 2026-07-21). Rule of construction: when any item in this gate blesses an evidence class by name ("trace-id table", "envelope log"), it means the class's *resolvable instance*, never its bare tokens — a blessed class name is otherwise the next costume. Emit-time trigger: `pr-evidence-gate.py` class `bare-identifier`; converse-of-gate note: the prior gate *whitelisted* `trace_ids?` as beyond-CI payload and its own fix-message recommended "trace-id table" unqualified — second occurrence of "audit the gate for whitelists of the violating shape." 13. **Terminal exhibits are reader-native — a live link or a visual; a dump behind a link is still an opaque reference** — item 12 makes every pointer resolve in one click; this item constrains what it may resolve *to*. A positive verdict's terminal artifact is one of the two media a reviewer natively consumes: a **live link into the resolving system** (Sentry trace/event permalink, absolute-windowed Discover query pre-filtered to the claim) or a **visual capture** (screenshot/recording, annotated or cropped to the discriminating region). Raw files (`.log`/`.json`/`.har`, MB-scale dumps) are **appendix-only** — linked once for auditability, never the exhibit a claim rests on: a link whose target is a raw dump passes item 12 and fails item 9 one click later; the digging moved a hop away, it did not disappear (extension#43931 *second* remediation, 2026-07-21: the `bare-identifier` fix shipped a ✅ row whose sole resolver was a re-hosted ~70KB run log). Two corollaries: (a) **the gate items are conjunctive** — a fix for the newest item must re-pass all prior items; satisfying resolvability with an artifact that fails legibility is the generator's next costume; (b) **ascertain the terminal medium at step zero and pick the capture lane that can produce it** — a local intercept (mockttp envelope forwarder) can never yield a live Sentry permalink, so for Sentry-observable claims it is the supplementary falsifier lane and live ingest (dev build → `SENTRY_DSN_DEV`/test-metamask) is primary, precisely because it terminates in permalinks + screenshots; choosing a lane that cannot produce the terminal medium silently displaces it. Emit-time trigger: `pr-evidence-gate.py` class `dump-resolver`, with the remediated extension#43931 row as the regression case (2026-07-21). -14. **Manual testing steps are the validation contract — steps present ⇒ live evidence definitionally required; an impossibility waiver contradicting an executable step is invalid** — the PR's own **Manual testing steps** are the author's assertion that the claimed behavior *is* live-observable, and how: each numbered step is an executability proof (a step a human reviewer can run, `/pr-validate` can run) and its text is the capture spec. Item 11 derives the lanes from the steps top-down; this item closes the other side of the hatch — a lane derived from a step may not then be *waived by argument*. Emit-time procedure: build the per-step coverage map (step → executed-output artifact); for any step without one, the only admissible state is a **per-step** ⏳ + tracker whose blocker is that step's *own* unmet precondition, checked against the step text. Three waiver-inflation patterns from the producing instance (extension#43228/#42869/#44538, 2026-07-21 — all three shipped articulate impossibility rationales in the same body whose Manual testing steps asserted the opposite): (a) **borrowed impossibility** — the excuse imported from a different mechanism or sibling PR (#43228 waived its live lane citing the async remote-flag read race, which is #44538's mechanism; #43228's overrides are build-time env vars, and its steps 1–4 are directly executable in a dev build); (b) **lane-limitation universalized** — one harness's gap stated as global impossibility (#42869: "the e2e harness emits no error `event` envelope, so … not capturable pre-merge" — the step says *dev build with Sentry enabled, trigger an error*, which ingests error events into test-metamask without the e2e harness); (c) **blocked-scope inflation** — a genuinely blocked precondition of one half of the claim expanded to waive the whole lane (#44538: LaunchDarkly provisioning (tracked internally) blocks only the *prod-flag* half; the step's own text names the dev-injectable alternative — "or inject it into persisted `RemoteFeatureFlagController` state"). If a step is *truly* non-executable, the waiver is still inadmissible alone: the Manual testing steps are then wrong and are corrected in the same edit — a document may not simultaneously instruct a reviewer to observe X and declare X unobservable. The honest-⏳ lane blessed throughout this gate is for *not yet done*, never for *argued away*: an eloquent impossibility rationale is the cheapest token sequence that satisfies every prior item (no fake capture, no CI link, no bare id, no dump) — the generator's costume for the coverage axis. Emit-time trigger: `pr-evidence-gate.py` class `step-waiver` ("not demonstrable / capturable / observable", "not separately captured", "not attached", "rests entirely/solely on the falsifiers/unit/revert" — unconditional in validation scope; no tracker or artifact excuses it), with the three shipped waiver paragraphs as regression cases and the gate verified-blocking on all three live bodies (2026-07-21). Detection gap: the per-step consistency check (does a deferral's blocker match the step's own precondition?) stays procedural — the gate sees vocabulary, not step semantics. +14. **Manual testing steps are the validation contract — steps present ⇒ live evidence definitionally required; an impossibility waiver contradicting an executable step is invalid** — the PR's own **Manual testing steps** are the author's assertion that the claimed behavior *is* live-observable, and how: each numbered step is an executability proof (a step a human reviewer can run, `/pr-validate` can run) and its text is the capture spec. Item 11 derives the lanes from the steps top-down; this item closes the other side of the hatch — a lane derived from a step may not then be *waived by argument*. Emit-time procedure: build the per-step coverage map (step → executed-output artifact); for any step without one, the only admissible state is a **per-step** ⏳ + tracker whose blocker is that step's *own* unmet precondition, checked against the step text. Three waiver-inflation patterns from the producing instance (extension#43228/#42869/#44538, 2026-07-21 — all three shipped articulate impossibility rationales in the same body whose Manual testing steps asserted the opposite): (a) **borrowed impossibility** — the excuse imported from a different mechanism or sibling PR (#43228 waived its live lane citing the async remote-flag read race, which is #44538's mechanism; #43228's overrides are build-time env vars, and its steps 1–4 are directly executable in a dev build); (b) **lane-limitation universalized** — one harness's gap stated as global impossibility (#42869: "the e2e harness emits no error `event` envelope, so … not capturable pre-merge" — the step says *dev build with Sentry enabled, trigger an error*, which ingests error events into test-metamask without the e2e harness); (c) **blocked-scope inflation** — a genuinely blocked precondition of one half of the claim expanded to waive the whole lane (#44538: LaunchDarkly provisioning [#7482](https://github.com/MetaMask/MetaMask-planning/issues/7482) blocks only the *prod-flag* half; the step's own text names the dev-injectable alternative — "or inject it into persisted `RemoteFeatureFlagController` state"). If a step is *truly* non-executable, the waiver is still inadmissible alone: the Manual testing steps are then wrong and are corrected in the same edit — a document may not simultaneously instruct a reviewer to observe X and declare X unobservable. The honest-⏳ lane blessed throughout this gate is for *not yet done*, never for *argued away*: an eloquent impossibility rationale is the cheapest token sequence that satisfies every prior item (no fake capture, no CI link, no bare id, no dump) — the generator's costume for the coverage axis. Emit-time trigger: `pr-evidence-gate.py` class `step-waiver` ("not demonstrable / capturable / observable", "not separately captured", "not attached", "rests entirely/solely on the falsifiers/unit/revert" — unconditional in validation scope; no tracker or artifact excuses it), with the three shipped waiver paragraphs as regression cases and the gate verified-blocking on all three live bodies (2026-07-21). Detection gap: the per-step consistency check (does a deferral's blocker match the step's own precondition?) stays procedural — the gate sees vocabulary, not step semantics. 15. **The exhibit lives in the body — link AND visual; a live link alone is the verification path, not the exhibit** — item 13 blessed the terminal media as a *disjunction* (live link OR visual), and generation took the cheaper disjunct: a Discover permalink is producible from the API token alone, a screenshot needs a browser session — so extension#44540's live-ingestion exhibit shipped as a permalink + prose counts, with nothing in the PR body a reader could look at (2026-07-21: "only sentry link and not screenshot that makes it immediately obvious how evidence validates pr"). A live link defers validation behind **click + auth + query rendering + column interpretation** — the dump-resolver displacement one hop further, with the mountain now behind a login: it fails item 9's ~30-second test at the moment of the click, and for any reader *without* Sentry org access (most PR reviewers) a link-only exhibit degrades to a bare identifier (item 12) behind an auth wall. The repaired rule is a **conjunction**: a positive verdict's headline exhibit is an **embedded visual** — screenshot/recording of the linked resolving view (Discover result rows, trace waterfall), cropped/annotated to the discriminating region, captioned with what it should show — **and** the co-located live permalink (absolute-windowed) as the independent-verification path. Neither substitutes for the other: link-only hides the exhibit; visual-only is independently unverifiable. The 2026-07-16 clause "screenshots ride along when a browser session is available; the API token alone yields links + JSON, which is the automatable minimum" was the self-authored escape hatch of this axis (family: the "as context" carve-out, the honest-⏳ waiver): the *automatable minimum* got promoted to the shipped standard because it was the cheapest compliant artifact. A capture lane that cannot screenshot its resolving view is a lane gap to fix before publish (drive a browser session to the Discover URL), never a licensed downgrade — deliberate exceptions route through the human. Emit-time trigger: `pr-evidence-gate.py` class `link-only-exhibit` (non-negated verdict + `sentry.io` link + no image/recording embed in the unit), with the shipped #44540 paragraph as the regression case and the prior suite's permalink-only ALLOW cases flipped/augmented — third occurrence of "an ALLOW case containing the violating tokens is a specification of the next costume." Detection gaps: verdict co-location is required, so a no-verdict link-only paragraph evades mechanically; the visual-without-link converse stays procedural under item 12. 16. **The audit chain is mechanical — quote, don't transcribe; pin, don't point** — an exhibit's inline data must be a **verbatim, greppable excerpt** of the artifact (full-length identifiers, raw capture lines quoted exactly), and every repo-hosted artifact link must be a **commit-pinned, line-anchored permalink** (`/blob/<sha>/…#Lx-Ly`) to the discriminating lines. Producing real artifacts and then hand-transcribing digests severs the claim→artifact bridge at every link: an ellipsized id (`24b1e2da…`) cannot be grepped against any artifact even when the log is linked in the same block; a reformatted data block cannot be distinguished from confabulation without redoing the dig, so it reads as *claims in the form of data*; a branch-ref `/blob/<branch>/` link is a mutable pointer whose target can be rewritten after review (not tamper-evident); a file-level link without line anchors lands the reader at the top of a 1,000-line dump. The producing instance (extension#43929 validation comment, 2026-07-21) had every number substantiated by four re-hosted run logs — real, included, resolvable — and still drew "still no immediately auditable evidence just claims in the form of data": each prior item individually near-passed while the exhibit↔artifact binding stayed **editorial** (transcription + file link) instead of **mechanical** (quotation + pinned line anchor). Emit-time procedure: for every inline datum, quote the raw capture line it comes from (fenced, verbatim, full ids) and anchor it (`#L<n>`); pin every evidence link to the SHA (press `y` on the GitHub file view). Emit-time trigger: `pr-evidence-gate.py` classes `truncated-identifier` (a co-located resolver does NOT excuse it — the resolver resolves the full id, not the fragment the reader holds; hash-equality prose exempt) and `mutable-ref`, plus the closed **surface hole**: the shipped comment was published via `gh api` PATCH, which the porcelain-only matcher (`gh pr|issue edit|create|comment`) never saw — the gate now scans `gh api` body writes (`-F body=@file`, `-f body=…`, `--input`); fifth occurrence of the converse-of-gate rule, one level down: audit the gate for *spellings of the write it cannot see*, not just tokens it excuses. Detection gaps: the line-anchor half of pinning and the paraphrase-vs-quotation judgment stay procedural — the gate sees ellipses and branch refs, not editorial fidelity. 17. **Evidence is captured in its environment — data alone is insufficient even when correct** — item 16 makes the data trustworthy as *transcription* (verbatim, greppable, pinned); this item polices what transcription can never carry: **liveness provenance**. A quoted `EVIDENCE trace_id=…` line, a re-hosted gist, a hand-assembled id table can all be correct and still show nothing about *where they came from* — extracted data is indistinguishable from data typed by hand, so it cannot make it immediately apparent that the evidence was captured **live** from a **functioning** system. The exhibit for a system-of-record-observable claim therefore includes an **in-environment capture**: a screenshot/recording of the resolving system's own UI (the Sentry Discover/trace view with the query, project/environment selectors, absolute time window, and result rows all in-frame) — the environmental chrome is not decoration, it *is* the provenance: it shows the query really ran, in the real dashboard, over the real window, and returned these rows. Correctness was never the failing dimension (2026-07-21: "just the data is insufficient even if correct — it needs to be immediately apparent that evidence was captured live and is functional"). Relation to prior items: item 15's link+visual conjunction fired only when a `sentry.io` link was present, and item 13's `NATIVE_MEDIUM` blessed an inline fenced excerpt as a terminal medium — so a no-link, quoted-data exhibit (the fidelity-remediated shape: full ids, verbatim excerpts, pinned line anchors, zero environment captures) passed the whole regime while carrying zero liveness provenance. The joint rule after this item: a telemetry-observable positive verdict always carries the in-environment visual (plus the live permalink per item 15); quoted excerpts, gists, and data files are appendix beside it, never the exhibit. Emit-time trigger: `pr-evidence-gate.py` class `data-only-exhibit` (non-negated verdict + telemetry-observation vocabulary + no image/recording embed + no sentry link — with a sentry link, `link-only-exhibit` already fires), with the re-hosted-gist ALLOW case flipped (fifth occurrence of "the ALLOW case was the next costume's spec") and the #43929 quoted-excerpt shape as a regression case. Detection gaps: vocabulary-scoped (telemetry-observation terms, not bare code tokens like `trace.test.ts`), so a claim phrased entirely without them evades mechanically; and the gate cannot see whether an embedded image actually shows the environment's chrome — screenshot content stays procedural (item 2's "eyeball it" applies: the capture must show the *resolving UI*, not a cropped data region indistinguishable from a spreadsheet). From 6459be40b0a39a446c30befb920077c00aabbbad Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 11:06:25 -0400 Subject: [PATCH 026/135] Move `memory-leak-hunt` from `coding` to a new `stability` domain The skill covers runtime retention behaviour, not code authoring, and `coding` reads as language- and style-level guidance. Registers `/domains/stability/` in CODEOWNERS alongside the other platform-owned domains. --- .github/CODEOWNERS | 1 + .../skills/memory-leak-hunt/references/heap-investigation.md | 0 .../skills/memory-leak-hunt/scripts/heap-over-cycles.example.ts | 0 .../skills/memory-leak-hunt/scripts/retention-scan.py | 0 domains/{coding => stability}/skills/memory-leak-hunt/skill.md | 0 5 files changed, 1 insertion(+) rename domains/{coding => stability}/skills/memory-leak-hunt/references/heap-investigation.md (100%) rename domains/{coding => stability}/skills/memory-leak-hunt/scripts/heap-over-cycles.example.ts (100%) rename domains/{coding => stability}/skills/memory-leak-hunt/scripts/retention-scan.py (100%) rename domains/{coding => stability}/skills/memory-leak-hunt/skill.md (100%) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index f156c522..9ba0f339 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -18,6 +18,7 @@ /domains/performance/ @MetaMask/extension-platform @MetaMask/mobile-platform /domains/perps/ @MetaMask/perps /domains/pr-workflow/ @MetaMask/extension-platform @MetaMask/mobile-platform +/domains/stability/ @MetaMask/extension-platform @MetaMask/mobile-platform /domains/swaps/ @MetaMask/swaps-engineers /domains/testing/ @MetaMask/qa /domains/ui/ @MetaMask/design-system-engineers diff --git a/domains/coding/skills/memory-leak-hunt/references/heap-investigation.md b/domains/stability/skills/memory-leak-hunt/references/heap-investigation.md similarity index 100% rename from domains/coding/skills/memory-leak-hunt/references/heap-investigation.md rename to domains/stability/skills/memory-leak-hunt/references/heap-investigation.md diff --git a/domains/coding/skills/memory-leak-hunt/scripts/heap-over-cycles.example.ts b/domains/stability/skills/memory-leak-hunt/scripts/heap-over-cycles.example.ts similarity index 100% rename from domains/coding/skills/memory-leak-hunt/scripts/heap-over-cycles.example.ts rename to domains/stability/skills/memory-leak-hunt/scripts/heap-over-cycles.example.ts diff --git a/domains/coding/skills/memory-leak-hunt/scripts/retention-scan.py b/domains/stability/skills/memory-leak-hunt/scripts/retention-scan.py similarity index 100% rename from domains/coding/skills/memory-leak-hunt/scripts/retention-scan.py rename to domains/stability/skills/memory-leak-hunt/scripts/retention-scan.py diff --git a/domains/coding/skills/memory-leak-hunt/skill.md b/domains/stability/skills/memory-leak-hunt/skill.md similarity index 100% rename from domains/coding/skills/memory-leak-hunt/skill.md rename to domains/stability/skills/memory-leak-hunt/skill.md From 8dc5b86b0da7720f0d6270f8a969a2570679eb74 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 11:22:59 -0400 Subject: [PATCH 027/135] Fold in `typescript-compiler-blindspots`, moved from the `testing` domain Was a separate PR against `domains/testing`. It belongs here: its subject is whether a hand-written type agrees with its authoritative source, which is the question `derive-types` answers from the authoring side, and it shares this domain's premise that a green `tsc` is not evidence the types are correct. Directory name and frontmatter `name` already agree; only the domain moved. --- .../references/false-negatives.md | 259 +++++++++++++++++ .../references/metamask-extension.md | 54 ++++ .../references/worked-example.md | 92 ++++++ .../scripts/substitution-ab.sh | 51 ++++ .../typescript-compiler-blindspots/skill.md | 272 ++++++++++++++++++ 5 files changed, 728 insertions(+) create mode 100644 domains/typescript/skills/typescript-compiler-blindspots/references/false-negatives.md create mode 100644 domains/typescript/skills/typescript-compiler-blindspots/references/metamask-extension.md create mode 100644 domains/typescript/skills/typescript-compiler-blindspots/references/worked-example.md create mode 100755 domains/typescript/skills/typescript-compiler-blindspots/scripts/substitution-ab.sh create mode 100644 domains/typescript/skills/typescript-compiler-blindspots/skill.md diff --git a/domains/typescript/skills/typescript-compiler-blindspots/references/false-negatives.md b/domains/typescript/skills/typescript-compiler-blindspots/references/false-negatives.md new file mode 100644 index 00000000..5bcf97b8 --- /dev/null +++ b/domains/typescript/skills/typescript-compiler-blindspots/references/false-negatives.md @@ -0,0 +1,259 @@ +# The standing blind spots — where `tsc` returns a false negative + +Defects the compiler cannot report, independent of any particular PR. Unlike the +restated-type class, these need no substitution to find: they are properties of +the language and the config, and they are present in every file. + +**Verified, not asserted.** The demonstration file below was typechecked against +`metamask-extension` at `7fafda0` with the repo's own `tsconfig.json`: + +``` +$ NODE_OPTIONS='--max-old-space-size=9216' npx tsc -p tsconfig.json --noEmit +$ echo $? +0 +``` + +Every block in it is wrong. `tsc` reports **zero errors** on all ten. + +--- + +## A. Unsoundness in the type system + +### 1. Index access is not `| undefined` + +```ts +const parts = host.split('.'); +return parts[9]; // typed `string`; `undefined` at runtime +``` + +`arr[i]` and `record[key]` are typed as if the element always exists. Easy to underestimate how much surface this covers: every `.split()[n]`, every +lookup table, every `find`-then-index is an instance. + +- **Flag:** `noUncheckedIndexedAccess` (not enabled in `metamask-extension`). +- **In review:** any index or dynamic key access on a path that can be empty or + short. `parts[parts.length - 1]` is only safe if the array is provably non-empty. + +### 2. Optional property vs. explicit `undefined` + +```ts +type PopupState = { currentPopupId?: number }; +const cleared: PopupState = { currentPopupId: undefined }; // accepted +``` + +`?:` means "may be absent"; without the strict flag it *also* accepts +present-and-undefined. Code that distinguishes the two (`'k' in obj`, +`Object.keys().length`, serialization that drops vs. writes `null`) breaks on a +distinction the type cannot express. + +- **Flag:** `exactOptionalPropertyTypes` (not enabled). +- **In review:** persisted state and message payloads, where absent and + `undefined` serialize differently. + +### 3. Method parameters are bivariant + +```ts +type MessageHandler = { handle(msg: { kind: 'booted' | 'connectivity' }): void }; +const narrow: MessageHandler = { handle(msg: { kind: 'booted' }) { … } }; // accepted +``` + +`strictFunctionTypes` makes function *properties* contravariant but exempts +**method shorthand** — deliberately, for DOM/array compatibility. So a handler +that only accepts a narrow subtype satisfies a wide handler type and receives +values it declared it would not. + +- **Fix:** declare callbacks as properties (`handle: (msg: …) => void`), which + *is* checked. +- **In review:** any interface with method-shorthand callbacks, especially + message/event handlers. + +### 4. Arrays are covariant + +```ts +const bases: Base[] = specials; // Special[] → Base[], accepted +bases.push(new Base()); // `specials` now holds a non-Special +``` + +- **In review:** a narrower array widened and then mutated. `readonly T[]` blocks it. + +### 5. Excess-property checking only fires on fresh literals + +```ts +const draft = { url: 'a', justification: 'b', reasosn: ['typo'] }; +const params: CreateParams = draft; // no error — not a fresh literal +``` + +Assigning the literal directly would catch the typo. Through a variable, the +extra property is silently ignored — and the intended one is missing. + +- **In review:** config/params objects built up in a variable before being passed. + This is how a misspelled option key survives to runtime. + +### 6. Structural typing erases domain distinctions + +```ts +type AccountAddress = string; +type TransactionHash = string; +fetchBalance(txHash); // accepted — both are `string` +``` + +Aliases are not nominal. Two semantically incompatible values are interchangeable +whenever their structure matches. + +- **Fix:** branded types, or a template-literal type where the format differs + (`Hex` = `` `0x${string}` `` genuinely does discriminate). +- **In review:** same-primitive parameters, especially adjacent ones in a + signature, where swapping the arguments would still compile. + +## B. Boundaries the compiler does not cross + +### 7. `any` absorbs any annotation + +```ts +declare function readPersisted(key: string): any; +const meta: VaultMeta = readPersisted('meta'); // asserted, never validated +meta.version.toFixed(2); // may be a string at runtime +``` + +An `any` satisfies every annotation silently. Sources: untyped dependencies, +`JSON.parse`, generics that default to `any`, and `as any`. + +- **In review:** trace where a confidently-typed value *entered* the program. If + it entered as `any`, its type is a wish. + +### 8. Ambient `declare module` is an unverified assertion + +```ts +declare module '@ensdomains/content-hash' { + const contentHash: { decode: (h: string) => string /* … */ }; + export default contentHash; +} +``` + +Hand-written module declarations are believed unconditionally — nothing compares +them to the package. Getting a return type wrong here is invisible forever, and +the declaration is **global**, so it also shadows any real types the package +later ships. + +- **In review:** read the package's actual source at the installed version when a + `declare module` is added or changed. Prefer `@types/*` or a PR upstream. + +### 9. External data is asserted, not validated + +```ts +const chainId = (rpcResult as { chainId: string }).chainId.slice(2); +``` + +Every `as` on data crossing a boundary — RPC responses, `fetch().json()`, +`chrome.storage` reads, persisted state written by an *older version of the app* — +is a claim the compiler cannot evaluate. + +- **In review:** highest stakes for persisted state and migrations, where the real + input was produced by code that no longer exists. A runtime validator + (`@metamask/superstruct`, zod) is the only thing that actually checks. + +### 10. A JS caller is not checked at all + +With `checkJs` off (the default, and the case in `metamask-extension`), a type +written for a function whose callers are still `.js` is compared against no call +site, ever. See the restated-type class in the main skill — this is why that class +exists. + +## C. Config-level blind spots + +Check these before trusting a green build. Values shown are `metamask-extension` +at the time of writing. + +| Setting | Effect | Here | +|---|---|---| +| `skipLibCheck` | Errors *inside* `.d.ts` files are suppressed, including conflicts between library types | **`true`** (from `@tsconfig/node22`) | +| `exclude` | Excluded files are never typechecked | `**/*.stories.tsx`, `**/*.stories.ts` | +| `include` | Anything outside it is invisible to `tsc` | `app`, `development`, `shared`, `test`, `types`, `ui`, `*.ts` | +| `checkJs` | Off ⇒ `.js` callers unchecked | unset | +| `noEmit` + bundler | `tsc` never produces the shipped artifact; webpack/swc transpiles **without** typechecking, so a type error cannot break the build — only the separate `lint:tsc` job reports it | `noEmit: true` | +| `@ts-expect-error` / `@ts-ignore` | Point suppressions | grep before trusting a clean file | + +The last row is worth stating plainly: **type errors do not break the build.** They +break a CI job. If that job is skipped, filtered, or its output is not read, the +types were never checked at all. + +--- + +## The demonstration file + +Drop this anywhere inside the `include` paths and typecheck. Zero errors is the +expected — and alarming — result. + +```ts +/* eslint-disable @typescript-eslint/no-explicit-any, @typescript-eslint/no-unused-vars */ + +// 1. Index access is not `| undefined` +function lastSegment(host: string): string { + const parts = host.split('.'); + return parts[9]; // typed `string`; `undefined` at runtime +} +export const seg = lastSegment('foo.eth').toUpperCase(); + +// 2. Record lookup claims the value always exists +declare const gateways: Record<string, { url: string }>; +export const gw = gateways.definitelyNotAKey.url; + +// 3. Optional property accepts an explicit undefined +type PopupState = { currentPopupId?: number }; +const cleared: PopupState = { currentPopupId: undefined }; +export const idPlusOne = (cleared.currentPopupId ?? 0) + 1; + +// 4. Method-shorthand parameters are bivariant +type MessageHandler = { handle(msg: { kind: 'booted' | 'connectivity' }): void }; +const narrow: MessageHandler = { handle(msg: { kind: 'booted' }) {} }; +export { narrow }; + +// 5. Arrays are covariant +class Base {} +class Special extends Base { + special() { + return 1; + } +} +const specials: Special[] = [new Special()]; +const bases: Base[] = specials; +bases.push(new Base()); +export const boom = () => specials.map((s) => s.special()); + +// 6. Excess-property checking only fires on fresh literals +type CreateParams = { url: string; justification: string }; +const draft = { url: 'a', justification: 'b', reasosn: ['typo'] }; +export const params: CreateParams = draft; + +// 7. `any` absorbs any annotation +declare function readPersisted(key: string): any; +type VaultMeta = { version: number; storageKind: 'data' | 'split' }; +export const meta: VaultMeta = readPersisted('meta'); +export const ver = meta.version.toFixed(2); + +// 8. An ambient `declare module` is an unverified assertion +import contentHash from '@ensdomains/content-hash'; + +export const decoded: string = contentHash.decode('0x'); + +// 9. Structural typing erases domain distinctions +type AccountAddress = string; +type TransactionHash = string; +declare function fetchBalance(addr: AccountAddress): Promise<string>; +declare const txHash: TransactionHash; +export const wrong = fetchBalance(txHash); + +// 10. A type assertion on external data is unchecked by construction +declare const rpcResult: unknown; +export const chainId = (rpcResult as { chainId: string }).chainId.slice(2); +``` + +## How to use the catalog in a review + +Don't run all ten as a checklist. Pick by what the diff touches: + +- **New indexing / destructuring** → 1, 2 +- **New message, event, or callback types** → 3, 5, 6 +- **New `declare module`, new dependency, `@types` change** → 8 +- **Anything reading persisted state, storage, or an RPC response** → 7, 9 +- **A JS→TS conversion** → 10, plus the restated-type class in the main skill +- **Any PR whose safety argument is "CI is green"** → section C, first diff --git a/domains/typescript/skills/typescript-compiler-blindspots/references/metamask-extension.md b/domains/typescript/skills/typescript-compiler-blindspots/references/metamask-extension.md new file mode 100644 index 00000000..36a67496 --- /dev/null +++ b/domains/typescript/skills/typescript-compiler-blindspots/references/metamask-extension.md @@ -0,0 +1,54 @@ +# Repo notes — metamask-extension + +Specifics for running the two-arm proof in `MetaMask/metamask-extension`. (Kept +here rather than in `repos/` on purpose: a `repos/` subdir containing only an +extension overlay would make this skill *skip* installs for mobile and core.) + +## Typecheck invocation + +```bash +NODE_OPTIONS='--max-old-space-size=9216' npx tsc -p tsconfig.json --noEmit +``` + +`package.json`'s `lint:tsc` uses `--max-old-space-size=6144`, which **OOMs** on a +full run on a 16 GB machine — and the OOM exits non-zero with no type diagnostics, +so a naive exit-code check reads it as "errors found." Raise the heap and read the +output. A full run takes roughly 3–5 minutes. + +## Where to put probes + +Anywhere under the `include` list — `app`, `development`, `shared`, `test`, +`types`, `ui`. `app/scripts/derive-probe/` works. Delete it afterwards; it is +inside the build's include paths. + +## What the compiler is *not* checking + +- **`checkJs` is unset** and there is no `// @ts-check` in `app/scripts/background.js`. + `background.js` is the sole caller of much of `app/scripts/lib/**`, so a type + written for those functions is validated against **nothing**. This is where + migration PRs accumulate silent divergence, and where this skill pays. +- `tsconfig.json` sets `lib: ["DOM", "es2023"]`, overriding the base. **`webworker` + is absent**, so service-worker globals (`clients`, `Clients`, `Client`) have no + authoritative type in scope — hand-declaring them is legitimate here. +- Strictness comes from `@tsconfig/node22` (`strict: true`), so `strictNullChecks` + is on and nullability divergences do surface in a probe. + +## Authoritative sources worth knowing + +| Looking for | Derive from | +|---|---| +| current chain id | `ReturnType<typeof getCurrentChainId>` (`shared/lib/selectors/networks.ts`) → `Hex`, not `string` | +| the EIP-1193 provider | `ReturnType<NetworkController['getProviderAndBlockTracker']>['provider']` — note the `\| undefined` | +| a controller method's params | `SomeController['methodName']` | +| persisted state root | `MetaMaskStorageStructure` (`shared/lib/stores/base-store.ts`) | +| a `browser.*` listener payload | `browser.WebRequest.OnErrorOccurredDetailsType` and siblings, from `webextension-polyfill` | +| offscreen message targets/events | the enums in `shared/constants/offscreen-communication.ts` | + +## Before "fixing" a `chrome.*` type error + +Read the declaration in `node_modules/@types/chrome/index.d.ts` first. Several +parameters are template-literal **string** types, not the enums they mirror — e.g. +`ContextFilter.contextTypes?: ` `` `${ContextType}`[] `` and +`CreateParameters.reasons: ` `` `${Reason}`[] ``. A plain string literal already +satisfies them, so swapping in `chrome.offscreen.Reason.X` is an unnecessary +runtime change, not a type fix. diff --git a/domains/typescript/skills/typescript-compiler-blindspots/references/worked-example.md b/domains/typescript/skills/typescript-compiler-blindspots/references/worked-example.md new file mode 100644 index 00000000..a7f1f978 --- /dev/null +++ b/domains/typescript/skills/typescript-compiler-blindspots/references/worked-example.md @@ -0,0 +1,92 @@ +# Worked example — a JS→TS migration PR + +[metamask-extension#44397](https://github.com/MetaMask/metamask-extension/pull/44397), +head `7fafda0`. 11 files converted, +153/−72, described as *"mostly mechanical +JS→TS with equivalent runtime logic"*, four files *"rename only"*. All CI green, +including `lint:tsc`. + +Twelve hand-written types. Nine had an authoritative source. Five disagreed with it. + +## Arm A + +``` +$ NODE_OPTIONS='--max-old-space-size=9216' npx tsc -p tsconfig.json --noEmit +$ echo $? +0 +``` + +Silent — so every diagnostic below is attributable to the substitution. + +## Arm B + +Six probe files, each substituting the derived type and calling it as the real +code does: + +``` +probe-1-get-obj-structure.ts(19,47): error TS2345: Argument of type + 'MetaMaskStorageStructure | undefined' is not assignable to parameter of type + 'Record<string, unknown>'. +probe-2-set-current-popup-id.ts(22,21): error TS2345: Argument of type 'undefined' + is not assignable to parameter of type 'number'. +probe-2-set-current-popup-id.ts(29,21): error TS2345: Argument of type + 'number | undefined' is not assignable to parameter of type 'number'. +probe-3-ens-provider.ts(27,32): error TS18048: 'provider' is possibly 'undefined'. +probe-3-ens-provider.ts(44,14): error TS2322: Type + 'SwappableProxy<ProxyWithAccessibleTarget<Provider>> | undefined' is not + assignable to type 'HandWrittenEthProvider'. +probe-4-offscreen-message.ts(43,14): error TS2322: Types of property 'target' are + incompatible. Type 'string' is not assignable to type 'OffscreenCommunicationTarget'. +probe-6-chain-id-widening.ts(31,50): error TS2322: Type '"1"' is not assignable to + type '`0x${string}`'. +``` + +A seventh probe compiled the PR's *original* string literals with **zero** errors — +which is how the two unnecessary runtime changes below were established. + +## Findings + +| Hand-written | Authoritative source | Shape | +|---|---|---| +| `_setCurrentPopupId: ((id: number \| undefined) => void)` | `AppStateController['setCurrentPopupId']` → `(id: number) => void` | widening | +| `getCurrentChainId: () => string` | `ReturnType<typeof getCurrentChainId>` → `` `0x${string}` `` | widening | +| `target: string` on a received message | the sender, TypeScript in-repo → `OffscreenCommunicationTarget` | widening | +| `EthProvider` (written twice, unshared) | `ReturnType<NetworkController['getProviderAndBlockTracker']>['provider']` | duplication + dropped nullability | +| `obj: Record<string, unknown>` | already in a JSDoc `@type` on the argument at the call site: `MetaMaskStorageStructure \| undefined` | placeholder + dropped nullability | + +Two of these had a consequence beyond tidiness: + +- The widened setter is what let `setter?.(undefined)` compile. Deriving it fails — + usefully, because it surfaces a genuine mismatch between a controller method's + declared parameter and how its callers actually use it. +- The dropped nullability sat in the same edit that deleted a `= {}` default + parameter, i.e. the guard that existed *for* the nullable case. (Traced: inert + today, because a fallback upstream guarantees an object by the time the path runs.) + +**Separately**, reading `@types/chrome` before trusting a type error found two +runtime changes the types never required: `contextTypes` and `reasons` are declared +as template-literal **string** types (`` `${ContextType}`[] ``, `` `${Reason}`[] ``), +so the original `['OFFSCREEN_DOCUMENT']` / `['IFRAME_SCRIPTING']` already compiled. +The PR replaced both with runtime enum lookups, and added a redundant cast. + +## Clearances — and one that mattered + +Five claims the probes **cleared**: + +- `Promise<string>` as a provider `request` return looked unsound. It isn't: the + real `request<Params, Result extends Json>` is generic in its result, so a call + site may legitimately fix `Result = string`. +- Two `declare module` blocks: neither package ships types, and no `@types/*` is + installed → no authoritative source exists, so hand-writing is correct. +- A hand-declared `clients?: { matchAll }`: the authoritative `Clients` lives in + `lib.webworker.d.ts`, which is not in this repo's `tsconfig.lib` → out of scope. +- Two dependency callbacks (`() => string`, `() => boolean`) matched their sources + exactly. +- A dropped argument (`_getPopup(id)` → `_getPopup()`) was a genuine no-op: the + base function declared no parameters, so the argument was already discarded. + +**The provider clearance is the reason Step 5 exists.** The first Arm B run +reported `TS18048: 'provider' is possibly 'undefined'` on that probe — an error on +the line the return-type claim lived on, which reads as confirmation if you count +exit codes. The nullability error fired one property *ahead* of the claim. Setting +it aside with `NonNullable<…>` and re-running showed the return type compiles +clean. Reported as a finding, it would have been wrong. diff --git a/domains/typescript/skills/typescript-compiler-blindspots/scripts/substitution-ab.sh b/domains/typescript/skills/typescript-compiler-blindspots/scripts/substitution-ab.sh new file mode 100755 index 00000000..32d76bd1 --- /dev/null +++ b/domains/typescript/skills/typescript-compiler-blindspots/scripts/substitution-ab.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# +# Two-arm type proof: does a hand-written type agree with the authoritative one? +# +# substitution-ab.sh <repo-path> <probe-dir> [probe-dest] +# +# <repo-path> repo checked out at the PR head, deps installed +# <probe-dir> directory of probe-*.ts files (see skill.md Step 3) +# [probe-dest] where to stage them, relative to <repo-path>; must sit inside +# the tsconfig `include` paths. Default: src/__type-probe__ +# +# Arm A must be silent. If it is not, stop — nothing in Arm B is attributable. +set -uo pipefail + +REPO=${1:?usage: substitution-ab.sh <repo-path> <probe-dir> [probe-dest]} +PROBES=${2:?usage: substitution-ab.sh <repo-path> <probe-dir> [probe-dest]} +DEST=${3:-src/__type-probe__} + +: "${NODE_OPTIONS:=--max-old-space-size=9216}" +export NODE_OPTIONS + +cd "$REPO" || exit 1 +[ -d "$PROBES" ] || { echo "no such probe dir: $PROBES" >&2; exit 1; } + +cleanup() { rm -rf "$REPO/$DEST"; } +trap cleanup EXIT INT TERM + +echo "=== Arm A — PR head as written (must be silent) ===" +A=$(npx tsc -p tsconfig.json --noEmit 2>&1) +A_STATUS=$? +if [ -n "$A" ]; then + echo "$A" + echo + echo "!! Arm A is NOT silent (exit $A_STATUS). The comparison is INCONCLUSIVE:" + echo "!! Arm B's diagnostics cannot be attributed to the substitution." + echo "!! Fix the baseline (toolchain, lockfile, heap, project scope) before reading Arm B." + exit 2 +fi +echo "0 diagnostics — baseline clean, Arm B is attributable." +echo + +echo "=== Arm B — same commit, derived types substituted ===" +mkdir -p "$DEST" +cp "$PROBES"/probe-*.ts "$DEST"/ 2>/dev/null || { + echo "no probe-*.ts found in $PROBES" >&2; exit 1; } + +npx tsc -p tsconfig.json --noEmit 2>&1 +echo +echo "=== Each diagnostic above is a disagreement the hand-written type concealed. ===" +echo "=== Before believing any of them: confirm the diagnostic is the one the ===" +echo "=== claim needs, not an earlier cause short-circuiting it (skill.md Step 5).===" diff --git a/domains/typescript/skills/typescript-compiler-blindspots/skill.md b/domains/typescript/skills/typescript-compiler-blindspots/skill.md new file mode 100644 index 00000000..90c1bc4f --- /dev/null +++ b/domains/typescript/skills/typescript-compiler-blindspots/skill.md @@ -0,0 +1,272 @@ +--- +name: typescript-compiler-blindspots +description: >- + Find the type defects `tsc` is structurally unable to report — a green build is + not evidence the types are correct. Covers the two classes: (1) hand-written + types that restate an authoritative source and disagree with it, caught by + substituting the derived type at a fixed commit and diffing `tsc` output; and + (2) the standing blind spots in the language and config — unchecked array/record + indexing, bivariant method parameters, covariant arrays, `any` absorption at + untyped boundaries, ambient `declare module` assertions, excess-property checks + that only fire on fresh literals, and external data asserted rather than + validated. Also audits typing edits that quietly change runtime behavior: + stripped `| undefined`, deleted default parameters, literals swapped for runtime + enum lookups, calls made optional so a throw becomes a silent no-op. Use when + reviewing a JS→TS migration, a PR that hand-writes types for values that already + have them, a "rename-only" refactor, or any PR claiming a change is mechanical. + Trigger phrases include "validate this TypeScript migration", "is this type + right", "does this type match the real shape", "why didn't CI catch this type", + "derive vs define", and "what can tsc not check". +maturity: experimental +--- + +# TypeScript compiler blind spots + +A hand-written type is a **claim about a value's shape**, and it compiles whether +or not the claim is true. `tsc` checks declarations for internal **consistency** — +never for **correspondence** to the source they restate. Across a JavaScript +boundary (`checkJs` off) it checks nothing at all. + +Those are the blind spots. This skill finds what is hiding in them. + +Two classes, two methods: + +| Class | What it is | Method | +|---|---|---| +| **Restated types** | A type hand-written to describe a value that already has an authoritative type | **Substitution A/B** — swap in the derived type, diff `tsc` output | +| **Standing blind spots** | Defects the language and config cannot report at all, in any codebase | **Targeted audit** — [references/false-negatives.md](references/false-negatives.md) | + +The second class is the one that surprises people: a file of genuinely broken +code can typecheck clean. The reference includes exactly that — a demonstration +file where every block is wrong and `tsc` reports zero errors. + +Companion to the authoring rule it enforces — *derive types from authoritative +sources instead of re-declaring them* in +[contributor-docs `docs/typescript.md`](https://github.com/MetaMask/contributor-docs/blob/main/docs/typescript.md). +This skill is the review-side proof; that document is the write-side guidance. + +## When to use + +- A **JS→TS migration** PR, or one whose body says *mechanical*, *rename-only*, or *no behavior change*. +- A PR that **hand-writes a type for a value that already has one** — a controller method's parameters, a selector's return, a message payload, a package's exported shape. +- A reviewer asks "is this type actually right?" and the answer so far is "it compiles." + +Out of scope: code correctness (use a normal review), runtime behavior (use e2e / +visual proof), and lint/format (CI owns those). + +## Prerequisites + +- The repo checked out at the PR head, dependencies installed, `tsc` runnable. +- Enough heap for a full typecheck on large repos (see Troubleshooting). +- A scratch directory inside the `tsconfig` `include` paths for probe files. + +## The core idea: two arms, one commit + +Both arms sit at the **same commit**. They differ by a *substitution*, not by a +ref — so there is no build, no rebase, and no merge boundary to confound. + +| | What it is | What it must show | +|---|---|---| +| **Arm A** | The PR exactly as written | **Silent.** Zero diagnostics | +| **Arm B** | Same tree + probes that use the *derived* type, exercised as the real code exercises it | Each new diagnostic = a disagreement the hand-written type concealed | + +**Arm A must be silent or the run is inconclusive.** If the untouched tree +already emits diagnostics, nothing in Arm B is attributable to the substitution — +"N errors in Arm B" is then a count, not a finding. Publish Arm A's result +verbatim as the delivery check. + +## Instructions + +### Step 1: Inventory every type the PR hand-wrote + +```bash +gh pr diff <pr> | grep -nE '^\+.*(type [A-Z]|interface [A-Z]|: (Record<|string|number|boolean|unknown|any)\b)' +``` + +List them. Each one is a claim you are about to test. + +### Step 2: Find the authoritative source for each + +Work down this list — the first hit wins. In the worked example 9 of the 12 +hand-written types had a source, each found in under a minute: + +1. **The call site.** What is actually passed? In a JS caller, check for a JSDoc + `@type {import('…').Foo}` annotation on the variable — the answer is sometimes + literally already written down there. +2. **The class or method being wrapped** → `MyController['someMethod']`. +3. **A package already imported in the same file** — e.g. `webextension-polyfill` + defines every listener payload; if the file calls the API, the type is in reach. +4. **The sender**, for a message or event payload. If the sender is TypeScript, the + shape is derivable, not guessable. +5. **A selector's return** → `ReturnType<typeof mySelector>`. +6. **`@types/*` for a platform API.** Read the actual declaration before "fixing" + a type error — many are template-literal *string* types (`` `${SomeEnum}`[] ``), + which already accept a plain string literal. +7. **No source exists** — an untyped dependency, a lib absent from `tsconfig.lib`, + a genuinely new boundary the repo owns. Hand-writing is then **correct**. Record + it as a cleared falsifier with the reason; do not report it as a finding. + +### Step 3: Write one probe per claim + +One file per claim, in a scratch dir inside the `include` paths. Each probe names +its authoritative source in a header comment and calls the derived type **the way +the real call site calls it**: + +```ts +// PROBE — src/thing.ts hand-wrote `setFoo: (id: number | undefined) => void`. +// Authoritative: FooController['setFoo'] (foo-controller.ts:120) — param is `number`. +// The real code calls it as below. +import type { FooController } from '../controllers/foo-controller'; + +declare const setFoo: FooController['setFoo']; + +export function asCalledByTheRealCode() { + setFoo(undefined); // thing.ts:105 +} +``` + +### Step 4: Run both arms + +```bash +./scripts/substitution-ab.sh <repo-path> <probe-dir> +``` + +Or by hand — Arm A first, and stop if it is not silent. + +### Step 5: Isolate diagnostics that fire for the wrong reason + +A checker reports the *first* failure it reaches, so an unrelated earlier cause +can short-circuit the claim under test — and an exit-code read scores that as a +confirmation. **Assert on the specific diagnostic** (code + message + line), and +where an earlier cause intervenes, neutralise it and re-probe: + +```ts +const defined = value as NonNullable<Derived>; // set nullability aside +// …now the return-type claim is the only thing left to fail +``` + +This is not hypothetical — see the worked example, where a claim that a return +type was unsound turned out **sound** once the nullability error ahead of it was +isolated. + +### Step 6: Report findings *and* clearances + +Give each claim a verdict, and say which ones the probes **cleared**. A +substitution sweep that only ever confirms is indistinguishable from one that +never isolated anything. + +## The four divergence shapes + +Four shapes to check for. All five divergences in the worked example were one of +these, which is a small sample — treat the list as a starting checklist, not a +partition: + +1. **Widening** — `string` for a `Hex`/template-literal type, `string` for an enum, + `number | undefined` for `number`. Admits values the real type rejects; worst + when a guard downstream depends on the narrower form. +2. **Dropped nullability** — the source says `| undefined`, the hand-written type + doesn't. Erases the compiler's record of why a runtime guard exists. +3. **Duplication** — the same shape written out in two files, unshared. Both copies + now need every future change. +4. **Placeholder** — `Record<string, unknown>`, `any`, or `unknown` standing in for + a shape that is known. Pushes a cast to every use site. + +## Escape hatches are the tell + +When a diff adds a hand-written type *and* an `as`, a `!`, a new `?.`, or an +`eslint-disable` in the same region, check whether the escape hatch exists to service +the type rather than the runtime. Count them — a cluster marks where to probe first. + +## A typing change should not change runtime behavior + +The second axis, and the one whose defects reach runtime rather than staying in +the type layer. A migration PR is +allowed to add annotations; it is not allowed to change what the program *does*. +Four patterns to grep the diff for, all of which look like typing work: + +1. **A literal replaced by a runtime lookup.** `['IFRAME_SCRIPTING']` becoming + `[SomeApi.Reason.IFRAME_SCRIPTING]` adds a dependency on that object existing at + runtime. **Read the declaration first** — if the parameter is a template-literal + string type (`` `${SomeEnum}`[] ``), the literal already type-checked and the + swap bought nothing. +2. **A default parameter or fallback deleted.** `function f(x = {})` → `function f(x: T)` + removes a guard. Ask what the guard was *for*: a `| undefined` the new type just + dropped is the first candidate. Then check reachability rather than assuming either way. +3. **A call made optional.** `obj.method()` → `obj.method?.()`, added to satisfy a + hand-written `| undefined`, converts a **throw into a silent no-op**. The loud + failure was load-bearing; now the same state produces no signal at all. +4. **A widened local to keep a check alive.** `let name: string | undefined` on a + value the authoritative type calls `string`, so that an `=== undefined` branch + still compiles. If the runtime check is genuinely needed, the *input* type is + wrong — fix that instead of widening downstream. + +For each hit: state whether it is reachable, and say so plainly either way. "I +traced it and it is inert today" is a useful review finding. "This might be a bug" +is not. + +### Silent failure modes deserve their own pass + +Ask where a newly-introduced failure would *surface*. A change inside a +`try { … } catch { captureException(e); return; }` degrades a feature without +crashing — nothing goes red, no test fails, and the only signal is an error-tracker +entry nobody is watching. The same edit in a hot path would be caught in minutes. +Weight findings by observability, not just by likelihood: **an unlikely failure in a +swallowed path can outrank a likely one in a loud path.** + +## Why the build stays green regardless + +- The hand-written type **compiles by construction** — that is why it was written. +- With `checkJs` off, a type written for a function whose callers are still `.js` + is checked against **nothing** and can drift indefinitely. +- A value that arrives as `any` silently satisfies any annotation. + +So cite the green build as the *premise* of the finding, never as counter-evidence. + +## Examples + +**Worked example** — 12 hand-written types across a JS→TS migration PR, 5 confirmed +divergences and 5 cleared falsifiers, with the verbatim two-arm output: +[references/worked-example.md](references/worked-example.md). + +**Repo notes** for MetaMask Extension (heap, probe location, `checkJs` status): +[references/metamask-extension.md](references/metamask-extension.md). + +``` +User: "Validate this TS migration PR — is it really mechanical?" +Agent: inventories the 12 new types → finds the authoritative source for 9 → + writes 6 probes → Arm A silent, Arm B reports 6 diagnostics → isolates + one that fired for the wrong reason → reports 5 findings, 5 clearances. +``` + +## Troubleshooting + +### Arm A is not silent + +**Problem:** the untouched tree already emits diagnostics, so Arm B is unattributable. +**Fix:** pin the toolchain, install against the PR's own lockfile, raise the heap, or +narrow the project. If it cannot be made silent, the lane is **inconclusive** — say +so; do not report Arm B's count as findings. + +### `tsc` runs out of memory + +**Problem:** `FATAL ERROR: Ineffective mark-compacts near heap limit`. +**Fix:** raise the heap — `NODE_OPTIONS='--max-old-space-size=9216'`. Note the OOM +exits non-zero *without* type diagnostics, so a naive exit-code check reads it as +"errors found." Always look at the output, not just the status. + +### A probe errors, but not for the claimed reason + +**Problem:** the diagnostic is about an earlier property, not the claim. +**Fix:** neutralise the earlier cause (`NonNullable<…>`, a narrow assertion) and +re-run. If the claim then compiles clean, the claim was **wrong** — report it as cleared. + +### There is no authoritative source + +Not a failure. Hand-writing is correct where nothing defines the shape; record the +reason (package ships no types, lib not in `tsconfig.lib`, new boundary) so the next +reviewer doesn't re-litigate it. + +## Related + +- [contributor-docs `docs/typescript.md`](https://github.com/MetaMask/contributor-docs/blob/main/docs/typescript.md) — the write-side rule this proves. +- `unit-testing`, `integration-test` — for behavior claims; this skill proves *types*. From a66e72ad2fec9c0fb798f3824b6efde84894173a Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 13:21:53 -0400 Subject: [PATCH 028/135] Move `falsifying-test` to the `testing` domain MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `pr-validate`'s other engines are placed by subject — `react-render-proof` in `performance`, `memory-leak-hunt` in `stability`, `supply-chain-audit` in `security`. This one was placed by its caller instead. Writing a test that fails on the base commit and passes on the branch is a testing technique, and `testing/` already holds techniques of that kind (`e2e-flakiness-patterns`, `test-layer-placement`, `performance-testing`), while `pr-workflow/` is uniformly PR-lifecycle stages. Both references to it are by name rather than path, so nothing needed updating. --- domains/{pr-workflow => testing}/skills/falsifying-test/skill.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename domains/{pr-workflow => testing}/skills/falsifying-test/skill.md (100%) diff --git a/domains/pr-workflow/skills/falsifying-test/skill.md b/domains/testing/skills/falsifying-test/skill.md similarity index 100% rename from domains/pr-workflow/skills/falsifying-test/skill.md rename to domains/testing/skills/falsifying-test/skill.md From e1cd0095c5d090eed8d3f66145d92b1d0c0942ef Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 13:22:30 -0400 Subject: [PATCH 029/135] Move the benchmarking skills out to the performance PR MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `benchmark-design` and `browser-extension-profiling` both cover E2E benchmarking with statistical rigor, and they sat in different domains here while the rest of the measurement work — `data-analysis`, `react-render-proof` — lives in the performance PR. Splitting one subject across two PRs made both harder to review. Moves `benchmark-design`, its `benchmark-statistical-hygiene` knowledge, and `browser-extension-profiling`. Neither skill referenced the other by path, so nothing needed rewriting. What remains is the extension-runtime work this PR is named for. --- .../browser-extension-profiling/skill.md | 65 ----------------- .../benchmark-statistical-hygiene.md | 45 ------------ .../testing/skills/benchmark-design/skill.md | 71 ------------------- 3 files changed, 181 deletions(-) delete mode 100644 domains/performance/skills/browser-extension-profiling/skill.md delete mode 100644 domains/testing/knowledge/benchmark-statistical-hygiene.md delete mode 100644 domains/testing/skills/benchmark-design/skill.md diff --git a/domains/performance/skills/browser-extension-profiling/skill.md b/domains/performance/skills/browser-extension-profiling/skill.md deleted file mode 100644 index d956dea8..00000000 --- a/domains/performance/skills/browser-extension-profiling/skill.md +++ /dev/null @@ -1,65 +0,0 @@ ---- -maturity: experimental -name: browser-extension-profiling -description: Compare browser extension performance between branches using WDYR, React DevTools Profiler, and E2E benchmarks with statistical rigor. ---- - -# Browser Extension Profiling - -Methodology for profiling and comparing extension performance across branches or commits. - -## When To Use - -- Validating that a refactor reduces unnecessary re-renders (needs before/after comparison) -- Establishing baseline metrics for a performance initiative -- Investigating a reported UI slowdown in the extension - -## Do Not Use When - -- Single-run comparisons — statistical significance requires ≥10 runs per scenario -- The change touches only non-render paths (background scripts, network with no UI impact) -- Target behavior is server-side latency, not UI rendering - -## Workflow - -1. **Build both branches** with `yarn build:test` on the same machine and Chrome version - -2. **WDYR profiling** (unnecessary re-render counts) - ```bash - ENABLE_WHY_DID_YOU_RENDER=true yarn start - ``` - Flags to watch: - - `different objects that are equal by value` → object recreation - - `different functions with the same name` → callback recreation - - `props object itself changed but values equal` → parent cascade - -3. **React DevTools Profiler** for flame graphs and commit timings - ```bash - yarn devtools:react - ``` - -4. **E2E benchmarks** for scenario durations - ```bash - yarn test:e2e:benchmark - ``` - -5. **Collect ≥10 runs** per scenario. Discard top/bottom 10%. Report mean, median, stddev, p75, p95. - -6. **Statistical threshold:** Cohen's d > 0.5 for a meaningful difference. - -## Common Pitfalls - -| Mistake | Correct Approach | -|---------|-----------------| -| Running branches on different machines or Chrome versions | Same machine, same Chrome, no other apps running | -| Pooling all runs including noisy late-session ones | Compute per-round stats first; report cleanest signal with explicit round attribution | -| Reporting absolute re-render counts without scenario context | Normalize per-action; cascade fixes show multiplied impact at root | -| Skipping cache and state reset between runs | Clear browser cache, reset extension state for each run | - -## Pre-Profiling Checklist - -- [ ] Both branches built with `yarn build:test` -- [ ] Same machine, same Chrome version -- [ ] No other tabs or applications running -- [ ] WDYR enabled: `ENABLE_WHY_DID_YOU_RENDER=true` -- [ ] Cache and extension state cleared between runs diff --git a/domains/testing/knowledge/benchmark-statistical-hygiene.md b/domains/testing/knowledge/benchmark-statistical-hygiene.md deleted file mode 100644 index 7015bbc5..00000000 --- a/domains/testing/knowledge/benchmark-statistical-hygiene.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -name: benchmark-statistical-hygiene -domain: testing -description: Three patterns for defensible A/B benchmark results: per-round best-subset reporting, fix-vector isolation, and artifact sort-order trap. ---- - -# Benchmark Statistical Hygiene - -Three patterns that prevent the most common classes of invalid benchmark conclusions. - -## Pattern: Per-Round Best-Subset Reporting - -Later benchmark rounds accumulate system noise (background load, memory pressure, I/O contention). Pooling all rounds blindly treats noisy late-session data equally with clean early-session data. - -**Instead:** Compute per-round statistics first, then report the cleanest signal per metric with explicit round attribution. - -``` -Round 1 (clean): metric X → treatment wins, p=0.04, d=-1.7 -Round 2 (moderate): metric X → treatment wins, p=0.08, d=-0.9 -Round 3 (noisy): metric X → no effect, p=0.90, d=+0.04 - -Pooled (all): metric X → no effect, p=0.50, d=-0.2 ← signal destroyed - -Correct report: "X improved 49% (Round 1, n=5, p=0.04, d=-1.7). - Pooled n=20 loses significance due to Round 3 outliers." -``` - -A small N with large effect size (|d| > 1.5, p < 0.05) is more defensible than a large N where noise has diluted significance to nothing. - -## Pattern: Isolate the Fix Vector - -Design each benchmark flow to exercise the optimization's specific input vector as its primary signal source. Incidental coverage produces fragile results where signal-to-noise depends on how much of the measured duration is optimization-affected. - -| | Weak | Strong | -|-|------|--------| -| Design | End-to-end flow that incidentally triggers target once among many other operations | Rapid sequence of actions each triggering the target with minimal other overhead | -| Optimization signal | ~5% of measured duration | ~80% of measured duration | - -## Pattern: Artifact Sort-Order Trap - -Unpadded iteration numbers in filenames break lexicographic sorting: `iteration-1, iteration-10, iteration-2, ...` interleaves data from different rounds when processed in glob order. - -**Rule:** When processing sequentially-numbered artifacts, extract the embedded timestamp or numeric value for sorting. Never rely on string sort order when numbers cross digit boundaries. - -**Diagnosis:** If pipeline results look implausible (p-values that are too perfect, round-level stats that don't match spot checks), print the actual file ordering the pipeline used. Check for lexicographic interleaving at digit boundaries. Re-sort by extracted timestamp or zero-padded key. diff --git a/domains/testing/skills/benchmark-design/skill.md b/domains/testing/skills/benchmark-design/skill.md deleted file mode 100644 index 578691df..00000000 --- a/domains/testing/skills/benchmark-design/skill.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -maturity: experimental -name: benchmark-design -description: Design, run, and analyze E2E performance benchmarks — session hygiene, per-round reporting, artifact grouping ---- - -# Benchmark Design - -## When To Use - -- Writing a new E2E benchmark flow -- Interpreting or presenting benchmark results -- Adding new metrics to existing benchmarks -- Diagnosing unexpected benchmark results - -## Do Not Use When - -- Adding unit, integration, or correctness E2E tests -- Profiling a single user-reported slowdown (use `selector-anti-pattern-review`) -- Writing micro-benchmarks outside the E2E harness - -## Workflow - -1. **Design the flow** — target ONE optimization vector per benchmark. Maximize ratio of optimization-affected time to total measured time. -2. **Run reference benchmarks first** in any session — session state degrades over time. -3. **Compute per-round statistics** before pooling. Check each round for stability (CV < 0.3 is a reasonable threshold). -4. **Group artifacts by timestamp**, not filename sort order. -5. **Report per-metric best subset** with explicit round attribution. Show pooled data as supplementary. - -## Flow Design by Optimization Type - -| Optimization | Primary cascade vector | Recommended flow | -|---|---|---| -| Selector memoization | State mutations | Multi-confirmation queue | -| Context memoization | Any state update | Account switching cycle | -| HOC stabilization | Route changes | Rapid route cycling (8+ transitions) | -| Dead code removal | Navigation | Return-to-home timer | - -## Session Hygiene - -System state degrades over long sessions — background load and memory pressure inflate variance and can **invert** treatment effects. - -- Run reference/critical benchmarks first -- If a late round contradicts clean earlier rounds, suspect session degradation before re-running the full suite - -## Artifact Grouping - -Filenames use `{test}-iteration-{N}-{ISO-timestamp}.json`. Unpadded N produces incorrect lexicographic sort. - -```javascript -// Extract seconds-since-midnight for round assignment -const match = filename.match(/T(\d{2})-(\d{2})-(\d{2})/); -const secondsOfDay = +match[1] * 3600 + +match[2] * 60 + +match[3]; -// Group by time range — never by filename position or array index -``` - -## Adding Metrics - -Extend `collectMetrics()` in `test/e2e/webdriver/driver.js` and register the metric key in `test/e2e/benchmarks/utils/constants.ts` → `ALL_METRICS`. - -- **Performance API metrics** (paint, navigation timing): collect directly inside `collectMetrics()` via `window.performance.getEntriesByType(...)`. -- **Long Task / TBT metrics**: already wired — `collectMetrics()` reads `window.stateHooks.getLongTaskMetricsWithTBT()`. Adding new long-task-derived metrics requires extending the `stateHooks` observer, not the driver. - -## Common Pitfalls - -| Mistake | Correct Approach | -|---------|-----------------| -| Pool all rounds before checking per-round stats | Per-round first — late-session noise can invert the treatment effect | -| Sort artifacts by filename | Extract ISO timestamp; sort by numeric time value | -| Benchmark flow that exercises multiple vectors | One vector per flow — mixed flows produce ambiguous signal | -| Report pooled p-value as primary result | Report cleanest per-metric signal with round attribution; pooled is supplementary | From 8f6762ff33a0308c700c52de58e954048222e1b7 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 13:22:51 -0400 Subject: [PATCH 030/135] Take in the benchmarking skills MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `benchmark-design` and `browser-extension-profiling` are the capture half of the measurement work already here: `data-analysis` turns raw numbers into a defensible before/after, and `react-render-proof` proves a specific change moved work. Both arrived from the platform PR, which shipped them alongside unrelated extension-runtime skills. `benchmark-design` stays in `testing` — that domain already owns benchmark methodology (`performance-testing`) — and brings its `benchmark-statistical-hygiene` knowledge with it. The PR spans two domains because the subject does, not because it is a grab bag. --- .../browser-extension-profiling/skill.md | 65 +++++++++++++++++ .../benchmark-statistical-hygiene.md | 45 ++++++++++++ .../testing/skills/benchmark-design/skill.md | 71 +++++++++++++++++++ 3 files changed, 181 insertions(+) create mode 100644 domains/performance/skills/browser-extension-profiling/skill.md create mode 100644 domains/testing/knowledge/benchmark-statistical-hygiene.md create mode 100644 domains/testing/skills/benchmark-design/skill.md diff --git a/domains/performance/skills/browser-extension-profiling/skill.md b/domains/performance/skills/browser-extension-profiling/skill.md new file mode 100644 index 00000000..d956dea8 --- /dev/null +++ b/domains/performance/skills/browser-extension-profiling/skill.md @@ -0,0 +1,65 @@ +--- +maturity: experimental +name: browser-extension-profiling +description: Compare browser extension performance between branches using WDYR, React DevTools Profiler, and E2E benchmarks with statistical rigor. +--- + +# Browser Extension Profiling + +Methodology for profiling and comparing extension performance across branches or commits. + +## When To Use + +- Validating that a refactor reduces unnecessary re-renders (needs before/after comparison) +- Establishing baseline metrics for a performance initiative +- Investigating a reported UI slowdown in the extension + +## Do Not Use When + +- Single-run comparisons — statistical significance requires ≥10 runs per scenario +- The change touches only non-render paths (background scripts, network with no UI impact) +- Target behavior is server-side latency, not UI rendering + +## Workflow + +1. **Build both branches** with `yarn build:test` on the same machine and Chrome version + +2. **WDYR profiling** (unnecessary re-render counts) + ```bash + ENABLE_WHY_DID_YOU_RENDER=true yarn start + ``` + Flags to watch: + - `different objects that are equal by value` → object recreation + - `different functions with the same name` → callback recreation + - `props object itself changed but values equal` → parent cascade + +3. **React DevTools Profiler** for flame graphs and commit timings + ```bash + yarn devtools:react + ``` + +4. **E2E benchmarks** for scenario durations + ```bash + yarn test:e2e:benchmark + ``` + +5. **Collect ≥10 runs** per scenario. Discard top/bottom 10%. Report mean, median, stddev, p75, p95. + +6. **Statistical threshold:** Cohen's d > 0.5 for a meaningful difference. + +## Common Pitfalls + +| Mistake | Correct Approach | +|---------|-----------------| +| Running branches on different machines or Chrome versions | Same machine, same Chrome, no other apps running | +| Pooling all runs including noisy late-session ones | Compute per-round stats first; report cleanest signal with explicit round attribution | +| Reporting absolute re-render counts without scenario context | Normalize per-action; cascade fixes show multiplied impact at root | +| Skipping cache and state reset between runs | Clear browser cache, reset extension state for each run | + +## Pre-Profiling Checklist + +- [ ] Both branches built with `yarn build:test` +- [ ] Same machine, same Chrome version +- [ ] No other tabs or applications running +- [ ] WDYR enabled: `ENABLE_WHY_DID_YOU_RENDER=true` +- [ ] Cache and extension state cleared between runs diff --git a/domains/testing/knowledge/benchmark-statistical-hygiene.md b/domains/testing/knowledge/benchmark-statistical-hygiene.md new file mode 100644 index 00000000..7015bbc5 --- /dev/null +++ b/domains/testing/knowledge/benchmark-statistical-hygiene.md @@ -0,0 +1,45 @@ +--- +name: benchmark-statistical-hygiene +domain: testing +description: Three patterns for defensible A/B benchmark results: per-round best-subset reporting, fix-vector isolation, and artifact sort-order trap. +--- + +# Benchmark Statistical Hygiene + +Three patterns that prevent the most common classes of invalid benchmark conclusions. + +## Pattern: Per-Round Best-Subset Reporting + +Later benchmark rounds accumulate system noise (background load, memory pressure, I/O contention). Pooling all rounds blindly treats noisy late-session data equally with clean early-session data. + +**Instead:** Compute per-round statistics first, then report the cleanest signal per metric with explicit round attribution. + +``` +Round 1 (clean): metric X → treatment wins, p=0.04, d=-1.7 +Round 2 (moderate): metric X → treatment wins, p=0.08, d=-0.9 +Round 3 (noisy): metric X → no effect, p=0.90, d=+0.04 + +Pooled (all): metric X → no effect, p=0.50, d=-0.2 ← signal destroyed + +Correct report: "X improved 49% (Round 1, n=5, p=0.04, d=-1.7). + Pooled n=20 loses significance due to Round 3 outliers." +``` + +A small N with large effect size (|d| > 1.5, p < 0.05) is more defensible than a large N where noise has diluted significance to nothing. + +## Pattern: Isolate the Fix Vector + +Design each benchmark flow to exercise the optimization's specific input vector as its primary signal source. Incidental coverage produces fragile results where signal-to-noise depends on how much of the measured duration is optimization-affected. + +| | Weak | Strong | +|-|------|--------| +| Design | End-to-end flow that incidentally triggers target once among many other operations | Rapid sequence of actions each triggering the target with minimal other overhead | +| Optimization signal | ~5% of measured duration | ~80% of measured duration | + +## Pattern: Artifact Sort-Order Trap + +Unpadded iteration numbers in filenames break lexicographic sorting: `iteration-1, iteration-10, iteration-2, ...` interleaves data from different rounds when processed in glob order. + +**Rule:** When processing sequentially-numbered artifacts, extract the embedded timestamp or numeric value for sorting. Never rely on string sort order when numbers cross digit boundaries. + +**Diagnosis:** If pipeline results look implausible (p-values that are too perfect, round-level stats that don't match spot checks), print the actual file ordering the pipeline used. Check for lexicographic interleaving at digit boundaries. Re-sort by extracted timestamp or zero-padded key. diff --git a/domains/testing/skills/benchmark-design/skill.md b/domains/testing/skills/benchmark-design/skill.md new file mode 100644 index 00000000..578691df --- /dev/null +++ b/domains/testing/skills/benchmark-design/skill.md @@ -0,0 +1,71 @@ +--- +maturity: experimental +name: benchmark-design +description: Design, run, and analyze E2E performance benchmarks — session hygiene, per-round reporting, artifact grouping +--- + +# Benchmark Design + +## When To Use + +- Writing a new E2E benchmark flow +- Interpreting or presenting benchmark results +- Adding new metrics to existing benchmarks +- Diagnosing unexpected benchmark results + +## Do Not Use When + +- Adding unit, integration, or correctness E2E tests +- Profiling a single user-reported slowdown (use `selector-anti-pattern-review`) +- Writing micro-benchmarks outside the E2E harness + +## Workflow + +1. **Design the flow** — target ONE optimization vector per benchmark. Maximize ratio of optimization-affected time to total measured time. +2. **Run reference benchmarks first** in any session — session state degrades over time. +3. **Compute per-round statistics** before pooling. Check each round for stability (CV < 0.3 is a reasonable threshold). +4. **Group artifacts by timestamp**, not filename sort order. +5. **Report per-metric best subset** with explicit round attribution. Show pooled data as supplementary. + +## Flow Design by Optimization Type + +| Optimization | Primary cascade vector | Recommended flow | +|---|---|---| +| Selector memoization | State mutations | Multi-confirmation queue | +| Context memoization | Any state update | Account switching cycle | +| HOC stabilization | Route changes | Rapid route cycling (8+ transitions) | +| Dead code removal | Navigation | Return-to-home timer | + +## Session Hygiene + +System state degrades over long sessions — background load and memory pressure inflate variance and can **invert** treatment effects. + +- Run reference/critical benchmarks first +- If a late round contradicts clean earlier rounds, suspect session degradation before re-running the full suite + +## Artifact Grouping + +Filenames use `{test}-iteration-{N}-{ISO-timestamp}.json`. Unpadded N produces incorrect lexicographic sort. + +```javascript +// Extract seconds-since-midnight for round assignment +const match = filename.match(/T(\d{2})-(\d{2})-(\d{2})/); +const secondsOfDay = +match[1] * 3600 + +match[2] * 60 + +match[3]; +// Group by time range — never by filename position or array index +``` + +## Adding Metrics + +Extend `collectMetrics()` in `test/e2e/webdriver/driver.js` and register the metric key in `test/e2e/benchmarks/utils/constants.ts` → `ALL_METRICS`. + +- **Performance API metrics** (paint, navigation timing): collect directly inside `collectMetrics()` via `window.performance.getEntriesByType(...)`. +- **Long Task / TBT metrics**: already wired — `collectMetrics()` reads `window.stateHooks.getLongTaskMetricsWithTBT()`. Adding new long-task-derived metrics requires extending the `stateHooks` observer, not the driver. + +## Common Pitfalls + +| Mistake | Correct Approach | +|---------|-----------------| +| Pool all rounds before checking per-round stats | Per-round first — late-session noise can invert the treatment effect | +| Sort artifacts by filename | Extract ISO timestamp; sort by numeric time value | +| Benchmark flow that exercises multiple vectors | One vector per flow — mixed flows produce ambiguous signal | +| Report pooled p-value as primary result | Report cleanest per-metric signal with round attribution; pooled is supplementary | From 0f740f2f3707ded5b29c180b021659fce51f87f9 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 13:26:17 -0400 Subject: [PATCH 031/135] Drop `resilient-api-collection` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Generic scripting guidance — paginate an API, handle rate limits, retry transient errors — with no MetaMask specificity and no relationship to this PR's extension-runtime subject. `coding/` otherwise holds MetaMask internals. Nothing referenced it and it referenced nothing, so removal is self-contained. The content stays in this branch's history if it is wanted later. --- .../skills/resilient-api-collection/skill.md | 188 ------------------ 1 file changed, 188 deletions(-) delete mode 100644 domains/coding/skills/resilient-api-collection/skill.md diff --git a/domains/coding/skills/resilient-api-collection/skill.md b/domains/coding/skills/resilient-api-collection/skill.md deleted file mode 100644 index 6d00ca51..00000000 --- a/domains/coding/skills/resilient-api-collection/skill.md +++ /dev/null @@ -1,188 +0,0 @@ ---- -name: resilient-api-collection -description: Build resilient data collection scripts that paginate APIs, handle rate limits, and retry transient errors. Use when writing scrapers, API collectors, data pipelines, or any script that fetches paginated data from external APIs (GitHub GraphQL, REST APIs, etc.). ---- - -# Resilient API Collection Scripts - -## Core Architecture - -Every collection script needs these layers: - -``` -run_query() → single request with retry + error classification -fetch_all_pages() → pagination loop with adaptive page sizing -main() → orchestration, dedup, persistence -``` - -## 1. Error Classification - -Classify errors **before** choosing a recovery strategy. Different errors need different fixes. - -| Error Type | Signal | Recovery | -|---|---|---| -| **Resource/complexity limit** | Query too expensive for server | Reduce page size | -| **Rate limit** (primary) | 429, `X-RateLimit-Remaining: 0` | Wait until reset time | -| **Rate limit** (secondary) | 403 + "secondary rate limit" | Exponential backoff (start 60s) | -| **Transient server error** | 502, 503, 504, stream reset | Retry with exponential backoff | -| **Client error** | 400, 401, 404 | Don't retry — fix the request | - -### CLI tools hide error details - -Tools like `gh`, `curl`, `httpie` surface errors differently than raw HTTP responses: - -- **`gh api graphql`**: "Resource limits exceeded" appears in `stderr` with non-zero exit code, NOT in the JSON response `errors` array. Always check `stderr` first, before checking `returncode`. -- Rate limit info may be in response headers (not visible via CLI) or in error messages. - -```python -# Check stderr BEFORE returncode — some errors are in stderr even on exit 0 -stderr = result.stderr.strip() - -if "Resource limits" in stderr or "resource limit" in stderr.lower(): - return RESOURCE_LIMIT_SIGNAL # caller reduces page size - -if result.returncode == 0: - data = json.loads(result.stdout) - # Also check JSON errors (some APIs put limits here) - if "errors" in data: - msg = data["errors"][0].get("message", "") - if "Resource limits" in msg or "timeout" in msg.lower(): - return RESOURCE_LIMIT_SIGNAL - return data - -# Classify non-zero exit -is_transient = any(s in stderr for s in [ - "502", "503", "504", "429", "rate limit", - "secondary", "stream error", "CANCEL" -]) -``` - -## 2. Retry with Exponential Backoff - -```python -MAX_RETRIES = 5 -INITIAL_BACKOFF = 5 # seconds - -for attempt in range(1, MAX_RETRIES + 1): - result = execute_request(...) - - if success: - return result - if is_resource_limit(error): - return RESOURCE_LIMIT_SIGNAL # don't retry, reduce page size - if not is_transient(error): - return None # permanent failure - if attempt == MAX_RETRIES: - return None # exhausted - - wait = INITIAL_BACKOFF * (2 ** (attempt - 1)) - log(f"Transient error (attempt {attempt}/{MAX_RETRIES}), retrying in {wait}s") - time.sleep(wait) -``` - -Key: resource-limit errors should NOT be retried — the same query will fail identically. Signal the caller to reduce page size instead. - -## 3. Adaptive Page Sizing - -Start conservatively. Halve on resource-limit errors. Set a floor. - -```python -MIN_PAGE_SIZE = 5 -MAX_REDUCTIONS = 4 -page_size = 50 # not 100 — nested sub-selections multiply complexity - -while has_more_pages: - data = run_query(..., page_size=page_size) - - if data == RESOURCE_LIMIT_SIGNAL: - reductions += 1 - if reductions > MAX_REDUCTIONS or page_size <= MIN_PAGE_SIZE: - break # can't go smaller - page_size = max(MIN_PAGE_SIZE, page_size // 2) - time.sleep(10) # cool down before retry - continue # retry same page with smaller size - - # process nodes, advance cursor... - time.sleep(2) # inter-page delay to avoid secondary rate limits -``` - -### Why 50, not 100? - -GraphQL query cost = `nodes × sub-selections`. A query fetching 100 PRs with `reviews(first:50)`, `participants(first:30)`, `commits(first:1)` easily exceeds GitHub's 500K node limit. Starting at 50 avoids most resource-limit errors. - -## 4. Deduplication and Incremental Collection - -Always dedup by natural key before writing. This lets re-runs extend existing data. - -```python -def dedup(existing, new, key_fn): - by_key = {} - for item in existing: - by_key[key_fn(item)] = item - for item in new: - by_key[key_fn(item)] = item # new overwrites old - return list(by_key.values()) - -# On write: -existing = load_json(path) if os.path.exists(path) else [] -final = dedup(existing, new_items, key_fn=lambda x: (x["repo"], x["number"])) -save_json(path, final) -``` - -## 5. Observability - -### Force unbuffered output - -Python buffers stdout when output is captured (subprocess, pipe, file redirect). Progress lines never appear. - -```python -import sys -sys.stdout.reconfigure(line_buffering=True) -# OR run with: python3 -u script.py -``` - -### Log structure for monitoring - -``` -=== repo-name (query-type) === - Page 1: 50 nodes, hasNext=True (size=50) - Page 2: 50 nodes, hasNext=True (size=50) - Resource limit exceeded (page_size=50), signaling page-size reduction - Reducing page size to 25 and retrying page 3 (reduction 1/4) - Page 3: 25 nodes, hasNext=True (size=25) - ... - Total: 430 items, collected 430, 3169 sub-items -``` - -Every log line should include: page number, items returned, whether there are more pages, and current page size. - -## 6. Inter-Page Delays - -GitHub's secondary rate limit triggers on sustained request volume, not individual request cost. Add 2-3s between pages. - -```python -PAGE_DELAY = 2 # seconds - -# After each successful page: -time.sleep(PAGE_DELAY) - -# After a resource-limit reduction: -time.sleep(INITIAL_BACKOFF * 2) # longer cooldown -``` - -## Checklist - -When writing a collection script, verify: - -- [ ] Error classification distinguishes resource-limit from rate-limit from transient -- [ ] Resource-limit errors reduce page size (not retry same query) -- [ ] Transient errors retry with exponential backoff -- [ ] Non-retryable errors fail fast -- [ ] Page size starts at 50 or lower for nested queries -- [ ] Page size has a floor (5-10) and max-reduction cap -- [ ] Inter-page delay prevents secondary rate limits -- [ ] Output is unbuffered (`-u` flag or `reconfigure`) -- [ ] Each log line includes page number, count, hasNext, page size -- [ ] Data is deduped by natural key before writing -- [ ] Re-runs merge with existing data (incremental collection) -- [ ] Collection log records run metadata (timestamps, repos, filters) From 45f78e1271e441383b31b70fe4aa842fbda021db Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 13:33:28 -0400 Subject: [PATCH 032/135] Make the keepalive claim re-verifiable instead of asserted MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The keepalive row was the one entry here resting on a live implementation detail rather than on an absent handler or persistent storage, and it was cited as `background.js:750-758` — a line range that drifts. If the interval grows past the idle timeout or the keepalive is removed, the conclusion inverts from "eviction is prevented" to "eviction happens routinely", and a skill still asserting the first is worse than no skill. Replaces the line range with a symbol grep (`saveTimestamp`, `SAVE_TIMESTAMP_INTERVAL_MS`), names the two conditions the conclusion depends on — sub-idle-timeout interval, and an extension API call rather than a bare timer — and records what was verified, against which commit. --- .../extension-lifecycle-decoupling/skill.md | 33 +++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/domains/platform/skills/extension-lifecycle-decoupling/skill.md b/domains/platform/skills/extension-lifecycle-decoupling/skill.md index ef908aed..cd7e5950 100644 --- a/domains/platform/skills/extension-lifecycle-decoupling/skill.md +++ b/domains/platform/skills/extension-lifecycle-decoupling/skill.md @@ -43,7 +43,35 @@ Before claiming a platform lifecycle event causes application behavior: | SW eviction triggers lock | No `onSuspend` lock handler — SW eviction does NOT trigger lock | | Timers lost on SW restart | Auto-lock uses Chrome Alarms API — persists across SW restarts | | State lost on SW restart | Wallet state persists in `chrome.storage.session` and IndexedDB | -| SW evicts frequently during active use | `background.js:750-758` calls `browser.storage.session.set` every 2s. Each `chrome.*`/`browser.*` call resets the 30s idle timer, so active-session eviction is effectively prevented. Cold starts (browser launch, extension reload) still happen. See `mv3-service-worker` knowledge for mechanism and verification discipline | +| SW evicts frequently during active use | A keepalive writes `browser.storage.session.set` on a short interval, and each `chrome.*`/`browser.*` call resets the 30s idle timer — so active-session eviction is effectively prevented. Cold starts (browser launch, extension reload) still happen. **Re-verify before relying on it — see below.** See `mv3-service-worker` knowledge for mechanism and verification discipline | + +### Re-verify the keepalive before reasoning from it + +This row is the only one that depends on a *current implementation detail* rather than on +absent handlers or persistent storage, and it is the one that inverts if the implementation +moves. If the interval grows past the idle timeout, or the keepalive is removed, the honest +answer flips from "eviction is prevented" to "eviction happens routinely" — and a skill that +still asserts the first would be worse than no skill. + +Confirm it in the target repo before drawing conclusions: + +```bash +# the keepalive writer and its cadence — symbol names, not line numbers +grep -rn "saveTimestamp\|SAVE_TIMESTAMP_INTERVAL_MS" app/scripts/background.js +``` + +Two things make the conclusion hold, and both must still be true: + +1. The interval is **well under the ~30s idle timeout** (last verified: `2 * 1000` ms). +2. The callback performs an **extension API call** — `browser.storage.session.set` — since it + is the API call that resets the timer, not the timer firing. + +If either has changed, treat active-session eviction as live and re-derive the rest of this +table's consequences. + +*Verified against `metamask-extension` at `d4dd55f300a` (2026-07-30): +`SAVE_TIMESTAMP_INTERVAL_MS = 2 * 1000`, `setInterval(saveTimestamp, …)`, +`saveTimestamp` calling `browser.storage.session.set`.* ## Common Pitfalls @@ -52,4 +80,5 @@ Before claiming a platform lifecycle event causes application behavior: | "SW evicts N times/day → event fires N times/day" | Check if application code has handler for eviction | | Assume frequency from platform behavior | Grep for actual handler chains in `background.js`, `app-state-controller.ts` | | Conflate platform restart with application reset | Check which state is persisted vs re-initialized | -| "Keepalive uses `chrome.alarms`" | It does not — keepalive uses `browser.storage.session.set` at 2s cadence. `chrome.alarms` is used separately for auto-lock timers that must persist across SW restart | +| "Keepalive uses `chrome.alarms`" | It does not — keepalive works by making an extension API call (`browser.storage.session.set`) on a sub-idle-timeout interval. `chrome.alarms` is used separately, for auto-lock timers that must persist across SW restart | +| Citing this skill's keepalive claim without re-checking | It is the one row here that tracks a live implementation detail. Run the grep above; the conclusion inverts if the interval or the API call changes | From b576c7ec1b3d6e79de2ab5a984b389f9a170ad08 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 13:41:40 -0400 Subject: [PATCH 033/135] Point the extension overlays at `main`, not `develop` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `metamask-extension` moved its default branch to `main`; `develop` still exists but its last commit is 2026-01-15, so six links in the extension overlays resolved to code roughly six months stale. They loaded, which is why nothing caught it — a frozen branch is worse than a dead one here, since the reader gets plausible but outdated source. All five cited paths verified present on `main` (HTTP 200): `ui/`, `ui/hooks/`, `ui/selectors/`, `shared/lib/selectors/selector-creators.ts`, and `app/scripts/metamask-controller.js`. --- .../repos/metamask-extension.md | 4 ++-- .../repos/metamask-extension.md | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/domains/performance/skills/effect-anti-pattern-review/repos/metamask-extension.md b/domains/performance/skills/effect-anti-pattern-review/repos/metamask-extension.md index 67d41395..532b74c6 100644 --- a/domains/performance/skills/effect-anti-pattern-review/repos/metamask-extension.md +++ b/domains/performance/skills/effect-anti-pattern-review/repos/metamask-extension.md @@ -5,8 +5,8 @@ parent: effect-anti-pattern-review ## Paths -- Component sources: [`ui/`](https://github.com/MetaMask/metamask-extension/tree/develop/ui) -- Shared hooks: [`ui/hooks/`](https://github.com/MetaMask/metamask-extension/tree/develop/ui/hooks) +- Component sources: [`ui/`](https://github.com/MetaMask/metamask-extension/tree/main/ui) +- Shared hooks: [`ui/hooks/`](https://github.com/MetaMask/metamask-extension/tree/main/ui/hooks) ## Commands diff --git a/domains/performance/skills/selector-anti-pattern-review/repos/metamask-extension.md b/domains/performance/skills/selector-anti-pattern-review/repos/metamask-extension.md index aafb8e9d..b3ca4992 100644 --- a/domains/performance/skills/selector-anti-pattern-review/repos/metamask-extension.md +++ b/domains/performance/skills/selector-anti-pattern-review/repos/metamask-extension.md @@ -5,10 +5,10 @@ parent: selector-anti-pattern-review ## Paths -- Selector definitions: [`ui/selectors/`](https://github.com/MetaMask/metamask-extension/tree/develop/ui/selectors) -- Selector creators: [`shared/lib/selectors/selector-creators.ts`](https://github.com/MetaMask/metamask-extension/blob/develop/shared/lib/selectors/selector-creators.ts) — source of truth for `createSelector`, `createDeepEqualSelector`, `createResultEqualSelector`, `createShallowResultSelector` -- Controller state shape: [`app/scripts/metamask-controller.js`](https://github.com/MetaMask/metamask-extension/blob/develop/app/scripts/metamask-controller.js) -- Component consumption sites: anywhere under [`ui/`](https://github.com/MetaMask/metamask-extension/tree/develop/ui) that calls `useSelector` +- Selector definitions: [`ui/selectors/`](https://github.com/MetaMask/metamask-extension/tree/main/ui/selectors) +- Selector creators: [`shared/lib/selectors/selector-creators.ts`](https://github.com/MetaMask/metamask-extension/blob/main/shared/lib/selectors/selector-creators.ts) — source of truth for `createSelector`, `createDeepEqualSelector`, `createResultEqualSelector`, `createShallowResultSelector` +- Controller state shape: [`app/scripts/metamask-controller.js`](https://github.com/MetaMask/metamask-extension/blob/main/app/scripts/metamask-controller.js) +- Component consumption sites: anywhere under [`ui/`](https://github.com/MetaMask/metamask-extension/tree/main/ui) that calls `useSelector` ## Commands From 78a93b0062f690aefabcad9cf989b10d14951ff7 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 14:38:06 -0400 Subject: [PATCH 034/135] Add `D6` substitution A/B lane and sync `pr-validate` references MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `D6` covers claims where a PR hand-writes an artifact that restates an existing source — a type, schema, vendored constant, or checked-in policy. Both arms sit at the same commit and differ by a substitution rather than a ref, so there is no build or merge boundary to confound the result. --- .../references/evidence-catalog.md | 44 +++++++--- .../references/evidence-publishing.md | 81 ++++++++++++------- .../references/evidence-trustworthiness.md | 11 ++- .../pr-validate/references/lane-assertions.md | 2 +- .../pr-workflow/skills/pr-validate/skill.md | 68 ++++++++++------ 5 files changed, 133 insertions(+), 73 deletions(-) diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md b/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md index b34a5e20..8c8b7680 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md @@ -2,7 +2,7 @@ The menu of evidence kinds for validating a MetaMask **extension** PR, with **what each proves**, **how to capture it (verified against the live repo)**, and **when to reach for it**. AEP is the primary autonomous engine; the rest are complementary. The skill's job is to **match evidence to the claim** and to **proactively suggest kinds the author didn't think of**. -Pick the evidence that would **falsify the claim if it were false**. Prefer a lane that yields an artifact a reviewer can independently re-check (a link, an image, a number, a replayable trace) over prose. Don't run the whole menu — match, then capture. Capture commands are written against the `metamask-extension` checkout; verify script names against its `package.json` (they drift). +Pick the evidence that would **falsify the claim if it were false**. Prefer a lane that yields an artifact a reviewer can independently re-check (a link, an image, a number, a replayable trace) over prose. Don't run the whole menu — match, then capture. Capture commands cite `~/Code/metamask/metamask-extension`; verify script names against its `package.json` (they drift). Legend: **first-class lanes** are `##`-headed; closely-related variants are sub-bullets. Capture marked *(manual)* has no repo helper — it's a DevTools/CDP action. @@ -12,12 +12,12 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- ## A1. visual_validation — before/after screenshots - **Proves:** a visible UI change on the real surface. Deterministic state seed + agent navigation; PNG artifacts in `evidenceBundle.artifactRefs`. -- **Capture:** `taskClass: visual_validation`, `payload.prUrl` + `description` hint. See the *Preflight* and *Run mechanics* sections of [skill.md](../skill.md). +- **Capture:** `taskClass: visual_validation`, `payload.prUrl` + `description` hint. See [aep-local-run.md](aep-local-run.md). - **Reach for it:** anything a human would screenshot for the PR's `### After`. ## A2. perf_validation — falsifiable network/static/smoke assertions - **Proves:** non-visible behavior (hover-preload, no double-fetch, chunk membership, smoke boot). CDP netlog / phase segmentation / source-map membership. -- **Capture:** `taskClass: perf_validation` (needs a `yarn webpack --test` build). Confirm the perf-validation graph is registered in your AEP checkout; falls back to C6/D2 manually if it isn't present. +- **Capture:** `taskClass: perf_validation` (local/uncommitted graph; needs `yarn webpack --test`). Falls back to C6/D2 manually if the graph isn't present. ## A3. AEP bundle byproducts (free with any run) - Test results (`executionResult`/`checkResults`), diff stats, automated `reviewResult` findings, and the **LangSmith trace** of the run. Include the relevant subset; link the trace for auditability. @@ -44,7 +44,8 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- - **Reach for it:** every bug-fix PR. If you can't write a test that fails on main, question whether the fix addresses the reported bug. ## B7. Deterministic interleaving test (concurrency / temporal-ordering) ⭐ -- **Proves:** an ordering guarantee under interleaving — retry, cancellation, supersession, debounce, locks, queues, async state machines — where the correctness *is* the ordering under races, not a value. +- **Engine:** `race-condition-proof` — run it rather than hand-rolling the harness. +- **Proves:** an ordering guarantee under interleaving — retry, cancellation, supersession, debounce, locks, queues, async state machines — where the correctness *is* the ordering under races, not a value. Full category: `exogram-daemon/artifacts/evidence-taxonomy/category-concurrency-temporal-ordering.md`. - **Capture:** force each race deterministically — `jest.useFakeTimers()` + `advanceTimersByTimeAsync(DELAY)` to fire the delayed action at a known point; `Promise.all([opA, opB])` to overlap operations; `advanceTimersByTimeAsync(0)` to step to a precise interleaving point; then assert the ordering/cancellation outcome for **each** guarantee, including asymmetric ones (one path canceled → its recovery event `.not.toHaveBeenCalled()`; another must complete → `.toHaveBeenCalledWith(...)`). Corroborate with transition telemetry; for the integration path, a live forced-race capture (C8/CDP, the #44610 technique). - **Trust-gate:** the test must **actually interleave** — time advanced into the pending window, the superseding op injected *during* it. A sequential run exercises no race and is a vacuous green. Verify the interleaving, not just the assertion. @@ -72,7 +73,7 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- ## C2. Web vitals — INP / FCP / LCP / CLS - **Proves:** a user-centric metric moved. `ui/helpers/utils/web-vitals.ts` via `web-vitals/attribution` (attribution names the causing element). - **Capture:** `window.stateHooks.getWebVitalsMetrics()` (test/debug) → `{inp, fcp, lcp, cls, *Rating}`. Thresholds: INP good<200/poor>500, FCP<1800/3000, LCP<2500/4000, CLS<0.1/0.25. -- **Caveat:** **INP fires on all pages; FCP/LCP/CLS do not fire on popup pages** (sidepanel/E2E only). For extensions, INP is the high-value runtime metric. +- **Caveat:** **INP fires on all pages; FCP/LCP/CLS do not fire on popup pages** (sidepanel/E2E only). For extensions, INP is the high-value runtime metric (per exogram `web-vitals-runtime-metrics`). ## C3. Long-task / TBT - **Proves:** main-thread blocking during an interaction dropped. This is where **TBT** lives (the web-vitals lib lane does *not* collect TBT). @@ -80,16 +81,16 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- ## C4. React render & selector proof - **Engine: the `react-render-proof` skill.** Delegate the measurement to it; it runs the source/delivery/metric gates, derives the needle from real build output, repeats the capture, and returns a band (or "not resolvable at this n" with an MDE). pr-validate packages the result. -- **Proves:** a component/selector stopped over-rendering (cascade-amplification before/after). +- **Proves:** a component/selector stopped over-rendering (cascade-amplification before/after — exogram `react-redux-performance`). - **Capture:** WDYR via `ENABLE_WHY_DID_YOU_RENDER` (`.metamaskrc` or env) — wired in `app/scripts/development/wdyr.ts` (`trackAllPureComponents`); console logs each unnecessary re-render. `yarn devtools:react` for the Profiler flame graph. Selectors use `reselect`'s `createSelector`, which **does expose a real `.recomputations()` counter** — read it (sample on an interval if the count should visibly climb) rather than injecting a log into the selector body; an injected log is an authored claim, a library API is an observation. *(This entry previously said there was no built-in counter. There is.)* - **Bar:** the delivery check comes before the number. An arm whose manipulation cannot be observed in the built bundle produces a null indistinguishable from "small effect" — and reports as the second. ## C5. Benchmark A/B - **Proves:** a startup/journey/interaction timing moved, with a distribution not one sample. - **Capture:** `yarn test:e2e:benchmark` (`test/e2e/benchmarks/run-benchmark.ts`); presets in `shared/constants/benchmarks.ts` (`startupStandardHome`, `sendTransactions`, `swap`, `dappPageLoad`, …). -- **Caveat:** the rolling baseline (`MetaMask/extension_benchmark_stats`) can **silently freeze** behind a green check (the `store-benchmark-stats` step is `continue-on-error`; happened 2026-04-02, PR #42947). Prefer a **paired A/B** (build both refs now, compare directly) over the stored baseline. +- **Caveat:** the rolling baseline (`MetaMask/extension_benchmark_stats`) can **silently freeze** behind a green check (the `store-benchmark-stats` step is `continue-on-error`; happened 2026-04-02, PR #42947). Prefer a **paired A/B** (build both refs now, compare directly) over the stored baseline. See exogram `benchmark-baseline-staleness-paired-ab`. - **Treatment check first** — before trusting any delta, confirm the mechanism under test is actually active in each arm (split chunk present in head and absent in base; the span emitted; the flag evaluated). An arm without the treatment delivered is a no-op, not a control (2026-07-22, #42795). -- **A null needs its power stated** — "no change" and "underpowered" print the same result. When the run-to-run spread exceeds the effect under test, report **not resolvable at this n** and name the smallest detectable effect; never let it read as "no effect". Correcting a known bias (discarding a warm-up, alternating the starting arm) removes *that* bias and nothing more — it is not a trust gate, and the confounds you did not enumerate (thermal drift, background load, ordering within a round) stay live. +- **A null needs its power stated** — "no change" and "underpowered" print the same result. When the run-to-run spread exceeds the effect under test, report **not resolvable at this n** and name the smallest detectable effect; never let it read as "no effect". Correcting a known bias (discarding a warm-up, alternating the starting arm) removes *that* bias and nothing more — it is not a trust gate, and the confounds you did not enumerate (thermal drift, background load, ordering within a round) stay live. See exogram `removing-a-bias-is-not-establishing-validity` (2026-07-24). ### Capturing an authenticated view (the in-situ requirement) @@ -137,7 +138,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ ## C9. Retention-path analysis — memory leak from code ⭐ *(static; lead for leak claims)* - **Engine: the `memory-leak-hunt` skill.** For a memory-leak claim, delegate the analysis to `memory-leak-hunt` — it runs Phase-1 static pairing (and Phase-2 heap investigation if a primitive can't be paired) and returns the paired/unpaired sites + verdict. pr-validate keeps **memory leak** as the evidence category: it invokes the skill on the diff and packages the result (in-situ scan capture, plus the lifecycle test / retainer graph if Phase 2 ran) as the category's evidence. The lane spec below is the method that skill implements. -- **Proves:** "X is retained past its lifecycle boundary" / "collection Y grows unboundedly" — argued from code, no runtime needed. This is the lane that works at **review time** (does this PR *introduce* retention?) and leads fix-side validation (does the fix *break* the retention path?). C7 is the runtime corroborator, not the lead — leaks need many cycles to exceed noise. +- **Proves:** "X is retained past its lifecycle boundary" / "collection Y grows unboundedly" — argued from code, no runtime needed. This is the lane that works at **review time** (does this PR *introduce* retention?) and leads fix-side validation (does the fix *break* the retention path?). C7 is the runtime corroborator, not the lead — leaks need many cycles to exceed noise. Full category: `exogram-daemon/artifacts/evidence-taxonomy/category-memory-retention-from-code.md`. - **Capture — the holder → held → boundary triple, per suspect:** (1) the **holder** (listener, closure, module singleton, accumulating collection, timer); (2) the **held set** — the *specific* objects pinned (list the closure's captures; note when a closure links two objects' GC); (3) the **outlived boundary** (`destroy()`, stream close, instance replacement, request completion). Method: **pair every acquire with its release site** (`on`↔`removeListener`, push↔drain, assign↔null) — the absence of the pair, cited at the acquire site, IS the finding. Four canonical shapes: unbounded accumulator (defeated guard, no drain) · stale-instance listeners on replacement · unremoved listener + capture set · retention past `destroy()`. - **Scope to the diff, or you invent findings.** Classify every flagged primitive as *introduced by this PR* (in the added lines) vs *pre-existing* (already in the file). Charge only the introduced ones to the PR; report pre-existing un-paired primitives separately and uncharged. On extension#40684 the two new stream listeners each had a `removeListener` on `onStreamClosed` (the exact fix a reviewer suggested) and the new pending-request Map had its `.delete` — no leak introduced — while three pre-existing un-torn-down listeners were surfaced but left uncharged, matching how the human/bot reviewers treated them in-thread. This lane *is* the retention review automated; a heap snapshot (C7) is warranted only for an introduced primitive it cannot pair. - **Corroborate:** a falsifying lifecycle test (force the boundary, assert release — listener count zero, singleton nulled, collection drained); C7 heap-over-flow with the **retainer graph naming the same path** the static argument named. @@ -151,7 +152,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ ## D3. LavaMoat policy / supply-chain capability diff - **Engine: the `supply-chain-audit` skill** (umbrella — lockfile/manifest diff, advisories, Socket Security, install scripts), which delegates capability grants to **`lavamoat-policy-diligence`** (per-grant call-site justification). Delegate the dependency change to the umbrella; it returns a disposition per lane. pr-validate keeps **supply-chain capability diff** as the evidence category and packages the output. Note the lanes are independent: a clean policy diff does not mean a safe dependency, and a known CVE never appears as a new grant. -- **Proves:** a dependency change (bump/add/lockfile) grants **no *unjustified* new capability** — the supply-chain-risk lane. Note the bar: for a bump the policy *will* change, so "empty diff" is the WRONG test; the right test is **every new grant is justified by the dep's function**. The framing generalizes past LavaMoat to any capability-containment mechanism. +- **Proves:** a dependency change (bump/add/lockfile) grants **no *unjustified* new capability** — the supply-chain-risk lane. Note the bar: for a bump the policy *will* change, so "empty diff" is the WRONG test; the right test is **every new grant is justified by the dep's function**. Full category (trust-boundary framing, generalizes past LavaMoat to any capability-containment mechanism): `exogram-daemon/artifacts/evidence-taxonomy/category-supply-chain-capability-diff.md`. - **Capture:** **Prefer the CI-generated policy whenever one is available.** `@metamaskbot update-policies` regenerates the policy files from a real run of the code and `validate-lavamoat-policies` fails the build on drift, so the committed policy on a bot-run PR *is* the authoritative artifact — diff that. Regenerating locally when a current CI policy exists only re-does a machine that is already trusted, and a local run's provenance is weaker (your node/OS/lockfile resolution, not CI's). **Local regen is the fallback**, for when the bot hasn't run yet, the branch is unpushed, or you need a variant CI didn't cover: `yarn webpack:lavamoat:policy:build` (`:mv2` / `:mv3` for variants) over `lavamoat/webpack/build/policy.json` (+ `policy-override.json`). Either way, `git diff` the policy across **all 8 variants** (mv{2,3}/{main,beta,flask,experimental}) — a grant can appear in one and not others. Then audit **grant-by-grant**: new **globals** (`fetch`, `importScripts`, `WebAssembly`) / **builtins** (`fs`, `child_process`) on a dep that shouldn't need them, new **packages** edges to powerful APIs, or an identifier substitution (`pkgC>name` replacing `pkgB>pkgA>name` = possible dep swap). Falsifier = a surprising grant ("I wonder what it's using this for"). Guide: lavamoat.github.io/guides/policy-diff/. `allowScripts` in `package.json` gates install scripts. ## D4. Manifest permissions diff @@ -162,6 +163,25 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **Proves:** the change works across build types, not just main. - **Capture:** `yarn build:test:flask` / `:beta` / `:mv2` (`ENABLE_MV3=false`, Firefox). Run the relevant lane per variant when behavior is build-type-gated. +## D6. Authored-vs-authoritative substitution A/B ⭐ *(fixed head; lead for "the artifact restates a source" claims)* +- **Proves:** whether an artifact the PR *hand-wrote* agrees with the source it restates — a type vs the value's real type, a hand-maintained schema vs the generated one, a vendored constant vs the upstream export, a checked-in policy vs `update-policies` output. The finding is the **delta in a checker's output**, not a reading of the diff. +- **Shape:** both arms sit at the **same commit**; they differ by a *substitution*, not by a ref — so there is no build, no rebase, and no merge boundary to confound. + - **Arm A** — the PR as written, run through the checker. Must be **silent**. A non-empty Arm A means the instrument is broken and Arm B is unreadable (see trustworthiness gate item 19). + - **Arm B** — same tree, with the authored artifact replaced by the **derived** equivalent, exercised exactly as the real code exercises it. Every new diagnostic is a disagreement the authored version concealed. +- **Capture (TypeScript worked example — extension#44397, 2026-07-30):** + ```bash + # Arm A — baseline. Expect zero errors. + NODE_OPTIONS='--max-old-space-size=9216' npx tsc -p tsconfig.json --noEmit + # Arm B — probe files that substitute the derived type and call it as the caller does. + mkdir -p app/scripts/derive-probe && cp probe-*.ts app/scripts/derive-probe/ + NODE_OPTIONS='--max-old-space-size=9216' npx tsc -p tsconfig.json --noEmit # diagnostics = the findings + rm -rf app/scripts/derive-probe + ``` + One probe per claim, each naming the authoritative source in a header comment and calling the derived type the way the real call site does. Keep the probes as the artifact — they are the re-runnable falsifier. +- **Why it finds what review and CI miss:** the authored artifact compiles, so CI is green *by construction*. In a partially-migrated repo the asymmetry is structural — with `checkJs` off, a type written for a function whose callers are still `.js` is checked against nothing, and drifts silently forever. Those boundaries are where the lane pays. +- **Traps:** (a) **a substitution can fail for the wrong reason** — a diagnostic on an earlier property short-circuits the one under test, and counting exit codes reads that as confirmation; assert on the *specific* diagnostic, and re-probe with the earlier cause neutralised (`NonNullable<…>`, a targeted assertion) to isolate each claim. Same hazard as B3's "fails on base for the wrong reason." (b) **no authoritative source may exist** — an unshipped package's types, a lib not in tsconfig `lib`, a genuinely new boundary the repo owns. Hand-writing is then *correct*; report it as a cleared falsifier, not a finding. +- **Pairs with:** [lane-assertions.md](lane-assertions.md) for the recipe form; D3 when the substituted artifact is a LavaMoat policy. + --- # E. Production telemetry @@ -224,7 +244,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **G2. Coverage delta** — `yarn test:unit:coverage` → `coverage/unit/` (and `yarn test:unit:webpack:coverage`); `codecov.yml`. Proves the new code is exercised. - **G3. Automated-reviewer output** — independent bot (e.g. cursor[bot]) found nothing blocking. Complements, never replaces, behavior evidence. - **G4. Manual reproduction steps** — human-followable steps that reproduce the fixed behavior; populates the PR template's Manual testing steps. -- **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-<X>=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: in a fork you control, push to a branch literally named **`main`** (or `stable`) — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets configured on that fork (the benchmark jobs need the Infura and test-account secrets; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo. +- **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-<X>=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: push to a branch literally named **`main`** (or `stable`) on `consensys-test/metamask-extension-test-majorlift` — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets on the fork (`INFURA_PROJECT_ID`, `TEST_SRP_*` for benchmarks; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo. Detail: `exogram-daemon/memory/ci-workflow-pr-self-validation-gap.md`. --- @@ -242,6 +262,8 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ | a memory leak fixed / introduced | **C9 retention-path from code** (holder → held → boundary) | C7 heap-over-flow + retainer graph; falsifying lifecycle test | | an error/crash fixed | E1 Sentry rate→0 | B3 test, A1 if visible | | a dep change is safe | D3 LavaMoat + D4 manifest | D1 size; supply-chain-audit's patch/resolutions/ignore lanes | +| a mechanical migration / "rename-only" refactor | **D6 substitution A/B** (authored artifact vs its authoritative source) | B3 if behavior-visible; D1 for accidental output change | +| a hand-written type/schema/policy restates a source | **D6 substitution A/B** | G1 checks (as the *premise*: it compiles, which is why nobody noticed) | | runtime containment / SES / scuttling | **F8 runtime containment** (on the shipped variant) | D3 policy; E1 for `Lockdown failed` events | | persisted-state change | **F1 migration** | F2 vault | | tx/confirmation behavior | F3 simulation | B2 e2e | diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md b/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md index b1a46e1c..6f031e99 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md @@ -2,7 +2,7 @@ How to take run artifacts + complementary evidence and write a clean, idempotent, reviewer-familiar section into the PR body — **matching AEP's own format** so a re-run replaces in place instead of stacking duplicates. -Canonical source for the format: `packages/github/src/pr-body-builder.ts` (`upsertVisualValidationSection`) in the [AEP repo](https://github.com/MetaMask/metamask-autonomous-engineering-platform). Mirror it. +Canonical source for the format: `~/Code/metamask/metamask-autonomous-engineering-platform/packages/github/src/pr-body-builder.ts` (`upsertVisualValidationSection`). Mirror it. > **Publishing is public and outward-facing. Always render the section and get explicit confirmation before writing the PR body. Use `publishEvidence: false` on the run; this manual flow is the only publish path.** @@ -10,29 +10,24 @@ Canonical source for the format: `packages/github/src/pr-body-builder.ts` (`upse Control-plane artifact URLs (`localhost:3000/v1/runs/:id/artifacts/:name`) won't render on GitHub. Re-host each artifact and link the hosted URL. -**Host: an object store or repo whose read access matches your audience.** Configure it once and -reuse it; the examples below assume an S3 bucket exposed through an environment variable: - -```bash -# set these to a bucket you control whose `public/` prefix allows anonymous GetObject -EVIDENCE_BUCKET=<your-bucket> -EVIDENCE_BASE="https://$EVIDENCE_BUCKET.s3.<region>.amazonaws.com" -``` +**Host: the S3 bucket `majorlift-artifacts-share`, prefix `public/`.** ``` -s3://$EVIDENCE_BUCKET/public/metamask/pr-<n>/<run-id>/<artifact-name> -$EVIDENCE_BASE/public/metamask/pr-<n>/<run-id>/<artifact-name> +s3://majorlift-artifacts-share/public/metamask/pr-<n>/<run-id>/<artifact-name> +https://majorlift-artifacts-share.s3.us-west-1.amazonaws.com/public/metamask/pr-<n>/<run-id>/<artifact-name> ``` -Allow anonymous `GetObject` under `public/*` but not bucket listing, so the prefix is not +Anonymous `GetObject` is allowed under `public/*`; bucket listing is not, so the prefix is not browsable — link individual files, and don't promise readers an index. -**Do NOT re-host to a personal repo.** A personal private repo returns 404 for every reader but -its owner, so every raw link to it is dead on arrival. +**Do NOT re-host to `MajorLift/metamask-extension-skills`.** It is a **personal private** repo: +every raw link to it returns 404 for every reader but its owner. That was the previous target +here, and this file simultaneously said links to it were unreachable — guidance that instructed +you to publish dead links. Verified live in a published artifact. -The test is **audience-reachability, not public-vs-private.** An org repo that is private but -readable by colleagues is fine for an internal-audience link. A personal repo is unreachable by -colleagues *and* by the public, so it fails for every audience. +The test is **audience-reachability, not public-vs-private.** A `MetaMask/*` org repo is private +but readable by colleagues, so an internal-audience link to one is fine. A `MajorLift/*` personal +repo is unreachable by colleagues *and* by the public, so it fails for every audience. - Path convention: `pr-<n>/<run-id>/<artifact-name>` keeps runs from colliding. - **Verify unauthenticated before shipping**: `curl -s -o /dev/null -w "%{http_code}"` on each @@ -40,8 +35,8 @@ colleagues *and* by the public, so it fails for every audience. ```bash RUN_ID=<id>; PR=<n>; CP=localhost:3000 -BUCKET="$EVIDENCE_BUCKET" -BASE="$EVIDENCE_BASE" +BUCKET=majorlift-artifacts-share +BASE="https://$BUCKET.s3.us-west-1.amazonaws.com" for name in <artifactName1> <artifactName2>; do curl -fsS "$CP/v1/runs/$RUN_ID/artifacts/$name" -o "/tmp/$name" key="public/metamask/pr-$PR/$RUN_ID/$name" @@ -109,9 +104,9 @@ Screenshots block (injected into `### After`, or appended under `### Screenshots <!-- AEP_SCREENSHOTS_START --> <details open><summary><artifact-name></summary> -<img alt="<artifact-name>" src="<hosted artifact URL>" width="420" /> +<img alt="<artifact-name>" src="<raw.githubusercontent URL>" width="420" /> -[Open full-size image](<hosted artifact URL>) +[Open full-size image](<raw URL>) </details> <!-- AEP_SCREENSHOTS_END --> @@ -121,15 +116,15 @@ Screenshots block (injected into `### After`, or appended under `### Screenshots ## Step 3 — Choose the surface by ownership, then publish -**Publish surface depends on your relationship to the PR.** Determine it FIRST: +**Publish surface depends on my relationship to the PR** (see exogram +`pr-validate-publish-surface-by-ownership`). Determine it FIRST: ```bash PR=<n>; REPO=MetaMask/metamask-extension -ME=$(gh api user --jq .login) -SURFACE=$(gh pr view "$PR" --repo "$REPO" --json author,commits --jq --arg me "$ME" ' - if .author.login==$me then "body" - elif ([.commits[] | select(.authors[].login==$me) - | select([.authors[].login] | map(select(.!=$me and .!="Copilot" and (test("claude|anthropic")|not))) | length == 0)] | length) > 0 +SURFACE=$(gh pr view "$PR" --repo "$REPO" --json author,commits --jq ' + if .author.login=="MajorLift" then "body" + elif ([.commits[] | select(.authors[].login=="MajorLift") + | select([.authors[].login] | map(select(.!="MajorLift" and .!="Copilot" and (test("claude|anthropic")|not))) | length == 0)] | length) > 0 then "comment" else "skip" end') ``` @@ -236,6 +231,7 @@ The common loop — a run refutes a claim, the author pushes a fix, `/pr-validat - New head → **new hosted artifact directory keyed to the fix commit** (`pr-<n>/fix-<sha>/`), commit-pinned raw URLs; never overwrite a prior run's published files. - Residuals the fix intentionally leaves get their own row/section — don't round a fixed-with-residual claim up to fully proven. +Source of truth: `exogram-core/memory/pr-validate-revalidation-delta-reports.md`. ## Lead with a lane-status ledger (no silent absence) @@ -270,7 +266,34 @@ Per-lane rendering: Multi-claim PRs get one sub-block per claim under the status section, each with its own ✅/❌/⚠️ verdict — mirror the Claim Cards. Keep the visual block (markers + `### After` injection) for the image lanes; render the rest as text beneath it. -**Per-scenario presentation (2026-07-21):** the same applies one level down — when the evidence spans multiple test scenarios (flag-on vs flag-off, control vs treatment in an A/B falsifier, numbered manual-testing steps), give each scenario its **own sub-section**: a heading naming the scenario in observation terms, one line on what it tests plus its verdict, and that scenario's artifacts co-located under it. Never bunch all scenarios' artifacts into one large evidence dump — the reviewer verifies "under condition X, artifact shows Y" one condition at a time, and a merged block destroys that mapping even when every artifact is real. For long artifact sets use a `<details>` block *per scenario*, not a merge. +### One comment per evidence *kind*, not one comment per PR (2026-07-30) + +Sub-blocks are for several claims **of the same kind**. When a PR draws two different +kinds — say an executed Validation Run *and* a read-level capability triage — they get +**separate comments**, each with its own header, its own marker pair, and its own format. + +| | Validation Run | LavaMoat policy diligence | +|---|---|---| +| header | `## 🧪 Validation Run` | `## 🔒 LavaMoat Grants — <pkg> <old> → <new>` | +| markers | `VALIDATION_RUN_*` | `LAVAMOAT_DILIGENCE_*` | +| opens on | `**Verdict:** ✅/⚠️/❌` | the finding; **no verdict at all** | +| body | lane ledger, artifacts per lane | deny candidates, enumeration folded | +| audience | whoever owns the PR's claim | whoever owns the policy | + +Merging them forces one frame onto both. A read-level triage has no run to verdict, so it +would land as `⚠️ inconclusive` on a header promising a run; and a `⏳ not-captured` lane +needs a tracker it does not have. The marker pairs also collide — a re-run replacing the +`VALIDATION_RUN` region would silently eat the diligence output sharing it. + +**So: choose the format from the evidence kind, not from this document's default.** The +canonical `## 🧪 Validation Run` header applies when a run produced artifacts. An engine +skill that defines its own output contract (`lavamoat-policy-diligence`) publishes in that +contract. `hooks/pr-evidence-gate.py` enforces the canonical literal only on bodies that +*claim* validation/evidence framing — a diligence comment that renders no verdict does not +trip it, which is the tell that the two are different artifacts rather than one with a +different skin. + +**Per-scenario presentation (2026-07-21):** the same applies one level down — when the evidence spans multiple test scenarios (flag-on vs flag-off, control vs treatment in an A/B falsifier, numbered manual-testing steps), give each scenario its **own sub-section**: a heading naming the scenario in observation terms, one line on what it tests plus its verdict, and that scenario's artifacts co-located under it. Never bunch all scenarios' artifacts into one large evidence dump — the reviewer verifies "under condition X, artifact shows Y" one condition at a time, and a merged block destroys that mapping even when every artifact is real. For long artifact sets use a `<details>` block *per scenario*, not a merge. (Preference: exogram-core `memory/pr-validate-present-scenarios-separately.md`; instance #44610.) ## Artifact contract (ADR-0058 alignment) @@ -283,7 +306,7 @@ To stay interoperable with the recipe-based verification system (MetaMask/decisi - [ ] Each lane passed the [trustworthiness gate](evidence-trustworthiness.md) (shows the claimed surface, signal > noise, could-have-failed) - [ ] Multi-scenario evidence rendered **per scenario** (own heading + verdict + co-located artifacts), not bunched into one block - [ ] **Automated-process voice, no first person** — published validation output never says "I ran/captured/verified"; attribute to the process ("Automated validation ran…", "the harness captured…") so readers know the evidence is machine-generated, not a manual account under the author's name -- [ ] Every image/GIF re-hosted to your configured evidence host; no localhost/local-path URLs in the body +- [ ] Every image/GIF re-hosted to `majorlift-artifacts-share/public/…`; no localhost/local-path URLs in the body - [ ] **Every published link curl'd unauthenticated and returning 200** — never a personal private repo - [ ] Work cited by **PR link** rather than tracking-ticket id, unless the ticket's own content (an RCA, a spec) is the referent - [ ] Narrative scrubbed of username/paths/internal hosts diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md b/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md index 482f01e5..06478a68 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md +++ b/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md @@ -2,10 +2,6 @@ A green result is not proof. An agent — or an eager run — can produce evidence that *looks* like it validates the claim but doesn't. Before believing or publishing any lane, run it through this gate. It extends the vacuous-pass trap to all lanes; the Claim Card's **Falsifier** is the anchor: trustworthy evidence is evidence that *could* have shown the falsifier and didn't. -> **Which items are mechanically enforced.** Several items below close with an *"Emit-time trigger: `pr-evidence-gate.py` class …"* note. Every such class is implemented in [`hooks/pr-evidence-gate.py`](../hooks/pr-evidence-gate.py), which runs as a `PreToolUse:Bash` hook and blocks the write: `verdict`, `observation`, `deferral`, `ci-restatement`, `inflated-verdict`, `bare-identifier`, `truncated-identifier`, `mutable-ref`, `dump-resolver`, `link-only-exhibit`, `data-only-exhibit`, `step-waiver`. It polices both the `gh pr|issue edit|create|comment` porcelain and `gh api` body writes, since a PATCH to a comment is the same publish with a different spelling. -> -> **What stays procedural.** The hook sees vocabulary, not semantics. It cannot tell whether an embedded screenshot actually shows the resolving UI's chrome (item 17), whether a deferral's stated blocker matches the step's own precondition (item 14), or whether inline data is a faithful quotation rather than a transcription (item 16). Those remain reader-applied. Setup: [evidence-gate-setup](evidence-gate-setup.md). - ## The gate (per lane, before publish) 1. **Non-empty & expected media** — the bundle has artifacts of the expected kind. Zero artifacts = not a pass (the vacuous-pass guard). @@ -13,7 +9,7 @@ A green result is not proof. An agent — or an eager run — can produce eviden 3. **Exercises the changed code** — the test/flow actually hits the diff. For a test: it **fails on `main`** (catalog B3). For a flow: the changed component/route is on the path. A green test that never imports the changed module proves nothing. 4. **Signal exceeds noise — and a null states its power** — a perf delta must be beyond run-to-run variance (paired A/B, multiple iterations); a 3% move on a noisy metric is not evidence. The same bar applies in reverse: when the spread is wider than the effect being looked for, the finding is **"not resolvable at this sample size"**, never "no change" — an underpowered run and a true null print the same word, and reporting the word alone lets the reader infer the stronger claim. State the smallest effect the design could have detected. - **Removing a bias is not establishing validity.** Correcting a flaw you found (discarding a warm-up, alternating the starting arm, pinning CPU governor) removes *that* bias and licenses no more than that. It is not a trust gate, because a trust gate names how the evidence could **still** be vacuous — residual risk, not completed work. List what remains uncontrolled (thermal drift, background load, ordering within a round); an unenumerated confound reads as a nonexistent one. - - **When correcting an overclaim, cut the certainty, not the evidence.** A falsifier that actually fired is the strongest thing on the page — downgrade the conclusion around it, don't delete it with the overclaim. + - **When correcting an overclaim, cut the certainty, not the evidence.** A falsifier that actually caught something is the strongest thing on the page — downgrade the conclusion around it, don't delete it with the overclaim. 5. **Could have failed** — the assertion has a reachable failure mode. Always-true assertions (`expect(true)`, a screenshot with no assertion, a Sentry query with no time bound) can't falsify anything. 6. **Right baseline** — "before" is the actual base ref / prior version / pre-window, not a stale or mismatched comparison. 7. **Artifacts are independent & honestly labeled** — checksum every capture set (`md5 *`). Byte-identical files across supposedly independent runs/cases cannot stand as separate observations: either explain the identity in the artifact bundle (deterministic fixture rendering) with per-run provenance that *does* differ (the harness state dump, timestamps, a manifest), or re-capture at distinct moments. Labels must describe the observation, not the interpretation — a file named for the state it *should* show under the claim (`steady-state`, `no-toast`) misleads when the capture shows the refutation. @@ -25,15 +21,18 @@ A green result is not proof. An agent — or an eager run — can produce eviden 13. **Terminal exhibits are reader-native — a live link or a visual; a dump behind a link is still an opaque reference** — item 12 makes every pointer resolve in one click; this item constrains what it may resolve *to*. A positive verdict's terminal artifact is one of the two media a reviewer natively consumes: a **live link into the resolving system** (Sentry trace/event permalink, absolute-windowed Discover query pre-filtered to the claim) or a **visual capture** (screenshot/recording, annotated or cropped to the discriminating region). Raw files (`.log`/`.json`/`.har`, MB-scale dumps) are **appendix-only** — linked once for auditability, never the exhibit a claim rests on: a link whose target is a raw dump passes item 12 and fails item 9 one click later; the digging moved a hop away, it did not disappear (extension#43931 *second* remediation, 2026-07-21: the `bare-identifier` fix shipped a ✅ row whose sole resolver was a re-hosted ~70KB run log). Two corollaries: (a) **the gate items are conjunctive** — a fix for the newest item must re-pass all prior items; satisfying resolvability with an artifact that fails legibility is the generator's next costume; (b) **ascertain the terminal medium at step zero and pick the capture lane that can produce it** — a local intercept (mockttp envelope forwarder) can never yield a live Sentry permalink, so for Sentry-observable claims it is the supplementary falsifier lane and live ingest (dev build → `SENTRY_DSN_DEV`/test-metamask) is primary, precisely because it terminates in permalinks + screenshots; choosing a lane that cannot produce the terminal medium silently displaces it. Emit-time trigger: `pr-evidence-gate.py` class `dump-resolver`, with the remediated extension#43931 row as the regression case (2026-07-21). 14. **Manual testing steps are the validation contract — steps present ⇒ live evidence definitionally required; an impossibility waiver contradicting an executable step is invalid** — the PR's own **Manual testing steps** are the author's assertion that the claimed behavior *is* live-observable, and how: each numbered step is an executability proof (a step a human reviewer can run, `/pr-validate` can run) and its text is the capture spec. Item 11 derives the lanes from the steps top-down; this item closes the other side of the hatch — a lane derived from a step may not then be *waived by argument*. Emit-time procedure: build the per-step coverage map (step → executed-output artifact); for any step without one, the only admissible state is a **per-step** ⏳ + tracker whose blocker is that step's *own* unmet precondition, checked against the step text. Three waiver-inflation patterns from the producing instance (extension#43228/#42869/#44538, 2026-07-21 — all three shipped articulate impossibility rationales in the same body whose Manual testing steps asserted the opposite): (a) **borrowed impossibility** — the excuse imported from a different mechanism or sibling PR (#43228 waived its live lane citing the async remote-flag read race, which is #44538's mechanism; #43228's overrides are build-time env vars, and its steps 1–4 are directly executable in a dev build); (b) **lane-limitation universalized** — one harness's gap stated as global impossibility (#42869: "the e2e harness emits no error `event` envelope, so … not capturable pre-merge" — the step says *dev build with Sentry enabled, trigger an error*, which ingests error events into test-metamask without the e2e harness); (c) **blocked-scope inflation** — a genuinely blocked precondition of one half of the claim expanded to waive the whole lane (#44538: LaunchDarkly provisioning [#7482](https://github.com/MetaMask/MetaMask-planning/issues/7482) blocks only the *prod-flag* half; the step's own text names the dev-injectable alternative — "or inject it into persisted `RemoteFeatureFlagController` state"). If a step is *truly* non-executable, the waiver is still inadmissible alone: the Manual testing steps are then wrong and are corrected in the same edit — a document may not simultaneously instruct a reviewer to observe X and declare X unobservable. The honest-⏳ lane blessed throughout this gate is for *not yet done*, never for *argued away*: an eloquent impossibility rationale is the cheapest token sequence that satisfies every prior item (no fake capture, no CI link, no bare id, no dump) — the generator's costume for the coverage axis. Emit-time trigger: `pr-evidence-gate.py` class `step-waiver` ("not demonstrable / capturable / observable", "not separately captured", "not attached", "rests entirely/solely on the falsifiers/unit/revert" — unconditional in validation scope; no tracker or artifact excuses it), with the three shipped waiver paragraphs as regression cases and the gate verified-blocking on all three live bodies (2026-07-21). Detection gap: the per-step consistency check (does a deferral's blocker match the step's own precondition?) stays procedural — the gate sees vocabulary, not step semantics. 15. **The exhibit lives in the body — link AND visual; a live link alone is the verification path, not the exhibit** — item 13 blessed the terminal media as a *disjunction* (live link OR visual), and generation took the cheaper disjunct: a Discover permalink is producible from the API token alone, a screenshot needs a browser session — so extension#44540's live-ingestion exhibit shipped as a permalink + prose counts, with nothing in the PR body a reader could look at (2026-07-21: "only sentry link and not screenshot that makes it immediately obvious how evidence validates pr"). A live link defers validation behind **click + auth + query rendering + column interpretation** — the dump-resolver displacement one hop further, with the mountain now behind a login: it fails item 9's ~30-second test at the moment of the click, and for any reader *without* Sentry org access (most PR reviewers) a link-only exhibit degrades to a bare identifier (item 12) behind an auth wall. The repaired rule is a **conjunction**: a positive verdict's headline exhibit is an **embedded visual** — screenshot/recording of the linked resolving view (Discover result rows, trace waterfall), cropped/annotated to the discriminating region, captioned with what it should show — **and** the co-located live permalink (absolute-windowed) as the independent-verification path. Neither substitutes for the other: link-only hides the exhibit; visual-only is independently unverifiable. The 2026-07-16 clause "screenshots ride along when a browser session is available; the API token alone yields links + JSON, which is the automatable minimum" was the self-authored escape hatch of this axis (family: the "as context" carve-out, the honest-⏳ waiver): the *automatable minimum* got promoted to the shipped standard because it was the cheapest compliant artifact. A capture lane that cannot screenshot its resolving view is a lane gap to fix before publish (drive a browser session to the Discover URL), never a licensed downgrade — deliberate exceptions route through the human. Emit-time trigger: `pr-evidence-gate.py` class `link-only-exhibit` (non-negated verdict + `sentry.io` link + no image/recording embed in the unit), with the shipped #44540 paragraph as the regression case and the prior suite's permalink-only ALLOW cases flipped/augmented — third occurrence of "an ALLOW case containing the violating tokens is a specification of the next costume." Detection gaps: verdict co-location is required, so a no-verdict link-only paragraph evades mechanically; the visual-without-link converse stays procedural under item 12. -16. **The audit chain is mechanical — quote, don't transcribe; pin, don't point** — an exhibit's inline data must be a **verbatim, greppable excerpt** of the artifact (full-length identifiers, raw capture lines quoted exactly), and every repo-hosted artifact link must be a **commit-pinned, line-anchored permalink** (`/blob/<sha>/…#Lx-Ly`) to the discriminating lines. Producing real artifacts and then hand-transcribing digests severs the claim→artifact bridge at every link: an ellipsized id (`24b1e2da…`) cannot be grepped against any artifact even when the log is linked in the same block; a reformatted data block cannot be distinguished from confabulation without redoing the dig, so it reads as *claims in the form of data*; a branch-ref `/blob/<branch>/` link is a mutable pointer whose target can be rewritten after review (not tamper-evident); a file-level link without line anchors lands the reader at the top of a 1,000-line dump. The producing instance (extension#43929 validation comment, 2026-07-21) had every number substantiated by four re-hosted run logs — real, included, resolvable — and still drew "still no immediately auditable evidence just claims in the form of data": each prior item individually near-passed while the exhibit↔artifact binding stayed **editorial** (transcription + file link) instead of **mechanical** (quotation + pinned line anchor). Emit-time procedure: for every inline datum, quote the raw capture line it comes from (fenced, verbatim, full ids) and anchor it (`#L<n>`); pin every evidence link to the SHA (press `y` on the GitHub file view). Emit-time trigger: `pr-evidence-gate.py` classes `truncated-identifier` (a co-located resolver does NOT excuse it — the resolver resolves the full id, not the fragment the reader holds; hash-equality prose exempt) and `mutable-ref`, plus the closed **surface hole**: the shipped comment was published via `gh api` PATCH, which the porcelain-only matcher (`gh pr|issue edit|create|comment`) never saw — the gate now scans `gh api` body writes (`-F body=@file`, `-f body=…`, `--input`); fifth occurrence of the converse-of-gate rule, one level down: audit the gate for *spellings of the write it cannot see*, not just tokens it excuses. Detection gaps: the line-anchor half of pinning and the paraphrase-vs-quotation judgment stay procedural — the gate sees ellipses and branch refs, not editorial fidelity. +16. **The audit chain is mechanical — quote, don't transcribe; pin, don't point** — an exhibit's inline data must be a **verbatim, greppable excerpt** of the artifact (full-length identifiers, raw capture lines quoted exactly), and every repo-hosted artifact link must be a **commit-pinned, line-anchored permalink** (`/blob/<sha>/…#Lx-Ly`) to the discriminating lines. Producing real artifacts and then hand-transcribing digests severs the claim→artifact bridge at every link: an ellipsized id (`24b1e2da…`) cannot be grepped against any artifact even when the log is linked in the same block; a reformatted data block cannot be distinguished from confabulation without redoing the dig, so it reads as *claims in the form of data*; a branch-ref `/blob/<branch>/` link is a mutable pointer whose target can be rewritten after review (not tamper-evident); a file-level link without line anchors lands the reader at the top of a 1,000-line dump. The producing instance (extension#43929 validation comment, 2026-07-21) had every number substantiated by four re-hosted run logs — real, included, resolvable — and still drew "still no immediately auditable evidence just claims in the form of data": each prior item individually near-passed while the exhibit↔artifact binding stayed **editorial** (transcription + file link) instead of **mechanical** (quotation + pinned line anchor). Emit-time procedure: for every inline datum, quote the raw capture line it comes from (fenced, verbatim, full ids) and anchor it (`#L<n>`); pin every evidence link to the SHA (press `y` on the GitHub file view). Emit-time trigger: `pr-evidence-gate.py` classes `truncated-identifier` (a co-located resolver does NOT excuse it — the resolver resolves the full id, not the fragment the reader holds; hash-equality prose exempt) and `mutable-ref`, plus the **surface hole** — the shipped comment was published via `gh api` PATCH, which the porcelain-only matcher (`gh pr|issue edit|create|comment`) never saw; fifth occurrence of the converse-of-gate rule, one level down: audit the gate for *spellings of the write it cannot see*, not just tokens it excuses. **CORRECTION 2026-07-30 — this hole was recorded as closed and is not.** Verified against the deployed `hooks/pr-evidence-gate.py` (259 lines): line 47 is the only command matcher, `\bgh\s+(?:pr|issue)\s+(?:edit|create|comment)\b`, so `gh api` body writes are still invisible; and the file implements essentially one check (verdict-needs-artifact), **not** the ~9 classes named across items 11–18 (`ci-restatement`, `bare-identifier`, `dump-resolver`, `link-only-exhibit`, `data-only-exhibit`, `step-waiver`, `truncated-identifier`, `mutable-ref`, `inflated-verdict`). Treat every "Emit-time trigger: `pr-evidence-gate.py` class …" line in this document as **specified, not implemented**, until re-verified in the code — a doc asserting a class the code lacks retires the vigilance it claims to replace, which is the failure this very item warns about. Consequence observed the same day: 14 unlinked `path:line` references shipped across 12 review comments via `gh api`, with the gate both classless for that shape and unwired in `settings.json`. Detection gaps: the line-anchor half of pinning and the paraphrase-vs-quotation judgment stay procedural — the gate sees ellipses and branch refs, not editorial fidelity. 17. **Evidence is captured in its environment — data alone is insufficient even when correct** — item 16 makes the data trustworthy as *transcription* (verbatim, greppable, pinned); this item polices what transcription can never carry: **liveness provenance**. A quoted `EVIDENCE trace_id=…` line, a re-hosted gist, a hand-assembled id table can all be correct and still show nothing about *where they came from* — extracted data is indistinguishable from data typed by hand, so it cannot make it immediately apparent that the evidence was captured **live** from a **functioning** system. The exhibit for a system-of-record-observable claim therefore includes an **in-environment capture**: a screenshot/recording of the resolving system's own UI (the Sentry Discover/trace view with the query, project/environment selectors, absolute time window, and result rows all in-frame) — the environmental chrome is not decoration, it *is* the provenance: it shows the query really ran, in the real dashboard, over the real window, and returned these rows. Correctness was never the failing dimension (2026-07-21: "just the data is insufficient even if correct — it needs to be immediately apparent that evidence was captured live and is functional"). Relation to prior items: item 15's link+visual conjunction fired only when a `sentry.io` link was present, and item 13's `NATIVE_MEDIUM` blessed an inline fenced excerpt as a terminal medium — so a no-link, quoted-data exhibit (the fidelity-remediated shape: full ids, verbatim excerpts, pinned line anchors, zero environment captures) passed the whole regime while carrying zero liveness provenance. The joint rule after this item: a telemetry-observable positive verdict always carries the in-environment visual (plus the live permalink per item 15); quoted excerpts, gists, and data files are appendix beside it, never the exhibit. Emit-time trigger: `pr-evidence-gate.py` class `data-only-exhibit` (non-negated verdict + telemetry-observation vocabulary + no image/recording embed + no sentry link — with a sentry link, `link-only-exhibit` already fires), with the re-hosted-gist ALLOW case flipped (fifth occurrence of "the ALLOW case was the next costume's spec") and the #43929 quoted-excerpt shape as a regression case. Detection gaps: vocabulary-scoped (telemetry-observation terms, not bare code tokens like `trace.test.ts`), so a claim phrased entirely without them evades mechanically; and the gate cannot see whether an embedded image actually shows the environment's chrome — screenshot content stays procedural (item 2's "eyeball it" applies: the capture must show the *resolving UI*, not a cropped data region indistinguishable from a spreadsheet). 18. **"Successful" is an evidence predicate, not a run status — and the default Sentry exhibit is fixed in advance** — a validation run may be scored/reported "successful"/"validated" only when its published surface already carries, for every Sentry-observable lane, the default exhibit pair: an **in-environment Sentry-UI screenshot** (item 17) **plus the co-located live permalink** (item 15). Completed runs, green falsifiers, staged drafts, and honest ⏳ lanes do not confer success — a run without the pair is at most "run-complete, evidence-owed." The default recipe needs no per-PR ascertainment: for Sentry, **generally capture actual screenshots from the Sentry UI and attach the link** — that pair is step zero's pre-computed answer for any Sentry-observable claim, never the terminus of axis-by-axis escalation. Capture-first ordering: the capture executes before any rule/gate/postmortem authoring may close a validation session — writing a new rule or gate class discharges nothing (2026-07-22: ten postmortems and 17 gate items shipped while zero Sentry-UI screenshots did; every "successful" run was claims-only, because success was assigned by run-completion and meta-work substituted for capture work). Emit-time trigger: `pr-evidence-gate.py` `VERDICT` vocabulary now includes the status spellings `successful`/`validated`/`live-proven`, so a claims-only unit scoring itself successful blocks like any bare "confirmed." Detection gap: the gate fires only on re-emit — already-shipped "successful" surfaces are audited by backward re-score, enumerated from live state, never from the ledger (the discharge-granularity rule applies to success statuses verbatim). +19. **A substitution A/B is readable only if the unmodified arm is silent — and only if each diagnostic fires for the reason claimed** — the substitution lane (catalog D6) derives its finding from the *delta* in a checker's output between the PR as written (Arm A) and the PR with one authored artifact replaced by its authoritative equivalent (Arm B). Two ways that delta lies, both of which look like a confirmed finding. (a) **A noisy Arm A destroys attribution.** If the unmodified tree already emits diagnostics, nothing in Arm B is attributable to the substitution — the reader cannot tell a concealed disagreement from ambient breakage, and "N errors in Arm B" is then a count, not a finding. Publish Arm A's result explicitly (`0 errors`, verbatim) as the delivery check; if it is non-empty, the instrument is broken and the lane is **inconclusive**, not a pass — fix the baseline (pin the toolchain, raise the heap, exclude the unrelated project) or drop the lane. This is the substitution analogue of item 1's vacuous-pass guard: item 1 asks whether the artifact exists, this asks whether the *comparison* means anything. (b) **A diagnostic can fire for the wrong reason.** A checker reports the first failure it reaches, so an earlier cause short-circuits the claim under test and an exit-code read scores it as confirmation — the same hazard item 3 polices for tests that fail on base for an import error rather than the bug. Producing instance (extension#44397, 2026-07-30): a probe asserting a hand-written provider return type was unsound errored on *nullability* one property earlier, and the return-type claim — re-probed with the nullability neutralised via `NonNullable<…>` — turned out to be **sound**, i.e. a finding that would have shipped as real. Emit-time procedure: for every substitution claim, assert on the *specific* expected diagnostic (code + message + line), not on non-zero exit; where an earlier cause intervenes, isolate it and re-run; and report the claims the re-probe **cleared** alongside the ones it confirmed — a substitution sweep that only ever confirms is indistinguishable from one that never isolated anything. Corollary on the negative case: when no authoritative source exists (a package that ships no types, a lib absent from tsconfig `lib`, a boundary the repo genuinely owns), hand-writing is *correct* — record it as a cleared falsifier with the reason, never as an unreported non-finding. Detection gap: procedural — a gate can see whether Arm A's result is published, not whether the diagnostic it cites is the one the claim needs. + ## Lane-specific traps - **Visual:** spinner/skeleton mistaken for the loaded state; the toggle (privacy/redaction) not actually flipped; a cached screenshot from a prior run; the fallback surface shown without saying so. - **Perf / benchmark:** stale frozen baseline (catalog C5 caveat); single sample; warm-vs-cold mismatch; measuring a different interaction than the claim. - **Test:** snapshot regenerated to match the bug (`--updateSnapshot` masking a regression); the test mocks out the changed path; it passes on `main` too (so it's not a regression test). +- **Substitution A/B (D6):** a non-silent Arm A (attribution destroyed); a diagnostic that fires one property earlier than the claim (isolate and re-probe); a substitution the checker never reaches because the caller is untyped JS or `any`; treating "no authoritative source exists" as a null result rather than a cleared falsifier. - **Telemetry:** query window excludes the release; the error regrouped under a different fingerprint; sample-rate makes "0 events" meaningless. - **Migration:** only the happy path asserted; `changedKeys` not checked against actual mutations; no real prior-version fixture. - **Coverage:** a line covered ≠ a behavior asserted (executed but never checked). diff --git a/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md b/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md index 31316c66..89622327 100644 --- a/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md +++ b/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md @@ -23,4 +23,4 @@ Maps each evidence-catalog lane to a declarative assertion form, so a Claim Card | F7 i18n | static: `verify-locales` exit 0 | out-of-band | | F8 runtime containment | `Object.isFrozen(Object.prototype)`; scuttled global throws + exception resolves; `typeof SNOW` | **yes** — `Runtime.evaluate`, but only against the SHIPPED build variant (dev is unscuttled, test's exception list is wider) | -**Takeaway.** UI-state and runtime-metric lanes (A/B-visual, C1–C3/C6, F3/F5) map cleanly to CDP recipe assertions — ADR-0058's sweet spot. Static (D, F7), test-runner (B3, C5, F1), and dashboard (E) lanes are **out-of-band**: the recipe should *reference* them as proof targets without executing them. That out-of-band reference is precisely the **non-UI scaling gap** raised in review on decisions#173 — a recipe schema that admits out-of-band assertion references (not only CDP actions) closes it. This table is the proposed taxonomy to contribute back (ITERATION item 10). +**Takeaway.** UI-state and runtime-metric lanes (A/B-visual, C1–C3/C6, F3/F5) map cleanly to CDP recipe assertions — ADR-0058's sweet spot. Static (D, F7), test-runner (B3, C5, F1), and dashboard (E) lanes are **out-of-band**: the recipe should *reference* them as proof targets without executing them. That out-of-band reference is precisely the **non-UI scaling gap** MajorLift's review of #173 flagged — a recipe schema that admits out-of-band assertion references (not only CDP actions) closes it. This table is the proposed taxonomy to contribute back (ITERATION item 10). diff --git a/domains/pr-workflow/skills/pr-validate/skill.md b/domains/pr-workflow/skills/pr-validate/skill.md index bb8e20f9..660ef602 100644 --- a/domains/pr-workflow/skills/pr-validate/skill.md +++ b/domains/pr-workflow/skills/pr-validate/skill.md @@ -1,7 +1,6 @@ --- name: pr-validate description: Validate a MetaMask PR with objective evidence — primarily the Autonomous Engineering Platform (AEP) harness (visual_validation for visible UI behavior, perf_validation for non-visible perf behavior), backed by complementary evidence (Sentry query links, screenshots, screen recordings→GIF, DevTools/CDP output, bundle-size, web-vitals, test/CI results). Drives the AEP local stack end to end: preflight (postgres + temporal + worker + control-plane) → submit POST /v1/tasks → poll GET /v1/runs/:id → fetch artifacts → assemble an evidence bundle → publish to the PR body (re-hosting images to a public repo, scrubbing local paths). Match evidence to the PR's specific falsifiable claim, not a fixed checklist. Triggers on /pr-validate, /pr-validate visual, /pr-validate perf, /pr-validate preflight, /pr-validate status, /pr-validate evidence, /pr-validate plan, or when the user mentions validating/proving a PR, AEP / visual validation / perf validation, capturing evidence for a PR, before/after screenshots, a screen recording or GIF for a PR, attaching Sentry links or DevTools output as proof, or publishing an evidence bundle to a PR body. -maturity: experimental --- # /pr-validate @@ -18,7 +17,7 @@ Twenty rules the rest of this skill implements. When a situation isn't covered b **What you may claim** - **Falsifiability** — name the observation that would disprove the claim, then go looking for it. A review that cannot fail is not a review. -- **Falsifier coverage is not exhaustive — human intervention point.** There is no fixed checklist, so there is no completeness guarantee: the falsifiers found are bounded by claim-extraction quality and by what the reviewer thought to test. "No falsifier fired" is not "no falsifier exists." A human judges whether the falsifier chosen matches the claim's actual risk, and whether a mixed or high-stakes claim needed more than one — this skill closes the falsifiers it finds, it does not attest that it found all of them. +- **Falsifier coverage is not exhaustive — human intervention point.** There is no fixed checklist, so there is no completeness guarantee: the falsifiers found are bounded by claim-extraction quality and by what the reviewer thought to test. "Nothing turned up" is not "there is nothing to find." A human judges whether the falsifier chosen matches the claim's actual risk, and whether a mixed or high-stakes claim needed more than one — this skill closes the falsifiers it finds, it does not attest that it found all of them. - **Diff-anchored** — the claim is what the code *can* do, not what the PR body promises. Drift between them is a finding, not a claim. - **Surface-specific, and a surface need not be a screen** — a job graph, a build artifact, a policy file, or a telemetry shape are all legitimate surfaces with their own falsifiers. @@ -74,10 +73,10 @@ A claim must be falsifiable, surface-specific, **anchored to the diff** (if the | Telemetry / error-rate / latency in prod | **E1 Sentry links** (before/after) | E2 Tempo; span-volume → `/sentry-quota` | | Bundle / build output | **D1 size · D2 chunk membership** | — | | A dependency change is safe | **D3 LavaMoat policy + D4 manifest diff** | D1 size | -| Runtime containment still holds | **F8 SES lockdown / scuttling, on the shipped variant** | D3 policy | | Persisted-state change | **⭐ F1 migration** (`changedKeys`, old→new state) | F2 vault round-trip | | Tx / dapp / flag / snap / i18n behavior | **F3 sim · F4 provider · F5 flag matrix · F6 snaps · F7 i18n** | B2 e2e trace | | Behavior with no UI | **B3 test + G4 repro** | G1 CI checks | +| Mechanical migration / "rename-only" refactor; a hand-written type/schema/constant/policy that restates a source | **⭐ D6 substitution A/B** at a fixed head (`compare` arm kind `substitution`) | B3 if behavior-visible; D1 for accidental output change | Lane IDs (A1, B3, …) index [references/evidence-catalog.md](references/evidence-catalog.md) — the full menu with verified capture commands and the complete matching guide. When a PR mixes claims (a UI fix that also shifts a metric), run more than one lane and assemble them into one bundle. @@ -97,35 +96,43 @@ Not for code-correctness review (use `/review`, `/code-review`) or span-quota re | `/pr-validate <pr>` | **Flagship.** Read the PR → state the claim → pick lanes → [preflight](#preflight) → run AEP lane(s) + gather complementary evidence → assemble bundle → **propose** the PR-body section and confirm before publishing. | | `/pr-validate plan <pr>` | Dry run: read the PR, state the claim, recommend lanes + targeting hints. No stack, no run. Cheap first step when unsure. | | `/pr-validate visual <pr>` | AEP `visual_validation` only. | -| `/pr-validate perf <pr>` | AEP `perf_validation` only — check the graph is present first, see [caveat](#perf_validation-caveat). | +| `/pr-validate perf <pr>` | AEP `perf_validation` only (local/uncommitted graph — see [caveat](#perf_validation-caveat)). | | `/pr-validate preflight` | Health-check the local stack; bring up what's down. No run. | | `/pr-validate status <run-id>` | Poll `GET /v1/runs/:id`; print stage timeline + `evidenceBundle.artifactRefs`. | | `/pr-validate evidence <pr> [--run <id>]` | Assemble + publish a bundle from an existing run and/or complementary sources (Sentry/screens/devtools). No new AEP run. | | `/pr-validate lane <id> <pr>` | Run a single [catalog](references/evidence-catalog.md) lane by id (e.g. `lane F1`, `lane C3`, `lane D3`) — for the non-AEP lanes where you know the claim type. | -| `/pr-validate compare <pr>` | Paired A/B for a perf or refactor claim: build base + head, capture the lane on both, diff. Avoids the stale-baseline trap (catalog C5). **Per-arm treatment check first:** verify the mechanism under test is actually active in each arm (chunk split present, span emitted, flag evaluated) before interpreting deltas — a null arm without delivered treatment is a no-op, not a control (2026-07-22, #42795 bisect lesson). | +| `/pr-validate compare <pr>` | Paired A/B for a perf or refactor claim. Two arm kinds — pick by what the claim varies: **`ref`** (default) builds base + head, captures the lane on both, diffs; avoids the stale-baseline trap (catalog C5). **`substitution`** holds a **fixed head** and varies one *artifact* instead of the ref — replace the PR's hand-written type/schema/constant/policy with the authoritative equivalent and diff a checker's output (catalog D6); no build, no rebase, no merge boundary. **Per-arm checks first, one per kind:** for `ref`, verify the mechanism under test is actually active in each arm (chunk split present, span emitted, flag evaluated) — a null arm without delivered treatment is a no-op, not a control (2026-07-22, #42795 bisect lesson). For `substitution`, verify the unmodified arm is **silent** and that each diagnostic fires for the reason claimed — a noisy Arm A destroys attribution, and a diagnostic tripping one property early scores as a confirmation it isn't (trustworthiness gate item 19; 2026-07-30, #44397). | `<pr>` is a number or URL on `MetaMask/metamask-extension` unless another repo is given. Every variant runs Step 1 (extract the Claim Card) first — the claim decides the lane, even when you named one. ## Preflight -The AEP harness runs as a local stack: postgres, a temporal server, a worker, and a control plane. Bring-up steps, required Node version, registry auth, and environment are documented in the [AEP repository](https://github.com/MetaMask/metamask-autonomous-engineering-platform) itself — follow its README rather than a copy here, which drifts. Health-check first and bring up only what is down. +The hosted AEP doesn't resolve (`aep.dev.web3factory.consensys.net` is dead as of 2026-06). Everything runs locally. Health-check, then bring up only what's down. **Full procedure + every gotcha: [references/aep-local-run.md](references/aep-local-run.md).** Skim it before a first run in a session — each bullet there cost a failed run. Fast checks: ```bash +AEP=~/Code/metamask/metamask-autonomous-engineering-platform curl -fsS localhost:3000/health >/dev/null && echo "control-plane up" || echo "control-plane DOWN" curl -fsS localhost:8233 >/dev/null && echo "temporal UI up" || echo "temporal DOWN" -docker ps --format '{{.Names}}' | grep -E 'aep-postgres|aep-temporal' +docker ps --format '{{.Names}}' | grep -E 'mm-aep-postgres-dev|mm-aep-temporal-dev' ``` -If the control plane answers on `localhost:3000/health`, the stack is ready and you can skip to *Run mechanics*. +Bring-up order (each in its own shell; details + env in the reference): +1. `yarn dev:postgres` (docker `postgres:16-alpine`, `mm-aep-postgres-dev`, port 5432) +2. `yarn dev:temporal` (temporal dev server; UI on 8233) +3. `yarn db:migrate` +4. **worker** — `yarn dev:worker` on **Node ≥ 24.13**, env `ANTHROPIC_API_KEY=host-subscription`, `CLAUDE_CODE_EXECUTABLE=~/.local/bin/claude`, `GITHUB_TOKEN="$(gh auth token)"`, `SANDBOX_PROVIDER=local` (needs JFrog `npm login` first; relies on uncommitted local patches) +5. `yarn dev:control-plane` (`localhost:3000`) + +If any of the local patches (`local-sandbox-adapter.ts` timeout, `claude-agent-runner.ts` auth, the `perf-validation/` graph) are missing from the working tree, the reference says how to restore them — `git status` in the AEP repo should show them modified/untracked. ## Teardown The stack is the heaviest thing this skill starts — postgres + temporal + a Node worker + control-plane — and the worker holds a live Claude session while the autonomous run itself spends tokens. It is **on-demand, not resident**: bring it up for the validation window, **tear it down when the run(s) finish**. Left up, it's the single largest reclaimable footprint on a shared host and quietly keeps a Claude seat warm. -- **If your host wraps the stack in a service manager**, use its own down command — it stops the services and removes the `--rm` postgres/temporal containers, so state resets on the next bring-up (fine, each run is fresh anyway). -- **Otherwise:** stop the `yarn dev:*` processes and remove the postgres/temporal containers. +- **On a host managed by `aep-stack` (systemd):** `aep-stack up` to preflight, **`aep-stack down` when done** — stops the services; the `--rm` postgres/temporal containers are removed, so state resets on the next `up` (fine — each run is fresh anyway). +- **Otherwise:** stop the `yarn dev:*` processes and `docker rm -f mm-aep-postgres-dev mm-aep-temporal-dev`. - **Tear down on every exit path** — pass, refutation, *or* abort. A failed or abandoned run leaves the stack up exactly as much as a passing one; the usual leak is walking away after a refutation without stopping it. ## Run mechanics (submit → poll → fetch) @@ -159,7 +166,7 @@ curl -fsS "$CP/v1/runs/$RUN_ID/artifacts/<artifactName>" -o /tmp/<artifactName> ### Concurrent runs (multiple agents / parallel lanes) -Five shared resources need per-run isolation on one machine — collisions cross-contaminate evidence *silently* (wrong session's logs attributed to a run), which is an integrity failure, not flakiness: **(1)** CDP debug ports — derive per run, never hardcode; **(2)** e2e harness service ports (anvil/proxy/fixture/mocha) — one e2e run at a time per worktree, one worktree per agent, and never rebuild `dist/` in a worktree with an active run; **(3)** artifact dirs — per-run namespaces; `test-artifacts/` is per-worktree shared state, harvest failure artifacts before the next run overwrites the same test-title dir; **(4)** evidence-repo uploads — run-scoped paths (`pr-<n>/<run-id>/`), retry-with-fresh-sha on 409, never overwrite another run's published files; **(5)** commit-pinning — pin only after your own final upload lands, verifying your files exist at that sha. Safe to share: registry auth, a read-only `dist/`, the AEP stack itself. +Five shared resources need per-run isolation on one machine — collisions cross-contaminate evidence *silently* (wrong session's logs attributed to a run), which is an integrity failure, not flakiness: **(1)** CDP debug ports — derive per run, never hardcode; **(2)** e2e harness service ports (anvil/proxy/fixture/mocha) — one e2e run at a time per worktree, one worktree per agent (`wt new`), and never rebuild `dist/` in a worktree with an active run; **(3)** artifact dirs — per-run namespaces; `test-artifacts/` is per-worktree shared state, harvest failure artifacts before the next run overwrites the same test-title dir; **(4)** evidence-repo uploads — run-scoped paths (`pr-<n>/<run-id>/`), retry-with-fresh-sha on 409, never overwrite another run's published files; **(5)** commit-pinning — pin only after your own final upload lands, verifying your files exist at that sha. Safe to share: JFrog login, a read-only `dist/`, the AEP stack itself. ### Trust the evidence (anti-reward-hacking) @@ -167,12 +174,7 @@ A green result is not proof. The vacuous-pass trap is the floor: if `promptCraft ### perf_validation caveat -The `perf-validation/` graph writes falsifiable network/static/smoke assertions and gives the tester deterministic `.aep/` helpers (CDP netlog, phase segmentation, source-map chunk membership). Two constraints worth knowing before a perf run: - -- It requires a `yarn webpack --test` build first — the browserify `build:test` has no code splitting, so `import()` never hits the network there. -- Temporal caps activity results at ~2MB, so artifact refs must be content-free; only `evidenceBundle` carries base64. - -**Check the graph is present in your AEP checkout before relying on it.** It is newer than the visual-validation graph and may not be in every version — if it isn't registered, perf runs silently won't dispatch, and the fallback is manual DevTools/CDP capture (see the catalog). +The `perf-validation/` graph is **uncommitted local AEP work** (added 2026-06-11). It writes falsifiable network/static/smoke assertions and gives the tester deterministic `.aep/` helpers (CDP netlog, phase segmentation, source-map chunk membership). It requires a `yarn webpack --test` build first (the browserify `build:test` has no code splitting, so `import()` never hits the network there). Temporal caps activity results at ~2MB — artifact refs must be content-free; only `evidenceBundle` carries base64. If the graph isn't in the working tree, perf runs won't register — fall back to manual DevTools/CDP capture (catalog). ## Complementary evidence @@ -182,7 +184,6 @@ AEP is primary but rarely sufficient alone. Pull whatever the claim needs — ** - **B. Behavior & flow** — mm-CLI visual, E2E trace+video, **⭐ falsifying regression test** (fails on main, passes on branch — the strongest bug proof), Storybook/component, a11y, flaky-stability rerun. - **C. Performance & render** — startup/custom traces, web-vitals (**INP/FCP/LCP/CLS** via `stateHooks`), long-task **TBT** (separate observer), React render/selector (WDYR), benchmark A/B (paired), DevTools/CDP profiling, memory-over-flow, **same-window app+DevTools capture** (C8 — UI + console evidence in one frame, OS-level region recording). - **D. Build output** — bundle-size, chunk membership, **LavaMoat policy diff**, manifest permissions diff, build-variant matrix. - (Runtime containment — SES lockdown, scuttling, Snow — is **F8**, not D: D is what the build *permits*, F8 is what the running artifact *enforces*.) - **E. Production telemetry** — Sentry links (span-volume → `/sentry-quota`), Tempo traces, error-event shape. - **F. Extension integrity** — **⭐ state migration**, vault/keyring, tx simulation, provider/dapp, feature-flag matrix, snaps, i18n. - **G. CI/review/process** — check links, coverage delta, reviewer bot, manual repro. @@ -196,7 +197,7 @@ Match the bar to the claim; stop when the claim's falsifier is closed. Don't ove - **One lead lane that closes the falsifier** is enough for low-risk, single-claim PRs (a copy fix → one screenshot; a bug fix → the falsifying test). - **Weigh AEP's cost before reaching for it.** A `visual_validation`/`perf_validation` run spins the full stack *and* burns autonomous-agent tokens — by far the most expensive lane. Use it when the claim genuinely needs autonomous capture of a reachable surface; when a lighter lane closes the same falsifier (a single `mm` screenshot, a falsifying test, a CDP capture, an artifact CI already produced), prefer it and skip the stack. Whenever you do start it, tear it down after (see [Teardown](#teardown)). - **Lead + one corroborator** for perf/telemetry (a number *and* its source) and for anything user-facing that also moves a metric. **For a perf-targeting PR the lead lane is the measured impact itself** — a paired A/B benchmark at the current head (C5) or equivalent — never mechanism evidence alone (chunk membership, netlog exclusion prove the improvement is *possible*, not that it *happened*). A perf PR also always carries correctness + non-regression lanes: changed-surface tests green at head, affected flows exercised, neutral profile within noise. (2026-07-22, #42795 lesson.) -- **Lead + integrity lane** for high-stakes surfaces regardless of size: persisted-state (migration + vault), money (tx simulation), permissions (LavaMoat + manifest), runtime containment (SES lockdown / scuttling), security/keyring. Size-S doesn't lower the bar here. +- **Lead + integrity lane** for high-stakes surfaces regardless of size: persisted-state (migration + vault), money (tx simulation), permissions (LavaMoat + manifest), security/keyring. Size-S doesn't lower the bar here. - **Per-claim** for mixed PRs — each Claim Card needs its own closed falsifier; a strong UI proof doesn't cover the metric it also shifts. - **Rely on CI for routine coverage — don't re-collect what CI already establishes.** Lint, build, typecheck, the full test suite, changelog validation: CI is the authoritative source; **cite the check result** (e.g. "423 pass / 0 fail at head") instead of re-running it locally. Spend independent evidence only on (a) the claim's load-bearing falsifier, (b) specifically important/noteworthy areas (security, money, permissions, the exact changed surface), or (c) where the trust-gate warns a green result could be vacuous/misattributed. This is the economy counterpart to *"don't trust green blindly"*: that gate polices the **claim-critical** lane; this rule spares the **routine** coverage — re-collecting what CI covers is bundle noise. (#9628: cited CI's pass matrix for build/test, ran independent evidence only for the load-bearing homogeneity + resolution lanes.) @@ -207,10 +208,10 @@ Stop when each claim has one trustworthy artifact that would have shown its fals **Public, outward-facing action — always confirm the rendered section with the user before writing the PR body.** Match AEP's own format so the section is idempotent and reviewer-familiar. Full recipe (markers, image re-hosting, recordings, the `### After` injection, privacy scrub): **[references/evidence-publishing.md](references/evidence-publishing.md).** Essentials: - **Canonical header — every validation output leads with the exact literal `## 🧪 Validation Run`.** Same string in a PR comment and in the PR-body section, never reworded or demoted — the constancy is what makes it scannable/Ctrl-F-able, like Copilot's fixed `## Pull request overview`. Line 2 is the meta line: `**Verdict:** ✅ proven — **Claim:** <one-liner>` then `head \`<sha>\` · <date> · lanes: <list>`. Enforced mechanically by `hooks/pr-evidence-gate.py` (a validation/verification/evidence heading or AEP marker without the literal blocks the `gh` write). -- **Post complete, once — and know which regime the surface is in.** Comments are **push** (audience notified once at post time; edits are silent): hold until every planned lane is present or consciously dropped, and put substantive additions or changed verdicts in a **new comment referencing the original**, never a silent edit. The PR **body** is **pull** (consulted at review time): the idempotent marker upsert on re-validation at a new head is correct there. Typo-level comment edits are fine. +- **Post complete, once — and know which regime the surface is in.** Comments are **push** (audience notified once at post time; edits are silent): hold until every planned lane is present or consciously dropped, and put substantive additions or changed verdicts in a **new comment referencing the original**, never a silent edit. The PR **body** is **pull** (consulted at review time): the idempotent marker upsert on re-validation at a new head is correct there. Typo-level comment edits are fine. (Decision: `exogram-core/decisions/2026-07-23-publish-complete-bundles.md`; framework: Reprise `push-pull-artifact-edit-regimes`.) - **Falsifier-forward.** After the meta line, foreground **what would have falsified the claim and how each falsifier is closed** — the falsifier is the load-bearing content, not a footnote. Structure the body as "what would make this false → the evidence that rules it out," not a lane inventory with a `falsifiers closed` line buried at the bottom. The reviewer should see the disproof attempt first. - **Don't restate CI results.** Lint/build/typecheck/test/changelog outcomes are already on the PR's Checks tab — the reviewer sees them. Cite a CI result in the comment only to **highlight something specific** they'd otherwise miss; otherwise reference "green in Checks" or omit it. Restating "423 pass / 0 fail" is bundle noise (the display-side counterpart to the catalog's *rely on CI* collection rule). -- **Re-host images first.** Control-plane artifact URLs are `localhost` and won't render on GitHub. Re-host each artifact somewhere **your readers can reach unauthenticated**, then link the hosted URL — see [evidence-publishing.md](references/evidence-publishing.md) for the host choice and the mandatory unauthenticated `curl` check. A personal repo or a private bucket fails this for every reader but you. +- **Re-host images first.** Control-plane artifact URLs are `localhost` and won't render on GitHub. Push to the public `MajorLift/metamask-extension-skills` repo, branch `aep-evidence`, via the contents API; link the `raw.githubusercontent.com` URLs. - **Use idempotency markers** so a re-run replaces in place: wrap the whole section in `<!-- VALIDATION_RUN_START -->` … `<!-- VALIDATION_RUN_END -->`; inside it, AEP's own `<!-- AEP_VISUAL_VALIDATION_START/END -->` for the status block and `<!-- AEP_SCREENSHOTS_START/END -->` for images, injected into the PR template's `### **After**` section (replacing the `<!-- [screenshots/recordings] -->` placeholder) when present. - **Verdict-first, lanes nested:** under the canonical header, hand-assembled AEP blocks demote to `### AEP Visual Validation` (leave AEP's own service-published `##` blocks untouched) with `**✅ Passed**` / `**❌ Failed**` / `ℹ️`, the long narrative in `<details><summary>Validation details</summary>`, a meta line `Run \`<id>\` · [LangSmith trace](…)`. - **Scrub** local paths and your username from any narrative before publishing — failure summaries leak them. @@ -258,7 +259,7 @@ PR claims privacy mode now hides the Perps balance (the demo bug #42683): 2. Lane = `visual_validation` (visible). Preflight stack. 3. Submit with `description: "Onboard, enable privacy mode in Settings, open the Perps tab, confirm the balance is masked. If the Perps tutorial modal blocks, use the Shield entry modal as the reachable surface."` + `publishEvidence:false`. 4. Poll to completion; assert `artifactRefs` has the before/after pair (not a vacuous skip). -5. Fetch the two PNGs; re-host them to your configured evidence host; assemble the `AEP_VISUAL_VALIDATION` section with the hosted URLs injected into the template's `### After`. +5. Fetch the two PNGs; re-host to `aep-evidence`; assemble the `AEP_VISUAL_VALIDATION` section with the `raw.githubusercontent` URLs injected into the template's `### After`. 6. Show the rendered section; on confirm, upsert the PR body. End-to-end examples for **non-visual** claims (perf, migration, flag-gated, refactor/no-op): **[references/worked-examples.md](references/worked-examples.md).** @@ -269,7 +270,7 @@ Three adjacent things; keep the boundary clear so they compose instead of collid - **AEP** — governed *fleet orchestration*: sandboxes, Temporal, autonomous runs at scale. The heavy engine. - **ADR-0058 recipes** ([decisions#173](https://github.com/MetaMask/decisions/pull/173)) — a *dev-machine inner-loop* proof artifact: a declarative per-PR recipe run against the live app over CDP, emitting `summary.json`/`trace.json`/manifest. -- **pr-validate** (this skill) — the *claim→evidence methodology + taxonomy* both draw on. The Claim Card is the bridge from a PR's claim to the right proof target; the [evidence catalog](references/evidence-catalog.md) is the lane vocabulary; [lane-assertions.md](references/lane-assertions.md) maps each lane to a recipe assertion (and flags the out-of-band, non-UI lanes — the gap raised in review on decisions#173). +- **pr-validate** (this skill) — the *claim→evidence methodology + taxonomy* both draw on. The Claim Card is the bridge from a PR's claim to the right proof target; the [evidence catalog](references/evidence-catalog.md) is the lane vocabulary; [lane-assertions.md](references/lane-assertions.md) maps each lane to a recipe assertion (and flags the out-of-band, non-UI lanes — the gap MajorLift's #173 review raised). pr-validate is the one a human drives; it can dispatch an AEP run or author a recipe as its capture step. @@ -287,7 +288,7 @@ Where pr-validate sits in the PR lifecycle (see the public `pr-workflow` sibling - **Executes, with a confirmation gate on publish.** It runs the harness and captures evidence autonomously; it does not write to the public PR body without showing you the section first. - **Local-only AEP.** No hosted instance. The skill drives the local stack. - **Proves behavior, not code.** Pair with `/review` / `/code-review` for correctness and `/sentry-quota` for span-volume risk. -- **No persisted state.** Each run is fresh. To keep a validation record, ask — nothing is written by default. +- **No persisted state.** Each run is fresh. To keep a validation record, ask — it can go to `exogram-daemon/`, but nothing writes by default. ## Related @@ -297,9 +298,24 @@ Where pr-validate sits in the PR lifecycle (see the public `pr-workflow` sibling - [references/evidence-publishing.md](references/evidence-publishing.md) — PR-body format, non-visual/multi-lane rendering, image re-hosting, recordings→GIF, privacy scrub, ADR-0058 artifact contract. - [references/worked-examples.md](references/worked-examples.md) — end-to-end runs for perf / migration / flag-gated / refactor claims. - [references/lane-assertions.md](references/lane-assertions.md) — lane → declarative recipe-assertion mapping (ADR-0058 bridge). -- [MetaMask/metamask-autonomous-engineering-platform](https://github.com/MetaMask/metamask-autonomous-engineering-platform) — the AEP repo: stack bring-up in its README, plus `docs/demo-runbook.md`, `packages/agent-chain/src/graphs/{visual,perf}-validation/`, and `packages/github/src/pr-body-builder.ts` (the canonical PR-body format this skill mirrors). +- [references/aep-local-run.md](references/aep-local-run.md) — full local-stack bring-up + every gotcha. +- `~/Code/metamask/metamask-autonomous-engineering-platform` — the AEP repo (`docs/demo-runbook.md`, `packages/agent-chain/src/graphs/{visual,perf}-validation/`, `packages/github/src/pr-body-builder.ts`). - `MetaMask/decisions#173` — ADR-0058 Recipe-Based Verification (the adjacent inner-loop proof system). - `/sentry-quota` — sibling skill for span-volume PR review; `/review`, `/code-review` — code correctness. -- `/memory-leak-hunt` — the engine behind the **memory leak** evidence category (C9). pr-validate delegates retention analysis to it and packages the verdict; it also runs standalone. +- **Engine skills — delegate the analysis, package the result.** Each owns a category in + [references/evidence-catalog.md](references/evidence-catalog.md); all run standalone too. + + | category | engine | + |---|---| + | B3 falsifying regression test | `/falsifying-test` | + | B7 deterministic interleaving (concurrency / ordering) | `/race-condition-proof` | + | C4 React render & selector proof | `/react-render-proof` | + | C9 memory leak | `/memory-leak-hunt` | + | D supply-chain / dependency change | `/supply-chain-audit` → delegates capability grants to `/lavamoat-policy-diligence` | + + **An engine that defines its own output contract publishes in it.** `lavamoat-policy-diligence` + is the live case: read-level triage, no verdict, its own header and marker pair. Do not + re-frame it as a Validation Run — see *One comment per evidence kind* in + [references/evidence-publishing.md](references/evidence-publishing.md). - [[reference_aep_local_run]] — the source memory this skill encodes. - [[reference_sentry_project_topology]] — Sentry project mapping for the telemetry-evidence lane. From 3fd9bdac212281cf4412b7ac9f1ecc81154e7329 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 14:52:07 -0400 Subject: [PATCH 035/135] Add `privacy-egress-diligence` skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `app/scripts/constants/sentry-state.ts` decides what user data leaves the machine. It carries 116 fields currently set to `true` — meaning the real value is copied and sent — and it is edited inside ordinary feature PRs (onboarding, swaps, rewards, the analytics controller) with no CODEOWNERS entry, so no privacy reviewer is automatically tagged. Same shape as `lavamoat-policy-diligence`: the diff is mechanical, the judgement is what each grant means. `git diff` finds every newly-`true` field exactly, so the deliverable is not "is it listed" but what the field holds at runtime — a mask path cannot distinguish `selectedTab` from `selectedAddress`. Sorts findings into safe / needs-narrowing / must-not-egress with the evidence and a proposed mask for each, and renders no accept verdict: that call belongs to privacy and legal, and a confident reviewer "this is fine" is precisely what lets an unreviewed field through. Also covers the sibling pipes a PR widens at the same time — new MetaMetrics or Segment properties, and error strings that interpolate runtime values, both of which egress regardless of the mask. --- .../skills/privacy-egress-diligence/skill.md | 113 ++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 domains/security/skills/privacy-egress-diligence/skill.md diff --git a/domains/security/skills/privacy-egress-diligence/skill.md b/domains/security/skills/privacy-egress-diligence/skill.md new file mode 100644 index 00000000..507e82ad --- /dev/null +++ b/domains/security/skills/privacy-egress-diligence/skill.md @@ -0,0 +1,113 @@ +--- +name: privacy-egress-diligence +maturity: experimental +description: >- + Triage a change to what user data leaves the device — the Sentry state masks in + `app/scripts/constants/sentry-state.ts`, new MetaMetrics/Segment event properties, and + error or breadcrumb strings that interpolate user values. Detection is mechanical (the + mask diff), so the deliverable is not "is each field listed" but what each newly + unmasked field actually holds at runtime: a bounded enum is not an account address, and + the mask cannot tell them apart. Sorts findings into safe / needs-narrowing / + must-not-egress with the evidence for each, and hands the accept decision to the people + who own it. Use when a PR touches sentry-state, adds a tracked event or property, or + widens what an error message includes. +--- + +# Privacy egress diligence + +`app/scripts/constants/sentry-state.ts` decides what leaves the user's machine. It is +~11.7KB of per-controller masks, **116 fields currently set to `true`**, and it is edited +inside ordinary feature PRs — onboarding, swaps, rewards, the analytics controller — with +**no CODEOWNERS entry**, so no privacy reviewer is automatically tagged. + +This skill reviews that egress surface the way `lavamoat-policy-diligence` reviews +capability grants: the diff is mechanical, the judgement is what each grant *means*. + +## When to use + +- A PR touches `app/scripts/constants/sentry-state.ts` (either mask). +- A PR adds or widens a MetaMetrics / Segment event property. +- A PR adds an error message, breadcrumb, or log line that interpolates runtime values. +- A new controller lands and its state gets a mask entry. + +## Do not use when + +- The change only *removes* fields or narrows a mask — that shrinks egress; note it and move on. +- The PR is a pure rename with no change to which values are copied. + +## The mechanic, and the trap + +`maskObject` (`shared/lib/object.utils.ts`) walks the mask: + +| Mask value | Effect on the field | +|---|---| +| `true` | **the real value is copied and sent** | +| `false`, `[]`, absent | leaf replaced with its `typeof` string | +| nested object | recurse | +| `[AllProperties]` | applies to dynamic keys — **the field names are not known at review time** | + +Unlisted is safe by default, so the risk direction is one-way: **a field promoted to +`true`.** That is the whole review surface, and `git diff` finds it exactly. + +**The trap:** "the field is in the mask, so someone decided it was fine." The mask *is* the +decision — appearing in it is not evidence that anyone weighed it. Every `true` was typed by +someone, usually while shipping an unrelated feature. Presence proves authorship, not review. + +## Procedure + +1. **Extract the newly-`true` set.** + + ```bash + git diff origin/main...HEAD -- app/scripts/constants/sentry-state.ts | grep -E '^\+.*:\s*true' + ``` + + Also flag any new `[AllProperties]`, which admits keys nobody has seen. + +2. **Resolve each field to its runtime type.** The mask names a path, not a type. Find the + controller field and read what it actually holds — the declaration, and a real value if + the state fixtures have one: + + ```bash + grep -rn "<fieldName>" app/scripts/controllers/ shared/ --include=*.ts + grep -rn "<fieldName>" test/e2e/default-fixture.js app/scripts/../test/**/mock-state.json 2>/dev/null + ``` + +3. **Sort into three buckets.** This is the deliverable. + + | Bucket | What it looks like | Action | + |---|---|---| + | **Safe** | bounded enum, boolean, count, duration, feature-flag name, error code | note the type that makes it safe | + | **Needs narrowing** | object whose *shape* is useful but whose *leaves* are not — a tx object, a quote, a network config | propose the nested mask that keeps the shape and drops the values | + | **Must not egress** | account address, ENS name, balance, token amount, private RPC URL, free text a user typed, anything keyed by address | propose `false`, or a derived non-identifying substitute (a count, a boolean, a hash) | + +4. **Check the sibling surfaces** the same PR may have widened: + - **Event properties** — a new MetaMetrics/Segment property carrying an address or amount + is the same defect on a different pipe. `analytics-instrumentation` covers whether the + event is *correct*; this covers whether its payload is *sendable*. + - **Error strings** — `throw new Error(\`... ${someValue}\`)` reaches Sentry as the message. + Interpolating an address or a balance leaks it regardless of the mask. + +5. **Report, do not rule.** Give each field its bucket, the evidence (declaration site, an + observed value), and a proposed mask. **Do not render an accept/reject verdict** — whether + a given field is acceptable to collect is a call for the privacy and legal owners, and a + confident-sounding "this is fine" from a reviewer is exactly the artifact that lets an + unreviewed field through. + +## Common pitfalls + +| Mistake | Correct approach | +|---|---| +| Treating mask membership as review | Presence proves someone typed it, not that anyone weighed it | +| Reading the mask path as a type | Resolve the field to its declaration; `selectedAddress` and `selectedTab` look alike in a mask | +| Ignoring `[AllProperties]` | Dynamic keys are unreviewable by construction — say so, and ask what generates them | +| Approving because a value is "usually" small | Report the worst case the type admits, not the common case | +| Rendering a verdict | Sort and evidence; the accept decision belongs to privacy/legal | +| Reviewing only `sentry-state.ts` | Event properties and interpolated error strings egress on other pipes | + +## Related + +- `lavamoat-policy-diligence` — same shape for capability grants; read it for the + diff-is-mechanical-judgement-is-not pattern. +- `analytics-instrumentation` — whether an event is correctly *identified* and *gated* + (`isOptIn`, `metaMetricsId`). This skill is about whether its payload is *sendable*. +- `supply-chain-audit` — the third diligence lane, for dependency capability. From dfd0012418997faf6cb46775b3b48ed74930a06f Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 14:54:01 -0400 Subject: [PATCH 036/135] Add `agent-run-cost` skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scripted automation announces its cost in wall-clock time; agentic automation does not. A fan-out of forty subagents and a single call are the same shape in a diff, and the difference surfaces later on a bill attributed to nothing in particular. The token-spend counterpart to `sentry-quota`, and deliberately the same posture: operate on code and PRs before the spend exists, and produce figures rather than adjectives. Names the amplifier triad — fan-out × trigger frequency × no-kill-switch — where one alone is usually fine and all three together is the shape that produces a surprise. Requires the arithmetic be shown, and the worst case stated separately from the expected case, since the budget conversation is about the PR that touches 400 files rather than the normal one. Renders no ship verdict: whether a cost is worth paying belongs to whoever owns the budget. Raised as an open question during ADR-0058 review (MetaMask/decisions#173), where agent token consumption had no estimate. --- .../agentic/skills/agent-run-cost/skill.md | 108 ++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 domains/agentic/skills/agent-run-cost/skill.md diff --git a/domains/agentic/skills/agent-run-cost/skill.md b/domains/agentic/skills/agent-run-cost/skill.md new file mode 100644 index 00000000..f44ee8b0 --- /dev/null +++ b/domains/agentic/skills/agent-run-cost/skill.md @@ -0,0 +1,108 @@ +--- +name: agent-run-cost +maturity: experimental +description: >- + Estimate what an agentic workflow costs to run before it merges — fan-out × trigger + frequency × no kill-switch — and say so in figures rather than adjectives. Agent token + spend is invisible in a diff: a workflow that spawns one agent and one that spawns forty + are the same few lines, and the difference only appears on a bill nobody reads during + review. Produces a per-run and per-month estimate with its arithmetic shown, flags the + three amplifiers, and proposes the cheapest mitigation that preserves the intent. Use + when a PR adds or widens an agentic workflow, an AEP task class, a verification recipe, + or a schedule that runs agents unattended. +--- + +# Agent run cost + +Scripted automation announces its cost in wall-clock time; agentic automation does not. +A fan-out of forty subagents and a single call are the same shape in a diff, and the +difference surfaces later, on a bill, attributed to nothing in particular. + +This is the token-spend counterpart to `sentry-quota`, which guards span volume. Same +posture: operate on **code and PRs**, before the spend exists, and produce figures. + +## When to use + +- A PR adds or widens an agentic workflow, AEP task class, or verification recipe. +- A workflow gains fan-out — an agent per file, per finding, per test, per PR. +- Something agentic moves from opt-in to automatic (a CI trigger, a cron, a git hook). +- An ADR or design proposes agents for work a script already does — the estimate is the + argument, and its absence is usually the tell. + +## Do not use when + +- The workflow is developer-invoked, one agent, no loop — the ceiling is a person's patience. +- The change only narrows fan-out or adds a gate. + +## The amplifier triad + +Cost is not the per-agent price. It is the product of three things, any one of which can be +the whole problem: + +| Amplifier | What it looks like | Effect | +|---|---|---| +| **Fan-out** | an agent per item — per file, per finding, per dimension, per round; nested `parallel` inside `pipeline`; a loop-until-dry with no ceiling | N× per run, and N is often data-dependent rather than fixed | +| **Trigger frequency** | runs on every push rather than on demand; a cron; a label that re-fires on each commit; a retry that respawns the fleet | turns a one-off into a rate | +| **No kill-switch** | no env var, feature flag, or budget cap; nothing to stop it mid-run; no way to disable without a revert | a runaway costs whatever it costs until someone merges a fix | + +One alone is usually fine. **Fan-out × frequency with no kill-switch is the shape that +produces a surprise**, and it is worth naming explicitly in review when all three are present. + +## Producing the estimate + +Show the arithmetic. An estimate whose derivation is hidden is an adjective. + +1. **Count agents per run.** Read the fan-out literally — how many items feed the widest + stage, and whether that number is bounded by the code or by the data. A `pipeline` over + changed files is unbounded by the code; `Array.from({length: 3})` is not. +2. **Estimate tokens per agent.** Prompt + the context it will read + its output. The context + dominates: an agent that reads three files is not an agent that greps a repo. +3. **Multiply, then apply frequency.** Per-run cost × runs per week. State the assumption + about run count — it is the number most likely to be wrong, and naming it lets a reviewer + correct it. +4. **State the worst case separately from the expected case.** The expected case is what it + costs on a normal PR; the worst case is what it costs on the PR that touches 400 files. + Budget conversations are about the second one. +5. **Compare against the alternative.** If a deterministic script covers the same ground, the + estimate belongs next to that script's cost. An agentic approach can still win — for + adversarial review, exploration, fuzzing, or anything with no fixed oracle — but the case + is made by the comparison, not by the capability. + +Report the figures, the assumptions behind them, and the mitigation. **Do not render a +ship/no-ship verdict** — whether a cost is worth paying is a budget decision, and it belongs +to whoever owns the budget. + +## Mitigation ladder + +Cheapest first; stop at the rung that fits. + +1. **Cap the fan-out.** A literal ceiling on items, with a `log()` of what was dropped — + silent truncation reads as full coverage and is worse than the cost. +2. **Narrow the trigger.** On-demand or label-gated instead of every push; on the changed + subset instead of the tree. +3. **Right-size the model per stage.** Mechanical stages rarely need the top tier; reserve it + for the judgement stages. +4. **Add a budget guard.** A token ceiling the workflow checks between stages, so it degrades + instead of running to completion at any price. +5. **Add a kill-switch.** An env var or flag that disables it without a revert. Cheap to add + up front and unavailable exactly when it is needed most. + +## Common pitfalls + +| Mistake | Correct approach | +|---|---| +| "It's just a few agents" | Count them. Data-dependent fan-out has no "just" | +| Estimating output tokens only | Context dominates — an agent that reads the repo costs more than one that answers at length | +| Quoting an average with no worst case | The worst case is the budget conversation | +| Treating a retry as free | A retried fleet is a second fleet | +| Assuming a concurrency cap bounds cost | It bounds *parallelism*, not total spend — queued agents still run | +| Adding a kill-switch after launch | It is needed during the incident it would have prevented | +| Comparing capability instead of cost | "Agents can do this" is not "agents should do this at this price" | + +## Related + +- `sentry-quota` — the same guard for span volume; `fan-out × ungated × no-kill-switch`. +- `pr-validate` — weighs AEP run cost when choosing an evidence lane, and tears the stack + down after; this skill is the review-side version for workflows others will run. +- [`MetaMask/decisions#173`](https://github.com/MetaMask/decisions/pull/173) — ADR-0058 + review, where the missing token-cost estimate was raised as an open question. From 36ea9c8f48a5a725de830133c7b08963b4239aca Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Thu, 30 Jul 2026 16:53:30 -0400 Subject: [PATCH 037/135] Move the AEP run procedure behind a reference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `pr-validate` cost ~9,059 tokens the moment an agent selected it — 5.6× the median of the 26 skills in the open PRs — and a quarter of that was the AEP local-run procedure, which most validations never touch. The skill's own Sufficiency section tells you to prefer a lighter lane; the body charged you for the heavy one regardless. Preflight, run mechanics, and teardown move to `references/aep-local-run.md`, which the body already linked twice and which did not exist. The link was dangling — the same defect class the `knowledge/` guard catches, on a path nothing checks. Publishing keeps the decisions (surface by ownership, post complete once, falsifier-forward, scrub) and points at `references/evidence-publishing.md` for the mechanics it already documents in full. Body 34,431 → 25,736 bytes, so a selected skill is ~6,812 tokens installed rather than ~9,059. Nothing is lost: it sits behind the same progressive disclosure boundary as the other seven references, read when an AEP run is actually warranted. Description trimmed 1,147 → 885 characters. It was over the 1,024 ceiling that #47 enforces, so it would have failed that check on merge. --- .../pr-validate/references/aep-local-run.md | 81 +++++++++++++ .../pr-workflow/skills/pr-validate/skill.md | 108 +++++------------- 2 files changed, 108 insertions(+), 81 deletions(-) create mode 100644 domains/pr-workflow/skills/pr-validate/references/aep-local-run.md diff --git a/domains/pr-workflow/skills/pr-validate/references/aep-local-run.md b/domains/pr-workflow/skills/pr-validate/references/aep-local-run.md new file mode 100644 index 00000000..cccb6b6e --- /dev/null +++ b/domains/pr-workflow/skills/pr-validate/references/aep-local-run.md @@ -0,0 +1,81 @@ +# Running AEP locally + +Everything the local Autonomous Engineering Platform run needs: bring-up, submit, poll, +fetch artifacts, tear down. The hosted instance (`aep.dev.web3factory.consensys.net`) has +not resolved since 2026-06, so local is the only path. + +The skill body links here rather than carrying this inline. An AEP run is the heaviest +lane in the catalog and most validations do not need it — a falsifying test, a single +screenshot, or an artifact CI already produced usually closes the same falsifier. Read +this when you have decided an AEP run is warranted. + +Every bullet below cost a failed run at least once. + +## Preflight — bring up what is down + +The hosted AEP doesn't resolve (`aep.dev.web3factory.consensys.net` is dead as of 2026-06). Everything runs locally. Health-check, then bring up only what's down. + +Fast checks: + +```bash +AEP=~/Code/metamask/metamask-autonomous-engineering-platform +curl -fsS localhost:3000/health >/dev/null && echo "control-plane up" || echo "control-plane DOWN" +curl -fsS localhost:8233 >/dev/null && echo "temporal UI up" || echo "temporal DOWN" +docker ps --format '{{.Names}}' | grep -E 'mm-aep-postgres-dev|mm-aep-temporal-dev' +``` + +Bring-up order (each in its own shell; details + env in the reference): +1. `yarn dev:postgres` (docker `postgres:16-alpine`, `mm-aep-postgres-dev`, port 5432) +2. `yarn dev:temporal` (temporal dev server; UI on 8233) +3. `yarn db:migrate` +4. **worker** — `yarn dev:worker` on **Node ≥ 24.13**, env `ANTHROPIC_API_KEY=host-subscription`, `CLAUDE_CODE_EXECUTABLE=~/.local/bin/claude`, `GITHUB_TOKEN="$(gh auth token)"`, `SANDBOX_PROVIDER=local` (needs JFrog `npm login` first; relies on uncommitted local patches) +5. `yarn dev:control-plane` (`localhost:3000`) + +If any of the local patches (`local-sandbox-adapter.ts` timeout, `claude-agent-runner.ts` auth, the `perf-validation/` graph) are missing from the working tree, the reference says how to restore them — `git status` in the AEP repo should show them modified/untracked. +## Run mechanics — submit, poll, fetch + +The control-plane is a thin REST shell. Submit a PR-validation task, poll the run, pull artifacts from the evidence bundle. + +```bash +CP=localhost:3000 +PR="https://github.com/MetaMask/metamask-extension/pull/<n>" + +# Submit (publishEvidence:false ALWAYS for local runs — the platform otherwise +# writes to the public PR body even on failure, leaking local paths/usernames) +RUN_ID=$(curl -fsS -X POST "$CP/v1/tasks" -H 'content-type: application/json' -d '{ + "repo": "MetaMask/metamask-extension", + "title": "Visual validation — PR #<n>", + "taskClass": "visual_validation", + "externalRef": "'"$PR"'", + "payload": { "prUrl": "'"$PR"'", "description": "<targeting hint>", "publishEvidence": false } +}' | node -e 'process.stdin.on("data",d=>console.log(JSON.parse(d).runId||JSON.parse(d).id))') + +# Poll +curl -fsS "$CP/v1/runs/$RUN_ID" | node -e 'const r=JSON.parse(require("fs").readFileSync(0));console.log(r.status); (r.evidenceBundle?.artifactRefs||[]).forEach(a=>console.log(a.name,a.mediaType))' + +# Fetch an artifact +curl -fsS "$CP/v1/runs/$RUN_ID/artifacts/<artifactName>" -o /tmp/<artifactName> +``` + +- `taskClass`: `visual_validation` or `perf_validation`. The worker auto-enriches the payload from `prUrl` (pulls headSha, base, diff, files, linked issues via the GitHub app) — you only supply `prUrl` + a `description` targeting hint. +- The **targeting hint** (`payload.description`) is how you steer the agent to the surface under test. Be specific: which screen, which control, what to toggle. For hard-to-reach surfaces, name the reachable fallback (e.g. the Shield entry modal stands in for the Perps tutorial modal, which is gated in the default fixture). +- Artifact regex allows **png/jpg/log/txt only** — no video. Screen recordings need the side-channel recipe (catalog + publishing reference). + +### Concurrent runs (multiple agents / parallel lanes) + +Five shared resources need per-run isolation on one machine — collisions cross-contaminate evidence *silently* (wrong session's logs attributed to a run), which is an integrity failure, not flakiness: **(1)** CDP debug ports — derive per run, never hardcode; **(2)** e2e harness service ports (anvil/proxy/fixture/mocha) — one e2e run at a time per worktree, one worktree per agent (`wt new`), and never rebuild `dist/` in a worktree with an active run; **(3)** artifact dirs — per-run namespaces; `test-artifacts/` is per-worktree shared state, harvest failure artifacts before the next run overwrites the same test-title dir; **(4)** evidence-repo uploads — run-scoped paths (`pr-<n>/<run-id>/`), retry-with-fresh-sha on 409, never overwrite another run's published files; **(5)** commit-pinning — pin only after your own final upload lands, verifying your files exist at that sha. Safe to share: JFrog login, a read-only `dist/`, the AEP stack itself. + +### Trust the evidence (anti-reward-hacking) + +A green result is not proof. The vacuous-pass trap is the floor: if `promptCrafter` errors, the chain "passes" via skip with **zero artifacts** — a pass is only real if `evidenceBundle.artifactRefs` is non-empty with the expected media. Beyond that, every lane must clear a trustworthiness gate before you believe or publish it: **does the artifact show the *claimed* surface** (not a spinner/wrong screen), **does the test exercise the *changed* code** (fails on `main`), **does the signal exceed noise**, **could the assertion have failed**? The Claim Card's Falsifier is the anchor. Full gate + per-lane traps: **[references/evidence-trustworthiness.md](references/evidence-trustworthiness.md).** + +### perf_validation caveat + +The `perf-validation/` graph is **uncommitted local AEP work** (added 2026-06-11). It writes falsifiable network/static/smoke assertions and gives the tester deterministic `.aep/` helpers (CDP netlog, phase segmentation, source-map chunk membership). It requires a `yarn webpack --test` build first (the browserify `build:test` has no code splitting, so `import()` never hits the network there). Temporal caps activity results at ~2MB — artifact refs must be content-free; only `evidenceBundle` carries base64. If the graph isn't in the working tree, perf runs won't register — fall back to manual DevTools/CDP capture (catalog). +## Teardown — always, on every exit path + +The stack is the heaviest thing this skill starts — postgres + temporal + a Node worker + control-plane — and the worker holds a live Claude session while the autonomous run itself spends tokens. It is **on-demand, not resident**: bring it up for the validation window, **tear it down when the run(s) finish**. Left up, it's the single largest reclaimable footprint on a shared host and quietly keeps a Claude seat warm. + +- **On a host managed by `aep-stack` (systemd):** `aep-stack up` to preflight, **`aep-stack down` when done** — stops the services; the `--rm` postgres/temporal containers are removed, so state resets on the next `up` (fine — each run is fresh anyway). +- **Otherwise:** stop the `yarn dev:*` processes and `docker rm -f mm-aep-postgres-dev mm-aep-temporal-dev`. +- **Tear down on every exit path** — pass, refutation, *or* abort. A failed or abandoned run leaves the stack up exactly as much as a passing one; the usual leak is walking away after a refutation without stopping it. diff --git a/domains/pr-workflow/skills/pr-validate/skill.md b/domains/pr-workflow/skills/pr-validate/skill.md index 660ef602..b2ae4b83 100644 --- a/domains/pr-workflow/skills/pr-validate/skill.md +++ b/domains/pr-workflow/skills/pr-validate/skill.md @@ -1,6 +1,6 @@ --- name: pr-validate -description: Validate a MetaMask PR with objective evidence — primarily the Autonomous Engineering Platform (AEP) harness (visual_validation for visible UI behavior, perf_validation for non-visible perf behavior), backed by complementary evidence (Sentry query links, screenshots, screen recordings→GIF, DevTools/CDP output, bundle-size, web-vitals, test/CI results). Drives the AEP local stack end to end: preflight (postgres + temporal + worker + control-plane) → submit POST /v1/tasks → poll GET /v1/runs/:id → fetch artifacts → assemble an evidence bundle → publish to the PR body (re-hosting images to a public repo, scrubbing local paths). Match evidence to the PR's specific falsifiable claim, not a fixed checklist. Triggers on /pr-validate, /pr-validate visual, /pr-validate perf, /pr-validate preflight, /pr-validate status, /pr-validate evidence, /pr-validate plan, or when the user mentions validating/proving a PR, AEP / visual validation / perf validation, capturing evidence for a PR, before/after screenshots, a screen recording or GIF for a PR, attaching Sentry links or DevTools output as proof, or publishing an evidence bundle to a PR body. +description: Validate a MetaMask PR with objective evidence — match the evidence to the PR's specific falsifiable claim rather than running a fixed checklist. Covers the full catalog: before/after screenshots, falsifying regression tests, perf and render proofs, bundle and LavaMoat diffs, Sentry and Tempo links, state-migration and vault checks, plus the Autonomous Engineering Platform (AEP) harness for autonomous visual and perf capture. Assembles an evidence bundle and publishes it to the PR body, images re-hosted and local paths scrubbed. Triggers on /pr-validate and its subcommands (visual, perf, preflight, status, evidence, plan, lane, compare), or when the user mentions validating or proving a PR, AEP or visual/perf validation, capturing evidence, before/after screenshots, a screen recording for a PR, attaching Sentry or DevTools output as proof, or publishing an evidence bundle. --- # /pr-validate @@ -93,7 +93,7 @@ Not for code-correctness review (use `/review`, `/code-review`) or span-quota re | Invocation | Behavior | |---|---| -| `/pr-validate <pr>` | **Flagship.** Read the PR → state the claim → pick lanes → [preflight](#preflight) → run AEP lane(s) + gather complementary evidence → assemble bundle → **propose** the PR-body section and confirm before publishing. | +| `/pr-validate <pr>` | **Flagship.** Read the PR → state the claim → pick lanes → [preflight](references/aep-local-run.md) → run AEP lane(s) + gather complementary evidence → assemble bundle → **propose** the PR-body section and confirm before publishing. | | `/pr-validate plan <pr>` | Dry run: read the PR, state the claim, recommend lanes + targeting hints. No stack, no run. Cheap first step when unsure. | | `/pr-validate visual <pr>` | AEP `visual_validation` only. | | `/pr-validate perf <pr>` | AEP `perf_validation` only (local/uncommitted graph — see [caveat](#perf_validation-caveat)). | @@ -105,76 +105,15 @@ Not for code-correctness review (use `/review`, `/code-review`) or span-quota re `<pr>` is a number or URL on `MetaMask/metamask-extension` unless another repo is given. Every variant runs Step 1 (extract the Claim Card) first — the claim decides the lane, even when you named one. -## Preflight +## Running AEP -The hosted AEP doesn't resolve (`aep.dev.web3factory.consensys.net` is dead as of 2026-06). Everything runs locally. Health-check, then bring up only what's down. **Full procedure + every gotcha: [references/aep-local-run.md](references/aep-local-run.md).** Skim it before a first run in a session — each bullet there cost a failed run. +The hosted instance is dead; everything runs locally. Bring-up, submit/poll/fetch, and +teardown are in **[references/aep-local-run.md](references/aep-local-run.md)** — read it +once you have decided an AEP run is warranted, not before. -Fast checks: - -```bash -AEP=~/Code/metamask/metamask-autonomous-engineering-platform -curl -fsS localhost:3000/health >/dev/null && echo "control-plane up" || echo "control-plane DOWN" -curl -fsS localhost:8233 >/dev/null && echo "temporal UI up" || echo "temporal DOWN" -docker ps --format '{{.Names}}' | grep -E 'mm-aep-postgres-dev|mm-aep-temporal-dev' -``` - -Bring-up order (each in its own shell; details + env in the reference): -1. `yarn dev:postgres` (docker `postgres:16-alpine`, `mm-aep-postgres-dev`, port 5432) -2. `yarn dev:temporal` (temporal dev server; UI on 8233) -3. `yarn db:migrate` -4. **worker** — `yarn dev:worker` on **Node ≥ 24.13**, env `ANTHROPIC_API_KEY=host-subscription`, `CLAUDE_CODE_EXECUTABLE=~/.local/bin/claude`, `GITHUB_TOKEN="$(gh auth token)"`, `SANDBOX_PROVIDER=local` (needs JFrog `npm login` first; relies on uncommitted local patches) -5. `yarn dev:control-plane` (`localhost:3000`) - -If any of the local patches (`local-sandbox-adapter.ts` timeout, `claude-agent-runner.ts` auth, the `perf-validation/` graph) are missing from the working tree, the reference says how to restore them — `git status` in the AEP repo should show them modified/untracked. - -## Teardown - -The stack is the heaviest thing this skill starts — postgres + temporal + a Node worker + control-plane — and the worker holds a live Claude session while the autonomous run itself spends tokens. It is **on-demand, not resident**: bring it up for the validation window, **tear it down when the run(s) finish**. Left up, it's the single largest reclaimable footprint on a shared host and quietly keeps a Claude seat warm. - -- **On a host managed by `aep-stack` (systemd):** `aep-stack up` to preflight, **`aep-stack down` when done** — stops the services; the `--rm` postgres/temporal containers are removed, so state resets on the next `up` (fine — each run is fresh anyway). -- **Otherwise:** stop the `yarn dev:*` processes and `docker rm -f mm-aep-postgres-dev mm-aep-temporal-dev`. -- **Tear down on every exit path** — pass, refutation, *or* abort. A failed or abandoned run leaves the stack up exactly as much as a passing one; the usual leak is walking away after a refutation without stopping it. - -## Run mechanics (submit → poll → fetch) - -The control-plane is a thin REST shell. Submit a PR-validation task, poll the run, pull artifacts from the evidence bundle. - -```bash -CP=localhost:3000 -PR="https://github.com/MetaMask/metamask-extension/pull/<n>" - -# Submit (publishEvidence:false ALWAYS for local runs — the platform otherwise -# writes to the public PR body even on failure, leaking local paths/usernames) -RUN_ID=$(curl -fsS -X POST "$CP/v1/tasks" -H 'content-type: application/json' -d '{ - "repo": "MetaMask/metamask-extension", - "title": "Visual validation — PR #<n>", - "taskClass": "visual_validation", - "externalRef": "'"$PR"'", - "payload": { "prUrl": "'"$PR"'", "description": "<targeting hint>", "publishEvidence": false } -}' | node -e 'process.stdin.on("data",d=>console.log(JSON.parse(d).runId||JSON.parse(d).id))') - -# Poll -curl -fsS "$CP/v1/runs/$RUN_ID" | node -e 'const r=JSON.parse(require("fs").readFileSync(0));console.log(r.status); (r.evidenceBundle?.artifactRefs||[]).forEach(a=>console.log(a.name,a.mediaType))' - -# Fetch an artifact -curl -fsS "$CP/v1/runs/$RUN_ID/artifacts/<artifactName>" -o /tmp/<artifactName> -``` - -- `taskClass`: `visual_validation` or `perf_validation`. The worker auto-enriches the payload from `prUrl` (pulls headSha, base, diff, files, linked issues via the GitHub app) — you only supply `prUrl` + a `description` targeting hint. -- The **targeting hint** (`payload.description`) is how you steer the agent to the surface under test. Be specific: which screen, which control, what to toggle. For hard-to-reach surfaces, name the reachable fallback (e.g. the Shield entry modal stands in for the Perps tutorial modal, which is gated in the default fixture). -- Artifact regex allows **png/jpg/log/txt only** — no video. Screen recordings need the side-channel recipe (catalog + publishing reference). - -### Concurrent runs (multiple agents / parallel lanes) - -Five shared resources need per-run isolation on one machine — collisions cross-contaminate evidence *silently* (wrong session's logs attributed to a run), which is an integrity failure, not flakiness: **(1)** CDP debug ports — derive per run, never hardcode; **(2)** e2e harness service ports (anvil/proxy/fixture/mocha) — one e2e run at a time per worktree, one worktree per agent (`wt new`), and never rebuild `dist/` in a worktree with an active run; **(3)** artifact dirs — per-run namespaces; `test-artifacts/` is per-worktree shared state, harvest failure artifacts before the next run overwrites the same test-title dir; **(4)** evidence-repo uploads — run-scoped paths (`pr-<n>/<run-id>/`), retry-with-fresh-sha on 409, never overwrite another run's published files; **(5)** commit-pinning — pin only after your own final upload lands, verifying your files exist at that sha. Safe to share: JFrog login, a read-only `dist/`, the AEP stack itself. - -### Trust the evidence (anti-reward-hacking) - -A green result is not proof. The vacuous-pass trap is the floor: if `promptCrafter` errors, the chain "passes" via skip with **zero artifacts** — a pass is only real if `evidenceBundle.artifactRefs` is non-empty with the expected media. Beyond that, every lane must clear a trustworthiness gate before you believe or publish it: **does the artifact show the *claimed* surface** (not a spinner/wrong screen), **does the test exercise the *changed* code** (fails on `main`), **does the signal exceed noise**, **could the assertion have failed**? The Claim Card's Falsifier is the anchor. Full gate + per-lane traps: **[references/evidence-trustworthiness.md](references/evidence-trustworthiness.md).** - -### perf_validation caveat - -The `perf-validation/` graph is **uncommitted local AEP work** (added 2026-06-11). It writes falsifiable network/static/smoke assertions and gives the tester deterministic `.aep/` helpers (CDP netlog, phase segmentation, source-map chunk membership). It requires a `yarn webpack --test` build first (the browserify `build:test` has no code splitting, so `import()` never hits the network there). Temporal caps activity results at ~2MB — artifact refs must be content-free; only `evidenceBundle` carries base64. If the graph isn't in the working tree, perf runs won't register — fall back to manual DevTools/CDP capture (catalog). +It is the heaviest lane here: the full stack plus autonomous-agent tokens. Weigh that +against a lighter lane that closes the same falsifier (see [Sufficiency](#sufficiency--how-much-is-enough)), +and **tear the stack down on every exit path** — pass, refutation, or abort. ## Complementary evidence @@ -195,7 +134,7 @@ Screen recordings (motion a still can't prove): `mm` + a Playwright `recordVideo Match the bar to the claim; stop when the claim's falsifier is closed. Don't over-instrument a copy fix; don't under-prove a high-stakes claim. - **One lead lane that closes the falsifier** is enough for low-risk, single-claim PRs (a copy fix → one screenshot; a bug fix → the falsifying test). -- **Weigh AEP's cost before reaching for it.** A `visual_validation`/`perf_validation` run spins the full stack *and* burns autonomous-agent tokens — by far the most expensive lane. Use it when the claim genuinely needs autonomous capture of a reachable surface; when a lighter lane closes the same falsifier (a single `mm` screenshot, a falsifying test, a CDP capture, an artifact CI already produced), prefer it and skip the stack. Whenever you do start it, tear it down after (see [Teardown](#teardown)). +- **Weigh AEP's cost before reaching for it.** A `visual_validation`/`perf_validation` run spins the full stack *and* burns autonomous-agent tokens — by far the most expensive lane. Use it when the claim genuinely needs autonomous capture of a reachable surface; when a lighter lane closes the same falsifier (a single `mm` screenshot, a falsifying test, a CDP capture, an artifact CI already produced), prefer it and skip the stack. Whenever you do start it, tear it down after (see [references/aep-local-run.md](references/aep-local-run.md)). - **Lead + one corroborator** for perf/telemetry (a number *and* its source) and for anything user-facing that also moves a metric. **For a perf-targeting PR the lead lane is the measured impact itself** — a paired A/B benchmark at the current head (C5) or equivalent — never mechanism evidence alone (chunk membership, netlog exclusion prove the improvement is *possible*, not that it *happened*). A perf PR also always carries correctness + non-regression lanes: changed-surface tests green at head, affected flows exercised, neutral profile within noise. (2026-07-22, #42795 lesson.) - **Lead + integrity lane** for high-stakes surfaces regardless of size: persisted-state (migration + vault), money (tx simulation), permissions (LavaMoat + manifest), security/keyring. Size-S doesn't lower the bar here. - **Per-claim** for mixed PRs — each Claim Card needs its own closed falsifier; a strong UI proof doesn't cover the metric it also shifts. @@ -205,16 +144,23 @@ Stop when each claim has one trustworthy artifact that would have shown its fals ## Publishing the evidence bundle -**Public, outward-facing action — always confirm the rendered section with the user before writing the PR body.** Match AEP's own format so the section is idempotent and reviewer-familiar. Full recipe (markers, image re-hosting, recordings, the `### After` injection, privacy scrub): **[references/evidence-publishing.md](references/evidence-publishing.md).** Essentials: - -- **Canonical header — every validation output leads with the exact literal `## 🧪 Validation Run`.** Same string in a PR comment and in the PR-body section, never reworded or demoted — the constancy is what makes it scannable/Ctrl-F-able, like Copilot's fixed `## Pull request overview`. Line 2 is the meta line: `**Verdict:** ✅ proven — **Claim:** <one-liner>` then `head \`<sha>\` · <date> · lanes: <list>`. Enforced mechanically by `hooks/pr-evidence-gate.py` (a validation/verification/evidence heading or AEP marker without the literal blocks the `gh` write). -- **Post complete, once — and know which regime the surface is in.** Comments are **push** (audience notified once at post time; edits are silent): hold until every planned lane is present or consciously dropped, and put substantive additions or changed verdicts in a **new comment referencing the original**, never a silent edit. The PR **body** is **pull** (consulted at review time): the idempotent marker upsert on re-validation at a new head is correct there. Typo-level comment edits are fine. (Decision: `exogram-core/decisions/2026-07-23-publish-complete-bundles.md`; framework: Reprise `push-pull-artifact-edit-regimes`.) -- **Falsifier-forward.** After the meta line, foreground **what would have falsified the claim and how each falsifier is closed** — the falsifier is the load-bearing content, not a footnote. Structure the body as "what would make this false → the evidence that rules it out," not a lane inventory with a `falsifiers closed` line buried at the bottom. The reviewer should see the disproof attempt first. -- **Don't restate CI results.** Lint/build/typecheck/test/changelog outcomes are already on the PR's Checks tab — the reviewer sees them. Cite a CI result in the comment only to **highlight something specific** they'd otherwise miss; otherwise reference "green in Checks" or omit it. Restating "423 pass / 0 fail" is bundle noise (the display-side counterpart to the catalog's *rely on CI* collection rule). -- **Re-host images first.** Control-plane artifact URLs are `localhost` and won't render on GitHub. Push to the public `MajorLift/metamask-extension-skills` repo, branch `aep-evidence`, via the contents API; link the `raw.githubusercontent.com` URLs. -- **Use idempotency markers** so a re-run replaces in place: wrap the whole section in `<!-- VALIDATION_RUN_START -->` … `<!-- VALIDATION_RUN_END -->`; inside it, AEP's own `<!-- AEP_VISUAL_VALIDATION_START/END -->` for the status block and `<!-- AEP_SCREENSHOTS_START/END -->` for images, injected into the PR template's `### **After**` section (replacing the `<!-- [screenshots/recordings] -->` placeholder) when present. -- **Verdict-first, lanes nested:** under the canonical header, hand-assembled AEP blocks demote to `### AEP Visual Validation` (leave AEP's own service-published `##` blocks untouched) with `**✅ Passed**` / `**❌ Failed**` / `ℹ️`, the long narrative in `<details><summary>Validation details</summary>`, a meta line `Run \`<id>\` · [LangSmith trace](…)`. -- **Scrub** local paths and your username from any narrative before publishing — failure summaries leak them. +**Public, outward-facing — always confirm the rendered section with the user before writing +a PR body.** Full recipe, markers, image re-hosting, recordings, and the privacy scrub: +**[references/evidence-publishing.md](references/evidence-publishing.md).** + +The parts that decide *whether* to publish, rather than how: + +- **Surface follows ownership** — the PR body when you authored it, a comment when validating + someone else's. Never publish a failure to another author's PR unprompted. +- **Post complete, once.** A comment is push: its audience is notified at post time and edits + are silent, so hold until every planned lane is present, and put changed verdicts in a new + comment referencing the original. The PR body is pull, so an idempotent marker upsert is + correct there. +- **Lead with the canonical header** `## 🧪 Validation Run`, then verdict and claim. +- **Falsifier-forward** — what would have made this false, and what rules it out, before any + lane inventory. +- **Don't restate CI.** Lint, build, and test results are already on the Checks tab. +- **Scrub** local paths and usernames; failure summaries leak them. ## Validation output format From 2af1db1e11586ee2f0c85a67380382d4edc76852 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 03:25:01 -0400 Subject: [PATCH 038/135] Restore the full `pr-validate` description MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Trimmed to 885 characters to fit a 1,024 budget that turned out to be unverified — no operator observed rejects or truncates a longer description, and several over 1,024 load today. Back to 1,147, under the 1,536 budget. The description is the discovery surface, so the 262 characters were trigger cues: the subcommand list and the phrasings that route a request here rather than to another skill. Cutting them made the skill harder to select, which is a functional loss and not a cosmetic one. The body restructure is unaffected — that removed duplication behind a reference, which costs nothing at selection time. From 0b4bb26445ddabd54d8215c428921f113c5629d1 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 04:28:32 -0400 Subject: [PATCH 039/135] Actually restore the `pr-validate` description MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 2af1db1 carries the reasoning but not the change: its `git add` was chained into a command the commit guard rejected, so the commit was created from an already staged tree and landed empty — same 885-character description as its parent. This applies it. Back to 1,147 characters, under the 1,536 budget. The 262 characters are trigger cues — the subcommand list and the phrasings that route a request here rather than to a sibling skill — so losing them made the skill harder to select, which is a functional loss rather than a cosmetic one. --- domains/pr-workflow/skills/pr-validate/skill.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/pr-validate/skill.md b/domains/pr-workflow/skills/pr-validate/skill.md index b2ae4b83..351667db 100644 --- a/domains/pr-workflow/skills/pr-validate/skill.md +++ b/domains/pr-workflow/skills/pr-validate/skill.md @@ -1,6 +1,6 @@ --- name: pr-validate -description: Validate a MetaMask PR with objective evidence — match the evidence to the PR's specific falsifiable claim rather than running a fixed checklist. Covers the full catalog: before/after screenshots, falsifying regression tests, perf and render proofs, bundle and LavaMoat diffs, Sentry and Tempo links, state-migration and vault checks, plus the Autonomous Engineering Platform (AEP) harness for autonomous visual and perf capture. Assembles an evidence bundle and publishes it to the PR body, images re-hosted and local paths scrubbed. Triggers on /pr-validate and its subcommands (visual, perf, preflight, status, evidence, plan, lane, compare), or when the user mentions validating or proving a PR, AEP or visual/perf validation, capturing evidence, before/after screenshots, a screen recording for a PR, attaching Sentry or DevTools output as proof, or publishing an evidence bundle. +description: Validate a MetaMask PR with objective evidence — primarily the Autonomous Engineering Platform (AEP) harness (visual_validation for visible UI behavior, perf_validation for non-visible perf behavior), backed by complementary evidence (Sentry query links, screenshots, screen recordings→GIF, DevTools/CDP output, bundle-size, web-vitals, test/CI results). Drives the AEP local stack end to end: preflight (postgres + temporal + worker + control-plane) → submit POST /v1/tasks → poll GET /v1/runs/:id → fetch artifacts → assemble an evidence bundle → publish to the PR body (re-hosting images to a public repo, scrubbing local paths). Match evidence to the PR's specific falsifiable claim, not a fixed checklist. Triggers on /pr-validate, /pr-validate visual, /pr-validate perf, /pr-validate preflight, /pr-validate status, /pr-validate evidence, /pr-validate plan, or when the user mentions validating/proving a PR, AEP / visual validation / perf validation, capturing evidence for a PR, before/after screenshots, a screen recording or GIF for a PR, attaching Sentry links or DevTools output as proof, or publishing an evidence bundle to a PR body. --- # /pr-validate From 95e6d8b698d2165b4455b68c419a3709c8f0901e Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 06:46:39 -0400 Subject: [PATCH 040/135] Add `race-condition-proof` skill for ordering guarantees under concurrency The falsifier for a concurrency claim is a test that never interleaved: a sequential run exercises no race and produces a green indistinguishable from a real pass. The skill therefore treats showing the interleaving occurred as the proof obligation, not the assertion passing. Lands in `stability/` beside `memory-leak-hunt`, the other defect-class engine `pr-validate` delegates to. --- .../skills/race-condition-proof/skill.md | 110 ++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100644 domains/stability/skills/race-condition-proof/skill.md diff --git a/domains/stability/skills/race-condition-proof/skill.md b/domains/stability/skills/race-condition-proof/skill.md new file mode 100644 index 00000000..3ff6d2d3 --- /dev/null +++ b/domains/stability/skills/race-condition-proof/skill.md @@ -0,0 +1,110 @@ +--- +name: race-condition-proof +description: Prove an ordering guarantee under concurrency — that when B arrives during A's pending window, A is canceled, or completes first, or the two commit in a defined order. Covers race conditions, retries, cancellation, supersession, debounce/throttle, locks, queues, and async state machines, where correctness IS the interleaving rather than a value. Builds a deterministic interleaving harness (fake timers advanced into the pending window, concurrent launch, microtask stepping) and asserts each guarantee separately, including asymmetric ones where two paths deliberately differ. The falsifier is a test that never interleaved — operations run to completion in sequence exercise no race and produce a vacuous green indistinguishable from a real pass, so the proof obligation is to show the interleaving occurred, not that the assertion passed. Triggers on /race-condition-proof, or when asked to prove a race condition is fixed, test cancellation or supersession, validate retry or debounce ordering, write a deterministic interleaving test, or check whether a concurrency test actually exercises the race. Callable by pr-validate as the engine behind its B7 deterministic-interleaving evidence category, and named by falsifying-test as its sibling for ordering bugs. +maturity: experimental +--- + +# /race-condition-proof + +A race is nondeterministic in the wild, so you cannot validate "the stale retry was canceled" by +running the code and hoping the interleaving occurs. The evidence has to **make the race +deterministic** — force the exact interleaving, then assert the outcome. + +The claim shape is distinctive: not a value, not a behavior, but an *ordering guarantee*. "When a +newer write supersedes a pending retry, the stale retry is dropped." No screenshot, benchmark, or +value assertion touches that. + +> **Falsifier.** A test that never interleaved. If the operations ran to completion in sequence, +> no race was exercised and the green is vacuous — and it looks identical to a real pass. The +> proof obligation is to show **the interleaving happened**, not that the assertion passed. + +This is the reward-hack specific to the category, and it is easy to write by accident: `await` +the first operation, then start the second, then assert. Every assertion passes. Nothing was +tested. + +## Method + +1. **State each guarantee separately, in interleaving terms.** Not "retries work" but "when B + arrives during A's pending window, A's recovery event does not fire." One sentence per + guarantee, each naming the arriving operation, the window, and the expected outcome. + + **Asymmetric guarantees are usually the crux.** Two paths that deliberately behave differently + — a primary retry that *is* cancelable by a newer write, a backup retry that is *not* because a + split write could leave backed-up keys stale — need one forced interleaving each. A harness + that proves the symmetric half and assumes the other has proven the easy one. + +2. **Force the interleaving.** Control time and ordering rather than waiting for them: + + | technique | purpose | + |---|---| + | `jest.useFakeTimers()` + `advanceTimersByTimeAsync(DELAY)` | fire the delayed action at a known point | + | `Promise.all([opA, opB])` | overlap operations rather than sequencing them | + | `advanceTimersByTimeAsync(0)` | step to a precise interleaving point between overlapping ops | + + The shape that matters: launch A, advance time *into* its pending window, inject B *during* + that window, then assert. Never `await opA` before starting `opB`. + +3. **Verify the interleaving before believing the assertion.** This is step 2's trust-gate and it + is not optional — confirm time was advanced into the pending window and the superseding op was + launched concurrently. Reading the assertion tells you nothing; a sequential test asserts the + same things and passes. + + The cheap check: **break the implementation and confirm the test fails.** Revert the ordering + logic, keep the test file byte-identical, re-run. A test that still passes never exercised the + race. Show both runs — that mutation pair is the evidence, not the green run alone. + +4. **Assert the negative side explicitly.** Cancellation guarantees are proven by absence: + `expect(recoveryEvent).not.toHaveBeenCalled()`. A suite that only asserts things happened + cannot detect a stale operation that ran when it should have been dropped. Pair every + "must complete" (`.toHaveBeenCalledWith(...)`) with its "must not" counterpart. + +5. **Corroborate the integration path if the claim reaches beyond the unit.** The deterministic + harness is a *model* — exhaustive and fast, but a model. For a high-stakes claim, add one live + forced-race capture in the real runtime (CDP/injection, the force-the-unobservable technique) + to show the race exists where the model says it does. Unit harness for coverage, live capture + for reality; use both when the cost of being wrong is high. + +6. **Report transition telemetry with enough labeling to distinguish branches.** `retry-recovered` + is ambiguous when there are two retry paths; `set-retry-recovered` vs + `set-backup-retry-recovered` is not. If the observable can't tell the branches apart, it can't + witness an asymmetric guarantee. + +## Output + +``` +Ordering guarantees — <component> <claim> + +| guarantee | forced how | assertion | result | +|---|---|---|---| +| B during A's window cancels A | advance <DELAY>, inject B via Promise.all | recovery .not.toHaveBeenCalled() | pass | +| backup completes despite newer write | advance <DELAY>, inject B | .toHaveBeenCalledWith(...) | pass | + +Interleaving verified: <how time was advanced / where the concurrent op was injected> +Mutation check: <impl reverted> → <N failures>, test file unchanged +Live corroboration: <capture> | not run +``` + +Lead with the guarantee table — one row per guarantee, each naming how the interleaving was +forced. A row without a forcing mechanism is a sequential test wearing the category's clothes. +Report the mutation pair (head green / reverted red) as the evidence that the harness discriminates. + +## Scope — what this is NOT + +- **Not the generic falsifying test.** These *are* falsifying tests, but the category is the + *technique* (forced deterministic interleaving) and the *claim shape* (ordering, not values). + `falsifying-test` names this skill as its sibling for ordering bugs; use that one when the claim + is a value or a behavior and the base/head arms are the whole story. +- **Not flake diagnosis.** A test that fails intermittently is a different problem from a + guarantee that needs proving. Determinism here is the *method*, not the goal. +- **Not performance under load.** Throughput and contention are timing questions; this is about + ordering correctness at a specific interleaving. + +## Notes + +Correctness of the *reasoning* about a race is not something to assert from reading. Where the +guarantee depends on runtime semantics — what an `AbortController` actually cancels, whether a +microtask runs before a timer callback — cite the behavior or demonstrate it in the harness rather +than describing it. + +Engine for `pr-validate` category **B7 (deterministic interleaving)**; the full category note +lives at `exogram-daemon/artifacts/evidence-taxonomy/category-concurrency-temporal-ordering.md`. From 7032425994745a9cacd90d8ed05aab53006f78fe Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 07:13:16 -0400 Subject: [PATCH 041/135] Add `debug` orchestrator as the symptom-first sibling of `pr-validate` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `pr-validate` is handed a claim and looks for the observation that would falsify it. Debugging starts from a symptom and has to generate the hypothesis first, which is where the expensive failure lives: the theory is yours, nobody else is positioned to challenge it, and confirmation is cheap. Routes the symptom to the engine that owns its defect class rather than reimplementing any investigation, so both orchestrators share one set of engines. Carries `pr-validate`'s trust gates, which bind harder here — a weak instrument in review yields a claim someone challenges, in debugging it yields a theory nobody checks. --- domains/pr-workflow/skills/debug/skill.md | 101 ++++++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100644 domains/pr-workflow/skills/debug/skill.md diff --git a/domains/pr-workflow/skills/debug/skill.md b/domains/pr-workflow/skills/debug/skill.md new file mode 100644 index 00000000..556a1b72 --- /dev/null +++ b/domains/pr-workflow/skills/debug/skill.md @@ -0,0 +1,101 @@ +--- +name: debug +description: Locate the cause of a symptom you cannot yet explain — a crash, a leak, a flake, a production error spike, a number that moved. The sibling of pr-validate: where pr-validate is handed a claim and looks for the observation that would falsify it, this is handed a symptom and must generate the hypothesis first, then kill it. Classifies the symptom into a defect class, routes to the engine skill that owns that class (memory-leak-hunt, race-condition-proof, react-render-proof, sentry-grafana-cross-ref, extension-errors-debugging, typescript-compiler-blindspots, supply-chain-audit), and holds the investigation to the same evidence bar pr-validate applies — an instrument that cannot fail is not evidence, a null needs its sensitivity stated, and a finding is scoped to what the change introduced versus what pre-existed. Stops when the cause is located or the class is excluded, not when a plausible story is available. Triggers on /debug, or when asked to debug, diagnose, or investigate a symptom, find why something is slow, leaking, flaky, or erroring, chase a production alert to its cause, or reproduce a bug that cannot be reproduced by hand. +maturity: experimental +--- + +# /debug + +`pr-validate` is given a claim and looks for the observation that would prove it false. +This is given a **symptom** and has to produce the hypothesis before anything can falsify it. + +That difference is the whole skill. In review, the claim is someone else's and the social +pressure runs toward scepticism. In debugging, the hypothesis is *yours*, nobody else is +positioned to challenge it, and the expensive failure is building three hours of work on the +first theory that fit the first observation. + +## When To Use + +- A symptom with no established cause: a crash, a hang, a leak, an intermittent test, an + error-rate step change, a metric that moved without a deploy that explains it. +- A bug you cannot reproduce by hand and therefore cannot yet observe. +- A production signal that needs chasing back to code. + +## Do Not Use When + +- The PR states a claim and you need it settled — that is `/pr-validate`. +- The cause is known and you are validating the fix — that is `/pr-validate`, or the engine + skill directly. +- You want an after-the-fact writeup of a resolved failure — that is a postmortem, not this. + +## Workflow + +1. **State the symptom as an observation, not a theory.** "Popup memory grows ~105 MB per + open/close cycle" — not "the popup leaks because of the snow hook". The theory is the + output, never the input. +2. **Classify into a defect class** (table below). If two classes fit, run both; do not pick + the one you find more interesting. +3. **Delegate to the engine.** Each owns its own method and its own falsifier. This skill + routes and holds the bar; it does not re-implement the investigation. +4. **Kill the hypothesis before extending it.** Name the observation that would rule it out, + and go looking for that observation specifically. A hypothesis that has only ever been + confirmed has not been tested. +5. **Stop on a located cause or an excluded class.** A plausible story is not a stop condition. + +## Symptom → engine + +| Symptom | Class | Engine | +|---|---|---| +| Memory grows across a repeated flow; tab or worker dies over time | retention | `memory-leak-hunt` | +| Intermittent failure; passes on rerun; order-dependent | interleaving | `race-condition-proof` | +| UI janks, re-renders excessively, selector recomputes | wasted render work | `react-render-proof` | +| Production error spike, latency change, or a metric that moved | production signal | `sentry-grafana-cross-ref` | +| Extension-specific: MV3 vs MV2, background vs UI context, service-worker lifecycle | platform | `extension-errors-debugging` | +| Runtime value disagrees with its declared type; green typecheck, wrong behaviour | type/reality drift | `typescript-compiler-blindspots` | +| Started after a dependency change; new capability or transitive edge | supply chain | `supply-chain-audit` | +| None of the above, or several | — | bisect to a change first, then re-classify | + +## The evidence bar carries over + +The engines are shared with `pr-validate`, and so are its trust gates. They matter more here, +because in review a weak instrument produces a weak claim someone else will challenge — in +debugging it produces a wrong theory nobody checks. + +- **An instrument that cannot fail is not evidence.** Before trusting a measurement, establish + it can report the negative: a positive control that must move, a base arm that must fail. +- **A null needs its sensitivity stated.** "No difference" and "could not have detected one" + print identically. Calibrate, then report the zero against what the instrument demonstrably + resolves. +- **Scope to the change.** Classify each finding as introduced-here versus pre-existing. + Report pre-existing separately and uncharged, or you will attribute an old defect to a new + diff. +- **A negative result carries the scope of its search.** "No leak found", "nothing in the logs", + "the artifact does not exist" are claims about where you looked. Name the stores searched in the + finding itself — filesystem, artifact bucket, issue tracker, the other process's logs. If you + cannot name them, the search is not finished. +- **Measure on an isolated machine.** Timing- and GC-sensitive numbers taken on a contended host + are not noisy-but-usable, they are *stably wrong* — several runs will agree with each other and + disagree with reality. Replicate across hosts, not just across runs, before trusting a figure. +- **Collect the whole battery, not the discriminating member.** Instruments that stay flat are + data: the joint pattern localises the defect in a way no single reading does. + +## Output + +A short investigation record, not a narrative: + +``` +SYMPTOM observation, as measured +CLASS defect class, and why (with the classes considered and dropped) +HYPOTHESES each with the observation that would kill it, and whether that was found +CAUSE located mechanism, at file:line — or "class excluded", which is a real result +NOT CAUSE hypotheses killed, kept so the next person does not re-walk them +``` + +Killed hypotheses are part of the deliverable. Deleting them makes the surviving one look +inevitable and hands the next investigator the same dead ends. + +## Scope — what this is NOT + +- Not a fix. It locates and evidences the cause; the change is a separate act. +- Not a replacement for the engines. Each owns its method; this chooses among them. +- Not an incident-management process. No severity, comms, or timeline. From 704128a61661a0af629beb0e3a6210a4a656f596 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 08:22:30 -0400 Subject: [PATCH 042/135] Rename `pr-validate` to `evidence` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two of the three assumptions in the old name were false. The skill runs in the inner loop against uncommitted changes, and on a symptom with no claim and no PR at all, so `pr-` narrowed it to one of three modes. `validate` named an activity where the deliverable is an artifact, and read as a checklist exercise — the posture the skill spends its opening section arguing against. `evidence` names what it produces, and covers a refutation as naturally as a confirmation. The verdicts are proven, refuted, and inconclusive; a name promising proof would make two of those read as failure. The description is rewritten rather than search-replaced. It now states all three modes, since the old one described only the PR case and so under-selected for the other two, and it names the trigger as `mms-evidence` — the form the installer actually emits. Eleven skills across the repo still promise the unprefixed `/<name>` in their descriptions while installing prefixed; this corrects the one being renamed. --- .../hooks/pr-evidence-gate.py | 0 .../references/aep-local-run.md | 0 .../references/claim-extraction.md | 2 +- .../references/evidence-catalog.md | 6 ++-- .../references/evidence-gate-setup.md | 8 ++--- .../references/evidence-publishing.md | 10 +++--- .../references/evidence-trustworthiness.md | 2 +- .../references/lane-assertions.md | 0 .../references/worked-examples.md | 0 .../skills/{pr-validate => evidence}/skill.md | 34 +++++++++---------- .../testing/skills/falsifying-test/skill.md | 4 +-- 11 files changed, 33 insertions(+), 33 deletions(-) rename domains/pr-workflow/skills/{pr-validate => evidence}/hooks/pr-evidence-gate.py (100%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/aep-local-run.md (100%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/claim-extraction.md (94%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/evidence-catalog.md (98%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/evidence-gate-setup.md (87%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/evidence-publishing.md (97%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/evidence-trustworthiness.md (90%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/lane-assertions.md (100%) rename domains/pr-workflow/skills/{pr-validate => evidence}/references/worked-examples.md (100%) rename domains/pr-workflow/skills/{pr-validate => evidence}/skill.md (84%) diff --git a/domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py b/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py similarity index 100% rename from domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py rename to domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py diff --git a/domains/pr-workflow/skills/pr-validate/references/aep-local-run.md b/domains/pr-workflow/skills/evidence/references/aep-local-run.md similarity index 100% rename from domains/pr-workflow/skills/pr-validate/references/aep-local-run.md rename to domains/pr-workflow/skills/evidence/references/aep-local-run.md diff --git a/domains/pr-workflow/skills/pr-validate/references/claim-extraction.md b/domains/pr-workflow/skills/evidence/references/claim-extraction.md similarity index 94% rename from domains/pr-workflow/skills/pr-validate/references/claim-extraction.md rename to domains/pr-workflow/skills/evidence/references/claim-extraction.md index fdd90417..9dde89d9 100644 --- a/domains/pr-workflow/skills/pr-validate/references/claim-extraction.md +++ b/domains/pr-workflow/skills/evidence/references/claim-extraction.md @@ -1,6 +1,6 @@ # Claim extraction -The linchpin of pr-validate: before choosing any lane, turn the PR into a **falsifiable, surface-specific claim**. Every lane is only as good as the claim it tests. A vague claim ("improves perf", "fixes the bug") can't be proven or refuted; a sharp claim names the precondition, action, observable outcome, and what would disprove it. +The linchpin of evidence: before choosing any lane, turn the PR into a **falsifiable, surface-specific claim**. Every lane is only as good as the claim it tests. A vague claim ("improves perf", "fixes the bug") can't be proven or refuted; a sharp claim names the precondition, action, observable outcome, and what would disprove it. ## Read these, in order diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md similarity index 98% rename from domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md rename to domains/pr-workflow/skills/evidence/references/evidence-catalog.md index 8c8b7680..d3578ca3 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md @@ -80,7 +80,7 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- - **Capture:** `ui/helpers/utils/performance-observers.ts`; `window.stateHooks.getLongTaskMetricsWithTBT()` → `{count, totalDuration, maxDuration, tbt, tbtRating}`. TBT good<200 / needs-improvement<600 / poor>600. Sampled 10% prod / 100% test. ## C4. React render & selector proof - - **Engine: the `react-render-proof` skill.** Delegate the measurement to it; it runs the source/delivery/metric gates, derives the needle from real build output, repeats the capture, and returns a band (or "not resolvable at this n" with an MDE). pr-validate packages the result. + - **Engine: the `react-render-proof` skill.** Delegate the measurement to it; it runs the source/delivery/metric gates, derives the needle from real build output, repeats the capture, and returns a band (or "not resolvable at this n" with an MDE). evidence packages the result. - **Proves:** a component/selector stopped over-rendering (cascade-amplification before/after — exogram `react-redux-performance`). - **Capture:** WDYR via `ENABLE_WHY_DID_YOU_RENDER` (`.metamaskrc` or env) — wired in `app/scripts/development/wdyr.ts` (`trackAllPureComponents`); console logs each unnecessary re-render. `yarn devtools:react` for the Profiler flame graph. Selectors use `reselect`'s `createSelector`, which **does expose a real `.recomputations()` counter** — read it (sample on an interval if the count should visibly climb) rather than injecting a log into the selector body; an injected log is an authored claim, a library API is an observation. *(This entry previously said there was no built-in counter. There is.)* - **Bar:** the delivery check comes before the number. An arm whose manipulation cannot be observed in the built bundle produces a null indistinguishable from "small effect" — and reports as the second. @@ -137,7 +137,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ # D. Build output ## C9. Retention-path analysis — memory leak from code ⭐ *(static; lead for leak claims)* -- **Engine: the `memory-leak-hunt` skill.** For a memory-leak claim, delegate the analysis to `memory-leak-hunt` — it runs Phase-1 static pairing (and Phase-2 heap investigation if a primitive can't be paired) and returns the paired/unpaired sites + verdict. pr-validate keeps **memory leak** as the evidence category: it invokes the skill on the diff and packages the result (in-situ scan capture, plus the lifecycle test / retainer graph if Phase 2 ran) as the category's evidence. The lane spec below is the method that skill implements. +- **Engine: the `memory-leak-hunt` skill.** For a memory-leak claim, delegate the analysis to `memory-leak-hunt` — it runs Phase-1 static pairing (and Phase-2 heap investigation if a primitive can't be paired) and returns the paired/unpaired sites + verdict. evidence keeps **memory leak** as the evidence category: it invokes the skill on the diff and packages the result (in-situ scan capture, plus the lifecycle test / retainer graph if Phase 2 ran) as the category's evidence. The lane spec below is the method that skill implements. - **Proves:** "X is retained past its lifecycle boundary" / "collection Y grows unboundedly" — argued from code, no runtime needed. This is the lane that works at **review time** (does this PR *introduce* retention?) and leads fix-side validation (does the fix *break* the retention path?). C7 is the runtime corroborator, not the lead — leaks need many cycles to exceed noise. Full category: `exogram-daemon/artifacts/evidence-taxonomy/category-memory-retention-from-code.md`. - **Capture — the holder → held → boundary triple, per suspect:** (1) the **holder** (listener, closure, module singleton, accumulating collection, timer); (2) the **held set** — the *specific* objects pinned (list the closure's captures; note when a closure links two objects' GC); (3) the **outlived boundary** (`destroy()`, stream close, instance replacement, request completion). Method: **pair every acquire with its release site** (`on`↔`removeListener`, push↔drain, assign↔null) — the absence of the pair, cited at the acquire site, IS the finding. Four canonical shapes: unbounded accumulator (defeated guard, no drain) · stale-instance listeners on replacement · unremoved listener + capture set · retention past `destroy()`. - **Scope to the diff, or you invent findings.** Classify every flagged primitive as *introduced by this PR* (in the added lines) vs *pre-existing* (already in the file). Charge only the introduced ones to the PR; report pre-existing un-paired primitives separately and uncharged. On extension#40684 the two new stream listeners each had a `removeListener` on `onStreamClosed` (the exact fix a reviewer suggested) and the new pending-request Map had its `.delete` — no leak introduced — while three pre-existing un-torn-down listeners were surfaced but left uncharged, matching how the human/bot reviewers treated them in-thread. This lane *is* the retention review automated; a heap snapshot (C7) is warranted only for an introduced primitive it cannot pair. @@ -151,7 +151,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **Proves:** a module moved to the intended (lazy) chunk and no longer ships on the critical path. Requires the webpack build. Mirrors AEP `perf-chunks`. ## D3. LavaMoat policy / supply-chain capability diff - - **Engine: the `supply-chain-audit` skill** (umbrella — lockfile/manifest diff, advisories, Socket Security, install scripts), which delegates capability grants to **`lavamoat-policy-diligence`** (per-grant call-site justification). Delegate the dependency change to the umbrella; it returns a disposition per lane. pr-validate keeps **supply-chain capability diff** as the evidence category and packages the output. Note the lanes are independent: a clean policy diff does not mean a safe dependency, and a known CVE never appears as a new grant. + - **Engine: the `supply-chain-audit` skill** (umbrella — lockfile/manifest diff, advisories, Socket Security, install scripts), which delegates capability grants to **`lavamoat-policy-diligence`** (per-grant call-site justification). Delegate the dependency change to the umbrella; it returns a disposition per lane. evidence keeps **supply-chain capability diff** as the evidence category and packages the output. Note the lanes are independent: a clean policy diff does not mean a safe dependency, and a known CVE never appears as a new grant. - **Proves:** a dependency change (bump/add/lockfile) grants **no *unjustified* new capability** — the supply-chain-risk lane. Note the bar: for a bump the policy *will* change, so "empty diff" is the WRONG test; the right test is **every new grant is justified by the dep's function**. Full category (trust-boundary framing, generalizes past LavaMoat to any capability-containment mechanism): `exogram-daemon/artifacts/evidence-taxonomy/category-supply-chain-capability-diff.md`. - **Capture:** **Prefer the CI-generated policy whenever one is available.** `@metamaskbot update-policies` regenerates the policy files from a real run of the code and `validate-lavamoat-policies` fails the build on drift, so the committed policy on a bot-run PR *is* the authoritative artifact — diff that. Regenerating locally when a current CI policy exists only re-does a machine that is already trusted, and a local run's provenance is weaker (your node/OS/lockfile resolution, not CI's). **Local regen is the fallback**, for when the bot hasn't run yet, the branch is unpushed, or you need a variant CI didn't cover: `yarn webpack:lavamoat:policy:build` (`:mv2` / `:mv3` for variants) over `lavamoat/webpack/build/policy.json` (+ `policy-override.json`). Either way, `git diff` the policy across **all 8 variants** (mv{2,3}/{main,beta,flask,experimental}) — a grant can appear in one and not others. Then audit **grant-by-grant**: new **globals** (`fetch`, `importScripts`, `WebAssembly`) / **builtins** (`fs`, `child_process`) on a dep that shouldn't need them, new **packages** edges to powerful APIs, or an identifier substitution (`pkgC>name` replacing `pkgB>pkgA>name` = possible dep swap). Falsifier = a surprising grant ("I wonder what it's using this for"). Guide: lavamoat.github.io/guides/policy-diff/. `allowScripts` in `package.json` gates install scripts. diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-gate-setup.md b/domains/pr-workflow/skills/evidence/references/evidence-gate-setup.md similarity index 87% rename from domains/pr-workflow/skills/pr-validate/references/evidence-gate-setup.md rename to domains/pr-workflow/skills/evidence/references/evidence-gate-setup.md index f1df5b9c..21a2c998 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-gate-setup.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-gate-setup.md @@ -23,7 +23,7 @@ Add a `PreToolUse` hook with matcher `Bash` that runs the script with `python3`. "hooks": [ { "type": "command", - "command": "python3 /absolute/path/to/pr-validate/hooks/pr-evidence-gate.py" + "command": "python3 /absolute/path/to/evidence/hooks/pr-evidence-gate.py" } ] } @@ -32,10 +32,10 @@ Add a `PreToolUse` hook with matcher `Bash` that runs the script with `python3`. } ``` -Resolve the path to wherever `pr-validate` lives on disk. Note that `tools/install` copies only the `references`/`scripts`/`assets`/`adapters` bundles into `~/.claude/skills/mms-pr-validate/` — the `hooks/` directory is **not** part of the installed bundle. Point the `command` at your checked-out skills repo instead: +Resolve the path to wherever `evidence` lives on disk. Note that `tools/install` copies only the `references`/`scripts`/`assets`/`adapters` bundles into `~/.claude/skills/mms-evidence/` — the `hooks/` directory is **not** part of the installed bundle. Point the `command` at your checked-out skills repo instead: ``` -<skills-repo>/domains/pr-workflow/skills/pr-validate/hooks/pr-evidence-gate.py +<skills-repo>/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py ``` **When it blocks:** the hook exits `2` and prints the reason (which claim, what artifact/tracker it needs) to stderr. Claude Code surfaces that to the model, which self-corrects — attaches the missing artifact/tracker or downgrades the verdict — and re-posts. No manual intervention needed. @@ -44,7 +44,7 @@ Resolve the path to wherever `pr-validate` lives on disk. Note that `tools/insta These are independent of the hook; the skill needs them whether or not you install the gate. -1. **`gh pr comment` must be permitted — pick a grant model.** pr-validate posts its evidence bundle as a PR review comment (`gh pr edit` if publishing into your own PR body). Four options, in descending order of standing safety: +1. **`gh pr comment` must be permitted — pick a grant model.** evidence posts its evidence bundle as a PR review comment (`gh pr edit` if publishing into your own PR body). Four options, in descending order of standing safety: | Model | How | Tradeoff | |---|---|---| diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md similarity index 97% rename from domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md rename to domains/pr-workflow/skills/evidence/references/evidence-publishing.md index 6f031e99..4316c52f 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-publishing.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md @@ -117,7 +117,7 @@ Screenshots block (injected into `### After`, or appended under `### Screenshots ## Step 3 — Choose the surface by ownership, then publish **Publish surface depends on my relationship to the PR** (see exogram -`pr-validate-publish-surface-by-ownership`). Determine it FIRST: +`evidence-publish-surface-by-ownership`). Determine it FIRST: ```bash PR=<n>; REPO=MetaMask/metamask-extension @@ -223,7 +223,7 @@ The platform can't collect video (artifact regex = png/jpg/log/txt). Capture out ## Re-validation runs: delta-first presentation, every verdict re-earned (2026-07-21) -The common loop — a run refutes a claim, the author pushes a fix, `/pr-validate` re-runs at the new head — gets a **delta report**, not a second full bundle: +The common loop — a run refutes a claim, the author pushes a fix, `/evidence` re-runs at the new head — gets a **delta report**, not a second full bundle: - **Presentation is delta-only.** Full exhibits only for lanes whose outcome changed (flipped verdict / new lane / new residual). Unchanged lanes collapse to a `Prior run | This run` ledger, each row with a fresh run-log link from the new head plus one link to the prior run's comment for the full exhibits — and say so ("unchanged rows re-run at `<head>`; full exhibits in the prior run"). - **Evidence is never delta.** Evidence is head-pinned: re-run every automated lane at the new head and re-earn every verdict with a fresh artifact. "Unchanged" is a conclusion from the re-run, never a carried-over assumption (the stale-baseline trap at report level). Re-running is cheap — the falsifier harness already exists from the first run. @@ -231,7 +231,7 @@ The common loop — a run refutes a claim, the author pushes a fix, `/pr-validat - New head → **new hosted artifact directory keyed to the fix commit** (`pr-<n>/fix-<sha>/`), commit-pinned raw URLs; never overwrite a prior run's published files. - Residuals the fix intentionally leaves get their own row/section — don't round a fixed-with-residual claim up to fully proven. -Source of truth: `exogram-core/memory/pr-validate-revalidation-delta-reports.md`. +Source of truth: `exogram-core/memory/evidence-revalidation-delta-reports.md`. ## Lead with a lane-status ledger (no silent absence) @@ -293,11 +293,11 @@ contract. `hooks/pr-evidence-gate.py` enforces the canonical literal only on bod trip it, which is the tell that the two are different artifacts rather than one with a different skin. -**Per-scenario presentation (2026-07-21):** the same applies one level down — when the evidence spans multiple test scenarios (flag-on vs flag-off, control vs treatment in an A/B falsifier, numbered manual-testing steps), give each scenario its **own sub-section**: a heading naming the scenario in observation terms, one line on what it tests plus its verdict, and that scenario's artifacts co-located under it. Never bunch all scenarios' artifacts into one large evidence dump — the reviewer verifies "under condition X, artifact shows Y" one condition at a time, and a merged block destroys that mapping even when every artifact is real. For long artifact sets use a `<details>` block *per scenario*, not a merge. (Preference: exogram-core `memory/pr-validate-present-scenarios-separately.md`; instance #44610.) +**Per-scenario presentation (2026-07-21):** the same applies one level down — when the evidence spans multiple test scenarios (flag-on vs flag-off, control vs treatment in an A/B falsifier, numbered manual-testing steps), give each scenario its **own sub-section**: a heading naming the scenario in observation terms, one line on what it tests plus its verdict, and that scenario's artifacts co-located under it. Never bunch all scenarios' artifacts into one large evidence dump — the reviewer verifies "under condition X, artifact shows Y" one condition at a time, and a merged block destroys that mapping even when every artifact is real. For long artifact sets use a `<details>` block *per scenario*, not a merge. (Preference: exogram-core `memory/evidence-present-scenarios-separately.md`; instance #44610.) ## Artifact contract (ADR-0058 alignment) -To stay interoperable with the recipe-based verification system (MetaMask/decisions#173), shape the bundle like its reviewer-visible contract where practical: a `summary.json` (claim → verdict → evidence refs), a `trace.json` (the run/assertion log), and an artifact manifest (names + media types), with screenshots/video as the confidence layer. Publishing then becomes "render `summary.json` into the PR section." This keeps pr-validate's output and a recipe's output the same shape — see [lane-assertions.md](lane-assertions.md). Don't hand-roll a divergent format. +To stay interoperable with the recipe-based verification system (MetaMask/decisions#173), shape the bundle like its reviewer-visible contract where practical: a `summary.json` (claim → verdict → evidence refs), a `trace.json` (the run/assertion log), and an artifact manifest (names + media types), with screenshots/video as the confidence layer. Publishing then becomes "render `summary.json` into the PR section." This keeps evidence's output and a recipe's output the same shape — see [lane-assertions.md](lane-assertions.md). Don't hand-roll a divergent format. ## Checklist before you publish diff --git a/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md b/domains/pr-workflow/skills/evidence/references/evidence-trustworthiness.md similarity index 90% rename from domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md rename to domains/pr-workflow/skills/evidence/references/evidence-trustworthiness.md index 06478a68..f398b2ef 100644 --- a/domains/pr-workflow/skills/pr-validate/references/evidence-trustworthiness.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-trustworthiness.md @@ -19,7 +19,7 @@ A green result is not proof. An agent — or an eager run — can produce eviden 11. **Lanes derive from the Manual testing steps — a CI-green row is not a lane** — the Validation Run's rows are generated top-down from the claim and the PR's own **Manual testing steps**, never bottom-up from whatever links already exist. For each step the claim depends on, the lane's payload is the **captured output of executing that step** (step "in Discover, group by `trace`" → a Discover permalink / **linked** trace-id table showing N rounds → N distinct `trace_id`s, per item 12), or an honest ⏳ naming the missing capture with a tracker. A row restating CI ("tests green at head `<sha>` in [CI run]") duplicates the Checks tab and is deleted — and a validation surface carries **zero** CI references, full stop: no `actions/runs` links, no "green at head" clauses, no "as context (only)" retention. The earlier carve-out here ("a CI link is admissible as context on a beyond-CI row") was itself the next costume: within a day all four sibling bodies (extension#43928–#43931) shipped restatements phrased as the exception — rows *leading* with "green at head … in [Unit tests CI]", the same link repeated 3× per body, the remediated row keeping it re-labeled "as context only" — while the gate's excuse regex matched the mere word "revert", so vocabulary, not evidence, discharged the class. The revert lane cites the revert **outcome** (which blocks failed, at which commit); its green-at-head half is the Checks tab's information and is omitted. A carve-out in an emit-time gate is an instruction to generation to phrase every violation as the exception — deliberate exceptions route through the human, never through an excuse predicate. Borrowed evidence — a sibling PR's capture, a unit falsifier standing in for the named live surface — never upgrades an uncaptured lane to ✅: "mechanism live-proven" co-located with "was not exercised" is an inflated verdict; downgrade it. Emit-time trigger: `pr-evidence-gate.py` classes `ci-restatement` (unconditional since 2026-07-21: any CI link / CI-green phrase in validation scope fires — no verdict co-location required, no beyond-CI excuse) and `inflated-verdict`, with the shipped extension#43928 rows and the carve-out-blessed "as context" shape as regression cases (2026-07-21). 12. **Identifiers resolve in one click — a bare id is a digging assignment** — trace ids, event ids, run ids, SHAs are *pointers into a system*, not evidence. Publishing a bunch of raw trace ids hands the reviewer the job of reconstructing project/environment/time window and querying Sentry themselves — it fails item 9's ~30-second test by construction (item 9 makes the signal *findable*; this item makes it *checkable*). Every identifier published as evidence is either hyperlinked to its resolving surface (the Sentry trace/event permalink, or an absolute-windowed Discover query pre-filtered to exactly those ids) or accompanied by the re-hosted captured output (query-result rows / envelope excerpt showing the discriminating fields) — ideally both. Special case that produced the rule: ids captured **locally** (mockttp forwarder, envelope intercept) never reached Sentry, so no permalink can exist — the re-hosted capture is the *only* admissible form, and pasting the id fragments plus a re-run recipe is the "spec necessary / output sufficient" violation wearing ids as decoration (extension#43931 Validation row, 2026-07-21). Rule of construction: when any item in this gate blesses an evidence class by name ("trace-id table", "envelope log"), it means the class's *resolvable instance*, never its bare tokens — a blessed class name is otherwise the next costume. Emit-time trigger: `pr-evidence-gate.py` class `bare-identifier`; converse-of-gate note: the prior gate *whitelisted* `trace_ids?` as beyond-CI payload and its own fix-message recommended "trace-id table" unqualified — second occurrence of "audit the gate for whitelists of the violating shape." 13. **Terminal exhibits are reader-native — a live link or a visual; a dump behind a link is still an opaque reference** — item 12 makes every pointer resolve in one click; this item constrains what it may resolve *to*. A positive verdict's terminal artifact is one of the two media a reviewer natively consumes: a **live link into the resolving system** (Sentry trace/event permalink, absolute-windowed Discover query pre-filtered to the claim) or a **visual capture** (screenshot/recording, annotated or cropped to the discriminating region). Raw files (`.log`/`.json`/`.har`, MB-scale dumps) are **appendix-only** — linked once for auditability, never the exhibit a claim rests on: a link whose target is a raw dump passes item 12 and fails item 9 one click later; the digging moved a hop away, it did not disappear (extension#43931 *second* remediation, 2026-07-21: the `bare-identifier` fix shipped a ✅ row whose sole resolver was a re-hosted ~70KB run log). Two corollaries: (a) **the gate items are conjunctive** — a fix for the newest item must re-pass all prior items; satisfying resolvability with an artifact that fails legibility is the generator's next costume; (b) **ascertain the terminal medium at step zero and pick the capture lane that can produce it** — a local intercept (mockttp envelope forwarder) can never yield a live Sentry permalink, so for Sentry-observable claims it is the supplementary falsifier lane and live ingest (dev build → `SENTRY_DSN_DEV`/test-metamask) is primary, precisely because it terminates in permalinks + screenshots; choosing a lane that cannot produce the terminal medium silently displaces it. Emit-time trigger: `pr-evidence-gate.py` class `dump-resolver`, with the remediated extension#43931 row as the regression case (2026-07-21). -14. **Manual testing steps are the validation contract — steps present ⇒ live evidence definitionally required; an impossibility waiver contradicting an executable step is invalid** — the PR's own **Manual testing steps** are the author's assertion that the claimed behavior *is* live-observable, and how: each numbered step is an executability proof (a step a human reviewer can run, `/pr-validate` can run) and its text is the capture spec. Item 11 derives the lanes from the steps top-down; this item closes the other side of the hatch — a lane derived from a step may not then be *waived by argument*. Emit-time procedure: build the per-step coverage map (step → executed-output artifact); for any step without one, the only admissible state is a **per-step** ⏳ + tracker whose blocker is that step's *own* unmet precondition, checked against the step text. Three waiver-inflation patterns from the producing instance (extension#43228/#42869/#44538, 2026-07-21 — all three shipped articulate impossibility rationales in the same body whose Manual testing steps asserted the opposite): (a) **borrowed impossibility** — the excuse imported from a different mechanism or sibling PR (#43228 waived its live lane citing the async remote-flag read race, which is #44538's mechanism; #43228's overrides are build-time env vars, and its steps 1–4 are directly executable in a dev build); (b) **lane-limitation universalized** — one harness's gap stated as global impossibility (#42869: "the e2e harness emits no error `event` envelope, so … not capturable pre-merge" — the step says *dev build with Sentry enabled, trigger an error*, which ingests error events into test-metamask without the e2e harness); (c) **blocked-scope inflation** — a genuinely blocked precondition of one half of the claim expanded to waive the whole lane (#44538: LaunchDarkly provisioning [#7482](https://github.com/MetaMask/MetaMask-planning/issues/7482) blocks only the *prod-flag* half; the step's own text names the dev-injectable alternative — "or inject it into persisted `RemoteFeatureFlagController` state"). If a step is *truly* non-executable, the waiver is still inadmissible alone: the Manual testing steps are then wrong and are corrected in the same edit — a document may not simultaneously instruct a reviewer to observe X and declare X unobservable. The honest-⏳ lane blessed throughout this gate is for *not yet done*, never for *argued away*: an eloquent impossibility rationale is the cheapest token sequence that satisfies every prior item (no fake capture, no CI link, no bare id, no dump) — the generator's costume for the coverage axis. Emit-time trigger: `pr-evidence-gate.py` class `step-waiver` ("not demonstrable / capturable / observable", "not separately captured", "not attached", "rests entirely/solely on the falsifiers/unit/revert" — unconditional in validation scope; no tracker or artifact excuses it), with the three shipped waiver paragraphs as regression cases and the gate verified-blocking on all three live bodies (2026-07-21). Detection gap: the per-step consistency check (does a deferral's blocker match the step's own precondition?) stays procedural — the gate sees vocabulary, not step semantics. +14. **Manual testing steps are the validation contract — steps present ⇒ live evidence definitionally required; an impossibility waiver contradicting an executable step is invalid** — the PR's own **Manual testing steps** are the author's assertion that the claimed behavior *is* live-observable, and how: each numbered step is an executability proof (a step a human reviewer can run, `/evidence` can run) and its text is the capture spec. Item 11 derives the lanes from the steps top-down; this item closes the other side of the hatch — a lane derived from a step may not then be *waived by argument*. Emit-time procedure: build the per-step coverage map (step → executed-output artifact); for any step without one, the only admissible state is a **per-step** ⏳ + tracker whose blocker is that step's *own* unmet precondition, checked against the step text. Three waiver-inflation patterns from the producing instance (extension#43228/#42869/#44538, 2026-07-21 — all three shipped articulate impossibility rationales in the same body whose Manual testing steps asserted the opposite): (a) **borrowed impossibility** — the excuse imported from a different mechanism or sibling PR (#43228 waived its live lane citing the async remote-flag read race, which is #44538's mechanism; #43228's overrides are build-time env vars, and its steps 1–4 are directly executable in a dev build); (b) **lane-limitation universalized** — one harness's gap stated as global impossibility (#42869: "the e2e harness emits no error `event` envelope, so … not capturable pre-merge" — the step says *dev build with Sentry enabled, trigger an error*, which ingests error events into test-metamask without the e2e harness); (c) **blocked-scope inflation** — a genuinely blocked precondition of one half of the claim expanded to waive the whole lane (#44538: LaunchDarkly provisioning [#7482](https://github.com/MetaMask/MetaMask-planning/issues/7482) blocks only the *prod-flag* half; the step's own text names the dev-injectable alternative — "or inject it into persisted `RemoteFeatureFlagController` state"). If a step is *truly* non-executable, the waiver is still inadmissible alone: the Manual testing steps are then wrong and are corrected in the same edit — a document may not simultaneously instruct a reviewer to observe X and declare X unobservable. The honest-⏳ lane blessed throughout this gate is for *not yet done*, never for *argued away*: an eloquent impossibility rationale is the cheapest token sequence that satisfies every prior item (no fake capture, no CI link, no bare id, no dump) — the generator's costume for the coverage axis. Emit-time trigger: `pr-evidence-gate.py` class `step-waiver` ("not demonstrable / capturable / observable", "not separately captured", "not attached", "rests entirely/solely on the falsifiers/unit/revert" — unconditional in validation scope; no tracker or artifact excuses it), with the three shipped waiver paragraphs as regression cases and the gate verified-blocking on all three live bodies (2026-07-21). Detection gap: the per-step consistency check (does a deferral's blocker match the step's own precondition?) stays procedural — the gate sees vocabulary, not step semantics. 15. **The exhibit lives in the body — link AND visual; a live link alone is the verification path, not the exhibit** — item 13 blessed the terminal media as a *disjunction* (live link OR visual), and generation took the cheaper disjunct: a Discover permalink is producible from the API token alone, a screenshot needs a browser session — so extension#44540's live-ingestion exhibit shipped as a permalink + prose counts, with nothing in the PR body a reader could look at (2026-07-21: "only sentry link and not screenshot that makes it immediately obvious how evidence validates pr"). A live link defers validation behind **click + auth + query rendering + column interpretation** — the dump-resolver displacement one hop further, with the mountain now behind a login: it fails item 9's ~30-second test at the moment of the click, and for any reader *without* Sentry org access (most PR reviewers) a link-only exhibit degrades to a bare identifier (item 12) behind an auth wall. The repaired rule is a **conjunction**: a positive verdict's headline exhibit is an **embedded visual** — screenshot/recording of the linked resolving view (Discover result rows, trace waterfall), cropped/annotated to the discriminating region, captioned with what it should show — **and** the co-located live permalink (absolute-windowed) as the independent-verification path. Neither substitutes for the other: link-only hides the exhibit; visual-only is independently unverifiable. The 2026-07-16 clause "screenshots ride along when a browser session is available; the API token alone yields links + JSON, which is the automatable minimum" was the self-authored escape hatch of this axis (family: the "as context" carve-out, the honest-⏳ waiver): the *automatable minimum* got promoted to the shipped standard because it was the cheapest compliant artifact. A capture lane that cannot screenshot its resolving view is a lane gap to fix before publish (drive a browser session to the Discover URL), never a licensed downgrade — deliberate exceptions route through the human. Emit-time trigger: `pr-evidence-gate.py` class `link-only-exhibit` (non-negated verdict + `sentry.io` link + no image/recording embed in the unit), with the shipped #44540 paragraph as the regression case and the prior suite's permalink-only ALLOW cases flipped/augmented — third occurrence of "an ALLOW case containing the violating tokens is a specification of the next costume." Detection gaps: verdict co-location is required, so a no-verdict link-only paragraph evades mechanically; the visual-without-link converse stays procedural under item 12. 16. **The audit chain is mechanical — quote, don't transcribe; pin, don't point** — an exhibit's inline data must be a **verbatim, greppable excerpt** of the artifact (full-length identifiers, raw capture lines quoted exactly), and every repo-hosted artifact link must be a **commit-pinned, line-anchored permalink** (`/blob/<sha>/…#Lx-Ly`) to the discriminating lines. Producing real artifacts and then hand-transcribing digests severs the claim→artifact bridge at every link: an ellipsized id (`24b1e2da…`) cannot be grepped against any artifact even when the log is linked in the same block; a reformatted data block cannot be distinguished from confabulation without redoing the dig, so it reads as *claims in the form of data*; a branch-ref `/blob/<branch>/` link is a mutable pointer whose target can be rewritten after review (not tamper-evident); a file-level link without line anchors lands the reader at the top of a 1,000-line dump. The producing instance (extension#43929 validation comment, 2026-07-21) had every number substantiated by four re-hosted run logs — real, included, resolvable — and still drew "still no immediately auditable evidence just claims in the form of data": each prior item individually near-passed while the exhibit↔artifact binding stayed **editorial** (transcription + file link) instead of **mechanical** (quotation + pinned line anchor). Emit-time procedure: for every inline datum, quote the raw capture line it comes from (fenced, verbatim, full ids) and anchor it (`#L<n>`); pin every evidence link to the SHA (press `y` on the GitHub file view). Emit-time trigger: `pr-evidence-gate.py` classes `truncated-identifier` (a co-located resolver does NOT excuse it — the resolver resolves the full id, not the fragment the reader holds; hash-equality prose exempt) and `mutable-ref`, plus the **surface hole** — the shipped comment was published via `gh api` PATCH, which the porcelain-only matcher (`gh pr|issue edit|create|comment`) never saw; fifth occurrence of the converse-of-gate rule, one level down: audit the gate for *spellings of the write it cannot see*, not just tokens it excuses. **CORRECTION 2026-07-30 — this hole was recorded as closed and is not.** Verified against the deployed `hooks/pr-evidence-gate.py` (259 lines): line 47 is the only command matcher, `\bgh\s+(?:pr|issue)\s+(?:edit|create|comment)\b`, so `gh api` body writes are still invisible; and the file implements essentially one check (verdict-needs-artifact), **not** the ~9 classes named across items 11–18 (`ci-restatement`, `bare-identifier`, `dump-resolver`, `link-only-exhibit`, `data-only-exhibit`, `step-waiver`, `truncated-identifier`, `mutable-ref`, `inflated-verdict`). Treat every "Emit-time trigger: `pr-evidence-gate.py` class …" line in this document as **specified, not implemented**, until re-verified in the code — a doc asserting a class the code lacks retires the vigilance it claims to replace, which is the failure this very item warns about. Consequence observed the same day: 14 unlinked `path:line` references shipped across 12 review comments via `gh api`, with the gate both classless for that shape and unwired in `settings.json`. Detection gaps: the line-anchor half of pinning and the paraphrase-vs-quotation judgment stay procedural — the gate sees ellipses and branch refs, not editorial fidelity. 17. **Evidence is captured in its environment — data alone is insufficient even when correct** — item 16 makes the data trustworthy as *transcription* (verbatim, greppable, pinned); this item polices what transcription can never carry: **liveness provenance**. A quoted `EVIDENCE trace_id=…` line, a re-hosted gist, a hand-assembled id table can all be correct and still show nothing about *where they came from* — extracted data is indistinguishable from data typed by hand, so it cannot make it immediately apparent that the evidence was captured **live** from a **functioning** system. The exhibit for a system-of-record-observable claim therefore includes an **in-environment capture**: a screenshot/recording of the resolving system's own UI (the Sentry Discover/trace view with the query, project/environment selectors, absolute time window, and result rows all in-frame) — the environmental chrome is not decoration, it *is* the provenance: it shows the query really ran, in the real dashboard, over the real window, and returned these rows. Correctness was never the failing dimension (2026-07-21: "just the data is insufficient even if correct — it needs to be immediately apparent that evidence was captured live and is functional"). Relation to prior items: item 15's link+visual conjunction fired only when a `sentry.io` link was present, and item 13's `NATIVE_MEDIUM` blessed an inline fenced excerpt as a terminal medium — so a no-link, quoted-data exhibit (the fidelity-remediated shape: full ids, verbatim excerpts, pinned line anchors, zero environment captures) passed the whole regime while carrying zero liveness provenance. The joint rule after this item: a telemetry-observable positive verdict always carries the in-environment visual (plus the live permalink per item 15); quoted excerpts, gists, and data files are appendix beside it, never the exhibit. Emit-time trigger: `pr-evidence-gate.py` class `data-only-exhibit` (non-negated verdict + telemetry-observation vocabulary + no image/recording embed + no sentry link — with a sentry link, `link-only-exhibit` already fires), with the re-hosted-gist ALLOW case flipped (fifth occurrence of "the ALLOW case was the next costume's spec") and the #43929 quoted-excerpt shape as a regression case. Detection gaps: vocabulary-scoped (telemetry-observation terms, not bare code tokens like `trace.test.ts`), so a claim phrased entirely without them evades mechanically; and the gate cannot see whether an embedded image actually shows the environment's chrome — screenshot content stays procedural (item 2's "eyeball it" applies: the capture must show the *resolving UI*, not a cropped data region indistinguishable from a spreadsheet). diff --git a/domains/pr-workflow/skills/pr-validate/references/lane-assertions.md b/domains/pr-workflow/skills/evidence/references/lane-assertions.md similarity index 100% rename from domains/pr-workflow/skills/pr-validate/references/lane-assertions.md rename to domains/pr-workflow/skills/evidence/references/lane-assertions.md diff --git a/domains/pr-workflow/skills/pr-validate/references/worked-examples.md b/domains/pr-workflow/skills/evidence/references/worked-examples.md similarity index 100% rename from domains/pr-workflow/skills/pr-validate/references/worked-examples.md rename to domains/pr-workflow/skills/evidence/references/worked-examples.md diff --git a/domains/pr-workflow/skills/pr-validate/skill.md b/domains/pr-workflow/skills/evidence/skill.md similarity index 84% rename from domains/pr-workflow/skills/pr-validate/skill.md rename to domains/pr-workflow/skills/evidence/skill.md index 351667db..53c3c87e 100644 --- a/domains/pr-workflow/skills/pr-validate/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -1,9 +1,9 @@ --- -name: pr-validate -description: Validate a MetaMask PR with objective evidence — primarily the Autonomous Engineering Platform (AEP) harness (visual_validation for visible UI behavior, perf_validation for non-visible perf behavior), backed by complementary evidence (Sentry query links, screenshots, screen recordings→GIF, DevTools/CDP output, bundle-size, web-vitals, test/CI results). Drives the AEP local stack end to end: preflight (postgres + temporal + worker + control-plane) → submit POST /v1/tasks → poll GET /v1/runs/:id → fetch artifacts → assemble an evidence bundle → publish to the PR body (re-hosting images to a public repo, scrubbing local paths). Match evidence to the PR's specific falsifiable claim, not a fixed checklist. Triggers on /pr-validate, /pr-validate visual, /pr-validate perf, /pr-validate preflight, /pr-validate status, /pr-validate evidence, /pr-validate plan, or when the user mentions validating/proving a PR, AEP / visual validation / perf validation, capturing evidence for a PR, before/after screenshots, a screen recording or GIF for a PR, attaching Sentry links or DevTools output as proof, or publishing an evidence bundle to a PR body. +name: evidence +description: Produce reviewer-grade evidence that a claim is true — or that it is not. Matches the evidence to the specific falsifiable claim rather than running a fixed checklist, across a catalog of 41 lanes: before/after screenshots, falsifying regression tests, render and selector proofs, bundle and LavaMoat diffs, Sentry and Tempo links, state-migration and vault checks, plus the Autonomous Engineering Platform (AEP) harness for autonomous visual and perf capture. Assembles an evidence bundle and publishes it, images re-hosted and local paths scrubbed. Runs three ways: on a PR whose claim someone else made, in the inner loop against uncommitted changes before a reviewer sees them, and on a symptom with no claim yet, where the hypothesis to kill is your own. Triggers on the evidence command and its subcommands (visual, perf, preflight, status, plan, lane, compare) — installed as mms-evidence — or when the user mentions validating or proving a PR, capturing evidence, before/after screenshots, a screen recording for a PR, attaching Sentry or DevTools output as proof, AEP or visual/perf validation, or publishing an evidence bundle. --- -# /pr-validate +# /evidence Prove a PR does what it claims with **objective, reviewer-grade evidence**. The primary engine is the **Autonomous Engineering Platform (AEP)** harness run locally — `visual_validation` for visible UI behavior, `perf_validation` for non-visible perf behavior — augmented by whatever complementary evidence the claim demands (Sentry query links, screenshots, screen recordings, DevTools/CDP output, bundle/web-vitals/test results). @@ -93,15 +93,15 @@ Not for code-correctness review (use `/review`, `/code-review`) or span-quota re | Invocation | Behavior | |---|---| -| `/pr-validate <pr>` | **Flagship.** Read the PR → state the claim → pick lanes → [preflight](references/aep-local-run.md) → run AEP lane(s) + gather complementary evidence → assemble bundle → **propose** the PR-body section and confirm before publishing. | -| `/pr-validate plan <pr>` | Dry run: read the PR, state the claim, recommend lanes + targeting hints. No stack, no run. Cheap first step when unsure. | -| `/pr-validate visual <pr>` | AEP `visual_validation` only. | -| `/pr-validate perf <pr>` | AEP `perf_validation` only (local/uncommitted graph — see [caveat](#perf_validation-caveat)). | -| `/pr-validate preflight` | Health-check the local stack; bring up what's down. No run. | -| `/pr-validate status <run-id>` | Poll `GET /v1/runs/:id`; print stage timeline + `evidenceBundle.artifactRefs`. | -| `/pr-validate evidence <pr> [--run <id>]` | Assemble + publish a bundle from an existing run and/or complementary sources (Sentry/screens/devtools). No new AEP run. | -| `/pr-validate lane <id> <pr>` | Run a single [catalog](references/evidence-catalog.md) lane by id (e.g. `lane F1`, `lane C3`, `lane D3`) — for the non-AEP lanes where you know the claim type. | -| `/pr-validate compare <pr>` | Paired A/B for a perf or refactor claim. Two arm kinds — pick by what the claim varies: **`ref`** (default) builds base + head, captures the lane on both, diffs; avoids the stale-baseline trap (catalog C5). **`substitution`** holds a **fixed head** and varies one *artifact* instead of the ref — replace the PR's hand-written type/schema/constant/policy with the authoritative equivalent and diff a checker's output (catalog D6); no build, no rebase, no merge boundary. **Per-arm checks first, one per kind:** for `ref`, verify the mechanism under test is actually active in each arm (chunk split present, span emitted, flag evaluated) — a null arm without delivered treatment is a no-op, not a control (2026-07-22, #42795 bisect lesson). For `substitution`, verify the unmodified arm is **silent** and that each diagnostic fires for the reason claimed — a noisy Arm A destroys attribution, and a diagnostic tripping one property early scores as a confirmation it isn't (trustworthiness gate item 19; 2026-07-30, #44397). | +| `/evidence <pr>` | **Flagship.** Read the PR → state the claim → pick lanes → [preflight](references/aep-local-run.md) → run AEP lane(s) + gather complementary evidence → assemble bundle → **propose** the PR-body section and confirm before publishing. | +| `/evidence plan <pr>` | Dry run: read the PR, state the claim, recommend lanes + targeting hints. No stack, no run. Cheap first step when unsure. | +| `/evidence visual <pr>` | AEP `visual_validation` only. | +| `/evidence perf <pr>` | AEP `perf_validation` only (local/uncommitted graph — see [caveat](#perf_validation-caveat)). | +| `/evidence preflight` | Health-check the local stack; bring up what's down. No run. | +| `/evidence status <run-id>` | Poll `GET /v1/runs/:id`; print stage timeline + `evidenceBundle.artifactRefs`. | +| `/evidence evidence <pr> [--run <id>]` | Assemble + publish a bundle from an existing run and/or complementary sources (Sentry/screens/devtools). No new AEP run. | +| `/evidence lane <id> <pr>` | Run a single [catalog](references/evidence-catalog.md) lane by id (e.g. `lane F1`, `lane C3`, `lane D3`) — for the non-AEP lanes where you know the claim type. | +| `/evidence compare <pr>` | Paired A/B for a perf or refactor claim. Two arm kinds — pick by what the claim varies: **`ref`** (default) builds base + head, captures the lane on both, diffs; avoids the stale-baseline trap (catalog C5). **`substitution`** holds a **fixed head** and varies one *artifact* instead of the ref — replace the PR's hand-written type/schema/constant/policy with the authoritative equivalent and diff a checker's output (catalog D6); no build, no rebase, no merge boundary. **Per-arm checks first, one per kind:** for `ref`, verify the mechanism under test is actually active in each arm (chunk split present, span emitted, flag evaluated) — a null arm without delivered treatment is a no-op, not a control (2026-07-22, #42795 bisect lesson). For `substitution`, verify the unmodified arm is **silent** and that each diagnostic fires for the reason claimed — a noisy Arm A destroys attribution, and a diagnostic tripping one property early scores as a confirmation it isn't (trustworthiness gate item 19; 2026-07-30, #44397). | `<pr>` is a number or URL on `MetaMask/metamask-extension` unless another repo is given. Every variant runs Step 1 (extract the Claim Card) first — the claim decides the lane, even when you named one. @@ -210,23 +210,23 @@ PR claims privacy mode now hides the Perps balance (the demo bug #42683): End-to-end examples for **non-visual** claims (perf, migration, flag-gated, refactor/no-op): **[references/worked-examples.md](references/worked-examples.md).** -## Positioning: AEP vs recipes vs pr-validate +## Positioning: AEP vs recipes vs evidence Three adjacent things; keep the boundary clear so they compose instead of collide: - **AEP** — governed *fleet orchestration*: sandboxes, Temporal, autonomous runs at scale. The heavy engine. - **ADR-0058 recipes** ([decisions#173](https://github.com/MetaMask/decisions/pull/173)) — a *dev-machine inner-loop* proof artifact: a declarative per-PR recipe run against the live app over CDP, emitting `summary.json`/`trace.json`/manifest. -- **pr-validate** (this skill) — the *claim→evidence methodology + taxonomy* both draw on. The Claim Card is the bridge from a PR's claim to the right proof target; the [evidence catalog](references/evidence-catalog.md) is the lane vocabulary; [lane-assertions.md](references/lane-assertions.md) maps each lane to a recipe assertion (and flags the out-of-band, non-UI lanes — the gap MajorLift's #173 review raised). +- **evidence** (this skill) — the *claim→evidence methodology + taxonomy* both draw on. The Claim Card is the bridge from a PR's claim to the right proof target; the [evidence catalog](references/evidence-catalog.md) is the lane vocabulary; [lane-assertions.md](references/lane-assertions.md) maps each lane to a recipe assertion (and flags the out-of-band, non-UI lanes — the gap MajorLift's #173 review raised). -pr-validate is the one a human drives; it can dispatch an AEP run or author a recipe as its capture step. +evidence is the one a human drives; it can dispatch an AEP run or author a recipe as its capture step. ## Workflow integration -Where pr-validate sits in the PR lifecycle (see the public `pr-workflow` siblings): +Where evidence sits in the PR lifecycle (see the public `pr-workflow` siblings): - **After `create-pr`, before `pr-review-queue`:** validate the claim, attach the bundle, *then* request review — reviewers get the before/after up front. - **On force-push / requested-change:** re-run the affected lane(s); re-validation keeps a stale evidence section honest. -- **`/triage` push items:** a `push`-state PR isn't done until its claim is proven; pr-validate produces the evidence that lets it move. +- **`/triage` push items:** a `push`-state PR isn't done until its claim is proven; evidence produces the evidence that lets it move. - **Not a CI gate** (same scope line as ADR-0058) — it's the author's inner loop, complementing unit/e2e, not replacing them. ## Boundaries diff --git a/domains/testing/skills/falsifying-test/skill.md b/domains/testing/skills/falsifying-test/skill.md index 922991f5..e0a2f367 100644 --- a/domains/testing/skills/falsifying-test/skill.md +++ b/domains/testing/skills/falsifying-test/skill.md @@ -1,6 +1,6 @@ --- name: falsifying-test -description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /falsifying-test, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by pr-validate as the engine behind its falsifying regression test evidence category. +description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /falsifying-test, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by evidence as the engine behind its falsifying regression test evidence category. maturity: experimental --- @@ -80,7 +80,7 @@ Falsifying test — <test name> (Fixes #N) ## Related -- `pr-validate` — packages this skill's output as its B3 evidence category; B7 (deterministic +- `evidence` — packages this skill's output as its B3 evidence category; B7 (deterministic interleaving) is the sibling for concurrency and temporal-ordering bugs. - `react-render-proof` — the same before/after discipline applied to a measured quantity rather than a boolean. From 6e15178ae445de7616ab29beda77bfa6d8b66ea4 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 08:30:33 -0400 Subject: [PATCH 043/135] Shorten and normalise skill names `browser-extension-profiling` drops `browser-`, which distinguishes nothing: an extension is a browser extension, and the `extension-` half is what separates it from the mobile work this domain also covers. `anti-pattern` loses its hyphen in identifiers, matching what `main` already ships in `review-antipatterns.md` and `mm-redux-antipatterns.md`. Both skills and both knowledge files move together, since a skill and its knowledge sharing a stem is what makes the by-name citation convention resolvable. Prose inside the two renamed skills is normalised with them so each file agrees with its own name; hyphenated prose elsewhere is left alone as pre-existing and outside this change. --- ...ect-anti-patterns.md => effect-antipatterns.md} | 4 ++-- ...r-anti-patterns.md => selector-antipatterns.md} | 2 +- .../repos/metamask-extension.md | 2 +- .../repos/metamask-mobile.md | 2 +- .../skill.md | 12 ++++++------ .../skill.md | 2 +- .../references/mm-hook-dependency-arrays.md | 2 +- .../performance/references/mm-selector-cascade.md | 2 +- .../references/mm-selector-memoization.md | 2 +- .../references/mm-state-normalization.md | 2 +- .../references/mm-useeffect-antipatterns.md | 2 +- .../repos/metamask-extension.md | 2 +- .../repos/metamask-mobile.md | 2 +- .../skill.md | 14 +++++++------- domains/testing/skills/benchmark-design/skill.md | 2 +- 15 files changed, 27 insertions(+), 27 deletions(-) rename domains/performance/knowledge/{effect-anti-patterns.md => effect-antipatterns.md} (97%) rename domains/performance/knowledge/{selector-anti-patterns.md => selector-antipatterns.md} (99%) rename domains/performance/skills/{effect-anti-pattern-review => effect-antipattern-review}/repos/metamask-extension.md (96%) rename domains/performance/skills/{effect-anti-pattern-review => effect-antipattern-review}/repos/metamask-mobile.md (96%) rename domains/performance/skills/{effect-anti-pattern-review => effect-antipattern-review}/skill.md (87%) rename domains/performance/skills/{browser-extension-profiling => extension-profiling}/skill.md (98%) rename domains/performance/skills/{selector-anti-pattern-review => selector-antipattern-review}/repos/metamask-extension.md (98%) rename domains/performance/skills/{selector-anti-pattern-review => selector-antipattern-review}/repos/metamask-mobile.md (97%) rename domains/performance/skills/{selector-anti-pattern-review => selector-antipattern-review}/skill.md (87%) diff --git a/domains/performance/knowledge/effect-anti-patterns.md b/domains/performance/knowledge/effect-antipatterns.md similarity index 97% rename from domains/performance/knowledge/effect-anti-patterns.md rename to domains/performance/knowledge/effect-antipatterns.md index 3b74afe0..2b24006d 100644 --- a/domains/performance/knowledge/effect-anti-patterns.md +++ b/domains/performance/knowledge/effect-antipatterns.md @@ -1,5 +1,5 @@ --- -name: effect-anti-patterns +name: effect-antipatterns domain: performance description: The React `useEffect` patterns that cause unnecessary renders, memory leaks, or race conditions — the canonical, platform-agnostic taxonomy that per-repo effect references instantiate --- @@ -145,5 +145,5 @@ Pick one and apply it consistently. ## Related - `render-cascade` — how effect-driven re-renders propagate through the component graph. -- `selector-anti-patterns` — the store-side counterpart; an unstable selector result is a +- `selector-antipatterns` — the store-side counterpart; an unstable selector result is a common source of the unstable dependency in pattern 1. diff --git a/domains/performance/knowledge/selector-anti-patterns.md b/domains/performance/knowledge/selector-antipatterns.md similarity index 99% rename from domains/performance/knowledge/selector-anti-patterns.md rename to domains/performance/knowledge/selector-antipatterns.md index 8bf32e9f..df65609f 100644 --- a/domains/performance/knowledge/selector-anti-patterns.md +++ b/domains/performance/knowledge/selector-antipatterns.md @@ -1,5 +1,5 @@ --- -name: selector-anti-patterns +name: selector-antipatterns domain: performance description: The Redux selector patterns that break memoization and cause render cascades — the canonical, platform-agnostic taxonomy that per-repo selector references instantiate --- diff --git a/domains/performance/skills/effect-anti-pattern-review/repos/metamask-extension.md b/domains/performance/skills/effect-antipattern-review/repos/metamask-extension.md similarity index 96% rename from domains/performance/skills/effect-anti-pattern-review/repos/metamask-extension.md rename to domains/performance/skills/effect-antipattern-review/repos/metamask-extension.md index 532b74c6..af9b4298 100644 --- a/domains/performance/skills/effect-anti-pattern-review/repos/metamask-extension.md +++ b/domains/performance/skills/effect-antipattern-review/repos/metamask-extension.md @@ -1,6 +1,6 @@ --- repo: metamask-extension -parent: effect-anti-pattern-review +parent: effect-antipattern-review --- ## Paths diff --git a/domains/performance/skills/effect-anti-pattern-review/repos/metamask-mobile.md b/domains/performance/skills/effect-antipattern-review/repos/metamask-mobile.md similarity index 96% rename from domains/performance/skills/effect-anti-pattern-review/repos/metamask-mobile.md rename to domains/performance/skills/effect-antipattern-review/repos/metamask-mobile.md index dde98072..6f19aaed 100644 --- a/domains/performance/skills/effect-anti-pattern-review/repos/metamask-mobile.md +++ b/domains/performance/skills/effect-antipattern-review/repos/metamask-mobile.md @@ -1,6 +1,6 @@ --- repo: metamask-mobile -parent: effect-anti-pattern-review +parent: effect-antipattern-review --- ## Paths diff --git a/domains/performance/skills/effect-anti-pattern-review/skill.md b/domains/performance/skills/effect-antipattern-review/skill.md similarity index 87% rename from domains/performance/skills/effect-anti-pattern-review/skill.md rename to domains/performance/skills/effect-antipattern-review/skill.md index 1ecc8d86..5f2ab763 100644 --- a/domains/performance/skills/effect-anti-pattern-review/skill.md +++ b/domains/performance/skills/effect-antipattern-review/skill.md @@ -1,12 +1,12 @@ --- maturity: experimental -name: effect-anti-pattern-review -description: Review PR diffs that add or modify `useEffect` for the systemic React effect anti-patterns +name: effect-antipattern-review +description: Review PR diffs that add or modify `useEffect` for the systemic React effect antipatterns --- # Effect Anti-Pattern Review -**Scope:** Pre-merge review of PRs that add or modify `useEffect` calls. The workflow is a grep-driven checklist against the patterns catalogued in the **`effect-anti-patterns`** knowledge file, which is the single source for their definitions and fixes (installed alongside this skill under `knowledge/`). +**Scope:** Pre-merge review of PRs that add or modify `useEffect` calls. The workflow is a grep-driven checklist against the patterns catalogued in the **`effect-antipatterns`** knowledge file, which is the single source for their definitions and fixes (installed alongside this skill under `knowledge/`). Applies to both `metamask-extension` and `metamask-mobile`. See overlays for repo-specific paths. @@ -18,7 +18,7 @@ Applies to both `metamask-extension` and `metamask-mobile`. See overlays for rep ## Do Not Use When -- Reviewing selector or render-cascade issues (use [`selector-anti-pattern-review`](../selector-anti-pattern-review/skill.md)) +- Reviewing selector or render-cascade issues (use [`selector-antipattern-review`](../selector-antipattern-review/skill.md)) - Reviewing non-React code (background scripts, workers, test utilities) - Reviewing an effect that is intentionally one-shot with no async work or timers (check patterns below anyway, but most do not apply) @@ -26,14 +26,14 @@ Applies to both `metamask-extension` and `metamask-mobile`. See overlays for rep 1. **List changed files with `useEffect`.** `git diff --name-only origin/main...HEAD | xargs grep -l 'useEffect'` 2. **Run the [grep checklist](#grep-checklist)** against the changed files. -3. **For each hit, map to a pattern** in `effect-anti-patterns` and apply the fix from the knowledge file. +3. **For each hit, map to a pattern** in `effect-antipatterns` and apply the fix from the knowledge file. 4. **Block on unstable dependency identity.** `JSON.stringify` in a dependency array is always broken. Do not merge. 5. **Block on a timer without cleanup.** Any `setInterval` / `setTimeout` without a matching `clearInterval` / `clearTimeout` in the cleanup function is blocking. 6. **Require cancellation for async effects.** Any `fetch` / network call inside `useEffect` must use `AbortController`. ## Grep Checklist -| Pattern (`effect-anti-patterns` §) | Detection | +| Pattern (`effect-antipatterns` §) | Detection | |---|---| | §1 Unstable dependency identity | `grep -rnE 'useEffect.*\[.*JSON\.stringify' <source-dir>`, plus inline `{`/`[` literals in the dep position | | §2 Wrong dependencies | Hand review — empty deps that read state (stale closure), or deps that read nothing | diff --git a/domains/performance/skills/browser-extension-profiling/skill.md b/domains/performance/skills/extension-profiling/skill.md similarity index 98% rename from domains/performance/skills/browser-extension-profiling/skill.md rename to domains/performance/skills/extension-profiling/skill.md index d956dea8..c2a80272 100644 --- a/domains/performance/skills/browser-extension-profiling/skill.md +++ b/domains/performance/skills/extension-profiling/skill.md @@ -1,6 +1,6 @@ --- maturity: experimental -name: browser-extension-profiling +name: extension-profiling description: Compare browser extension performance between branches using WDYR, React DevTools Profiler, and E2E benchmarks with statistical rigor. --- diff --git a/domains/performance/skills/performance/references/mm-hook-dependency-arrays.md b/domains/performance/skills/performance/references/mm-hook-dependency-arrays.md index 5ff27dc0..117cc1dd 100644 --- a/domains/performance/skills/performance/references/mm-hook-dependency-arrays.md +++ b/domains/performance/skills/performance/references/mm-hook-dependency-arrays.md @@ -8,7 +8,7 @@ tags: useEffect, useMemo, useCallback, dependencies, JSON.stringify Dependency arrays decide when `useEffect`/`useMemo`/`useCallback` re-run. The most common MetaMask problem is **`JSON.stringify` inside a dependency array** — it runs a synchronous serialization on every render just to compute the dependency key, which is both expensive and a sign the upstream reference is unstable. -> **Scope.** This file is the *dependency* half of effect performance, instantiated for this codebase. The platform-agnostic taxonomy — unstable dependency identity, wrong dependencies, derived state via effect, cascading effect chains, missing cleanup, uncancelled async — lives in the **`effect-anti-patterns`** knowledge file, installed alongside this skill under `knowledge/`. Read that for the general shape; read this for the verified instances and the repo's lint gaps. +> **Scope.** This file is the *dependency* half of effect performance, instantiated for this codebase. The platform-agnostic taxonomy — unstable dependency identity, wrong dependencies, derived state via effect, cascading effect chains, missing cleanup, uncancelled async — lives in the **`effect-antipatterns`** knowledge file, installed alongside this skill under `knowledge/`. Read that for the general shape; read this for the verified instances and the repo's lint gaps. ## Pattern — `JSON.stringify` in deps diff --git a/domains/performance/skills/performance/references/mm-selector-cascade.md b/domains/performance/skills/performance/references/mm-selector-cascade.md index 387e9cbe..5892e599 100644 --- a/domains/performance/skills/performance/references/mm-selector-cascade.md +++ b/domains/performance/skills/performance/references/mm-selector-cascade.md @@ -8,7 +8,7 @@ tags: reselect, cascade, dependency-graph, isEqual, structural-sharing, react-co > **Scope.** What a broken root selector does to the component graph is defined generically > in the **`render-cascade`** knowledge file, and the selector patterns that cause it in -> **`selector-anti-patterns`** — both installed alongside this skill under `knowledge/`. +> **`selector-antipatterns`** — both installed alongside this skill under `knowledge/`. > This file is the MetaMask Mobile instance: the real dependency graph, its blast radius, > and the repair order. diff --git a/domains/performance/skills/performance/references/mm-selector-memoization.md b/domains/performance/skills/performance/references/mm-selector-memoization.md index c6731a62..414c48b0 100644 --- a/domains/performance/skills/performance/references/mm-selector-memoization.md +++ b/domains/performance/skills/performance/references/mm-selector-memoization.md @@ -15,7 +15,7 @@ Broken or absent memoization in widely-used selectors is the single highest-impa ## The patterns, and what they look like here -The pattern taxonomy itself lives in the **`selector-anti-patterns`** knowledge file, +The pattern taxonomy itself lives in the **`selector-antipatterns`** knowledge file, installed alongside this skill under `knowledge/`. It is the single source — read it for the full definition, the worked before/after of each, and the selector-creator decision tree. This section maps each pattern onto *this* codebase. diff --git a/domains/performance/skills/performance/references/mm-state-normalization.md b/domains/performance/skills/performance/references/mm-state-normalization.md index 6a8b0bc1..0bcaa57c 100644 --- a/domains/performance/skills/performance/references/mm-state-normalization.md +++ b/domains/performance/skills/performance/references/mm-state-normalization.md @@ -6,7 +6,7 @@ tags: redux, normalization, selectors, O(1)-lookups, cache-thrashing, useSelecto # Skill: State Normalization & Selector Shape -> **Scope.** The generic form of the O(n)-lookup problem is `selector-anti-patterns` §7, +> **Scope.** The generic form of the O(n)-lookup problem is `selector-antipatterns` §7, > in the knowledge file installed alongside this skill under `knowledge/`. This file is the > MetaMask Mobile instance, plus the parameterized-selector cache-thrashing and > view-selector consolidation work that is specific to this store's shape. diff --git a/domains/performance/skills/performance/references/mm-useeffect-antipatterns.md b/domains/performance/skills/performance/references/mm-useeffect-antipatterns.md index 7b24d982..ad54d055 100644 --- a/domains/performance/skills/performance/references/mm-useeffect-antipatterns.md +++ b/domains/performance/skills/performance/references/mm-useeffect-antipatterns.md @@ -8,7 +8,7 @@ tags: useEffect, setState, cleanup, AbortController, unmount, memory-leaks > **Scope.** The platform-agnostic taxonomy — unstable dependency identity, wrong > dependencies, derived state via effect, cascading effect chains, missing timer cleanup, -> uncancelled async — is the single source in the **`effect-anti-patterns`** knowledge file, +> uncancelled async — is the single source in the **`effect-antipatterns`** knowledge file, > installed alongside this skill under `knowledge/`. This file is the MetaMask Mobile > instance of its lifecycle half: the verified instances, the repo's own idioms, and the > fix recipes. diff --git a/domains/performance/skills/selector-anti-pattern-review/repos/metamask-extension.md b/domains/performance/skills/selector-antipattern-review/repos/metamask-extension.md similarity index 98% rename from domains/performance/skills/selector-anti-pattern-review/repos/metamask-extension.md rename to domains/performance/skills/selector-antipattern-review/repos/metamask-extension.md index b3ca4992..81dceb6b 100644 --- a/domains/performance/skills/selector-anti-pattern-review/repos/metamask-extension.md +++ b/domains/performance/skills/selector-antipattern-review/repos/metamask-extension.md @@ -1,6 +1,6 @@ --- repo: metamask-extension -parent: selector-anti-pattern-review +parent: selector-antipattern-review --- ## Paths diff --git a/domains/performance/skills/selector-anti-pattern-review/repos/metamask-mobile.md b/domains/performance/skills/selector-antipattern-review/repos/metamask-mobile.md similarity index 97% rename from domains/performance/skills/selector-anti-pattern-review/repos/metamask-mobile.md rename to domains/performance/skills/selector-antipattern-review/repos/metamask-mobile.md index cb28ddab..b77ab8c6 100644 --- a/domains/performance/skills/selector-anti-pattern-review/repos/metamask-mobile.md +++ b/domains/performance/skills/selector-antipattern-review/repos/metamask-mobile.md @@ -1,6 +1,6 @@ --- repo: metamask-mobile -parent: selector-anti-pattern-review +parent: selector-antipattern-review --- ## Paths diff --git a/domains/performance/skills/selector-anti-pattern-review/skill.md b/domains/performance/skills/selector-antipattern-review/skill.md similarity index 87% rename from domains/performance/skills/selector-anti-pattern-review/skill.md rename to domains/performance/skills/selector-antipattern-review/skill.md index 0b541d08..a98d65f5 100644 --- a/domains/performance/skills/selector-anti-pattern-review/skill.md +++ b/domains/performance/skills/selector-antipattern-review/skill.md @@ -1,12 +1,12 @@ --- maturity: experimental -name: selector-anti-pattern-review -description: Review and diagnose Redux selector anti-patterns that cause render cascades, pre-merge and post-merge +name: selector-antipattern-review +description: Review and diagnose Redux selector antipatterns that cause render cascades, pre-merge and post-merge --- # Selector Anti-Pattern Review -**Scope:** Redux selector anti-patterns are the dominant cause of React render cascades in the MetaMask UI. This skill covers both review phases: pre-merge PR review (grep-driven checklist) and post-merge diagnosis (WDYR-driven workflow). Both modes resolve to the same root cause and the same fix set, catalogued in the **`selector-anti-patterns`** and **`render-cascade`** knowledge files — the single source for their definitions (installed alongside this skill under `knowledge/`). +**Scope:** Redux selector antipatterns are the dominant cause of React render cascades in the MetaMask UI. This skill covers both review phases: pre-merge PR review (grep-driven checklist) and post-merge diagnosis (WDYR-driven workflow). Both modes resolve to the same root cause and the same fix set, catalogued in the **`selector-antipatterns`** and **`render-cascade`** knowledge files — the single source for their definitions (installed alongside this skill under `knowledge/`). Both `metamask-extension` and `metamask-mobile` share the same React + Redux architecture; this skill applies to both (see overlays for repo-specific paths). @@ -18,7 +18,7 @@ Both `metamask-extension` and `metamask-mobile` share the same React + Redux arc ## Do Not Use When -- Non-selector performance concerns (effects → use `effect-anti-pattern-review`, context providers, virtualization) +- Non-selector performance concerns (effects → use `effect-antipattern-review`, context providers, virtualization) - Network-bound slowness (use the Network panel, not WDYR) - Startup or initial-mount perf (use startup profiling) - Non-React trees (worker messaging, background script perf) @@ -27,8 +27,8 @@ Both `metamask-extension` and `metamask-mobile` share the same React + Redux arc 1. **List changed selector/consumer files.** `git diff --name-only origin/main...HEAD | grep -E '(selectors|useSelector)'` 2. **Run the [grep checklist](#grep-checklist)** against the changed files. -3. **Match each hit to a pattern** in `selector-anti-patterns` or to one of the [team-specific workarounds](#team-specific-workarounds) below. -4. **Block on Jest warning.** If the PR's test run surfaces `"result function returned its own inputs"`, the PR introduces an identity/passthrough result (`selector-anti-patterns` §2). Do not merge. +3. **Match each hit to a pattern** in `selector-antipatterns` or to one of the [team-specific workarounds](#team-specific-workarounds) below. +4. **Block on Jest warning.** If the PR's test run surfaces `"result function returned its own inputs"`, the PR introduces an identity/passthrough result (`selector-antipatterns` §2). Do not merge. 5. **Require a fix, not a justification.** None of the five patterns have a valid use case. See [Pitfalls](#common-pitfalls) for the narrow `createDeepEqualSelector` exception. ## Mode B: Post-Merge Diagnosis (WDYR-driven) @@ -46,7 +46,7 @@ Both `metamask-extension` and `metamask-mobile` share the same React + Redux arc ## Grep Checklist -| Pattern (`selector-anti-patterns` §) | Detection | +| Pattern (`selector-antipatterns` §) | Detection | |---|---| | §1 Unmemoized selector | `grep -rE 'export function get' <selectors-dir>/` | | §2 Identity / passthrough result | Jest warning `result function returned its own inputs` | diff --git a/domains/testing/skills/benchmark-design/skill.md b/domains/testing/skills/benchmark-design/skill.md index 578691df..641ef6c5 100644 --- a/domains/testing/skills/benchmark-design/skill.md +++ b/domains/testing/skills/benchmark-design/skill.md @@ -16,7 +16,7 @@ description: Design, run, and analyze E2E performance benchmarks — session hyg ## Do Not Use When - Adding unit, integration, or correctness E2E tests -- Profiling a single user-reported slowdown (use `selector-anti-pattern-review`) +- Profiling a single user-reported slowdown (use `selector-antipattern-review`) - Writing micro-benchmarks outside the E2E harness ## Workflow From 615c3abb7247e08f1ba183b876a3dda823c5482c Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 08:31:14 -0400 Subject: [PATCH 044/135] Rename `memory-leak-hunt` to `memory-leak` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `hunt` disambiguated nothing. No sibling skill targets memory leaks, and none would: this one already covers both halves — the static retention read from the diff and the heap investigation when the read cannot settle it — so there is no detection/diagnosis split for the suffix to mark. Installed as `mms-memory-leak`. --- .../references/heap-investigation.md | 0 .../scripts/heap-over-cycles.example.ts | 0 .../scripts/retention-scan.py | 0 .../skills/{memory-leak-hunt => memory-leak}/skill.md | 6 +++--- 4 files changed, 3 insertions(+), 3 deletions(-) rename domains/stability/skills/{memory-leak-hunt => memory-leak}/references/heap-investigation.md (100%) rename domains/stability/skills/{memory-leak-hunt => memory-leak}/scripts/heap-over-cycles.example.ts (100%) rename domains/stability/skills/{memory-leak-hunt => memory-leak}/scripts/retention-scan.py (100%) rename domains/stability/skills/{memory-leak-hunt => memory-leak}/skill.md (96%) diff --git a/domains/stability/skills/memory-leak-hunt/references/heap-investigation.md b/domains/stability/skills/memory-leak/references/heap-investigation.md similarity index 100% rename from domains/stability/skills/memory-leak-hunt/references/heap-investigation.md rename to domains/stability/skills/memory-leak/references/heap-investigation.md diff --git a/domains/stability/skills/memory-leak-hunt/scripts/heap-over-cycles.example.ts b/domains/stability/skills/memory-leak/scripts/heap-over-cycles.example.ts similarity index 100% rename from domains/stability/skills/memory-leak-hunt/scripts/heap-over-cycles.example.ts rename to domains/stability/skills/memory-leak/scripts/heap-over-cycles.example.ts diff --git a/domains/stability/skills/memory-leak-hunt/scripts/retention-scan.py b/domains/stability/skills/memory-leak/scripts/retention-scan.py similarity index 100% rename from domains/stability/skills/memory-leak-hunt/scripts/retention-scan.py rename to domains/stability/skills/memory-leak/scripts/retention-scan.py diff --git a/domains/stability/skills/memory-leak-hunt/skill.md b/domains/stability/skills/memory-leak/skill.md similarity index 96% rename from domains/stability/skills/memory-leak-hunt/skill.md rename to domains/stability/skills/memory-leak/skill.md index bdd9c312..fc18f6ac 100644 --- a/domains/stability/skills/memory-leak-hunt/skill.md +++ b/domains/stability/skills/memory-leak/skill.md @@ -1,10 +1,10 @@ --- -name: memory-leak-hunt -description: Find and investigate memory leaks / retention issues in JavaScript/TypeScript. Two phases. (1) Static identification from a diff — enumerate the retention primitives the change introduces (event listeners, timers, pending-request registries, subscriptions, module singletons, growing collections), pair every acquire with its release site, and scope findings to what the diff adds versus what pre-exists. (2) Runtime investigation, only for a primitive that cannot be paired statically — DevTools/CDP heap snapshots over N cycles, the retainer graph, detached-node count, and a falsifying lifecycle test. Leads with the cheap static read (the retention review a reviewer already performs) and escalates to a heap snapshot only where the read is inconclusive. Triggers on /memory-leak-hunt, or when asked to find or investigate a memory leak, check listener/subscription/timer cleanup, review a diff for retention, or take and read a heap snapshot. Callable by pr-validate as the engine behind its memory-leak evidence category. +name: memory-leak +description: Find and investigate memory leaks / retention issues in JavaScript/TypeScript. Two phases. (1) Static identification from a diff — enumerate the retention primitives the change introduces (event listeners, timers, pending-request registries, subscriptions, module singletons, growing collections), pair every acquire with its release site, and scope findings to what the diff adds versus what pre-exists. (2) Runtime investigation, only for a primitive that cannot be paired statically — DevTools/CDP heap snapshots over N cycles, the retainer graph, detached-node count, and a falsifying lifecycle test. Leads with the cheap static read (the retention review a reviewer already performs) and escalates to a heap snapshot only where the read is inconclusive. Triggers on /memory-leak, or when asked to find or investigate a memory leak, check listener/subscription/timer cleanup, review a diff for retention, or take and read a heap snapshot. Callable by pr-validate as the engine behind its memory-leak evidence category. maturity: experimental --- -# /memory-leak-hunt +# /memory-leak Find where an object outlives its purpose — and prove it, or prove it doesn't. A memory leak is a **retention path**: something acquires a reference (a listener, a timer, a map From ab926b82bfbe0ca4abe2cb29ea1d981169ef32f7 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 08:32:19 -0400 Subject: [PATCH 045/135] Rename `typescript-compiler-blindspots` to `compiler-blindspots` The skill lives in the `typescript` domain, so the prefix repeated information already carried by the path and by every discovery surface that shows it. Installed as `mms-compiler-blindspots`. --- .../references/false-negatives.md | 0 .../references/metamask-extension.md | 0 .../references/worked-example.md | 0 .../scripts/substitution-ab.sh | 0 .../skill.md | 2 +- 5 files changed, 1 insertion(+), 1 deletion(-) rename domains/typescript/skills/{typescript-compiler-blindspots => compiler-blindspots}/references/false-negatives.md (100%) rename domains/typescript/skills/{typescript-compiler-blindspots => compiler-blindspots}/references/metamask-extension.md (100%) rename domains/typescript/skills/{typescript-compiler-blindspots => compiler-blindspots}/references/worked-example.md (100%) rename domains/typescript/skills/{typescript-compiler-blindspots => compiler-blindspots}/scripts/substitution-ab.sh (100%) rename domains/typescript/skills/{typescript-compiler-blindspots => compiler-blindspots}/skill.md (99%) diff --git a/domains/typescript/skills/typescript-compiler-blindspots/references/false-negatives.md b/domains/typescript/skills/compiler-blindspots/references/false-negatives.md similarity index 100% rename from domains/typescript/skills/typescript-compiler-blindspots/references/false-negatives.md rename to domains/typescript/skills/compiler-blindspots/references/false-negatives.md diff --git a/domains/typescript/skills/typescript-compiler-blindspots/references/metamask-extension.md b/domains/typescript/skills/compiler-blindspots/references/metamask-extension.md similarity index 100% rename from domains/typescript/skills/typescript-compiler-blindspots/references/metamask-extension.md rename to domains/typescript/skills/compiler-blindspots/references/metamask-extension.md diff --git a/domains/typescript/skills/typescript-compiler-blindspots/references/worked-example.md b/domains/typescript/skills/compiler-blindspots/references/worked-example.md similarity index 100% rename from domains/typescript/skills/typescript-compiler-blindspots/references/worked-example.md rename to domains/typescript/skills/compiler-blindspots/references/worked-example.md diff --git a/domains/typescript/skills/typescript-compiler-blindspots/scripts/substitution-ab.sh b/domains/typescript/skills/compiler-blindspots/scripts/substitution-ab.sh similarity index 100% rename from domains/typescript/skills/typescript-compiler-blindspots/scripts/substitution-ab.sh rename to domains/typescript/skills/compiler-blindspots/scripts/substitution-ab.sh diff --git a/domains/typescript/skills/typescript-compiler-blindspots/skill.md b/domains/typescript/skills/compiler-blindspots/skill.md similarity index 99% rename from domains/typescript/skills/typescript-compiler-blindspots/skill.md rename to domains/typescript/skills/compiler-blindspots/skill.md index 90c1bc4f..0cfaa6c3 100644 --- a/domains/typescript/skills/typescript-compiler-blindspots/skill.md +++ b/domains/typescript/skills/compiler-blindspots/skill.md @@ -1,5 +1,5 @@ --- -name: typescript-compiler-blindspots +name: compiler-blindspots description: >- Find the type defects `tsc` is structurally unable to report — a green build is not evidence the types are correct. Covers the two classes: (1) hand-written From 4d330e411d50b88e2846444361a82b026e5b10ee Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 08:33:08 -0400 Subject: [PATCH 046/135] Rename `analytics-instrumentation` to `instrumentation` The skill lives in the `analytics` domain, so the prefix repeated what the path and every discovery surface already show. Installed as `mms-instrumentation`. --- .../repos/metamask-extension.md | 2 +- .../{analytics-instrumentation => instrumentation}/skill.md | 2 +- domains/analytics/skills/sentry-quota/skill.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) rename domains/analytics/skills/{analytics-instrumentation => instrumentation}/repos/metamask-extension.md (98%) rename domains/analytics/skills/{analytics-instrumentation => instrumentation}/skill.md (99%) diff --git a/domains/analytics/skills/analytics-instrumentation/repos/metamask-extension.md b/domains/analytics/skills/instrumentation/repos/metamask-extension.md similarity index 98% rename from domains/analytics/skills/analytics-instrumentation/repos/metamask-extension.md rename to domains/analytics/skills/instrumentation/repos/metamask-extension.md index 5175f96e..d88ddaec 100644 --- a/domains/analytics/skills/analytics-instrumentation/repos/metamask-extension.md +++ b/domains/analytics/skills/instrumentation/repos/metamask-extension.md @@ -1,6 +1,6 @@ --- repo: metamask-extension -parent: analytics-instrumentation +parent: instrumentation --- ## Key Files diff --git a/domains/analytics/skills/analytics-instrumentation/skill.md b/domains/analytics/skills/instrumentation/skill.md similarity index 99% rename from domains/analytics/skills/analytics-instrumentation/skill.md rename to domains/analytics/skills/instrumentation/skill.md index eab3328f..987ef8f3 100644 --- a/domains/analytics/skills/analytics-instrumentation/skill.md +++ b/domains/analytics/skills/instrumentation/skill.md @@ -1,6 +1,6 @@ --- maturity: experimental -name: analytics-instrumentation +name: instrumentation description: Create and update Sentry spans, MetaMetrics events, and Segment events — methodology, policies, common pitfalls --- diff --git a/domains/analytics/skills/sentry-quota/skill.md b/domains/analytics/skills/sentry-quota/skill.md index 6ae2e1b1..8ef9e4a5 100644 --- a/domains/analytics/skills/sentry-quota/skill.md +++ b/domains/analytics/skills/sentry-quota/skill.md @@ -18,7 +18,7 @@ Find and fix custom Sentry span instrumentation that blows the project span budg ## Do Not Use When - Reading the live span counts themselves — that's `sentry-mcp-queries` (Volume Estimation). -- Product-analytics events (Segment / `trackEvent`) — that's `analytics-instrumentation` + `segment-governance`. +- Product-analytics events (Segment / `trackEvent`) — that's `instrumentation` + `segment-governance`. - The span is already behind a per-trace sample gate **and** a kill-switch — already mitigated. ## Breach Triad From 952d54362fc3018409326a9e7e55979129c36172 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 08:39:59 -0400 Subject: [PATCH 047/135] Ship `hooks/`, and check that a description names the installed command MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `hooks/` was absent from the bundle list, so a skill shipping a hook delivered everything except the hook. This is the fourth instance of one root cause — a directory that exists in source and is not in `copy_bundle_dirs` — after `pr-validate`'s hooks, domain `knowledge/`, and `workflows/`. The `BUNDLE_DIRS` drift test covers it now, so a fifth cannot land silently. Separately, the installer prefixes every emitted skill, so a description advertising `/<name>` names a command no operator exposes. The description is the discovery surface, which makes a wrong trigger string a selection failure rather than a typo. The check requires the `mms-` form. Its first version flagged `@metamask/gator-cli` — a scoped package, not a slash command. A negative lookbehind now excludes `@scope/name` and path-like forms, and that case is a test rather than a note. --- .github/scripts/lint-skill-entry.mjs | 16 ++++++++++++++++ test/lint-skill-entry.test.mjs | 23 +++++++++++++++++++++++ tools/install | 2 +- tools/skill-schema.mjs | 2 +- 4 files changed, 41 insertions(+), 2 deletions(-) diff --git a/.github/scripts/lint-skill-entry.mjs b/.github/scripts/lint-skill-entry.mjs index 8de11ba1..c737a0ce 100644 --- a/.github/scripts/lint-skill-entry.mjs +++ b/.github/scripts/lint-skill-entry.mjs @@ -111,6 +111,22 @@ export function lintSkill(skill) { } } + // The installer prefixes every emitted skill, so a description advertising `/<name>` + // names a command no operator exposes. The description IS the discovery surface, so a + // wrong trigger string is a selection failure, not a typo. + // + // The lookbehind keeps a scoped package (`@metamask/gator-cli`) or a path + // (`skills/gator-cli`) from being read as a slash command. + if (raw.description && raw.name) { + const bare = new RegExp(`(?<![\\w@/-])/${raw.name}\\b`, 'u'); + const prefixed = new RegExp(`(?<![\\w@/-])/${INSTALLED_PREFIX}${raw.name}\\b`, 'u'); + if (bare.test(raw.description) && !prefixed.test(raw.description)) { + errors.push( + `\`description\` advertises \`/${raw.name}\` but the installer emits \`${INSTALLED_PREFIX}${raw.name}\`; name the installed form`, + ); + } + } + for (const section of RECOMMENDED_SECTIONS) { // A trailing `\b` let `## When To Use Cases` satisfy `When To Use` — a different // section. Anchoring to end-of-line fixes that but rejects `## Workflows` and diff --git a/test/lint-skill-entry.test.mjs b/test/lint-skill-entry.test.mjs index 744096ce..947021af 100644 --- a/test/lint-skill-entry.test.mjs +++ b/test/lint-skill-entry.test.mjs @@ -268,3 +268,26 @@ describe('changed-files mode', () => { assert.match(output, /over the \d+-char budget/u); }); }); + +describe('description names the installed command', () => { + test('a bare /<name> trigger fails', () => { + const root = makeRoot(); + writeSkill(root, 'testing', 'demo', 'name: demo\ndescription: Triggers on /demo when asked.'); + const { code, output } = lint(root); + assert.equal(code, 1, output); + assert.match(output, /advertises `\/demo` but the installer emits `mms-demo`/u); + }); + + test('the prefixed form passes', () => { + const root = makeRoot(); + writeSkill(root, 'testing', 'demo', 'name: demo\ndescription: Triggers on /mms-demo when asked.'); + assert.equal(lint(root).code, 0); + }); + + test('a scoped package name is not a slash command', () => { + // `@metamask/gator-cli` tripped the first version of this rule. + const root = makeRoot(); + writeSkill(root, 'web3-tools', 'gator-cli', 'name: gator-cli\ndescription: Operate the @metamask/gator-cli package.'); + assert.equal(lint(root).code, 0); + }); +}); diff --git a/tools/install b/tools/install index af56d281..2df02e44 100755 --- a/tools/install +++ b/tools/install @@ -346,7 +346,7 @@ write_user_codex() { copy_bundle_dirs() { local skill_dir="$1" dest_dir="$2" label="$3" local bundle - for bundle in references scripts assets adapters workflows; do + for bundle in references scripts assets adapters workflows hooks; do if [[ -d "$skill_dir/$bundle" ]]; then action "$label/$bundle/" $DRY_RUN && continue diff --git a/tools/skill-schema.mjs b/tools/skill-schema.mjs index 290b4222..2d471844 100644 --- a/tools/skill-schema.mjs +++ b/tools/skill-schema.mjs @@ -14,7 +14,7 @@ export const MATURITY_VALUES = ['experimental', 'stable', 'deprecated']; export const SCOPE_VALUES = ['user', 'project']; // Directories the installer copies alongside skill.md (see tools/install). -export const BUNDLE_DIRS = ['references', 'scripts', 'assets', 'adapters', 'workflows']; +export const BUNDLE_DIRS = ['references', 'scripts', 'assets', 'adapters', 'workflows', 'hooks']; // Directories allowed beside skill.md: the bundle dirs plus the repo-overlay // dir. Anything else is rejected, because the installer does not ship it and any From 14196b2d9fe6500a7934de93db0aaec17d88f0b6 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 09:40:05 -0400 Subject: [PATCH 048/135] Rename the antipattern skills from `-review` to `-scan` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `scan` says what they do. Both walk a diff looking for a known set of shapes and report what they find; `review` implied a judgement they do not make and overlapped with the correctness review these deliberately are not. The suffix still carries its original job of keeping each skill distinct from the knowledge file it cites — `selector-antipatterns.md` and `effect-antipatterns.md` — which the by-name citation resolver needs, since it matches on filename. Installed as `mms-selector-antipattern-scan` and `mms-effect-antipattern-scan`. --- .../repos/metamask-extension.md | 2 +- .../repos/metamask-mobile.md | 2 +- .../skill.md | 4 ++-- .../repos/metamask-extension.md | 2 +- .../repos/metamask-mobile.md | 2 +- .../skill.md | 4 ++-- domains/testing/skills/benchmark-design/skill.md | 2 +- 7 files changed, 9 insertions(+), 9 deletions(-) rename domains/performance/skills/{effect-antipattern-review => effect-antipattern-scan}/repos/metamask-extension.md (96%) rename domains/performance/skills/{effect-antipattern-review => effect-antipattern-scan}/repos/metamask-mobile.md (96%) rename domains/performance/skills/{effect-antipattern-review => effect-antipattern-scan}/skill.md (97%) rename domains/performance/skills/{selector-antipattern-review => selector-antipattern-scan}/repos/metamask-extension.md (98%) rename domains/performance/skills/{selector-antipattern-review => selector-antipattern-scan}/repos/metamask-mobile.md (97%) rename domains/performance/skills/{selector-antipattern-review => selector-antipattern-scan}/skill.md (98%) diff --git a/domains/performance/skills/effect-antipattern-review/repos/metamask-extension.md b/domains/performance/skills/effect-antipattern-scan/repos/metamask-extension.md similarity index 96% rename from domains/performance/skills/effect-antipattern-review/repos/metamask-extension.md rename to domains/performance/skills/effect-antipattern-scan/repos/metamask-extension.md index af9b4298..ee5fb3b2 100644 --- a/domains/performance/skills/effect-antipattern-review/repos/metamask-extension.md +++ b/domains/performance/skills/effect-antipattern-scan/repos/metamask-extension.md @@ -1,6 +1,6 @@ --- repo: metamask-extension -parent: effect-antipattern-review +parent: effect-antipattern-scan --- ## Paths diff --git a/domains/performance/skills/effect-antipattern-review/repos/metamask-mobile.md b/domains/performance/skills/effect-antipattern-scan/repos/metamask-mobile.md similarity index 96% rename from domains/performance/skills/effect-antipattern-review/repos/metamask-mobile.md rename to domains/performance/skills/effect-antipattern-scan/repos/metamask-mobile.md index 6f19aaed..b005c307 100644 --- a/domains/performance/skills/effect-antipattern-review/repos/metamask-mobile.md +++ b/domains/performance/skills/effect-antipattern-scan/repos/metamask-mobile.md @@ -1,6 +1,6 @@ --- repo: metamask-mobile -parent: effect-antipattern-review +parent: effect-antipattern-scan --- ## Paths diff --git a/domains/performance/skills/effect-antipattern-review/skill.md b/domains/performance/skills/effect-antipattern-scan/skill.md similarity index 97% rename from domains/performance/skills/effect-antipattern-review/skill.md rename to domains/performance/skills/effect-antipattern-scan/skill.md index 5f2ab763..f890fcc8 100644 --- a/domains/performance/skills/effect-antipattern-review/skill.md +++ b/domains/performance/skills/effect-antipattern-scan/skill.md @@ -1,6 +1,6 @@ --- maturity: experimental -name: effect-antipattern-review +name: effect-antipattern-scan description: Review PR diffs that add or modify `useEffect` for the systemic React effect antipatterns --- @@ -18,7 +18,7 @@ Applies to both `metamask-extension` and `metamask-mobile`. See overlays for rep ## Do Not Use When -- Reviewing selector or render-cascade issues (use [`selector-antipattern-review`](../selector-antipattern-review/skill.md)) +- Reviewing selector or render-cascade issues (use [`selector-antipattern-scan`](../selector-antipattern-scan/skill.md)) - Reviewing non-React code (background scripts, workers, test utilities) - Reviewing an effect that is intentionally one-shot with no async work or timers (check patterns below anyway, but most do not apply) diff --git a/domains/performance/skills/selector-antipattern-review/repos/metamask-extension.md b/domains/performance/skills/selector-antipattern-scan/repos/metamask-extension.md similarity index 98% rename from domains/performance/skills/selector-antipattern-review/repos/metamask-extension.md rename to domains/performance/skills/selector-antipattern-scan/repos/metamask-extension.md index 81dceb6b..82801a08 100644 --- a/domains/performance/skills/selector-antipattern-review/repos/metamask-extension.md +++ b/domains/performance/skills/selector-antipattern-scan/repos/metamask-extension.md @@ -1,6 +1,6 @@ --- repo: metamask-extension -parent: selector-antipattern-review +parent: selector-antipattern-scan --- ## Paths diff --git a/domains/performance/skills/selector-antipattern-review/repos/metamask-mobile.md b/domains/performance/skills/selector-antipattern-scan/repos/metamask-mobile.md similarity index 97% rename from domains/performance/skills/selector-antipattern-review/repos/metamask-mobile.md rename to domains/performance/skills/selector-antipattern-scan/repos/metamask-mobile.md index b77ab8c6..1b324e3a 100644 --- a/domains/performance/skills/selector-antipattern-review/repos/metamask-mobile.md +++ b/domains/performance/skills/selector-antipattern-scan/repos/metamask-mobile.md @@ -1,6 +1,6 @@ --- repo: metamask-mobile -parent: selector-antipattern-review +parent: selector-antipattern-scan --- ## Paths diff --git a/domains/performance/skills/selector-antipattern-review/skill.md b/domains/performance/skills/selector-antipattern-scan/skill.md similarity index 98% rename from domains/performance/skills/selector-antipattern-review/skill.md rename to domains/performance/skills/selector-antipattern-scan/skill.md index a98d65f5..be3b81fe 100644 --- a/domains/performance/skills/selector-antipattern-review/skill.md +++ b/domains/performance/skills/selector-antipattern-scan/skill.md @@ -1,6 +1,6 @@ --- maturity: experimental -name: selector-antipattern-review +name: selector-antipattern-scan description: Review and diagnose Redux selector antipatterns that cause render cascades, pre-merge and post-merge --- @@ -18,7 +18,7 @@ Both `metamask-extension` and `metamask-mobile` share the same React + Redux arc ## Do Not Use When -- Non-selector performance concerns (effects → use `effect-antipattern-review`, context providers, virtualization) +- Non-selector performance concerns (effects → use `effect-antipattern-scan`, context providers, virtualization) - Network-bound slowness (use the Network panel, not WDYR) - Startup or initial-mount perf (use startup profiling) - Non-React trees (worker messaging, background script perf) diff --git a/domains/testing/skills/benchmark-design/skill.md b/domains/testing/skills/benchmark-design/skill.md index 641ef6c5..5f343006 100644 --- a/domains/testing/skills/benchmark-design/skill.md +++ b/domains/testing/skills/benchmark-design/skill.md @@ -16,7 +16,7 @@ description: Design, run, and analyze E2E performance benchmarks — session hyg ## Do Not Use When - Adding unit, integration, or correctness E2E tests -- Profiling a single user-reported slowdown (use `selector-antipattern-review`) +- Profiling a single user-reported slowdown (use `selector-antipattern-scan`) - Writing micro-benchmarks outside the E2E harness ## Workflow From 81dcf484656b0d01285a411286451a6daa631cfd Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 10:20:19 -0400 Subject: [PATCH 049/135] fix(race-condition-proof): name the evidence category instead of indexing it The description read "the engine behind evidence's B7 deterministic-interleaving evidence category". "B7" is an address into evidence-catalog.md, not a name: it carries no meaning to a reader who has not opened the catalog, and a frontmatter description cannot link out to one. - Drop the lane id from the description; name the role instead. - Replace the trailing Notes reference with a Related section that links the catalog by URL. A relative path would not survive installation, which flattens skills to mms-<name>/. - Drop the exogram-daemon path, which is a private repo the reader cannot open. - Update the stale pr-validate name to evidence. --- .../stability/skills/race-condition-proof/skill.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/domains/stability/skills/race-condition-proof/skill.md b/domains/stability/skills/race-condition-proof/skill.md index 3ff6d2d3..5c8b7906 100644 --- a/domains/stability/skills/race-condition-proof/skill.md +++ b/domains/stability/skills/race-condition-proof/skill.md @@ -1,6 +1,6 @@ --- name: race-condition-proof -description: Prove an ordering guarantee under concurrency — that when B arrives during A's pending window, A is canceled, or completes first, or the two commit in a defined order. Covers race conditions, retries, cancellation, supersession, debounce/throttle, locks, queues, and async state machines, where correctness IS the interleaving rather than a value. Builds a deterministic interleaving harness (fake timers advanced into the pending window, concurrent launch, microtask stepping) and asserts each guarantee separately, including asymmetric ones where two paths deliberately differ. The falsifier is a test that never interleaved — operations run to completion in sequence exercise no race and produce a vacuous green indistinguishable from a real pass, so the proof obligation is to show the interleaving occurred, not that the assertion passed. Triggers on /race-condition-proof, or when asked to prove a race condition is fixed, test cancellation or supersession, validate retry or debounce ordering, write a deterministic interleaving test, or check whether a concurrency test actually exercises the race. Callable by pr-validate as the engine behind its B7 deterministic-interleaving evidence category, and named by falsifying-test as its sibling for ordering bugs. +description: Prove an ordering guarantee under concurrency — that when B arrives during A's pending window, A is canceled, or completes first, or the two commit in a defined order. Covers race conditions, retries, cancellation, supersession, debounce/throttle, locks, queues, and async state machines, where correctness IS the interleaving rather than a value. Builds a deterministic interleaving harness (fake timers advanced into the pending window, concurrent launch, microtask stepping) and asserts each guarantee separately, including asymmetric ones where two paths deliberately differ. The falsifier is a test that never interleaved — operations run to completion in sequence exercise no race and produce a vacuous green indistinguishable from a real pass, so the proof obligation is to show the interleaving occurred, not that the assertion passed. Triggers on /race-condition-proof, or when asked to prove a race condition is fixed, test cancellation or supersession, validate retry or debounce ordering, write a deterministic interleaving test, or check whether a concurrency test actually exercises the race. Callable by `evidence` as its deterministic-interleaving engine, and named by `falsifying-test` as its sibling for ordering bugs. maturity: experimental --- @@ -106,5 +106,12 @@ guarantee depends on runtime semantics — what an `AbortController` actually ca microtask runs before a timer callback — cite the behavior or demonstrate it in the harness rather than describing it. -Engine for `pr-validate` category **B7 (deterministic interleaving)**; the full category note -lives at `exogram-daemon/artifacts/evidence-taxonomy/category-concurrency-temporal-ordering.md`. +## Related + +- `evidence` — this skill is its deterministic-interleaving engine: `evidence` decides that a + concurrency claim needs an interleaving proof, and calls here to produce one. The category note + is [`deterministic interleaving` in the evidence catalog](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md). + A relative path would not survive installation — skills flatten to `mms-<name>/`, so a link + out of one skill into another only resolves as a URL. +- `falsifying-test` — the sibling engine for ordering bugs that reproduce without a forced + interleaving. From 843a49ed99226bc2bb19194241bebf27b5081808 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 10:26:02 -0400 Subject: [PATCH 050/135] Name the evidence category in `react-render-proof` instead of indexing it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `C4` is an address into `evidence-catalog.md`. A reader who has not opened the catalog cannot resolve it, and the frontmatter `description` cannot link out to one. Both sites now name the category and link the catalog by URL — a relative path would not survive installation, which flattens skills to `mms-<name>/`. Also updates two sibling names that no longer resolve: `pr-validate` is now `evidence`, and `memory-leak-hunt` is now `memory-leak`. --- domains/performance/skills/react-render-proof/skill.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/domains/performance/skills/react-render-proof/skill.md b/domains/performance/skills/react-render-proof/skill.md index 4cb4fa2c..3f613696 100644 --- a/domains/performance/skills/react-render-proof/skill.md +++ b/domains/performance/skills/react-render-proof/skill.md @@ -1,6 +1,6 @@ --- name: react-render-proof -description: Prove a React rendering or memoization change actually reduced work, with a delivery gate and a reported band. Covers re-render counts (why-did-you-render), selector recomputes (reselect's real `.recomputations()` API), and A/B arms toggled at a FIXED commit rather than across a merge boundary. The falsifier is an arm whose treatment never reached the built bundle — a null from undelivered treatment is indistinguishable from a null from a small effect and reports as the second. Triggers on /react-render-proof, or when asked to prove a component stopped over-rendering, measure selector recomputation, validate a memoization/React Compiler change, run a render-count A/B, or interpret a re-render benchmark. Callable by pr-validate as the engine behind its React render & selector proof evidence category. +description: Prove a React rendering or memoization change actually reduced work, with a delivery gate and a reported band. Covers re-render counts (why-did-you-render), selector recomputes (reselect's real `.recomputations()` API), and A/B arms toggled at a FIXED commit rather than across a merge boundary. The falsifier is an arm whose treatment never reached the built bundle — a null from undelivered treatment is indistinguishable from a null from a small effect and reports as the second. Triggers on /react-render-proof, or when asked to prove a component stopped over-rendering, measure selector recomputation, validate a memoization/React Compiler change, run a render-count A/B, or interpret a re-render benchmark. Callable by `evidence` as its React render & selector proof engine. maturity: experimental --- @@ -48,8 +48,8 @@ reporting a difference between arms that never differed. **`.recomputations()`** on memoized selectors — a genuine API, not a patch. Read it (sample on an interval if the count should visibly climb). An injected `console.log` you added to a selector body is an authored claim, not an observation; reach for it only when no real API - exists, and say so when you do. *(Note: pr-validate's C4 entry long claimed there was "no - built-in selector-call counter". There is.)* + exists, and say so when you do. *(Note: the evidence catalog's render-and-selector entry long claimed there + was "no built-in selector-call counter". There is.)* 5. **Toggle at a fixed commit, not across a merge boundary.** Same tree in both arms, one thing different. A commit boundary drags in unrelated change you will then be unable to @@ -108,5 +108,5 @@ and their absence is what makes a number unfalsifiable. ## Related -- `pr-validate` — packages this skill's output as its C4 evidence category. -- `memory-leak-hunt`, `supply-chain-audit` — sibling engines behind other categories. +- `evidence` — packages this skill's output as its [React render & selector proof category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md). +- `memory-leak`, `supply-chain-audit` — sibling engines behind other categories. From d9c6231a11ff782ecf0b8c54978cad03124a9856 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 10:26:20 -0400 Subject: [PATCH 051/135] Update `pr-validate` references to `evidence` in the security skills `pr-validate` was renamed to `evidence`; both skills still named the old one, in a section heading, prose, and a `## Related` entry. `supply-chain-audit` now also links its evidence category in the catalog rather than naming it bare. --- domains/security/skills/lavamoat-policy-diligence/skill.md | 4 ++-- domains/security/skills/supply-chain-audit/skill.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/domains/security/skills/lavamoat-policy-diligence/skill.md b/domains/security/skills/lavamoat-policy-diligence/skill.md index a1e3491f..4a86838d 100644 --- a/domains/security/skills/lavamoat-policy-diligence/skill.md +++ b/domains/security/skills/lavamoat-policy-diligence/skill.md @@ -223,9 +223,9 @@ A clean policy diff does not mean a safe dependency, and a known CVE does not sh grant. Run the umbrella skill when the question is "is this bump safe"; run this one when the question is "why does it need that". -## Called by supply-chain-audit and pr-validate +## Called by supply-chain-audit and evidence -`supply-chain-audit` delegates its capability-containment lane here. pr-validate keeps +`supply-chain-audit` delegates its capability-containment lane here. `evidence` keeps **supply-chain** as an evidence category and packages the per-grant justification (accept / reject, each with its permalink) posted where the policy is reviewed. Engine helper: `scripts/policy-audit.py`. Usable standalone whenever a policy grant needs a reason. diff --git a/domains/security/skills/supply-chain-audit/skill.md b/domains/security/skills/supply-chain-audit/skill.md index 32dbffd2..8a8cb91c 100644 --- a/domains/security/skills/supply-chain-audit/skill.md +++ b/domains/security/skills/supply-chain-audit/skill.md @@ -1,6 +1,6 @@ --- name: supply-chain-audit -description: Assess whether a dependency change is safe to take, across every detector that answers a different part of that question — Socket Security (malicious/anomalous package behavior, install scripts, new maintainers), `yarn npm audit` and advisories (known vulnerabilities), lockfile and manifest diffs (what actually changed, including transitive and resolution swaps), and LavaMoat policy grants (new capabilities, delegated to `lavamoat-policy-diligence`). Also covers the fronts no upstream scanner sees because they are things your own repo does to dependencies afterwards: yarn patches that modify dependency source at install, `resolutions` that force or stub versions, `npmAuditIgnoreAdvisories` suppression lists, CI actions riding mutable tags instead of pinned SHAs, and yarn plugins that execute at install. The falsifier is a lane whose finding is unaccounted for — a flagged package, an unresolved advisory, or a grant with no call site. Detection belongs to the tools; the job is disposition, and handing it to the humans who own the dependency. Triggers on /supply-chain-audit, or when asked whether a dependency bump is safe, to review a lockfile or package.json change, to triage a Socket or audit finding, or to assess supply-chain risk of a change. Callable by pr-validate as the engine behind its supply-chain evidence category. +description: Assess whether a dependency change is safe to take, across every detector that answers a different part of that question — Socket Security (malicious/anomalous package behavior, install scripts, new maintainers), `yarn npm audit` and advisories (known vulnerabilities), lockfile and manifest diffs (what actually changed, including transitive and resolution swaps), and LavaMoat policy grants (new capabilities, delegated to `lavamoat-policy-diligence`). Also covers the fronts no upstream scanner sees because they are things your own repo does to dependencies afterwards: yarn patches that modify dependency source at install, `resolutions` that force or stub versions, `npmAuditIgnoreAdvisories` suppression lists, CI actions riding mutable tags instead of pinned SHAs, and yarn plugins that execute at install. The falsifier is a lane whose finding is unaccounted for — a flagged package, an unresolved advisory, or a grant with no call site. Detection belongs to the tools; the job is disposition, and handing it to the humans who own the dependency. Triggers on /supply-chain-audit, or when asked whether a dependency bump is safe, to review a lockfile or package.json change, to triage a Socket or audit finding, or to assess supply-chain risk of a change. Callable by `evidence` as its supply-chain engine. maturity: experimental --- @@ -155,4 +155,4 @@ unresolved and what would settle it. ## Related - `lavamoat-policy-diligence` — the capability-containment engine this skill delegates to. -- `pr-validate` — packages this skill's output as its supply-chain evidence category. +- `evidence` — packages this skill's output as its [supply-chain evidence category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md). From e7e924814a791c172499a8d3c3bfd4cf6ae78c65 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 10:26:40 -0400 Subject: [PATCH 052/135] Repair unresolvable references in `evidence` and `falsifying-test` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The engine table still routed the memory-leak category to `/memory-leak-hunt`, which was renamed to `/memory-leak` — the rename missed the PR that performed it. Two `[[snake_case]]` entries were wiki links to a private authoring vault; they render as literal brackets here and resolve for no reader. One had a real counterpart in `references/` and now links it; the other pointed at a file that does not exist and is dropped rather than left dangling. `falsifying-test` named its evidence categories as `B3` and `B7`. Those are addresses into `evidence-catalog.md`, not names, so both now use the category name and link the catalog. --- domains/pr-workflow/skills/evidence/skill.md | 5 ++--- domains/testing/skills/falsifying-test/skill.md | 5 +++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 53c3c87e..46dc7efe 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -256,12 +256,11 @@ Where evidence sits in the PR lifecycle (see the public `pr-workflow` siblings): | B3 falsifying regression test | `/falsifying-test` | | B7 deterministic interleaving (concurrency / ordering) | `/race-condition-proof` | | C4 React render & selector proof | `/react-render-proof` | - | C9 memory leak | `/memory-leak-hunt` | + | C9 memory leak | `/memory-leak` | | D supply-chain / dependency change | `/supply-chain-audit` → delegates capability grants to `/lavamoat-policy-diligence` | **An engine that defines its own output contract publishes in it.** `lavamoat-policy-diligence` is the live case: read-level triage, no verdict, its own header and marker pair. Do not re-frame it as a Validation Run — see *One comment per evidence kind* in [references/evidence-publishing.md](references/evidence-publishing.md). -- [[reference_aep_local_run]] — the source memory this skill encodes. -- [[reference_sentry_project_topology]] — Sentry project mapping for the telemetry-evidence lane. +- [references/aep-local-run.md](references/aep-local-run.md) — the local-run procedure this skill encodes. diff --git a/domains/testing/skills/falsifying-test/skill.md b/domains/testing/skills/falsifying-test/skill.md index e0a2f367..97219111 100644 --- a/domains/testing/skills/falsifying-test/skill.md +++ b/domains/testing/skills/falsifying-test/skill.md @@ -80,7 +80,8 @@ Falsifying test — <test name> (Fixes #N) ## Related -- `evidence` — packages this skill's output as its B3 evidence category; B7 (deterministic - interleaving) is the sibling for concurrency and temporal-ordering bugs. +- `evidence` — packages this skill's output as its [falsifying regression test category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md). + The deterministic-interleaving category is the sibling for concurrency and temporal-ordering + bugs; `race-condition-proof` drives it. - `react-render-proof` — the same before/after discipline applied to a measured quantity rather than a boolean. From 698d4f10df7820937793a2e2632d7acbf6211d27 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 10:26:58 -0400 Subject: [PATCH 053/135] Drop references to a skill that does not exist from `agent-run-cost` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `sentry-quota` was named twice — once as the framing analogy, once in `## Related` — but it exists on no branch of this repo. The analogy is kept as a description of the guard; the `## Related` entry is removed. Also updates `pr-validate` to its current name, `evidence`. --- domains/agentic/skills/agent-run-cost/skill.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/domains/agentic/skills/agent-run-cost/skill.md b/domains/agentic/skills/agent-run-cost/skill.md index f44ee8b0..b37dbc30 100644 --- a/domains/agentic/skills/agent-run-cost/skill.md +++ b/domains/agentic/skills/agent-run-cost/skill.md @@ -18,7 +18,7 @@ Scripted automation announces its cost in wall-clock time; agentic automation do A fan-out of forty subagents and a single call are the same shape in a diff, and the difference surfaces later, on a bill, attributed to nothing in particular. -This is the token-spend counterpart to `sentry-quota`, which guards span volume. Same +This is the token-spend counterpart to a span-volume quota guard. Same posture: operate on **code and PRs**, before the spend exists, and produce figures. ## When to use @@ -101,8 +101,7 @@ Cheapest first; stop at the rung that fits. ## Related -- `sentry-quota` — the same guard for span volume; `fan-out × ungated × no-kill-switch`. -- `pr-validate` — weighs AEP run cost when choosing an evidence lane, and tears the stack +- `evidence` — weighs AEP run cost when choosing an evidence lane, and tears the stack down after; this skill is the review-side version for workflows others will run. - [`MetaMask/decisions#173`](https://github.com/MetaMask/decisions/pull/173) — ADR-0058 review, where the missing token-cost estimate was raised as an open question. From e170b394c277906fb5ba3b17e62d60eca7542fee Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 10:44:06 -0400 Subject: [PATCH 054/135] Rename `compiler-blindspots` to `tsc-blindspots` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Skills install flat as `mms-<name>`, so `domains/typescript/` does not disambiguate the name at the callsite — and "compiler" reads as React Compiler in a repo where that is a live subject. The skill is about `tsc` specifically: its own first clause is "the type defects `tsc` is structurally unable to report". Also adds the slash trigger to the description, which listed only prose phrases. --- .../references/false-negatives.md | 0 .../references/metamask-extension.md | 0 .../references/worked-example.md | 0 .../scripts/substitution-ab.sh | 0 .../{compiler-blindspots => tsc-blindspots}/skill.md | 9 +++++---- 5 files changed, 5 insertions(+), 4 deletions(-) rename domains/typescript/skills/{compiler-blindspots => tsc-blindspots}/references/false-negatives.md (100%) rename domains/typescript/skills/{compiler-blindspots => tsc-blindspots}/references/metamask-extension.md (100%) rename domains/typescript/skills/{compiler-blindspots => tsc-blindspots}/references/worked-example.md (100%) rename domains/typescript/skills/{compiler-blindspots => tsc-blindspots}/scripts/substitution-ab.sh (100%) rename domains/typescript/skills/{compiler-blindspots => tsc-blindspots}/skill.md (98%) diff --git a/domains/typescript/skills/compiler-blindspots/references/false-negatives.md b/domains/typescript/skills/tsc-blindspots/references/false-negatives.md similarity index 100% rename from domains/typescript/skills/compiler-blindspots/references/false-negatives.md rename to domains/typescript/skills/tsc-blindspots/references/false-negatives.md diff --git a/domains/typescript/skills/compiler-blindspots/references/metamask-extension.md b/domains/typescript/skills/tsc-blindspots/references/metamask-extension.md similarity index 100% rename from domains/typescript/skills/compiler-blindspots/references/metamask-extension.md rename to domains/typescript/skills/tsc-blindspots/references/metamask-extension.md diff --git a/domains/typescript/skills/compiler-blindspots/references/worked-example.md b/domains/typescript/skills/tsc-blindspots/references/worked-example.md similarity index 100% rename from domains/typescript/skills/compiler-blindspots/references/worked-example.md rename to domains/typescript/skills/tsc-blindspots/references/worked-example.md diff --git a/domains/typescript/skills/compiler-blindspots/scripts/substitution-ab.sh b/domains/typescript/skills/tsc-blindspots/scripts/substitution-ab.sh similarity index 100% rename from domains/typescript/skills/compiler-blindspots/scripts/substitution-ab.sh rename to domains/typescript/skills/tsc-blindspots/scripts/substitution-ab.sh diff --git a/domains/typescript/skills/compiler-blindspots/skill.md b/domains/typescript/skills/tsc-blindspots/skill.md similarity index 98% rename from domains/typescript/skills/compiler-blindspots/skill.md rename to domains/typescript/skills/tsc-blindspots/skill.md index 0cfaa6c3..495142b4 100644 --- a/domains/typescript/skills/compiler-blindspots/skill.md +++ b/domains/typescript/skills/tsc-blindspots/skill.md @@ -1,5 +1,5 @@ --- -name: compiler-blindspots +name: tsc-blindspots description: >- Find the type defects `tsc` is structurally unable to report — a green build is not evidence the types are correct. Covers the two classes: (1) hand-written @@ -14,9 +14,10 @@ description: >- enum lookups, calls made optional so a throw becomes a silent no-op. Use when reviewing a JS→TS migration, a PR that hand-writes types for values that already have them, a "rename-only" refactor, or any PR claiming a change is mechanical. - Trigger phrases include "validate this TypeScript migration", "is this type - right", "does this type match the real shape", "why didn't CI catch this type", - "derive vs define", and "what can tsc not check". + Triggers on /tsc-blindspots, or on phrases like "validate this TypeScript + migration", "is this type right", "does this type match the real shape", + "why didn't CI catch this type", "derive vs define", and "what can tsc not + check". maturity: experimental --- From 0da684bdca7fc54d284ce7d813e37c2a5df47d09 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 11:47:10 -0400 Subject: [PATCH 055/135] Rename `sentry-grafana-cross-ref` to `sentry-grafana-correlation` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Cross-ref" describes the mechanism and reads as a generic pointer. The skill produces a correlation: one trace joined across two stores by `trace_id`, plus a reading of which absences are expected. The vendor names stay. They are load-bearing here rather than incidental — the skill is about the split between Sentry and Grafana Tempo specifically, and is not source-agnostic. --- domains/analytics/skills/grafana-tempo-queries/skill.md | 4 ++-- .../skill.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) rename domains/analytics/skills/{sentry-grafana-cross-ref => sentry-grafana-correlation}/skill.md (99%) diff --git a/domains/analytics/skills/grafana-tempo-queries/skill.md b/domains/analytics/skills/grafana-tempo-queries/skill.md index cc720dbf..99fee6ad 100644 --- a/domains/analytics/skills/grafana-tempo-queries/skill.md +++ b/domains/analytics/skills/grafana-tempo-queries/skill.md @@ -6,7 +6,7 @@ maturity: experimental # grafana-tempo-queries -Tempo holds **backend** spans. Client spans from the extension and mobile go to Sentry via the SDK's own transport and never appear here — so a Tempo trace normally starts at an inbound server span, and a missing root is expected rather than broken. To join the two halves, see `sentry-grafana-cross-ref`. +Tempo holds **backend** spans. Client spans from the extension and mobile go to Sentry via the SDK's own transport and never appear here — so a Tempo trace normally starts at an inbound server span, and a missing root is expected rather than broken. To join the two halves, see `sentry-grafana-correlation`. ## Setup @@ -122,4 +122,4 @@ Prefer an absolute `from`/`to` when the link needs to outlive the event; a relat | Tag-values returns 502 | High cardinality | Inspect traces directly | | Id from search not found elsewhere | Leading zeros stripped | Zero-pad to 32 chars | | Kind filter matches nothing | Comparing to a number | Compare to `SPAN_KIND_*` | -| Trace has no root | Root is a client span | Expected; see `sentry-grafana-cross-ref` | +| Trace has no root | Root is a client span | Expected; see `sentry-grafana-correlation` | diff --git a/domains/analytics/skills/sentry-grafana-cross-ref/skill.md b/domains/analytics/skills/sentry-grafana-correlation/skill.md similarity index 99% rename from domains/analytics/skills/sentry-grafana-cross-ref/skill.md rename to domains/analytics/skills/sentry-grafana-correlation/skill.md index 56e8c327..0ff7135a 100644 --- a/domains/analytics/skills/sentry-grafana-cross-ref/skill.md +++ b/domains/analytics/skills/sentry-grafana-correlation/skill.md @@ -1,10 +1,10 @@ --- -name: sentry-grafana-cross-ref +name: sentry-grafana-correlation description: Join one trace across Sentry and Grafana Tempo by trace id to see the whole client-to-backend path, and diagnose why a half is missing. Covers the split-store model (client spans reach Sentry through the SDK and survive only head sampling; backend spans reach Tempo through tail sampling and Sentry through environment routing), the classification of both-halves / client-only / backend-only outcomes with the sampling and routing rule that causes each, and the id-padding, time-window, and query-syntax traps that make a present trace look absent. Use when a trace looks truncated, a backend span has no parent, per-hop latency needs attributing across the seam, or you need to know which store should hold a given span. Triggers on cross-stack trace, orphaned span, trace id lookup, client-backend correlation, split waterfall, or "where did the rest of the trace go". maturity: experimental --- -# sentry-grafana-cross-ref +# sentry-grafana-correlation One request produces spans in two stores, joined only by `trace_id`. Reading a trace end to end means querying both and knowing which absences are expected. From 6a9343423cace346504e9daae3cf40ef1a42b4ba Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 11:53:15 -0400 Subject: [PATCH 056/135] Rename `race-condition-proof` to `race-condition-repro` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit As a noun suffix, `-proof` is privative in English — waterproof, bulletproof, tamper-proof all mean "immune to". So `race-condition-proof` parses as "immune to race conditions" rather than "produces a proof about ordering", and that misreading is plausible enough not to self-correct. `-repro` names what the harness produces and carries no such inversion. --- .../{race-condition-proof => race-condition-repro}/skill.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename domains/stability/skills/{race-condition-proof => race-condition-repro}/skill.md (98%) diff --git a/domains/stability/skills/race-condition-proof/skill.md b/domains/stability/skills/race-condition-repro/skill.md similarity index 98% rename from domains/stability/skills/race-condition-proof/skill.md rename to domains/stability/skills/race-condition-repro/skill.md index 5c8b7906..47b8de16 100644 --- a/domains/stability/skills/race-condition-proof/skill.md +++ b/domains/stability/skills/race-condition-repro/skill.md @@ -1,10 +1,10 @@ --- -name: race-condition-proof -description: Prove an ordering guarantee under concurrency — that when B arrives during A's pending window, A is canceled, or completes first, or the two commit in a defined order. Covers race conditions, retries, cancellation, supersession, debounce/throttle, locks, queues, and async state machines, where correctness IS the interleaving rather than a value. Builds a deterministic interleaving harness (fake timers advanced into the pending window, concurrent launch, microtask stepping) and asserts each guarantee separately, including asymmetric ones where two paths deliberately differ. The falsifier is a test that never interleaved — operations run to completion in sequence exercise no race and produce a vacuous green indistinguishable from a real pass, so the proof obligation is to show the interleaving occurred, not that the assertion passed. Triggers on /race-condition-proof, or when asked to prove a race condition is fixed, test cancellation or supersession, validate retry or debounce ordering, write a deterministic interleaving test, or check whether a concurrency test actually exercises the race. Callable by `evidence` as its deterministic-interleaving engine, and named by `falsifying-test` as its sibling for ordering bugs. +name: race-condition-repro +description: Prove an ordering guarantee under concurrency — that when B arrives during A's pending window, A is canceled, or completes first, or the two commit in a defined order. Covers race conditions, retries, cancellation, supersession, debounce/throttle, locks, queues, and async state machines, where correctness IS the interleaving rather than a value. Builds a deterministic interleaving harness (fake timers advanced into the pending window, concurrent launch, microtask stepping) and asserts each guarantee separately, including asymmetric ones where two paths deliberately differ. The falsifier is a test that never interleaved — operations run to completion in sequence exercise no race and produce a vacuous green indistinguishable from a real pass, so the proof obligation is to show the interleaving occurred, not that the assertion passed. Triggers on /race-condition-repro, or when asked to prove a race condition is fixed, test cancellation or supersession, validate retry or debounce ordering, write a deterministic interleaving test, or check whether a concurrency test actually exercises the race. Callable by `evidence` as its deterministic-interleaving engine, and named by `falsifying-test` as its sibling for ordering bugs. maturity: experimental --- -# /race-condition-proof +# /race-condition-repro A race is nondeterministic in the wild, so you cannot validate "the stale retry was canceled" by running the code and hoping the interleaving occurs. The evidence has to **make the race From c18c751a79157d2cbfe3a3de69d07a94a8396548 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 11:53:26 -0400 Subject: [PATCH 057/135] Rename `react-render-proof` to `react-render-delta` `-proof` as a noun suffix reads as "immune to", so the old name parsed as "immune to React renders". `-delta` names what the skill actually produces, and matches the skill's own insistence that its output is a measured quantity rather than a boolean. --- .../{react-render-proof => react-render-delta}/skill.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename domains/performance/skills/{react-render-proof => react-render-delta}/skill.md (98%) diff --git a/domains/performance/skills/react-render-proof/skill.md b/domains/performance/skills/react-render-delta/skill.md similarity index 98% rename from domains/performance/skills/react-render-proof/skill.md rename to domains/performance/skills/react-render-delta/skill.md index 3f613696..919c17c3 100644 --- a/domains/performance/skills/react-render-proof/skill.md +++ b/domains/performance/skills/react-render-delta/skill.md @@ -1,10 +1,10 @@ --- -name: react-render-proof -description: Prove a React rendering or memoization change actually reduced work, with a delivery gate and a reported band. Covers re-render counts (why-did-you-render), selector recomputes (reselect's real `.recomputations()` API), and A/B arms toggled at a FIXED commit rather than across a merge boundary. The falsifier is an arm whose treatment never reached the built bundle — a null from undelivered treatment is indistinguishable from a null from a small effect and reports as the second. Triggers on /react-render-proof, or when asked to prove a component stopped over-rendering, measure selector recomputation, validate a memoization/React Compiler change, run a render-count A/B, or interpret a re-render benchmark. Callable by `evidence` as its React render & selector proof engine. +name: react-render-delta +description: Prove a React rendering or memoization change actually reduced work, with a delivery gate and a reported band. Covers re-render counts (why-did-you-render), selector recomputes (reselect's real `.recomputations()` API), and A/B arms toggled at a FIXED commit rather than across a merge boundary. The falsifier is an arm whose treatment never reached the built bundle — a null from undelivered treatment is indistinguishable from a null from a small effect and reports as the second. Triggers on /react-render-delta, or when asked to prove a component stopped over-rendering, measure selector recomputation, validate a memoization/React Compiler change, run a render-count A/B, or interpret a re-render benchmark. Callable by `evidence` as its React render & selector proof engine. maturity: experimental --- -# /react-render-proof +# /react-render-delta A render-count number is worthless until two things are true: the **treatment reached the artifact the browser executes**, and the number is reported as a **band** rather than a point. From 1e927d5833af17a3b93a5fe3ec751fe0faf40855 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 11:53:39 -0400 Subject: [PATCH 058/135] Follow the engine renames in the catalog and `falsifying-test` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `race-condition-proof` is now `race-condition-repro` and `react-render-proof` is now `react-render-delta`. Both are named here as engines, in the catalog, the engine table, and the sibling reference — none of which the renaming branches can reach. --- .../skills/evidence/references/evidence-catalog.md | 4 ++-- domains/pr-workflow/skills/evidence/skill.md | 4 ++-- domains/testing/skills/falsifying-test/skill.md | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md index d3578ca3..ae90d30f 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md @@ -44,7 +44,7 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- - **Reach for it:** every bug-fix PR. If you can't write a test that fails on main, question whether the fix addresses the reported bug. ## B7. Deterministic interleaving test (concurrency / temporal-ordering) ⭐ -- **Engine:** `race-condition-proof` — run it rather than hand-rolling the harness. +- **Engine:** `race-condition-repro` — run it rather than hand-rolling the harness. - **Proves:** an ordering guarantee under interleaving — retry, cancellation, supersession, debounce, locks, queues, async state machines — where the correctness *is* the ordering under races, not a value. Full category: `exogram-daemon/artifacts/evidence-taxonomy/category-concurrency-temporal-ordering.md`. - **Capture:** force each race deterministically — `jest.useFakeTimers()` + `advanceTimersByTimeAsync(DELAY)` to fire the delayed action at a known point; `Promise.all([opA, opB])` to overlap operations; `advanceTimersByTimeAsync(0)` to step to a precise interleaving point; then assert the ordering/cancellation outcome for **each** guarantee, including asymmetric ones (one path canceled → its recovery event `.not.toHaveBeenCalled()`; another must complete → `.toHaveBeenCalledWith(...)`). Corroborate with transition telemetry; for the integration path, a live forced-race capture (C8/CDP, the #44610 technique). - **Trust-gate:** the test must **actually interleave** — time advanced into the pending window, the superseding op injected *during* it. A sequential run exercises no race and is a vacuous green. Verify the interleaving, not just the assertion. @@ -80,7 +80,7 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- - **Capture:** `ui/helpers/utils/performance-observers.ts`; `window.stateHooks.getLongTaskMetricsWithTBT()` → `{count, totalDuration, maxDuration, tbt, tbtRating}`. TBT good<200 / needs-improvement<600 / poor>600. Sampled 10% prod / 100% test. ## C4. React render & selector proof - - **Engine: the `react-render-proof` skill.** Delegate the measurement to it; it runs the source/delivery/metric gates, derives the needle from real build output, repeats the capture, and returns a band (or "not resolvable at this n" with an MDE). evidence packages the result. + - **Engine: the `react-render-delta` skill.** Delegate the measurement to it; it runs the source/delivery/metric gates, derives the needle from real build output, repeats the capture, and returns a band (or "not resolvable at this n" with an MDE). evidence packages the result. - **Proves:** a component/selector stopped over-rendering (cascade-amplification before/after — exogram `react-redux-performance`). - **Capture:** WDYR via `ENABLE_WHY_DID_YOU_RENDER` (`.metamaskrc` or env) — wired in `app/scripts/development/wdyr.ts` (`trackAllPureComponents`); console logs each unnecessary re-render. `yarn devtools:react` for the Profiler flame graph. Selectors use `reselect`'s `createSelector`, which **does expose a real `.recomputations()` counter** — read it (sample on an interval if the count should visibly climb) rather than injecting a log into the selector body; an injected log is an authored claim, a library API is an observation. *(This entry previously said there was no built-in counter. There is.)* - **Bar:** the delivery check comes before the number. An arm whose manipulation cannot be observed in the built bundle produces a null indistinguishable from "small effect" — and reports as the second. diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 46dc7efe..55a585ed 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -254,8 +254,8 @@ Where evidence sits in the PR lifecycle (see the public `pr-workflow` siblings): | category | engine | |---|---| | B3 falsifying regression test | `/falsifying-test` | - | B7 deterministic interleaving (concurrency / ordering) | `/race-condition-proof` | - | C4 React render & selector proof | `/react-render-proof` | + | B7 deterministic interleaving (concurrency / ordering) | `/race-condition-repro` | + | C4 React render & selector proof | `/react-render-delta` | | C9 memory leak | `/memory-leak` | | D supply-chain / dependency change | `/supply-chain-audit` → delegates capability grants to `/lavamoat-policy-diligence` | diff --git a/domains/testing/skills/falsifying-test/skill.md b/domains/testing/skills/falsifying-test/skill.md index 97219111..7cfeb160 100644 --- a/domains/testing/skills/falsifying-test/skill.md +++ b/domains/testing/skills/falsifying-test/skill.md @@ -82,6 +82,6 @@ Falsifying test — <test name> (Fixes #N) - `evidence` — packages this skill's output as its [falsifying regression test category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md). The deterministic-interleaving category is the sibling for concurrency and temporal-ordering - bugs; `race-condition-proof` drives it. -- `react-render-proof` — the same before/after discipline applied to a measured quantity + bugs; `race-condition-repro` drives it. +- `react-render-delta` — the same before/after discipline applied to a measured quantity rather than a boolean. From affac8f1fb25e96fd65f332394652537406b8322 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 11:53:56 -0400 Subject: [PATCH 059/135] Update `pr-validate` references to `evidence` in `memory-leak` Missed when the rename swept the other branches: the description, the section heading, and two prose references all still named `pr-validate`. The evidence category is now linked to the catalog rather than named bare. --- domains/stability/skills/memory-leak/skill.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/domains/stability/skills/memory-leak/skill.md b/domains/stability/skills/memory-leak/skill.md index fc18f6ac..e59966b9 100644 --- a/domains/stability/skills/memory-leak/skill.md +++ b/domains/stability/skills/memory-leak/skill.md @@ -1,6 +1,6 @@ --- name: memory-leak -description: Find and investigate memory leaks / retention issues in JavaScript/TypeScript. Two phases. (1) Static identification from a diff — enumerate the retention primitives the change introduces (event listeners, timers, pending-request registries, subscriptions, module singletons, growing collections), pair every acquire with its release site, and scope findings to what the diff adds versus what pre-exists. (2) Runtime investigation, only for a primitive that cannot be paired statically — DevTools/CDP heap snapshots over N cycles, the retainer graph, detached-node count, and a falsifying lifecycle test. Leads with the cheap static read (the retention review a reviewer already performs) and escalates to a heap snapshot only where the read is inconclusive. Triggers on /memory-leak, or when asked to find or investigate a memory leak, check listener/subscription/timer cleanup, review a diff for retention, or take and read a heap snapshot. Callable by pr-validate as the engine behind its memory-leak evidence category. +description: Find and investigate memory leaks / retention issues in JavaScript/TypeScript. Two phases. (1) Static identification from a diff — enumerate the retention primitives the change introduces (event listeners, timers, pending-request registries, subscriptions, module singletons, growing collections), pair every acquire with its release site, and scope findings to what the diff adds versus what pre-exists. (2) Runtime investigation, only for a primitive that cannot be paired statically — DevTools/CDP heap snapshots over N cycles, the retainer graph, detached-node count, and a falsifying lifecycle test. Leads with the cheap static read (the retention review a reviewer already performs) and escalates to a heap snapshot only where the read is inconclusive. Triggers on /memory-leak, or when asked to find or investigate a memory leak, check listener/subscription/timer cleanup, review a diff for retention, or take and read a heap snapshot. Callable by `evidence` as its memory-leak engine. maturity: experimental --- @@ -145,11 +145,11 @@ it, go straight to Phase 2, and let the intervention test carry the causal claim Phase-2 counterpart to #40684: the same discipline that *proves the absence* of a leak (#40684, the read settles it) *proves the presence and cause* of one here. -## Called by pr-validate +## Called by `evidence` -pr-validate keeps **memory leak** as an evidence category and delegates the analysis here: +`evidence` keeps [**memory leak**](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md) as an evidence category and delegates the analysis here: it invokes this skill on the PR's diff, takes the verdict + the paired/unpaired sites, and packages them as the category's evidence (an in-situ capture of the scan, plus the lifecycle -test or retainer graph if Phase 2 ran). This skill is the engine; pr-validate is the +test or retainer graph if Phase 2 ran). This skill is the engine; `evidence` is the orchestrator that publishes the result. Usable standalone for any leak hunt, in review or in an incident, PR or not. From c57959197f6759524557ce1056aa056a24412ed9 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 00:59:28 +0900 Subject: [PATCH 060/135] feat(cli): resolve hook registration at install, and on demand (#100) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stacked on #99, which is what makes `hooks/` reach a consumer at all. **Base is #99's branch, not `main`** — review that one first. ## The gap Copying a hook does not activate it. Claude Code runs one only once it is registered in `settings.json`, and the path to register is **absolute** — different per machine, per consumer repo, and per `mms-` prefixed skill directory. So the setup reference could only ever say: ``` python3 /absolute/path/to/evidence/hooks/pr-evidence-gate.py ``` and leave the reader to work out what that is. Meanwhile `evidence` cites the hook twice in its body as its enforcement mechanism, so someone installing it reasonably assumes the gate is live. It isn't. ## Two surfaces, one output **At install** — when any installed skill ships a hook, `tools/install` prints the registration with every path resolved against the actual install: ``` Note: 1 skill(s) ship a hook. Copying the file does not activate it — Claude Code runs a hook only once it is registered in settings.json. Add this to ~/.claude/settings.json (or <target>/.claude/settings.json): { "hooks": { "PreToolUse": [ { "matcher": "Bash", "hooks": [ { "type": "command", "command": "python3 <target>/.claude/skills/mms-gatekeeper/hooks/evidence-gate.py" } ] } ] } } ``` **On demand** — `metamask-skills hooks [--target <path>]` prints the same thing, for anyone who scrolled past it or is re-registering later. Says *"No installed skill ships a hook"* rather than printing empty JSON, and exits non-zero only when the target has no installed skills at all. ## What it deliberately does not do **Neither surface writes to `settings.json`.** Editing a user's operator config is a materially larger permission than "copy files into the repo you pointed me at", and it should be decided deliberately rather than inherited as a side effect of shipping one hook. This is also the only hook in the corpus — a sample of one is thin evidence for automating a write to `$HOME`. ## Test plan - [x] `yarn test` — 69 pass / 0 fail across three files - [x] Fixture skill with `hooks/evidence-gate.py`: file delivered, registration printed, path resolved - [x] Printed registration **parses as JSON** — asserted by `JSON.parse`, matcher and command checked - [x] `metamask-skills hooks` emits the same registration - [x] Target with skills but no hooks → message, exit 0 - [x] Target with no installed skills → warning, exit 1 ## Note An earlier version emitted a trailing comma and told the reader to delete it. Handing someone JSON that doesn't parse is worse than handing them none, so the entries are joined properly and the output is valid as printed. --- bin/metamask-skills.mjs | 52 ++++++++++++++++++++++++++++++ test/cli.test.mjs | 70 +++++++++++++++++++++++++++++++++++++++++ tools/install | 45 ++++++++++++++++++++++++++ 3 files changed, 167 insertions(+) diff --git a/bin/metamask-skills.mjs b/bin/metamask-skills.mjs index af1ec09a..ea4cb0a0 100755 --- a/bin/metamask-skills.mjs +++ b/bin/metamask-skills.mjs @@ -22,6 +22,7 @@ Usage: metamask-skills describe <skill|domain/skill> [options] metamask-skills sync [options] metamask-skills postinstall [options] + metamask-skills hooks [options] metamask-skills install [options] Options: @@ -799,6 +800,55 @@ function invokedDirectly() { } } + +/** + * Print the Claude Code registration for every hook an installed skill ships. + * + * The installer copies `hooks/` like any other bundle directory, but a hook does nothing + * until it is registered in settings.json — and the path to register is absolute, so it + * differs per machine and per consumer repo and cannot be documented as a constant. This + * resolves it against the actual install. + */ +function printHookRegistration(args) { + const { target } = parseGlobalArgs(args); + const skillsDir = path.join(target, '.claude', 'skills'); + + let entries = []; + try { + for (const skill of readdirSync(skillsDir, { withFileTypes: true })) { + if (!skill.isDirectory()) continue; + const hooks = path.join(skillsDir, skill.name, 'hooks'); + if (!dirExists(hooks)) continue; + for (const file of readdirSync(hooks)) { + if (file.endsWith('.py')) entries.push(path.join(hooks, file)); + } + } + } catch { + warn(`no installed skills found under ${skillsDir}`); + return 1; + } + + if (entries.length === 0) { + process.stdout.write('No installed skill ships a hook.\n'); + return 0; + } + + const commands = entries + .map((f) => ` { "type": "command", "command": "python3 ${f}" }`) + .join(',\n'); + + process.stdout.write( + `${entries.length} hook(s) installed. Copying a hook does not activate it — Claude Code\n` + + `runs one only once it is registered. Add this to ~/.claude/settings.json, or to\n` + + `${path.join(target, '.claude', 'settings.json')} to scope it to this repo:\n\n` + + ' {\n "hooks": {\n "PreToolUse": [\n {\n "matcher": "Bash",\n "hooks": [\n' + + `${commands}\n` + + ' ]\n }\n ]\n }\n }\n', + ); + return 0; +} + + if (invokedDirectly()) { const [command, ...args] = process.argv.slice(2); if (!command || command === '-h' || command === '--help') { @@ -817,6 +867,8 @@ if (invokedDirectly()) { exitCode = sync(args); } else if (command === 'postinstall') { exitCode = postinstall(args); + } else if (command === 'hooks') { + exitCode = printHookRegistration(args); } else if (command === 'install') { exitCode = install(args); } else { diff --git a/test/cli.test.mjs b/test/cli.test.mjs index e4829a10..d14fbf43 100644 --- a/test/cli.test.mjs +++ b/test/cli.test.mjs @@ -240,3 +240,73 @@ describe('managed skill pruning', () => { assert.equal(existsSync(stale), true); }); }); + +describe('hook registration', () => { + // Copying a hook does not activate it: Claude Code runs one only once it is registered + // in settings.json, and the path to register is absolute — different per machine and per + // consumer repo, so it cannot be documented as a constant. Both surfaces resolve it. + let root; + let source; + let target; + + before(() => { + root = mkdtempSync(path.join(os.tmpdir(), 'mms-hooks-')); + source = path.join(root, 'source'); + target = path.join(root, 'target'); + const dir = path.join(source, 'domains', 'testing', 'skills', 'gatekeeper'); + mkdirSync(path.join(dir, 'hooks'), { recursive: true }); + mkdirSync(path.join(source, 'tools'), { recursive: true }); + symlinkSync(INSTALL, path.join(source, 'tools', 'install')); + mkdirSync(target, { recursive: true }); + writeFileSync( + path.join(dir, 'skill.md'), + ['---', 'name: gatekeeper', 'description: Gate writes', 'maturity: stable', '---', 'Body.'].join('\n'), + ); + writeFileSync(path.join(dir, 'hooks', 'evidence-gate.py'), 'print("gate")\n'); + const r = spawnSync('bash', [INSTALL, '--target', target, '--repo', 'core', '--source', source], { + encoding: 'utf8', + }); + assert.equal(r.status, 0, r.stderr); + installOutput = r.stdout; + }); + + let installOutput = ''; + + after(() => { + rmSync(root, { recursive: true, force: true }); + }); + + test('the hook file is delivered', () => { + assert.ok( + existsSync(path.join(target, '.claude/skills', 'mms-gatekeeper', 'hooks', 'evidence-gate.py')), + ); + }); + + test('install prints a registration with the path resolved', () => { + assert.match(installOutput, /Copying the file does not activate it/u); + assert.match(installOutput, /mms-gatekeeper\/hooks\/evidence-gate\.py/u); + }); + + test('the printed registration is valid JSON', () => { + const body = installOutput.slice(installOutput.indexOf('{'), installOutput.lastIndexOf('}') + 1); + const parsed = JSON.parse(body); + assert.equal(parsed.hooks.PreToolUse[0].matcher, 'Bash'); + assert.match(parsed.hooks.PreToolUse[0].hooks[0].command, /^python3 \//u); + }); + + test('the hooks subcommand prints the same registration on demand', () => { + const r = spawnSync(process.execPath, [BIN, 'hooks', '--target', target], { encoding: 'utf8' }); + assert.equal(r.status, 0, r.stderr); + const body = r.stdout.slice(r.stdout.indexOf('{'), r.stdout.lastIndexOf('}') + 1); + assert.equal(JSON.parse(body).hooks.PreToolUse[0].hooks.length, 1); + }); + + test('a target with no hooks says so rather than printing empty JSON', () => { + const bare = mkdtempSync(path.join(os.tmpdir(), 'mms-nohooks-')); + mkdirSync(path.join(bare, '.claude', 'skills', 'mms-x'), { recursive: true }); + const r = spawnSync(process.execPath, [BIN, 'hooks', '--target', bare], { encoding: 'utf8' }); + assert.equal(r.status, 0); + assert.match(r.stdout, /No installed skill ships a hook/u); + rmSync(bare, { recursive: true, force: true }); + }); +}); diff --git a/tools/install b/tools/install index 2df02e44..816c4dfa 100755 --- a/tools/install +++ b/tools/install @@ -386,6 +386,9 @@ copy_domain_knowledge() { copy_project_bundles() { local skill_dir="$1" out_name="$2" + # A hook is inert until an operator registers it, so remember which skills shipped one + # and print the registration at the end rather than leaving the file to be discovered. + [[ -d "$skill_dir/hooks" ]] && HOOK_SKILLS+=("$out_name") copy_bundle_dirs "$skill_dir" "$CLAUDE_DIR/$out_name" ".claude/skills/$out_name" copy_domain_knowledge "$skill_dir" "$CLAUDE_DIR/$out_name" ".claude/skills/$out_name" copy_bundle_dirs "$skill_dir" "$CURSOR_DIR/$out_name" ".cursor/rules/$out_name" @@ -592,6 +595,7 @@ RESOLVED_KEYS=() RESOLVED_DIRS=() RESOLVED_DOMAINS=() SKIPPED_USER_SKILLS=() +HOOK_SKILLS=() FAILED_SKILLS=() EXPECTED_PROJECT_SKILLS=() @@ -660,6 +664,47 @@ $PRUNE_STALE && remove_stale_project_skills echo $DRY_RUN && echo "Dry run complete. No files written." || echo "Install complete." +set +u +HOOK_COUNT=${#HOOK_SKILLS[@]} +set -u +if (( HOOK_COUNT > 0 )) && ! $DRY_RUN; then + echo + echo "Note: $HOOK_COUNT skill(s) ship a hook. Copying the file does not activate it —" + echo "Claude Code runs a hook only once it is registered in settings.json." + echo + echo "Add this to ~/.claude/settings.json (or $TARGET/.claude/settings.json):" + echo + + hook_entries=() + for out_name in "${HOOK_SKILLS[@]}"; do + for hook_file in "$CLAUDE_DIR/$out_name/hooks"/*.py; do + [[ -f "$hook_file" ]] || continue + hook_entries+=(" { \"type\": \"command\", \"command\": \"python3 $hook_file\" }") + done + done + + echo ' {' + echo ' "hooks": {' + echo ' "PreToolUse": [' + echo ' {' + echo ' "matcher": "Bash",' + echo ' "hooks": [' + for i in "${!hook_entries[@]}"; do + if (( i < ${#hook_entries[@]} - 1 )); then + echo "${hook_entries[$i]}," + else + echo "${hook_entries[$i]}" + fi + done + echo ' ]' + echo ' }' + echo ' ]' + echo ' }' + echo ' }' + echo + echo "Re-run \`metamask-skills hooks\` to print this again." +fi + set +u SKIPPED_USER_COUNT=${#SKIPPED_USER_SKILLS[@]} set -u From d04bd59932ab5ce10b99a98247b57c83cd320468 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 12:34:08 -0400 Subject: [PATCH 061/135] Restore the `sentry-quota` references MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removed in the previous commit on the claim that no such skill existed. It does — `sentry-quota` ships in the analytics domain. The check that "proved" its absence passed a bare PR number where a ref was required, so every lookup errored into a silenced zero and the skill appeared to exist nowhere. A skill defined in a concurrent pull request is a forward reference that resolves on merge, which is why the reference linter treats it as a warning rather than an error. --- domains/agentic/skills/agent-run-cost/skill.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/domains/agentic/skills/agent-run-cost/skill.md b/domains/agentic/skills/agent-run-cost/skill.md index b37dbc30..7feb8060 100644 --- a/domains/agentic/skills/agent-run-cost/skill.md +++ b/domains/agentic/skills/agent-run-cost/skill.md @@ -18,7 +18,7 @@ Scripted automation announces its cost in wall-clock time; agentic automation do A fan-out of forty subagents and a single call are the same shape in a diff, and the difference surfaces later, on a bill, attributed to nothing in particular. -This is the token-spend counterpart to a span-volume quota guard. Same +This is the token-spend counterpart to `sentry-quota`, which guards span volume. Same posture: operate on **code and PRs**, before the spend exists, and produce figures. ## When to use @@ -101,6 +101,7 @@ Cheapest first; stop at the rung that fits. ## Related +- `sentry-quota` — the same guard for span volume; `fan-out × ungated × no-kill-switch`. - `evidence` — weighs AEP run cost when choosing an evidence lane, and tears the stack down after; this skill is the review-side version for workflows others will run. - [`MetaMask/decisions#173`](https://github.com/MetaMask/decisions/pull/173) — ADR-0058 From 14e670a02b73c3b39c8519de27146e14e7a3342a Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 12:52:45 -0400 Subject: [PATCH 062/135] Add a lane index to the evidence catalog, and fix lane placement MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The catalog is 41 full lane specs with no summary, so a reader arriving from a link lands mid-document with no way to see the shape of it. Adds a generated "Lanes at a glance" table: family, count, and every lane id with its title. Two placement bugs: - `C9` sat inside the `# D. Build output` section, so scanning family C missed the lane backing `memory-leak`, and scanning D found a stranger. - `B7` sat between `B3` and `B4`. All 41 lanes now read in order. Also removes six pointers into a private authoring vault — four inline `exogram` references and two full `exogram-daemon/...` paths. They resolve for no reader of a public repository. Every substantive claim they were attached to is kept; only the dangling pointer is dropped. `memory-leak-hunt` updated to `memory-leak`. --- .../evidence/references/evidence-catalog.md | 48 ++++++++++++------- domains/pr-workflow/skills/evidence/skill.md | 2 +- 2 files changed, 33 insertions(+), 17 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md index ae90d30f..01fa1665 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md @@ -8,6 +8,22 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- --- +## Lanes at a glance + +41 lanes in 7 families. Each lane below has a full spec in its own section — what it proves, how to capture it, and its trust gate. Family G is written as one-liners rather than full sections, because those lanes are links and counts rather than captures. + +| Family | Lanes | | +|---|---|---| +| **A. AEP harness (primary, autonomous)** | 3 | `A1` visual_validation · `A2` perf_validation · `A3` AEP bundle byproducts | +| **B. Behavior & flow proof** | 7 | `B1` Visual before/after via the mm CLI · `B2` E2E trace + video · `B3` Falsifying regression test · `B4` Component / Storybook visual · `B5` Accessibility · `B6` Flaky-stability rerun · `B7` Deterministic interleaving test | +| **C. Performance & render** | 9 | `C1` Startup / custom traces + phase segmentation · `C2` Web vitals · `C3` Long-task / TBT · `C4` React render & selector proof · `C5` Benchmark A/B · `C6` DevTools / CDP profiling · `C7` Memory stability over a flow · `C8` Same-window app + DevTools capture · `C9` Retention-path analysis | +| **D. Build output** | 6 | `D1` Bundle-size diff · `D2` Chunk membership / source-map · `D3` LavaMoat policy / supply-chain capability diff · `D4` Manifest permissions diff · `D5` Build-variant matrix · `D6` Authored-vs-authoritative substitution A/B | +| **E. Production telemetry** | 3 | `E1` Sentry query links · `E2` Tempo distributed traces · `E3` Sentry error-event / breadcrumb shape | +| **F. Extension integrity (high-stakes, extension-specific)** | 8 | `F1` State migration / upgrade · `F2` Vault / keyring round-trip · `F3` Transaction simulation / gas · `F4` Provider / dapp connectivity · `F5` Feature-flag matrix · `F6` Snaps / multichain execution · `F7` i18n usage · `F8` SES lockdown / runtime containment | +| **G. CI, review & process** | 5 | `G1` CI check links · `G2` Coverage delta · `G3` Automated-reviewer output · `G4` Manual reproduction steps · `G5` CI-workflow change, run on a test fork | + +--- + # A. AEP harness (primary, autonomous) ## A1. visual_validation — before/after screenshots @@ -43,12 +59,6 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- - **Capture:** add the test, run it on the PR branch (pass) and on the PR's **merge-base** (fail) — pin the base, don't use whatever `main` points at today. Pair with the PR's `Fixes #N`. **Read the base failure's message, not its exit code:** it must fail on the assertion that encodes the bug. A `ModuleNotFoundError`, a missing fixture, or an unrelated pre-existing red produces an identical non-zero exit and falsifies nothing. - **Reach for it:** every bug-fix PR. If you can't write a test that fails on main, question whether the fix addresses the reported bug. -## B7. Deterministic interleaving test (concurrency / temporal-ordering) ⭐ -- **Engine:** `race-condition-repro` — run it rather than hand-rolling the harness. -- **Proves:** an ordering guarantee under interleaving — retry, cancellation, supersession, debounce, locks, queues, async state machines — where the correctness *is* the ordering under races, not a value. Full category: `exogram-daemon/artifacts/evidence-taxonomy/category-concurrency-temporal-ordering.md`. -- **Capture:** force each race deterministically — `jest.useFakeTimers()` + `advanceTimersByTimeAsync(DELAY)` to fire the delayed action at a known point; `Promise.all([opA, opB])` to overlap operations; `advanceTimersByTimeAsync(0)` to step to a precise interleaving point; then assert the ordering/cancellation outcome for **each** guarantee, including asymmetric ones (one path canceled → its recovery event `.not.toHaveBeenCalled()`; another must complete → `.toHaveBeenCalledWith(...)`). Corroborate with transition telemetry; for the integration path, a live forced-race capture (C8/CDP, the #44610 technique). -- **Trust-gate:** the test must **actually interleave** — time advanced into the pending window, the superseding op injected *during* it. A sequential run exercises no race and is a vacuous green. Verify the interleaving, not just the assertion. - ## B4. Component / Storybook visual - **Proves:** a component renders across states/props in isolation. - **Capture:** `.storybook/` present; `yarn storybook` (port 6006), `yarn storybook:build`, `yarn test-storybook` (visual + a11y via `@storybook/addon-a11y`). Jest snapshot diffs for serialized output. @@ -64,6 +74,12 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- --- +## B7. Deterministic interleaving test (concurrency / temporal-ordering) ⭐ +- **Engine:** `race-condition-repro` — run it rather than hand-rolling the harness. +- **Proves:** an ordering guarantee under interleaving — retry, cancellation, supersession, debounce, locks, queues, async state machines — where the correctness *is* the ordering under races, not a value. +- **Capture:** force each race deterministically — `jest.useFakeTimers()` + `advanceTimersByTimeAsync(DELAY)` to fire the delayed action at a known point; `Promise.all([opA, opB])` to overlap operations; `advanceTimersByTimeAsync(0)` to step to a precise interleaving point; then assert the ordering/cancellation outcome for **each** guarantee, including asymmetric ones (one path canceled → its recovery event `.not.toHaveBeenCalled()`; another must complete → `.toHaveBeenCalledWith(...)`). Corroborate with transition telemetry; for the integration path, a live forced-race capture (C8/CDP, the #44610 technique). +- **Trust-gate:** the test must **actually interleave** — time advanced into the pending window, the superseding op injected *during* it. A sequential run exercises no race and is a vacuous green. Verify the interleaving, not just the assertion. + # C. Performance & render ## C1. Startup / custom traces + phase segmentation @@ -73,7 +89,7 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- ## C2. Web vitals — INP / FCP / LCP / CLS - **Proves:** a user-centric metric moved. `ui/helpers/utils/web-vitals.ts` via `web-vitals/attribution` (attribution names the causing element). - **Capture:** `window.stateHooks.getWebVitalsMetrics()` (test/debug) → `{inp, fcp, lcp, cls, *Rating}`. Thresholds: INP good<200/poor>500, FCP<1800/3000, LCP<2500/4000, CLS<0.1/0.25. -- **Caveat:** **INP fires on all pages; FCP/LCP/CLS do not fire on popup pages** (sidepanel/E2E only). For extensions, INP is the high-value runtime metric (per exogram `web-vitals-runtime-metrics`). +- **Caveat:** **INP fires on all pages; FCP/LCP/CLS do not fire on popup pages** (sidepanel/E2E only). For extensions, INP is the high-value runtime metric. ## C3. Long-task / TBT - **Proves:** main-thread blocking during an interaction dropped. This is where **TBT** lives (the web-vitals lib lane does *not* collect TBT). @@ -81,16 +97,16 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- ## C4. React render & selector proof - **Engine: the `react-render-delta` skill.** Delegate the measurement to it; it runs the source/delivery/metric gates, derives the needle from real build output, repeats the capture, and returns a band (or "not resolvable at this n" with an MDE). evidence packages the result. -- **Proves:** a component/selector stopped over-rendering (cascade-amplification before/after — exogram `react-redux-performance`). +- **Proves:** a component/selector stopped over-rendering (cascade-amplification before/after). - **Capture:** WDYR via `ENABLE_WHY_DID_YOU_RENDER` (`.metamaskrc` or env) — wired in `app/scripts/development/wdyr.ts` (`trackAllPureComponents`); console logs each unnecessary re-render. `yarn devtools:react` for the Profiler flame graph. Selectors use `reselect`'s `createSelector`, which **does expose a real `.recomputations()` counter** — read it (sample on an interval if the count should visibly climb) rather than injecting a log into the selector body; an injected log is an authored claim, a library API is an observation. *(This entry previously said there was no built-in counter. There is.)* - **Bar:** the delivery check comes before the number. An arm whose manipulation cannot be observed in the built bundle produces a null indistinguishable from "small effect" — and reports as the second. ## C5. Benchmark A/B - **Proves:** a startup/journey/interaction timing moved, with a distribution not one sample. - **Capture:** `yarn test:e2e:benchmark` (`test/e2e/benchmarks/run-benchmark.ts`); presets in `shared/constants/benchmarks.ts` (`startupStandardHome`, `sendTransactions`, `swap`, `dappPageLoad`, …). -- **Caveat:** the rolling baseline (`MetaMask/extension_benchmark_stats`) can **silently freeze** behind a green check (the `store-benchmark-stats` step is `continue-on-error`; happened 2026-04-02, PR #42947). Prefer a **paired A/B** (build both refs now, compare directly) over the stored baseline. See exogram `benchmark-baseline-staleness-paired-ab`. +- **Caveat:** the rolling baseline (`MetaMask/extension_benchmark_stats`) can **silently freeze** behind a green check (the `store-benchmark-stats` step is `continue-on-error`; happened 2026-04-02, PR #42947). Prefer a **paired A/B** (build both refs now, compare directly) over the stored baseline. - **Treatment check first** — before trusting any delta, confirm the mechanism under test is actually active in each arm (split chunk present in head and absent in base; the span emitted; the flag evaluated). An arm without the treatment delivered is a no-op, not a control (2026-07-22, #42795). -- **A null needs its power stated** — "no change" and "underpowered" print the same result. When the run-to-run spread exceeds the effect under test, report **not resolvable at this n** and name the smallest detectable effect; never let it read as "no effect". Correcting a known bias (discarding a warm-up, alternating the starting arm) removes *that* bias and nothing more — it is not a trust gate, and the confounds you did not enumerate (thermal drift, background load, ordering within a round) stay live. See exogram `removing-a-bias-is-not-establishing-validity` (2026-07-24). +- **A null needs its power stated** — "no change" and "underpowered" print the same result. When the run-to-run spread exceeds the effect under test, report **not resolvable at this n** and name the smallest detectable effect; never let it read as "no effect". Correcting a known bias (discarding a warm-up, alternating the starting arm) removes *that* bias and nothing more — it is not a trust gate, and the confounds you did not enumerate (thermal drift, background load, ordering within a round) stay live. ### Capturing an authenticated view (the in-situ requirement) @@ -134,16 +150,16 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ --- -# D. Build output - ## C9. Retention-path analysis — memory leak from code ⭐ *(static; lead for leak claims)* -- **Engine: the `memory-leak-hunt` skill.** For a memory-leak claim, delegate the analysis to `memory-leak-hunt` — it runs Phase-1 static pairing (and Phase-2 heap investigation if a primitive can't be paired) and returns the paired/unpaired sites + verdict. evidence keeps **memory leak** as the evidence category: it invokes the skill on the diff and packages the result (in-situ scan capture, plus the lifecycle test / retainer graph if Phase 2 ran) as the category's evidence. The lane spec below is the method that skill implements. -- **Proves:** "X is retained past its lifecycle boundary" / "collection Y grows unboundedly" — argued from code, no runtime needed. This is the lane that works at **review time** (does this PR *introduce* retention?) and leads fix-side validation (does the fix *break* the retention path?). C7 is the runtime corroborator, not the lead — leaks need many cycles to exceed noise. Full category: `exogram-daemon/artifacts/evidence-taxonomy/category-memory-retention-from-code.md`. +- **Engine: the `memory-leak` skill.** For a memory-leak claim, delegate the analysis to `memory-leak` — it runs Phase-1 static pairing (and Phase-2 heap investigation if a primitive can't be paired) and returns the paired/unpaired sites + verdict. evidence keeps **memory leak** as the evidence category: it invokes the skill on the diff and packages the result (in-situ scan capture, plus the lifecycle test / retainer graph if Phase 2 ran) as the category's evidence. The lane spec below is the method that skill implements. +- **Proves:** "X is retained past its lifecycle boundary" / "collection Y grows unboundedly" — argued from code, no runtime needed. This is the lane that works at **review time** (does this PR *introduce* retention?) and leads fix-side validation (does the fix *break* the retention path?). C7 is the runtime corroborator, not the lead — leaks need many cycles to exceed noise. - **Capture — the holder → held → boundary triple, per suspect:** (1) the **holder** (listener, closure, module singleton, accumulating collection, timer); (2) the **held set** — the *specific* objects pinned (list the closure's captures; note when a closure links two objects' GC); (3) the **outlived boundary** (`destroy()`, stream close, instance replacement, request completion). Method: **pair every acquire with its release site** (`on`↔`removeListener`, push↔drain, assign↔null) — the absence of the pair, cited at the acquire site, IS the finding. Four canonical shapes: unbounded accumulator (defeated guard, no drain) · stale-instance listeners on replacement · unremoved listener + capture set · retention past `destroy()`. - **Scope to the diff, or you invent findings.** Classify every flagged primitive as *introduced by this PR* (in the added lines) vs *pre-existing* (already in the file). Charge only the introduced ones to the PR; report pre-existing un-paired primitives separately and uncharged. On extension#40684 the two new stream listeners each had a `removeListener` on `onStreamClosed` (the exact fix a reviewer suggested) and the new pending-request Map had its `.delete` — no leak introduced — while three pre-existing un-torn-down listeners were surfaced but left uncharged, matching how the human/bot reviewers treated them in-thread. This lane *is* the retention review automated; a heap snapshot (C7) is warranted only for an introduced primitive it cannot pair. - **Corroborate:** a falsifying lifecycle test (force the boundary, assert release — listener count zero, singleton nulled, collection drained); C7 heap-over-flow with the **retainer graph naming the same path** the static argument named. - **Trust-gate:** the triple must be specific ("this listener holds `patchStore` after `patchStore.destroy()`", not "might leak"); distinguish **bounded staleness vs unbounded growth** (severity differs); attribute **introduced vs pre-existing** honestly. +# D. Build output + ## D1. Bundle-size diff - **Proves:** the build grew/shrank by a measured amount. Use the bundle-size CI output or a local build size comparison. @@ -152,7 +168,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ ## D3. LavaMoat policy / supply-chain capability diff - **Engine: the `supply-chain-audit` skill** (umbrella — lockfile/manifest diff, advisories, Socket Security, install scripts), which delegates capability grants to **`lavamoat-policy-diligence`** (per-grant call-site justification). Delegate the dependency change to the umbrella; it returns a disposition per lane. evidence keeps **supply-chain capability diff** as the evidence category and packages the output. Note the lanes are independent: a clean policy diff does not mean a safe dependency, and a known CVE never appears as a new grant. -- **Proves:** a dependency change (bump/add/lockfile) grants **no *unjustified* new capability** — the supply-chain-risk lane. Note the bar: for a bump the policy *will* change, so "empty diff" is the WRONG test; the right test is **every new grant is justified by the dep's function**. Full category (trust-boundary framing, generalizes past LavaMoat to any capability-containment mechanism): `exogram-daemon/artifacts/evidence-taxonomy/category-supply-chain-capability-diff.md`. +- **Proves:** a dependency change (bump/add/lockfile) grants **no *unjustified* new capability** — the supply-chain-risk lane. Note the bar: for a bump the policy *will* change, so "empty diff" is the WRONG test; the right test is **every new grant is justified by the dep's function**.. - **Capture:** **Prefer the CI-generated policy whenever one is available.** `@metamaskbot update-policies` regenerates the policy files from a real run of the code and `validate-lavamoat-policies` fails the build on drift, so the committed policy on a bot-run PR *is* the authoritative artifact — diff that. Regenerating locally when a current CI policy exists only re-does a machine that is already trusted, and a local run's provenance is weaker (your node/OS/lockfile resolution, not CI's). **Local regen is the fallback**, for when the bot hasn't run yet, the branch is unpushed, or you need a variant CI didn't cover: `yarn webpack:lavamoat:policy:build` (`:mv2` / `:mv3` for variants) over `lavamoat/webpack/build/policy.json` (+ `policy-override.json`). Either way, `git diff` the policy across **all 8 variants** (mv{2,3}/{main,beta,flask,experimental}) — a grant can appear in one and not others. Then audit **grant-by-grant**: new **globals** (`fetch`, `importScripts`, `WebAssembly`) / **builtins** (`fs`, `child_process`) on a dep that shouldn't need them, new **packages** edges to powerful APIs, or an identifier substitution (`pkgC>name` replacing `pkgB>pkgA>name` = possible dep swap). Falsifier = a surprising grant ("I wonder what it's using this for"). Guide: lavamoat.github.io/guides/policy-diff/. `allowScripts` in `package.json` gates install scripts. ## D4. Manifest permissions diff @@ -244,7 +260,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **G2. Coverage delta** — `yarn test:unit:coverage` → `coverage/unit/` (and `yarn test:unit:webpack:coverage`); `codecov.yml`. Proves the new code is exercised. - **G3. Automated-reviewer output** — independent bot (e.g. cursor[bot]) found nothing blocking. Complements, never replaces, behavior evidence. - **G4. Manual reproduction steps** — human-followable steps that reproduce the fixed behavior; populates the PR template's Manual testing steps. -- **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-<X>=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: push to a branch literally named **`main`** (or `stable`) on `consensys-test/metamask-extension-test-majorlift` — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets on the fork (`INFURA_PROJECT_ID`, `TEST_SRP_*` for benchmarks; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo. Detail: `exogram-daemon/memory/ci-workflow-pr-self-validation-gap.md`. +- **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-<X>=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: push to a branch literally named **`main`** (or `stable`) on `consensys-test/metamask-extension-test-majorlift` — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets on the fork (`INFURA_PROJECT_ID`, `TEST_SRP_*` for benchmarks; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo.. --- diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 55a585ed..916fb6e6 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -234,7 +234,7 @@ Where evidence sits in the PR lifecycle (see the public `pr-workflow` siblings): - **Executes, with a confirmation gate on publish.** It runs the harness and captures evidence autonomously; it does not write to the public PR body without showing you the section first. - **Local-only AEP.** No hosted instance. The skill drives the local stack. - **Proves behavior, not code.** Pair with `/review` / `/code-review` for correctness and `/sentry-quota` for span-volume risk. -- **No persisted state.** Each run is fresh. To keep a validation record, ask — it can go to `exogram-daemon/`, but nothing writes by default. +- **No persisted state.** Each run is fresh. To keep a validation record, ask — nothing writes by default. ## Related From eceaf37d81b128981cda563f651c43201b316571 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 14:11:43 -0400 Subject: [PATCH 063/135] Add build-duration lanes `D7` and `G6` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The catalog had 41 lanes and none for how long a build takes. Family D covered build *output* — size, chunks, policy, permissions, variants — and `C5` covers runtime, so a toolchain change had no category to publish into even though a skill for measuring one is specified in #102. `D7` is the dev-loop half: paired A/B, cold and warm as separate numbers, with the four confounds that each return a favourable result when uncontrolled — warm cache leaking into the cold arm, worker-pool startup amortised away, core count that does not transfer off the measuring machine, and watch rebuilds presented as cold builds. `G6` is the CI half, in family G because its dominant confound is a process one: `get-requirements.yml` skips jobs when build output matches base, so a measured speedup is often a skipped job. Family D is retitled from "Build output" to "Build", since it now covers both. The at-a-glance index is regenerated rather than hand-patched — it is derived from the headings, and hand-editing it is how it drifts. --- .../evidence/references/evidence-catalog.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md index 01fa1665..728aae6f 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md @@ -10,18 +10,17 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- ## Lanes at a glance -41 lanes in 7 families. Each lane below has a full spec in its own section — what it proves, how to capture it, and its trust gate. Family G is written as one-liners rather than full sections, because those lanes are links and counts rather than captures. +43 lanes in 7 families. Each lane below has a full spec in its own section — what it proves, how to capture it, and its trust gate. Family G is written as one-liners rather than full sections, because those lanes are links and counts rather than captures. | Family | Lanes | | |---|---|---| | **A. AEP harness (primary, autonomous)** | 3 | `A1` visual_validation · `A2` perf_validation · `A3` AEP bundle byproducts | | **B. Behavior & flow proof** | 7 | `B1` Visual before/after via the mm CLI · `B2` E2E trace + video · `B3` Falsifying regression test · `B4` Component / Storybook visual · `B5` Accessibility · `B6` Flaky-stability rerun · `B7` Deterministic interleaving test | | **C. Performance & render** | 9 | `C1` Startup / custom traces + phase segmentation · `C2` Web vitals · `C3` Long-task / TBT · `C4` React render & selector proof · `C5` Benchmark A/B · `C6` DevTools / CDP profiling · `C7` Memory stability over a flow · `C8` Same-window app + DevTools capture · `C9` Retention-path analysis | -| **D. Build output** | 6 | `D1` Bundle-size diff · `D2` Chunk membership / source-map · `D3` LavaMoat policy / supply-chain capability diff · `D4` Manifest permissions diff · `D5` Build-variant matrix · `D6` Authored-vs-authoritative substitution A/B | +| **D. Build** | 7 | `D1` Bundle-size diff · `D2` Chunk membership / source-map · `D3` LavaMoat policy / supply-chain capability diff · `D4` Manifest permissions diff · `D5` Build-variant matrix · `D6` Authored-vs-authoritative substitution A/B · `D7` Build & rebuild duration A/B | | **E. Production telemetry** | 3 | `E1` Sentry query links · `E2` Tempo distributed traces · `E3` Sentry error-event / breadcrumb shape | | **F. Extension integrity (high-stakes, extension-specific)** | 8 | `F1` State migration / upgrade · `F2` Vault / keyring round-trip · `F3` Transaction simulation / gas · `F4` Provider / dapp connectivity · `F5` Feature-flag matrix · `F6` Snaps / multichain execution · `F7` i18n usage · `F8` SES lockdown / runtime containment | -| **G. CI, review & process** | 5 | `G1` CI check links · `G2` Coverage delta · `G3` Automated-reviewer output · `G4` Manual reproduction steps · `G5` CI-workflow change, run on a test fork | - +| **G. CI, review & process** | 6 | `G1` CI check links · `G2` Coverage delta · `G3` Automated-reviewer output · `G4` Manual reproduction steps · `G5` CI-workflow change, run on a test fork · `G6` CI job-duration delta | --- # A. AEP harness (primary, autonomous) @@ -158,7 +157,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **Corroborate:** a falsifying lifecycle test (force the boundary, assert release — listener count zero, singleton nulled, collection drained); C7 heap-over-flow with the **retainer graph naming the same path** the static argument named. - **Trust-gate:** the triple must be specific ("this listener holds `patchStore` after `patchStore.destroy()`", not "might leak"); distinguish **bounded staleness vs unbounded growth** (severity differs); attribute **introduced vs pre-existing** honestly. -# D. Build output +# D. Build ## D1. Bundle-size diff - **Proves:** the build grew/shrank by a measured amount. Use the bundle-size CI output or a local build size comparison. @@ -200,6 +199,14 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ --- +## D7. Build & rebuild duration A/B *(paired; lead for toolchain-change claims)* +- **Proves:** what a toolchain change costs or saves in the **dev loop** — a loader, transform, linter, or bundler swap. Distinct from `C5`, which times the shipped app at runtime; this times the build that produces it. The two move independently and in opposite directions often enough that measuring one and inferring the other is the failure this lane exists to prevent (`React Compiler` builds slower and runs faster; `thread-loader` builds faster and runs identically). +- **Shape:** paired A/B, both arms built now, on one machine, alternating order. **Cold and warm are separate questions and get separate numbers** — never one figure labelled "build time". +- **Capture:** N ≥ 5 per arm per mode, alternating. Cold: clear the cache explicitly between arms (`node_modules/.cache`, webpack `cache.cacheDirectory`) and state what was cleared. Warm: touch one source file, rebuild, discard the first result as pool warmup. Report median **and spread**; a median without spread hides a bimodal cache effect. +- **Falsifiers — each returns a favourable number when uncontrolled:** warm cache leaking into the "cold" arm (the largest confound, and the easiest to introduce by running arms in sequence); worker-pool startup counted once and amortised across rebuilds; core count, since parallel loaders scale with the runner and a laptop result does not transfer; watch-rebuild numbers presented as cold-build numbers. +- **Trust-gate:** state machine, core count, N, and cache handling per arm, or the number is unreproducible. A null result states the smallest effect the sample could have detected — "no difference" from N=3 is not a finding. Renders **no ship verdict**: a change that costs build time and buys runtime is a trade, and pricing it is not the same as taking it. +- **Corroborate:** `G6` for the CI half (different machine, different confounds), `C5` for the runtime half. A toolchain claim is not closed by one surface. + # E. Production telemetry ## E1. Sentry query links (before/after) @@ -261,6 +268,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **G3. Automated-reviewer output** — independent bot (e.g. cursor[bot]) found nothing blocking. Complements, never replaces, behavior evidence. - **G4. Manual reproduction steps** — human-followable steps that reproduce the fixed behavior; populates the PR template's Manual testing steps. - **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-<X>=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: push to a branch literally named **`main`** (or `stable`) on `consensys-test/metamask-extension-test-majorlift` — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets on the fork (`INFURA_PROJECT_ID`, `TEST_SRP_*` for benchmarks; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo.. +- **G6. CI job-duration delta** — compare job wall-clock across arms in the Actions UI or `gh run view`. **Falsifier: build reuse.** `get-requirements.yml` skips jobs when build output matches base, so a measured "speedup" is often a skipped job — confirm each arm actually ran the work before comparing. Runner class and queue time vary independently of the change; report job time, not wall-clock from push. Pairs with `D7`, which measures the same change on a machine you control. --- From 7b3eeaa04a60a739306621ed417cba5407c1f95d Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 20:06:07 -0400 Subject: [PATCH 064/135] =?UTF-8?q?Widen=20the=20=C2=A73=20grep=20to=20cat?= =?UTF-8?q?ch=20result=20functions=20returning=20fresh=20literals?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by running the skill against real merged PRs: the §3 detection matched only named collection constructors, so a result function returning an object literal directly went undetected. `(metamask) => ({ userRegion: ..., ... })` builds a new object on every recompute and matches none of `new Set`, `new Map`, `Object.values`, `?? {}`, or `?? []`. Adds `=> ({` and `=> [` as alternates, with the reason recorded beside the table so the next person does not narrow it again. --- domains/performance/skills/selector-antipattern-scan/skill.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/domains/performance/skills/selector-antipattern-scan/skill.md b/domains/performance/skills/selector-antipattern-scan/skill.md index be3b81fe..ee32d7d7 100644 --- a/domains/performance/skills/selector-antipattern-scan/skill.md +++ b/domains/performance/skills/selector-antipattern-scan/skill.md @@ -50,13 +50,15 @@ Both `metamask-extension` and `metamask-mobile` share the same React + Redux arc |---|---| | §1 Unmemoized selector | `grep -rE 'export function get' <selectors-dir>/` | | §2 Identity / passthrough result | Jest warning `result function returned its own inputs` | -| §3 New collection in the result function | `grep -rnE 'new Set\|new Map\|Object\.(values\|keys\|entries)\|\?\? \{\}\|\?\? \[\]' <selectors-dir>/` | +| §3 New collection in the result function | `grep -rnE 'new Set\|new Map\|Object\.(values\|keys\|entries)\|\?\? \{\}\|\?\? \[\]\|=> \(\{\|=> \[' <selectors-dir>/` | | §4 Mutation in the result function | `grep -rnE '\.sort\(\|\.reverse\(\|\.push\(\|\.splice\(' <selectors-dir>/` | | §5 Over-broad input | `grep -rn 'state) => state\b' <selectors-dir>/` | | §6 Unnecessary deep equality | `grep -rn 'createDeepEqualSelector' <selectors-dir>/` then verify each input is genuinely unstable | | §7 O(n) lookup | `grep -rnE '\.find\(.*=>.*address' <selectors-dir>/` | | §8 Chained unmemoized transforms | `grep -rnE 'export function get.*\{' <selectors-dir>/ -A5`, then look for several `.filter/.map/.sort` without memoization | +The `=> ({` and `=> [` alternates catch a result function that *returns* a fresh literal rather than constructing a named collection. A trial run missed a real instance without them: `(metamask) => ({ userRegion: ..., ... })` builds a new object every recompute and matches none of the collection constructors. + See the repo overlay for the concrete `<selectors-dir>` path. ## Team-Specific Workarounds From 90d70821aa9be06603742624de1b691aafd692c4 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 21:32:58 -0400 Subject: [PATCH 065/135] Inline the publishing non-negotiables, which a real run ignored entirely MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An 18-comment trial run met none of this skill's output requirements. The cause was structural, not behavioural: `VALIDATION_RUN_START` and the in-situ capture rule occurred zero times in skill.md and only in a reference costing ~5x the body to open, described there as "image re-hosting and the privacy scrub". The publish gate checks none of them either. All three layers failed open. Moves six non-negotiables and the canonical output shape into the body, where they load with the work: 1. Ship an artifact the reader can check without trusting you. Pasted terminal text is indistinguishable from invented terminal text — running the check justifies your belief, not the reader's. 2. `proven` requires execution; reading gives shape, never power. Run arm B against your own probe: one that passes with the mechanism deleted is measuring something else. 3. No "what would close it" section — that is an unfinished run formatted to look finished. Imperative-mood prose means the artifact does not exist. 4. Write to the reviewer who arrives, not whoever commissioned the run. 5. Delete findings whose entire content is test quality, unless critical. 6. Route privacy and security findings to the private tracker. Derived from eight postmortems in exogram-core; the reference keeps the full recipe. --- domains/pr-workflow/skills/evidence/skill.md | 53 +++++++++++++++++++- 1 file changed, 52 insertions(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 916fb6e6..d88a6c98 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -145,7 +145,58 @@ Stop when each claim has one trustworthy artifact that would have shown its fals ## Publishing the evidence bundle **Public, outward-facing — always confirm the rendered section with the user before writing -a PR body.** Full recipe, markers, image re-hosting, recordings, and the privacy scrub: +a PR body.** + +### Non-negotiables — these are here, not in a reference, because a requirement you have to fetch is advisory + +**1. Ship an artifact the reader can check without trusting you.** Terminal text you pasted is +indistinguishable from terminal text you invented; it carries the weight of your assertion, not +of a measurement. Running the check justifies *your* belief. It becomes *evidence* only when the +reader can confirm it independently: a committed test CI executes, a link to a run, a capture with +visual provenance, an artifact at a URL. **If every character of the output is one you typed, you +have published an assertion.** + +**2. `proven` requires execution; reading yields `unverified`.** Reading a test establishes its +shape, never its power. A test is evidence when it *fails* on the base arm — so run arm B, including +against your own probe. A probe that passes with the mechanism deleted is measuring something else. + +**3. There is no "what would close it" section.** If you know what would close the falsifier, close +it. Three legal endings: proven with artifact attached · unproven, stated flatly and nothing +prescribed · an open question that is genuinely a human's product decision. Imperative-mood prose +(*run*, *switch*, *assert*) means the artifact does not exist. + +**4. Write to the reviewer who arrives, not whoever commissioned the run.** They have a stake in +this PR and none in your tooling. Cut calibration rationale, prior hypotheses, and corrections to +drafts they never saw. One line of disclosure that the output is automated and needs no action is +for them; everything explaining why you are running this is not. + +**5. Delete findings whose entire content is test quality** — code correct, test weak — unless the +untested path touches funds, keys, persisted state, user-visible wrongness, or silent corruption. + +**6. Privacy and security findings are routed, never published here.** File them in the private +planning tracker. Subject matter triggers this, not severity: the code cannot distinguish a missing +gate from a deliberate one. + +### Canonical output shape + +Every validation-run output — PR comment *or* PR-body section — uses exactly this, so re-runs +replace idempotently instead of accumulating: + +```markdown +<!-- VALIDATION_RUN_START --> +## 🧪 Validation Run + +**Verdict:** ✅ proven — **Claim:** <one-line falsifiable behavior under test> +head `<sha>` · <YYYY-MM-DD> · lanes: <lane ids> + +<claim → artifact table; every claim binds its artifact> +<!-- VALIDATION_RUN_END --> +``` + +Verdict icons: `✅` proven · `❌` failed · `ℹ️` otherwise. Never `❌` for a gap in *evidence* — that +reads as a verdict on the author's work. + +Full recipe — image re-hosting, recordings, AEP mirroring, the privacy scrub: **[references/evidence-publishing.md](references/evidence-publishing.md).** The parts that decide *whether* to publish, rather than how: From d412caf5a4169203feebff19f55f9ff6193930f8 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 21:39:23 -0400 Subject: [PATCH 066/135] =?UTF-8?q?Add=20`falsify-probe.sh`=20=E2=80=94=20?= =?UTF-8?q?the=20runner=20that=20makes=20a=20lane=20reproducible?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A recipe returns as many answers as it has operators, and any output an operator retypes carries the operator's provenance rather than the measurement's. This ships the mechanism instead. Runs arm A, mutates one line, runs arm B, restores the source, and writes `falsify-<label>.{json,md}` plus both raw logs itself — nothing is transcribed. The exit code is the verdict, so CI gates on it directly: 0 falsifying, 1 vacuous, 2 arm A already failing, 3 usage error. Every artifact pins HEAD, node version, yarn.lock hash, and the tracked-change count, so two operators either produce comparable results or visibly do not. Verified against both outcomes on metamask-extension at 796685ce7b7: the perps coalescing suite reports `falsifying` (10 passed, 2 failed under mutation), and the token-search suite reports `vacuous` (3 passed both arms, so it does not test the abort it appears to test). --- .../skills/evidence/scripts/falsify-probe.sh | 134 ++++++++++++++++++ domains/pr-workflow/skills/evidence/skill.md | 23 +++ 2 files changed, 157 insertions(+) create mode 100755 domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh new file mode 100755 index 00000000..8edd34c7 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -0,0 +1,134 @@ +#!/usr/bin/env bash +# +# falsify-probe — prove a test is falsifying, by mutation rather than by reading. +# +# A test is evidence only if it FAILS when the mechanism it guards is removed. +# Reading the test establishes its shape; only this establishes its power. +# +# Runs two arms against the same tree: +# Arm A baseline — the suite as committed +# Arm B mutant — one line replaced, suite re-run, source restored +# +# Emits a captured artifact (JSON + markdown) written by this script, not +# transcribed by an operator. Exit code IS the verdict, so CI can gate on it. +# +# 0 falsifying arm A passed, arm B failed → the test has power +# 1 vacuous arm A passed, arm B ALSO passed → the test proves nothing +# 2 broken arm A failed → nothing to conclude +# 3 usage/env error +# +# Usage: +# falsify-probe.sh --test <path> --source <path> --line <n> --replace <text> +# [--label <slug>] [--out <dir>] [--runner "<cmd>"] +# +# Example: +# falsify-probe.sh \ +# --test ui/hooks/perps/coalesceBackgroundRequest.test.ts \ +# --source ui/hooks/perps/coalesceBackgroundRequest.ts \ +# --line 54 --replace ' const existing = undefined as Promise<TResult> | undefined;' \ +# --label coalesce-inflight +set -uo pipefail + +RUNNER="yarn jest" +OUT_DIR="evidence-artifacts" +LABEL="" +TEST="" SOURCE="" LINE="" REPLACE="" + +die() { printf 'falsify-probe: %s\n' "$1" >&2; exit 3; } + +while [ $# -gt 0 ]; do + case "$1" in + --test) TEST="${2:-}"; shift 2 ;; + --source) SOURCE="${2:-}"; shift 2 ;; + --line) LINE="${2:-}"; shift 2 ;; + --replace) REPLACE="${2:-}"; shift 2 ;; + --label) LABEL="${2:-}"; shift 2 ;; + --out) OUT_DIR="${2:-}"; shift 2 ;; + --runner) RUNNER="${2:-}"; shift 2 ;; + -h|--help) sed -n '2,32p' "$0"; exit 0 ;; + *) die "unknown argument: $1" ;; + esac +done + +[ -n "$TEST" ] || die "--test is required" +[ -n "$SOURCE" ] || die "--source is required" +[ -n "$LINE" ] || die "--line is required" +[ -n "$REPLACE" ] || die "--replace is required (use '' only if deleting the line)" +[ -f "$TEST" ] || die "test not found: $TEST" +[ -f "$SOURCE" ] || die "source not found: $SOURCE" +case "$LINE" in ''|*[!0-9]*) die "--line must be numeric: $LINE" ;; esac +[ "$LINE" -le "$(wc -l < "$SOURCE")" ] || die "--line $LINE is past the end of $SOURCE" + +LABEL="${LABEL:-$(basename "$SOURCE" | sed 's/\.[^.]*$//')-L$LINE}" +mkdir -p "$OUT_DIR" || die "cannot create $OUT_DIR" +STAMP="$OUT_DIR/falsify-$LABEL" + +# --- environment pin: two operators on different machines must be comparable --- +HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" +DIRTY="$(git status --porcelain 2>/dev/null | grep -v '^??' | wc -l | tr -d ' ')" +NODE_V="$(node -v 2>/dev/null || echo unknown)" +LOCK_SHA="$( { sha256sum yarn.lock 2>/dev/null || shasum -a 256 yarn.lock 2>/dev/null; } | cut -c1-16)" +ORIGINAL_LINE="$(sed -n "${LINE}p" "$SOURCE")" + +BACKUP="$(mktemp)" || die "mktemp failed" +cp "$SOURCE" "$BACKUP" +restore() { cp "$BACKUP" "$SOURCE"; rm -f "$BACKUP"; } +trap restore EXIT INT TERM + +run_arm() { # $1=logfile ; prints "passed|failed" + if $RUNNER "$TEST" > "$1" 2>&1; then echo passed; else echo failed; fi +} + +ARM_A="$(run_arm "$STAMP-armA.log")" + +if [ "$ARM_A" != "passed" ]; then + VERDICT="broken"; CODE=2; ARM_B="not-run" + : > "$STAMP-armB.log" +else + # Mutate exactly one line. `.bak` form keeps this portable across GNU/BSD sed. + awk -v n="$LINE" -v r="$REPLACE" 'NR==n{print r; next}{print}' "$SOURCE" > "$SOURCE.tmp" \ + && mv "$SOURCE.tmp" "$SOURCE" || die "mutation failed" + ARM_B="$(run_arm "$STAMP-armB.log")" + restore; trap - EXIT INT TERM + if [ "$ARM_B" = "failed" ]; then VERDICT="falsifying"; CODE=0; else VERDICT="vacuous"; CODE=1; fi +fi + +summarise() { grep -E '^(Tests|Test Suites):' "$1" 2>/dev/null | tr '\n' ' ' | sed 's/ */ /g'; } +A_SUM="$(summarise "$STAMP-armA.log")" +B_SUM="$(summarise "$STAMP-armB.log")" +FAILED_NAMES="$(grep -E '^\s+●[^›]*›' "$STAMP-armB.log" 2>/dev/null | sed 's/^ *//' | head -10)" + +cat > "$STAMP.json" <<JSON +{ + "verdict": "$VERDICT", + "exit": $CODE, + "test": "$TEST", + "mutation": { "source": "$SOURCE", "line": $LINE, + "from": $(printf '%s' "$ORIGINAL_LINE" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))'), + "to": $(printf '%s' "$REPLACE" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))') }, + "armA": { "result": "$ARM_A", "summary": "$A_SUM", "log": "$STAMP-armA.log" }, + "armB": { "result": "$ARM_B", "summary": "$B_SUM", "log": "$STAMP-armB.log" }, + "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, "node": "$NODE_V", "yarn_lock_sha256_16": "$LOCK_SHA" } +} +JSON + +{ + echo "### Falsification probe — \`$VERDICT\`" + echo + echo "| Arm | Mutation | Result |" + echo "|---|---|---|" + echo "| A — baseline | none | \`$A_SUM\` |" + echo "| B — mutant | \`$SOURCE:$LINE\` replaced | \`$B_SUM\` |" + echo + case "$VERDICT" in + falsifying) echo "The suite **fails when the mechanism is removed** and passes when restored. The test has power." ;; + vacuous) echo "The suite **passes with the mechanism removed**. It does not test what it appears to test." ;; + broken) echo "Arm A did not pass, so arm B was not run. No conclusion." ;; + esac + [ -n "$FAILED_NAMES" ] && { echo; echo "Failing under mutation:"; echo; printf '%s\n' "$FAILED_NAMES" | sed 's/^/- /'; } + echo + echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" +} > "$STAMP.md" + +printf 'falsify-probe: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 +exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index d88a6c98..1877eb77 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -177,6 +177,29 @@ untested path touches funds, keys, persisted state, user-visible wrongness, or s planning tracker. Subject matter triggers this, not severity: the code cannot distinguish a missing gate from a deliberate one. +### The runner, not the recipe + +`scripts/falsify-probe.sh` proves a test is falsifying by mutation rather than by reading, and +**writes the artifact itself** — the operator never transcribes output: + +```bash +scripts/falsify-probe.sh \ + --test ui/hooks/perps/coalesceBackgroundRequest.test.ts \ + --source ui/hooks/perps/coalesceBackgroundRequest.ts \ + --line 54 --replace ' const existing = undefined as Promise<TResult> | undefined;' +``` + +Runs arm A, mutates one line, runs arm B, restores the source, and emits +`evidence-artifacts/falsify-<label>.{json,md}` plus both raw logs. **The exit code is the +verdict**, so CI can gate on it: `0` falsifying · `1` vacuous · `2` arm A already failing · `3` +usage error. + +Every artifact pins `HEAD`, node version, `yarn.lock` hash, and the tracked-change count, so two +operators on different machines produce comparable results or visibly do not. + +Prefer this over a hand-run test in every case. A hand-run test yields a number you then retype, +which returns the provenance to you and reintroduces exactly the problem the probe solves. + ### Canonical output shape Every validation-run output — PR comment *or* PR-body section — uses exactly this, so re-runs From 559b4c074a401fb52dd7421f779f9ce6ef7ec2d4 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 21:48:31 -0400 Subject: [PATCH 067/135] Add `capture.sh` so the C9 and D3 analyses stop needing an operator to retype them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `retention-scan.py` and `policy-audit.py` already do the analysis well; both print to stdout, which makes the operator the capture device and returns provenance to whoever pasted the output. This wraps any command so the tool writes the artifact. Emits <label>.log verbatim, plus .json and an attachable .md that quotes the log rather than summarising it, with HEAD, tracked-change count, node, python, and yarn.lock hash pinned in each. The wrapped exit code passes through for CI. `--verdict` is stated by the caller, never inferred from the exit code. The first run of this script proved why: it labelled a policy audit "pass" while the output listed sixteen newly granted capabilities, because policy-audit.py exits 0 regardless. With no --verdict it now says "ran to completion, no verdict asserted". Verified on both scripts against real PRs, including a deliberately wrong invocation — which produces an artifact containing the traceback rather than a fabricated finding. --- .../skills/evidence/scripts/capture.sh | 112 ++++++++++++++++++ domains/pr-workflow/skills/evidence/skill.md | 23 ++++ 2 files changed, 135 insertions(+) create mode 100755 domains/pr-workflow/skills/evidence/scripts/capture.sh diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh new file mode 100755 index 00000000..064ed404 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -0,0 +1,112 @@ +#!/usr/bin/env bash +# +# capture — turn any analysis command into a contract-compliant evidence artifact. +# +# The analysis scripts in this repo (retention-scan.py, policy-audit.py, a jest +# probe, a selector recomputation counter) all print to stdout. Printing to stdout +# means the operator is the capture device: they read it, retype some of it into a +# comment, and the result carries their provenance rather than the measurement's. +# +# This wraps any command so the ARTIFACT is written by the tool. Nothing is retyped. +# +# capture.sh --label <slug> --lane <id> --claim "<under test>" [--verdict <word>] -- <cmd...> +# +# --verdict is stated by the caller, never inferred from the exit code: a wrapped +# tool's exit convention is its own, and guessing prints "pass" over real findings. +# +# Emits, under --out (default evidence-artifacts/): +# <label>.log raw stdout+stderr of the command, unmodified +# <label>.json machine-readable: verdict, exit code, env pin, claim +# <label>.md the block to attach, quoting the log rather than summarising it +# +# Exit code is the wrapped command's own, so CI gates on it unchanged. +# +# Example: +# capture.sh --label defi-retention --lane C9 \ +# --claim "every retention primitive this diff introduces is released" \ +# -- python3 retention-scan.py ui/store/background-connection.ts pr.patch +set -uo pipefail + +OUT_DIR="evidence-artifacts"; LABEL=""; LANE=""; CLAIM=""; MAXLOG=120; VERDICT="" +die() { printf 'capture: %s\n' "$1" >&2; exit 3; } + +while [ $# -gt 0 ]; do + case "$1" in + --label) LABEL="${2:-}"; shift 2 ;; + --lane) LANE="${2:-}"; shift 2 ;; + --claim) CLAIM="${2:-}"; shift 2 ;; + --verdict) VERDICT="${2:-}"; shift 2 ;; + --out) OUT_DIR="${2:-}"; shift 2 ;; + --max-log-lines) MAXLOG="${2:-}"; shift 2 ;; + -h|--help) sed -n '2,26p' "$0"; exit 0 ;; + --) shift; break ;; + *) die "unknown argument: $1 (did you forget -- before the command?)" ;; + esac +done + +[ -n "$LABEL" ] || die "--label is required" +[ -n "$CLAIM" ] || die "--claim is required: name the falsifiable thing under test" +[ $# -gt 0 ] || die "no command given after --" + +mkdir -p "$OUT_DIR" || die "cannot create $OUT_DIR" +STAMP="$OUT_DIR/$LABEL" + +HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" +DIRTY="$(git status --porcelain 2>/dev/null | grep -v '^??' | wc -l | tr -d ' ')" +NODE_V="$(node -v 2>/dev/null || echo n/a)" +PY_V="$(python3 -V 2>&1 || echo n/a)" +LOCK_SHA="$( { sha256sum yarn.lock 2>/dev/null || shasum -a 256 yarn.lock 2>/dev/null; } | cut -c1-16)" +[ -n "$LOCK_SHA" ] || LOCK_SHA="n/a" +CMD_STR="$*" + +# Run it. Never interpret the output — capture it verbatim. +"$@" > "$STAMP.log" 2>&1 +CODE=$? + +LINES="$(wc -l < "$STAMP.log" | tr -d ' ')" +# No verdict is inferred from the exit code. A wrapped tool's convention is its own — +# policy-audit.py exits 0 while listing sixteen new capability grants, so guessing here +# would print "pass" over a page of findings. The caller states the verdict or none is claimed. +[ -n "$VERDICT" ] || VERDICT="completed" + +jstr() { printf '%s' "${1-}" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))'; } + +cat > "$STAMP.json" <<JSON +{ + "label": $(jstr "$LABEL"), + "lane": $(jstr "$LANE"), + "claim": $(jstr "$CLAIM"), + "command": $(jstr "$CMD_STR"), + "verdict": "$VERDICT", + "exit": $CODE, + "log": "$STAMP.log", + "log_lines": $LINES, + "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, + "node": "$NODE_V", "python": "$PY_V", "yarn_lock_sha256_16": "$LOCK_SHA" } +} +JSON + +{ + if [ "$VERDICT" = "completed" ]; then + echo "### ${LANE:+$LANE — }ran to completion (exit $CODE) — read the output, no verdict asserted" + else + echo "### ${LANE:+$LANE — }\`$VERDICT\` (exit $CODE)" + fi + echo + echo "**Claim under test:** $CLAIM" + echo + echo '```console' + echo "\$ $CMD_STR" + if [ "$LINES" -gt "$MAXLOG" ]; then + head -n "$MAXLOG" "$STAMP.log" + echo "… $((LINES - MAXLOG)) further lines in $STAMP.log" + else + cat "$STAMP.log" + fi + echo '```' + echo + echo "<sub>Captured by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. Raw log: \`$STAMP.log\`.</sub>" +} > "$STAMP.md" + +printf 'capture: %s (exit %s)\n %s\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.log" "$STAMP.json" "$STAMP.md" >&2 +exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 1877eb77..aba3c70f 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -200,6 +200,29 @@ operators on different machines produce comparable results or visibly do not. Prefer this over a hand-run test in every case. A hand-run test yields a number you then retype, which returns the provenance to you and reintroduces exactly the problem the probe solves. +### `capture.sh` — for every lane that already has an analysis script + +`retention-scan.py` (C9), `policy-audit.py` (D3), and any jest or selector probe all print to +stdout, which makes the operator the capture device. Wrap them instead: + +```bash +scripts/capture.sh --label bgconn-retention --lane "C9 retention-path analysis" \ + --claim "every retention primitive this diff introduces is paired with a release" \ + -- python3 retention-scan.py "ui/store/background-connection.ts:pr.patch" +``` + +Writes `<label>.log` (verbatim), `<label>.json`, and `<label>.md` — the attachable block, quoting +the log rather than summarising it — with `HEAD`, tracked-change count, node, python, and +`yarn.lock` hash pinned in each. The wrapped command's exit code passes through unchanged. + +**`--verdict` is stated by the caller, never inferred from the exit code.** A wrapped tool's exit +convention is its own: `policy-audit.py` exits `0` while listing sixteen newly granted +capabilities, so inferring would print "pass" over a page of findings. With no `--verdict`, the +artifact says *ran to completion — read the output, no verdict asserted*, which is the honest +default. + +A crashing command produces an artifact containing the traceback, not a fabricated result. + ### Canonical output shape Every validation-run output — PR comment *or* PR-body section — uses exactly this, so re-runs From 3a21ee7e440915ac2f3893f10c4875d11af6bdf2 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 22:04:59 -0400 Subject: [PATCH 068/135] Detect named-subscription listeners, which the scanner could not see MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The listener pass matched only `.on('event', handler)` and `.addListener('event', handler)` — a method named exactly `on` or `addListener` with a quoted event name. Any form carrying the event in the method name was invisible. Running it on extension#42823 returned "no retention path INTRODUCED" for a file containing `background.onNotification(routeMessengerEventNotification)` with zero `removeOnNotification` call sites anywhere in `ui/`. A clean verdict over a real unpaired listener is the worst output this script can produce, because it reports what the pattern can see as though it were what is there. Adds `onXxx(handler)`, `subscribe(handler)`, `addEventListener`, and `addXxxListener` forms, each paired against its corresponding release (`removeOnXxx`/`offXxx`, `unsubscribe`, `removeEventListener`, `removeXxx`). The same file now reports the primitive as NEW and OPEN. Verified no regression: `client.on('connected', connected)` and its siblings in qr-sync-controller.ts are still detected by the quoted-event pass. Also drops a hardcoded `PR #40684` from the header, which printed on every run whatever was scanned, and a re-run hint naming a script that does not exist. --- .../memory-leak/scripts/retention-scan.py | 36 ++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/domains/stability/skills/memory-leak/scripts/retention-scan.py b/domains/stability/skills/memory-leak/scripts/retention-scan.py index aad49597..ce767cef 100644 --- a/domains/stability/skills/memory-leak/scripts/retention-scan.py +++ b/domains/stability/skills/memory-leak/scripts/retention-scan.py @@ -33,6 +33,40 @@ def scan(src_path, patch_path): f"removeListener L{rln}"+(" on stream close" if onclose else ""))) else: rows.append((new,'OPEN',f"{emitter}.on('{ev}', {handler})",ln,"no removeListener in file")) + # named-subscription listeners: onXxx(handler) / subscribe(handler) / addXxxListener(handler). + # The quoted-event form above cannot see these — the method name carries the event, and + # there is no event-name argument to pair on. Missing them yields a clean verdict over a + # real unpaired listener (observed: background.onNotification on extension#42823). + for m in re.finditer( + r'(\w+)\.(on[A-Z]\w*|subscribe|addEventListener|add[A-Z]\w*Listener)\(\s*([\w.]+)\s*[,)]', + src): + emitter, method, handler = m.groups() + if method in ('on', 'addListener'): + continue # already covered by the quoted-event pass + ln = src[:m.start()].count('\n') + 1 + new = lines[ln - 1].strip() in added + # Release forms that correspond to this acquire form. + if method.startswith('on'): + rel = ['remove' + method[0].upper() + method[1:], 'off' + method[2:]] + elif method == 'subscribe': + rel = ['unsubscribe'] + elif method == 'addEventListener': + rel = ['removeEventListener'] + else: + rel = ['remove' + method[3:]] + found = None + for r in rel: + rm = re.search(re.escape(r) + r'\(', src) + if rm: + found = (r, src[:rm.start()].count('\n') + 1) + break + label = f"{emitter}.{method}({handler})" + if found: + rows.append((new, 'ok', label, ln, f"{found[0]} L{found[1]}")) + else: + rows.append((new, 'OPEN', label, ln, + "no " + "/".join(rel) + " in file")) + # pending registries: Map with set paired with delete for m in re.finditer(r'(#?\w*[Pp]ending\w*|#?\w*[Rr]equests?\w*)\s*[=:][^\n]*new Map', src): name=m.group(1); ln=src[:m.start()].count('\n')+1 @@ -46,7 +80,7 @@ def scan(src_path, patch_path): rows.append((new,'OPEN',f"{name} (.set L{sln})",ln,"no .delete — entries accumulate")) return rows -print("RETENTION REVIEW — PR #40684, scoped to the diff (re-run: retention-scoped.py <file> <patch>)") +print("RETENTION REVIEW — scoped to the supplied diff (re-run: retention-scan.py <file>:<patch> [...])") print("="*74) new_open=0 for pair in sys.argv[1:]: From 85a575c3a7fbe591aa2a8f65536e8951a2b8d60f Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 22:08:46 -0400 Subject: [PATCH 069/135] =?UTF-8?q?Add=20`selector-recompute.sh`=20?= =?UTF-8?q?=E2=80=94=20lane=20C4=20gets=20a=20runner?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A memoization claim is a claim about a count, and reselect publishes the count. This generates a probe, runs it, deletes it, and writes the artifact, so the number never passes through an operator's hands. Three conditions; the middle one discriminates. A selector built on narrowed input selectors is unmoved by a write it does not read, while one taking `state.metamask` wholesale recomputes on every unrelated write in the app. Verified against both shapes on main, so the runner is shown to distinguish them rather than only to report success: getWalletsWithAccounts 1 / 1 / 6 narrowed selectRampsControllerState 1 / 6 / 11 recomputes on unrelated writes That completes runner coverage for the catalog lanes with engine skills: B3 and B7 via falsify-probe, C4 here, C9 and D3 via capture around their existing analysis scripts. --- .../evidence/scripts/selector-recompute.sh | 151 ++++++++++++++++++ domains/pr-workflow/skills/evidence/skill.md | 24 +++ 2 files changed, 175 insertions(+) create mode 100755 domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh new file mode 100755 index 00000000..5abb05b9 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -0,0 +1,151 @@ +#!/usr/bin/env bash +# +# selector-recompute — measure how often a reselect selector actually recomputes. +# +# Lane C4. A memoization claim ("avoids recomputation", "stops deep traversal", +# "prevents re-renders") is a claim about a COUNT. `reselect` exposes that count +# natively via `.recomputations()`, so no instrumentation and no profiler is +# needed — and no operator judgement either. +# +# Generates a throwaway probe test, runs it, captures the counter under three +# conditions, removes the probe, and writes the artifact itself. +# +# A identical state reference, repeated → memoized floor (expect 1) +# B fresh enclosing slice, unrelated field changed → does an unrelated write cost a recompute? +# C a real input key perturbed → does a relevant write cost one? (expect +1 each) +# +# B is the discriminating condition. A selector taking narrowed inputs is +# unmoved by B; one reading a whole slice recomputes on every unrelated write. +# +# Usage: +# selector-recompute.sh --module <import path> --export <name> \ +# --fixture <json path> --slice <key> --perturb <key> [--n 5] [--label <slug>] +# +# Example: +# selector-recompute.sh \ +# --module ui/selectors/multichain-accounts/account-tree \ +# --export getWalletsWithAccounts \ +# --fixture test/data/mock-state.json --slice metamask --perturb pinnedAccountList +set -uo pipefail + +N=5; OUT_DIR="evidence-artifacts"; LABEL=""; MODULE=""; EXPORT=""; FIXTURE=""; SLICE="metamask"; PERTURB="" +die() { printf 'selector-recompute: %s\n' "$1" >&2; exit 3; } + +while [ $# -gt 0 ]; do + case "$1" in + --module) MODULE="${2:-}"; shift 2 ;; + --export) EXPORT="${2:-}"; shift 2 ;; + --fixture) FIXTURE="${2:-}"; shift 2 ;; + --slice) SLICE="${2:-}"; shift 2 ;; + --perturb) PERTURB="${2:-}"; shift 2 ;; + --n) N="${2:-}"; shift 2 ;; + --label) LABEL="${2:-}"; shift 2 ;; + --out) OUT_DIR="${2:-}"; shift 2 ;; + -h|--help) sed -n '2,27p' "$0"; exit 0 ;; + *) die "unknown argument: $1" ;; + esac +done + +[ -n "$MODULE" ] || die "--module is required (import path, no extension)" +[ -n "$EXPORT" ] || die "--export is required (the selector's exported name)" +[ -n "$FIXTURE" ] || die "--fixture is required (a JSON state fixture)" +[ -n "$PERTURB" ] || die "--perturb is required (an input key the selector genuinely reads)" +[ -f "$FIXTURE" ] || die "fixture not found: $FIXTURE" +[ -f "$MODULE.ts" ] || [ -f "$MODULE.js" ] || die "module not found: $MODULE.{ts,js}" + +LABEL="${LABEL:-recompute-$EXPORT}" +mkdir -p "$OUT_DIR" || die "cannot create $OUT_DIR" +STAMP="$OUT_DIR/$LABEL" +PROBE="$(dirname "$MODULE")/__recompute_probe__.test.ts" + +# Relative import from the probe back to the module, and up to the fixture. +MOD_BASE="./$(basename "$MODULE")" +DEPTH="$(dirname "$MODULE" | tr -cd '/' | wc -c | tr -d ' ')" +UP=""; i=0; while [ "$i" -le "$DEPTH" ]; do UP="../$UP"; i=$((i+1)); done + +cleanup() { rm -f "$PROBE"; } +trap cleanup EXIT INT TERM + +cat > "$PROBE" <<PROBEEOF +import { $EXPORT } from '$MOD_BASE'; +import fixture from '$UP$FIXTURE'; + +describe('$EXPORT recomputation probe', () => { + it('counts recomputations across three conditions', () => { + const base = fixture as never as { $SLICE: Record<string, unknown> }; + const call = (s: unknown) => ($EXPORT as (x: never) => unknown)(s as never); + + ($EXPORT as unknown as { resetRecomputations: () => void }).resetRecomputations(); + const count = () => ($EXPORT as unknown as { recomputations: () => number }).recomputations(); + + for (let i = 0; i < $N; i++) call(base); + const a = count(); + + for (let i = 0; i < $N; i++) { + call({ ...base, $SLICE: { ...base.$SLICE, __unrelated__: i } }); + } + const b = count(); + + for (let i = 0; i < $N; i++) { + call({ ...base, $SLICE: { ...base.$SLICE, $PERTURB: [\`0x\${i}\`] } }); + } + const c = count(); + + // eslint-disable-next-line no-console + console.log(\`RECOMPUTE_PROBE identical=\${a} unrelated=\${b} inputChanged=\${c} n=$N\`); + expect(c).toBeGreaterThanOrEqual(b); + }); +}); +PROBEEOF + +yarn jest "$PROBE" > "$STAMP.log" 2>&1 +CODE=$? +cleanup; trap - EXIT INT TERM + +LINE="$(grep -o 'RECOMPUTE_PROBE .*' "$STAMP.log" | head -1)" +A="$(printf '%s' "$LINE" | sed -n 's/.*identical=\([0-9]*\).*/\1/p')" +B="$(printf '%s' "$LINE" | sed -n 's/.*unrelated=\([0-9]*\).*/\1/p')" +C="$(printf '%s' "$LINE" | sed -n 's/.*inputChanged=\([0-9]*\).*/\1/p')" + +if [ -z "$A" ]; then + VERDICT="probe-failed" +elif [ "$B" -gt "$A" ]; then + VERDICT="recomputes on unrelated writes" +else + VERDICT="narrowed — unrelated writes cost nothing" +fi + +HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" +DIRTY="$(git status --porcelain 2>/dev/null | grep -v '^??' | wc -l | tr -d ' ')" +NODE_V="$(node -v 2>/dev/null || echo unknown)" + +cat > "$STAMP.json" <<JSON +{ "selector": "$EXPORT", "module": "$MODULE", "verdict": "$VERDICT", "exit": $CODE, + "n_calls_per_condition": $N, + "recomputations": { "identical": ${A:-null}, "unrelated_write": ${B:-null}, "input_changed": ${C:-null} }, + "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, "node": "$NODE_V" }, + "log": "$STAMP.log" } +JSON + +{ + echo "### C4 — \`$EXPORT\` recomputation count" + echo + echo "**Verdict:** $VERDICT" + echo + echo "| Condition | Calls | Recomputations |" + echo "|---|---|---|" + echo "| Identical state reference | $N | ${A:-?} |" + echo "| Fresh \`$SLICE\` slice, unrelated field | $N | ${B:-?} |" + echo "| \`$PERTURB\` changed (a real input) | $N | ${C:-?} |" + echo + echo '```console' + echo "\$ yarn jest <generated probe>" + echo "$LINE" + echo '```' + echo + echo "<sub>Measured by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Log: \`$STAMP.log\`.</sub>" +} > "$STAMP.md" + +printf 'selector-recompute: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 +[ -n "$A" ] || exit 2 +exit 0 diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index aba3c70f..e384f4a8 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -223,6 +223,30 @@ default. A crashing command produces an artifact containing the traceback, not a fabricated result. +### `selector-recompute.sh` — lane C4 + +A memoization claim is a claim about a count, and `reselect` publishes the count. Generates a +probe, runs it, deletes it, writes the artifact: + +```bash +scripts/selector-recompute.sh --module ui/selectors/multichain-accounts/account-tree \ + --export getWalletsWithAccounts --fixture test/data/mock-state.json \ + --slice metamask --perturb pinnedAccountList +``` + +Three conditions, of which the middle one discriminates: + +| Condition | narrowed inputs | whole-slice input | +|---|---|---| +| identical state reference | 1 | 1 | +| fresh slice, **unrelated** field | **1** | **6** | +| a real input changed | 6 | 11 | + +A selector taking narrowed input selectors is unmoved by an unrelated write; one reading +`state.metamask` wholesale recomputes on every unrelated write in the app. Both rows above are +measured, not illustrative — `getWalletsWithAccounts` and `selectRampsControllerState` on +`main`. + ### Canonical output shape Every validation-run output — PR comment *or* PR-body section — uses exactly this, so re-runs From 94a764079f9bb037328e8951b32e1e8177c8cb87 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 22:16:33 -0400 Subject: [PATCH 070/135] =?UTF-8?q?Add=20`tsc-substitution.sh`=20=E2=80=94?= =?UTF-8?q?=20a=20runner=20for=20the=20tsc-blindspots=20lane?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Whether a hand-written type agrees with the source it restates is a question only the compiler can settle. Arm A typechecks the baseline, arm B applies the substitution, and the finding is the error diff. Source is restored on exit, including on interrupt. Carries the warning the lane most needs: a silent arm B is not proof of agreement. Indexing and `.match()` compile against `string` and `string[]` alike, so without `--probe` injecting a deliberately-typed sink, the lane reports false clean on exactly the divergence it exists to find. When arm A already fails it stops and says nothing was established, alongside the module/export error count — but it does not classify from that ratio. The first real run had 124 of 280 errors as install artifacts while tripping no majority rule, because other codes are downstream of the same missing types. A threshold there would be a number I could not justify, so it reports the breakdown and leaves the judgement with the operator. --- .../evidence/scripts/tsc-substitution.sh | 140 ++++++++++++++++++ domains/pr-workflow/skills/evidence/skill.md | 25 ++++ 2 files changed, 165 insertions(+) create mode 100755 domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh new file mode 100755 index 00000000..23e84c2d --- /dev/null +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -0,0 +1,140 @@ +#!/usr/bin/env bash +# +# tsc-substitution — arm A/B against the type checker. +# +# A hand-written type that restates an authoritative source either agrees with it +# or does not, and `tsc` is the only thing that can settle which. Reading the two +# declarations side by side does not: TypeScript's assignability rules are not +# obvious by inspection, which is the entire reason the lane exists. +# +# Arm A baseline typecheck, errors recorded +# Arm B the substitution applied — the hand-written type replaced by the derived +# one, or a cast removed — typecheck re-run, errors diffed +# +# The finding is the DIFF: error codes present in B and absent in A are what the +# hand-written type or the cast was concealing. +# +# 0 divergence surfaced new errors in arm B → the local type disagrees +# 1 no divergence identical error sets → substitution is silent +# 2 arm A already failing → nothing to conclude +# 3 usage/env error +# +# A silent result is NOT proof of agreement. Existing call sites may type-check +# against both shapes (indexing a `string` and a `string[]` both compile), so use +# --probe to inject a deliberately-typed sink that only one shape satisfies. +# +# Usage: +# tsc-substitution.sh --file <path> --line <n> --replace <text> +# [--probe-line <n> --probe <text>] [--label <slug>] +# [--tsc "<command>"] +set -uo pipefail + +TSC="yarn lint:tsc"; OUT_DIR="evidence-artifacts"; LABEL="" +FILE=""; LINE=""; REPLACE=""; PROBE_LINE=""; PROBE="" +die() { printf 'tsc-substitution: %s\n' "$1" >&2; exit 3; } + +while [ $# -gt 0 ]; do + case "$1" in + --file) FILE="${2:-}"; shift 2 ;; + --line) LINE="${2:-}"; shift 2 ;; + --replace) REPLACE="${2:-}"; shift 2 ;; + --probe-line) PROBE_LINE="${2:-}"; shift 2 ;; + --probe) PROBE="${2:-}"; shift 2 ;; + --label) LABEL="${2:-}"; shift 2 ;; + --out) OUT_DIR="${2:-}"; shift 2 ;; + --tsc) TSC="${2:-}"; shift 2 ;; + -h|--help) sed -n '2,30p' "$0"; exit 0 ;; + *) die "unknown argument: $1" ;; + esac +done + +[ -n "$FILE" ] || die "--file is required" +[ -f "$FILE" ] || die "file not found: $FILE" +[ -n "$LINE" ] || [ -n "$PROBE" ] || die "give --line/--replace, or --probe-line/--probe, or both" +LABEL="${LABEL:-tsc-$(basename "$FILE" | sed 's/\.[^.]*$//')}" +mkdir -p "$OUT_DIR" || die "cannot create $OUT_DIR" +STAMP="$OUT_DIR/$LABEL" + +BACKUP="$(mktemp)" || die "mktemp failed" +cp "$FILE" "$BACKUP" +restore() { cp "$BACKUP" "$FILE"; rm -f "$BACKUP"; } +trap restore EXIT INT TERM + +# tsc exits non-zero on any error, so the error SET is the signal, not the exit code. +errors_of() { grep -oE "error TS[0-9]+" "$1" 2>/dev/null | sort | uniq -c | sed 's/^ *//'; } + +$TSC > "$STAMP-armA.log" 2>&1 +A_ERRS="$(errors_of "$STAMP-armA.log")" +A_COUNT="$(grep -c "error TS" "$STAMP-armA.log" 2>/dev/null || echo 0)" + +if [ "$A_COUNT" -gt 0 ]; then + # Distinguish a genuinely failing repo from an incomplete local install. A baseline + # dominated by TS2305/TS2724/TS2307 ("has no exported member" / "cannot find module") + # means dependency types were never generated — `yarn install --mode=skip-build` does + # exactly this — and says nothing about the code. Reporting both as "baseline failing" + # would send the operator hunting a repo defect that is not there. + # Report the module/export share; do not classify from it. A threshold here would be + # a number I cannot justify — 124/280 on this repo is plainly an install artifact, yet + # trips no majority rule, because TS2339 and TS7006 are themselves downstream of the + # missing types. Surface the signal, leave the judgement with the operator. + ENVISH="$(grep -coE "error TS(2305|2307|2724)" "$STAMP-armA.log" || echo 0)" + VERDICT="baseline failing — no conclusion available" + CODE=2; B_COUNT="not-run"; NEW_ERRS="" + : > "$STAMP-armB.log" +else + # Apply substitution and/or probe, highest line first so numbering holds. + apply() { awk -v n="$1" -v r="$2" 'NR==n{print r; next}{print}' "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"; } + insert() { awk -v n="$1" -v r="$2" 'NR==n{print; print r; next}{print}' "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"; } + if [ -n "$PROBE" ] && [ -n "$PROBE_LINE" ] && [ -n "$LINE" ] && [ "$PROBE_LINE" -gt "$LINE" ]; then + insert "$PROBE_LINE" "$PROBE"; apply "$LINE" "$REPLACE" + else + [ -n "$LINE" ] && apply "$LINE" "$REPLACE" + [ -n "$PROBE" ] && [ -n "$PROBE_LINE" ] && insert "$PROBE_LINE" "$PROBE" + fi + + $TSC > "$STAMP-armB.log" 2>&1 + B_COUNT="$(grep -c "error TS" "$STAMP-armB.log" 2>/dev/null || echo 0)" + NEW_ERRS="$(grep -oE "error TS[0-9]+.*" "$STAMP-armB.log" 2>/dev/null | sort -u | head -12)" + restore; trap - EXIT INT TERM + if [ "$B_COUNT" -gt 0 ]; then VERDICT="divergence surfaced"; CODE=0; else VERDICT="substitution silent"; CODE=1; fi +fi + +HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" +DIRTY="$(git status --porcelain 2>/dev/null | grep -v '^??' | wc -l | tr -d ' ')" +TS_V="$(yarn tsc --version 2>/dev/null | tail -1 || echo unknown)" + +cat > "$STAMP.json" <<JSON +{ "verdict": "$VERDICT", "exit": $CODE, "file": "$FILE", + "arm_a_errors": $A_COUNT, "arm_b_errors": ${B_COUNT:-null}, + "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, "typescript": "$TS_V" }, + "logs": ["$STAMP-armA.log", "$STAMP-armB.log"] } +JSON + +{ + echo "### D6 — authored-vs-authoritative substitution · \`$VERDICT\`" + echo + echo "| Arm | Change | \`tsc\` errors |" + echo "|---|---|---|" + echo "| A — baseline | none | $A_COUNT |" + echo "| B — substituted | \`$FILE\`${LINE:+:$LINE}${PROBE:+ + typed sink} | ${B_COUNT} |" + echo + if [ "$CODE" = "2" ] && [ "${ENVISH:-0}" -gt 0 ]; then + echo "Arm A did not pass, so arm B was not run and **nothing about the types is established**." + echo + echo "\`$ENVISH\` of \`$A_COUNT\` baseline errors are module/export resolution" + echo "(TS2305/TS2307/TS2724). Those usually mean dependency types were never generated —" + echo "a skipped install step — rather than a defect in this repo, and other codes can be" + echo "downstream of the same cause. Confirm the toolchain is complete before reading" + echo "anything into this lane." + elif [ -n "$NEW_ERRS" ]; then + echo "Errors surfaced only under substitution:"; echo; echo '```'; printf '%s\n' "$NEW_ERRS"; echo '```' + elif [ "$CODE" = "1" ]; then + echo "**Silent — this is not proof of agreement.** Existing call sites may satisfy both shapes." + echo "Re-run with \`--probe\` to inject a sink only the authoritative type accepts." + fi + echo + echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" +} > "$STAMP.md" + +printf 'tsc-substitution: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 +exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index e384f4a8..6a258b30 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -247,6 +247,31 @@ A selector taking narrowed input selectors is unmoved by an unrelated write; one measured, not illustrative — `getWalletsWithAccounts` and `selectRampsControllerState` on `main`. +### `tsc-substitution.sh` — lane D6 (`tsc-blindspots`) + +Whether a hand-written type agrees with the source it restates is a question only the compiler +can settle; assignability is not obvious by inspection, which is the reason the lane exists. + +```bash +scripts/tsc-substitution.sh --file shared/lib/transactions-controller-utils.ts \ + --line 146 --replace ' topics?: string;' \ + --probe-line 150 --probe ' const _probe: string[] = txReceiptLogs[0].topics;' +``` + +Arm A typechecks the baseline, arm B applies the substitution, and the finding is the **error +diff**. Source is restored on exit including on interrupt. + +**A silent arm B is not proof of agreement.** Existing call sites often satisfy both shapes — +indexing and `.match()` compile against `string` and `string[]` alike — so use `--probe` to +inject a deliberately-typed sink that only the authoritative shape accepts. Without one this +lane reports false clean. + +If arm A already fails, the run stops and states that nothing was established, alongside the +count of module/export errors (TS2305/TS2307/TS2724), which usually indicate an incomplete +install rather than a repo defect. It does not classify from that ratio — on a real run 124 of +280 errors were install artifacts while tripping no majority rule, because other codes are +downstream of the same cause. + ### Canonical output shape Every validation-run output — PR comment *or* PR-body section — uses exactly this, so re-runs From 6e38d6da9e4b5a706bb6fd20c7f11417b0aa3650 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Fri, 31 Jul 2026 22:31:28 -0400 Subject: [PATCH 071/135] Diff the error sets rather than requiring a clean baseline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Gating on arm A being error-free was wrong: the finding was always the diff, so a single unrelated pre-existing error — a local work-in-progress file, in the run that exposed this — vetoed the whole lane. Baseline errors are now subtracted and only errors new under substitution count. Verified end to end on shared/lib/transactions-controller-utils.ts, where the local `LogWithTopicsArray` declares `topics?: string[]` against an upstream `topics?: string`: substitution alone 1 -> 1 errors, 0 new silent substitution + typed sink 1 -> 2 errors, 1 new TS2322 at the sink That is the lane's central caution demonstrated rather than asserted. Indexing and `.match()` compile against both shapes, so the obvious probe reports a false clean; only a deliberately-typed sink surfaces the divergence. A silent arm B means the probe was too weak, not that the types agree. Also fixes a `grep -c ... || echo 0` double-fire that produced "0\n0" and an integer-comparison error — the same shape already fixed once in this script's tracked-change count. --- .../evidence/scripts/tsc-substitution.sh | 88 +++++++++---------- 1 file changed, 41 insertions(+), 47 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh index 23e84c2d..4dc4ea81 100755 --- a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -16,7 +16,7 @@ # # 0 divergence surfaced new errors in arm B → the local type disagrees # 1 no divergence identical error sets → substitution is silent -# 2 arm A already failing → nothing to conclude +# 2 usage/env error (baseline errors are subtracted, not disqualifying) # 3 usage/env error # # A silent result is NOT proof of agreement. Existing call sites may type-check @@ -64,39 +64,34 @@ trap restore EXIT INT TERM errors_of() { grep -oE "error TS[0-9]+" "$1" 2>/dev/null | sort | uniq -c | sed 's/^ *//'; } $TSC > "$STAMP-armA.log" 2>&1 -A_ERRS="$(errors_of "$STAMP-armA.log")" -A_COUNT="$(grep -c "error TS" "$STAMP-armA.log" 2>/dev/null || echo 0)" +# Baseline errors are subtracted, not disqualifying. A local WIP file or an +# unrelated pre-existing error must not veto the lane — the finding was always the +# DIFF, so compare error SETS and let anything already present fall out. +grep -oE "^[^ ]+\([0-9]+,[0-9]+\): error TS[0-9]+" "$STAMP-armA.log" 2>/dev/null | sort -u > "$STAMP-armA.set" +A_COUNT="$(wc -l < "$STAMP-armA.set" | tr -d ' ')" +ENVISH="$(grep -cE "error TS(2305|2307|2724)" "$STAMP-armA.log" 2>/dev/null)"; ENVISH="${ENVISH:-0}" -if [ "$A_COUNT" -gt 0 ]; then - # Distinguish a genuinely failing repo from an incomplete local install. A baseline - # dominated by TS2305/TS2724/TS2307 ("has no exported member" / "cannot find module") - # means dependency types were never generated — `yarn install --mode=skip-build` does - # exactly this — and says nothing about the code. Reporting both as "baseline failing" - # would send the operator hunting a repo defect that is not there. - # Report the module/export share; do not classify from it. A threshold here would be - # a number I cannot justify — 124/280 on this repo is plainly an install artifact, yet - # trips no majority rule, because TS2339 and TS7006 are themselves downstream of the - # missing types. Surface the signal, leave the judgement with the operator. - ENVISH="$(grep -coE "error TS(2305|2307|2724)" "$STAMP-armA.log" || echo 0)" - VERDICT="baseline failing — no conclusion available" - CODE=2; B_COUNT="not-run"; NEW_ERRS="" - : > "$STAMP-armB.log" +apply() { awk -v n="$1" -v r="$2" 'NR==n{print r; next}{print}' "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"; } +insert() { awk -v n="$1" -v r="$2" 'NR==n{print; print r; next}{print}' "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"; } +if [ -n "$PROBE" ] && [ -n "$PROBE_LINE" ] && [ -n "$LINE" ] && [ "$PROBE_LINE" -gt "$LINE" ]; then + insert "$PROBE_LINE" "$PROBE"; apply "$LINE" "$REPLACE" else - # Apply substitution and/or probe, highest line first so numbering holds. - apply() { awk -v n="$1" -v r="$2" 'NR==n{print r; next}{print}' "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"; } - insert() { awk -v n="$1" -v r="$2" 'NR==n{print; print r; next}{print}' "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"; } - if [ -n "$PROBE" ] && [ -n "$PROBE_LINE" ] && [ -n "$LINE" ] && [ "$PROBE_LINE" -gt "$LINE" ]; then - insert "$PROBE_LINE" "$PROBE"; apply "$LINE" "$REPLACE" - else - [ -n "$LINE" ] && apply "$LINE" "$REPLACE" - [ -n "$PROBE" ] && [ -n "$PROBE_LINE" ] && insert "$PROBE_LINE" "$PROBE" - fi + [ -n "$LINE" ] && apply "$LINE" "$REPLACE" + [ -n "$PROBE" ] && [ -n "$PROBE_LINE" ] && insert "$PROBE_LINE" "$PROBE" +fi + +$TSC > "$STAMP-armB.log" 2>&1 +grep -oE "^[^ ]+\([0-9]+,[0-9]+\): error TS[0-9]+" "$STAMP-armB.log" 2>/dev/null | sort -u > "$STAMP-armB.set" +B_COUNT="$(wc -l < "$STAMP-armB.set" | tr -d ' ')" +restore; trap - EXIT INT TERM + +NEW_ERRS="$(comm -13 "$STAMP-armA.set" "$STAMP-armB.set" | head -12)" +NEW_COUNT="$(comm -13 "$STAMP-armA.set" "$STAMP-armB.set" | wc -l | tr -d ' ')" - $TSC > "$STAMP-armB.log" 2>&1 - B_COUNT="$(grep -c "error TS" "$STAMP-armB.log" 2>/dev/null || echo 0)" - NEW_ERRS="$(grep -oE "error TS[0-9]+.*" "$STAMP-armB.log" 2>/dev/null | sort -u | head -12)" - restore; trap - EXIT INT TERM - if [ "$B_COUNT" -gt 0 ]; then VERDICT="divergence surfaced"; CODE=0; else VERDICT="substitution silent"; CODE=1; fi +if [ "$NEW_COUNT" -gt 0 ]; then + VERDICT="divergence surfaced"; CODE=0 +else + VERDICT="substitution silent"; CODE=1 fi HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" @@ -105,7 +100,7 @@ TS_V="$(yarn tsc --version 2>/dev/null | tail -1 || echo unknown)" cat > "$STAMP.json" <<JSON { "verdict": "$VERDICT", "exit": $CODE, "file": "$FILE", - "arm_a_errors": $A_COUNT, "arm_b_errors": ${B_COUNT:-null}, + "arm_a_errors": $A_COUNT, "arm_b_errors": $B_COUNT, "new_under_substitution": $NEW_COUNT, "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, "typescript": "$TS_V" }, "logs": ["$STAMP-armA.log", "$STAMP-armB.log"] } JSON @@ -113,27 +108,26 @@ JSON { echo "### D6 — authored-vs-authoritative substitution · \`$VERDICT\`" echo - echo "| Arm | Change | \`tsc\` errors |" + echo "| Arm | Change | distinct \`tsc\` errors |" echo "|---|---|---|" echo "| A — baseline | none | $A_COUNT |" - echo "| B — substituted | \`$FILE\`${LINE:+:$LINE}${PROBE:+ + typed sink} | ${B_COUNT} |" + echo "| B — substituted | \`$FILE\`${LINE:+:$LINE}${PROBE:+ + typed sink} | $B_COUNT |" + echo "| **new under substitution** | | **$NEW_COUNT** |" echo - if [ "$CODE" = "2" ] && [ "${ENVISH:-0}" -gt 0 ]; then - echo "Arm A did not pass, so arm B was not run and **nothing about the types is established**." - echo - echo "\`$ENVISH\` of \`$A_COUNT\` baseline errors are module/export resolution" - echo "(TS2305/TS2307/TS2724). Those usually mean dependency types were never generated —" - echo "a skipped install step — rather than a defect in this repo, and other codes can be" - echo "downstream of the same cause. Confirm the toolchain is complete before reading" - echo "anything into this lane." - elif [ -n "$NEW_ERRS" ]; then - echo "Errors surfaced only under substitution:"; echo; echo '```'; printf '%s\n' "$NEW_ERRS"; echo '```' - elif [ "$CODE" = "1" ]; then - echo "**Silent — this is not proof of agreement.** Existing call sites may satisfy both shapes." + if [ "$NEW_COUNT" -gt 0 ]; then + echo "Errors present in B and absent in A — what the local type was concealing:" + echo; echo '```'; printf '%s\n' "$NEW_ERRS"; echo '```' + else + echo "**Silent — this is not proof of agreement.** Existing call sites may satisfy both" + echo "shapes; indexing and \`.match()\` compile against \`string\` and \`string[]\` alike." echo "Re-run with \`--probe\` to inject a sink only the authoritative type accepts." fi + if [ "$A_COUNT" -gt 0 ]; then + echo + echo "<sub>Baseline carried $A_COUNT pre-existing error(s) (${ENVISH} module/export). These are" + echo "subtracted, not disqualifying — only errors new under substitution are the finding.</sub>" + fi echo - echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" } > "$STAMP.md" printf 'tsc-substitution: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 From 3f861c716fe4bc2e814aadf200db1abb88a6ee71 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 07:25:25 -0400 Subject: [PATCH 072/135] =?UTF-8?q?Add=20`attest-gate.sh`=20=E2=80=94=20ei?= =?UTF-8?q?ght=20mechanical=20checks=20before=20anything=20is=20published?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 0 of the /attest command. Everything greppable is checked before a model is asked for judgement, because a model asked "is this good evidence?" answers from inside the frame that produced the text. Marker pair, canonical header, verdict line, environment pin, a captured artifact, no "what would close it", no first-person process narration, and `proven` only where an execution artifact exists. Check 5 carries the weight: if every character of the output is one the operator typed, the run published an assertion. `--reference` compares capture density against a known-good artifact. Verified in both directions. A retracted run-1 comment is BLOCKED on three checks — no captured artifact, a "what would close it" section, and an unearned `proven`. A runner-produced artifact passes all eight. Building it reproduced two bugs it exists to catch: `hasre -i '<pat>'` passed `-i` as the pattern, so three checks silently grepped for the literal string and returned false passes; and an over-escaped backtick made the environment-pin check never match. Both found by running the gate against a file whose expected verdict was already known. --- .../skills/evidence/scripts/attest-gate.sh | 88 +++++++++++++++++++ domains/pr-workflow/skills/evidence/skill.md | 22 +++++ 2 files changed, 110 insertions(+) create mode 100755 domains/pr-workflow/skills/evidence/scripts/attest-gate.sh diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh new file mode 100755 index 00000000..1fe9e1d4 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# +# attest-gate — phase 0 of /attest. Mechanical, no model, hard fails only. +# +# Everything checkable is checked before anything is asked of a model, because a +# model asked "is this good evidence?" answers from inside the frame that produced +# the text. These eight are greppable, so they are not a matter of judgement. +# +# Usage: attest-gate.sh <artifact.md> [--reference <showcase.html>] +# +# 0 all checks pass → proceed to the dispatched passes +# 1 one or more failed → BLOCKED, do not publish +# 2 usage error +set -uo pipefail + +FILE="${1:-}"; REF="" +[ $# -ge 2 ] && [ "${2:-}" = "--reference" ] && REF="${3:-}" +[ -n "$FILE" ] || { echo "usage: attest-gate.sh <artifact.md> [--reference <file>]" >&2; exit 2; } +[ -f "$FILE" ] || { echo "attest-gate: not found: $FILE" >&2; exit 2; } + +FAILED=0 +pass() { printf ' PASS %s\n' "$1"; } +fail() { printf ' FAIL %s\n %s\n' "$1" "$2"; FAILED=$((FAILED+1)); } +has() { grep -qF "$1" "$FILE"; } +hasre(){ grep -qE "$1" "$FILE"; } +hasi() { grep -qiE "$1" "$FILE"; } # case-insensitive; a separate function because + # `hasre -i '<pat>'` silently greps for "-i". + +echo "attest-gate: $FILE" +echo + +has 'VALIDATION_RUN_START' && has 'VALIDATION_RUN_END' \ + && pass "1 marker pair" \ + || fail "1 marker pair" "no VALIDATION_RUN_START/_END — a re-run appends a duplicate instead of replacing" + +has '## 🧪 Validation Run' \ + && pass "2 canonical header" \ + || fail "2 canonical header" "missing '## 🧪 Validation Run'" + +hasre '^\*\*Verdict:\*\*.*\*\*Claim:\*\*' \ + && pass "3 verdict line" \ + || fail "3 verdict line" "no '**Verdict:** … — **Claim:** …' — valence is not legible at a glance" + +hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `' \ + && pass "4 environment pinned" \ + || fail "4 environment pinned" "no head SHA, toolchain version, or lockfile hash" + +# 5 — the one that matters. A tool-written log, a run link, or an image; not typed prose. +if hasre '!\[|<img|data:image|actions/runs|/gist\.|evidence-artifacts/|Captured by|Produced by \`'; then + pass "5 captured artifact" +else + fail "5 captured artifact" "every block appears operator-typed; no tool-written log, run link, or image referenced" +fi + +if hasi 'what would close it|what would prove it|closing it requires'; then + fail "6 no prescriptions" "contains a 'what would close it' section — that is an unfinished run, formatted to look finished" +elif hasre '^\s*(Run|Switch|Assert|Scroll|Compare) '; then + fail "6 no prescriptions" "imperative-mood instructions to the reader — the artifact does not exist" +else + pass "6 no prescriptions" +fi + +if hasi "I originally|correction to my earlier|filed by me|hard to calibrate|I withdraw|my earlier comment"; then + fail "7 no process narration" "contains first-person process commentary — the reader did not see the earlier draft, and the byline may not be yours" +else + pass "7 no process narration" +fi + +if hasi '\*\*Verdict:\*\*.*proven' && ! hasre 'Captured by|Produced by \`|actions/runs|evidence-artifacts/'; then + fail "8 verdict is earned" "claims 'proven' with no execution artifact — reading yields 'unverified'" +else + pass "8 verdict is earned" +fi + +if [ -n "$REF" ] && [ -f "$REF" ]; then + r=$(grep -coE '!\[|<img|data:image' "$REF"); c=$(grep -coE '!\[|<img|data:image|evidence-artifacts/|Captured by' "$FILE") + echo + printf ' ratio reference captures: %s | this artifact: %s\n' "$r" "$c" + [ "$c" -eq 0 ] && [ "$r" -gt 0 ] && printf ' reference is capture-led and this is prose-only — see check 5\n' +fi + +echo +if [ "$FAILED" -eq 0 ]; then + echo "attest-gate: phase 0 clean — proceed to /outframe ‖ /missing ‖ /press" + exit 0 +fi +echo "attest-gate: BLOCKED — $FAILED check(s) failed. Do not publish." +exit 1 diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 6a258b30..55a53d38 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -272,6 +272,28 @@ install rather than a repo defect. It does not classify from that ratio — on a 280 errors were install artifacts while tripping no majority rule, because other codes are downstream of the same cause. +### `attest-gate.sh` — run this before publishing anything + +Eight mechanical checks over the artifact as it will ship. No model is asked anything until +these pass, because a model asked "is this good evidence?" answers from inside the frame that +produced the text. + +```bash +scripts/attest-gate.sh comment.md # exit 0 = proceed, 1 = BLOCKED +``` + +Marker pair · canonical header · verdict line · environment pinned · **a captured artifact** · +no "what would close it" · no first-person process narration · `proven` only with an execution +artifact. + +Check 5 is the one that matters and the easiest to slip past: if every character of the output +is one the operator typed, the run published an assertion. Pass `--reference <showcase>` to +compare capture density against a known-good artifact. + +This is phase 0 of [`/attest`](https://github.com/MajorLift/Reprise); phases 1 and 2 dispatch +`/outframe ‖ /missing ‖ /press` then `/trim` to fresh instances, because those passes cannot be +self-run — the author is positionally the wrong reader. + ### Canonical output shape Every validation-run output — PR comment *or* PR-body section — uses exactly this, so re-runs From 2c0f2869071aab16edeac66c6cef79b85bbea8c5 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 09:38:18 -0400 Subject: [PATCH 073/135] Standardise the provenance marker across every runner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three runners wrote three different provenance verbs — "Produced by", "Captured by", "Measured by" — so the gate had to know each synonym and silently blocked the one it did not. That is a false BLOCK on valid evidence, the most expensive direction for a gate to fail in. All four now emit `Produced by <script>`, and the gate matches one marker. `tsc-substitution.sh` emitted none at all: an earlier rewrite of its markdown body dropped the footer, so its artifacts failed the captured-artifact check despite being fully machine-produced. Also removes the last escaped backticks from the gate. In ERE a backtick is not special, so `Produced by \`` matched literal-backslash-backtick and never fired — the same defect already fixed once in the environment-pin check and not generalised then. Fifth appearance of this class today; now eliminated rather than patched per-check. Verified by regating five wrapped comments through every fix: three blocked on the synonym mismatch, all six pass once the marker is uniform. --- domains/pr-workflow/skills/evidence/scripts/attest-gate.sh | 6 +++--- domains/pr-workflow/skills/evidence/scripts/capture.sh | 2 +- .../skills/evidence/scripts/selector-recompute.sh | 2 +- .../pr-workflow/skills/evidence/scripts/tsc-substitution.sh | 2 ++ 4 files changed, 7 insertions(+), 5 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 1fe9e1d4..f377697b 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -46,7 +46,7 @@ hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `' \ || fail "4 environment pinned" "no head SHA, toolchain version, or lockfile hash" # 5 — the one that matters. A tool-written log, a run link, or an image; not typed prose. -if hasre '!\[|<img|data:image|actions/runs|/gist\.|evidence-artifacts/|Captured by|Produced by \`'; then +if hasre '!\[|<img|data:image|actions/runs|/gist\.|evidence-artifacts/|Produced by '; then pass "5 captured artifact" else fail "5 captured artifact" "every block appears operator-typed; no tool-written log, run link, or image referenced" @@ -66,14 +66,14 @@ else pass "7 no process narration" fi -if hasi '\*\*Verdict:\*\*.*proven' && ! hasre 'Captured by|Produced by \`|actions/runs|evidence-artifacts/'; then +if hasi '\*\*Verdict:\*\*.*proven' && ! hasre 'Produced by |actions/runs|evidence-artifacts/'; then fail "8 verdict is earned" "claims 'proven' with no execution artifact — reading yields 'unverified'" else pass "8 verdict is earned" fi if [ -n "$REF" ] && [ -f "$REF" ]; then - r=$(grep -coE '!\[|<img|data:image' "$REF"); c=$(grep -coE '!\[|<img|data:image|evidence-artifacts/|Captured by' "$FILE") + r=$(grep -coE '!\[|<img|data:image' "$REF"); c=$(grep -coE '!\[|<img|data:image|evidence-artifacts/|Produced by' "$FILE") echo printf ' ratio reference captures: %s | this artifact: %s\n' "$r" "$c" [ "$c" -eq 0 ] && [ "$r" -gt 0 ] && printf ' reference is capture-led and this is prose-only — see check 5\n' diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh index 064ed404..fc18c4fd 100755 --- a/domains/pr-workflow/skills/evidence/scripts/capture.sh +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -105,7 +105,7 @@ JSON fi echo '```' echo - echo "<sub>Captured by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. Raw log: \`$STAMP.log\`.</sub>" + echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. Raw log: \`$STAMP.log\`.</sub>" } > "$STAMP.md" printf 'capture: %s (exit %s)\n %s\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.log" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index 5abb05b9..54f68bcd 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -143,7 +143,7 @@ JSON echo "$LINE" echo '```' echo - echo "<sub>Measured by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Log: \`$STAMP.log\`.</sub>" + echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Log: \`$STAMP.log\`.</sub>" } > "$STAMP.md" printf 'selector-recompute: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh index 4dc4ea81..7bb81c81 100755 --- a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -128,6 +128,8 @@ JSON echo "subtracted, not disqualifying — only errors new under substitution are the finding.</sub>" fi echo + echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" + echo } > "$STAMP.md" printf 'tsc-substitution: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 From 51d6f947dedcb3b0cf5a8fef6748bf79fc17077c Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 10:21:01 -0400 Subject: [PATCH 074/135] =?UTF-8?q?Add=20`render-count.sh`=20=E2=80=94=20t?= =?UTF-8?q?he=20component=20half=20of=20lane=20C4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `selector-recompute` answers how often a selector recomputes. This answers the other C4 question: how many times a named consumer actually renders. A memoisation claim about context or props is a claim about that count, and a count of call sites is not it — 149 consumers can mean 149 avoided renders or none. Runs the probe, defeats the memo at a given line, re-runs, reverts. Verified on a synthetic provider: 1 consumer render across 6 parent updates, rising to 6 with the memo defeated. The probe is supplied rather than generated, deliberately. A provider's mount requirements are specific to the component, and a generated probe would either be wrong or need every prop on the command line. That distinction also bounds what this can currently prove about extension#39310: a probe mounting its own memo demonstrates the mechanism but does not measure that PR's provider, which needs the app's store and router. No run was posted there — a synthetic stand-in presented against a real claim is the substitution this lane exists to catch. --- .../skills/evidence/scripts/render-count.sh | 116 ++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100755 domains/pr-workflow/skills/evidence/scripts/render-count.sh diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh new file mode 100755 index 00000000..fff2c0d1 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# +# render-count — lane C4, the component half. +# +# `selector-recompute` answers "how often does this selector recompute". This +# answers the other C4 question: "how many times does a consumer actually +# render". A memoization claim about context or props is a claim about that +# count, and a count of call sites is not it — 149 consumers can mean 149 +# avoided renders or none. +# +# Generates a probe that mounts a provider with a counting consumer, forces the +# parent to re-render N times with the memoised value unchanged, and reports the +# consumer's render count. Arm B re-runs with the memo defeated, so the delta is +# attributable rather than assumed. +# +# Usage: +# render-count.sh --probe <probe.test.tsx> [--defeat <file> --defeat-line <n> --defeat-with <text>] +# [--label <slug>] [--out <dir>] +# +# The probe is supplied rather than generated: a provider's mount requirements +# are specific to the component, and a generated one would either be wrong or +# would need every prop passed on the command line. Write it once, keep it. +# It must print a line of the form: +# +# RENDER_COUNT consumer=<n> parentRenders=<m> +# +# 0 measured counts captured for both arms (or arm A alone if no --defeat) +# 1 no delta arm B identical to arm A — the memo is not doing what is claimed +# 2 probe did not emit RENDER_COUNT +# 3 usage error +set -uo pipefail + +OUT_DIR="evidence-artifacts"; LABEL=""; PROBE=""; DEFEAT=""; DEFEAT_LINE=""; DEFEAT_WITH="" +die() { printf 'render-count: %s\n' "$1" >&2; exit 3; } + +while [ $# -gt 0 ]; do + case "$1" in + --probe) PROBE="${2:-}"; shift 2 ;; + --defeat) DEFEAT="${2:-}"; shift 2 ;; + --defeat-line) DEFEAT_LINE="${2:-}"; shift 2 ;; + --defeat-with) DEFEAT_WITH="${2:-}"; shift 2 ;; + --label) LABEL="${2:-}"; shift 2 ;; + --out) OUT_DIR="${2:-}"; shift 2 ;; + -h|--help) sed -n '2,30p' "$0"; exit 0 ;; + *) die "unknown argument: $1" ;; + esac +done + +[ -n "$PROBE" ] || die "--probe is required" +[ -f "$PROBE" ] || die "probe not found: $PROBE" +LABEL="${LABEL:-render-$(basename "$PROBE" | sed 's/\..*$//')}" +mkdir -p "$OUT_DIR" || die "cannot create $OUT_DIR" +STAMP="$OUT_DIR/$LABEL" + +counts_from() { grep -o 'RENDER_COUNT .*' "$1" | head -1; } +consumer_of() { printf '%s' "$1" | sed -n 's/.*consumer=\([0-9]*\).*/\1/p'; } + +yarn jest "$PROBE" > "$STAMP-armA.log" 2>&1 +A_LINE="$(counts_from "$STAMP-armA.log")" +A="$(consumer_of "$A_LINE")" +[ -n "$A" ] || { printf 'render-count: probe emitted no RENDER_COUNT line\n' >&2; exit 2; } + +B=""; B_LINE="" +if [ -n "$DEFEAT" ] && [ -n "$DEFEAT_LINE" ]; then + [ -f "$DEFEAT" ] || die "defeat target not found: $DEFEAT" + BACKUP="$(mktemp)"; cp "$DEFEAT" "$BACKUP" + restore() { cp "$BACKUP" "$DEFEAT"; rm -f "$BACKUP"; } + trap restore EXIT INT TERM + awk -v n="$DEFEAT_LINE" -v r="$DEFEAT_WITH" 'NR==n{print r; next}{print}' "$DEFEAT" > "$DEFEAT.tmp" && mv "$DEFEAT.tmp" "$DEFEAT" + yarn jest "$PROBE" > "$STAMP-armB.log" 2>&1 + B_LINE="$(counts_from "$STAMP-armB.log")" + B="$(consumer_of "$B_LINE")" + restore; trap - EXIT INT TERM +else + : > "$STAMP-armB.log" +fi + +if [ -n "$B" ] && [ "$B" = "$A" ]; then VERDICT="no delta — memo not attributable"; CODE=1 +elif [ -n "$B" ]; then VERDICT="delta measured: $A → $B renders with the memo defeated"; CODE=0 +else VERDICT="baseline only: $A consumer renders"; CODE=0; fi + +HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" +DIRTY="$(git status --porcelain 2>/dev/null | grep -v '^??' | wc -l | tr -d ' ')" +NODE_V="$(node -v 2>/dev/null || echo unknown)" + +cat > "$STAMP.json" <<JSON +{ "probe": "$PROBE", "verdict": "$VERDICT", "exit": $CODE, + "consumer_renders": { "armA": ${A:-null}, "armB": ${B:-null} }, + "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, "node": "$NODE_V" }, + "logs": ["$STAMP-armA.log", "$STAMP-armB.log"] } +JSON + +{ + echo "### C4 — consumer render count" + echo + echo "**Verdict:** $VERDICT" + echo + echo "| Arm | Change | consumer renders |" + echo "|---|---|---|" + echo "| A — as committed | none | ${A:-?} |" + [ -n "$B" ] && echo "| B — memo defeated | \`$DEFEAT:$DEFEAT_LINE\` | $B |" + echo + echo '```console' + echo "\$ yarn jest $PROBE" + echo "$A_LINE" + [ -n "$B_LINE" ] && { echo "\$ yarn jest $PROBE # memo defeated"; echo "$B_LINE"; } + echo '```' + echo + echo "This counts renders of one named consumer across a defined interaction. It is not a count" + echo "of consumers, and a larger consumer count does not imply a larger effect." + echo + echo "<sub>Produced by \`render-count.sh\`; the defeat edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" +} > "$STAMP.md" + +printf 'render-count: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 +exit "$CODE" From 73d4b1734148e454eafb23581ea58ead8ef62ae0 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 10:56:01 -0400 Subject: [PATCH 075/135] Audit override scope, and escalate rather than rule on the critical grants MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds `--override`: overrides are where containment is widened by a person rather than observed by the toolchain, and the decision persists across regenerations, so it outlives the reason for it. On the extension's mv3/main policy: 42 tightened, 83 persisting, 9 narrowable, 2 write. Three things came from reading lavamoat-core/src/mergePolicy.js rather than inferring them, each of which had been wrong: - The effective policy is mergePolicy(generated, override). The two files are DESIGNED not to align — the generated one is regenerated on dependency updates while the override persists — so an override entry absent from the generated policy is the normal case. Calling that "never observed" was alarmist and wrong; 83 of 87 entries are in that state by construction. - `validateHierarchy` throws when both `X` and `X.y` are present, so the first version of the narrowing suggestion would have produced a policy that fails to build. The documented form denies the parent: `"X": false, "X.y": true`. - Escalation flagged every global in any package matching a name hint, labelling `Array` and `Object` as "critical class". Intrinsics are now excluded and the reason string is true of the row it appears on — an escalation list that is mostly noise trains its reader to skip it. Critical grants and write access get RAISE WITH A HUMAN and no verdict. Correctness there depends on intent and threat model, neither of which is in the policy files, and an audit that silently resolves them has substituted a guess for the thing it was asked to check. --- .../scripts/policy-audit.py | 183 ++++++++++++++++-- 1 file changed, 170 insertions(+), 13 deletions(-) diff --git a/domains/security/skills/lavamoat-policy-diligence/scripts/policy-audit.py b/domains/security/skills/lavamoat-policy-diligence/scripts/policy-audit.py index 6c7331e8..c6a39221 100644 --- a/domains/security/skills/lavamoat-policy-diligence/scripts/policy-audit.py +++ b/domains/security/skills/lavamoat-policy-diligence/scripts/policy-audit.py @@ -1,17 +1,56 @@ #!/usr/bin/env python3 -"""Turn a LavaMoat policy base/head pair into a per-grant justification worklist. +"""Turn a LavaMoat policy base/head pair into a per-grant justification worklist, +and audit hand-written overrides for scope. Detection is LavaMoat's job: `@metamaskbot update-policies` regenerates the policy from a real run of the code and CI fails on drift. This script does NOT re-derive or classify that diff — it enumerates every capability newly granted so each can be JUSTIFIED with a permalink to the dependency's own source (accept), or REJECTED where no call site uses it. -Usage: policy-audit.py <base/policy.json> <head/policy.json> +With --override it also audits `policy-override.json`. Per lavamoat-core/src/mergePolicy.js the +effective policy is `mergePolicy(generated, override)`, with priority to stricter decisions in +the override. The two files are DESIGNED not to align: the generated policy is regenerated on +dependency updates while the override persists, which is the whole point of separating them. +So an override entry absent from the generated policy is the normal case, not a finding. + +What the audit reports is therefore scope, not divergence: which entries persist without a +recorded reason, which are broader than the generated policy shows a need for, and which grant +write. Narrowing a whole-object grant requires setting the parent to `false` — `validateHierarchy` +throws if `X` and `X.y` are both present. + +Usage: + policy-audit.py <base/policy.json> <head/policy.json> + [--override <policy-override.json>] [--generated <policy.json>] + Falsifier: a listed grant for which no upstream call site can be found. """ import json import sys +# Capability classes where a wrong call is not recoverable by a follow-up PR. The script +# refuses a verdict on these by design — it reports and escalates. Naming them explicitly +# rather than scoring them keeps the escalation auditable. +CRITICAL = { + "child_process", "fs", "vm", "worker_threads", "module", "process", + "eval", "Function", "WebAssembly", "importScripts", "SharedArrayBuffer", + "fetch", "XMLHttpRequest", "WebSocket", "crypto", "indexedDB", + "localStorage", "sessionStorage", "chrome", "browser", +} +CRITICAL_PKG_HINTS = ("keyring", "vault", "snap", "lavamoat", "seed", "wallet") + +# ECMAScript and DOM intrinsics. Granting these is unremarkable — a package that renders +# anything touches Element and Object — so they never escalate on their own. Without this +# a sensitive-package hint floods the list with `Object`, `String`, `Array`, and an +# escalation list that is mostly noise trains its reader to skip it. +BENIGN = { + "Array", "Object", "String", "Number", "Boolean", "Symbol", "BigInt", "Math", + "JSON", "Date", "RegExp", "Map", "Set", "WeakMap", "WeakSet", "Weakmap", + "Promise", "Error", "TypeError", "Proxy", "Reflect", "Intl", + "Document", "DocumentFragment", "Element", "Event", "EventTarget", "Node", + "NavigateEvent", "NavigationDestination", "Clipboard", "CSS", "Text", + "console", "queueMicrotask", "structuredClone", +} + def resources(path): with open(path) as f: @@ -29,11 +68,78 @@ def newly_granted(head, base): return out +def root_of(cap): + return cap.split(".", 1)[0] + + +def criticality(pkg, cap): + """Return a reason string, or None. The reason is reported verbatim, so it must be + true of THIS row — a package-level hint is not a claim about the capability.""" + root = root_of(cap) + if root in CRITICAL: + return f"critical capability: {root}" + if root in BENIGN: + return None + if any(h in pkg.lower() for h in CRITICAL_PKG_HINTS): + return "non-intrinsic grant in a security-sensitive package" + return None + + +def is_critical(pkg, cap): + return criticality(pkg, cap) is not None + + +def audit_overrides(over, gen): + """Classify each override entry against what the generated policy observed. + + widened granted here, not observed by the toolchain — a human decision needing a reason + tightened explicit false over an observed grant — containment narrowed, no action + broad whole-object grant where only specific members were observed — narrowable + write write access; read may suffice, and only call sites can settle it + """ + widened, tightened, broad, write = [], [], [], [] + for pkg, cfg in over.items(): + gpkg = gen.get(pkg, {}) + for kind, caps in cfg.items(): + if not isinstance(caps, dict): + continue + gcaps = gpkg.get(kind) or {} + for cap, val in caps.items(): + if val == "write": + write.append((pkg, kind, cap)) + continue + if val is False: + if gcaps.get(cap): + tightened.append((pkg, kind, cap)) + continue + if not gcaps.get(cap): + # NOT "never observed" — the generated policy is regenerated + # independently, so absence here is expected. This is only a list of + # entries whose justification lives outside both files. + widened.append((pkg, kind, cap)) + if "." not in cap: + members = sorted( + c for c in gcaps if c.startswith(cap + ".") and gcaps.get(c) + ) + if members and not gcaps.get(cap): + broad.append((pkg, kind, cap, tuple(members))) + return widened, tightened, broad, write + + def main(): - if len(sys.argv) != 3: - sys.exit("usage: policy-audit.py <base/policy.json> <head/policy.json>") - base = resources(sys.argv[1]) - head = resources(sys.argv[2]) + args = sys.argv[1:] + if len(args) < 2: + sys.exit("usage: policy-audit.py <base/policy.json> <head/policy.json> " + "[--override <file>] [--generated <file>]") + base_p, head_p = args[0], args[1] + over_p = gen_p = None + for i, a in enumerate(args): + if a == "--override" and i + 1 < len(args): + over_p = args[i + 1] + if a == "--generated" and i + 1 < len(args): + gen_p = args[i + 1] + + base, head = resources(base_p), resources(head_p) grants = sorted(newly_granted(head, base)) print("PER-GRANT JUSTIFICATION WORKLIST") @@ -44,16 +150,67 @@ def main(): if not grants: print(" (no new grants between base and head — nothing to justify)") + else: + for pkg, kind, cap in grants: + mark = "!" if is_critical(pkg, cap) else " " + print(f" {mark}[ ] {pkg[:38]:40s} {kind[:3]}:{cap:22s}" + " reason: <upstream file#Ln @ tag> verdict: accept|REJECT") + crit = [g for g in grants if is_critical(g[0], g[2])] + print(f"\n {len(grants)} grant(s) to justify" + + (f"; {len(crit)} marked ! for escalation." if crit else ".")) + print(" A grant with no locatable call site is the finding — reject it.") + + if not over_p: return + gen = resources(gen_p) if gen_p else head + over = resources(over_p) + widened, tightened, broad, write = audit_overrides(over, gen) + + print("\n\nOVERRIDE SCOPE AUDIT") + print("=" * 74) + print("Effective policy = mergePolicy(generated, override), stricter decisions winning.") + print("The files are meant to differ: the generated one is regenerated on dependency") + print("updates while the override persists. Entries below are scoped, not \"unobserved\".") + print(f"{len(over)} package(s) overridden.\n") + print(f" tightened {len(tightened):3d} explicit false over an observed grant — containment narrowed") + print(f" persisting {len(widened):3d} in the override only — expected, but each needs a standing reason") + print(f" broad {len(broad):3d} whole-object grant where only members were observed") + print(f" write {len(write):3d} write access — read may suffice") - for pkg, kind, cap in grants: - print( - f" [ ] {pkg[:38]:40s} {kind[:3]}:{cap:22s}" - " reason: <upstream file#Ln @ tag> verdict: accept|REJECT" - ) + if broad: + print("\n\nSUGGESTED TIGHTENINGS — evidence-based, functionality-preserving") + print("-" * 74) + print("A whole-object grant where the generated policy shows only members in use.") + print("`validateHierarchy` REJECTS a policy containing both `X` and `X.y`, so narrowing") + print("requires denying the parent explicitly — that is LavaMoat's documented form:") + print(" \"You could set the parent to false if you intended a less permissive policy.\"") + print("Regenerate and re-run the app after applying; an over-narrowed grant fails loudly.\n") + for pkg, kind, cap, members in sorted(broad)[:20]: + print(f" {pkg}") + narrowed = {cap: False} + narrowed.update({m: True for m in members}) + print(f" now: \"{cap}\": true") + print(f" → " + json.dumps(narrowed)[1:-1]) + if len(broad) > 20: + print(f"\n … {len(broad) - 20} further narrowable grant(s).") - print(f"\n {len(grants)} grant(s) to justify.") - print(" A grant with no locatable call site is the finding — reject it.") + escalate = sorted(set( + [(p, k, c) for p, k, c in widened if is_critical(p, c)] + + [(p, k, c) for p, k, c in write] + )) + if escalate: + write_set = set(write) + print("\n\nRAISE WITH A HUMAN — no verdict offered") + print("-" * 74) + print("These widen a capability class where a wrong call is not recoverable by a") + print("follow-up PR, or grant write where read may suffice. Whether each is correct") + print("depends on intent and threat model, neither of which is in the policy files.") + print("This script stops here deliberately rather than guessing.\n") + for pkg, kind, cap in escalate: + why = "write access" if (pkg, kind, cap) in write_set else criticality(pkg, cap) + print(f" [?] {pkg[:42]:44s} {kind[:3]}:{cap:20s} {why}") + print(f"\n {len(escalate)} decision(s) for a human. An audit that silently resolves") + print(" these has substituted a guess for the thing it was asked to check.") if __name__ == "__main__": From e5b6ce1e097e7a80845944911ab5164cadf25d22 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:06:03 -0400 Subject: [PATCH 076/135] Guard the runners against the failures that masquerade as findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit falsify-probe reported `falsifying, exit 0` for a mutation that broke syntax. Arm B "failed" because the module would not load and zero tests ran — identical to a real falsification by exit code, and it would have published "the suite fails when the mechanism is removed" about a suite that never executed. Arm B must now run the same test count as arm A and fail on assertions; a load error or a dropped count reports "mutation broke the module, nothing falsified". selector-recompute now gates on correctness before reporting counts. The probe captures the selector's value across all three conditions, and a value that moves under a write the selector does not declare as an input fails the run outright. A memoisation change that alters output is a breaking change the count would never reveal. attest-gate gains check 9: the wrapper's verdict must not contradict the artifact it embeds. Comments are assembled by hand around machine output, and the hand-written header is exactly where a "vacuous" result acquires a "proven" label. Verified by relabelling a real comment — blocked. All three verified in both directions: the poisoned mutation is refused and the genuine one still reports falsifying; the mislabelled comment is blocked and the correct one passes. --- .../skills/evidence/scripts/attest-gate.sh | 11 ++++++ .../skills/evidence/scripts/falsify-probe.sh | 35 +++++++++++++++---- .../evidence/scripts/selector-recompute.sh | 34 ++++++++++++++++-- 3 files changed, 70 insertions(+), 10 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index f377697b..13d0bc39 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -72,6 +72,17 @@ else pass "8 verdict is earned" fi +# 9 — the wrapper's verdict must not contradict the artifact it embeds. A comment is +# assembled by hand around machine output, and the hand-written header is exactly where +# a "vacuous" result acquires a "proven" label. +HDR="$(grep -m1 '^\*\*Verdict:\*\*' "$FILE" | tr 'A-Z' 'a-z')" +BODY="$(grep -ioE 'vacuous|value unstable|no delta|nothing falsified|broke the module|substitution silent|probe-failed' "$FILE" | head -1 | tr 'A-Z' 'a-z')" +if printf '%s' "$HDR" | grep -q 'proven' && [ -n "$BODY" ]; then + fail "9 verdict matches artifact" "header claims 'proven' while the embedded artifact reports '$BODY'" +else + pass "9 verdict matches artifact" +fi + if [ -n "$REF" ] && [ -f "$REF" ]; then r=$(grep -coE '!\[|<img|data:image' "$REF"); c=$(grep -coE '!\[|<img|data:image|evidence-artifacts/|Produced by' "$FILE") echo diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh index 8edd34c7..29616922 100755 --- a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -12,11 +12,16 @@ # Emits a captured artifact (JSON + markdown) written by this script, not # transcribed by an operator. Exit code IS the verdict, so CI can gate on it. # -# 0 falsifying arm A passed, arm B failed → the test has power -# 1 vacuous arm A passed, arm B ALSO passed → the test proves nothing -# 2 broken arm A failed → nothing to conclude +# 0 falsifying arm A passed, arm B failed ON ASSERTIONS → the test has power +# 1 vacuous arm A passed, arm B ALSO passed → the test proves nothing +# 2 broken arm A failed, or arm B did not run → nothing to conclude # 3 usage/env error # +# Arm B failing is NOT sufficient. A mutation that breaks syntax fails every test in +# the file, which looks identical to a falsification and is worth nothing: the suite +# never executed. So arm B must run the SAME number of tests as arm A and fail some of +# them. A dropped test count means the mutation broke the module, not the mechanism. +# # Usage: # falsify-probe.sh --test <path> --source <path> --line <n> --replace <text> # [--label <slug>] [--out <dir>] [--runner "<cmd>"] @@ -79,10 +84,13 @@ run_arm() { # $1=logfile ; prints "passed|failed" if $RUNNER "$TEST" > "$1" 2>&1; then echo passed; else echo failed; fi } +total_tests() { sed -n 's/.*Tests:.*[^0-9]\([0-9][0-9]*\) total.*/\1/p' "$1" | head -1; } +load_failed() { grep -qiE "SyntaxError|Cannot find module|Unexpected token|Transform failed" "$1"; } + ARM_A="$(run_arm "$STAMP-armA.log")" if [ "$ARM_A" != "passed" ]; then - VERDICT="broken"; CODE=2; ARM_B="not-run" + VERDICT="baseline-already-failing"; CODE=2; ARM_B="not-run" : > "$STAMP-armB.log" else # Mutate exactly one line. `.bak` form keeps this portable across GNU/BSD sed. @@ -90,7 +98,19 @@ else && mv "$SOURCE.tmp" "$SOURCE" || die "mutation failed" ARM_B="$(run_arm "$STAMP-armB.log")" restore; trap - EXIT INT TERM - if [ "$ARM_B" = "failed" ]; then VERDICT="falsifying"; CODE=0; else VERDICT="vacuous"; CODE=1; fi + A_TOTAL="$(total_tests "$STAMP-armA.log")"; A_TOTAL="${A_TOTAL:-0}" + B_TOTAL="$(total_tests "$STAMP-armB.log")"; B_TOTAL="${B_TOTAL:-0}" + if [ "$ARM_B" != "failed" ]; then + VERDICT="vacuous"; CODE=1 + elif load_failed "$STAMP-armB.log" || [ "$B_TOTAL" -lt "$A_TOTAL" ]; then + # The suite did not execute under mutation, so nothing was falsified. Reported as + # broken rather than falsifying: a module that will not load fails every test, which + # is indistinguishable from a real failure by exit code alone. + VERDICT="mutation broke the module — suite ran $B_TOTAL of $A_TOTAL tests, nothing falsified" + CODE=2 + else + VERDICT="falsifying"; CODE=0 + fi fi summarise() { grep -E '^(Tests|Test Suites):' "$1" 2>/dev/null | tr '\n' ' ' | sed 's/ */ /g'; } @@ -121,9 +141,10 @@ JSON echo "| B — mutant | \`$SOURCE:$LINE\` replaced | \`$B_SUM\` |" echo case "$VERDICT" in - falsifying) echo "The suite **fails when the mechanism is removed** and passes when restored. The test has power." ;; + falsifying) echo "The suite **fails when the mechanism is removed** and passes when restored, running the same $A_TOTAL tests in both arms. The test has power." ;; vacuous) echo "The suite **passes with the mechanism removed**. It does not test what it appears to test." ;; - broken) echo "Arm A did not pass, so arm B was not run. No conclusion." ;; + baseline-already-failing) echo "Arm A did not pass, so arm B was not run. No conclusion." ;; + *) echo "**No conclusion.** $VERDICT — a module that will not load fails every test, which an exit code cannot tell apart from a real falsification." ;; esac [ -n "$FAILED_NAMES" ] && { echo; echo "Failing under mutation:"; echo; printf '%s\n' "$FAILED_NAMES" | sed 's/^/- /'; } echo diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index 54f68bcd..c8cd109f 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -78,13 +78,21 @@ describe('$EXPORT recomputation probe', () => { ($EXPORT as unknown as { resetRecomputations: () => void }).resetRecomputations(); const count = () => ($EXPORT as unknown as { recomputations: () => number }).recomputations(); - for (let i = 0; i < $N; i++) call(base); + const seen: string[] = []; + const snap = (v: unknown) => { try { return JSON.stringify(v); } catch { return '<unserialisable>'; } }; + + for (let i = 0; i < $N; i++) seen.push(snap(call(base))); const a = count(); + const stableIdentical = new Set(seen).size === 1; + const unrelatedSeen: string[] = []; for (let i = 0; i < $N; i++) { - call({ ...base, $SLICE: { ...base.$SLICE, __unrelated__: i } }); + unrelatedSeen.push(snap(call({ ...base, $SLICE: { ...base.$SLICE, __unrelated__: i } }))); } const b = count(); + // A write the selector does not read must not change what it returns. If it does, + // the memoisation is not the story — the selector has an input it does not declare. + const stableUnrelated = new Set(unrelatedSeen).size === 1 && unrelatedSeen[0] === seen[0]; for (let i = 0; i < $N; i++) { call({ ...base, $SLICE: { ...base.$SLICE, $PERTURB: [\`0x\${i}\`] } }); @@ -92,8 +100,14 @@ describe('$EXPORT recomputation probe', () => { const c = count(); // eslint-disable-next-line no-console - console.log(\`RECOMPUTE_PROBE identical=\${a} unrelated=\${b} inputChanged=\${c} n=$N\`); + console.log( + \`RECOMPUTE_PROBE identical=\${a} unrelated=\${b} inputChanged=\${c} n=$N\` + + \` valueStable=\${stableIdentical && stableUnrelated}\`, + ); expect(c).toBeGreaterThanOrEqual(b); + // Correctness gates the measurement: an unstable value makes the count meaningless. + expect(stableIdentical).toBe(true); + expect(stableUnrelated).toBe(true); }); }); PROBEEOF @@ -103,12 +117,17 @@ CODE=$? cleanup; trap - EXIT INT TERM LINE="$(grep -o 'RECOMPUTE_PROBE .*' "$STAMP.log" | head -1)" +STABLE="$(printf '%s' "$LINE" | sed -n 's/.*valueStable=\([a-z]*\).*/\1/p')" A="$(printf '%s' "$LINE" | sed -n 's/.*identical=\([0-9]*\).*/\1/p')" B="$(printf '%s' "$LINE" | sed -n 's/.*unrelated=\([0-9]*\).*/\1/p')" C="$(printf '%s' "$LINE" | sed -n 's/.*inputChanged=\([0-9]*\).*/\1/p')" if [ -z "$A" ]; then VERDICT="probe-failed" +elif [ "$STABLE" = "false" ]; then + # Correctness first. A selector whose value moves under a write it does not read has an + # undeclared input, and no recomputation count means anything until that is resolved. + VERDICT="VALUE UNSTABLE — breaking behaviour, count not meaningful" elif [ "$B" -gt "$A" ]; then VERDICT="recomputes on unrelated writes" else @@ -138,6 +157,15 @@ JSON echo "| Fresh \`$SLICE\` slice, unrelated field | $N | ${B:-?} |" echo "| \`$PERTURB\` changed (a real input) | $N | ${C:-?} |" echo + if [ "$STABLE" = "true" ]; then + echo "**Correctness:** the returned value is identical across all calls above, so the count" + echo "measures memoisation rather than a change in behaviour." + else + echo "**Correctness: FAILED.** The returned value changed under a write the selector does not" + echo "declare as an input. That is a behavioural difference, not a performance one, and it" + echo "makes the recomputation count meaningless — resolve it before reading the numbers." + fi + echo echo '```console' echo "\$ yarn jest <generated probe>" echo "$LINE" From 477009f50f7fc6a4cb92449ab0b41516de5dc6fd Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:12:31 -0400 Subject: [PATCH 077/135] =?UTF-8?q?Add=20`egress-delta.py`=20=E2=80=94=20w?= =?UTF-8?q?hat=20a=20diff=20newly=20exposes,=20and=20what=20it=20stopped?= =?UTF-8?q?=20protecting?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two halves. INTRODUCED lists added lines that send, store, or log off-device, tagged by the sensitivity of the payload near the call. WORSENED lists removed consent gates, sanitisers, sampling gates, and validations — and that half is the reason the script exists, because a deleted protection adds no code and so is invisible to anything that scans what a diff introduces. Verified against two real PRs. On extension#42519 it ranks the compliance call first of sixteen egress sites, tagged `identifier` — the same site found by hand, found here without being told where to look. On extension#43869 it reports five removed consent gates including `if (!canSubmitAnalytics(...))`. Two defects the runs exposed, both of which had it reporting nothing: - Payload sensitivity was read from the egress line alone, but a call and its argument sit on different lines — `submitRequestToBackground<T>(` then `[addresses]`. Now read from a window around the call site. - `\baddress\b` does not match `addresses`, nor `\btoken\b` match `tokenList`. Identifiers appear pluralised and camel-cased far more often than bare, so a closing word boundary missed the entire payload on real call sites. Guards removed and re-added in the same diff are excluded as refactors. No verdict is offered: whether a flow is acceptable depends on disclosure, jurisdiction, and intent, and a screening check a user can decline screens nobody — so an absent consent gate is not automatically a defect. Anything sensitive belongs in a private tracker rather than a public comment. --- .../scripts/egress-delta.py | 183 ++++++++++++++++++ 1 file changed, 183 insertions(+) create mode 100644 domains/security/skills/privacy-egress-diligence/scripts/egress-delta.py diff --git a/domains/security/skills/privacy-egress-diligence/scripts/egress-delta.py b/domains/security/skills/privacy-egress-diligence/scripts/egress-delta.py new file mode 100644 index 00000000..3b9df8e6 --- /dev/null +++ b/domains/security/skills/privacy-egress-diligence/scripts/egress-delta.py @@ -0,0 +1,183 @@ +#!/usr/bin/env python3 +"""Security and privacy surface delta for a diff — what this change newly exposes, +and what protection it removed. + +Two halves, and the second is the one nothing else catches: + + INTRODUCED added lines that send, store, or log something that leaves the + device or outlives the session + WORSENED removed lines that were a consent gate, a sanitiser, a redaction, + or a validation — a protection deleted is a regression that no + "scan the new code" pass can see, because there is no new code + +Reports and escalates. It does not rule on whether a flow is acceptable: that +depends on disclosure, jurisdiction, and intent, none of which are in a diff. +A finding here is a question for a human, and for anything sensitive it belongs +in a private tracker rather than a public comment. + +Usage: egress-delta.py <patch-file> [--context <repo-root>] +Falsifier: an added egress site with no corresponding gate anywhere in its call path. +""" +import re +import sys +from collections import defaultdict + +# Things that move data off-device or persist it beyond the session. +EGRESS = [ + (r"\bfetch\s*\(", "network: fetch"), + (r"\bXMLHttpRequest\b", "network: XHR"), + (r"\bnew WebSocket\s*\(", "network: websocket"), + (r"\bsendBeacon\s*\(", "network: beacon"), + (r"\baxios\.\w+\s*\(", "network: axios"), + (r"\bsubmitRequestToBackground\s*(<[^>]*>)?\s*\(", "background RPC"), + (r"\btrackEvent\s*\(", "telemetry: event"), + (r"\bcaptureException\s*\(|\bcaptureMessage\s*\(", "telemetry: sentry"), + (r"\baddBreadcrumb\s*\(", "telemetry: breadcrumb"), + (r"\bstartSpan\w*\s*\(|\btrace\s*\(\s*\{", "telemetry: span"), + (r"\blocalStorage\.setItem\s*\(|\bsessionStorage\.setItem\s*\(", "storage: web"), + (r"\bchrome\.storage\.\w+\.set\s*\(", "storage: extension"), + (r"\bindexedDB\.open\s*\(", "storage: indexeddb"), + (r"\bconsole\.(log|info|warn|error)\s*\(", "log: console"), +] + +# Identifier-shaped payloads. Presence on an egress line is what makes it interesting. +# Trailing \w* rather than \b: identifiers appear pluralised and camel-cased far more +# often than bare — `addresses`, `accountIds`, `tokenList`. A closing \b silently misses +# every one of those, which is the whole payload on a real call site. +SENSITIVE = [ + (r"\bprivate\w*[Kk]ey|\bmnemonic\w*|\bseed\w*[Pp]hrase", "SECRET"), + (r"\bvault\w*|\bkeyring\w*|\bencryptionKey\w*", "SECRET"), + (r"\baddress\w*|\baccount\w*|\bpublicKey\w*", "identifier"), + (r"\bemail\w*|\bipAddress\w*|\buserId\w*|\bdeviceId\w*", "identifier"), + (r"\bjwt\w*|\btoken\w*|\bbearer\w*|\bapiKey\w*|\bsecret\w*", "credential"), + (r"\bbalance\w*|\btxHash\w*|\btransaction\w*", "activity"), +] + +# Protections whose REMOVAL is the finding. +GUARDS = [ + (r"\buseExternalServices\b|\bbasicFunctionality\b", "basic-functionality gate"), + (r"\bparticipateInMetaMetrics\b|\bcanSubmitAnalytics\b|\boptedIn\b|\boptIn\b", "consent gate"), + (r"\bisEnabled\b|\bfeatureFlag\w*\b|\bremoteFeatureFlags\b", "feature gate"), + (r"\bsanitiz|\bredact|\bmask\b|\bscrub\b|\banonymi", "sanitiser"), + (r"\bvalidate\w*\s*\(|\bassert\w*\s*\(|\bisValid\w*\s*\(", "validation"), + (r"\bbeforeSend\b|\btracesSampleRate\b|\bsampleRate\b", "sampling gate"), + (r"\bencrypt\w*\s*\(|\bhash\w*\s*\(", "encryption/hashing"), +] + +HOST = re.compile(r"https?://([A-Za-z0-9.\-]+)") + + +def parse(patch_path): + """Yield (file, sign, text) for +/- lines, tracking the current file.""" + cur = None + with open(patch_path, errors="replace") as f: + for line in f: + if line.startswith("+++ b/"): + cur = line[6:].strip() + continue + if line.startswith("--- ") or line.startswith("+++ "): + continue + if line.startswith("+") or line.startswith("-"): + yield cur, line[0], line[1:].rstrip("\n") + + +def classify(text, table): + return [label for pat, label in table if re.search(pat, text)] + + +def main(): + if len(sys.argv) < 2: + sys.exit("usage: egress-delta.py <patch-file>") + + introduced = [] # (file, kind, sensitivity, host, text) + worsened = defaultdict(list) # guard -> [(file, text)] + removed_egress = 0 + + # An egress call and its payload are usually on different lines — + # `submitRequestToBackground<T>(` on one, `[addresses]` on the next. Matching a + # single line therefore misses precisely the argument that makes the call + # interesting, so sensitivity is read from a window around the call site. + WINDOW = 3 + rows = [(p, sg, t) for p, sg, t in parse(sys.argv[1]) + if p and not p.endswith((".md", ".json", ".lock", ".snap"))] + + for i, (path, sign, text) in enumerate(rows): + stripped = text.strip() + if not stripped or stripped.startswith(("//", "*", "/*")): + continue + + kinds = classify(text, EGRESS) + if sign == "+" and kinds: + lo, hi = max(0, i - WINDOW), min(len(rows), i + WINDOW + 1) + near = " ".join(t for p2, sg2, t in rows[lo:hi] if p2 == path and sg2 == "+") + sens = classify(near, SENSITIVE) + host = HOST.search(near) + introduced.append((path, kinds[0], sens, host.group(1) if host else None, stripped)) + elif sign == "-" and kinds: + removed_egress += 1 + + if sign == "-": + for g in classify(text, GUARDS): + worsened[g].append((path, stripped)) + + # A guard removed in the same hunk that re-adds it is a refactor, not a regression. + readded = set() + for path, sign, text in parse(sys.argv[1]): + if sign == "+": + for g in classify(text, GUARDS): + readded.add((path, g)) + worsened = {g: [(p, t) for p, t in v if (p, g) not in readded] + for g, v in worsened.items()} + worsened = {g: v for g, v in worsened.items() if v} + + print("SECURITY / PRIVACY SURFACE DELTA") + print("=" * 74) + print("Introduced = added lines that send, store, or log off-device.") + print("Worsened = removed protections. Nothing that scans new code can see these,") + print(" because a deleted guard adds no code.\n") + + if not introduced and not worsened: + print(" (no egress added and no protection removed in this diff)") + return + + if introduced: + print(f"INTRODUCED — {len(introduced)} site(s)") + print("-" * 74) + sens_first = sorted(introduced, key=lambda r: (not r[2], r[0])) + for path, kind, sens, host, text in sens_first[:25]: + tag = ("/".join(sens) if sens else "—") + print(f" [{tag:>12}] {kind:<22} {path}") + print(f" {text[:96]}") + if host: + print(f" → host: {host}") + if len(introduced) > 25: + print(f"\n … {len(introduced) - 25} further site(s).") + print() + + if worsened: + n = sum(len(v) for v in worsened.values()) + print(f"WORSENED — {n} protection(s) removed and not re-added in this diff") + print("-" * 74) + for guard, rows in sorted(worsened.items()): + print(f" {guard} ({len(rows)})") + for path, text in rows[:4]: + print(f" {path}") + print(f" - {text[:92]}") + print() + + if removed_egress: + print(f" ({removed_egress} egress line(s) also removed — a move or a deletion, " + "check before reading the counts above as net-new.)\n") + + print("RAISE WITH A HUMAN — no verdict offered") + print("-" * 74) + print("Whether a flow is acceptable depends on disclosure, jurisdiction, and intent,") + print("none of which are in a diff. A screening check that a user can decline screens") + print("nobody, so an absent consent gate is not automatically a defect — and that is") + print("exactly the judgement this script must not make.") + print() + print("Anything sensitive here belongs in a private tracker, not a public comment.") + + +if __name__ == "__main__": + main() From d84f3048a450852fe162750cdb9e866ab7c41395 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:15:48 -0400 Subject: [PATCH 078/135] Give the orchestrator a runner registry, with limits and synthesis rules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seven runners existed with nothing telling the orchestrator which serves which claim, so routing was left to judgement at the exact point where judgement is what the tooling replaces. The registry pairs every runner with what it CANNOT establish, because that column is where a run stops being evidence: falsify-probe shows a test has power and says nothing about whether the fix is correct; a silent tsc-substitution is not agreement; retention-scan pairs by name and cannot show the release site is reachable; policy-audit and egress-delta cannot rule on acceptability at all. Synthesis rules, in priority order: a lead lane is required and a corroborator never substitutes for one; correctness gates measurement, since a performance number over changed behaviour is a missed regression rather than a result; an exit-2 from any runner caps the whole run at unproven and is reported on its own line rather than averaged away; security and privacy findings route privately whatever the other lanes say; and the uncovered part of the claim is named in the artifact so the covered part cannot imply coverage. Records a known gap rather than papering over it. No runner checks a diff against an architectural decision record. A merged PR added deeplinks accepting unsigned parameters — justified as "read-only screens, so unsigned routing params are safe" — and was reverted. Signed links skip the warning interstitial, so an unsigned parameter inherits the signature's trust without being covered by it, which is exploitable whether or not the destination writes anything. Nothing in the table would have caught that: it is a rule in a document, violated by code that looks unremarkable. ADR-governed surfaces route to a human until a conformance runner exists, and the artifact must say so. --- domains/pr-workflow/skills/evidence/skill.md | 50 ++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 55a53d38..9380b946 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -294,6 +294,56 @@ This is phase 0 of [`/attest`](https://github.com/MajorLift/Reprise); phases 1 a `/outframe ‖ /missing ‖ /press` then `/trim` to fresh instances, because those passes cannot be self-run — the author is positionally the wrong reader. +### Runner registry — what each establishes, and what it cannot + +Route a claim to a runner by what the claim asserts. **Read the limit column before quoting a +result**: every runner has a shape of claim it cannot reach, and reporting past that line is how +a run stops being evidence. + +| Runner | Establishes | Cannot establish | +|---|---|---| +| `falsify-probe.sh` | a test fails when its mechanism is removed | that the fix is *correct* — only that the test has power | +| `selector-recompute.sh` | recomputation counts across three input conditions | component render counts; a selector without `.recomputations()` | +| `render-count.sh` | renders of one named consumer over one interaction | that other consumers behave the same; needs a hand-written probe | +| `tsc-substitution.sh` | a hand-written type disagrees with its source | agreement — **a silent arm B means the probe was too weak** | +| `retention-scan.py` | acquire/release pairing within one file | that the release site is *reachable* from the acquire | +| `policy-audit.py` | capability delta and override scope | whether a grant is acceptable — intent is not in the files | +| `egress-delta.py` | egress added, protections removed | whether a flow is acceptable, or what happens off-diff | +| `capture.sh` | a verbatim artifact for any command | any verdict — the caller states it or none is claimed | +| `attest-gate.sh` | eight mechanical publication checks | whether the claim under test was the right one to test | + +Exit codes are uniform: `0` the checked property holds · `1` it does not · `2` no conclusion +available · `3` usage error. **A `2` from any runner caps the whole run at unproven** — one +inconclusive arm is not offset by another lane passing. + +### Synthesising a run from several runners + +1. **Lead lane first.** Pick the runner whose output *is* the claim. A corroborator strengthens + a lead; it never substitutes for one. +2. **Correctness gates measurement.** `selector-recompute` fails outright on an unstable value, + and that ordering generalises: a performance number over changed behaviour is not a + performance result, it is a missed regression. +3. **A `2` is load-bearing.** Report it as its own line. Averaging it away, or quoting the lanes + that passed, converts "we could not tell" into "it is fine". +4. **Security and privacy findings route privately** regardless of what the other lanes say. A + green performance lane does not make an egress finding publishable here. +5. **State the residue.** Name the part of the claim no runner reached, in the artifact, rather + than letting the covered part imply coverage. + +### Known gap: conformance to a written decision + +No runner here checks a diff against an architectural decision record. That class is real and +expensive — a merged PR added deeplinks accepting unsigned parameters, justified as "read-only +screens, so unsigned routing params are safe", and was reverted after review. The justification +misreads the model: signed links skip the warning interstitial, so an unsigned parameter inherits +the signature's trust without being covered by it, which is exploitable regardless of whether the +destination writes anything. + +Nothing in the table above would have found that. It is not a measurement, a count, or a diff +delta — it is a rule stated in a document, violated by code that looks unremarkable. Until a +conformance runner exists, **route ADR-governed surfaces to a human reviewer and say in the +artifact that you did**. + ### Canonical output shape Every validation-run output — PR comment *or* PR-body section — uses exactly this, so re-runs From 0d8935f3e4ac22308635cb6abb171413a1f2ade8 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:30:23 -0400 Subject: [PATCH 079/135] State the bar: float concerns, do not close them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A run succeeds when it puts concerns, falsifiers, and avenues of deeper inquiry in front of a reviewer. Not every concern caught, none required to reach a conclusion. Lower than being right; much higher than staying silent unless certain. Replaces the "known gap" framing, which had the bar wrong. That no runner checks ADR conformance is not a deficiency to apologise for — a run naming the governed surface and its falsifier has done its job while resolving nothing. Held to correctness-to-conclusion the tooling is useless precisely where review matters most, because those cases turn on intent, threat model, or a written decision that is not in the diff, and a tool held to that bar either goes silent or guesses. Guards the obvious failure mode: floating a concern still requires naming what would settle it. "This might be unsafe" is noise; "unsigned parameters on a signed link skip the interstitial, check whether the signature covers them" is actionable. The difference is whether the next step is stated. Keeps the deeplink revert as the worked case, now as an illustration of a sufficient run rather than a missed one. --- domains/pr-workflow/skills/evidence/skill.md | 47 ++++++++++++++------ 1 file changed, 34 insertions(+), 13 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 9380b946..7c7d1a72 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -330,19 +330,40 @@ inconclusive arm is not offset by another lane passing. 5. **State the residue.** Name the part of the claim no runner reached, in the artifact, rather than letting the covered part imply coverage. -### Known gap: conformance to a written decision - -No runner here checks a diff against an architectural decision record. That class is real and -expensive — a merged PR added deeplinks accepting unsigned parameters, justified as "read-only -screens, so unsigned routing params are safe", and was reverted after review. The justification -misreads the model: signed links skip the warning interstitial, so an unsigned parameter inherits -the signature's trust without being covered by it, which is exploitable regardless of whether the -destination writes anything. - -Nothing in the table above would have found that. It is not a measurement, a count, or a diff -delta — it is a rule stated in a document, violated by code that looks unremarkable. Until a -conformance runner exists, **route ADR-governed surfaces to a human reviewer and say in the -artifact that you did**. +### The bar: float concerns, do not close them + +A run succeeds when it puts **concerns, falsifiers, and avenues of deeper inquiry** in front of a +reviewer. It is not required to catch every concern, resolve each correctly, or carry any to a +conclusion. That is a lower bar than being right, and a much higher one than staying silent +unless certain. + +Three consequences worth being explicit about: + +**A coverage gap is not a failed run.** No runner here checks a diff against an architectural +decision record. A run that says *"this touches deeplinks, which are ADR-governed; the falsifier +is whether these parameters are covered by the signature"* has done its job while resolving +nothing. + +**Incomplete analysis is reportable, not suppressible.** Withholding anything short of fully +established throws away the run's actual product. An unresolved concern with a named falsifier is +the deliverable. + +**This does not license speculation.** Floating a concern still requires naming what would settle +it. *"This might be unsafe"* is noise. *"Unsigned parameters on a signed link skip the +interstitial — check whether the signature covers them"* is actionable. The difference is whether +the next step is stated. + +The runners raise questions with evidence attached; they are not oracles. Their limits are +publishable content, which is why the table above lists what each cannot establish. + +**Worked case.** A merged PR added deeplinks accepting unsigned parameters, justified as +"read-only screens, so unsigned routing params are safe". It was reverted after review. The +justification misreads the model: signed links skip the warning interstitial, so an unsigned +parameter inherits the signature's trust without being covered by it — exploitable whether or not +the destination writes anything, via navigation hijacking, request forgery, phishing through +trusted chrome, or attribution poisoning. No runner would have caught it. A run that merely +flagged *"deeplink surface, ADR-0011 governs parameter signing, is this param in the signed +set?"* would have been enough. ### Canonical output shape From 6bdffd13afc2c2e588ab97399b81f10572f12a89 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:35:19 -0400 Subject: [PATCH 080/135] Report what each measurement runner did not cover A runner's job is to put concerns in front of a reviewer, not to close them. The four measurement runners exited 0 or 1 and said nothing about the surface they left unmeasured, which reads as a clean bill of health for the whole mechanism rather than for the one property tested. --- .../pr-workflow/skills/evidence/scripts/falsify-probe.sh | 8 ++++++++ .../pr-workflow/skills/evidence/scripts/render-count.sh | 5 +++++ .../skills/evidence/scripts/selector-recompute.sh | 5 +++++ .../skills/evidence/scripts/tsc-substitution.sh | 5 +++++ 4 files changed, 23 insertions(+) diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh index 29616922..cd46cf9c 100755 --- a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -148,6 +148,14 @@ JSON esac [ -n "$FAILED_NAMES" ] && { echo; echo "Failing under mutation:"; echo; printf '%s\n' "$FAILED_NAMES" | sed 's/^/- /'; } echo + echo "**Open for review** — this run mutated one line of one file. It says nothing about" + echo "other paths into the same mechanism, whether the mechanism is reachable in production," + echo "or whether the behaviour it guards is the right behaviour. A falsifying test proves the" + echo "test has power, not that the fix is correct." + case "$VERDICT" in + vacuous) echo "Worth a look: the mechanism is unguarded by this suite — what else depends on it?" ;; + esac + echo echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" } > "$STAMP.md" diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh index fff2c0d1..a0784724 100755 --- a/domains/pr-workflow/skills/evidence/scripts/render-count.sh +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -109,6 +109,11 @@ JSON echo "This counts renders of one named consumer across a defined interaction. It is not a count" echo "of consumers, and a larger consumer count does not imply a larger effect." echo + echo "**Open for review** — one named consumer, one interaction. Other consumers of the same" + echo "provider are unmeasured, and a consumer that renders once here may render freely under" + echo "an interaction this probe does not perform. The probe also does not check that the" + echo "consumer renders the same OUTPUT, only that it renders fewer times." + echo echo "<sub>Produced by \`render-count.sh\`; the defeat edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" } > "$STAMP.md" diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index c8cd109f..e08788bb 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -171,6 +171,11 @@ JSON echo "$LINE" echo '```' echo + echo "**Open for review** — measured against one fixture with one perturbed key. A selector" + echo "unmoved here can still recompute under state this fixture does not reach, and the count" + echo "says nothing about the cost of each recomputation. Worth a look if the fixture is thin" + echo "relative to the shapes this selector sees in production." + echo echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Log: \`$STAMP.log\`.</sub>" } > "$STAMP.md" diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh index 7bb81c81..6fe4e987 100755 --- a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -128,6 +128,11 @@ JSON echo "subtracted, not disqualifying — only errors new under substitution are the finding.</sub>" fi echo + echo "**Open for review** — the compiler answers only what the probe asks. A silent arm B means" + echo "no call site in this tree distinguishes the two shapes, which is not agreement: a" + echo "divergence reachable only at runtime, or only from a caller outside this repo, will not" + echo "appear here. Worth a look at whether the authoritative type is itself correct." + echo echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" echo } > "$STAMP.md" From 30a03b6ae3b2c078900567ca5dd4c0b260e57140 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:36:43 -0400 Subject: [PATCH 081/135] Require a validation run to float something for review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `capture.sh` gains `--open`, stated by the caller like `--verdict` and never inferred; when omitted the artifact says so rather than reading as full coverage. `attest-gate.sh` gains check 10, the positive counterpart to check 6 — check 6 rejects handing the reader the run's own unfinished work, check 10 rejects an artifact that names no limit at all. --- .../skills/evidence/scripts/attest-gate.sh | 13 ++++++++++++- .../skills/evidence/scripts/capture.sh | 18 ++++++++++++++++-- 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 13d0bc39..4ceeed83 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -4,7 +4,7 @@ # # Everything checkable is checked before anything is asked of a model, because a # model asked "is this good evidence?" answers from inside the frame that produced -# the text. These eight are greppable, so they are not a matter of judgement. +# the text. These are greppable, so they are not a matter of judgement. # # Usage: attest-gate.sh <artifact.md> [--reference <showcase.html>] # @@ -83,6 +83,17 @@ else pass "9 verdict matches artifact" fi +# 10 — the positive counterpart to check 6. A run succeeds by putting concerns in front +# of a reviewer, so an artifact that floats nothing has reported only what it happened to +# measure and called that the whole picture. This is NOT satisfied by a "what would close +# it" section, which check 6 rejects: that hands the reader the run's own unfinished work, +# whereas this names a limit or a question the run is right to leave open. +if hasi 'open for review|raise with a human|falsifier|worth a look|left unmeasured|not covered by this run|no verdict offered'; then + pass "10 floats something for review" +else + fail "10 floats something for review" "no limit, open question, or falsifier named — an artifact that floats nothing implies its measurement was the whole surface" +fi + if [ -n "$REF" ] && [ -f "$REF" ]; then r=$(grep -coE '!\[|<img|data:image' "$REF"); c=$(grep -coE '!\[|<img|data:image|evidence-artifacts/|Produced by' "$FILE") echo diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh index fc18c4fd..e40e2ef5 100755 --- a/domains/pr-workflow/skills/evidence/scripts/capture.sh +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -9,11 +9,16 @@ # # This wraps any command so the ARTIFACT is written by the tool. Nothing is retyped. # -# capture.sh --label <slug> --lane <id> --claim "<under test>" [--verdict <word>] -- <cmd...> +# capture.sh --label <slug> --lane <id> --claim "<under test>" [--verdict <word>] +# [--open "<what this run leaves open>"] -- <cmd...> # # --verdict is stated by the caller, never inferred from the exit code: a wrapped # tool's exit convention is its own, and guessing prints "pass" over real findings. # +# --open is the same discipline pointed the other way. A run succeeds by putting +# concerns in front of a reviewer, not by closing them, so what the wrapped tool +# could not reach is publishable content. Omitting it is recorded, not hidden. +# # Emits, under --out (default evidence-artifacts/): # <label>.log raw stdout+stderr of the command, unmodified # <label>.json machine-readable: verdict, exit code, env pin, claim @@ -27,7 +32,7 @@ # -- python3 retention-scan.py ui/store/background-connection.ts pr.patch set -uo pipefail -OUT_DIR="evidence-artifacts"; LABEL=""; LANE=""; CLAIM=""; MAXLOG=120; VERDICT="" +OUT_DIR="evidence-artifacts"; LABEL=""; LANE=""; CLAIM=""; MAXLOG=120; VERDICT=""; OPEN="" die() { printf 'capture: %s\n' "$1" >&2; exit 3; } while [ $# -gt 0 ]; do @@ -36,6 +41,7 @@ while [ $# -gt 0 ]; do --lane) LANE="${2:-}"; shift 2 ;; --claim) CLAIM="${2:-}"; shift 2 ;; --verdict) VERDICT="${2:-}"; shift 2 ;; + --open) OPEN="${2:-}"; shift 2 ;; --out) OUT_DIR="${2:-}"; shift 2 ;; --max-log-lines) MAXLOG="${2:-}"; shift 2 ;; -h|--help) sed -n '2,26p' "$0"; exit 0 ;; @@ -78,6 +84,7 @@ cat > "$STAMP.json" <<JSON "claim": $(jstr "$CLAIM"), "command": $(jstr "$CMD_STR"), "verdict": "$VERDICT", + "open_for_review": $(jstr "$OPEN"), "exit": $CODE, "log": "$STAMP.log", "log_lines": $LINES, @@ -105,6 +112,13 @@ JSON fi echo '```' echo + if [ -n "$OPEN" ]; then + echo "**Open for review** — $OPEN" + else + echo "**Open for review** — none stated. This tool answered one question; what it does" + echo "not cover was not recorded, which is not the same as it covering everything." + fi + echo echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. Raw log: \`$STAMP.log\`.</sub>" } > "$STAMP.md" From d5c6a61d0db01e65e3ba2617853813ab33e710de Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:53:20 -0400 Subject: [PATCH 082/135] Stop the runners from cutting the part a reader needed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `capture.sh` elided the tail when output exceeded the line budget, which drops `policy-audit.py`'s RAISE WITH A HUMAN section and leaves a wall of checkboxes in its place — a tool that escalates does it last. It now elides the middle. `render-count.sh` hard-coded arm B as "memo defeated"; when the PR under test is the suspect rather than the fix, that prints the reading backwards, so the label is caller-stated via `--arm-b`. --- .../skills/evidence/scripts/capture.sh | 11 ++++++++-- .../skills/evidence/scripts/render-count.sh | 21 ++++++++++++------- 2 files changed, 23 insertions(+), 9 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh index e40e2ef5..954e01a8 100755 --- a/domains/pr-workflow/skills/evidence/scripts/capture.sh +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -105,8 +105,15 @@ JSON echo '```console' echo "\$ $CMD_STR" if [ "$LINES" -gt "$MAXLOG" ]; then - head -n "$MAXLOG" "$STAMP.log" - echo "… $((LINES - MAXLOG)) further lines in $STAMP.log" + # Elide the middle, never the end. A tool that escalates does it last: + # policy-audit.py prints a per-grant worklist first and its RAISE WITH A HUMAN + # section at the bottom, so head-truncation cuts exactly the rows that needed a + # reader and leaves a wall of checkboxes in their place. + H=$(( MAXLOG * 2 / 3 )); T=$(( MAXLOG - H )) + head -n "$H" "$STAMP.log" + printf '\n… %s lines elided from the middle — full output in %s\n\n' \ + "$((LINES - MAXLOG))" "$STAMP.log" + tail -n "$T" "$STAMP.log" else cat "$STAMP.log" fi diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh index a0784724..d2a550c5 100755 --- a/domains/pr-workflow/skills/evidence/scripts/render-count.sh +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -10,12 +10,17 @@ # # Generates a probe that mounts a provider with a counting consumer, forces the # parent to re-render N times with the memoised value unchanged, and reports the -# consumer's render count. Arm B re-runs with the memo defeated, so the delta is +# consumer's render count. Arm B re-runs with one line changed, so the delta is # attributable rather than assumed. # # Usage: # render-count.sh --probe <probe.test.tsx> [--defeat <file> --defeat-line <n> --defeat-with <text>] -# [--label <slug>] [--out <dir>] +# [--arm-b <label>] [--label <slug>] [--out <dir>] +# +# Arm B is "the memo defeated" by default, which is the shape when a PR ADDS +# memoisation. When a PR is the one under suspicion the arms invert — arm B applies +# the candidate fix — and calling that "defeated" prints the reading backwards. So +# the label is caller-stated, like every other verdict word in this suite. # # The probe is supplied rather than generated: a provider's mount requirements # are specific to the component, and a generated one would either be wrong or @@ -31,6 +36,7 @@ set -uo pipefail OUT_DIR="evidence-artifacts"; LABEL=""; PROBE=""; DEFEAT=""; DEFEAT_LINE=""; DEFEAT_WITH="" +ARM_B="memo defeated" die() { printf 'render-count: %s\n' "$1" >&2; exit 3; } while [ $# -gt 0 ]; do @@ -39,6 +45,7 @@ while [ $# -gt 0 ]; do --defeat) DEFEAT="${2:-}"; shift 2 ;; --defeat-line) DEFEAT_LINE="${2:-}"; shift 2 ;; --defeat-with) DEFEAT_WITH="${2:-}"; shift 2 ;; + --arm-b) ARM_B="${2:-}"; shift 2 ;; --label) LABEL="${2:-}"; shift 2 ;; --out) OUT_DIR="${2:-}"; shift 2 ;; -h|--help) sed -n '2,30p' "$0"; exit 0 ;; @@ -75,8 +82,8 @@ else : > "$STAMP-armB.log" fi -if [ -n "$B" ] && [ "$B" = "$A" ]; then VERDICT="no delta — memo not attributable"; CODE=1 -elif [ -n "$B" ]; then VERDICT="delta measured: $A → $B renders with the memo defeated"; CODE=0 +if [ -n "$B" ] && [ "$B" = "$A" ]; then VERDICT="no delta — arm B changed nothing measurable"; CODE=1 +elif [ -n "$B" ]; then VERDICT="delta measured: $A → $B renders with $ARM_B"; CODE=0 else VERDICT="baseline only: $A consumer renders"; CODE=0; fi HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" @@ -98,12 +105,12 @@ JSON echo "| Arm | Change | consumer renders |" echo "|---|---|---|" echo "| A — as committed | none | ${A:-?} |" - [ -n "$B" ] && echo "| B — memo defeated | \`$DEFEAT:$DEFEAT_LINE\` | $B |" + [ -n "$B" ] && echo "| B — $ARM_B | \`$DEFEAT:$DEFEAT_LINE\` | $B |" echo echo '```console' echo "\$ yarn jest $PROBE" echo "$A_LINE" - [ -n "$B_LINE" ] && { echo "\$ yarn jest $PROBE # memo defeated"; echo "$B_LINE"; } + [ -n "$B_LINE" ] && { echo "\$ yarn jest $PROBE # $ARM_B"; echo "$B_LINE"; } echo '```' echo echo "This counts renders of one named consumer across a defined interaction. It is not a count" @@ -114,7 +121,7 @@ JSON echo "an interaction this probe does not perform. The probe also does not check that the" echo "consumer renders the same OUTPUT, only that it renders fewer times." echo - echo "<sub>Produced by \`render-count.sh\`; the defeat edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" + echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" } > "$STAMP.md" printf 'render-count: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 From a3f15900c6d03c0e19ff6e07446e09a09e84729b Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 11:55:55 -0400 Subject: [PATCH 083/135] Reference artifacts by name in the publishable block An absolute `--out` path put the operator's home directory into the `<sub>` line of every artifact, which is the one part of the run that gets pasted into a public comment. The `.json` keeps full paths for machine use. --- domains/pr-workflow/skills/evidence/scripts/capture.sh | 2 +- domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh | 2 +- domains/pr-workflow/skills/evidence/scripts/render-count.sh | 2 +- .../pr-workflow/skills/evidence/scripts/selector-recompute.sh | 2 +- domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh index 954e01a8..027f9eff 100755 --- a/domains/pr-workflow/skills/evidence/scripts/capture.sh +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -126,7 +126,7 @@ JSON echo "not cover was not recorded, which is not the same as it covering everything." fi echo - echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. Raw log: \`$STAMP.log\`.</sub>" + echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. Raw log: \`${STAMP##*/}.log\`.</sub>" } > "$STAMP.md" printf 'capture: %s (exit %s)\n %s\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.log" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh index cd46cf9c..5b1531da 100755 --- a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -156,7 +156,7 @@ JSON vacuous) echo "Worth a look: the mechanism is unguarded by this suite — what else depends on it?" ;; esac echo - echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" + echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes. Logs: \`${STAMP##*/}-armA.log\`, \`${STAMP##*/}-armB.log\`.</sub>" } > "$STAMP.md" printf 'falsify-probe: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh index d2a550c5..511a0e5a 100755 --- a/domains/pr-workflow/skills/evidence/scripts/render-count.sh +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -121,7 +121,7 @@ JSON echo "an interaction this probe does not perform. The probe also does not check that the" echo "consumer renders the same OUTPUT, only that it renders fewer times." echo - echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" + echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Logs: \`${STAMP##*/}-armA.log\`, \`${STAMP##*/}-armB.log\`.</sub>" } > "$STAMP.md" printf 'render-count: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index e08788bb..d0f4c7e5 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -176,7 +176,7 @@ JSON echo "says nothing about the cost of each recomputation. Worth a look if the fixture is thin" echo "relative to the shapes this selector sees in production." echo - echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Log: \`$STAMP.log\`.</sub>" + echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Log: \`${STAMP##*/}.log\`.</sub>" } > "$STAMP.md" printf 'selector-recompute: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh index 6fe4e987..38ba7fa0 100755 --- a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -133,7 +133,7 @@ JSON echo "divergence reachable only at runtime, or only from a caller outside this repo, will not" echo "appear here. Worth a look at whether the authoritative type is itself correct." echo - echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. Logs: \`$STAMP-armA.log\`, \`$STAMP-armB.log\`.</sub>" + echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. Logs: \`${STAMP##*/}-armA.log\`, \`${STAMP##*/}-armB.log\`.</sub>" echo } > "$STAMP.md" From d55b32430f2fb98814fb4ab0d1dc22e5f374061a Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 12:02:36 -0400 Subject: [PATCH 084/135] Stop the gate failing runs whose environment is not the repo's MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Check 4 recognised only repo-toolchain pins — head SHA, lockfile hash, node version — so a browser-memory run pinned to `Firefox 153.0 headless` was told its pinned environment was unpinned. Check 10's vocabulary missed a run that stated its limit as "what it does not establish". Both were false negatives against real published runs, not missing content. --- .../skills/evidence/scripts/attest-gate.sh | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 4ceeed83..a3dcf266 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -41,9 +41,13 @@ hasre '^\*\*Verdict:\*\*.*\*\*Claim:\*\*' \ && pass "3 verdict line" \ || fail "3 verdict line" "no '**Verdict:** … — **Claim:** …' — valence is not legible at a glance" -hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `' \ +# A run outside the repo's toolchain pins a different thing. A browser-memory lane +# names "Firefox 153.0"; a repo lane names a head SHA and a lockfile hash. Both are +# pins, and a check that only knows the second one fails every run of the first — +# telling an author their pinned environment is unpinned. +hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `|[Ff]irefox [0-9]+\.[0-9]|[Cc]hrom(e|ium) [0-9]+\.|[Ss]afari [0-9]+\.|[Nn]ode v?[0-9]+\.[0-9]' \ && pass "4 environment pinned" \ - || fail "4 environment pinned" "no head SHA, toolchain version, or lockfile hash" + || fail "4 environment pinned" "no head SHA, lockfile hash, or pinned toolchain/browser version" # 5 — the one that matters. A tool-written log, a run link, or an image; not typed prose. if hasre '!\[|<img|data:image|actions/runs|/gist\.|evidence-artifacts/|Produced by '; then @@ -88,7 +92,12 @@ fi # measure and called that the whole picture. This is NOT satisfied by a "what would close # it" section, which check 6 rejects: that hands the reader the run's own unfinished work, # whereas this names a limit or a question the run is right to leave open. -if hasi 'open for review|raise with a human|falsifier|worth a look|left unmeasured|not covered by this run|no verdict offered'; then +# +# The vocabulary is a fixed list because this phase asks no model anything. That makes +# it blind to a limit phrased outside the list — a real run stated its limit as "what it +# does not establish" and the check called it absent. Add phrases when that happens; +# judging whether the stated limit is substantive is the dispatched passes' job. +if hasi 'open for review|raise with a human|falsifier|worth a look|left unmeasured|not covered by this run|no verdict offered|does not establish|what it does not|cannot attribute'; then pass "10 floats something for review" else fail "10 floats something for review" "no limit, open question, or falsifier named — an artifact that floats nothing implies its measurement was the whole surface" From 4ae53736d32197f09fbc288dcfc850796528511e Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 12:14:54 -0400 Subject: [PATCH 085/135] Let the caller say where its tool puts the finding Two thirds head, one third tail is a guess. `policy-audit.py` prints 1200 lines of worklist before its escalation section, so the default budget spent itself on checkboxes; `--head-lines 10 --tail-lines 36` cut the embedded block from 9K to 5K with more of the part a reader needs. The elision notice also printed an absolute path, same leak the `<sub>` line had. --- .../skills/evidence/scripts/capture.sh | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh index 027f9eff..f51d8391 100755 --- a/domains/pr-workflow/skills/evidence/scripts/capture.sh +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -10,7 +10,8 @@ # This wraps any command so the ARTIFACT is written by the tool. Nothing is retyped. # # capture.sh --label <slug> --lane <id> --claim "<under test>" [--verdict <word>] -# [--open "<what this run leaves open>"] -- <cmd...> +# [--open "<what this run leaves open>"] +# [--max-log-lines N | --head-lines N --tail-lines N] -- <cmd...> # # --verdict is stated by the caller, never inferred from the exit code: a wrapped # tool's exit convention is its own, and guessing prints "pass" over real findings. @@ -33,6 +34,7 @@ set -uo pipefail OUT_DIR="evidence-artifacts"; LABEL=""; LANE=""; CLAIM=""; MAXLOG=120; VERDICT=""; OPEN="" +HEADL=""; TAILL="" die() { printf 'capture: %s\n' "$1" >&2; exit 3; } while [ $# -gt 0 ]; do @@ -44,6 +46,11 @@ while [ $# -gt 0 ]; do --open) OPEN="${2:-}"; shift 2 ;; --out) OUT_DIR="${2:-}"; shift 2 ;; --max-log-lines) MAXLOG="${2:-}"; shift 2 ;; + # Two thirds head / one third tail is a guess about where the finding is. For a + # tool that escalates at the bottom the useful split is the other way round, and + # the caller knows which shape its tool has. + --head-lines) HEADL="${2:-}"; shift 2 ;; + --tail-lines) TAILL="${2:-}"; shift 2 ;; -h|--help) sed -n '2,26p' "$0"; exit 0 ;; --) shift; break ;; *) die "unknown argument: $1 (did you forget -- before the command?)" ;; @@ -104,15 +111,16 @@ JSON echo echo '```console' echo "\$ $CMD_STR" - if [ "$LINES" -gt "$MAXLOG" ]; then + BUDGET=$(( ${HEADL:-0} + ${TAILL:-0} )); [ "$BUDGET" -gt 0 ] || BUDGET="$MAXLOG" + if [ "$LINES" -gt "$BUDGET" ]; then # Elide the middle, never the end. A tool that escalates does it last: # policy-audit.py prints a per-grant worklist first and its RAISE WITH A HUMAN # section at the bottom, so head-truncation cuts exactly the rows that needed a # reader and leaves a wall of checkboxes in their place. - H=$(( MAXLOG * 2 / 3 )); T=$(( MAXLOG - H )) + H="${HEADL:-$(( MAXLOG * 2 / 3 ))}"; T="${TAILL:-$(( MAXLOG - H ))}" head -n "$H" "$STAMP.log" printf '\n… %s lines elided from the middle — full output in %s\n\n' \ - "$((LINES - MAXLOG))" "$STAMP.log" + "$((LINES - BUDGET))" "${STAMP##*/}.log" tail -n "$T" "$STAMP.log" else cat "$STAMP.log" From d32bdf703525234fc05ce5bb505c4b610aa15388 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 12:25:08 -0400 Subject: [PATCH 086/135] Write for the reviewer, not for a user of this skill The published runs led with a summary table and buried the captured blocks, which inverts what a reviewer opens the comment for, and each exhibit carried a lane id, a paragraph of generic instrument limits, and two log filenames pointing at files nobody can open. Three consecutive blocks meant the same boilerplate three times. The exhibits now lead and should outweigh the prose. Generic limits go to stderr and the `.json` so the orchestrator can read them and synthesise one question about the diff in front of it. Restating an artifact's number in prose is called out as its own defect: a figure that appears only in a typed sentence is a figure on the author's word. --- .../skills/evidence/scripts/capture.sh | 21 ++++----- .../skills/evidence/scripts/falsify-probe.sh | 18 ++++---- .../skills/evidence/scripts/render-count.sh | 16 ++++--- .../evidence/scripts/selector-recompute.sh | 16 ++++--- .../evidence/scripts/tsc-substitution.sh | 16 ++++--- domains/pr-workflow/skills/evidence/skill.md | 43 ++++++++++++++++++- 6 files changed, 89 insertions(+), 41 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh index f51d8391..60666fa6 100755 --- a/domains/pr-workflow/skills/evidence/scripts/capture.sh +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -102,9 +102,9 @@ JSON { if [ "$VERDICT" = "completed" ]; then - echo "### ${LANE:+$LANE — }ran to completion (exit $CODE) — read the output, no verdict asserted" + echo "### Ran to completion (exit $CODE) — read the output, no verdict asserted" else - echo "### ${LANE:+$LANE — }\`$VERDICT\` (exit $CODE)" + echo "### \`$VERDICT\` (exit $CODE)" fi echo echo "**Claim under test:** $CLAIM" @@ -127,15 +127,16 @@ JSON fi echo '```' echo - if [ -n "$OPEN" ]; then - echo "**Open for review** — $OPEN" - else - echo "**Open for review** — none stated. This tool answered one question; what it does" - echo "not cover was not recorded, which is not the same as it covering everything." - fi - echo - echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. Raw log: \`${STAMP##*/}.log\`.</sub>" + echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`.</sub>" } > "$STAMP.md" printf 'capture: %s (exit %s)\n %s\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.log" "$STAMP.json" "$STAMP.md" >&2 +# Stated limits reach the orchestrator, not the pasted exhibit: one open question per +# comment, about this diff, beats the same sentence repeated under every block. +if [ -n "$OPEN" ]; then + printf 'limits: %s\n' "$OPEN" >&2 +else + printf 'limits: none stated. This tool answered one question; what it does not cover was +not recorded, which is not the same as it covering everything.\n' >&2 +fi exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh index 5b1531da..2677e199 100755 --- a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -148,16 +148,16 @@ JSON esac [ -n "$FAILED_NAMES" ] && { echo; echo "Failing under mutation:"; echo; printf '%s\n' "$FAILED_NAMES" | sed 's/^/- /'; } echo - echo "**Open for review** — this run mutated one line of one file. It says nothing about" - echo "other paths into the same mechanism, whether the mechanism is reachable in production," - echo "or whether the behaviour it guards is the right behaviour. A falsifying test proves the" - echo "test has power, not that the fix is correct." - case "$VERDICT" in - vacuous) echo "Worth a look: the mechanism is unguarded by this suite — what else depends on it?" ;; - esac - echo - echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes. Logs: \`${STAMP##*/}-armA.log\`, \`${STAMP##*/}-armB.log\`.</sub>" + echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes.</sub>" } > "$STAMP.md" printf 'falsify-probe: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 +# The limits below are identical on every run: they describe the instrument, not the +# change under review. Pasted into a PR comment they read as boilerplate to a reviewer +# who has no stake in this tooling, so they go to stderr and to the .json instead. The +# orchestrator reads them and writes ONE open question about THIS diff. +printf 'limits: one line of one file was mutated. Says nothing about other paths into the +same mechanism, whether it is reachable in production, or whether the guarded behaviour is +correct. A falsifying test proves the test has power, not that the fix is right.%s\n' \ + "$([ "$VERDICT" = vacuous ] && printf '\n vacuous: the mechanism is unguarded by this suite — what else depends on it?')" >&2 exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh index 511a0e5a..19c5d8fc 100755 --- a/domains/pr-workflow/skills/evidence/scripts/render-count.sh +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -98,7 +98,7 @@ cat > "$STAMP.json" <<JSON JSON { - echo "### C4 — consumer render count" + echo "### Consumer render count" echo echo "**Verdict:** $VERDICT" echo @@ -116,13 +116,15 @@ JSON echo "This counts renders of one named consumer across a defined interaction. It is not a count" echo "of consumers, and a larger consumer count does not imply a larger effect." echo - echo "**Open for review** — one named consumer, one interaction. Other consumers of the same" - echo "provider are unmeasured, and a consumer that renders once here may render freely under" - echo "an interaction this probe does not perform. The probe also does not check that the" - echo "consumer renders the same OUTPUT, only that it renders fewer times." - echo - echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Logs: \`${STAMP##*/}-armA.log\`, \`${STAMP##*/}-armB.log\`.</sub>" + echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`.</sub>" } > "$STAMP.md" printf 'render-count: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 +# The limits below are identical on every run: they describe the instrument, not the +# change under review. Pasted into a PR comment they read as boilerplate to a reviewer +# who has no stake in this tooling, so they go to stderr and to the .json instead. The +# orchestrator reads them and writes ONE open question about THIS diff. +printf 'limits: one named consumer, one interaction. Other consumers are unmeasured, and one +that renders once here may render freely under an interaction this probe does not perform. +Counts renders, not whether the output is equivalent.\n' >&2 exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index d0f4c7e5..8bd8e69d 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -147,7 +147,7 @@ cat > "$STAMP.json" <<JSON JSON { - echo "### C4 — \`$EXPORT\` recomputation count" + echo "### \`$EXPORT\` recomputation count" echo echo "**Verdict:** $VERDICT" echo @@ -171,14 +171,16 @@ JSON echo "$LINE" echo '```' echo - echo "**Open for review** — measured against one fixture with one perturbed key. A selector" - echo "unmoved here can still recompute under state this fixture does not reach, and the count" - echo "says nothing about the cost of each recomputation. Worth a look if the fixture is thin" - echo "relative to the shapes this selector sees in production." - echo - echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. Log: \`${STAMP##*/}.log\`.</sub>" + echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`.</sub>" } > "$STAMP.md" printf 'selector-recompute: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 +# The limits below are identical on every run: they describe the instrument, not the +# change under review. Pasted into a PR comment they read as boilerplate to a reviewer +# who has no stake in this tooling, so they go to stderr and to the .json instead. The +# orchestrator reads them and writes ONE open question about THIS diff. +printf 'limits: one fixture, one perturbed key. A selector unmoved here can still recompute +under state this fixture does not reach, and the count says nothing about the cost of each +recomputation.\n' >&2 [ -n "$A" ] || exit 2 exit 0 diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh index 38ba7fa0..0c397eae 100755 --- a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -106,7 +106,7 @@ cat > "$STAMP.json" <<JSON JSON { - echo "### D6 — authored-vs-authoritative substitution · \`$VERDICT\`" + echo "### Authored type vs authoritative source · \`$VERDICT\`" echo echo "| Arm | Change | distinct \`tsc\` errors |" echo "|---|---|---|" @@ -128,14 +128,16 @@ JSON echo "subtracted, not disqualifying — only errors new under substitution are the finding.</sub>" fi echo - echo "**Open for review** — the compiler answers only what the probe asks. A silent arm B means" - echo "no call site in this tree distinguishes the two shapes, which is not agreement: a" - echo "divergence reachable only at runtime, or only from a caller outside this repo, will not" - echo "appear here. Worth a look at whether the authoritative type is itself correct." - echo - echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. Logs: \`${STAMP##*/}-armA.log\`, \`${STAMP##*/}-armB.log\`.</sub>" + echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V.</sub>" echo } > "$STAMP.md" printf 'tsc-substitution: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 +# The limits below are identical on every run: they describe the instrument, not the +# change under review. Pasted into a PR comment they read as boilerplate to a reviewer +# who has no stake in this tooling, so they go to stderr and to the .json instead. The +# orchestrator reads them and writes ONE open question about THIS diff. +printf 'limits: the compiler answers only what the probe asks. A silent arm B means no call +site in this tree distinguishes the two shapes, which is not agreement — a divergence +reachable only at runtime, or from a caller outside this repo, will not appear here.\n' >&2 exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 7c7d1a72..81119fde 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -377,10 +377,30 @@ replace idempotently instead of accumulating: **Verdict:** ✅ proven — **Claim:** <one-line falsifiable behavior under test> head `<sha>` · <YYYY-MM-DD> · lanes: <lane ids> -<claim → artifact table; every claim binds its artifact> +<one sentence: what kind of evidence follows> + +<the captured artifacts, unfolded> + +**Follows from the above** +<terse bullets — each one a consequence of a number in an artifact above> + +**Open for review:** <the single question this run hands to a human, about THIS diff> <!-- VALIDATION_RUN_END --> ``` +**The exhibits are the comment.** A reviewer opens this to see a measurement, so the +captured blocks go in the body, not behind a `<details>`, and they should outweigh your +prose — 70% exhibit is a reasonable floor. Everything you write around them is a caption. + +**Never restate an artifact's number in your own prose.** A figure that appears only in a +sentence you typed is a figure on your word, which is the one thing this whole skill exists +to avoid. Cite by pointing at the block; a summary table above the exhibits duplicates the +artifact's own table and downgrades it. + +**Prose is the failure mode.** Lead with the conclusion, then the exhibits, then bullets. +Paragraphs of explanation read as an infodump and bury the finding; if a bullet needs three +sentences the exhibit is not doing its job. + Verdict icons: `✅` proven · `❌` failed · `ℹ️` otherwise. Never `❌` for a gap in *evidence* — that reads as a verdict on the author's work. @@ -401,6 +421,27 @@ The parts that decide *whether* to publish, rather than how: - **Don't restate CI.** Lint, build, and test results are already on the Checks tab. - **Scrub** local paths and usernames; failure summaries leak them. +### The reader is a reviewer on this PR, not a user of this skill + +They have a stake in the change and none in the tooling. Everything internal to how the +evidence was produced is noise to them, and several of these leaked into a published run +before anyone noticed: + +| Leaks | Publish instead | +|---|---| +| Lane ids — `B3`, `C4`, `D3` | The category in words: *falsifying test*, *render count* | +| A runner's generic limits, identical on every run | One open question about **this** diff | +| The runner's own name as though it means something | `Produced by <tool>` provenance, and nothing more | +| Your process — drafts, retractions, what you tried first | The measurement as it stands now | +| Anything calibrating the skill rather than the change | Nothing; delete it | + +The runners cooperate with this: their generic limits go to stderr and to the `.json`, not +into the `.md` exhibit, precisely so a reviewer never reads the same paragraph about the +instrument under three consecutive blocks. Read them there and synthesise **one** question. + +The test: would this sentence still be worth reading if the skill did not exist? If it is +only interesting to someone who knows how the tool works, cut it. + ## Validation output format When reporting back (before publishing), lead with the verdict and the claim it tests: From 7dee1f937da381757000dc285bed19ef7902f9ae Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 12:28:47 -0400 Subject: [PATCH 087/135] Move the output template into the skill, where corrections can land Three rounds of format corrections were applied to three draft comments and to nothing else, because the template they were assembled from lived in a scratch directory. A generator outside the repo makes fixing the instance and fixing the target two separate acts, and only the first one is visible, so the second gets skipped and the correction arrives again on the next run. `references/output-templates.md` is now the generator and says so. --- .../evidence/references/output-templates.md | 72 +++++++++++++++++++ domains/pr-workflow/skills/evidence/skill.md | 5 ++ 2 files changed, 77 insertions(+) create mode 100644 domains/pr-workflow/skills/evidence/references/output-templates.md diff --git a/domains/pr-workflow/skills/evidence/references/output-templates.md b/domains/pr-workflow/skills/evidence/references/output-templates.md new file mode 100644 index 00000000..607ef807 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/references/output-templates.md @@ -0,0 +1,72 @@ +# Output templates + +The shape a validation run ships in. **This file is the generator.** A correction to how a +run reads is a defect here, not in the comment it was noticed on — fix it here and regenerate, +or the same correction arrives again on the next run. + +Drafting a template in a scratch directory is how that goes wrong: the comment gets better and +nothing else does. + +## The template + +```markdown +<!-- VALIDATION_RUN_START --> +## 🧪 Validation Run + +**Verdict:** <icon> <the conclusion, in words a reviewer can act on> — **Claim:** <the +falsifiable thing under test> head `<sha>` · <YYYY-MM-DD> · <check name in words> + +<one sentence: what kind of evidence follows, and how it was arranged> + +<captured artifact> + +<one sentence, only if a second exhibit needs a transition> + +<captured artifact> + +**Follows from the above** + +- <a consequence of a number in an exhibit above> +- <another> + +**Open for review:** <the single question this run hands to a human, about THIS change> + +<sub>Trial run of the <a href="...">MetaMask evidence skills</a> — feedback welcome. Not a +review verdict; nothing here blocks the PR.</sub> +<!-- VALIDATION_RUN_END --> +``` + +## What each slot is for + +**Verdict line.** The conclusion, not the topic. *"one of the two conjuncts is tested"* and +*"six renders where one would do"* are conclusions; *"tested the hash predicate"* is a topic. +Icons: `✅` proven · `⚠️` partial or scoped · `📋` measured, no verdict asserted · `❌` failed. +Never `❌` for a gap in *evidence* — that reads as a verdict on the author's work. + +**Check name, in words.** *falsifying-test check*, *render-count check*, +*dependency-containment check*. Never the lane id: `B3` is an address into +[evidence-catalog.md](evidence-catalog.md), which the reviewer cannot open. + +**The exhibits.** Whatever the runner wrote, pasted whole and unfolded. They should outweigh +everything else in the comment; 70% is a reasonable floor. Do not summarise them above +themselves — a table of your own restating theirs turns a measurement into your word for it. + +**Follows from the above.** Bullets, each traceable to a number in an exhibit. If a bullet +needs three sentences, the exhibit is not carrying its weight. + +**Open for review.** One question, about this change. The runners' generic limits go to stderr +and the `.json` precisely so they do not end up here three times over; read them, and write +the thing a human should actually look at. + +## Assembly + +Templates carry `@@TOKEN@@` placeholders, one per exhibit, substituted with the runner's `.md` +verbatim. Substitution — never retyping — is what keeps the provenance line attached to the +numbers it vouches for. + +Before posting, `scripts/attest-gate.sh <file>` must exit 0. + +## Worked instantiations + +Three runs assembled from this template, with the reasoning behind each choice, are in +[worked-examples.md](worked-examples.md). diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 81119fde..65a907f2 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -404,6 +404,11 @@ sentences the exhibit is not doing its job. Verdict icons: `✅` proven · `❌` failed · `ℹ️` otherwise. Never `❌` for a gap in *evidence* — that reads as a verdict on the author's work. +The template itself, slot by slot, and how a run is assembled from it: +**[references/output-templates.md](references/output-templates.md)** — that file is the +generator, so a correction to how a run reads belongs there rather than in the comment it was +noticed on. + Full recipe — image re-hosting, recordings, AEP mirroring, the privacy scrub: **[references/evidence-publishing.md](references/evidence-publishing.md).** From f0eaf4111c3a3eec9cfe286f6a9dfe28be79b429 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 12:39:16 -0400 Subject: [PATCH 088/135] Separate the grants someone chose from the ones a new package brought MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A single escalation list mixes two different questions. A capability granted to a package the base policy did not contain arrives because the package arrived — the question there is whether the package belongs in the bundle. A capability newly granted to a package already contained is somebody's decision about that capability. Mixed together on one mv3 policy, 14 of the 24 rows were the first kind, and a list that is mostly not actionable teaches its reader to skim. Write access is exempt from the row cap: it is the smallest and highest-signal category, and truncating it hides the row that most needed a reader. --- .../scripts/policy-audit.py | 42 ++++++++++++++++++- 1 file changed, 41 insertions(+), 1 deletion(-) diff --git a/domains/security/skills/lavamoat-policy-diligence/scripts/policy-audit.py b/domains/security/skills/lavamoat-policy-diligence/scripts/policy-audit.py index c6a39221..e87e45d7 100644 --- a/domains/security/skills/lavamoat-policy-diligence/scripts/policy-audit.py +++ b/domains/security/skills/lavamoat-policy-diligence/scripts/policy-audit.py @@ -126,6 +126,21 @@ def audit_overrides(over, gen): return widened, tightened, broad, write +def partition_escalations(rows, base, write_set): + """Split a human's worklist into decisions and consequences. + + A capability granted to a package the base policy did not contain at all arrives + because the package arrived — a bundle-graph change, not a choice anyone made about + that capability. A capability newly granted to a package already contained is + somebody's decision. Under one heading the second is buried in the first: a list + where 23 of 24 rows are not actionable teaches its reader to skim past the one that is. + """ + chosen, inherited = [], [] + for pkg, kind, cap in rows: + (inherited if pkg not in base else chosen).append((pkg, kind, cap)) + return chosen, inherited + + def main(): args = sys.argv[1:] if len(args) < 2: @@ -200,15 +215,40 @@ def main(): )) if escalate: write_set = set(write) + chosen, inherited = partition_escalations(escalate, base, write_set) print("\n\nRAISE WITH A HUMAN — no verdict offered") print("-" * 74) print("These widen a capability class where a wrong call is not recoverable by a") print("follow-up PR, or grant write where read may suffice. Whether each is correct") print("depends on intent and threat model, neither of which is in the policy files.") print("This script stops here deliberately rather than guessing.\n") - for pkg, kind, cap in escalate: + def row(pkg, kind, cap): why = "write access" if (pkg, kind, cap) in write_set else criticality(pkg, cap) print(f" [?] {pkg[:42]:44s} {kind[:3]}:{cap:20s} {why}") + + print(f"\nCHOSEN HERE — {len(chosen)} row(s), on a package the base already contained") + if chosen: + for pkg, kind, cap in chosen: + row(pkg, kind, cap) + else: + print(" (none — every escalation below arrived with a new package)") + + if inherited: + print(f"\nARRIVES WITH A NEWLY-CONTAINED PACKAGE — {len(inherited)} row(s)") + print(" The package is new to this policy, so the grant follows from containing") + print(" it. The question is whether the package belongs in this bundle, not") + print(" whether the capability was correctly chosen.") + # Write access is never truncated. It is the smallest and highest-signal + # category, and a cap that hides it turns the section into a list whose + # most important row is the one the reader cannot see. + w = [r for r in inherited if r in write_set] + rest = [r for r in inherited if r not in write_set] + for pkg, kind, cap in w: + row(pkg, kind, cap) + for pkg, kind, cap in rest[:10]: + row(pkg, kind, cap) + if len(rest) > 10: + print(f" … {len(rest) - 10} further read-only row(s) of the same kind.") print(f"\n {len(escalate)} decision(s) for a human. An audit that silently resolves") print(" these has substituted a guess for the thing it was asked to check.") From 2c72486606b0d2090b409495ec8a931a312d5936 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 13:15:02 -0400 Subject: [PATCH 089/135] Require a finding, not a printout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A run was published reading "measured, no verdict asserted" over a policy delta and a 24-row escalation list, leaving the reader to work out whether any of it was good news. Withholding the conclusion feels like the rigorous move under this skill's own standards, but the numbers came from an instrument the reader does not have, and the run is the only party holding the context to read them. Records the usual cause — a number with no baseline to hold it against — and the fix, which is to find the comparison rather than to hedge. --- domains/pr-workflow/skills/evidence/skill.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 65a907f2..eef4b9ed 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -401,6 +401,24 @@ artifact's own table and downgrades it. Paragraphs of explanation read as an infodump and bury the finding; if a bullet needs three sentences the exhibit is not doing its job. +**There must be a finding.** A run ends with something the reader can agree or disagree with — +this holds, or it does not. Running the instruments and publishing what they printed is not +that. `📋 measured, no verdict asserted` is not a verdict, and neither is a headline that +reports a delta: *"494 → 651 packages, 24 rows escalated"* is a measurement in a verdict's +clothes, leaving the reader to work out whether it is good news. + +The tell is a verdict line you cannot restate as a sentence with a subject and a verb. +Withholding the conclusion feels like the rigorous move under this skill's standards, but the +numbers came from an instrument the reader does not have; the run is the only party holding the +context to interpret them, and declining to is offloading rather than restraint. State the +conclusion **and** its limits — floating a concern is the residue of a finding, not a +substitute for having one. + +When no finding presents itself, the usual cause is a missing comparison rather than a +genuinely inconclusive result: a number with nothing to hold it against. Find the baseline that +turns it into a claim — a sibling artifact, the other build target, the previous release, the +arm the change did not touch. + Verdict icons: `✅` proven · `❌` failed · `ℹ️` otherwise. Never `❌` for a gap in *evidence* — that reads as a verdict on the author's work. From 41329516f660c47f2dd02fbff0dfb972733b243f Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 14:53:41 -0400 Subject: [PATCH 090/135] Check where the disclaimer is, not just that it is there Three rounds of editing for density demoted the trial-run disclaimer from a callout under the verdict to `<sub>` at the foot of the page. It survived in every comment and did nothing in any of them: it is the frame a reviewer needs before reading a verdict on their own PR, and from the bottom it arrives after the reaction it exists to shape. A fifth comment had none at all. A disclaimer is not content. Content earns its place by density and a frame earns it by arriving first, so a uniform compression pass will always demote it. Check 11 compares its line against the first exhibit's; existence alone passed in all four demoted comments. --- .../evidence/references/output-templates.md | 13 +++++++++++-- .../skills/evidence/scripts/attest-gate.sh | 16 ++++++++++++++++ 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/output-templates.md b/domains/pr-workflow/skills/evidence/references/output-templates.md index 607ef807..0aae9992 100644 --- a/domains/pr-workflow/skills/evidence/references/output-templates.md +++ b/domains/pr-workflow/skills/evidence/references/output-templates.md @@ -16,6 +16,11 @@ nothing else does. **Verdict:** <icon> <the conclusion, in words a reviewer can act on> — **Claim:** <the falsifiable thing under test> head `<sha>` · <YYYY-MM-DD> · <check name in words> +> [!NOTE] +> Trial run of the [MetaMask evidence skills](<link>) — feedback welcome, on the finding or +> on whether this format is useful to a reviewer. Not a review verdict; nothing here blocks +> the PR. + <one sentence: what kind of evidence follows, and how it was arranged> <captured artifact> @@ -31,8 +36,6 @@ falsifiable thing under test> head `<sha>` · <YYYY-MM-DD> · <check name in wor **Open for review:** <the single question this run hands to a human, about THIS change> -<sub>Trial run of the <a href="...">MetaMask evidence skills</a> — feedback welcome. Not a -review verdict; nothing here blocks the PR.</sub> <!-- VALIDATION_RUN_END --> ``` @@ -54,6 +57,12 @@ themselves — a table of your own restating theirs turns a measurement into you **Follows from the above.** Bullets, each traceable to a number in an exhibit. If a bullet needs three sentences, the exhibit is not carrying its weight. +**The disclaimer sits directly under the verdict, and stays a callout.** It is the frame a +reviewer needs *before* they read a verdict on their own PR from a source they have not seen +before — where feedback goes, and that nothing here blocks them. Edited by the same rules as +prose it drifts to the foot of the page in `<sub>`, where it arrives after the reaction it +exists to shape. Check 11 tests its position, not just its presence. + **Open for review.** One question, about this change. The runners' generic limits go to stderr and the `.json` precisely so they do not end up here three times over; read them, and write the thing a human should actually look at. diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index a3dcf266..2cb0d395 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -103,6 +103,22 @@ else fail "10 floats something for review" "no limit, open question, or falsifier named — an artifact that floats nothing implies its measurement was the whole surface" fi +# 11 — the trial-run disclaimer, and its POSITION. This is not content, it is the frame +# the reader needs before they read a verdict on their own PR from an unfamiliar source. +# Compressed and moved to the foot of the page — which is what happens when it is edited +# by the same rules as prose — it arrives after the reaction it exists to shape. +DISC="$(grep -n -i 'trial run' "$FILE" | head -1 | cut -d: -f1)" +FIRST_EXHIBIT="$(grep -n '^```' "$FILE" | head -1 | cut -d: -f1)" +if [ -z "$DISC" ]; then + fail "11 disclaimer present and early" "no trial-run disclaimer — a reviewer cannot tell what this is or where to send feedback" +elif ! grep -qi 'trial run' "$FILE" || ! grep -q 'skills/pull/\|MetaMask/skills' "$FILE"; then + fail "11 disclaimer present and early" "disclaimer does not link the skills PR, so feedback has nowhere to go" +elif [ -n "$FIRST_EXHIBIT" ] && [ "$DISC" -gt "$FIRST_EXHIBIT" ]; then + fail "11 disclaimer present and early" "disclaimer is at line $DISC, after the first exhibit at line $FIRST_EXHIBIT — it frames nothing from there" +else + pass "11 disclaimer present and early" +fi + if [ -n "$REF" ] && [ -f "$REF" ]; then r=$(grep -coE '!\[|<img|data:image' "$REF"); c=$(grep -coE '!\[|<img|data:image|evidence-artifacts/|Produced by' "$FILE") echo From e58d1c0e1c5100e8bc245269d6163cd08f527e49 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 14:56:34 -0400 Subject: [PATCH 091/135] Stop a script's summary of itself passing as a capture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A run shipped a table the script composed, one grepped line, and the command `yarn jest <generated probe>` — a path that had already been deleted. The tool's real output was in the `.log` beside it and never shown. The gate passed it because `Produced by` was present, and a marker attests who wrote a block rather than what the block contains: it is equally true of a paraphrase. That is the failure this suite exists to prevent — an operator retyping output — displaced one layer down, where the retyping is done by the script and carries a machine's byline. `selector-recompute` now prints the real command and passes jest's own stdout through, and keeps the probe beside the artifact instead of deleting the file its command line names. Check 5 fails a `$` line carrying a placeholder: nothing else in a console block advertises "composed, not captured" so plainly, and it greps. --- .../skills/evidence/scripts/attest-gate.sh | 14 +++++++++++--- .../skills/evidence/scripts/selector-recompute.sh | 15 +++++++++++---- 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 2cb0d395..1cbcb72b 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -50,10 +50,18 @@ hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `|[Ff]irefox [0-9]+\.[0-9]|[ || fail "4 environment pinned" "no head SHA, lockfile hash, or pinned toolchain/browser version" # 5 — the one that matters. A tool-written log, a run link, or an image; not typed prose. -if hasre '!\[|<img|data:image|actions/runs|/gist\.|evidence-artifacts/|Produced by '; then - pass "5 captured artifact" -else +# +# `Produced by` attests who WROTE the block, not that the block is the tool's own output. +# A script that composes a summary table and stamps itself passes on the marker alone — +# which is how a run shipped with a table the script had written, one grepped line, and a +# command reading `yarn jest <generated probe>`. A `$` line carrying a placeholder is the +# tell: it looks reproducible and cannot be run. +if ! hasre '!\[|<img|data:image|actions/runs|/gist\.|evidence-artifacts/|Produced by '; then fail "5 captured artifact" "every block appears operator-typed; no tool-written log, run link, or image referenced" +elif grep -qE '^\$ .*<[a-z][a-z ._-]*>' "$FILE"; then + fail "5 captured artifact" "a console command contains a placeholder — $(grep -m1 -oE '^\$ .*' "$FILE") is not a command a reader can run" +else + pass "5 captured artifact" fi if hasi 'what would close it|what would prove it|closing it requires'; then diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index 8bd8e69d..46b756d2 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -63,7 +63,12 @@ MOD_BASE="./$(basename "$MODULE")" DEPTH="$(dirname "$MODULE" | tr -cd '/' | wc -c | tr -d ' ')" UP=""; i=0; while [ "$i" -le "$DEPTH" ]; do UP="../$UP"; i=$((i+1)); done -cleanup() { rm -f "$PROBE"; } +# The probe used to be deleted on exit, which left the exhibit quoting +# `yarn jest <generated probe>` — a command line that cannot be run, printed where a +# reader expects a reproducible one. It is kept alongside the artifact instead, and +# removed from the working tree so the repo is left clean. +KEEP_PROBE="$STAMP.probe.test.ts" +cleanup() { [ -f "$PROBE" ] && cp "$PROBE" "$KEEP_PROBE"; rm -f "$PROBE"; } trap cleanup EXIT INT TERM cat > "$PROBE" <<PROBEEOF @@ -167,11 +172,13 @@ JSON fi echo echo '```console' - echo "\$ yarn jest <generated probe>" - echo "$LINE" + echo "\$ yarn jest $PROBE" + # The tool's own output, not a line this script composed. A summary a script writes + # about its own run carries the script's word; the runner's stdout carries the run's. + grep -E "RECOMPUTE_PROBE |^Test Suites:|^Tests: |^Time: " "$STAMP.log" | head -8 echo '```' echo - echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and deleted. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`.</sub>" + echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and kept beside this artifact. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`.</sub>" } > "$STAMP.md" printf 'selector-recompute: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 From 3c86466852aa5cc79a53c7b6360f28a7776a509a Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 19:20:26 -0400 Subject: [PATCH 092/135] Ask for a medium, not for better text MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Check 5 has been rewritten four times today and each time it tested a property of the plaintext — a provenance marker, a placeholder in the command, a local path. Each caught one defect and missed the next, because every property of text is forgeable by whatever emits the text. Four runs shipped through it. It now requires a capture the reader verifies without going through the author: an image of the tool's surface, a link that re-executes, or a hosted artifact. A fenced block sits beside one of those and is never the evidence itself. --- .../skills/evidence/scripts/attest-gate.sh | 32 +++++++++++++++++-- domains/pr-workflow/skills/evidence/skill.md | 20 ++++++++++++ 2 files changed, 49 insertions(+), 3 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 1cbcb72b..cc454e32 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -49,17 +49,43 @@ hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `|[Ff]irefox [0-9]+\.[0-9]|[ && pass "4 environment pinned" \ || fail "4 environment pinned" "no head SHA, lockfile hash, or pinned toolchain/browser version" -# 5 — the one that matters. A tool-written log, a run link, or an image; not typed prose. +# 5 — the one that matters, and it asks for a MEDIUM, not for better text. +# +# Every earlier version of this check tested a property of the plaintext: does it carry a +# provenance marker, does the command contain a placeholder, is the path local. Each caught +# one defect and missed the next, because every property of plaintext is forgeable by +# whatever emits the plaintext. Four runs shipped that way. +# +# So the block below is necessary but is no longer the evidence. The evidence is an image +# of the tool's own surface, a link that re-executes, or a hosted artifact the reader +# fetches without going through the author. If the artifact is small, nothing was attached. # # `Produced by` attests who WROTE the block, not that the block is the tool's own output. # A script that composes a summary table and stamps itself passes on the marker alone — # which is how a run shipped with a table the script had written, one grepped line, and a # command reading `yarn jest <generated probe>`. A `$` line carrying a placeholder is the # tell: it looks reproducible and cannot be run. -if ! hasre '!\[|<img|data:image|actions/runs|/gist\.|evidence-artifacts/|Produced by '; then - fail "5 captured artifact" "every block appears operator-typed; no tool-written log, run link, or image referenced" +# An image, a re-executing link, or a hosted artifact — verification that does not route +# through the author. `Produced by` and `evidence-artifacts/` are provenance, not this. +if ! hasre '!\[[^]]*\]\(https?://|<img [^>]*src="https?://|actions/runs/[0-9]|/gist\.|https?://[^ )]+\.(png|jpg|jpeg|gif|svg|txt|log|json)\b'; then + fail "5 captured artifact" "no reader-verifiable capture — an image of the tool surface, a run link, or a hosted artifact. A fenced block is the author\'s transcription, whatever produced it" + # No separate attribution test: a hosted artifact the reader fetches is its own + # attribution, and requiring `Produced by` on top of it only fails runs whose + # evidence is stronger than a stamped fenced block. elif grep -qE '^\$ .*<[a-z][a-z ._-]*>' "$FILE"; then fail "5 captured artifact" "a console command contains a placeholder — $(grep -m1 -oE '^\$ .*' "$FILE") is not a command a reader can run" +elif grep -qE '^\$ .*(/tmp/|/home/|/Users/)' "$FILE"; then + # A helper script in /tmp, or any absolute local path, is unreproducible by + # construction. `capture.sh` records the command honestly — but honestly + # recording `bash /tmp/dup.sh` still publishes a recipe nobody else can follow. + # Inline the commands, or ship the helper where the reader can reach it. + fail "5 captured artifact" "a console command references a local-only path — $(grep -m1 -oE '^\$ .*(/tmp/|/home/|/Users/)[^ ]*' "$FILE") cannot be run by a reader" +elif [ "$(grep -cE '^\$ ' "$FILE")" -gt 1 ] && \ + [ "$(grep -E '^\$ ' "$FILE" | sed 's/ *#.*$//' | sort -u | wc -l)" -lt "$(grep -cE '^\$ ' "$FILE")" ]; then + # Two identical commands shown as producing different outputs. The difference + # came from an edit made between runs, so the block misstates its own cause: + # running it twice reproduces the first number twice. + fail "5 captured artifact" "two console commands are identical but shown with different output — the block does not say what actually differed between them" else pass "5 captured artifact" fi diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index eef4b9ed..f400bfb7 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -392,6 +392,26 @@ head `<sha>` · <YYYY-MM-DD> · lanes: <lane ids> captured blocks go in the body, not behind a `<details>`, and they should outweigh your prose — 70% exhibit is a reasonable floor. Everything you write around them is a caption. +**A fenced block is not the evidence.** Nothing in it distinguishes real stdout from invented +stdout, or from real stdout that has since drifted from its source — and whatever would +fabricate it is what formats it. Where fabrication, hallucination, or drift is a concern at +all, and that is nearly all plaintext, the medium is wrong. + +The qualifying media are the ones where verification does not route through you: + +- an **image of the tool's own surface** — the run page, the Discover view, the waterfall; +- a **link that re-executes or re-renders** — a CI run, a query permalink, a dashboard; +- a **hosted artifact the reader fetches** — the log at a URL, not a quotation of it. + +Paste the fenced block *beside* one of those, never instead of one. This is also why a +plaintext check can never close the gap: every property of text is forgeable by whatever emits +the text, so the gate asks for a different medium rather than for better text. + +**If the artifact is small, nothing was attached.** The reference showcase runs to 2 MB because +it carries 31 embedded captures. Uploading costs a step and a decision about what may be +published; that cost is the price of the reader not having to trust you, and a pipeline with no +upload step has no evidence step. + **Never restate an artifact's number in your own prose.** A figure that appears only in a sentence you typed is a figure on your word, which is the one thing this whole skill exists to avoid. Cite by pointing at the block; a summary table above the exhibits duplicates the From dbc275429d92d00daf4c9138edabe93f526fc26b Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 20:26:16 -0400 Subject: [PATCH 093/135] Move the measurement to CI, where the run URL is the capture A local run's only witness is its author, so it cannot satisfy the requirement that a reader verify without going through them. Every defect class this suite has shipped was a local-environment one: a helper in `/tmp`, a probe deleted after the run, an absolute path, a drifted toolchain, a contended host whose numbers were published and retracted. None is expressible in CI. Two controls ride along because both were learned the hard way. A `baseline` input: twice a run reported no finding when what it lacked was a comparison. A determinism check: the head arm runs twice and the numbers are not endorsed if they move. Inputs reach the shell through `env`, never spliced into the script text. --- .../skills/evidence/assets/evidence-run.yml | 169 ++++++++++++++++++ domains/pr-workflow/skills/evidence/skill.md | 12 ++ 2 files changed, 181 insertions(+) create mode 100644 domains/pr-workflow/skills/evidence/assets/evidence-run.yml diff --git a/domains/pr-workflow/skills/evidence/assets/evidence-run.yml b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml new file mode 100644 index 00000000..cdbdbab9 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml @@ -0,0 +1,169 @@ +# Evidence runner — install into the consumer repo as +# `.github/workflows/evidence-run.yml`. +# +# Why this exists rather than running the measurement locally: a validation run is a +# claim that a command produced an output, and the reader has to be able to check that +# without going through the author. A local run's only witness is the author. Every +# failure class this suite has shipped was a local-environment failure — a helper script +# in /tmp, a probe deleted after the run, an absolute path, a toolchain that drifted, a +# contended host producing numbers that had to be retracted. +# +# In CI none of those is expressible. The workflow file is the recipe, the workspace is +# the repo, the run records its own ref, and the run URL is itself the capture — +# `actions/runs/<id>` is what the publish gate accepts. +# +# Trigger from the CLI: +# gh workflow run evidence-run.yml \ +# -f runner=falsify-probe \ +# -f ref=<sha> \ +# -f args='--test path/to.test.ts --source path/to.ts --line 9 --replace " return x;"' +# +# Then cite the run URL in the comment. The artifacts are attached to the run; the +# orchestrator reads them to write the finding. +name: Evidence run + +'on': + workflow_dispatch: + inputs: + runner: + description: Runner to execute + required: true + type: choice + options: + - falsify-probe + - selector-recompute + - render-count + - tsc-substitution + - capture + ref: + description: Commit SHA to measure. Pin it — a branch name makes the run unrepeatable. + required: true + type: string + args: + description: Arguments passed to the runner, verbatim + required: true + type: string + baseline: + description: >- + Second SHA to measure identically. Twice now a run reported "no finding" when it + had no comparison, so the baseline arm is offered here rather than left to memory. + required: false + type: string + +permissions: + contents: read + +jobs: + measure: + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - name: Reject a moving ref + env: + REF: ${{ inputs.ref }} + BASE: ${{ inputs.baseline }} + run: | + # A branch name makes the artifact unrepeatable, which is the property this + # workflow exists to provide. Checked before anything is fetched. + for r in "$REF" ${BASE:+"$BASE"}; do + case "$r" in + *[!0-9a-f]* | "") echo "::error::'$r' is not a commit SHA"; exit 1 ;; + esac + [ ${#r} -eq 40 ] || { echo "::error::'$r' must be the full 40-char SHA"; exit 1; } + done + + - name: Checkout at the measured ref + uses: actions/checkout@v6 + with: + ref: ${{ inputs.ref }} + fetch-depth: 2 # the runners diff against the parent + + - uses: actions/setup-node@v4 + with: + node-version-file: .nvmrc + cache: yarn + + - name: Install + run: yarn --immutable + + - name: Fetch the runners at a pinned version + uses: actions/checkout@v6 + with: + repository: MetaMask/skills + ref: ${{ vars.EVIDENCE_SKILLS_REF || 'main' }} + path: .evidence-skills + sparse-checkout: domains/pr-workflow/skills/evidence/scripts + + - name: Run + id: run + continue-on-error: true # the exit code IS the verdict; a finding is not a failure + env: + RUNNER: ${{ inputs.runner }} + ARGS: ${{ inputs.args }} + run: | + RUNNERS=.evidence-skills/domains/pr-workflow/skills/evidence/scripts + mkdir -p evidence-artifacts + set +e + eval "bash \"$RUNNERS/$RUNNER.sh\" $ARGS --label \"$RUNNER-head\" --out evidence-artifacts" + code=$? + set -e + echo "head_exit=$code" >> "$GITHUB_OUTPUT" + echo "runner exited $code — the exit code is the verdict, not a build failure" \ + >> "$GITHUB_STEP_SUMMARY" + + - name: Baseline arm + if: inputs.baseline != '' + continue-on-error: true + env: + RUNNER: ${{ inputs.runner }} + ARGS: ${{ inputs.args }} + BASE: ${{ inputs.baseline }} + run: | + git checkout --detach "$BASE" + yarn --immutable + RUNNERS=.evidence-skills/domains/pr-workflow/skills/evidence/scripts + eval "bash \"$RUNNERS/$RUNNER.sh\" $ARGS --label \"$RUNNER-base\" --out evidence-artifacts" + + - name: Determinism check + # Contention produced numbers that were published and then retracted. Running the + # head arm twice and diffing costs one repeat and turns that into a pre-publish + # signal rather than a correction. + continue-on-error: true + env: + RUNNER: ${{ inputs.runner }} + ARGS: ${{ inputs.args }} + REF: ${{ inputs.ref }} + run: | + RUNNERS=.evidence-skills/domains/pr-workflow/skills/evidence/scripts + git checkout --detach "$REF" + eval "bash \"$RUNNERS/$RUNNER.sh\" $ARGS --label \"$RUNNER-repeat\" --out evidence-artifacts" || true + A="evidence-artifacts/$RUNNER-head.json" + B="evidence-artifacts/$RUNNER-repeat.json" + if [ -f "$A" ] && [ -f "$B" ]; then + # env block differs by design (timing); compare the measurement only + if diff <(jq -S 'del(.env)' "$A") <(jq -S 'del(.env)' "$B") > determinism.diff; then + echo "deterministic across two runs" | tee -a "$GITHUB_STEP_SUMMARY" + else + echo "::warning::runner is NOT deterministic at this ref — do not publish these numbers" + cat determinism.diff >> "$GITHUB_STEP_SUMMARY" + fi + fi + + - name: Publish the run summary + if: always() + run: | + for f in evidence-artifacts/*.md; do + [ -f "$f" ] || continue + { echo; cat "$f"; } >> "$GITHUB_STEP_SUMMARY" + done + + - name: Upload artifacts + if: always() + uses: actions/upload-artifact@v4 + with: + name: evidence-${{ inputs.runner }}-${{ inputs.ref }} + path: | + evidence-artifacts/** + determinism.diff + retention-days: 90 + if-no-files-found: error diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index f400bfb7..20ffa882 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -412,6 +412,18 @@ it carries 31 embedded captures. Uploading costs a step and a decision about wha published; that cost is the price of the reader not having to trust you, and a pipeline with no upload step has no evidence step. +**Run the measurement in CI, not locally.** `assets/evidence-run.yml` installs into the consumer +repo and dispatches any runner at a pinned SHA. This is not about convenience: a local run's +only witness is you, so it cannot meet the requirement above, and every defect class this suite +has shipped was a local-environment one — a helper in `/tmp`, a probe deleted after the run, an +absolute path, a drifted toolchain, a contended host whose numbers had to be retracted. None of +those is expressible in CI, where the workflow file is the recipe, the workspace is the repo, +and the run URL is itself the capture. + +It also carries two controls worth having by default: a `baseline` input, because twice a run +reported "no finding" when what it lacked was a comparison; and a determinism check that runs +the head arm twice and refuses to endorse numbers that move. + **Never restate an artifact's number in your own prose.** A figure that appears only in a sentence you typed is a figure on your word, which is the one thing this whole skill exists to avoid. Cite by pointing at the block; a summary table above the exhibits duplicates the From bdf5f39ec61a8e99858bb0d1526bfffd20c1f504 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sat, 1 Aug 2026 20:28:25 -0400 Subject: [PATCH 094/135] Pin the runner source, and say so when it is missing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The workflow fetched the runners from the skills repo's `main`, where they do not exist — they are on this branch. Every dispatch would have failed at that step, and a sparse checkout of an absent path succeeds with an empty directory, so the failure would have surfaced two steps later as "No such file" with no hint that the ref was the cause. Pinned to a commit for the same reason the measured ref must be, with an explicit check that names EVIDENCE_SKILLS_REF as the thing to change. --- .../skills/evidence/assets/evidence-run.yml | 22 ++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/evidence/assets/evidence-run.yml b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml index cdbdbab9..1ce37316 100644 --- a/domains/pr-workflow/skills/evidence/assets/evidence-run.yml +++ b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml @@ -86,14 +86,34 @@ jobs: - name: Install run: yarn --immutable + # Pinned to a commit, for the same reason the measured ref must be: a branch name + # makes the run unrepeatable, and "which version of the runner produced this" is + # exactly the question a reader asks. Override per-repo with the EVIDENCE_SKILLS_REF + # variable; bump the default when the runners land on the skills repo's main. - name: Fetch the runners at a pinned version uses: actions/checkout@v6 with: repository: MetaMask/skills - ref: ${{ vars.EVIDENCE_SKILLS_REF || 'main' }} + ref: ${{ vars.EVIDENCE_SKILLS_REF || 'dbc275429d92d00daf4c9138edabe93f526fc26b' }} path: .evidence-skills sparse-checkout: domains/pr-workflow/skills/evidence/scripts + - name: Verify the runners arrived + env: + RUNNER: ${{ inputs.runner }} + run: | + # A sparse checkout of a path that does not exist on the chosen ref succeeds and + # produces an empty directory, so the next step would fail with "No such file" + # and no indication that the REF was the problem. + F=".evidence-skills/domains/pr-workflow/skills/evidence/scripts/$RUNNER.sh" + [ -f "$F" ] || { + echo "::error::$RUNNER.sh not present at the pinned skills ref." + echo "::error::Set the EVIDENCE_SKILLS_REF repository variable to a commit that has it." + exit 1 + } + echo "runner $RUNNER.sh sourced from skills @ ${{ vars.EVIDENCE_SKILLS_REF || 'pinned default' }}" \ + >> "$GITHUB_STEP_SUMMARY" + - name: Run id: run continue-on-error: true # the exit code IS the verdict; a finding is not a failure From 5c41bfbf8ab23901075e074194c33e9381b19238 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sun, 2 Aug 2026 08:06:45 -0400 Subject: [PATCH 095/135] Require the target repo, make install opt-out, drop the consumer-install framing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three corrections to the runner workflow, all found by pointing it somewhere. `target_repo` is required with no default: its stated purpose is to be proved somewhere harmless first, and a default pointed the first dispatch at the repo under review. `needs_install` skips a ten-minute yarn install for runners that only read files. And the header still described installing this into the repo under review, which stopped being true once the target became an input — that stale sentence is what made a fork of the extension look necessary. --- .../skills/evidence/assets/evidence-run.yml | 43 +++++++++++++++---- domains/pr-workflow/skills/evidence/skill.md | 5 ++- 2 files changed, 37 insertions(+), 11 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/assets/evidence-run.yml b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml index 1ce37316..b3db840f 100644 --- a/domains/pr-workflow/skills/evidence/assets/evidence-run.yml +++ b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml @@ -1,5 +1,8 @@ -# Evidence runner — install into the consumer repo as -# `.github/workflows/evidence-run.yml`. +# Evidence runner — lives in ONE repo and measures any other. +# +# It does not need to be installed in the repo under review, and there is no reason to +# fork that repo either: `target_repo` is an input and the job checks it out read-only. +# Put this in whatever repo you want the runs and artifacts to belong to. # # Why this exists rather than running the measurement locally: a validation run is a # claim that a command produced an output, and the reader has to be able to check that @@ -35,6 +38,13 @@ name: Evidence run - render-count - tsc-substitution - capture + target_repo: + description: >- + Repository to measure. No default on purpose: a default here is a standing + decision about what every unthinking dispatch touches, and this workflow's whole + argument is that it should be proved somewhere harmless first. + required: true + type: string ref: description: Commit SHA to measure. Pin it — a branch name makes the run unrepeatable. required: true @@ -43,6 +53,13 @@ name: Evidence run description: Arguments passed to the runner, verbatim required: true type: string + needs_install: + description: >- + Install the target's dependencies. Required for the jest and tsc runners; a waste + of ten minutes for `capture` wrapping git or a policy audit, which read files only. + required: false + default: true + type: boolean baseline: description: >- Second SHA to measure identically. Twice now a run reported "no finding" when it @@ -72,18 +89,21 @@ jobs: [ ${#r} -eq 40 ] || { echo "::error::'$r' must be the full 40-char SHA"; exit 1; } done - - name: Checkout at the measured ref + - name: Checkout the target at the measured ref uses: actions/checkout@v6 with: + repository: ${{ inputs.target_repo }} ref: ${{ inputs.ref }} fetch-depth: 2 # the runners diff against the parent - uses: actions/setup-node@v4 + if: inputs.needs_install with: node-version-file: .nvmrc cache: yarn - name: Install + if: inputs.needs_install run: yarn --immutable # Pinned to a commit, for the same reason the measured ref must be: a branch name @@ -124,7 +144,7 @@ jobs: RUNNERS=.evidence-skills/domains/pr-workflow/skills/evidence/scripts mkdir -p evidence-artifacts set +e - eval "bash \"$RUNNERS/$RUNNER.sh\" $ARGS --label \"$RUNNER-head\" --out evidence-artifacts" + eval "bash \"$RUNNERS/$RUNNER.sh\" --label \"$RUNNER-head\" --out evidence-artifacts $ARGS" code=$? set -e echo "head_exit=$code" >> "$GITHUB_OUTPUT" @@ -140,9 +160,9 @@ jobs: BASE: ${{ inputs.baseline }} run: | git checkout --detach "$BASE" - yarn --immutable + if [ "${{ inputs.needs_install }}" = "true" ]; then yarn --immutable; fi RUNNERS=.evidence-skills/domains/pr-workflow/skills/evidence/scripts - eval "bash \"$RUNNERS/$RUNNER.sh\" $ARGS --label \"$RUNNER-base\" --out evidence-artifacts" + eval "bash \"$RUNNERS/$RUNNER.sh\" --label \"$RUNNER-base\" --out evidence-artifacts $ARGS" - name: Determinism check # Contention produced numbers that were published and then retracted. Running the @@ -156,12 +176,15 @@ jobs: run: | RUNNERS=.evidence-skills/domains/pr-workflow/skills/evidence/scripts git checkout --detach "$REF" - eval "bash \"$RUNNERS/$RUNNER.sh\" $ARGS --label \"$RUNNER-repeat\" --out evidence-artifacts" || true + eval "bash \"$RUNNERS/$RUNNER.sh\" --label \"$RUNNER-repeat\" --out evidence-artifacts $ARGS" || true A="evidence-artifacts/$RUNNER-head.json" B="evidence-artifacts/$RUNNER-repeat.json" if [ -f "$A" ] && [ -f "$B" ]; then - # env block differs by design (timing); compare the measurement only - if diff <(jq -S 'del(.env)' "$A") <(jq -S 'del(.env)' "$B") > determinism.diff; then + # `label` and `log` name the arm, and `env` carries timing — all three differ + # between the two runs by construction. Comparing them makes the check fire on + # every run, which is the same as not having it. + if diff <(jq -S 'del(.env, .label, .log)' "$A") \ + <(jq -S 'del(.env, .label, .log)' "$B") > determinism.diff; then echo "deterministic across two runs" | tee -a "$GITHUB_STEP_SUMMARY" else echo "::warning::runner is NOT deterministic at this ref — do not publish these numbers" @@ -182,6 +205,8 @@ jobs: uses: actions/upload-artifact@v4 with: name: evidence-${{ inputs.runner }}-${{ inputs.ref }} + # the artifact name carries what was measured, so a downloaded zip is + # self-describing rather than needing the run page to interpret path: | evidence-artifacts/** determinism.diff diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 20ffa882..664f747a 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -412,8 +412,9 @@ it carries 31 embedded captures. Uploading costs a step and a decision about wha published; that cost is the price of the reader not having to trust you, and a pipeline with no upload step has no evidence step. -**Run the measurement in CI, not locally.** `assets/evidence-run.yml` installs into the consumer -repo and dispatches any runner at a pinned SHA. This is not about convenience: a local run's +**Run the measurement in CI, not locally.** `assets/evidence-run.yml` lives in one repo and +measures any other — `target_repo` is an input and the checkout is read-only, so the repo under +review needs no workflow, no fork, and no change of any kind. This is not about convenience: a local run's only witness is you, so it cannot meet the requirement above, and every defect class this suite has shipped was a local-environment one — a helper in `/tmp`, a probe deleted after the run, an absolute path, a drifted toolchain, a contended host whose numbers had to be retracted. None of From 8d1ec29542945da113ebfec9dbde2b89b07a75b6 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sun, 2 Aug 2026 08:07:58 -0400 Subject: [PATCH 096/135] Make every runner say whether a reader can verify it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Putting the showcase runs and the trial runs in one table shows the causation running the wrong way: eight of eleven hand-built runs attach a capture a reader can open, against one of twelve built by these runners. Nothing was neglected. A runner emits clean stdout, clean stdout formats beautifully into a fenced block, and a fenced block looks like evidence — so automating the measurement automated away the part that made it checkable. The hand-built runs had no such thing to reach for and went and got a real capture. Each runner's provenance line now prints the run URL under CI, and under a local run prints that there is no reader-verifiable capture and the workflow should be used before publishing. The confession belongs in the exhibit, not in a gate that has to remember to look for it. --- .../pr-workflow/skills/evidence/scripts/capture.sh | 14 +++++++++++++- .../skills/evidence/scripts/falsify-probe.sh | 14 +++++++++++++- .../skills/evidence/scripts/render-count.sh | 14 +++++++++++++- .../skills/evidence/scripts/selector-recompute.sh | 14 +++++++++++++- .../skills/evidence/scripts/tsc-substitution.sh | 14 +++++++++++++- domains/pr-workflow/skills/evidence/skill.md | 11 +++++++++++ 6 files changed, 76 insertions(+), 5 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/capture.sh b/domains/pr-workflow/skills/evidence/scripts/capture.sh index 60666fa6..298cd619 100755 --- a/domains/pr-workflow/skills/evidence/scripts/capture.sh +++ b/domains/pr-workflow/skills/evidence/scripts/capture.sh @@ -33,6 +33,18 @@ # -- python3 retention-scan.py ui/store/background-connection.ts pr.patch set -uo pipefail +# A run's artifact has to say whether a reader can verify it. In CI the run URL is that +# verification; locally there is none, and the artifact says so rather than leaving the +# omission for a gate to catch later. +capture_provenance() { + if [ -n "${GITHUB_RUN_ID:-}" ]; then + printf 'Run: %s/%s/actions/runs/%s — logs and artifacts attached there.' \ + "${GITHUB_SERVER_URL:-https://github.com}" "${GITHUB_REPOSITORY:-}" "$GITHUB_RUN_ID" + else + printf 'Produced on a local machine: no reader-verifiable capture. Re-run through the evidence workflow before publishing.' + fi +} + OUT_DIR="evidence-artifacts"; LABEL=""; LANE=""; CLAIM=""; MAXLOG=120; VERDICT=""; OPEN="" HEADL=""; TAILL="" die() { printf 'capture: %s\n' "$1" >&2; exit 3; } @@ -127,7 +139,7 @@ JSON fi echo '```' echo - echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`.</sub>" + echo "<sub>Produced by \`capture.sh\`, not transcribed. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\` · \`$PY_V\` · yarn.lock \`$LOCK_SHA\`. $(capture_provenance)</sub>" } > "$STAMP.md" printf 'capture: %s (exit %s)\n %s\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.log" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh index 2677e199..afec5d71 100755 --- a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -34,6 +34,18 @@ # --label coalesce-inflight set -uo pipefail +# A run's artifact has to say whether a reader can verify it. In CI the run URL is that +# verification; locally there is none, and the artifact says so rather than leaving the +# omission for a gate to catch later. +capture_provenance() { + if [ -n "${GITHUB_RUN_ID:-}" ]; then + printf 'Run: %s/%s/actions/runs/%s — logs and artifacts attached there.' \ + "${GITHUB_SERVER_URL:-https://github.com}" "${GITHUB_REPOSITORY:-}" "$GITHUB_RUN_ID" + else + printf 'Produced on a local machine: no reader-verifiable capture. Re-run through the evidence workflow before publishing.' + fi +} + RUNNER="yarn jest" OUT_DIR="evidence-artifacts" LABEL="" @@ -148,7 +160,7 @@ JSON esac [ -n "$FAILED_NAMES" ] && { echo; echo "Failing under mutation:"; echo; printf '%s\n' "$FAILED_NAMES" | sed 's/^/- /'; } echo - echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes.</sub>" + echo "<sub>Produced by \`falsify-probe.sh\` at \`$HEAD_SHA\` · node \`$NODE_V\` · yarn.lock \`$LOCK_SHA\` · $DIRTY tracked changes. $(capture_provenance)</sub>" } > "$STAMP.md" printf 'falsify-probe: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh index 19c5d8fc..eaf0ee3d 100755 --- a/domains/pr-workflow/skills/evidence/scripts/render-count.sh +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -35,6 +35,18 @@ # 3 usage error set -uo pipefail +# A run's artifact has to say whether a reader can verify it. In CI the run URL is that +# verification; locally there is none, and the artifact says so rather than leaving the +# omission for a gate to catch later. +capture_provenance() { + if [ -n "${GITHUB_RUN_ID:-}" ]; then + printf 'Run: %s/%s/actions/runs/%s — logs and artifacts attached there.' \ + "${GITHUB_SERVER_URL:-https://github.com}" "${GITHUB_REPOSITORY:-}" "$GITHUB_RUN_ID" + else + printf 'Produced on a local machine: no reader-verifiable capture. Re-run through the evidence workflow before publishing.' + fi +} + OUT_DIR="evidence-artifacts"; LABEL=""; PROBE=""; DEFEAT=""; DEFEAT_LINE=""; DEFEAT_WITH="" ARM_B="memo defeated" die() { printf 'render-count: %s\n' "$1" >&2; exit 3; } @@ -116,7 +128,7 @@ JSON echo "This counts renders of one named consumer across a defined interaction. It is not a count" echo "of consumers, and a larger consumer count does not imply a larger effect." echo - echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`.</sub>" + echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. $(capture_provenance)</sub>" } > "$STAMP.md" printf 'render-count: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index 46b756d2..5cf6e757 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -28,6 +28,18 @@ # --fixture test/data/mock-state.json --slice metamask --perturb pinnedAccountList set -uo pipefail +# A run's artifact has to say whether a reader can verify it. In CI the run URL is that +# verification; locally there is none, and the artifact says so rather than leaving the +# omission for a gate to catch later. +capture_provenance() { + if [ -n "${GITHUB_RUN_ID:-}" ]; then + printf 'Run: %s/%s/actions/runs/%s — logs and artifacts attached there.' \ + "${GITHUB_SERVER_URL:-https://github.com}" "${GITHUB_REPOSITORY:-}" "$GITHUB_RUN_ID" + else + printf 'Produced on a local machine: no reader-verifiable capture. Re-run through the evidence workflow before publishing.' + fi +} + N=5; OUT_DIR="evidence-artifacts"; LABEL=""; MODULE=""; EXPORT=""; FIXTURE=""; SLICE="metamask"; PERTURB="" die() { printf 'selector-recompute: %s\n' "$1" >&2; exit 3; } @@ -178,7 +190,7 @@ JSON grep -E "RECOMPUTE_PROBE |^Test Suites:|^Tests: |^Time: " "$STAMP.log" | head -8 echo '```' echo - echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and kept beside this artifact. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`.</sub>" + echo "<sub>Produced by \`selector-recompute.sh\` via reselect's own counter; the probe is generated, run, and kept beside this artifact. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. $(capture_provenance)</sub>" } > "$STAMP.md" printf 'selector-recompute: %s\n %s\n %s\n' "$VERDICT" "$STAMP.json" "$STAMP.md" >&2 diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh index 0c397eae..cd8e4e03 100755 --- a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -29,6 +29,18 @@ # [--tsc "<command>"] set -uo pipefail +# A run's artifact has to say whether a reader can verify it. In CI the run URL is that +# verification; locally there is none, and the artifact says so rather than leaving the +# omission for a gate to catch later. +capture_provenance() { + if [ -n "${GITHUB_RUN_ID:-}" ]; then + printf 'Run: %s/%s/actions/runs/%s — logs and artifacts attached there.' \ + "${GITHUB_SERVER_URL:-https://github.com}" "${GITHUB_REPOSITORY:-}" "$GITHUB_RUN_ID" + else + printf 'Produced on a local machine: no reader-verifiable capture. Re-run through the evidence workflow before publishing.' + fi +} + TSC="yarn lint:tsc"; OUT_DIR="evidence-artifacts"; LABEL="" FILE=""; LINE=""; REPLACE=""; PROBE_LINE=""; PROBE="" die() { printf 'tsc-substitution: %s\n' "$1" >&2; exit 3; } @@ -128,7 +140,7 @@ JSON echo "subtracted, not disqualifying — only errors new under substitution are the finding.</sub>" fi echo - echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V.</sub>" + echo "<sub>Produced by \`tsc-substitution.sh\`; source restored after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · $TS_V. $(capture_provenance)</sub>" echo } > "$STAMP.md" diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 664f747a..81c1e391 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -407,6 +407,17 @@ Paste the fenced block *beside* one of those, never instead of one. This is also plaintext check can never close the gap: every property of text is forgeable by whatever emits the text, so the gate asks for a different medium rather than for better text. +**Automation is what removes the capture — watch for it.** The runs built by hand, before these +runners existed, attached images and hosted logs: eight of eleven carry a capture a reader can +open. The runs built by the runners attached one in twelve. Nothing was neglected; the causation +runs the other way. A runner emits clean stdout, clean stdout formats beautifully into a fenced +block, and a fenced block looks like evidence. The hand-built runs had no such thing to reach +for, so they went and got a real one. + +Every runner now states this in its own artifact: in CI it prints the run URL, and on a local +machine it prints *"no reader-verifiable capture — re-run through the evidence workflow before +publishing."* The confession is in the exhibit rather than left for a gate to catch. + **If the artifact is small, nothing was attached.** The reference showcase runs to 2 MB because it carries 31 embedded captures. Uploading costs a step and a decision about what may be published; that cost is the price of the reader not having to trust you, and a pipeline with no From 56578cee0f679881e6f928177ef3cf6d45a5bfec Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sun, 2 Aug 2026 08:11:50 -0400 Subject: [PATCH 097/135] Cite what exists; capture what you ran MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Plaintext is the wrong medium" was too broad. The rule is about where verification routes, and a line-level permalink routes it away from the author exactly as an image does — the reader clicks and sees what you saw. That is the normal case for the audit lanes, whose findings are facts about code that exists rather than results of running something. A screenshot of a policy diff is less checkable than a permalink to it, not more. The bar there is comprehensive linking: a call site per grant, a re-runnable search per claimed absence, a file and line per version claim. A row naming a package and a capability with no link is a claim on the author's word in a technical register. --- domains/pr-workflow/skills/evidence/skill.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 81c1e391..26fc0bbc 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -407,6 +407,26 @@ Paste the fenced block *beside* one of those, never instead of one. This is also plaintext check can never close the gap: every property of text is forgeable by whatever emits the text, so the gate asks for a different medium rather than for better text. +**The exception — plaintext where every claim is a citation.** The rule is about where +verification routes, not about pixels. Line-level links are externally verifiable: the reader +clicks and sees exactly what you saw. That is the *normal* case for the audit lanes — +`supply-chain-audit`, `lavamoat-policy-diligence`, `privacy-egress-diligence` — whose findings +are facts about code that exists rather than results of running something. There an image would +be worse: a screenshot of a policy diff is less checkable than a permalink to it. + +The bar for those lanes is comprehensive linking, not a link somewhere nearby: + +- every capability grant → a permalink to its **call site** in the dependency's source, at the + installed version, with the line; +- every *"no call site uses this"* → the **search that establishes the absence**, re-runnable; +- every version, advisory, or policy claim → the file and line it came from. + +An audit row naming a package and a capability with no link is the same defect as a bare +console block — a claim on your word, wearing a technical register. + +The general form: ask what the reader must do to check a claim. *Trust the transcription* means +the medium is wrong whatever it looks like. **Cite what exists; capture what you ran.** + **Automation is what removes the capture — watch for it.** The runs built by hand, before these runners existed, attached images and hosted logs: eight of eleven carry a capture a reader can open. The runs built by the runners attached one in twelve. Nothing was neglected; the causation From f6d76e87980444d2ee5c4f81957b29f21665da45 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sun, 2 Aug 2026 09:55:22 -0400 Subject: [PATCH 098/135] Fold the runner-workflow fixes back from where they were found MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four corrections, each from a run that failed rather than from review. `corepack enable` before `setup-node`: a target pinning its package manager via `packageManager` makes setup-node's `cache: yarn` probe run under the runner's global yarn 1.22, which refuses — every jest runner died at setup with nothing measured. `logs` added to the determinism exclusions, since `render-count` is the one runner writing the plural key and so failed that check on every run while reporting identical counts; a warning always wrong for one runner teaches its reader to publish through it. A `probe_path` input, because `render-count` takes a hand-written probe and a probe living only on the author's disk is the exact defect a run URL exists to remove. And `skills_repo`/`skills_ref`, because a runner fix and the run that needs it cannot both wait on a review. The sparse checkout now also pulls `domains/security/skills`, so the analysis scripts wrapped by `capture` are on disk without being runners themselves. --- .../skills/evidence/assets/evidence-run.yml | 75 +++++++++++++++++-- 1 file changed, 67 insertions(+), 8 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/assets/evidence-run.yml b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml index b3db840f..1a65b24d 100644 --- a/domains/pr-workflow/skills/evidence/assets/evidence-run.yml +++ b/domains/pr-workflow/skills/evidence/assets/evidence-run.yml @@ -66,6 +66,30 @@ name: Evidence run had no comparison, so the baseline arm is offered here rather than left to memory. required: false type: string + skills_repo: + description: >- + Where to source the runners. Defaults to upstream. Overridable because a runner + fix and the run that needs it cannot both wait on a review: point this at a fork + branch, and say in the artifact that you did. + required: false + default: MetaMask/skills + type: string + skills_ref: + description: >- + Ref within skills_repo. Overrides the EVIDENCE_SKILLS_REF variable. + required: false + type: string + probe_path: + description: >- + Path, within skills_repo, of a probe file to copy into the target tree before the + runner executes. `render-count` takes a hand-written probe, and a probe that lives + only on the author's disk is the exact defect the run URL exists to remove. + required: false + type: string + probe_dest: + description: Where in the target tree to place probe_path. Required with probe_path. + required: false + type: string permissions: contents: read @@ -96,6 +120,15 @@ jobs: ref: ${{ inputs.ref }} fetch-depth: 2 # the runners diff against the parent + # Before setup-node, not after. A target that pins its package manager through + # `packageManager` in package.json makes setup-node's `cache: yarn` probe run + # `yarn cache dir` under the runner's global yarn 1.22, which refuses and fails + # the step — so every jest runner died at setup with nothing measured. The target + # repo's own workflows order it exactly this way. + - name: Enable corepack + if: inputs.needs_install + run: corepack enable + - uses: actions/setup-node@v4 if: inputs.needs_install with: @@ -113,10 +146,18 @@ jobs: - name: Fetch the runners at a pinned version uses: actions/checkout@v6 with: - repository: MetaMask/skills - ref: ${{ vars.EVIDENCE_SKILLS_REF || 'dbc275429d92d00daf4c9138edabe93f526fc26b' }} + repository: ${{ inputs.skills_repo || 'MetaMask/skills' }} + ref: ${{ inputs.skills_ref || vars.EVIDENCE_SKILLS_REF || '56578cee0f679881e6f928177ef3cf6d45a5bfec' }} path: .evidence-skills - sparse-checkout: domains/pr-workflow/skills/evidence/scripts + # The security-domain analysis scripts (policy-audit.py and its siblings) are + # wrapped by the `capture` runner rather than being runners themselves, so they + # need no entry in the `runner` choice list — but they do need to be on disk. + # A sparse path absent from the chosen ref is silently empty, so listing it here + # costs nothing when it is not there. + sparse-checkout: | + domains/pr-workflow/skills/evidence/scripts + domains/pr-workflow/skills/evidence/probes + domains/security/skills - name: Verify the runners arrived env: @@ -131,9 +172,24 @@ jobs: echo "::error::Set the EVIDENCE_SKILLS_REF repository variable to a commit that has it." exit 1 } - echo "runner $RUNNER.sh sourced from skills @ ${{ vars.EVIDENCE_SKILLS_REF || 'pinned default' }}" \ + echo "runner $RUNNER.sh sourced from ${{ inputs.skills_repo || 'MetaMask/skills' }} @ ${{ inputs.skills_ref || vars.EVIDENCE_SKILLS_REF || 'pinned default' }}" \ >> "$GITHUB_STEP_SUMMARY" + - name: Place the probe + if: inputs.probe_path != '' + env: + SRC: .evidence-skills/${{ inputs.probe_path }} + DEST: ${{ inputs.probe_dest }} + run: | + # Copied from the runners checkout, so the probe has a permalink of its own and + # the reader can see the file that produced the count rather than taking the + # count on the author's word. + [ -n "$DEST" ] || { echo "::error::probe_dest is required with probe_path"; exit 1; } + [ -f "$SRC" ] || { echo "::error::probe not found at $SRC on the chosen skills ref"; exit 1; } + mkdir -p "$(dirname "$DEST")" + cp "$SRC" "$DEST" + echo "probe $SRC -> $DEST" >> "$GITHUB_STEP_SUMMARY" + - name: Run id: run continue-on-error: true # the exit code IS the verdict; a finding is not a failure @@ -180,11 +236,14 @@ jobs: A="evidence-artifacts/$RUNNER-head.json" B="evidence-artifacts/$RUNNER-repeat.json" if [ -f "$A" ] && [ -f "$B" ]; then - # `label` and `log` name the arm, and `env` carries timing — all three differ + # `label`, `log` and `logs` name the arm, and `env` carries timing — all differ # between the two runs by construction. Comparing them makes the check fire on - # every run, which is the same as not having it. - if diff <(jq -S 'del(.env, .label, .log)' "$A") \ - <(jq -S 'del(.env, .label, .log)' "$B") > determinism.diff; then + # every run, which is the same as not having it. `logs` was missing from this + # list, so `render-count` — the only runner that writes the plural key — failed + # the check on every run while reporting identical counts. A warning that is + # always wrong for one runner teaches the operator to publish through it. + if diff <(jq -S 'del(.env, .label, .log, .logs)' "$A") \ + <(jq -S 'del(.env, .label, .log, .logs)' "$B") > determinism.diff; then echo "deterministic across two runs" | tee -a "$GITHUB_STEP_SUMMARY" else echo "::warning::runner is NOT deterministic at this ref — do not publish these numbers" From 0b874cf1e3681a1c0580cb5512617a453cd3abdb Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sun, 2 Aug 2026 16:10:08 -0400 Subject: [PATCH 099/135] Name the commit in the command line, so an A/B pair is not one line twice A base arm and a head arm both printed `yarn jest <probe>`, leaving the reader no way to tell which commit produced which number. On a clean result, where the two arms agree, the block reads as a single measurement printed twice. The publish gate fails a pair of identical `$` lines for exactly this reason and caught it on a real comment. Fixing the check's input rather than the check. --- domains/pr-workflow/skills/evidence/scripts/render-count.sh | 2 +- .../pr-workflow/skills/evidence/scripts/selector-recompute.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh index eaf0ee3d..05de89a1 100755 --- a/domains/pr-workflow/skills/evidence/scripts/render-count.sh +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -120,7 +120,7 @@ JSON [ -n "$B" ] && echo "| B — $ARM_B | \`$DEFEAT:$DEFEAT_LINE\` | $B |" echo echo '```console' - echo "\$ yarn jest $PROBE" + echo "\$ git checkout --detach $HEAD_SHA && yarn jest $PROBE" echo "$A_LINE" [ -n "$B_LINE" ] && { echo "\$ yarn jest $PROBE # $ARM_B"; echo "$B_LINE"; } echo '```' diff --git a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh index 5cf6e757..029d5409 100755 --- a/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh +++ b/domains/pr-workflow/skills/evidence/scripts/selector-recompute.sh @@ -184,7 +184,7 @@ JSON fi echo echo '```console' - echo "\$ yarn jest $PROBE" + echo "\$ git checkout --detach $HEAD_SHA && yarn jest $PROBE" # The tool's own output, not a line this script composed. A summary a script writes # about its own run carries the script's word; the runner's stdout carries the run's. grep -E "RECOMPUTE_PROBE |^Test Suites:|^Tests: |^Time: " "$STAMP.log" | head -8 From a64ddb83d3de55066c99a81a18046d6a15ff4419 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sun, 2 Aug 2026 16:38:55 -0400 Subject: [PATCH 100/135] Stop a broken substitution reading as a divergence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First CI run of this runner, and it reported `divergence surfaced` with six new errors — all TS1109/TS1011/TS1128, the syntactic family. The substitution had landed one line above the type declaration and broken parsing, so arm B never type-checked at all. By error count that is indistinguishable from the local type genuinely disagreeing with its authoritative source. `falsify-probe` has carried the equivalent guard since a syntax-breaking mutation looked like a falsification. This runner shipped without one and had never executed in CI, which is where the gap surfaced. --- .../skills/evidence/scripts/tsc-substitution.sh | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh index cd8e4e03..48308f98 100755 --- a/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh +++ b/domains/pr-workflow/skills/evidence/scripts/tsc-substitution.sh @@ -100,7 +100,16 @@ restore; trap - EXIT INT TERM NEW_ERRS="$(comm -13 "$STAMP-armA.set" "$STAMP-armB.set" | head -12)" NEW_COUNT="$(comm -13 "$STAMP-armA.set" "$STAMP-armB.set" | wc -l | tr -d ' ')" -if [ "$NEW_COUNT" -gt 0 ]; then +# TS1xxx is the syntactic family — "expression expected", "declaration expected". A +# substitution that lands on the wrong line breaks parsing and produces a pile of them, +# which reads as a large divergence and is worth nothing: the file never type-checked. +# `falsify-probe` has carried this guard since a broken mutation looked like a +# falsification; this runner shipped without it and reported six syntax errors as a +# divergence on its first CI run. +SYNTAX="$(comm -13 "$STAMP-armA.set" "$STAMP-armB.set" | grep -cE 'error TS1[0-9]{3}')" +if [ "$NEW_COUNT" -gt 0 ] && [ "$SYNTAX" -eq "$NEW_COUNT" ]; then + VERDICT="substitution broke parsing — $NEW_COUNT syntax error(s), nothing type-checked"; CODE=2 +elif [ "$NEW_COUNT" -gt 0 ]; then VERDICT="divergence surfaced"; CODE=0 else VERDICT="substitution silent"; CODE=1 @@ -127,8 +136,14 @@ JSON echo "| **new under substitution** | | **$NEW_COUNT** |" echo if [ "$NEW_COUNT" -gt 0 ]; then + if [ "$CODE" -eq 2 ]; then + echo "**No conclusion.** Every new error is syntactic, so arm B never type-checked —" + echo "the substitution landed on the wrong line or produced invalid TypeScript. This is" + echo "indistinguishable from a real divergence by error count alone." + else echo "Errors present in B and absent in A — what the local type was concealing:" echo; echo '```'; printf '%s\n' "$NEW_ERRS"; echo '```' + fi else echo "**Silent — this is not proof of agreement.** Existing call sites may satisfy both" echo "shapes; indexing and \`.match()\` compile against \`string\` and \`string[]\` alike." From bfdf6b4a37d9eabe510daa02ebf9c2409977716b Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Sun, 2 Aug 2026 17:50:06 -0400 Subject: [PATCH 101/135] Require a run to measure the PR's range, and to say where its reach ends MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four runs measured `$SHA^..$SHA` on branches of five to twenty-nine commits and produced clean artifacts for a fraction of each change — same runner, same green run, nothing in the output distinguishing it from a finished measurement. So the range is now a non-negotiable, with the compare endpoint's `merge_base_commit.sha` named because `.base.sha` is the base branch tip and moves. Two rules alongside it, from the same batch: what a run could not see is a finding to state rather than a gap to omit, and the label on a number is caller-stated for the same reason the verdict is — a probe counting distinct context values published under a fixed "renders" heading passes every check while naming the wrong quantity. --- domains/pr-workflow/skills/evidence/skill.md | 26 ++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 26fc0bbc..584342f5 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -177,6 +177,32 @@ untested path touches funds, keys, persisted state, user-visible wrongness, or s planning tracker. Subject matter triggers this, not severity: the code cannot distinguish a missing gate from a deliberate one. +**7. Measure the pull request, which is a range, not a commit.** `$SHA^..$SHA` is one commit's +diff. On a twenty-six-commit branch it is a twenty-sixth of the change, and it looks exactly like a +finished measurement — same runner, same green run, same artifact. Take the head from +`.head.sha` and the base from `merge_base_commit.sha` on the compare endpoint, and say the range in +the comment so a reader can see what was covered. `.base.sha` is the base branch's tip, which moves +under you and is not where the branch left. + +**8. The label on a number is part of the number.** A runner reads a field out of a line its probe +printed; it knows the field's name and not what was counted. When a probe for a claim about value +identity counts distinct values, publishing that under a fixed heading of "renders" ships a correct +measurement of the wrong quantity, and every check passes. Whatever names the number is caller- +stated, like the verdict — and the comment points at the probe, which is the definition. + +### The claim is scoped to what the run could see + +A run measures a diff, a file, a probe. What sits behind an interface it calls is not in the +measurement, and a comment that speaks past that boundary is asserting rather than reporting. + +The instrumentation lanes make this concrete: a diff can show that a span is created and that a +flag gates it, and cannot show how often the surrounding package invokes the callback. That is not +a gap to apologise for — it is the finding. *"Cost scales with a call frequency decided in another +package, so nothing here bounds it"* is a real conclusion, and the reviewer is the person who knows +the number. + +Say where the edge is, in the comment, in the reviewer's terms. + ### The runner, not the recipe `scripts/falsify-probe.sh` proves a test is falsifying by mutation rather than by reading, and From faff610d0f642ba8de515a07a3b1b43f045a9f27 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 03:30:13 -0400 Subject: [PATCH 102/135] Require an instrument to publish the effect it had, not the instruction it took A mutation runner echoed its `--replace` argument into the artifact, so the two could never disagree. They did: an `awk -v` assignment escape-processed the value and wrote a different line than the one requested, narrowing a regex meant to be widened. The suite ran the same test count in both arms, a different test failed than the one targeted, and the run reported power over a mechanism it never touched. Nothing in the artifact could have shown it. --- domains/pr-workflow/skills/evidence/skill.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 584342f5..334133f6 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -190,6 +190,16 @@ identity counts distinct values, publishing that under a fixed heading of "rende measurement of the wrong quantity, and every check passes. Whatever names the number is caller- stated, like the verdict — and the comment points at the probe, which is the definition. +**9. An instrument reports what it did, never what it was asked to do.** A mutation runner that +echoes its `--replace` argument into the artifact cannot detect its own misfire, because the two +are the same string by construction. They came apart once: `awk -v r="$REPLACE"` escape-processes +the assignment, so a replacement of `/^[\s\S]{1,4096}$/u` was written to the file as +`/^[sS]{1,4096}$/u` — narrowing the regex it was meant to widen. Arm B ran the same test count as +arm A, so every guard was satisfied, a different test failed than the one targeted, and the run +reported the suite as having power over a mechanism it never touched. Read the mutated line back +off disk and publish that; keep the requested text beside it. The rule generalises past mutation: +wherever a runner takes an instruction and performs an effect, the artifact carries the effect. + ### The claim is scoped to what the run could see A run measures a diff, a file, a probe. What sits behind an interface it calls is not in the From 5086ffe2a71268a278c3c74a0010cc04ea895e56 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 05:37:49 -0400 Subject: [PATCH 103/135] Add two PR-audit skills that read the description in opposite orders MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `falsifiers-first` seals the description until its hypotheses are fixed. A run that takes its falsifiers from the author's sentences probes the mechanisms the author already phrased clearly, mirrors their method so agreement carries no information, and has no bucket for effects nobody claimed — which is where the findings worth having usually sit. `unintended-breakage` reads the description first, because out-of-scope is a relation between code and a stated intent rather than a property of code. It fixes the envelope, enumerates effects against it, and requires a mechanical check per candidate: an export removed still has importers, a locale key still has lookups, a renamed event still has a dashboard. Its most valuable tier is the one nothing fails on — renamed events, flipped defaults, changed state shapes. Each names the other and says why the order differs, so neither gets harmonised into the other later. --- .../skills/falsifiers-first/skill.md | 131 ++++++++++++++++++ .../skills/unintended-breakage/skill.md | 118 ++++++++++++++++ 2 files changed, 249 insertions(+) create mode 100644 domains/pr-workflow/skills/falsifiers-first/skill.md create mode 100644 domains/pr-workflow/skills/unintended-breakage/skill.md diff --git a/domains/pr-workflow/skills/falsifiers-first/skill.md b/domains/pr-workflow/skills/falsifiers-first/skill.md new file mode 100644 index 00000000..ef1a17e5 --- /dev/null +++ b/domains/pr-workflow/skills/falsifiers-first/skill.md @@ -0,0 +1,131 @@ +--- +name: falsifiers-first +description: Derive what could break from the diff before reading the PR description, test those hypotheses, and only then compare the results against what the PR claims. Inverts the usual order so the author's wording cannot decide which mechanisms get probed, and produces a bucket no description-led run can — effects the change has that nobody claimed. Use when validating a PR whose description makes testable assertions, when a previous run confirmed everything and taught nobody anything, or when the change is security-relevant and the interesting failure is the one not mentioned. Pairs with the evidence skill's runners for execution and the attest gate before publishing. +--- + +# /falsifiers-first + +Read the diff. Write down what could break. Test it. **Then** open the description. + +The order is the whole skill. Everything else here exists to keep it. + +## Why the order is load-bearing + +A description-led run picks its falsifiers out of the author's sentences, so it probes the +mechanisms the author already thought about and phrased clearly. That produces three failures +at once, and they compound: + +**The probe lands on the most legible line, not the weakest one.** A condition the description +names is a condition the tests already isolate, so mutating it is the experiment most likely to +be caught and least likely to teach anything. + +**Your method mirrors theirs.** If the description says "a diff audit found no X" and you audit +the diff for X, a blind spot in their grep is a blind spot in yours, and agreement carries +almost no information. Two people running the same check is one check. + +**There is no bucket for what the description omits.** This is the expensive one. A run whose +hypotheses come from the claims can, structurally, only confirm or refute claims. The most +valuable finding on a change is routinely something nobody wrote down — and it is unreachable +from a method that starts by reading what was written down. + +Sealing is what buys all three back. Fixing the hypotheses before seeing what they will be +compared against is the same discipline as pre-registering a study, and for the same reason. + +## Phase A — the diff alone + +Do not open the description. Do not read the linked ticket. If the PR title is on screen, +ignore it; a title is a claim. + +Read the change and answer, in this order: + +1. **What does this actually do?** Mechanism by mechanism, in your words, from the code. +2. **Where can each mechanism fail?** Not "is it wrong" — *what would have to be true for it to + be wrong*, stated so it could be checked. +3. **For each, what is the smallest defeat?** Usually one line: a condition inverted, a guard + disabled, a bound widened, a call replaced by a constant. +4. **Which test should fail when you apply it?** By name. This is a prediction, and writing it + down is what makes the run falsifiable rather than decorative. + +Write the list to a file before anything runs. A hypothesis you can revise after seeing the +result is not a hypothesis. + +**Rank by where refutation would be most surprising**, not by what is easiest to reach. A +mechanism with one obvious guard and three tests around it is well covered by construction; the +mechanism assembled from two files and no direct test is where the run earns its cost. + +### Choosing defeats that mean something + +- **Prefer a value or a call over a boolean.** Flipping `||` to `&&` probes operands the suite + usually isolates already. Widening a validation pattern, or replacing a verification call + with a constant, reaches code that guards rather than code that branches. +- **A mechanism split across two lines can only be attacked at one end.** Set-here, check-there + guards need a decision about which end, and the answer is usually the one with no test + pointing at it. +- **A defeat that breaks parsing proves nothing.** Every test fails, which looks identical to a + falsification by exit code. Keep every identifier referenced so the module still loads. + +## Phase B — run them + +Each hypothesis becomes one probe with its prediction attached. The runners in the evidence +skill take the prediction directly, and a probe that fires somewhere other than predicted is +its own outcome rather than a pass. + +Record every result, including the ones that refuse to reproduce. A hypothesis that survives +its defeat is a finding — the mechanism is better covered than it looked — and dropping it +because it was not interesting is how a run becomes a highlight reel. + +**Nothing about the description enters here.** If a result makes you want to check what the +author said, that is the phase working; write the impulse down and keep it sealed. + +## Phase C — open the description, and sort + +Now read it. Sort every measured result into one of three buckets, and report all three: + +| bucket | meaning | +|---|---| +| **Supported** | the measurement and the claim agree | +| **Contradicted** | the measurement and the claim disagree | +| **Unclaimed** | the change does this, and the description does not say so | + +**The third bucket is the product.** It is why the phases are ordered this way and it is the +only one a description-led run cannot produce. It catches undersold changes as well as oversold +ones — a mechanism that is more careful than advertised belongs there too, and saying so is +worth as much to a reviewer as catching an overstatement. + +### Scope discipline in phase C + +Diff-first search reaches code the PR did not touch, and it will find real things there. Report +them, and do not report them as marks against this change. Two rules: + +- A finding outside the diff is attributed to the system, never to the author. "These three + routes forward raw parameters" is publishable; "your PR fails to handle" is not, when the PR + never went near them. +- If a finding outside the diff is security-relevant, it leaves the PR entirely and goes to the + private tracker. Venue is decided by subject, not by severity. + +## Failure modes this skill has, and what they look like + +**Cost.** Enumerating falsifiers across a large diff is a search problem; reading four sentences +is not. On a change past a few hundred lines, phase A is most of the run. Scope by mechanism +rather than by file — a 5,000-line diff often has four mechanisms. + +**Lost signal.** A description's manual-testing steps frequently name the exact edge the author +was worried about, which is genuine information you are declining to use during phase A. You +get it back in phase C. Sealed is not discarded. + +**Phase A rationalising.** The tell is a hypothesis phrased so that any outcome confirms it. +If you cannot name the observation that would make you say "no, it holds", it is not a +hypothesis, it is a suspicion. + +**Leakage.** CI check names, review comments, commit messages and branch names all carry the +author's framing. Perfect sealing is not achievable; note what leaked rather than pretending it +did not. + +## Related + +- [`evidence`](../evidence/skill.md) — the runners that execute phase B, and the gate that + decides whether phase C's writeup is publishable +- [`unintended-breakage`](../unintended-breakage/skill.md) — the mirror of this skill, which + reads the description *first* because "out of scope" is undefinable without a stated scope +- [`pr-readiness-check`](../pr-readiness-check/skill.md) — checklist-shaped review, which this + deliberately is not diff --git a/domains/pr-workflow/skills/unintended-breakage/skill.md b/domains/pr-workflow/skills/unintended-breakage/skill.md new file mode 100644 index 00000000..c0634c8e --- /dev/null +++ b/domains/pr-workflow/skills/unintended-breakage/skill.md @@ -0,0 +1,118 @@ +--- +name: unintended-breakage +description: Find the things a PR breaks that it did not set out to change. Reads the description first to fix the intended scope, then reads the diff for effects outside it, then tests whether any of those effects breaks a consumer — a removed export still imported, a deleted locale key still referenced, a persisted state shape with no migration, an analytics event renamed out from under a dashboard, a default flipped. Use before merging a refactor, a re-land, a rename, or any change whose description says it preserves behaviour. Reports breakage by who it breaks and how it was detected, not by suspicion. +--- + +# /unintended-breakage + +A change that breaks something it meant to break is a decision. A change that breaks something +it never mentions is a defect, and it is invisible to every review that starts from the diff — +because the diff looks intentional all the way through. + +This skill finds the second kind. + +## Read the description first — and why that is the opposite of the sibling skill + +[`falsifiers-first`](../falsifiers-first/skill.md) seals the description, because letting the +author's claims choose your hypotheses produces confirmation. Here the order inverts, and the +inversion is principled rather than a matter of taste: + +**"Out of scope" is not a property of code. It is a relation between code and a stated +intent.** You cannot detect a departure from an envelope you have not read. So the description +comes first, and its job is not to be tested — it is to fix the boundary that everything after +is measured against. + +Do not harmonise the two skills. Different question, different order. + +## Phase 1 — fix the envelope + +From the description, the linked ticket and the title, write down: + +- **What is meant to change**, in mechanisms and in surfaces. +- **Who is meant to notice.** Users, reviewers, a downstream package, a dashboard, nobody. +- **What is declared behaviour-preserving.** "Refactor", "re-land", "no functional change", + "converts X to Y" — each is a promise that the observable stays put. +- **What is explicitly excluded.** A PR that says it leaves something alone has handed you a + falsifiable statement and a boundary at once. + +A description that promises preservation is the strongest input this skill takes. It converts +every behavioural difference into a finding. + +## Phase 2 — enumerate effects, ignoring intent + +Now read the diff and list what it *does*, with the envelope out of mind. You are building the +left side of a subtraction; judgement comes after. + +The surfaces where unintended breakage actually lives, in rough order of how often they bite: + +| surface | what to look for | how it breaks | +|---|---|---| +| **Removed or renamed exports** | symbols deleted from a module's public surface | an importer that was not updated | +| **Deleted files** | any file removed, including tests | something still imports it, or the only coverage of a path just left | +| **Localisation keys** | message keys removed or renamed | a lookup at runtime that now renders a key or nothing | +| **Persisted state** | controller state shape, storage keys, defaults | old state read by new code with no migration | +| **Event and property names** | analytics events, their properties, their values | a dashboard, funnel or alert keyed on the old string | +| **Public types** | a widened parameter, a narrowed return, a new required field | a consumer that compiled yesterday | +| **Defaults** | feature flags, config, optional parameters gaining a value | behaviour changes for everyone who never opted in | +| **Dependency ranges** | a bump, a peer widened, a resolution pinned | a transitive consumer resolves differently | +| **DOM and test hooks** | test ids, class names, aria roles other code selects on | an e2e suite or a sibling app's selector | + +Two of these deserve special weight in a monorepo-adjacent codebase: **persisted state** and +**event names**, because both cross a boundary the type system does not see. A renamed event is +a silent break — nothing fails to compile, nothing fails a test, and a dashboard goes flat. + +## Phase 3 — subtract, then test what is left + +Effects minus envelope is your candidate list. Now the part that separates this from a hunch: +**every candidate has a mechanical check, and a candidate without one is not reportable.** + +| candidate | the check | +|---|---| +| export removed | search the tree at head for remaining importers | +| file deleted | same, plus whether its tests covered a path nothing else does | +| locale key removed | search for the key in code and in the other locales | +| state shape changed | round-trip old persisted state through the new code; is there a migration, does it run | +| event renamed | diff the event constants between base and head, and search the analytics surface for the old name | +| type changed | typecheck a consumer against the new signature | +| default flipped | read the default at base and head, and find who reads it | +| dependency bumped | resolve the lockfile both ways and diff the resolved versions | + +Run the check. Report what it returned. A candidate whose check comes back clean is still worth +one line — it says the surface was examined, which is what stops the next reviewer re-deriving +it. + +**Do not report a candidate you could not check.** Say the surface exists and the check was not +available; that is honest and useful. A list of things that might break, with nothing run +against any of them, is a worry masquerading as a review. + +## What counts as breaking + +Breakage is defined by a consumer, so name one. "This export was removed" is not a finding. +"This export was removed and these three files still import it" is. If you cannot name who +notices, you have found a change, not a break — and changes are the author's business. + +Three tiers worth distinguishing in the report: + +- **Breaks now** — a consumer in this repository is already inconsistent at head. +- **Breaks on contact** — nothing in this repository consumes it, but a published surface + changed, so a downstream consumer will find out later. +- **Breaks silently** — nothing fails; an observable moves. Renamed events, changed defaults, + altered state shapes. These are the ones worth the most and get flagged the least. + +## Report shape + +Group by surface, not by file. For each finding: what changed, who consumes it, what the check +returned, and which tier. Lead with anything in the **breaks silently** tier, because it is the +tier a reviewer cannot find by reading. + +Close with the surfaces you checked and found clean. A review that lists only hits reads as a +search that stopped when it found something. + +## Related + +- [`falsifiers-first`](../falsifiers-first/skill.md) — the mirror: seals the description, + because it is testing whether the claims hold rather than where the change exceeds them +- [`evidence`](../evidence/skill.md) — runners for the checks in phase 3, and the gate that + decides whether the report is publishable +- [`pr-changelog`](../pr-changelog/skill.md) — where a confirmed breaking change has to end up + once it is established From f046789e905308050ff6e065c5da5cc0af1000b1 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 05:40:53 -0400 Subject: [PATCH 104/135] Add `silent-failure`, which tests detectability rather than correctness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Asks whether a mechanism announces its own failure, which is a separate property from whether it can fail and is almost never tested. Since a silent failure cannot be observed, the method induces it and watches for a signal that never arrives — and the verdict reads opposite to a normal mutation probe, where a green suite means a vacuous test. Here green means the failure is undetectable, which is the result. Findings rank by distance to discovery rather than by severity: a small error nobody can see outranks a large one that pages someone in a minute. Half the listed shapes are instrument failures — a name that does not match what is counted, a pattern that cannot match what it searches for — because a measurement that fails quietly gets published. --- .../skills/silent-failure/skill.md | 111 ++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 domains/pr-workflow/skills/silent-failure/skill.md diff --git a/domains/pr-workflow/skills/silent-failure/skill.md b/domains/pr-workflow/skills/silent-failure/skill.md new file mode 100644 index 00000000..94016f62 --- /dev/null +++ b/domains/pr-workflow/skills/silent-failure/skill.md @@ -0,0 +1,111 @@ +--- +name: silent-failure +description: Find the paths where this code can fail without anything saying so, and prove it by inducing the failure and watching for a signal that never comes. Asks a different question from correctness — not "can this break" but "if it breaks, would we know" — and treats detectability as a property to be tested rather than assumed. Use on error handling that swallows, fallbacks that substitute a default, caches and guards that fail open, retries that mask, instrumentation whose absence looks like health, and any measurement whose wrong answer is well-formed. Inverts the usual mutation verdict: a suite that stays green under an induced failure is the result, not a disappointment. +--- + +# /silent-failure + +A loud failure is a solved problem: someone sees it, someone fixes it. The expensive failures +are the ones where the system keeps going, reports success, and the only evidence is an +observable nobody is watching. + +This skill hunts those, and it proves them rather than suspecting them. + +## The question + +For every mechanism on the path, ask **not** "can this be wrong" but: + +> If this were wrong right now, what would be different? + +Three answers, and only one of them is comfortable: + +- **A test fails, a log appears, a metric moves.** Detectable. Move on. +- **A different, plausible value is produced.** Silent. This is the dangerous one, because the + output is well-formed and the reader has no reason to doubt it. +- **Nothing whatsoever.** Silent and unbounded, and it will be discovered by a user. + +The second answer is where most of the real ones live, and it is why "can it fail" is the wrong +question. Nobody ships a mechanism that explodes. They ship one that returns `0`, or `[]`, or +the previous value, or the default. + +## Where to look + +These are the shapes that produce silence, ordered by how often they turn out to be real: + +| shape | why it goes quiet | +|---|---| +| `catch` that returns a default | the error is the signal, and it was consumed | +| optional chaining on the thing that does the work | `a?.b?.()` is a no-op when the dependency is missing, and a no-op looks like success | +| a fallback that substitutes a plausible value | the wrong answer is well-formed | +| a guard that fails open | the unprotected path is also the working path | +| a cache or memo whose key is incomplete | a stale value is a valid value | +| a bounded buffer that evicts | the evicted item is indistinguishable from one that completed | +| a retry that eventually succeeds | the failures never reach anyone | +| instrumentation that is conditionally off | absent data reads as healthy, not as unmeasured | +| a name that does not match what is counted | the number is right and describes something else | +| a check whose pattern cannot match the failure | absence of matches reads as absence of the thing | + +The last three are about the *instruments*, and they belong here for a reason: a measurement +that fails silently is worse than one that fails loudly, because its output gets published. + +## Inducing it + +You cannot observe a silent failure — that is its definition. So make it happen and watch for a +signal that never comes. + +For each candidate: + +1. **Name the signal you expect.** A test by name, a log line, a metric, a thrown error. If you + cannot name one before you start, you have already found the answer. +2. **Induce the failure at its source**, minimally, without breaking parsing. Make the + dependency absent, the guard fail, the cache return stale, the buffer evict, the flag off. +3. **Run everything that could plausibly notice** — not just the module's own tests. The point + is breadth of detection, so the widest suite that could reasonably fire is the right one. +4. **Record what went red.** Nothing going red is the finding. + +**The verdict is inverted from a normal falsification probe.** There, a suite that stays green +under mutation means the test is vacuous and you go fix the test. Here, a suite that stays green +under an induced failure means *the failure is undetectable*, which is a property of the system +and a legitimate result to report. Say which reading you are applying, in the artifact, because +the same green output supports both and they are opposite conclusions. + +## Reporting + +For each confirmed silent path, three facts and no adjectives: + +- **The induced failure** — what was made to go wrong, and where. +- **What was watched** — the suites, logs and metrics that had a chance to notice. +- **What happened** — ideally a captured run showing green under a broken mechanism. + +Then one line on **who finds out, and when**. That is the sentence a reviewer acts on. "Nothing +detects this; it surfaces as a support ticket" and "nothing detects this; the value is wrong by +a factor of two in a dashboard" are different findings even though the mechanism is identical. + +Do not rank by severity of the failure. Rank by **distance to discovery** — how long the wrong +state persists before anyone can see it. A small error nobody can detect outranks a large one +that pages someone in a minute. + +## What this skill is not + +**Not error handling review.** A `catch` that swallows is a candidate, not a finding. It +becomes a finding when the induced failure produces no signal anywhere. + +**Not the same as unintended breakage.** [`unintended-breakage`](../unintended-breakage/skill.md) +asks what a change broke that nobody meant, and one of its tiers is breakage that surfaces +quietly. This skill asks whether a mechanism — new or ten years old, changed or untouched — +announces its own failure. Different question, different input, and they overlap only on the +one tier. + +**Not a search for missing tests.** A path with no test but a loud runtime error is detectable. +A path with full coverage that returns a default on failure is not. Coverage and detectability +are independent, and confusing them produces a report about test quality when the finding was +about observability. + +## Related + +- [`falsifiers-first`](../falsifiers-first/skill.md) — supplies the induction technique; this + skill reads its verdict in the opposite direction +- [`unintended-breakage`](../unintended-breakage/skill.md) — overlaps only at its + breaks-silently tier, and starts from a change rather than from a mechanism +- [`evidence`](../evidence/skill.md) — the runners that induce the failure and capture the + green output that proves nothing noticed From 4fa12d7bde18e84596801074f1000d841f7f90b2 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 05:51:35 -0400 Subject: [PATCH 105/135] Add five reasoning audits and two diagnostic skills MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each encodes a failure that actually shipped, rather than a principle that sounded right. `unmeasured-join` targets the costliest one — a conclusion assembled from true facts through a step nobody measured, which survives review because the facts check out and the join reads as prose. `instrument-check` requires a positive and a negative control before a measurement counts, after a mutation runner reported a falsification for a line it never wrote. `scope-of-search` makes a negative carry the pattern that produced it, after a grep keyed on a variable name declared absent what its own output printed two blocks later. `coverage-partition` replaces "the test has power" with which cases guard which mechanism. `selection-audit` separates a count from the rule that produced it, after two different selections landed on the same number and the match closed the question. `distinguishing-observation` and `observability-gap` point the same discipline at debugging: design the observation that separates candidates rather than confirms the favourite, and establish what signal exists on a path before reading more of it. The `evidence` links are forward references — that skill ships in #84 and is not on main yet. --- .../distinguishing-observation/skill.md | 106 +++++++++++++++ .../coding/skills/observability-gap/skill.md | 114 ++++++++++++++++ .../skills/coverage-partition/skill.md | 128 ++++++++++++++++++ .../skills/instrument-check/skill.md | 125 +++++++++++++++++ .../skills/scope-of-search/skill.md | 122 +++++++++++++++++ .../skills/selection-audit/skill.md | 101 ++++++++++++++ .../skills/unmeasured-join/skill.md | 126 +++++++++++++++++ 7 files changed, 822 insertions(+) create mode 100644 domains/coding/skills/distinguishing-observation/skill.md create mode 100644 domains/coding/skills/observability-gap/skill.md create mode 100644 domains/pr-workflow/skills/coverage-partition/skill.md create mode 100644 domains/pr-workflow/skills/instrument-check/skill.md create mode 100644 domains/pr-workflow/skills/scope-of-search/skill.md create mode 100644 domains/pr-workflow/skills/selection-audit/skill.md create mode 100644 domains/pr-workflow/skills/unmeasured-join/skill.md diff --git a/domains/coding/skills/distinguishing-observation/skill.md b/domains/coding/skills/distinguishing-observation/skill.md new file mode 100644 index 00000000..fd11ab7b --- /dev/null +++ b/domains/coding/skills/distinguishing-observation/skill.md @@ -0,0 +1,106 @@ +--- +name: distinguishing-observation +description: Enumerate every mechanism that could produce a symptom, then design the observation that separates them — instead of instrumenting the one mechanism you already suspect. An observation your favourite hypothesis predicts, and the alternatives predict too, costs a debugging cycle and buys nothing. Use when a bug has more than one plausible cause, when you are about to add a log line to confirm a suspicion, when a fix landed and the symptom did not move, when every result so far "is consistent with" the theory you started with, or when the symptom looks impossible given your model of the system. Ranks observations by how much they split the candidate set rather than by how easy they are to collect, requires a per-candidate prediction written before looking, and records survivors as not-yet-distinguished rather than ruled out. +--- + +# /distinguishing-observation + +Given a symptom, the instinct is to instrument the mechanism you already suspect. That produces +evidence consistent with your hypothesis — and equally consistent with three others you never +wrote down. + +This is the diagnostic mirror of hypothesis-first validation. There, you fix the hypothesis +before seeing what it will be compared against. Here, you fix the *candidate set* before +choosing what to measure, because the value of an observation is a property of the whole set and +cannot be judged against one member of it. + +## The information is in the split + +A confirming observation feels like progress and usually is not. If four mechanisms could produce +this symptom and your log line fires under all four, you have learned that the code ran. You +already knew that; the symptom told you. + +The observation worth making is the one whose outcome you cannot predict, because the candidates +disagree about it. That is the only kind that costs a cycle and returns a cycle's worth of +information. Debugging that never converges is almost always a sequence of observations each of +which was compatible with everything. + +## The discipline + +1. **List the candidates before instrumenting.** Three to six mechanisms that could produce this + symptom. A list of one is not a list, it is a conclusion — and you will spend the next hour + collecting support for it. + +2. **For each pair, write what differs.** Not what you believe about each; what the world would + look like differently. If two candidates predict identical observations *everywhere*, they are + not distinguishable by observation at all, and you need either a different pair or a different + axis — often a level lower, where the two mechanisms stop coinciding. + +3. **Rank observations by how much they split the field**, not by how easy they are to collect. + The best observation halves the candidate set. The worst confirms the favourite. Cheapness is + worth something, but a cheap observation with no discriminating power is not cheap, it is free + and worthless. + +4. **Predict before you look.** Write down what each candidate predicts for the observation you + are about to make, then make it. Doing this after the fact is how every result becomes + consistent with the hypothesis you started with — the prediction is elastic until it is + written down, and reading the output first sets it. + +5. **A candidate that survives is not eliminated.** Say "not distinguished by this observation", + never "ruled out". The observation constrained what it constrained. This wording is not + pedantry: when the bug comes back in three weeks, a list of things "ruled out" is a list you + will not revisit, and the real mechanism is usually on it. + +## Pairs that look identical from outside + +These shapes recur, and knowing them saves the cycle you would spend rediscovering that your +evidence does not separate them. In each case the fix is to add the separating signal *before* +continuing — which is routinely faster than more reading. + +| indistinguishable pair | why the evidence coincides | what separates them | +|---|---|---| +| an error swallowed by a `catch` vs. a code path never reached | both produce no output, no error, and no trace | count entries to the `try`, not exits from the `catch`: entered-and-never-completed is the first, never-entered is the second | +| a cache hit vs. a correct recomputation | the returned value is the same value | poison the entry with a marker only a hit could return, or count invocations of the compute function | +| a retry that succeeded vs. a call that never failed | both end in one success log | log the attempt number, not the outcome — success on attempt 1 and success on attempt 3 are different worlds | +| a timing-dependent bug vs. a state-dependent bug | both reproduce "sometimes" | hold one axis fixed: a fresh process per run under varying load isolates timing; repeated runs in one process isolate accumulated state | +| the wrong value vs. the right value from the wrong source | the assertion fails the same way | print provenance alongside the value — which module, which config, which build | +| a change that had no effect vs. a change that never shipped | the symptom is unmoved either way | verify delivery first (hash, timestamp, a deliberate marker in the artifact); an undelivered treatment reads exactly like a null result | + +The last row generalises: **before concluding that a mechanism does not matter, prove the +mechanism was present.** Otherwise "no effect" and "not applied" are the same measurement. + +## The anti-pattern: the observation that always fires + +The tell is a log line you added, that printed, and that made you feel confirmed. Ask what would +have had to appear instead for you to abandon the hypothesis. If the answer is "nothing" — if +every candidate on your list predicts this exact output — the observation had no capacity to +discriminate and the confidence it produced is manufactured. + +This is why step 4 is ordered where it is. A prediction table written first makes an +always-fires observation obvious before you spend the cycle: the column is identical all the way +down, and you go find a different one. + +## When the candidate set is empty + +Sometimes you enumerate and get nothing: the symptom is impossible given your model of the +system. That is not a dead end, it is the most informative result available, because it means the +model is wrong and you now know it. + +Switch the question from "which mechanism did this" to **"what would have to be true for this to +happen at all"**, and enumerate *those*. The answers are usually assumptions you did not know you +were making — the built artifact is not the source you are editing, two copies of the module are +loaded, the process you are reading logs from is not the process serving the request, the +environment differs from the one you configured. Each is checkable, and one of them is the bug. + +## Related + +- [`flaky-test-detection`](../flaky-test-detection/skill.md) — the timing-vs-state pair applied + to one domain, where "reproduces sometimes" is the starting symptom rather than a row in a table +- [`falsifiers-first`](../../../pr-workflow/skills/falsifiers-first/skill.md) — the same sealing + discipline pointed at a change instead of a symptom: fix the hypotheses before seeing what they + will be compared against +- [`silent-failure`](../../../pr-workflow/skills/silent-failure/skill.md) — supplies the first + row of the table as a subject in its own right, and asks whether a mechanism announces its own + failure at all +- [`evidence`](../../../pr-workflow/skills/evidence/skill.md) — the runners that collect the + chosen observation and attach the prediction made before it diff --git a/domains/coding/skills/observability-gap/skill.md b/domains/coding/skills/observability-gap/skill.md new file mode 100644 index 00000000..35e5d1b1 --- /dev/null +++ b/domains/coding/skills/observability-gap/skill.md @@ -0,0 +1,114 @@ +--- +name: observability-gap +description: Before debugging a path, establish what signal already exists on it — logs, metrics, error reporting, test coverage, user-visible state — and treat the blanks in that inventory as the first finding. A bug you cannot see is a bug you will fix by guessing, so once reading has stopped narrowing the search, the productive move is to install signal rather than read further. Separates absent signal from suppressed signal — filtered by level, sample rate, a feature flag, or an error-swallowing wrapper — because they are different problems with different fixes, and the suppressed one is both more common and more expensive. Use when a bug reproduces but its cause is invisible, when a report arrives with no trace attached, when reading code has stopped eliminating candidates, or when instrumentation appears to exist and the environment where the bug happens is emitting none of it. +--- + +# /observability-gap + +The first question about a bug is not "where is it". It is **what would have told me**. + +A path you cannot see is a path you will fix by guessing, and a guess that happens to make the +symptom go away is indistinguishable from a fix until it comes back. The opening move on an +unobservable path is usually to make it observable — not because instrumentation is virtuous, +but because every subsequent step is cheaper once the path reports on itself. + +## Inventory the signal before the code + +For the path under investigation, write the list before reading further: + +| signal | what to check | a blank here means | +|---|---|---| +| logs | is anything written on this path, at what level | the path runs and leaves no trace | +| metrics / traces | is the operation counted, timed, spanned | you cannot tell how often, or whether it is getting worse | +| error reporting | does a failure here reach Sentry or equivalent | failures are counted by users, not by you | +| tests | does anything execute this path at all | you cannot reproduce without the full system | +| user-visible state | does the UI or the API response differ when this goes wrong | the only detector is a human noticing | + +The list is not the deliverable. **The blanks are the finding**, and a path with five blanks is +not a hard bug, it is an unobservable one — a different problem with a different first move. + +## Absent is not suppressed + +No log line, and a log line nobody sees, look identical from where you are sitting. They are +not the same problem: + +- **Absent** — the code never emits. The fix is to write the emission, and it lands in the diff. +- **Suppressed** — the code emits and something eats it: a level filter, a sample rate, a + feature flag or env gate, a transport pointed at a sink nobody reads, or a `catch` that + consumes the error before anything can report it. The fix is usually a config change, often + one line, sometimes in a repo you do not own. + +Suppressed is the more common case and by far the more frustrating, because the codebase reads +as instrumented. Grep found the log line. The line is there. It is just not reaching you, and +every minute spent explaining why the code "should" be logging is spent on the wrong question. +Establish which of the two you have before proposing anything. + +## Read until it stops narrowing, then install + +Reading has a point of diminishing returns and it is easy to blow past, because reading feels +like progress in a way that writing a log line does not. + +The tell is mechanical: **two consecutive passes over the same files that eliminate no +candidate**. At that point more reading is not going to produce the answer, and the cheapest +remaining move is to add signal and run it again. One log line at the right boundary routinely +settles a question that an hour of reading left open, because it reports what actually +happened rather than what the code permits to happen. + +## Instrument the boundary, not the suspect + +Put signal at the **edges of the subsystem**, not on the line you suspect. + +A boundary tells you whether the problem is inside or outside, which halves the search +regardless of whether your hypothesis was right. Signal on your favourite line tells you about +that line only, and only in the case where you had already guessed correctly — which is the +case where you needed the least help. Instrument in and out first; narrow after the halving. + +## The gaps worth naming + +| class | why it costs you | +|---|---| +| a failure path with no error reporting | the failure is real and the count is zero | +| an async boundary that loses context | the error surfaces detached from its cause, pointing at the awaiting frame instead of the failing one | +| a conditional whose branch is not recorded | you cannot tell which way it went, so both explanations survive | +| state mutated with no trace of the mutator | you can see the wrong value and not who wrote it | +| a third-party call whose failure mode is a default return | a degraded dependency is indistinguishable from an empty result | +| instrumentation that is off in the environment with the bug | the signal appears to exist | + +The last one is the most expensive in this table, and the reason is in the phrasing: the others +announce themselves as gaps once you look, and this one does not. You find the log line, you +assume the path is covered, and you spend the afternoon reasoning about why the covered path +produced no output. + +### The environment check + +Confirm the signal is on **in the environment where the bug happens**, not in the one where you +are reading the code. A metric emitted only in production and a log emitted only in development +are both silence exactly where you need them — and each looks like working instrumentation from +the other side. + +Concretely, for each signal you are counting on: which env vars, flags, log levels, sample +rates and build modes gate it, and what are their values *on the machine that reproduced the +bug*. If you cannot answer that, you do not know that the signal exists there; you know it +exists in the source. + +## Keeping what you added + +A signal you add to find a bug is a signal the next person needs. Decide deliberately before +removing it, and default to keeping it: the path was hard to debug **because** it was +unobservable, and reverting the instrumentation restores precisely that condition for whoever +arrives next. + +Reasons to remove are real but specific — a per-iteration log in a hot loop, output containing +user data, a metric whose cardinality is unbounded. "It was only for debugging" is not one of +them. If the volume is the problem, lower the level or gate it behind a sample rate rather than +deleting it, so the next person can turn it back on instead of rediscovering the gap. + +## Related + +- [`silent-failure`](../../../pr-workflow/skills/silent-failure/skill.md) — the review-facing + sibling. Same property, opposite end: it asks whether a mechanism would announce its own + failure, this one starts from a failure that already happened and nobody saw +- [`falsifiers-first`](../../../pr-workflow/skills/falsifiers-first/skill.md) — once the path + reports on itself, hypotheses about it become testable rather than arguable +- [`flaky-test-detection`](../flaky-test-detection/skill.md) — the same gap inside a suite, + where the missing signal is what the test observed on the run that failed diff --git a/domains/pr-workflow/skills/coverage-partition/skill.md b/domains/pr-workflow/skills/coverage-partition/skill.md new file mode 100644 index 00000000..15e53fa3 --- /dev/null +++ b/domains/pr-workflow/skills/coverage-partition/skill.md @@ -0,0 +1,128 @@ +--- +name: coverage-partition +description: Measure which cases in a suite guard which mechanism, by defeating each mechanism in turn and recording the exact set of cases that go red. Reports the partition rather than the total, because "the suite has power" is a boolean while a suite's power is a distribution — and a suite credited with covering five behaviours routinely has one case standing between a mechanism and silence. Names why each survivor survived, since testing something else, being shielded by an upstream step, and being genuinely unaffected are three different facts that a count collapses into one. Use when a suite is offered as evidence for a specific claim, when one description credits one test set with covering several mechanisms, when deciding whether a green suite can be trusted to guard a security check, or when a mutation run reported a number and stopped there. Costs one full suite run per mechanism, two arms each. +--- + +# /coverage-partition + +A mutation run that reports "4 of 7 tests failed" has answered a question nobody asked. **Which** +four is the reviewable fact, and it costs the same probe to find out. + +"The test has power" is a boolean. A suite's power is a distribution, and the distribution is +almost never the one the author's sentence implies. + +## Why a total is the wrong number + +Totals compose badly. Seven cases that each defeat one mechanism and seven cases that all defeat +the same mechanism produce the same count and describe opposite suites. The count also hides its +own shape: a mechanism guarded by exactly one case looks identical, in the total, to a mechanism +guarded by four. + +And a total cannot be checked against a claim. "This suite covers signing and verification" is an +assertion about *which* mechanisms the cases reach — it is refuted or supported by the partition +and is untouched by the number. + +## The probe + +For each mechanism the suite is credited with guarding: + +1. **Defeat it minimally**, at its source, without breaking parsing — invert the condition, widen + the pattern, replace the verification call with a constant that succeeds. +2. **Run the whole suite** and record the exact set of cases that go red, by name. +3. **Name why each survivor survived.** This is the step that turns a count into a map, and it is + the step that gets skipped. + +Then report the sets, one row per case, one column per mechanism. + +## A worked partition + +A seven-case suite, described by its author as covering "real ECDSA signing and verification +across valid signed, unsigned, tampered, malformed, and invalid-signature cases". Three +mechanisms, three probes: + +| case | strip condition weakened | verification stubbed to succeed | value-format pattern widened | +|---|---|---|---| +| positive forward | — | — | — | +| legacy-signature | **fail** | — | — | +| missing-signature | **fail** | — | — | +| tampered | **fail** | **fail** | — | +| invalid-signature | **fail** | **fail** | — | +| unlisted-parameter | — | — | — | +| malformed-value | — | — | **fail** | +| | **4 of 7** | **2 of 7** | **1 of 7** | + +Five cases pass with signature verification entirely disabled. + +### Why each survivor survived + +The three cases that survive the strip mutation survive for three unrelated reasons, and the +distinction is the finding: + +- **The positive case is supposed to forward.** It tests the other side of the branch. Not a gap. +- **The unlisted-parameter case never reaches the strip** — an earlier canonicalization step + already dropped that parameter, so the mutated condition does not run on it. This one is + shielded, and it would keep passing no matter how badly the strip broke. +- **The malformed-value case is rejected by the format check first**, regardless of the strip. It + is genuinely unaffected, and it is load-bearing for a different mechanism. + +Three survivors, three facts. A count says "3 passed" and loses all of them. Shielded cases are +the ones worth naming out loud, because they read as coverage in a case list and provide none. + +### What the partition said that the total could not + +The suite's power is real and it is distributed — but most of it sits on the parameter strip, and +**exactly two cases would notice if signature verification stopped working entirely**. The +author's sentence reads as though all five case classes exercise verification. Five of them do +not. + +The format check has one guarding case, and that case asserts three keys at once. + +## Reading a partition + +**Name why each survivor survives.** A case that passes under mutation is testing something else, +or shielded by a step upstream, or genuinely unaffected. Those are different facts and only the +second one is a problem — but you cannot tell which you have without looking. + +**A mechanism with one guarding case is a finding, even when that case passes.** One case is one +refactor, one skip, one flaky quarantine away from zero, and nothing in a green run announces the +drop from one to none. Report it as a finding, not as coverage. + +**A case that asserts several things at once counts as thin.** When it goes red you cannot tell +which assertion fired, so it cannot serve as the guard for any one of them. Its column entry +should be read as "something in here broke", which is a weaker fact than it looks. + +**Overlap matters as much as coverage.** Cases that all fail under the same mutation are +redundant with each other under that mutation, however different their names and fixtures are. +Four cases failing on the strip is one guard with four expressions of it, and it will survive +deleting three of them. + +## When to reach for it + +When someone credits a suite as evidence for a claim — a PR description, a review reply, a +security sign-off. **"It has power" answers whether the suite is decorative. The partition answers +whether it has power over the mechanism named in the claim**, which is a different question and +usually an unasked one. + +Reach for it also when a mutation run has already produced a number, because the expensive part is +already paid for and the partition is what that run was capable of reporting all along. + +## Cost + +One full suite run per mechanism, two arms each — mutated and clean, since a case already red on +the clean arm is not evidence about anything. Three mechanisms is six suite runs. That is the +honest price, and it scales with mechanisms rather than with cases, so a large suite over three +mechanisms costs the same number of runs as a small one. + +Scope by mechanism, and pick them before running: the mechanisms the claim names, plus any +mechanism whose failure would be silent. + +## Related + +- [`falsifiers-first`](../falsifiers-first/skill.md) — supplies the defeats; this skill changes + what gets recorded when they run +- [`silent-failure`](../silent-failure/skill.md) — a mechanism with zero guarding cases is a + silent path by construction, and the partition is how the zero gets found +- [`evidence`](../evidence/skill.md) — the runners that execute the arms and capture the per-case + results the partition is built from +- [`unintended-breakage`](../unintended-breakage/skill.md) — reads the same per-case results in + the other direction, asking which cases went red that nobody meant to touch diff --git a/domains/pr-workflow/skills/instrument-check/skill.md b/domains/pr-workflow/skills/instrument-check/skill.md new file mode 100644 index 00000000..de617673 --- /dev/null +++ b/domains/pr-workflow/skills/instrument-check/skill.md @@ -0,0 +1,125 @@ +--- +name: instrument-check +description: Prove the instrument works before its output counts as evidence — plant a defect it should catch (positive control), and run it where nothing is wrong (negative control). A runner broken in a plausible direction emits well-formed numbers, and well-formed numbers get published. Encodes the standing rule that an instrument reports the effect it had, never the instruction it was given: the change that landed is read back off disk and published beside the change that was requested, so the two are capable of disagreeing. Use before trusting a mutation runner, a counter, a grep- or diff-based check, an A/B harness, or any probe whose green result would be indistinguishable from having measured nothing. +--- + +# /instrument-check + +A measurement is not evidence. A measurement from an instrument that has been shown to work is +evidence. The gap between those two sentences is where published wrong answers come from. + +The failure is not that instruments break loudly. A broken instrument that throws gets fixed in +the same minute. The dangerous one is **broken in a plausible direction**: it still runs, still +emits a number of the right shape, still satisfies every guard around it, and the number is +wrong. Nothing downstream can tell, because nothing downstream ever sees the instrument — only +its output. + +## What this looks like when it happens + +A mutation runner applied its one-line replacement through `awk -v r="$REPLACE"`. `awk` +escape-processes a `-v` assignment before the program ever sees it, so a replacement of +`/^[\s\S]{1,4096}$/u` was written to the file as `/^[sS]{1,4096}$/u`. The `\s` and `\S` were +eaten. A pattern intended to be **widened** to accept anything was **narrowed** to accept a +string of `s` and `S` characters. + +Then read what the run reported, and how each layer of protection cooperated: + +- The suite went red — but on a **different test** than the mutation targeted, because the + probe had landed on a mechanism nobody was studying. +- Arm B ran the **same test count** as arm A, so the arms-differ guard was satisfied. +- The file changed on disk, so the mutation-applied guard was satisfied. +- The runner emitted `falsifying` for a mechanism it had never touched. +- The artifact's `mutation applied` field echoed the **requested** text. The requested and the + applied could not disagree, because they were the same string printed twice. + +Every guard was a real check, honestly implemented. Collectively they described a run that never +happened. Nothing in the system was in a position to notice, and the defect had been there since +the first commit; it was found weeks later, by accident. + +A second one from the same session, cheaper and just as total: a probe file was copied into a +target tree at a path where none of its imports resolved. The suite failed to load. The run +reported **success** and produced an artifact that measured nothing at all. + +## Two controls, both required + +Before a measurement counts, the instrument passes both. One without the other is half a check, +and the halves catch opposite defects. + +**Positive control — plant a defect the instrument should detect.** Break something on purpose, +in a way you are certain about, and confirm the instrument fires. If it stays quiet, it is not +measuring what you think it is measuring, and every clean result it has ever produced is a clean +result about nothing. + +**Negative control — run it where nothing is wrong.** An empty range, an untouched tree, a +fixture with a known-good answer. If it fires anyway, every positive result is suspect, because +you now know the instrument can produce a finding without a cause. + +The positive control catches an instrument that is deaf. The negative catches one that is +hallucinating. The awk defect was, precisely, an instrument that was deaf and hallucinating at +once — it did not do what it was asked, and it reported a finding regardless. + +## The standing rule + +> An instrument reports the effect it had, never the instruction it was given. + +Wherever a runner takes an input and performs an action, the artifact carries **what happened**, +not what was asked. The mutation artifact publishes the line read back off disk after the write, +alongside the line that was requested. Two fields, from two sources, and the entire value is +that they are **capable of disagreeing**. A field that echoes its own input is decoration; it +can only ever confirm that the program remembers what it was told. + +The second fix from the same session generalises the same way: the caller now states **which +test names should fail**. The instrument no longer gets to interpret any red as its red. A probe +that fires somewhere unintended becomes its own outcome — reported as a probe that missed — +rather than silently passing as a falsification. An instrument that cannot say "I hit the wrong +thing" will never say it. + +Both fixes are one move: give the instrument a way to contradict its own operator. + +## What a control looks like, by instrument + +| instrument | positive control | negative control | +|---|---|---| +| **mutation runner** | mutate something with an obvious, named guard — an assertion with a dedicated test. Does *that* test fail? If not, the write is not reaching the file the suite loads | apply an identity mutation, or none. Suite must be green and the arms identical | +| **counter** | count a fixture whose answer you counted by hand. Off-by-one and double-count both show here and nowhere else | count an empty input. A counter that returns anything but zero is measuring its own scaffolding | +| **grep-based check** | search for a string you know is present. A pattern that cannot match *anything* returns clean, which is character-for-character the output of a clean tree | search for a string you know is absent. A pattern that matches everything reads as a wall of findings nobody triages | +| **diff-based check** | run it across a commit you know introduces the thing. Silence means the range, the filter, or the parser is wrong | run it on an empty range — a commit against itself. Any finding at all is manufactured | +| **A/B harness** | make the arms differ in a way that must move the metric. If the arms report the same number, the treatment is not reaching the built artifact | run A against A. A non-zero delta is the harness's noise floor, and it bounds every result you will publish | + +Where an instrument copies files, executes in another tree, or shells out, add one more: **prove +the target actually ran.** Load failures, missing imports, and empty test selections all exit +in ways that look like a fast, clean pass. Assert a non-zero test count from the run's own +output, not from the count you intended to run. + +## The failure mode this skill exists to prevent + +**Controls that were run once, at build time, and never again.** + +This is the one that gets everybody, because it feels like diligence. The instrument was +validated. There is a commit that proves it. That commit validated a *different instrument* — +one composed of the shell quoting, file paths, environment, working directory, and dependency +versions of that day. + +A control is standing procedure, not a milestone. Run it in the same invocation that produces +the measurement, through the same code path, with the same plumbing, so its result and the +measurement's result share a fate. Cheap enough to run every time is a design requirement of the +control, not a nice property; a control too expensive to run per-measurement will quietly become +a control that ran once. + +The awk defect passed review. It shipped. It ran for weeks producing artifacts that read as +rigorous. The number of runs it corrupted is unknown, and the honest report of that period is +not "the findings were wrong" but **"the findings were not measurements"** — which is worse, +because it cannot be corrected, only discarded and redone. + +## Related + +- [`falsifiers-first`](../falsifiers-first/skill.md) — supplies the probes this skill validates; + its predictions ("which test should fail, by name") are exactly the field that lets a probe + report having hit the wrong thing +- [`silent-failure`](../silent-failure/skill.md) — the same question aimed at the system under + test rather than at the tooling; its last three shapes are instrument defects, and this is + where they get run down +- [`unintended-breakage`](../unintended-breakage/skill.md) — depends on this one, since a scan + that cannot find breakage and a change that broke nothing produce identical output +- [`evidence`](../evidence/skill.md) — the runners that carry the requested-vs-applied fields, + and the gate that should refuse an artifact with no control attached diff --git a/domains/pr-workflow/skills/scope-of-search/skill.md b/domains/pr-workflow/skills/scope-of-search/skill.md new file mode 100644 index 00000000..89b961f0 --- /dev/null +++ b/domains/pr-workflow/skills/scope-of-search/skill.md @@ -0,0 +1,122 @@ +--- +name: scope-of-search +description: Treat a negative result as a fact about the search rather than about the thing searched — "no occurrence" means this pattern did not match, which is a much smaller claim than the one it gets read as. Publish the pattern alongside the negative, name what the pattern structurally cannot see, search for the concept before the identifier, and run a positive control before believing an absence. Use when a diff audit, grep or validation run is about to report that something is not present, when checking a PR description's claim that a category of code is absent, or when a 404, an empty result set or a quiet run is about to be published as evidence of non-existence. +--- + +# /scope-of-search + +A negative result is a fact about the search, not about the thing searched. + +"No occurrence" means: *this pattern, run over this text, matched nothing.* That is a much +smaller claim than "the thing is not there", and the gap between the two is where negatives go +wrong. + +## The tell + +You are about to write **no**, **none**, **nothing** or **never** about a body of code. + +Whatever produced that word has a scope, and the scope belongs in the sentence. + +## Why negatives need their own discipline + +The consequences are asymmetric. A false positive gets investigated and dies — someone opens +the file, sees it does not say what the finding said, and the finding is gone inside a minute. +A false negative closes the question. Nobody returns to it, because there is nothing to return +to: the record says it was checked. + +So a negative should cost more to publish than a positive. In practice it costs less, because a +negative is what a run produces when it does nothing. + +## What this looks like when it fails + +A PR body claimed the diff contained "no original/unsigned handler parameter access". A +validation run tested that claim with + +``` +grep -nE "sig_params|SIG_PARAMS_PARAM|searchParams[.]get|url[.]searchParams" +``` + +over the added lines, and reported `(no occurrence)`. + +The pattern keys on the variable name `searchParams`. The claim was about the *access* — a query +parameter read before signature verification. Naming and behaviour are different things, and the +pattern only knew one of them. Two blocks later the same artifact printed five +`params.get('utm_source')`-style reads it had just declared absent. The clearest instance of all +was in a file the pattern never covered: a router handing raw URL query straight into a provider. + +The forms that pattern could not see are the general shape of the problem: + +- a `URLSearchParams` constructed under any other variable name — `params`, `q`, `qs`, a + destructured field +- `.getAll(` rather than `.get(` +- destructuring, where no accessor call appears at all +- a value assembled outside the diff and passed in, so the access is real and the added lines + only receive it + +Same session, opposite direction: a link was reported broken on the strength of an anonymous +fetch returning 404. The search was scoped to anonymous access; the claim was about existence. +A private, moved, or auth-gated resource returns exactly the same 404 as one that was never +there. + +Both failures have one shape. **The scope of the search was narrower than the scope of the +claim, and the report used the claim's words.** + +## The rules + +**Publish the pattern with the negative.** A reader cannot assess "no occurrence" without seeing +what was searched for. The pattern is part of the finding, not an implementation detail of how +it was reached. If showing it feels like clutter, that is the signal it is load-bearing — a +negative whose pattern is not worth publishing is a negative not worth believing. + +**State what the search structurally cannot see.** Every pattern has a shape it misses: an +identifier-keyed pattern misses renames, a line-oriented one misses anything spanning lines, an +added-lines scan misses everything the diff merely calls. Name the blind spot yourself, in the +finding. Otherwise someone names it later, in review, at much higher cost — and by then the +negative has been relied on. + +**Search for the concept, then for the name.** A pattern keyed on an identifier finds one +spelling of an idea. If the claim is about behaviour — "nothing here reads unverified input" — +then at least one search has to be about behaviour: the sinks, the call shapes, the boundary the +data crosses. Identifier searches then narrow what the behavioural search surfaced. In that +order, because the reverse only ever confirms the spelling you already guessed. + +**A negative wants a positive control.** Run the same pattern against somewhere you know the +thing exists — another file, an earlier revision, a line written to bait it. If it does not match +there, the negative elsewhere means nothing; you have measured your regex, not the code. This is +what catches the entire class where the pattern was never capable of matching: wrong escaping, +wrong flags, a path filter that excluded the file set, a revision that was never checked out. + +**Scope is part of the claim.** "Not in the added lines", "not in this file", "not in the +repository" and "not reachable from this entry point" are four different findings, routinely +reported in the same three words. Say which one you have. If the claim you are checking is wider +than the search you ran, report *that mismatch* — it is the real result, and it is more useful +than the negative. + +## Reporting a negative + +Four facts, and it need not run longer than a positive: + +- **what was searched for** — the pattern, verbatim +- **where** — file set, revision, added lines versus whole tree +- **the control** — where the pattern did match, proving it can +- **the blind spot** — the forms this could not have found + +Then the finding, scoped to exactly that: "no occurrence of `X` in the added lines of these four +files; control matched at `a/b.ts:31`; would not catch a renamed binding, or a value assembled +outside the diff and passed in." + +If one of the four is missing, the honest report is **not checked**. That is a legitimate thing +to publish, and a better one than a negative nobody can assess. + +## Related + +- [`falsifiers-first`](../falsifiers-first/skill.md) — its warning that mirroring the author's + method makes agreement uninformative is this failure in the large: their grep's blind spot + becomes yours +- [`silent-failure`](../silent-failure/skill.md) — its "a check whose pattern cannot match the + failure" row is this skill's whole subject, and it supplies the positive control as an + induction: make the thing exist, then see whether the check notices +- [`unintended-breakage`](../unintended-breakage/skill.md) — "we found no regressions" is a + negative with a scope, and the scope is usually the suite that ran +- [`evidence`](../evidence/skill.md) — where the pattern, the scope and the control get attached + to the finding rather than discarded once the run is over diff --git a/domains/pr-workflow/skills/selection-audit/skill.md b/domains/pr-workflow/skills/selection-audit/skill.md new file mode 100644 index 00000000..44d62fb4 --- /dev/null +++ b/domains/pr-workflow/skills/selection-audit/skill.md @@ -0,0 +1,101 @@ +--- +name: selection-audit +description: A count is the output of a selection rule, so two counts agree only when their rules do — report the rule that produced the number rather than the number, and never read a matching count as corroboration until both selections have been written out and compared. Catches the failure where a PR claims N affected suites and a validation run measures the same N from a narrower rule, and the coincidence closes a question nobody then asks. Use when checking a count against someone else's, when a report says "N of them" — tests, files, call sites, consumers, events — and the interesting question is which N, or when a word like affected, related, relevant or impacted is quietly doing the selecting. +--- + +# /selection-audit + +A count is the output of a selection rule. The number is the cheap part; the rule is the claim. + +Two people can produce the same number from different rules and read it as agreement. That is +worse than disagreeing, because a disagreement gets investigated and a match closes the +question. + +## The case + +A PR body claimed **18 affected Perps suites / 730 tests passed**. A validation run measured +**18 suites, 731 tests**, and read the exact suite-count match as corroboration — off by one +test, dead on for suites, so the selections must be the same set. + +They were not. They were two different rules landing on the same number: + +- **The run's rule** was every `*.test.*` file in the PR's own diff: 19 such files, of which 18 + still exist at head, because the PR deletes one. +- **The author's word was "affected"**, which selects something else entirely — a changed source + file affects suites that do not themselves change. At head, 31 test files reference the + controller package whose Jest stub this PR grows by 156 lines, and 15 reference the events + constants module. + +So "18" was reproducible, defensible, and meant a different thing in each mouth. The match was a +coincidence of two boundaries, and it carried exactly zero information about whether the run had +covered what the author claimed. Nineteen minus a deletion is not thirty-one. + +The one-test difference has its own lesson. It was explained away by a commit landing after the +pin the author had measured at — an explanation asserted from a *line* count in that commit +rather than a test count, and it stood unchallenged until someone actually counted. A plausible +mechanism offered for a numeric gap is a hypothesis about numbers, and it is checkable with the +same effort it took to say. + +## The rule + +**State the rule, not the count.** "Every changed test file that still exists at head" is a +claim someone can reproduce or dispute. "18 suites" is not — it is the residue of a claim, with +the claim removed. Write the selection into the report at the point where the number appears. + +**Matching numbers from unstated rules are not agreement.** Before treating a match as +corroboration, make both rules explicit and check they are the same rule. If you cannot state +the other party's rule, you have not confirmed anything; you have found that two unknown sets +happen to be the same size. + +**"Affected", "related", "relevant", "impacted" are selection rules in disguise** — and almost +always broader ones than whatever got run, because they reach through the change into code that +did not change. When someone else's count uses one of these words, the question is not whether +their number is right. It is what set the word names. + +**Name what the rule excludes.** A selection is defined by its boundary, and the excluded set is +where the missing coverage lives. "Every changed test file" excludes every unchanged test that +exercises the changed source — which is the entire population a reviewer cares about. Reporting +the boundary costs one sentence and is the only part of the count that can surprise anyone. + +**A count measured at a different commit is a different count.** Say where you measured. If you +are comparing two numbers, say whether they were measured at the same place, and treat "they +differ by one" as unexplained until the commits match or the difference is counted rather than +narrated. + +## The tell + +You are about to write **"which matches"**, **"as claimed"**, **"consistent with the PR body"**, +or **"confirming the author's figure"** about a number. + +Stop there. Two numbers agreeing tells you nothing until two rules agree, and the sentence you +were about to write is the one that converts a coincidence into a finding. Write the two rules +side by side first; if they turn out to differ, the match is the finding, and a more interesting +one than the confirmation would have been. + +## Beyond test counts + +The shape is general, and the phrase to watch for is "N of them": + +| the count | the rule hiding inside it | +|---|---| +| files changed | tracked? generated? vendored? renames as one or two? | +| call sites | static references, or reachable at runtime? through re-exports? | +| consumers | packages that import it, or packages that ship it to users? | +| events emitted | distinct names, or instances, and over what window? | +| failing tests | at which commit, with which shard, retries counted or collapsed? | + +Every row is a place where two competent people report different integers and neither is wrong. +Anywhere a report says "N of them", the number is an answer to a question that was never +written down — and your job is to write it down, especially when the number looks right. + +## Related + +- [`falsifiers-first`](../falsifiers-first/skill.md) — a count that agrees with the claim is the + most comfortable result available, and the least examined; this is that failure in its + numeric form +- [`coverage-partition`](../coverage-partition/skill.md) — takes the excluded set seriously as + its own object, rather than as the remainder of a selection +- [`scope-of-search`](../scope-of-search/skill.md) — the same discipline for what a search + looked at, where an empty result and an unsearched region are indistinguishable +- [`evidence`](../evidence/skill.md) — where the rule gets recorded alongside the number, so the + count remains reproducible after the run diff --git a/domains/pr-workflow/skills/unmeasured-join/skill.md b/domains/pr-workflow/skills/unmeasured-join/skill.md new file mode 100644 index 00000000..75b5b759 --- /dev/null +++ b/domains/pr-workflow/skills/unmeasured-join/skill.md @@ -0,0 +1,126 @@ +--- +name: unmeasured-join +description: Audit a finding for the step between its facts — the inference that turns two true observations into a conclusion, which is usually the only part nobody instrumented. Decomposes a finding into steps, marks each measured or asserted, and requires an instrument on the connecting step, because the facts are almost always fine and the join is where it fails. Use before publishing a verdict, when reviewing another run's report, when a finding reads as obviously right, and especially when its conclusion is correct — a right answer through an unmeasured step leaves nothing downstream to contradict it. Triggers on any finding whose statement needs "so", "therefore", "which means" or "hence", on a conclusion assembled from a grep plus an interpretation, and on set-membership claims hidden in possessives like "the recipe's coverage of X". +--- + +# /unmeasured-join + +A finding is two or more facts and a step between them. The facts get checked. The step is the +finding. + +That asymmetry is the whole failure. Every fact in a bad report can be individually true and the +report still wrong, because the error lives in the connective — and the connective is the one +part that never got an instrument pointed at it. + +## Why it survives review + +A reader who checks the facts and finds them solid stops checking. This is not laziness; it is +the correct heuristic applied to the wrong target. Verifying a fact *feels* like diligence — you +run something, you read output, you tick it off. Passing over the "so" between two verified +facts feels like reading, not like skipping a step, because grammar presents it as connective +tissue rather than as a claim. + +So the join gets the least scrutiny of anything in the finding, while being the cheapest thing in +it to check. Re-deriving the facts is the expensive part. The join is usually one grep, one +count, or one repeated request under a different identity. + +## Four that shipped + +Each of these had facts that held up. Each failed at the join. + +**A stale evidence bundle that did not matter.** Facts: the bundle pins commit A; three commits +landed after it; one of them changes close-all transport error tracking; close-all error tracking +is analytics emission; the evidence recipe measures analytics. Every one true. **Join asserted:** +that the recipe's analytics coverage *includes close-all*. Measured afterwards: +`grep -icE 'close[-_ ]all'` returns 0 across the recipe, the report and the trace — zero of 201 +nodes would differ. The verdict was withdrawn. Note where the claim hid: "the recipe's analytics +coverage" is a possessive, and it is carrying a set-membership assertion that a grep settles in +seconds. + +**730 versus 731.** A run explained the one-test difference with "commit X adds 40 lines to that +test file". The conclusion was right — the commit does add one test — and a line count cannot +yield a test count. Forty lines is two `it()` blocks and a deleted one, or a fixture and no +tests at all. The commit happens to add exactly one `it()`, which nobody counted until an +adversarial pass did. The instrument was `grep -c 'it('` on one file. + +**Signing without verifying.** A run credited a PR's claim of "real ECDSA signing and +verification" on a grep whose output contained `webcrypto`, `subtle.sign` and +`subtle.generateKey`. The pattern also included `subtle.importKey` and `subtle.verify`; both +matched nothing. **Join asserted:** signing present therefore verification exercised. A suite +that signs with a real key and never verifies produces exactly that output. The instrument had +already run here — the disconfirming half of its own pattern was in the file and went unread, +because a compound pattern reports the union and the reader takes the union as the answer. + +**A link that was not broken.** A run reported a documentation link as broken because it returned +404 to an anonymous fetch. It returns 404 to an authenticated token too. That is what an +org-internal repository looks like from outside the org, not what a broken link looks like. The +fetch measured reachability *from this identity*; the finding needed existence, and 404 does not +distinguish the two. One repeat request under different credentials separates them. + +## The rule + +Decompose the finding into its steps. Mark each one: + +- **Measured** — an instrument ran, and you read its output *including the part that did not + match*. A pattern with five alternatives produces five results, not one. +- **Asserted** — you supplied it. Domain knowledge, a reasonable reading, a thing that is + obviously true. All of those are assertions. + +Then: **the connecting step must be measured.** Not the facts — the step that turns them into a +conclusion. Facts arrive already instrumented, which is why they are facts; the join is the part +you wrote. + +An instrument on a join is small by construction, because a join is a small claim. Membership: +grep for the term in the artifact. Quantity: count the thing you are claiming changed, not a +proxy for it. Identity or environment: repeat the observation with the variable changed. If you +cannot name the one command, the step is not yet stated precisely enough to be a step. + +### The tell + +If a finding needs the word **"so"**, **"therefore"**, **"which means"** or **"hence"**, that is +where the instrument goes. Those words mark the seam where facts become a conclusion, and the +seam is the failure surface. + +They are the easy case. Joins also hide where there is no connective at all — in a possessive +("the recipe's coverage"), in an apposition, in a noun phrase that quietly names a category +membership ("close-all error tracking is analytics emission" is a fact; "the analytics the recipe +measures" is a set, and putting the two beside each other is an argument). When a sentence has no +"therefore" but the conclusion still moved, something joined silently. Find it. + +## When the conclusion is right + +This is the hardest case, and it is worth being honest about why: nothing downstream contradicts +it. The test count really was 731. No later step trips, no reviewer notices, no CI job disagrees. +The reasoning is defective and the artifact is clean. + +Which means it can only be caught by **auditing the reasoning rather than the result** — and that +audit has to be scheduled, because nothing about a correct answer prompts one. If the 730/731 +commit had added forty lines of fixtures and two `it()` blocks, the identical sentence produces +the identical confidence and the answer is wrong. The method was already broken when it was +right; the outcome just did not report it. + +Treat a correct conclusion reached through an asserted join as a defect of the same class as a +wrong one, and say so in the writeup. A run that only records where it was wrong has no way to +learn that it was right by accident. + +## What this is not + +**Not a demand that everything be measured.** Assertions are how a finding gets written at all. +The rule is narrow and it is positional: the *connecting* step, the one doing the inferential +work, is the one that must be measured. Marking the facts asserted where they plainly hold is +fine and costs nothing. + +**Not fact-checking.** A run that re-verifies every fact and never names the join has done the +expensive half of the work and skipped the half that fails. The facts being solid is the +precondition for this skill applying, not evidence against it. + +## Related + +- [`falsifiers-first`](../falsifiers-first/skill.md) — fixes hypotheses before seeing the claims; + this skill audits the inference inside a hypothesis once it has produced a finding +- [`silent-failure`](../silent-failure/skill.md) — its "a check whose pattern cannot match the + failure" row is the ECDSA case seen from the instrument's side +- [`evidence`](../evidence/skill.md) — the runners that execute a join's instrument, and the gate + a finding passes before it is published +- [`unintended-breakage`](../unintended-breakage/skill.md) — findings about a change's blast + radius are assembled from exactly this shape, and the radius is the asserted step From 46372afee7ef855474a93b9554fc9328a0e95b78 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 06:05:55 -0400 Subject: [PATCH 106/135] Point `falsifying-test` at the harness, and separate it from its sibling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 6 asked for both runs shown and left the operator to produce them, which is the shape that yields a retyped paste indistinguishable from output nobody ran. The run workflow already takes `ref` and `baseline`, executes the same command at both commits, and attaches the artifacts to a URL a reader can open — so the step names it. The larger risk was confusion with `falsify-probe.sh`, which shares the two-arm shape and answers a different question: base-against-branch asks whether a test is connected to the reported bug, one-commit-with-a-mutated-line asks whether it notices the mechanism going away. A test can pass either and fail the other. The skill now states the distinction in a table rather than leaving two similarly named things to be conflated, and notes that the runner's guards mechanise this skill's own falsifier — a red arm that ran fewer tests, or failed to load, is refused rather than counted. --- .../testing/skills/falsifying-test/skill.md | 43 ++++++++++++++++--- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/domains/testing/skills/falsifying-test/skill.md b/domains/testing/skills/falsifying-test/skill.md index 7cfeb160..65f36e42 100644 --- a/domains/testing/skills/falsifying-test/skill.md +++ b/domains/testing/skills/falsifying-test/skill.md @@ -44,10 +44,16 @@ this skill lives. file or suite, scope the run to the new test (by name/path) so the red is attributable. A suite that was already red proves nothing about your assertion. -6. **Show both runs.** Base: the assertion failure, verbatim. Branch: the pass. Same command, - same filter, both commits identified. Captured terminal output beats a transcription — - retyped output is a self-report, and a real capture has caught errors that careful prose - missed. +6. **Show both runs, and let a tool write them down.** Base: the assertion failure, verbatim. + Branch: the pass. Same command, same filter, both commits identified. Retyped output is a + self-report — indistinguishable from output that was never produced — so the two arms want + to come out of a runner rather than a paste buffer. + + `evidence` ships the mechanism: its run workflow takes `ref` and `baseline`, checks out both + commits, executes the same command at each, and attaches the artifacts to a run URL a reader + can open without going through you. Wrapping the test command in `capture.sh` gets the same + property locally, minus the reader-verifiable half — and that runner's footer says so, in + the artifact, rather than leaving the gap for a reviewer to notice. 7. **Pair it with the issue.** The PR's `Fixes #N` plus a test named for the behaviour makes the causal chain checkable by a reader who runs nothing. @@ -78,10 +84,33 @@ Falsifying test — <test name> (Fixes #N) scoped: <how the run was limited to this test> ``` +## The sibling experiment, and why it is not this one + +`evidence` also ships `falsify-probe.sh`, which has the same two-arm shape and answers a +different question. The distinction is worth holding, because conflating them produces a proof +of the wrong thing: + +| | arms | question | +|---|---|---| +| **this skill** | base commit, branch commit — same test | is the test causally connected to the reported bug? | +| **`falsify-probe.sh`** | one commit, one line mutated | does the test fail when the mechanism it guards is removed? | + +A test can pass this skill and fail that one: it fails on base because the fix was not there, +and passes under mutation because it asserts something adjacent to the mechanism. The reverse +also happens. On a bug-fix PR you usually want both — the first proves the test is about *this +bug*, the second proves it will keep noticing. + +What the runner does mechanise is step 2. Its guards refuse to call a red arm a falsification +when the suite ran fewer tests than the baseline, or failed to load at all — which is this +skill's falsifier, enforced rather than remembered. It also takes the names of the tests you +expect to fail, so a red run in the wrong place is reported as such instead of passing as a +falsification. + ## Related -- `evidence` — packages this skill's output as its [falsifying regression test category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md). - The deterministic-interleaving category is the sibling for concurrency and temporal-ordering - bugs; `race-condition-repro` drives it. +- `evidence` — packages this skill's output as its [falsifying regression test category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md), + and supplies the two-commit run harness step 6 asks for. The deterministic-interleaving + category is the sibling for concurrency and temporal-ordering bugs; `race-condition-repro` + drives it. - `react-render-delta` — the same before/after discipline applied to a measured quantity rather than a boolean. From 1e7ef54804ab65a1680f4ec258daa62aa4f69c2d Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 06:09:26 -0400 Subject: [PATCH 107/135] Rename `falsifying-test` to `red-on-base` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things in this PR shared a root and a two-arm shape while answering different questions: the skill compares a base commit against a branch to show a test is connected to the reported bug, and `falsify-probe.sh` mutates one line at one commit to show a suite notices the mechanism going away. A test can satisfy either and fail the other, so the names had to stop rhyming. `red-on-base` names the skill's own discipline — the test must be red on the base commit — and shares no root with the runner. The evidence category keeps its name: a falsifying regression test is the artifact, and this is the procedure that produces one. The runner's stderr described its own result as "a falsifying test", which was the same collision inside the output of the thing causing it; it now says what it proves and points at the other experiment by name. --- .../skills/evidence/references/claim-extraction.md | 2 +- .../skills/evidence/references/evidence-catalog.md | 2 +- .../skills/evidence/references/output-templates.md | 2 +- .../pr-workflow/skills/evidence/scripts/falsify-probe.sh | 4 +++- domains/pr-workflow/skills/evidence/skill.md | 2 +- .../skills/{falsifying-test => red-on-base}/skill.md | 6 +++--- 6 files changed, 10 insertions(+), 8 deletions(-) rename domains/testing/skills/{falsifying-test => red-on-base}/skill.md (92%) diff --git a/domains/pr-workflow/skills/evidence/references/claim-extraction.md b/domains/pr-workflow/skills/evidence/references/claim-extraction.md index 9dde89d9..e5654608 100644 --- a/domains/pr-workflow/skills/evidence/references/claim-extraction.md +++ b/domains/pr-workflow/skills/evidence/references/claim-extraction.md @@ -42,7 +42,7 @@ A good claim is **falsifiable** (observable outcome + clear falsifier), **surfac |---|---| | "Improves performance" | "Opening the Activity tab: TBT drops below 200ms (was >600ms)" — name the interaction, metric, threshold | | "Fixes the bug" | "With privacy mode on, the Perps tab balance is masked" — observable behavior + precondition + surface | -| "Refactor, no behavior change" | Negation claim: "behavior of `<surface>` is unchanged" → prove via falsifying-test-stays-green / snapshot / identical output, **not** a screenshot | +| "Refactor, no behavior change" | Negation claim: "behavior of `<surface>` is unchanged" → prove via a red-on-base test that stays green / snapshot / identical output, **not** a screenshot | | "Adds a null check" (restates the diff) | "No crash when `<field>` is null on `<surface>`" — the behavior, not the code | | Body promises X, diff does Y | Not a claim — **flag the drift** to the author | diff --git a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md index 728aae6f..d7f82692 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md @@ -54,7 +54,7 @@ Legend: **first-class lanes** are `##`-headed; closely-related variants are sub- ## B3. Falsifying regression test ⭐ - **Proves — strongest single proof a fix targets the bug:** a new test that **fails on `main` and passes on the branch**. Show both runs. - - **Engine: the `falsifying-test` skill.** + - **Engine: the `red-on-base` skill.** - **Capture:** add the test, run it on the PR branch (pass) and on the PR's **merge-base** (fail) — pin the base, don't use whatever `main` points at today. Pair with the PR's `Fixes #N`. **Read the base failure's message, not its exit code:** it must fail on the assertion that encodes the bug. A `ModuleNotFoundError`, a missing fixture, or an unrelated pre-existing red produces an identical non-zero exit and falsifies nothing. - **Reach for it:** every bug-fix PR. If you can't write a test that fails on main, question whether the fix addresses the reported bug. diff --git a/domains/pr-workflow/skills/evidence/references/output-templates.md b/domains/pr-workflow/skills/evidence/references/output-templates.md index 0aae9992..752ac449 100644 --- a/domains/pr-workflow/skills/evidence/references/output-templates.md +++ b/domains/pr-workflow/skills/evidence/references/output-templates.md @@ -46,7 +46,7 @@ falsifiable thing under test> head `<sha>` · <YYYY-MM-DD> · <check name in wor Icons: `✅` proven · `⚠️` partial or scoped · `📋` measured, no verdict asserted · `❌` failed. Never `❌` for a gap in *evidence* — that reads as a verdict on the author's work. -**Check name, in words.** *falsifying-test check*, *render-count check*, +**Check name, in words.** *red-on-base check*, *render-count check*, *dependency-containment check*. Never the lane id: `B3` is an address into [evidence-catalog.md](evidence-catalog.md), which the reviewer cannot open. diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh index afec5d71..d7e8753e 100755 --- a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -170,6 +170,8 @@ printf 'falsify-probe: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.js # orchestrator reads them and writes ONE open question about THIS diff. printf 'limits: one line of one file was mutated. Says nothing about other paths into the same mechanism, whether it is reachable in production, or whether the guarded behaviour is -correct. A falsifying test proves the test has power, not that the fix is right.%s\n' \ +correct. This probe proves the suite notices one mutated line, which is not the same as the +base-against-branch proof that a test is connected to the reported bug -- see the red-on-base +skill for that experiment.%s\n' \ "$([ "$VERDICT" = vacuous ] && printf '\n vacuous: the mechanism is unguarded by this suite — what else depends on it?')" >&2 exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 334133f6..9ac32af0 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -656,7 +656,7 @@ Where evidence sits in the PR lifecycle (see the public `pr-workflow` siblings): | category | engine | |---|---| - | B3 falsifying regression test | `/falsifying-test` | + | B3 falsifying regression test | `/red-on-base` | | B7 deterministic interleaving (concurrency / ordering) | `/race-condition-repro` | | C4 React render & selector proof | `/react-render-delta` | | C9 memory leak | `/memory-leak` | diff --git a/domains/testing/skills/falsifying-test/skill.md b/domains/testing/skills/red-on-base/skill.md similarity index 92% rename from domains/testing/skills/falsifying-test/skill.md rename to domains/testing/skills/red-on-base/skill.md index 65f36e42..0c7066af 100644 --- a/domains/testing/skills/falsifying-test/skill.md +++ b/domains/testing/skills/red-on-base/skill.md @@ -1,10 +1,10 @@ --- -name: falsifying-test -description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /falsifying-test, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by evidence as the engine behind its falsifying regression test evidence category. +name: red-on-base +description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /red-on-base, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by evidence as the engine behind its falsifying regression test evidence category. Named for its own discipline rather than for that category, so it is not mistaken for `falsify-probe.sh`, which shares the two-arm shape and answers a different question. maturity: experimental --- -# /falsifying-test +# /red-on-base Reach for this on **every bug-fix PR**. A test that passes on the branch proves the branch is green. A test that **fails on base and passes on the branch** proves the change is causally From 50b52dc90485e1086aaefa6c376e1cfe6df2c3d7 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 06:13:01 -0400 Subject: [PATCH 108/135] Move `red-on-base` out; this PR is the instrument MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every other engine `evidence` names ships in its own pull request — `memory-leak`, `race-condition-repro`, `supply-chain-audit`, `react-render-delta`, `agent-run-cost`. This one was the sole exception, and the reason was authoring order rather than design: it was written alongside the orchestrator before the split between the instrument and the reasoning that points it existed as a concept. It lands in #106 with the other reasoning skills, whose substance is the same kind — what counts as proof, and how a proof can look right while testing the wrong thing. What stays here is the machinery: the runners, the run workflow, the gate, the hooks. The B3 engine cell now names a skill that arrives in #106, which is a dangling name in a table rather than a broken link, and resolves whichever order the two merge. --- domains/testing/skills/red-on-base/skill.md | 116 -------------------- 1 file changed, 116 deletions(-) delete mode 100644 domains/testing/skills/red-on-base/skill.md diff --git a/domains/testing/skills/red-on-base/skill.md b/domains/testing/skills/red-on-base/skill.md deleted file mode 100644 index 0c7066af..00000000 --- a/domains/testing/skills/red-on-base/skill.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -name: red-on-base -description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /red-on-base, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by evidence as the engine behind its falsifying regression test evidence category. Named for its own discipline rather than for that category, so it is not mistaken for `falsify-probe.sh`, which shares the two-arm shape and answers a different question. -maturity: experimental ---- - -# /red-on-base - -Reach for this on **every bug-fix PR**. A test that passes on the branch proves the branch is -green. A test that **fails on base and passes on the branch** proves the change is causally -connected to the reported bug. Only the second is evidence, and the gap between them is where -this skill lives. - -> **Falsifier.** A test that fails on base for a reason unrelated to the bug. A missing import, -> a fixture the base commit doesn't have, a helper introduced by the branch, an unrelated -> pre-existing failure — every one produces a red run and a non-zero exit code that looks -> exactly like a correct falsification. **The exit code is not the evidence; the assertion -> message is.** - -## Method - -1. **Write the test against the reported behaviour, not the diff.** Start from the issue's - reproduction. A test derived from reading the fix tends to assert the fix's mechanism and - will pass on base the moment the mechanism is reachable by other means — or fail on base - for structural reasons rather than behavioural ones. - -2. **Run it on base FIRST, and read the failure output.** Not the exit code — the message. It - must fail on the **assertion that encodes the bug**: an expected value that differs, a state - that wasn't reached, an event that didn't fire. If base fails with a - `ModuleNotFoundError`, a syntax error, or a helper that doesn't exist yet, you have not - falsified anything; you have discovered that the test can't run there. - -3. **Pin the base explicitly.** Use the PR's actual merge-base, not whatever `main` points at - today. `main` moves; a re-run weeks later against a drifted `main` is a different - experiment and may fail for reasons that have nothing to do with the fix. - -4. **Make the test runnable on base.** When the test needs a helper or fixture the branch - introduces, split it: land the scaffolding in a form that exists on both sides, or inline - the setup so the test file is self-contained. If that's impossible, say so and downgrade the - claim — a test that *cannot* run on base gives a branch-only pass, which is a weaker piece - of evidence and should not be presented as a falsifying one. - -5. **Confirm it fails for one reason, not several.** If base has unrelated failures in the same - file or suite, scope the run to the new test (by name/path) so the red is attributable. A - suite that was already red proves nothing about your assertion. - -6. **Show both runs, and let a tool write them down.** Base: the assertion failure, verbatim. - Branch: the pass. Same command, same filter, both commits identified. Retyped output is a - self-report — indistinguishable from output that was never produced — so the two arms want - to come out of a runner rather than a paste buffer. - - `evidence` ships the mechanism: its run workflow takes `ref` and `baseline`, checks out both - commits, executes the same command at each, and attaches the artifacts to a run URL a reader - can open without going through you. Wrapping the test command in `capture.sh` gets the same - property locally, minus the reader-verifiable half — and that runner's footer says so, in - the artifact, rather than leaving the gap for a reviewer to notice. - -7. **Pair it with the issue.** The PR's `Fixes #N` plus a test named for the behaviour makes - the causal chain checkable by a reader who runs nothing. - -## When you can't write one - -This is a finding, not a gap to paper over. If no test fails on base, one of these is true: - -- **The bug isn't where the fix is.** The most common case, and the reason to run this check - before review rather than after. -- **The reported behaviour isn't reproducible in the harness** — timing, environment, or a - real-device dependency. Say which, and reach for a different evidence category (a - deterministic interleaving test for ordering bugs, an e2e trace for environment-dependent - ones). -- **The fix is a refactor or hardening change, not a bug fix.** Fine — then the PR's claim - should say that, and this category doesn't apply. - -State which one. "No test added" with no explanation reads as an omission; the diagnosis is -useful information about the change. - -## Output - -``` -Falsifying test — <test name> (Fixes #N) - base <sha> FAIL <the assertion line, verbatim> - branch <sha> PASS - command: <exact command, same on both> - scoped: <how the run was limited to this test> -``` - -## The sibling experiment, and why it is not this one - -`evidence` also ships `falsify-probe.sh`, which has the same two-arm shape and answers a -different question. The distinction is worth holding, because conflating them produces a proof -of the wrong thing: - -| | arms | question | -|---|---|---| -| **this skill** | base commit, branch commit — same test | is the test causally connected to the reported bug? | -| **`falsify-probe.sh`** | one commit, one line mutated | does the test fail when the mechanism it guards is removed? | - -A test can pass this skill and fail that one: it fails on base because the fix was not there, -and passes under mutation because it asserts something adjacent to the mechanism. The reverse -also happens. On a bug-fix PR you usually want both — the first proves the test is about *this -bug*, the second proves it will keep noticing. - -What the runner does mechanise is step 2. Its guards refuse to call a red arm a falsification -when the suite ran fewer tests than the baseline, or failed to load at all — which is this -skill's falsifier, enforced rather than remembered. It also takes the names of the tests you -expect to fail, so a red run in the wrong place is reported as such instead of passing as a -falsification. - -## Related - -- `evidence` — packages this skill's output as its [falsifying regression test category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md), - and supplies the two-commit run harness step 6 asks for. The deterministic-interleaving - category is the sibling for concurrency and temporal-ordering bugs; `race-condition-repro` - drives it. -- `react-render-delta` — the same before/after discipline applied to a measured quantity - rather than a boolean. From ca9111fa74b7635fb499edb7d627ee6986670c59 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 06:13:30 -0400 Subject: [PATCH 109/135] Take `red-on-base` from #84, where it was the odd one out MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Its substance is the same as everything else here: what counts as proof, and how a proof can look right while testing the wrong thing. Its falsifier — a base commit that fails for the wrong reason produces an identical exit code and proves nothing — is the same move `unmeasured-join` and `scope-of-search` make on different material. It sat in #84 because it was written alongside the orchestrator, before the instrument and the reasoning that points it were separate ideas. Every other engine already ships in its own pull request. Renamed from `falsifying-test` before the move, because it and `falsify-probe.sh` shared a root while answering different questions — base-against-branch asks whether a test is connected to the reported bug, one-commit-with-a-mutated-line asks whether a suite notices the mechanism going away. --- domains/testing/skills/red-on-base/skill.md | 116 ++++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100644 domains/testing/skills/red-on-base/skill.md diff --git a/domains/testing/skills/red-on-base/skill.md b/domains/testing/skills/red-on-base/skill.md new file mode 100644 index 00000000..0c7066af --- /dev/null +++ b/domains/testing/skills/red-on-base/skill.md @@ -0,0 +1,116 @@ +--- +name: red-on-base +description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /red-on-base, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by evidence as the engine behind its falsifying regression test evidence category. Named for its own discipline rather than for that category, so it is not mistaken for `falsify-probe.sh`, which shares the two-arm shape and answers a different question. +maturity: experimental +--- + +# /red-on-base + +Reach for this on **every bug-fix PR**. A test that passes on the branch proves the branch is +green. A test that **fails on base and passes on the branch** proves the change is causally +connected to the reported bug. Only the second is evidence, and the gap between them is where +this skill lives. + +> **Falsifier.** A test that fails on base for a reason unrelated to the bug. A missing import, +> a fixture the base commit doesn't have, a helper introduced by the branch, an unrelated +> pre-existing failure — every one produces a red run and a non-zero exit code that looks +> exactly like a correct falsification. **The exit code is not the evidence; the assertion +> message is.** + +## Method + +1. **Write the test against the reported behaviour, not the diff.** Start from the issue's + reproduction. A test derived from reading the fix tends to assert the fix's mechanism and + will pass on base the moment the mechanism is reachable by other means — or fail on base + for structural reasons rather than behavioural ones. + +2. **Run it on base FIRST, and read the failure output.** Not the exit code — the message. It + must fail on the **assertion that encodes the bug**: an expected value that differs, a state + that wasn't reached, an event that didn't fire. If base fails with a + `ModuleNotFoundError`, a syntax error, or a helper that doesn't exist yet, you have not + falsified anything; you have discovered that the test can't run there. + +3. **Pin the base explicitly.** Use the PR's actual merge-base, not whatever `main` points at + today. `main` moves; a re-run weeks later against a drifted `main` is a different + experiment and may fail for reasons that have nothing to do with the fix. + +4. **Make the test runnable on base.** When the test needs a helper or fixture the branch + introduces, split it: land the scaffolding in a form that exists on both sides, or inline + the setup so the test file is self-contained. If that's impossible, say so and downgrade the + claim — a test that *cannot* run on base gives a branch-only pass, which is a weaker piece + of evidence and should not be presented as a falsifying one. + +5. **Confirm it fails for one reason, not several.** If base has unrelated failures in the same + file or suite, scope the run to the new test (by name/path) so the red is attributable. A + suite that was already red proves nothing about your assertion. + +6. **Show both runs, and let a tool write them down.** Base: the assertion failure, verbatim. + Branch: the pass. Same command, same filter, both commits identified. Retyped output is a + self-report — indistinguishable from output that was never produced — so the two arms want + to come out of a runner rather than a paste buffer. + + `evidence` ships the mechanism: its run workflow takes `ref` and `baseline`, checks out both + commits, executes the same command at each, and attaches the artifacts to a run URL a reader + can open without going through you. Wrapping the test command in `capture.sh` gets the same + property locally, minus the reader-verifiable half — and that runner's footer says so, in + the artifact, rather than leaving the gap for a reviewer to notice. + +7. **Pair it with the issue.** The PR's `Fixes #N` plus a test named for the behaviour makes + the causal chain checkable by a reader who runs nothing. + +## When you can't write one + +This is a finding, not a gap to paper over. If no test fails on base, one of these is true: + +- **The bug isn't where the fix is.** The most common case, and the reason to run this check + before review rather than after. +- **The reported behaviour isn't reproducible in the harness** — timing, environment, or a + real-device dependency. Say which, and reach for a different evidence category (a + deterministic interleaving test for ordering bugs, an e2e trace for environment-dependent + ones). +- **The fix is a refactor or hardening change, not a bug fix.** Fine — then the PR's claim + should say that, and this category doesn't apply. + +State which one. "No test added" with no explanation reads as an omission; the diagnosis is +useful information about the change. + +## Output + +``` +Falsifying test — <test name> (Fixes #N) + base <sha> FAIL <the assertion line, verbatim> + branch <sha> PASS + command: <exact command, same on both> + scoped: <how the run was limited to this test> +``` + +## The sibling experiment, and why it is not this one + +`evidence` also ships `falsify-probe.sh`, which has the same two-arm shape and answers a +different question. The distinction is worth holding, because conflating them produces a proof +of the wrong thing: + +| | arms | question | +|---|---|---| +| **this skill** | base commit, branch commit — same test | is the test causally connected to the reported bug? | +| **`falsify-probe.sh`** | one commit, one line mutated | does the test fail when the mechanism it guards is removed? | + +A test can pass this skill and fail that one: it fails on base because the fix was not there, +and passes under mutation because it asserts something adjacent to the mechanism. The reverse +also happens. On a bug-fix PR you usually want both — the first proves the test is about *this +bug*, the second proves it will keep noticing. + +What the runner does mechanise is step 2. Its guards refuse to call a red arm a falsification +when the suite ran fewer tests than the baseline, or failed to load at all — which is this +skill's falsifier, enforced rather than remembered. It also takes the names of the tests you +expect to fail, so a red run in the wrong place is reported as such instead of passing as a +falsification. + +## Related + +- `evidence` — packages this skill's output as its [falsifying regression test category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md), + and supplies the two-commit run harness step 6 asks for. The deterministic-interleaving + category is the sibling for concurrency and temporal-ordering bugs; `race-condition-repro` + drives it. +- `react-render-delta` — the same before/after discipline applied to a measured quantity + rather than a boolean. From c30e57582c37e1c7923c725b0b23ea00944b0bb0 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 07:57:53 -0400 Subject: [PATCH 110/135] Say what this skill does not cover It reviews what a field carries and not whether the send is permitted, and a reader who runs it has covered half the question while believing otherwise. An unstated scope on a review is the same defect as an unstated scope on a search: the negative reads wider than the thing that produced it. The other axis lives in the private repo, because naming the conditions under which a control does not run is a different kind of document from describing the control. --- .../security/skills/privacy-egress-diligence/skill.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/domains/security/skills/privacy-egress-diligence/skill.md b/domains/security/skills/privacy-egress-diligence/skill.md index 507e82ad..7dbb3b89 100644 --- a/domains/security/skills/privacy-egress-diligence/skill.md +++ b/domains/security/skills/privacy-egress-diligence/skill.md @@ -93,6 +93,17 @@ someone, usually while shipping an unrelated feature. Presence proves authorship confident-sounding "this is fine" from a reviewer is exactly the artifact that lets an unreviewed field through. +## What this does not cover + +This reviews **what** a field carries. It does not review whether the send is permitted at all — +consent state, basic functionality, compliance and region gates, and what happens to data +buffered before a user decided. A correctly masked field sent without consent is the worse +failure of the two, and nothing here can see it. + +That axis is reviewed separately, in the private skills repo, because naming the conditions +under which a control does not run is a different kind of document from describing the control. +Run both on a change that adds collection; either alone leaves half the question open. + ## Common pitfalls | Mistake | Correct approach | From 0558f9e11b4a36af3bb39385c6484946b48b4397 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 08:16:26 -0400 Subject: [PATCH 111/135] Point the engine table at the renamed skills --- domains/pr-workflow/skills/debug/skill.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/domains/pr-workflow/skills/debug/skill.md b/domains/pr-workflow/skills/debug/skill.md index 556a1b72..1a6458a7 100644 --- a/domains/pr-workflow/skills/debug/skill.md +++ b/domains/pr-workflow/skills/debug/skill.md @@ -1,12 +1,12 @@ --- name: debug -description: Locate the cause of a symptom you cannot yet explain — a crash, a leak, a flake, a production error spike, a number that moved. The sibling of pr-validate: where pr-validate is handed a claim and looks for the observation that would falsify it, this is handed a symptom and must generate the hypothesis first, then kill it. Classifies the symptom into a defect class, routes to the engine skill that owns that class (memory-leak-hunt, race-condition-proof, react-render-proof, sentry-grafana-cross-ref, extension-errors-debugging, typescript-compiler-blindspots, supply-chain-audit), and holds the investigation to the same evidence bar pr-validate applies — an instrument that cannot fail is not evidence, a null needs its sensitivity stated, and a finding is scoped to what the change introduced versus what pre-existed. Stops when the cause is located or the class is excluded, not when a plausible story is available. Triggers on /debug, or when asked to debug, diagnose, or investigate a symptom, find why something is slow, leaking, flaky, or erroring, chase a production alert to its cause, or reproduce a bug that cannot be reproduced by hand. +description: Locate the cause of a symptom you cannot yet explain — a crash, a leak, a flake, a production error spike, a number that moved. The sibling of evidence: where evidence is handed a claim and looks for the observation that would falsify it, this is handed a symptom and must generate the hypothesis first, then kill it. Classifies the symptom into a defect class, routes to the engine skill that owns that class (memory-leak, race-condition-repro, react-render-proof, sentry-grafana-correlation, extension-errors-debugging, tsc-blindspots, supply-chain-audit), and holds the investigation to the same evidence bar evidence applies — an instrument that cannot fail is not evidence, a null needs its sensitivity stated, and a finding is scoped to what the change introduced versus what pre-existed. Stops when the cause is located or the class is excluded, not when a plausible story is available. Triggers on /debug, or when asked to debug, diagnose, or investigate a symptom, find why something is slow, leaking, flaky, or erroring, chase a production alert to its cause, or reproduce a bug that cannot be reproduced by hand. maturity: experimental --- # /debug -`pr-validate` is given a claim and looks for the observation that would prove it false. +`evidence` is given a claim and looks for the observation that would prove it false. This is given a **symptom** and has to produce the hypothesis before anything can falsify it. That difference is the whole skill. In review, the claim is someone else's and the social @@ -23,8 +23,8 @@ first theory that fit the first observation. ## Do Not Use When -- The PR states a claim and you need it settled — that is `/pr-validate`. -- The cause is known and you are validating the fix — that is `/pr-validate`, or the engine +- The PR states a claim and you need it settled — that is `/evidence`. +- The cause is known and you are validating the fix — that is `/evidence`, or the engine skill directly. - You want an after-the-fact writeup of a resolved failure — that is a postmortem, not this. @@ -46,18 +46,18 @@ first theory that fit the first observation. | Symptom | Class | Engine | |---|---|---| -| Memory grows across a repeated flow; tab or worker dies over time | retention | `memory-leak-hunt` | -| Intermittent failure; passes on rerun; order-dependent | interleaving | `race-condition-proof` | +| Memory grows across a repeated flow; tab or worker dies over time | retention | `memory-leak` | +| Intermittent failure; passes on rerun; order-dependent | interleaving | `race-condition-repro` | | UI janks, re-renders excessively, selector recomputes | wasted render work | `react-render-proof` | -| Production error spike, latency change, or a metric that moved | production signal | `sentry-grafana-cross-ref` | +| Production error spike, latency change, or a metric that moved | production signal | `sentry-grafana-correlation` | | Extension-specific: MV3 vs MV2, background vs UI context, service-worker lifecycle | platform | `extension-errors-debugging` | -| Runtime value disagrees with its declared type; green typecheck, wrong behaviour | type/reality drift | `typescript-compiler-blindspots` | +| Runtime value disagrees with its declared type; green typecheck, wrong behaviour | type/reality drift | `tsc-blindspots` | | Started after a dependency change; new capability or transitive edge | supply chain | `supply-chain-audit` | | None of the above, or several | — | bisect to a change first, then re-classify | ## The evidence bar carries over -The engines are shared with `pr-validate`, and so are its trust gates. They matter more here, +The engines are shared with `evidence`, and so are its trust gates. They matter more here, because in review a weak instrument produces a weak claim someone else will challenge — in debugging it produces a wrong theory nobody checks. From 594a1a7f03a238a0ca4711285f43f118a95af303 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 08:46:51 -0400 Subject: [PATCH 112/135] Check where the run is going, not only what it says MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eleven checks tested properties of the comment text. Text can be perfect and land somewhere nobody will read it, and that is what happened: across one register of published runs, 22 of 27 comments went onto pull requests that had already merged when they were posted — median 22 days after the merge, one 178 days after. The gate passed every one, because no property of a comment reveals the state of its destination. Check 12 takes `--target owner/repo#N` and blocks on anything that is not open. Omitting the target fails rather than passes: an unchecked destination is the condition that produced all 22. Without `gh` on PATH it reports UNVERIFIED and still refuses, since the point is that silence here is indistinguishable from success. --- .../skills/evidence/scripts/attest-gate.sh | 37 +++++++++++++++++-- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index cc454e32..3f058d5f 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -10,12 +10,22 @@ # # 0 all checks pass → proceed to the dispatched passes # 1 one or more failed → BLOCKED, do not publish +# +# --target owner/repo#N is how check 12 learns where this is going. Without it the gate +# cannot tell a live review from a merged one, and the difference is the whole point. # 2 usage error set -uo pipefail -FILE="${1:-}"; REF="" -[ $# -ge 2 ] && [ "${2:-}" = "--reference" ] && REF="${3:-}" -[ -n "$FILE" ] || { echo "usage: attest-gate.sh <artifact.md> [--reference <file>]" >&2; exit 2; } +FILE="${1:-}"; REF=""; TARGET="" +shift || true +while [ $# -gt 0 ]; do + case "$1" in + --reference) REF="${2:-}"; shift 2 ;; + --target) TARGET="${2:-}"; shift 2 ;; + *) shift ;; + esac +done +[ -n "$FILE" ] || { echo "usage: attest-gate.sh <artifact.md> [--reference <file>] [--target <owner/repo#N>]" >&2; exit 2; } [ -f "$FILE" ] || { echo "attest-gate: not found: $FILE" >&2; exit 2; } FAILED=0 @@ -160,6 +170,27 @@ if [ -n "$REF" ] && [ -f "$REF" ]; then [ "$c" -eq 0 ] && [ "$r" -gt 0 ] && printf ' reference is capture-led and this is prose-only — see check 5\n' fi +# 12 — the destination. Every check above tests a property of the text, and text can be +# perfect while landing somewhere nobody will read it. Measured across one register of +# published runs: 22 of 27 comments went onto pull requests that had ALREADY merged when +# they were posted, median 22 days after the merge, one of them 178 days after. The gate +# was clean on every one. A finding delivered to a closed pull request changes nothing, +# and no property of the comment can reveal that. +echo +if [ -z "$TARGET" ]; then + fail "12 destination is open" "no --target given, so nobody checked whether the pull request is still open. Pass --target owner/repo#N." +elif ! command -v gh >/dev/null 2>&1; then + printf ' ???? %s\n %s\n' "12 destination is open" "gh not on PATH — the destination is UNVERIFIED, not passing. Check it by hand before publishing." +else + t_repo="${TARGET%%#*}"; t_num="${TARGET##*#}" + t_state="$(gh api "repos/$t_repo/pulls/$t_num" --jq 'if .merged_at then "merged" else .state end' 2>/dev/null || echo unknown)" + case "$t_state" in + open) pass "12 destination is open" ;; + unknown) fail "12 destination is open" "could not read $TARGET — do not publish to a destination you could not check" ;; + *) fail "12 destination is open" "$TARGET is $t_state. A run published to a closed pull request reaches no reviewer and changes no decision." ;; + esac +fi + echo if [ "$FAILED" -eq 0 ]; then echo "attest-gate: phase 0 clean — proceed to /outframe ‖ /missing ‖ /press" From d50d94fb71013440e22937836ce63c63eee9e7b0 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 12:18:50 -0400 Subject: [PATCH 113/135] Add exit-tracing, a runtime declaration check, and a dead-module check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four additions, each of which found something the existing sections did not point at, while reviewing a JS->TS conversion. `any` absorption gains its complement: the existing bullet traces where a confidently-typed value entered, which finds nothing when the value is a module's own return. Tracing where an `any` *exits* — assigning the return to two impossible types with a known-typed sibling as control — found `any` escaping a resolver into its caller. The control line is load-bearing: without it a silent probe is indistinguishable from one that cannot fail, and running outside the project tsconfig produces errors that are the harness rather than the finding. Ethers `Contract` dynamic methods are named as a source, since the ABI is runtime data and the call reads as an ordinary typed await. It is already tracked at #31973, where one consumer declares `Promise<any>` with a disable comment and another lets it infer; the second is the dangerous form. Ambient `declare module` verification moves from reading package source to requiring the package and checking exports, return `typeof`, and whether a default export is legitimate under esModuleInterop. Inventory now begins by checking the module is referenced at all — a dead module's types are unfalsifiable, and its conversion is a deletion candidate rather than a typing exercise. --- .../references/false-negatives.md | 46 +++++++++++++++++-- .../typescript/skills/tsc-blindspots/skill.md | 12 +++++ 2 files changed, 55 insertions(+), 3 deletions(-) diff --git a/domains/typescript/skills/tsc-blindspots/references/false-negatives.md b/domains/typescript/skills/tsc-blindspots/references/false-negatives.md index 5bcf97b8..fc494022 100644 --- a/domains/typescript/skills/tsc-blindspots/references/false-negatives.md +++ b/domains/typescript/skills/tsc-blindspots/references/false-negatives.md @@ -115,10 +115,38 @@ meta.version.toFixed(2); // may be a string at runtime ``` An `any` satisfies every annotation silently. Sources: untyped dependencies, -`JSON.parse`, generics that default to `any`, and `as any`. +`JSON.parse`, generics that default to `any`, `as any` — and the one that hides +best in a typed-looking file: **dynamic methods on an ethers `Contract`**, where +the ABI is runtime data so every call returns `any` while reading as an ordinary +typed `await`. This is a known source in `metamask-extension` +([#31973](https://github.com/MetaMask/metamask-extension/issues/31973)): +`shared/lib/token-util.ts` declares it as `Promise<any>` with a disable comment, +which is the honest form. The dangerous form is letting it infer — nothing +annotates it, so it propagates into the module's return type unflagged. - **In review:** trace where a confidently-typed value *entered* the program. If it entered as `any`, its type is a wish. +- **Also trace where an `any` *exits*.** The bullet above looks backwards from a + suspicious value; this looks forward from a module's public surface. Assign its + return to two impossible types, with a known-typed sibling as a control: + + ```ts + const { type, hash } = await resolveEnsToIpfsContentId(args); + const a: number = hash; // compiles ⇒ `hash` is any + const c: symbol = hash.whateverIWant.deeply; // compiles ⇒ ditto + const d: number = type; // MUST error ⇒ probe can discriminate + ``` + + If the nonsense compiles and the control errors, `any` is escaping into every + caller. **The control line is not optional** — without it, a probe that reports + nothing is indistinguishable from a probe that cannot fail. Run it under the + project's `tsconfig`, not a standalone `tsc` invocation, or missing ambient + declarations and `resolveJsonModule` will produce errors that are the harness + rather than the finding. +- **Precise signatures around an `any` source make it worse, not better.** A file + with a derived provider type and `hexValueIsEmpty(value: string | null | undefined)` + reads as a checked boundary while every value crossing it is unchecked. A + migration that adds those signatures is what creates the appearance. ### 8. Ambient `declare module` is an unverified assertion @@ -134,8 +162,20 @@ them to the package. Getting a return type wrong here is invisible forever, and the declaration is **global**, so it also shadows any real types the package later ships. -- **In review:** read the package's actual source at the installed version when a - `declare module` is added or changed. Prefer `@types/*` or a PR upstream. +- **In review:** verify the declaration against the installed package at runtime — + faster and more decisive than reading source: + + ```bash + node -e 'const m = require("@ensdomains/content-hash"); + console.log(Object.keys(m), "default:", typeof m.default); + console.log(typeof m.decode(m.encode("ipfs-ns", "Qm…")));' + ``` + + Check three things: **the exports exist**, **each declared signature's return + `typeof` matches**, and **whether `export default` is legitimate** — a CJS module + with `typeof m.default === 'undefined'` still warrants a default declaration *if* + `esModuleInterop` is on, and is a defect if it is not. Prefer `@types/*` or an + upstream PR over hand-writing. ### 9. External data is asserted, not validated diff --git a/domains/typescript/skills/tsc-blindspots/skill.md b/domains/typescript/skills/tsc-blindspots/skill.md index 495142b4..8cdab2cc 100644 --- a/domains/typescript/skills/tsc-blindspots/skill.md +++ b/domains/typescript/skills/tsc-blindspots/skill.md @@ -86,6 +86,18 @@ gh pr diff <pr> | grep -nE '^\+.*(type [A-Z]|interface [A-Z]|: (Record<|string|n List them. Each one is a claim you are about to test. +**First, check each converted module is still referenced:** + +```bash +grep -rn "moduleName" --include=*.ts --include=*.js . | grep -v node_modules | grep -v '\.test\.' +``` + +If the only hits are the module's own definition and its test, the file is dead — +every type on it is unfalsifiable, because nothing constrains it and no divergence +can ever surface. This is the highest value-per-second check in a migration, and +it reorders the work: a dead module's conversion is a deletion candidate, not a +typing exercise. + ### Step 2: Find the authoritative source for each Work down this list — the first hit wins. In the worked example 9 of the 12 From 539c58b25583dd0f12eb6d8371d5a7cd89804345 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 12:37:57 -0400 Subject: [PATCH 114/135] Add `lane-graphs`: route to a small executable graph, not a prose procedure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Determinism belongs on the envelope and not on the inquiry. Computing a merge-base, checking a probe loaded, verifying a mutation landed where it was aimed — none of that needs judgement, and all of it has been got wrong by hand. Which measurement answers a claim still does. Five nodes per graph, two of which exist only because they were missing when something published anyway: reading the treatment back off disk, and naming the observation it should produce. A run whose mutation silently changed shape, or that went red somewhere other than where it aimed, satisfied every other check. The router contract carries the part that keeps routing honest. A router over enough lanes always finds a best score, so no-match has to be a first-class result rather than a fallback into the nearest lane, and an unrouted read of the mechanism runs regardless of what matched — the catalog is a list of questions someone already thought of, and the findings worth having sit outside it. Three lanes converted from prose: mutation power, base-against-branch, and render delta. Each states its own blind spot in its output, because a green result that does not say what it declined to measure reads as broader than it is. --- .../assets/base-branch-proof.graph.json | 51 +++++++++++ .../assets/mutation-power.graph.json | 57 ++++++++++++ .../assets/render-delta.graph.json | 51 +++++++++++ .../pr-workflow/skills/lane-graphs/skill.md | 86 +++++++++++++++++++ 4 files changed, 245 insertions(+) create mode 100644 domains/pr-workflow/skills/lane-graphs/assets/base-branch-proof.graph.json create mode 100644 domains/pr-workflow/skills/lane-graphs/assets/mutation-power.graph.json create mode 100644 domains/pr-workflow/skills/lane-graphs/assets/render-delta.graph.json create mode 100644 domains/pr-workflow/skills/lane-graphs/skill.md diff --git a/domains/pr-workflow/skills/lane-graphs/assets/base-branch-proof.graph.json b/domains/pr-workflow/skills/lane-graphs/assets/base-branch-proof.graph.json new file mode 100644 index 00000000..ae069d53 --- /dev/null +++ b/domains/pr-workflow/skills/lane-graphs/assets/base-branch-proof.graph.json @@ -0,0 +1,51 @@ +{ + "lane": "B3 \u2014 is this test connected to the reported bug", + "claim_shape": "This test would have caught <bug>.", + "executor": "evidence-run.yml with ref and baseline, runner capture wrapping the suite", + "does_not_cover": [ + "whether the test notices the mechanism later regressing \u2014 that is mutation-power", + "bugs whose reproduction needs timing, a device, or an environment the harness lacks" + ], + "nodes": [ + { + "id": "pre-target-open", + "type": "precondition", + "check": "target pull request state == open" + }, + { + "id": "pre-merge-base", + "type": "precondition", + "check": "base is merge_base_commit.sha from the compare endpoint, not the base branch tip", + "why": "the base branch tip moves; a re-run weeks later is a different experiment" + }, + { + "id": "baseline", + "type": "baseline", + "run": "the new test at the merge base", + "expect": "fails, and the failure message names the assertion that encodes the bug", + "on_fail": [ + "passes \u2014 the test is not connected to the reported behaviour", + "fails on a load or fixture error \u2014 the test cannot run there, nothing is falsified" + ] + }, + { + "id": "treatment", + "type": "treatment", + "run": "the same test at head, same command, same filter" + }, + { + "id": "prediction", + "type": "prediction", + "expect": "passes at head; the base failure is an assertion, not an import" + }, + { + "id": "capture", + "type": "capture", + "emit": [ + "both arms verbatim", + "both SHAs", + "the exact command" + ] + } + ] +} \ No newline at end of file diff --git a/domains/pr-workflow/skills/lane-graphs/assets/mutation-power.graph.json b/domains/pr-workflow/skills/lane-graphs/assets/mutation-power.graph.json new file mode 100644 index 00000000..cc412837 --- /dev/null +++ b/domains/pr-workflow/skills/lane-graphs/assets/mutation-power.graph.json @@ -0,0 +1,57 @@ +{ + "lane": "B3 \u2014 does the suite notice the mechanism going away", + "claim_shape": "This test guards <mechanism>.", + "executor": "evidence-run.yml, runner falsify-probe", + "does_not_cover": [ + "other paths into the same mechanism \u2014 one line of one file is mutated", + "whether the guarded behaviour is correct, only that the suite reacts", + "whether the mechanism is reachable in production" + ], + "nodes": [ + { + "id": "pre-target-open", + "type": "precondition", + "check": "target pull request state == open", + "why": "a run published to a merged pull request reaches no reviewer; measured at 22 of 27" + }, + { + "id": "pre-ref-pinned", + "type": "precondition", + "check": "ref is a 40-char SHA, and is the pull request head", + "why": "a branch name makes the run unrepeatable" + }, + { + "id": "baseline", + "type": "baseline", + "run": "suite at ref, unmutated", + "expect": "passes, and the test count is recorded", + "on_fail": "baseline-already-failing \u2014 nothing can be concluded" + }, + { + "id": "treatment", + "type": "treatment", + "run": "replace exactly one line; read the line back off disk", + "expect": "the line on disk equals the line requested", + "on_fail": "the mutation was altered in transit \u2014 do not report a verdict" + }, + { + "id": "prediction", + "type": "prediction", + "expect": "named tests fail, and the arm ran the same test count as baseline", + "on_fail": [ + "fewer tests ran \u2014 the module broke, nothing was falsified", + "different tests failed \u2014 the mutation landed somewhere unintended" + ] + }, + { + "id": "capture", + "type": "capture", + "emit": [ + "json", + "log", + "markdown" + ], + "cite": "the workflow run URL" + } + ] +} \ No newline at end of file diff --git a/domains/pr-workflow/skills/lane-graphs/assets/render-delta.graph.json b/domains/pr-workflow/skills/lane-graphs/assets/render-delta.graph.json new file mode 100644 index 00000000..693d0a43 --- /dev/null +++ b/domains/pr-workflow/skills/lane-graphs/assets/render-delta.graph.json @@ -0,0 +1,51 @@ +{ + "lane": "C4 \u2014 did the work actually decrease", + "claim_shape": "This change stops <consumer> re-rendering / <selector> recomputing.", + "executor": "evidence-run.yml with ref and baseline, runner render-count or selector-recompute", + "does_not_cover": [ + "consumers other than the one the probe mounts", + "interactions the probe does not perform", + "whether the rendered output is equivalent \u2014 it counts, it does not compare" + ], + "nodes": [ + { + "id": "pre-target-open", + "type": "precondition", + "check": "target pull request state == open" + }, + { + "id": "pre-probe-placed", + "type": "precondition", + "check": "the probe resolves its imports at the destination it was copied to", + "why": "a probe whose imports climb past the repo root reports a clean run that measured nothing" + }, + { + "id": "baseline", + "type": "baseline", + "run": "probe at the merge base", + "expect": "emits the count line", + "on_fail": "probe-failed \u2014 no comparison exists" + }, + { + "id": "treatment", + "type": "treatment", + "run": "the same probe at head" + }, + { + "id": "prediction", + "type": "prediction", + "expect": "the counts differ in the direction the claim asserts", + "on_fail": "no delta \u2014 the mechanism is not doing what the claim says" + }, + { + "id": "capture", + "type": "capture", + "emit": [ + "both arms", + "the metric name, caller-stated", + "the probe's permalink" + ], + "why": "the runner reads a field out of a line the probe printed; it does not know what was counted" + } + ] +} \ No newline at end of file diff --git a/domains/pr-workflow/skills/lane-graphs/skill.md b/domains/pr-workflow/skills/lane-graphs/skill.md new file mode 100644 index 00000000..4760413f --- /dev/null +++ b/domains/pr-workflow/skills/lane-graphs/skill.md @@ -0,0 +1,86 @@ +--- +name: lane-graphs +description: Route a claim to a small executable graph that measures it, instead of running a procedure from prose. Each graph is a fixed sequence of preconditions, a capture step, and a prediction check — deterministic where determinism is cheap, so the judgement stays on which graph to run rather than on how to run it. Defines the router contract, including the two outcomes that make routing honest: no-match as a first-class result, and a mandatory unrouted read whose findings no graph was looking for. Use when a claim maps onto a known measurement kind, when a run needs to be repeatable by someone who did not design it, or when deciding whether a new lane deserves a graph at all. +maturity: experimental +--- + +# Lane graphs + +A prose lane says what to measure. A graph runs it. The difference is not rigour — it is that a +graph fails the same way every time, which is what makes a failure findable. + +The split this skill exists to hold: **routing is judgement, execution is not.** Choosing which +measurement answers a claim requires reading the change. Computing a merge-base, checking a probe +loaded, verifying the mutation landed where it was aimed — none of that requires reading anything, +and all of it has been got wrong by hand. + +## What a graph is + +Five nodes, in this order, and a graph missing any of them is a script rather than a graph: + +| node | job | fails when | +|---|---|---| +| **preconditions** | the range is the pull request's, the target is open, the tree is at a pinned SHA | any is false — before anything expensive runs | +| **baseline** | run the measurement unmutated | it does not produce the expected shape of output | +| **treatment** | apply exactly one change, read back what was applied | the applied thing differs from the requested thing | +| **prediction** | the named observation the treatment should produce | it is absent, or something else moved instead | +| **capture** | the tool writes the artifact; the run URL is the citation | nothing was written | + +Two of these exist because they were missing and something published anyway: reading back the +treatment, and naming the prediction. A run whose treatment silently changed shape, or that went +red somewhere other than where it aimed, satisfied every other node. + +## The router contract + +The router takes a claim and returns a graph, **or nothing**. Both are results. + +**1. It may return no match, and no-match is a first-class outcome.** A router over enough lanes +always finds a best score. That score is meaningless if nothing was actually a fit, and a graph +run on a claim it does not measure produces a clean green that answers a question nobody asked. +The router must be able to say the claim is not of a kind it measures, and that must be reportable +rather than a fallback into the nearest lane. + +**2. An unrouted read runs regardless of what matched.** Before or alongside the graph, read the +mechanism and write down what could break, without reference to the lane catalog. This is not +redundancy. The catalog is a list of questions someone already thought of, so anything outside it +is invisible to routing by construction — and that is exactly where the findings worth having +tend to sit. Its output feeds the unclaimed bucket. + +**3. The route is part of the artifact.** Publish which graph ran and why it was chosen. A reader +who disagrees with the routing can then say so, which is impossible if only the result is shown. + +**4. A graph reports what it does not cover.** Each states its own blind spot in its output — the +paths it does not traverse, the properties it does not assert. A green result that does not say +what it declined to measure reads as broader than it is. + +## When a lane should not get a graph + +Graphs cost maintenance and drift silently: a defect can sit in one for weeks while every run it +produces looks correct. So the bar is not "could this be automated". + +Build a graph when the measurement is **run more than a handful of times**, its **preconditions +have been got wrong by hand**, and its **result is checkable without interpretation**. Leave a +lane in prose when the interesting part is the reading — a policy diff, a retention review, a +supply-chain disposition — because there the procedure is trivial and the judgement is everything. + +## Two failure modes with the same shape + +**A deterministic graph makes wrongness consistent.** A flaky procedure fails visibly; a +deterministic one fails identically forever and reads as evidence. The published examples in this +repository's own trial runs bear this out — large node counts, reproducible artifacts, checksums, +and assertions that keyed on a per-case sentinel rather than on absence, so a regression leaking a +different value would have passed every run. + +**A large green total stops being read.** Fifty-three of fifty-three, two hundred and one of two +hundred and one. Report the partition instead: which nodes carry the claim, which are navigation, +and which assertions would notice a regression. A total is a summary of effort, not of coverage. + +## Related + +- [`evidence`](../evidence/skill.md) — owns the lane catalog these graphs execute, and the runners + that are their capture step +- [`falsifiers-first`](../falsifiers-first/skill.md) — supplies the unrouted read, and receives its + output as the unclaimed bucket +- [`instrument-check`](../instrument-check/skill.md) — the controls a graph needs before its + results count +- [`coverage-partition`](../coverage-partition/skill.md) — what to report instead of a total From 42270a0ba8a76bc4dd0889fbc7d68c0cf7f49a85 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 17:04:30 -0400 Subject: [PATCH 115/135] Name the installed command in `red-on-base`'s description MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The description advertised `/red-on-base` while the installer emits `mms-red-on-base`, so a reader who typed what the description told them would find nothing. Caught only with #99 merged alongside this branch — the check lives there, the skill lives here, and neither branch can fail on its own. --- domains/testing/skills/red-on-base/skill.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/domains/testing/skills/red-on-base/skill.md b/domains/testing/skills/red-on-base/skill.md index 0c7066af..a5671941 100644 --- a/domains/testing/skills/red-on-base/skill.md +++ b/domains/testing/skills/red-on-base/skill.md @@ -1,6 +1,6 @@ --- name: red-on-base -description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /red-on-base, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by evidence as the engine behind its falsifying regression test evidence category. Named for its own discipline rather than for that category, so it is not mistaken for `falsify-probe.sh`, which shares the two-arm shape and answers a different question. +description: Produce the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. The falsifier is a test that fails on base for the wrong reason (import error, missing fixture, unrelated breakage) rather than by asserting the bug; that failure looks identical in an exit code and proves nothing. Also covers the diagnostic case: if no test can be written that fails on base, the fix's connection to the reported bug is the thing in doubt. Triggers on /mms-red-on-base, or when asked to write a regression test for a fix, prove a bug fix works, show a test failing before and passing after, or check whether a PR's test actually covers its claim. Callable by evidence as the engine behind its falsifying regression test evidence category. Named for its own discipline rather than for that category, so it is not mistaken for `falsify-probe.sh`, which shares the two-arm shape and answers a different question. maturity: experimental --- From 0dee42cd2b7006838df9a0c4d7b9b32894d0c1bf Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Mon, 3 Aug 2026 17:07:38 -0400 Subject: [PATCH 116/135] Bring the runner fixes back from the branch CI was actually running MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CI workflow sources runners by `skills_ref`, and every run this week pointed at a fork branch. Four fixes were made there and never reached this one: the mutation now travels through `ENVIRON` instead of an escape-processed `awk -v` assignment, the artifact reports the line read back off disk beside the line requested, `--expect-fail` turns a red arm in the wrong place into its own outcome, and `--metric` lets the caller name what a probe counted. So the defect this PR's own description cites as the reason instruments must report their effect was, until now, still live in the instrument this PR ships. The probe comes across too, with the import fix that made it resolve at the destination the workflow copies it to. Two copies of the same scripts on two branches, edited in both directions — `attest-gate.sh` had a check the fork lacked, so it stays as it is here. --- .../probes/metametrics-context.test.tsx | 68 +++++++++++++++++++ .../skills/evidence/scripts/falsify-probe.sh | 51 ++++++++++++-- .../skills/evidence/scripts/render-count.sh | 33 ++++++--- 3 files changed, 136 insertions(+), 16 deletions(-) create mode 100644 domains/pr-workflow/skills/evidence/probes/metametrics-context.test.tsx diff --git a/domains/pr-workflow/skills/evidence/probes/metametrics-context.test.tsx b/domains/pr-workflow/skills/evidence/probes/metametrics-context.test.tsx new file mode 100644 index 00000000..ae61df61 --- /dev/null +++ b/domains/pr-workflow/skills/evidence/probes/metametrics-context.test.tsx @@ -0,0 +1,68 @@ +// Probe — MetaMetrics context value identity. +// +// PLACEMENT: copy to `ui/contexts/__render_probe__.test.tsx` in a metamask-extension tree. +// The imports below are relative to `ui/contexts/`, so a different destination resolves +// nothing and the suite fails to run with "Cannot find module" — which is a failed probe, +// not a measurement. `probe_dest` in the evidence workflow must match this path. +// +// The claim under test is about breadth: "all N consumers avoid unnecessary re-renders". +// `useContext` re-renders a consumer when the value's IDENTITY changes, and that is not a +// per-consumer property — so one distinct value across N parent renders means every +// consumer is spared, and N distinct values means none is. Counting distinct values is +// therefore the measurement the claim actually rests on; counting one consumer's renders +// would only ever describe that consumer. +// +// Resolves against both `metametrics.js` and `metametrics.tsx`, so the same file measures a +// base commit and a head commit that renamed it — the comparison is the point. +import React, { useContext, useRef, useState } from 'react'; +import { act } from '@testing-library/react'; +import configureStore from '../store/store'; +import { renderWithProvider } from '../../test/lib/render-helpers-navigate'; +import mockState from '../../test/data/mock-state.json'; +import { MetaMetricsContext, MetaMetricsProvider } from './metametrics'; + +let consumerRenders = 0; +let distinctValues = 0; +let bump: (() => void) | undefined; + +function Consumer() { + const value = useContext(MetaMetricsContext); + const last = useRef<unknown>(null); + if (last.current !== value) { + last.current = value; + distinctValues += 1; + } + consumerRenders += 1; + return null; +} + +function Parent() { + const [, setN] = useState(0); + bump = () => setN((n) => n + 1); + return ( + <MetaMetricsProvider> + <Consumer /> + </MetaMetricsProvider> + ); +} + +describe('MetaMetrics context value identity', () => { + it('counts distinct context values across parent re-renders', () => { + const PARENT_RENDERS = 5; + consumerRenders = 0; + distinctValues = 0; + + renderWithProvider(<Parent />, configureStore(mockState)); + for (let i = 0; i < PARENT_RENDERS; i++) { + act(() => { + bump?.(); + }); + } + + // eslint-disable-next-line no-console + console.log( + `RENDER_COUNT consumer=${distinctValues} parentRenders=${PARENT_RENDERS + 1} consumerRenders=${consumerRenders}`, + ); + expect(consumerRenders).toBeGreaterThan(0); + }); +}); diff --git a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh index d7e8753e..46fed7f0 100755 --- a/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh +++ b/domains/pr-workflow/skills/evidence/scripts/falsify-probe.sh @@ -24,6 +24,7 @@ # # Usage: # falsify-probe.sh --test <path> --source <path> --line <n> --replace <text> +# [--expect-fail <test name substring>]... # [--label <slug>] [--out <dir>] [--runner "<cmd>"] # # Example: @@ -50,6 +51,12 @@ RUNNER="yarn jest" OUT_DIR="evidence-artifacts" LABEL="" TEST="" SOURCE="" LINE="" REPLACE="" +# Which test names the caller predicts will fail. Caller-stated, like every other judgement +# word here, and checked rather than trusted: the guards ask whether arm B failed and whether +# it ran the same tests, never whether the RIGHT ones failed. A mutation silently corrupted +# before it reached the file failed a different case than it aimed at, ran the full suite, and +# was reported `falsifying` — a green verdict for a mechanism the run never touched. +EXPECT="" die() { printf 'falsify-probe: %s\n' "$1" >&2; exit 3; } @@ -62,6 +69,7 @@ while [ $# -gt 0 ]; do --label) LABEL="${2:-}"; shift 2 ;; --out) OUT_DIR="${2:-}"; shift 2 ;; --runner) RUNNER="${2:-}"; shift 2 ;; + --expect-fail) EXPECT="$EXPECT${EXPECT:+\n}${2:-}"; shift 2 ;; -h|--help) sed -n '2,32p' "$0"; exit 0 ;; *) die "unknown argument: $1" ;; esac @@ -105,9 +113,21 @@ if [ "$ARM_A" != "passed" ]; then VERDICT="baseline-already-failing"; CODE=2; ARM_B="not-run" : > "$STAMP-armB.log" else - # Mutate exactly one line. `.bak` form keeps this portable across GNU/BSD sed. - awk -v n="$LINE" -v r="$REPLACE" 'NR==n{print r; next}{print}' "$SOURCE" > "$SOURCE.tmp" \ - && mv "$SOURCE.tmp" "$SOURCE" || die "mutation failed" + # Mutate exactly one line. The replacement travels through the environment, not + # through `awk -v`: awk runs escape processing on a `-v` assignment, so `[\s\S]` + # arrived as `[sS]` and the mutation written to the file was not the mutation asked + # for — it narrowed the regex it was meant to widen, failed a different test, and the + # runner reported `falsifying` for a mechanism it never touched. `ENVIRON` does no + # such processing. + MUTANT_LINE="$REPLACE" awk -v n="$LINE" 'NR==n{print ENVIRON["MUTANT_LINE"]; next}{print}' \ + "$SOURCE" > "$SOURCE.tmp" && mv "$SOURCE.tmp" "$SOURCE" || die "mutation failed" + # What the artifact reports as the mutation is read back off disk, never taken from the + # argument. The two differed once and nothing in the output said so. + APPLIED_LINE="$(sed -n "${LINE}p" "$SOURCE")" + if [ "$APPLIED_LINE" != "$REPLACE" ]; then + printf 'falsify-probe: the line written differs from --replace\n asked: %s\n written: %s\n' \ + "$REPLACE" "$APPLIED_LINE" >&2 + fi ARM_B="$(run_arm "$STAMP-armB.log")" restore; trap - EXIT INT TERM A_TOTAL="$(total_tests "$STAMP-armA.log")"; A_TOTAL="${A_TOTAL:-0}" @@ -125,6 +145,23 @@ else fi fi +# Runs last, on the verdict the guards already reached: a mutation can only fail the wrong +# case if it failed something, so this narrows `falsifying` and never widens it. +MISSED="" +if [ "$CODE" -eq 0 ] && [ -n "$EXPECT" ]; then + FAILED_SO_FAR="$(grep -E "^[[:space:]]+.[^\u203a]*\u203a" "$STAMP-armB.log" 2>/dev/null)" + printf '%b\n' "$EXPECT" | while IFS= read -r want; do + [ -n "$want" ] || continue + printf '%s' "$FAILED_SO_FAR" | grep -qF "$want" || printf '%s\n' "$want" + done > "$STAMP.missed" + MISSED="$(tr '\n' '|' < "$STAMP.missed" | sed 's/|$//;s/|/, /g')" + rm -f "$STAMP.missed" + if [ -n "$MISSED" ]; then + VERDICT="falsified a different case — predicted failure absent: $MISSED" + CODE=2 + fi +fi + summarise() { grep -E '^(Tests|Test Suites):' "$1" 2>/dev/null | tr '\n' ' ' | sed 's/ */ /g'; } A_SUM="$(summarise "$STAMP-armA.log")" B_SUM="$(summarise "$STAMP-armB.log")" @@ -137,7 +174,9 @@ cat > "$STAMP.json" <<JSON "test": "$TEST", "mutation": { "source": "$SOURCE", "line": $LINE, "from": $(printf '%s' "$ORIGINAL_LINE" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))'), - "to": $(printf '%s' "$REPLACE" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))') }, + "to": $(printf '%s' "${APPLIED_LINE-$REPLACE}" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))'), + "to_requested": $(printf '%s' "$REPLACE" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))') }, + "predicted_failures_absent": $(printf '%s' "$MISSED" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))'), "armA": { "result": "$ARM_A", "summary": "$A_SUM", "log": "$STAMP-armA.log" }, "armB": { "result": "$ARM_B", "summary": "$B_SUM", "log": "$STAMP-armB.log" }, "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, "node": "$NODE_V", "yarn_lock_sha256_16": "$LOCK_SHA" } @@ -170,8 +209,6 @@ printf 'falsify-probe: %s (exit %s)\n %s\n %s\n' "$VERDICT" "$CODE" "$STAMP.js # orchestrator reads them and writes ONE open question about THIS diff. printf 'limits: one line of one file was mutated. Says nothing about other paths into the same mechanism, whether it is reachable in production, or whether the guarded behaviour is -correct. This probe proves the suite notices one mutated line, which is not the same as the -base-against-branch proof that a test is connected to the reported bug -- see the red-on-base -skill for that experiment.%s\n' \ +correct. A falsifying test proves the test has power, not that the fix is right.%s\n' \ "$([ "$VERDICT" = vacuous ] && printf '\n vacuous: the mechanism is unguarded by this suite — what else depends on it?')" >&2 exit "$CODE" diff --git a/domains/pr-workflow/skills/evidence/scripts/render-count.sh b/domains/pr-workflow/skills/evidence/scripts/render-count.sh index 05de89a1..ab3dcca2 100755 --- a/domains/pr-workflow/skills/evidence/scripts/render-count.sh +++ b/domains/pr-workflow/skills/evidence/scripts/render-count.sh @@ -15,7 +15,7 @@ # # Usage: # render-count.sh --probe <probe.test.tsx> [--defeat <file> --defeat-line <n> --defeat-with <text>] -# [--arm-b <label>] [--label <slug>] [--out <dir>] +# [--arm-b <label>] [--metric <words>] [--label <slug>] [--out <dir>] # # Arm B is "the memo defeated" by default, which is the shape when a PR ADDS # memoisation. When a PR is the one under suspicion the arms invert — arm B applies @@ -29,6 +29,10 @@ # # RENDER_COUNT consumer=<n> parentRenders=<m> # +# `consumer=` is the field name, not a promise about what was counted — a probe for a claim +# about context value identity counts distinct values there, and calling that "consumer +# renders" prints a different quantity than the one measured. Pass --metric to name it. +# # 0 measured counts captured for both arms (or arm A alone if no --defeat) # 1 no delta arm B identical to arm A — the memo is not doing what is claimed # 2 probe did not emit RENDER_COUNT @@ -49,6 +53,13 @@ capture_provenance() { OUT_DIR="evidence-artifacts"; LABEL=""; PROBE=""; DEFEAT=""; DEFEAT_LINE=""; DEFEAT_WITH="" ARM_B="memo defeated" +# What the probe's `consumer=` field counts, in the caller's words. Caller-stated for the +# same reason the verdict is: this script reads a number out of a line the probe printed and +# has no way to know what the probe counted. A probe that counts distinct context values — +# the right measurement when the claim is about value identity rather than one component's +# renders — was published under the fixed heading "consumer renders", which is a different +# quantity and was wrong. A wrong label on a correct number is still a wrong number. +METRIC="consumer renders" die() { printf 'render-count: %s\n' "$1" >&2; exit 3; } while [ $# -gt 0 ]; do @@ -58,6 +69,7 @@ while [ $# -gt 0 ]; do --defeat-line) DEFEAT_LINE="${2:-}"; shift 2 ;; --defeat-with) DEFEAT_WITH="${2:-}"; shift 2 ;; --arm-b) ARM_B="${2:-}"; shift 2 ;; + --metric) METRIC="${2:-}"; shift 2 ;; --label) LABEL="${2:-}"; shift 2 ;; --out) OUT_DIR="${2:-}"; shift 2 ;; -h|--help) sed -n '2,30p' "$0"; exit 0 ;; @@ -85,7 +97,9 @@ if [ -n "$DEFEAT" ] && [ -n "$DEFEAT_LINE" ]; then BACKUP="$(mktemp)"; cp "$DEFEAT" "$BACKUP" restore() { cp "$BACKUP" "$DEFEAT"; rm -f "$BACKUP"; } trap restore EXIT INT TERM - awk -v n="$DEFEAT_LINE" -v r="$DEFEAT_WITH" 'NR==n{print r; next}{print}' "$DEFEAT" > "$DEFEAT.tmp" && mv "$DEFEAT.tmp" "$DEFEAT" + # Through the environment, not `awk -v`: a `-v` assignment is escape-processed, so a + # replacement containing a backslash reaches the file altered. See falsify-probe.sh. + DEFEAT_LINE_TEXT="$DEFEAT_WITH" awk -v n="$DEFEAT_LINE" 'NR==n{print ENVIRON["DEFEAT_LINE_TEXT"]; next}{print}' "$DEFEAT" > "$DEFEAT.tmp" && mv "$DEFEAT.tmp" "$DEFEAT" yarn jest "$PROBE" > "$STAMP-armB.log" 2>&1 B_LINE="$(counts_from "$STAMP-armB.log")" B="$(consumer_of "$B_LINE")" @@ -95,8 +109,8 @@ else fi if [ -n "$B" ] && [ "$B" = "$A" ]; then VERDICT="no delta — arm B changed nothing measurable"; CODE=1 -elif [ -n "$B" ]; then VERDICT="delta measured: $A → $B renders with $ARM_B"; CODE=0 -else VERDICT="baseline only: $A consumer renders"; CODE=0; fi +elif [ -n "$B" ]; then VERDICT="delta measured: $A → $B $METRIC with $ARM_B"; CODE=0 +else VERDICT="baseline only: $A $METRIC"; CODE=0; fi HEAD_SHA="$(git rev-parse HEAD 2>/dev/null || echo unknown)" DIRTY="$(git status --porcelain 2>/dev/null | grep -v '^??' | wc -l | tr -d ' ')" @@ -104,17 +118,18 @@ NODE_V="$(node -v 2>/dev/null || echo unknown)" cat > "$STAMP.json" <<JSON { "probe": "$PROBE", "verdict": "$VERDICT", "exit": $CODE, - "consumer_renders": { "armA": ${A:-null}, "armB": ${B:-null} }, + "metric": "$METRIC", + "counts": { "armA": ${A:-null}, "armB": ${B:-null} }, "env": { "head": "$HEAD_SHA", "tracked_changes": $DIRTY, "node": "$NODE_V" }, "logs": ["$STAMP-armA.log", "$STAMP-armB.log"] } JSON { - echo "### Consumer render count" + echo "### Render probe — $METRIC" echo echo "**Verdict:** $VERDICT" echo - echo "| Arm | Change | consumer renders |" + echo "| Arm | Change | $METRIC |" echo "|---|---|---|" echo "| A — as committed | none | ${A:-?} |" [ -n "$B" ] && echo "| B — $ARM_B | \`$DEFEAT:$DEFEAT_LINE\` | $B |" @@ -125,8 +140,8 @@ JSON [ -n "$B_LINE" ] && { echo "\$ yarn jest $PROBE # $ARM_B"; echo "$B_LINE"; } echo '```' echo - echo "This counts renders of one named consumer across a defined interaction. It is not a count" - echo "of consumers, and a larger consumer count does not imply a larger effect." + echo "The number is \`$METRIC\` as printed by \`$PROBE\` — that file is what defines the" + echo "quantity. It is one probe under one interaction, not a property of the application." echo echo "<sub>Produced by \`render-count.sh\`; the arm-B edit is reverted after the run. head \`$HEAD_SHA\` · $DIRTY tracked changes · node \`$NODE_V\`. $(capture_provenance)</sub>" } > "$STAMP.md" From cae0c082c0f61ca44d5825f1121c9b5259b52572 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 02:58:46 -0400 Subject: [PATCH 117/135] Check that a figure in the prose traces to an exhibit Check 9 is called "verdict matches artifact" and compares verdict words. Nothing compared the numbers, and prose drifting from the exhibit beside it is the most common way one of these goes wrong. Found by building a demonstration artifact to test this gate: the prose read "0 errors over 48 skills" directly above an exhibit reading "47 skill(s) checked", and named a warning class with zero instances in the output it was describing. Every other check passed. Two independent readers caught both, which is the argument for moving it into the layer that always runs rather than the one that costs money and sometimes never reports. Narrow on purpose, because a noisy check is an ignored one. Two-plus digits only, and only those absent from every fenced block; whole URLs, issue refs, versions, dates, SHAs, file:line citations, hyphenated identifiers and regex quantifiers are excluded as references rather than measurements. Each exclusion was added after a control run flagged something that was not a figure. Across eight real artifacts it flags one, correctly: a verdict line quoting an author's "730 tests" beside an exhibit measuring 731, where nothing distinguishes the cited figure from the measured one. --- .../skills/evidence/scripts/attest-gate.sh | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 3f058d5f..fd7d8935 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -191,6 +191,34 @@ else esac fi +# 13 — a number in the prose that appears in no exhibit. Check 9 compares verdict WORDS; +# nothing compared the figures. Measured on a demonstration artifact built to test this +# gate: the prose said "0 errors over 48 skills" directly above an exhibit reading +# "47 skill(s) checked", and named a warning class with zero instances in the output it +# was describing. Both survived every other check. Prose drifts from the exhibit it sits +# beside, and it is the most common way one of these goes wrong. +# +# Deliberately narrow, because a noisy check is an ignored check: integers of two or more +# digits only, and only those absent from every fenced block. Excluded as references +# rather than measurements — whole URLs, issue refs, version strings, dates, SHAs, +# file:line citations, hyphenated identifiers like P-256, and regex quantifiers. Every +# one of those was added after a control run flagged something that was not a figure. +echo +NUM_ORPHANS="$( + awk '/^```/{f=!f; next} f{print}' "$FILE" > "$FILE.exh" 2>/dev/null + awk '/^```/{f=!f; next} !f{print}' "$FILE" \ + | sed -E 's#https?://[^ )]*##g' \ + | sed -E 's/#[0-9]+//g; s/\bv?[0-9]+\.[0-9]+(\.[0-9]+)?\b//g; s/\b[0-9]{4}-[0-9]{2}-[0-9]{2}\b//g; s/\b[0-9a-f]{7,}\b//g; s/:[0-9]+\b//g; s/[A-Za-z]+-[0-9]+//g; s/\{[0-9,]+\}//g' \ + | grep -oE '\b[0-9]{2,}\b' | sort -u \ + | while read -r n; do grep -qF "$n" "$FILE.exh" || printf '%s ' "$n"; done + rm -f "$FILE.exh" +)" +if [ -n "$NUM_ORPHANS" ]; then + fail "13 figures trace to an exhibit" "these appear in the prose and in no exhibit: $NUM_ORPHANS — either they came from somewhere the reader cannot see, or they disagree with what is shown" +else + pass "13 figures trace to an exhibit" +fi + echo if [ "$FAILED" -eq 0 ]; then echo "attest-gate: phase 0 clean — proceed to /outframe ‖ /missing ‖ /press" From a5bfb9872065529a6399cc2838cc32f1c363e71d Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 03:15:29 -0400 Subject: [PATCH 118/135] Stop check 12 passing when it could not run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With `gh` absent the check printed UNVERIFIED and exited 0, so on a machine without it — running locally — the destination check announced that it had not run and the gate reported clean. A control that cannot run is indistinguishable from one that passed unless the exit code says otherwise. --- domains/pr-workflow/skills/evidence/scripts/attest-gate.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index fd7d8935..5fbed0ef 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -180,7 +180,10 @@ echo if [ -z "$TARGET" ]; then fail "12 destination is open" "no --target given, so nobody checked whether the pull request is still open. Pass --target owner/repo#N." elif ! command -v gh >/dev/null 2>&1; then - printf ' ???? %s\n %s\n' "12 destination is open" "gh not on PATH — the destination is UNVERIFIED, not passing. Check it by hand before publishing." + # Blocks rather than warns. An earlier version printed UNVERIFIED and exited 0, so on a + # machine without `gh` — which is to say, running locally — this check announced that it + # had not run and passed anyway. That is the shape it exists to catch, one level up. + fail "12 destination is open" "gh not on PATH, so the destination was not checked. Unverified is not passing: install gh, or confirm the target is open and re-run." else t_repo="${TARGET%%#*}"; t_num="${TARGET##*#}" t_state="$(gh api "repos/$t_repo/pulls/$t_num" --jq 'if .merged_at then "merged" else .state end' 2>/dev/null || echo unknown)" From a1ea24a681952793ec683115e5417a1f94fda3fa Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 05:34:03 -0400 Subject: [PATCH 119/135] Add false precision as a divergence shape and the `IsAny` probe that finds it A precise annotation fed `any` at every call site is reportable by neither `tsc` nor `no-explicit-any`, and an ambient `declare module` in the path re-mints the `any` as a confident `string`. Both arms of the probe verified against `metamask-extension`. Also fixes `avoid-any`'s frontmatter, which did not parse as YAML. --- domains/typescript/skills/avoid-any/skill.md | 22 +++++++- .../references/false-negatives.md | 52 ++++++++++++++++++- .../typescript/skills/tsc-blindspots/skill.md | 47 ++++++++++++++--- 3 files changed, 112 insertions(+), 9 deletions(-) diff --git a/domains/typescript/skills/avoid-any/skill.md b/domains/typescript/skills/avoid-any/skill.md index b9256e87..0fb55dcf 100644 --- a/domains/typescript/skills/avoid-any/skill.md +++ b/domains/typescript/skills/avoid-any/skill.md @@ -1,6 +1,13 @@ --- name: avoid-any -description: Handle `any` correctly — it is not a type but a directive that disables type checking. Substitute by position (assignee → `unknown`, assigned → `never`). Two narrow exceptions: a generic constraint, and a callback parameter caught in a bivariant position between two fixed, irresolvable function-type constraints — both declared with an inline eslint-disable. +description: >- + Handle `any` correctly — it is not a type but a directive that disables type + checking. Substitute by position (assignee → `unknown`, assigned → `never`). + Two narrow exceptions: a generic constraint, and a callback parameter caught + in a bivariant position between two fixed, irresolvable function-type + constraints — both declared with an inline eslint-disable. Also covers the + `any` that is never written down: a precise signature fed `any` at every call + site, which no lint rule and no compiler check can report. maturity: experimental --- @@ -72,3 +79,16 @@ Like the generic-constraint case, this `any` is **not infectious** — it is sco Canonical instance: a messenger `registerActionHandler` slot typed `(...args: any[]) => any` — strongly-typed handlers flow inward at registration, strongly-typed argument tuples outward at dispatch; `unknown[]` fails registration, `never[]` fails dispatch. It encodes rank-N polymorphism (`∀α. (α) => R`) that TypeScript cannot express directly. When `any` still seems unavoidable, prefer the narrower, greppable escape hatches: `as unknown as` as a documented last resort, or `@ts-expect-error` with a TODO. Never reach for `any` to unblock feature work "to fix later." + +## Declared `any` beats absorbed `any` — and only one of them is countable + +The two exceptions above are both *declared*: the `any` is written down, an inline disable sits next to it, and a reviewer can find it with `grep`. The dangerous case is the one where **no `any` appears anywhere** and the value is `any` regardless: + +- 🚫 **Absorbed.** A precise annotation on a parameter or return that receives `any` at every call site — `hexValueIsEmpty(value: string | null | undefined)` fed ethers dynamic-method results. `no-explicit-any` never fires, because no `any` was written. `tsc` never fires, because `any` satisfies every annotation. The file reads as checked and none of it is. +- ✅ **Declared.** `): Promise<any>` with an inline disable and a linked issue, as `shared/lib/token-util.ts` does for the same ethers API. Nothing is safer at runtime, but the claim is now honest, greppable, and countable by CI. + +The absorbed form is strictly worse than the declared one, and it is what a JS→TS conversion produces by default: the writer annotates what the value *ought* to be, and `any` accepts the annotation without comment. + +**A conversion is where the boundary's type is chosen, so an absorbed `any` is a decision, not an inheritance.** With `checkJs` off the predecessor asserted nothing; the precise signature is new. "The `any` is pre-existing" is true of the library and false of the annotation next to it. + +Detect it with `IsAny<T>` at the call sites rather than by reading the signature — the probe, its controls, and the `declare module` composition that turns an `any` into a confident `string` are in the `tsc-blindspots` skill. The fix is to type or validate the value where it enters, so the precise annotations downstream are earned. diff --git a/domains/typescript/skills/tsc-blindspots/references/false-negatives.md b/domains/typescript/skills/tsc-blindspots/references/false-negatives.md index fc494022..ea093572 100644 --- a/domains/typescript/skills/tsc-blindspots/references/false-negatives.md +++ b/domains/typescript/skills/tsc-blindspots/references/false-negatives.md @@ -147,6 +147,28 @@ annotates it, so it propagates into the module's return type unflagged. with a derived provider type and `hexValueIsEmpty(value: string | null | undefined)` reads as a checked boundary while every value crossing it is unchecked. A migration that adds those signatures is what creates the appearance. +- **To ask "is this specific value `any`", use `IsAny` rather than a nonsense + assignment.** The exit probe above traces a module's public surface; this answers + the question at any single site, and its verdict is a compile error rather than an + absence of one: + + ```ts + type IsAny<T> = 0 extends 1 & T ? true : false; + + const resolverAddress = await registryContract.resolver(hash); + const a1: IsAny<typeof resolverAddress> = true; // silent ⇒ it IS any + const known = 'x' as string; + const a3: IsAny<typeof known> = true; // MUST error ⇒ probe discriminates + ``` + + `0 extends 1 & T` holds only for `any`, because `1 & any` is `any` and every type + extends `any`. **Read the polarity carefully: silence is the finding here**, which + is the reverse of every other probe in this file — so the known-`string` control + is what separates "this value is `any`" from "the probe never ran." + + Verified against `metamask-extension` with the repo's own `tsconfig.json`: four + arms — an ethers dynamic method and an explicit `as any` both silent, a known + `string` and a laundered `string` (below) both `TS2322`. ### 8. Ambient `declare module` is an unverified assertion @@ -177,6 +199,32 @@ later ships. `esModuleInterop` is on, and is a defect if it is not. Prefer `@types/*` or an upstream PR over hand-writing. +#### 7 + 8 compose: a declaration launders `any` into a confident type + +The two blind spots above are usually audited apart, and the defect lives in their +composition. §7 says an `any` propagates; §8 says a hand-written declaration is +believed. Put them in sequence and the propagation **stops** — replaced by a type +nobody checked, which every reader downstream then trusts: + +| hop | site | resulting type | who asserted it | +|---|---|---|---| +| 1 | `await resolverContract.contenthash(hash)` | `any` | ethers `readonly [key: string]: ContractFunction \| any` | +| 2 | `contentHash.getCodec(rawContentHash)` | **`string`** | a hand-written `declare module` | +| 3 | `return { type, hash: decoded }` | `{ type: string; hash: any }` | inferred from hop 2 | +| 4 | `` `https://${hash}.${type.slice(0, 4)}.${gateway}` `` | `.slice` on a `string` | inferred from hop 3 | + +Hop 2 is declared `(contentHash: string) => string` and **called with an `any`** — +so it neither rejects its input nor earns its output. By hop 4 the value is a URL +segment, and the only claim it was ever a string is a line a human wrote. + +- **Audit rule:** for every `declare module`, list its call sites and run `IsAny` on + each **argument**. A parameter declared `string` and passed `any` is the laundering + point, and it is where the annotation or the validator belongs — not at hop 4, + where the value already looks trustworthy. +- Verified in the demonstration run above: hop 1 silent under `IsAny` (it is `any`), + hop 2 `TS2322` (it is `string`), with the declaration block supplied locally so the + only variable between the two arms is the `declare module`. + ### 9. External data is asserted, not validated ```ts @@ -293,7 +341,9 @@ Don't run all ten as a checklist. Pick by what the diff touches: - **New indexing / destructuring** → 1, 2 - **New message, event, or callback types** → 3, 5, 6 -- **New `declare module`, new dependency, `@types` change** → 8 +- **New `declare module`, new dependency, `@types` change** → 8, then **7 + 8** on + its call sites — a declaration is audited against the package by default and + against its *arguments* almost never - **Anything reading persisted state, storage, or an RPC response** → 7, 9 - **A JS→TS conversion** → 10, plus the restated-type class in the main skill - **Any PR whose safety argument is "CI is green"** → section C, first diff --git a/domains/typescript/skills/tsc-blindspots/skill.md b/domains/typescript/skills/tsc-blindspots/skill.md index 8cdab2cc..e8006f05 100644 --- a/domains/typescript/skills/tsc-blindspots/skill.md +++ b/domains/typescript/skills/tsc-blindspots/skill.md @@ -7,9 +7,11 @@ description: >- substituting the derived type at a fixed commit and diffing `tsc` output; and (2) the standing blind spots in the language and config — unchecked array/record indexing, bivariant method parameters, covariant arrays, `any` absorption at - untyped boundaries, ambient `declare module` assertions, excess-property checks - that only fire on fresh literals, and external data asserted rather than - validated. Also audits typing edits that quietly change runtime behavior: + untyped boundaries, precise signatures fed `any` at every call site, ambient + `declare module` assertions that launder an `any` into a confident type, + excess-property checks that only fire on fresh literals, and external data + asserted rather than validated. Also audits typing edits that quietly change + runtime behavior: stripped `| undefined`, deleted default parameters, literals swapped for runtime enum lookups, calls made optional so a throw becomes a silent no-op. Use when reviewing a JS→TS migration, a PR that hand-writes types for values that already @@ -168,11 +170,34 @@ Give each claim a verdict, and say which ones the probes **cleared**. A substitution sweep that only ever confirms is indistinguishable from one that never isolated anything. -## The four divergence shapes +### Severity: "pre-existing" is about the upstream `any`, not about the annotation -Four shapes to check for. All five divergences in the worked example were one of -these, which is a small sample — treat the list as a starting checklist, not a -partition: +A conversion PR invites two reflexes that both understate a false-precision +finding, and the underlying `any` is what makes each of them sound reasonable: + +- **"It's pre-existing."** Split the claim. The library's `any` is genuinely + inherited — ethers has returned `any` from dynamic contract methods since long + before this diff. The *annotation beside it* was written here: check `git log + --diff-filter=A -- <path>` and, for a `declare module`, whether the block itself + is added by this PR. If the file is new, nothing in it is pre-existing, because + with `checkJs` off the predecessor asserted nothing at all. A conversion is the + moment the boundary's type is **chosen**. +- **"It's a nit."** A nit is a finding that is minor *in itself*. A value crossing + a boundary unchecked, under a signature that says it was checked, is a type-safety + defect — the class this whole skill exists to surface. Scope and severity are + separate axes: a substantive finding a PR need not fix is still substantive, and + saying so costs one sentence. + +The remedy follows from which one it is. Annotating the hole and filing a TODO is +right for the inherited `any`; it is not a fix for false precision, because the +misleading signature stays exactly as it was. Type or validate the value where it +enters, so the precise annotations downstream are earned. + +## The five divergence shapes + +Five shapes to check for. Treat the list as a starting checklist, not a partition — +the first four each account for at least one divergence in the worked example, and +the fifth was found on a later pass over the same PR: 1. **Widening** — `string` for a `Hex`/template-literal type, `string` for an enum, `number | undefined` for `number`. Admits values the real type rejects; worst @@ -183,6 +208,14 @@ partition: now need every future change. 4. **Placeholder** — `Record<string, unknown>`, `any`, or `unknown` standing in for a shape that is known. Pushes a cast to every use site. +5. **False precision** — the inverse of a placeholder: the annotation is *narrower* + than what actually arrives. `hexValueIsEmpty(value: string | null | undefined)` + on a parameter fed `any` at every call site. `tsc` cannot report it, because + `any` satisfies every annotation, and `no-explicit-any` cannot either, because + no `any` was written. The narrower the type, the more confident the file reads + and the less any of it is checked. Find it with `IsAny` at the call sites, not + by reading the signature — see + [references/false-negatives.md](references/false-negatives.md) §7. ## Escape hatches are the tell From 7da6ee12d876698dc1b499825bf7378de330c075 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 05:40:02 -0400 Subject: [PATCH 120/135] Point references at the renamed `lavamoat-policy` skill Renamed on the security-domain branch; installs as `mms-lavamoat-policy`. --- .../skills/evidence/references/evidence-catalog.md | 2 +- .../skills/evidence/references/evidence-publishing.md | 2 +- domains/pr-workflow/skills/evidence/skill.md | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md index d7f82692..103ab9b3 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md @@ -166,7 +166,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **Proves:** a module moved to the intended (lazy) chunk and no longer ships on the critical path. Requires the webpack build. Mirrors AEP `perf-chunks`. ## D3. LavaMoat policy / supply-chain capability diff - - **Engine: the `supply-chain-audit` skill** (umbrella — lockfile/manifest diff, advisories, Socket Security, install scripts), which delegates capability grants to **`lavamoat-policy-diligence`** (per-grant call-site justification). Delegate the dependency change to the umbrella; it returns a disposition per lane. evidence keeps **supply-chain capability diff** as the evidence category and packages the output. Note the lanes are independent: a clean policy diff does not mean a safe dependency, and a known CVE never appears as a new grant. + - **Engine: the `supply-chain-audit` skill** (umbrella — lockfile/manifest diff, advisories, Socket Security, install scripts), which delegates capability grants to **`lavamoat-policy`** (per-grant call-site justification). Delegate the dependency change to the umbrella; it returns a disposition per lane. evidence keeps **supply-chain capability diff** as the evidence category and packages the output. Note the lanes are independent: a clean policy diff does not mean a safe dependency, and a known CVE never appears as a new grant. - **Proves:** a dependency change (bump/add/lockfile) grants **no *unjustified* new capability** — the supply-chain-risk lane. Note the bar: for a bump the policy *will* change, so "empty diff" is the WRONG test; the right test is **every new grant is justified by the dep's function**.. - **Capture:** **Prefer the CI-generated policy whenever one is available.** `@metamaskbot update-policies` regenerates the policy files from a real run of the code and `validate-lavamoat-policies` fails the build on drift, so the committed policy on a bot-run PR *is* the authoritative artifact — diff that. Regenerating locally when a current CI policy exists only re-does a machine that is already trusted, and a local run's provenance is weaker (your node/OS/lockfile resolution, not CI's). **Local regen is the fallback**, for when the bot hasn't run yet, the branch is unpushed, or you need a variant CI didn't cover: `yarn webpack:lavamoat:policy:build` (`:mv2` / `:mv3` for variants) over `lavamoat/webpack/build/policy.json` (+ `policy-override.json`). Either way, `git diff` the policy across **all 8 variants** (mv{2,3}/{main,beta,flask,experimental}) — a grant can appear in one and not others. Then audit **grant-by-grant**: new **globals** (`fetch`, `importScripts`, `WebAssembly`) / **builtins** (`fs`, `child_process`) on a dep that shouldn't need them, new **packages** edges to powerful APIs, or an identifier substitution (`pkgC>name` replacing `pkgB>pkgA>name` = possible dep swap). Falsifier = a surprising grant ("I wonder what it's using this for"). Guide: lavamoat.github.io/guides/policy-diff/. `allowScripts` in `package.json` gates install scripts. diff --git a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md index 4316c52f..72ce8da4 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md @@ -287,7 +287,7 @@ needs a tracker it does not have. The marker pairs also collide — a re-run rep **So: choose the format from the evidence kind, not from this document's default.** The canonical `## 🧪 Validation Run` header applies when a run produced artifacts. An engine -skill that defines its own output contract (`lavamoat-policy-diligence`) publishes in that +skill that defines its own output contract (`lavamoat-policy`) publishes in that contract. `hooks/pr-evidence-gate.py` enforces the canonical literal only on bodies that *claim* validation/evidence framing — a diligence comment that renders no verdict does not trip it, which is the tell that the two are different artifacts rather than one with a diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 9ac32af0..4f3f0b77 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -446,7 +446,7 @@ the text, so the gate asks for a different medium rather than for better text. **The exception — plaintext where every claim is a citation.** The rule is about where verification routes, not about pixels. Line-level links are externally verifiable: the reader clicks and sees exactly what you saw. That is the *normal* case for the audit lanes — -`supply-chain-audit`, `lavamoat-policy-diligence`, `privacy-egress-diligence` — whose findings +`supply-chain-audit`, `lavamoat-policy`, `privacy-egress-diligence` — whose findings are facts about code that exists rather than results of running something. There an image would be worse: a screenshot of a policy diff is less checkable than a permalink to it. @@ -660,9 +660,9 @@ Where evidence sits in the PR lifecycle (see the public `pr-workflow` siblings): | B7 deterministic interleaving (concurrency / ordering) | `/race-condition-repro` | | C4 React render & selector proof | `/react-render-delta` | | C9 memory leak | `/memory-leak` | - | D supply-chain / dependency change | `/supply-chain-audit` → delegates capability grants to `/lavamoat-policy-diligence` | + | D supply-chain / dependency change | `/supply-chain-audit` → delegates capability grants to `/lavamoat-policy` | - **An engine that defines its own output contract publishes in it.** `lavamoat-policy-diligence` + **An engine that defines its own output contract publishes in it.** `lavamoat-policy` is the live case: read-level triage, no verdict, its own header and marker pair. Do not re-frame it as a Validation Run — see *One comment per evidence kind* in [references/evidence-publishing.md](references/evidence-publishing.md). From 1706f0f90d2123ccec96245df8ae7eb691787aa6 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 05:40:15 -0400 Subject: [PATCH 121/135] Point references at the renamed `lavamoat-policy` skill Renamed on the security-domain branch; installs as `mms-lavamoat-policy`. --- domains/security/skills/privacy-egress-diligence/skill.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/domains/security/skills/privacy-egress-diligence/skill.md b/domains/security/skills/privacy-egress-diligence/skill.md index 7dbb3b89..023651ef 100644 --- a/domains/security/skills/privacy-egress-diligence/skill.md +++ b/domains/security/skills/privacy-egress-diligence/skill.md @@ -20,7 +20,7 @@ description: >- inside ordinary feature PRs — onboarding, swaps, rewards, the analytics controller — with **no CODEOWNERS entry**, so no privacy reviewer is automatically tagged. -This skill reviews that egress surface the way `lavamoat-policy-diligence` reviews +This skill reviews that egress surface the way `lavamoat-policy` reviews capability grants: the diff is mechanical, the judgement is what each grant *means*. ## When to use @@ -117,7 +117,7 @@ Run both on a change that adds collection; either alone leaves half the question ## Related -- `lavamoat-policy-diligence` — same shape for capability grants; read it for the +- `lavamoat-policy` — same shape for capability grants; read it for the diff-is-mechanical-judgement-is-not pattern. - `analytics-instrumentation` — whether an event is correctly *identified* and *gated* (`isOptIn`, `metaMetricsId`). This skill is about whether its payload is *sendable*. From af1a58fdd2991927455b30a15d800da5600a1afe Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 05:39:54 -0400 Subject: [PATCH 122/135] Rename `lavamoat-policy-diligence` to `lavamoat-policy` and restore the breadth step MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Installs as `mms-lavamoat-policy`. A trim pass had deleted the step that asks how wide each grant is versus what the call actually uses, leaving only the reachability question. That step is what produces findings on grants that are reached and not stale — a bare `node:url` standing in for `fileURLToPath` and `pathToFileURL` clears every other check. Also corrects the stated mechanism. Policy is generated by static analysis (`lavamoat-tofu`, `@babel/parser`), not by observing a run, and the difference matters: a statically detected grant says nothing about whether our usage reaches it, which is where removal candidates come from. And scopes the tautology claim to the *existence* of a call site rather than its content — the policy records `"crypto": true` and discards what a reviewer needs, so recovering that is the job rather than something to skip. --- .../scripts/policy-audit.py | 0 .../skill.md | 86 +++++++++++++------ .../skills/supply-chain-audit/skill.md | 12 +-- 3 files changed, 68 insertions(+), 30 deletions(-) rename domains/security/skills/{lavamoat-policy-diligence => lavamoat-policy}/scripts/policy-audit.py (100%) rename domains/security/skills/{lavamoat-policy-diligence => lavamoat-policy}/skill.md (70%) diff --git a/domains/security/skills/lavamoat-policy-diligence/scripts/policy-audit.py b/domains/security/skills/lavamoat-policy/scripts/policy-audit.py similarity index 100% rename from domains/security/skills/lavamoat-policy-diligence/scripts/policy-audit.py rename to domains/security/skills/lavamoat-policy/scripts/policy-audit.py diff --git a/domains/security/skills/lavamoat-policy-diligence/skill.md b/domains/security/skills/lavamoat-policy/skill.md similarity index 70% rename from domains/security/skills/lavamoat-policy-diligence/skill.md rename to domains/security/skills/lavamoat-policy/skill.md index 4a86838d..9f4634ac 100644 --- a/domains/security/skills/lavamoat-policy-diligence/skill.md +++ b/domains/security/skills/lavamoat-policy/skill.md @@ -1,22 +1,31 @@ --- -name: lavamoat-policy-diligence -description: Triage a LavaMoat policy change for least privilege — which newly granted capabilities can be dropped without breaking anything. Detection is delegated to `@metamaskbot update-policies` plus CI, and because the policy is generated from a real run, every grant has a call site by construction, so "each addition is justified" is a tautology and not the deliverable. Instead read each grant's use at the installed version to find its gate — a config flag nobody sets, an API nobody calls, a branch our payloads never take, an error-only path — and sort into removable / removable-at-a-cost / load-bearing, with the removal test (drop it, rebuild, run e2e) proposed for the policy owners to run. Lead with removal candidates and anything the reading turned up that bears on security; never render an accept/reject verdict, that call is the reviewer's. Hand it over untagged while the workflow is in trial. Triggers on /lavamoat-policy-diligence, or when asked about a LavaMoat policy grant, policy.json diff, capability containment, scuttling, allowScripts, or why a package needs a global/builtin. The specialized engine behind `supply-chain-audit`'s capability-containment lane. +name: lavamoat-policy +description: Triage a LavaMoat policy change for least privilege — which newly granted capabilities can be dropped without breaking anything. Detection is delegated to `@metamaskbot update-policies` plus CI, and because the policy is generated by static analysis, every grant has a call site by construction, so reporting that one exists is a tautology and not the deliverable. `policy.json` records `"crypto": true` and nothing about what the code does with it or how much of the module it touches — recovering that discarded half is the job. Read each grant's use at the installed version to find its gate (a config flag nobody sets, an API nobody calls, a branch our payloads never take, an error-only path) and its breadth (a bare global standing in for one property is a finding even when reached), with the removal or narrowing test proposed for the policy owners to run. Lead with removal candidates and anything the reading turned up that bears on security; never render an accept/reject verdict, that call is the reviewer's. Hand it over untagged while the workflow is in trial. Triggers on /mms-lavamoat-policy, or when asked about a LavaMoat policy grant, policy.json diff, capability containment, scuttling, allowScripts, or why a package needs a global/builtin. The specialized engine behind `supply-chain-audit`'s capability-containment lane. maturity: experimental --- -# /lavamoat-policy-diligence +# /lavamoat-policy Detection is not the job. LavaMoat already tells you which capabilities a dependency change -grants: `@metamaskbot update-policies` regenerates the `policy.json` files from a real run of -the code, and CI's `validate-lavamoat-policies` fails the build if the committed policy drifts -from that regeneration. Re-deriving the diff by hand, or sorting the grants into -network/DOM/red-flag buckets, only re-does a machine that is already trusted. - -**Finding the call site is not the job either — that search cannot fail.** The policy is -*generated from a real run*, so every grant in it corresponds to something the bundled code -did. "Each addition has a call site, therefore each is justified" is a tautology dressed as an -audit; it will report 11 of 11 justified every time, and a check that cannot come back negative -carries no information. +grants: `@metamaskbot update-policies` regenerates the `policy.json` files, and CI's +`validate-lavamoat-policies` fails the build if the committed policy drifts from that +regeneration. Re-deriving the diff by hand, or sorting the grants into network/DOM/red-flag +buckets, only re-does a machine that is already trusted. + +**Reporting that a call site exists is not the job — that search cannot fail.** Policy is +generated by **static analysis**, not by observing a run: `lavamoat-tofu` parses each module with +`@babel/parser` and records global and builtin references (`inspectGlobals` / `inspectImports`, +[generatePolicy.js](https://github.com/LavaMoat/LavaMoat/blob/main/packages/core/src/generatePolicy.js)). +A grant means the identifier is textually present in a parsed module, so searching for it must +succeed. "Each addition has a call site, therefore each is justified" reports 11 of 11 every +time, and a check that cannot come back negative carries no information. + +**Finding the call site is the entire job — its existence is just the part not worth printing.** +`policy.json` records `"crypto": true` and stops. It does not record what the code does with the +capability, or how much of the module it touches, and that discarded half is what a reviewer +needs. Recovering it is what this skill is for. Static detection is also why removal candidates +are common: an identifier surviving into the bundle says nothing about whether our usage reaches +it — which a runtime trace would have. **The job is least privilege: which of these grants can be dropped without breaking anything?** A grant exists because an identifier appears in bundled source. That is *not* the @@ -31,7 +40,7 @@ a branch our usage never takes, is removable. So for each grant, ask what execut | **load-bearing** | our usage genuinely needs it → say so briefly and move on | The lead is the first two rows plus anything the reading turned up that bears on security. -Load-bearing grants still each get a row in the capability → call-site table (step 5) — they just +Load-bearing grants still each get a row in the capability → call-site table (step 6) — they just don't get paragraphs. > **Falsifier.** A grant you called load-bearing that a build with it removed still passes. @@ -87,12 +96,36 @@ droppable. Those observations are worth more than the grant inventory. Lead with the cleanest removal there is — and a capability behind one we *do* use is load-bearing, which is worth one line and no more. -4. **Cite it at a pinned tag, not a branch head.** A permalink to `…/blob/<tag>/<file>#Ln` is +4. **Record what each grant reaches versus what it uses — breadth is the usual finding.** + Reachability and width are independent. "Nothing calls it" finds removable grants and says + nothing about a grant that *is* called and hands over far more than the call needs. LavaMoat + grants whatever path you name, and + [dotted sub-paths are supported](https://github.com/LavaMoat/LavaMoat/blob/f5e52ab457c16c3aea72cc8a9dd0833547dd7d2c/packages/core/src/endowmentsToolkit.js#L101-L162) + for globals and builtins alike — the extension's own override already uses + `document.visibilityState`. So put each grant's used surface next to its call site, and when a + grant is wider than its use, name the narrowest path that covers every call: `crypto.getRandomValues` + not `crypto`, `node:url.fileURLToPath` not `node:url`, `document.visibilityState` not `document`. + A bare global whose only use is one property is a finding even though it is reached. + + Name the exploit-relevant grants present — code loading (`eval`, `Function`, `importScripts`, + `WebAssembly`, `Worker`, `Blob`+`createObjectURL`), network (`fetch`, `XMLHttpRequest`, + `WebSocket`, `sendBeacon`, `postMessage`), crypto/storage (bare `crypto`, `indexedDB`, + `localStorage`, `chrome.storage`), UI/navigation (`clipboard`, `window.open`, `location`, + `document`, `chrome.tabs`), Node builtins (`child_process`, `fs`, `net`, `http`, `vm`) — and + **hand them over without assessing them**. Do not explain what an attacker could do, do not + rank them. That is threat-model work owned by someone else. + + Do not inflate benign grants to look like findings. `clearTimeout` takes a timer id and cancels + it; it confers no reach. (On extension#45024 the `crypto` grant was caught only because it was + *unreachable* — had `random()` been called it would have been filed as needed and its `subtle` + breadth never mentioned. That is the gap this step closes.) + +5. **Cite it at a pinned tag, not a branch head.** A permalink to `…/blob/<tag>/<file>#Ln` is immutable; a branch-head link drifts out from under the citation. The permalink *is* the evidence — a reader clicks it and lands on the code, convinced without re-running anything. "It needs X" retyped into a table proves nothing about provenance. -5. **Lead with removal candidates and anything security-relevant — then give the full table.** +6. **Lead with removal candidates and anything security-relevant — then give the full table.** Open on what can be dropped and what the reading turned up, not on an inventory. But every grant still gets its own row in the capability → call-site table, load-bearing ones included: that mapping is what a reviewer came for, and a load-bearing row is one short row, not a @@ -118,7 +151,7 @@ droppable. Those observations are worth more than the grant inventory. Lead with usually means the identifier is present but the generator saw it in a path you haven't found — say what you searched rather than implying nothing uses it. -6. **Put it where the policy is reviewed — but do not tag anyone.** Post the justification as a +7. **Put it where the policy is reviewed — but do not tag anyone.** Post the justification as a comment on the PR carrying the `policy.json` change, so it lands in front of the people who own the policy rather than standing as a unilateral assertion elsewhere. @@ -140,7 +173,7 @@ droppable. Those observations are worth more than the grant inventory. Lead with ends, the tag goes in because the user says so, not because this line stops saying "hold". (Violated on extension#45024, 2026-07-30.) -7. **One reason covers the variants.** Extension builds carry several policy files +8. **One reason covers the variants.** Extension builds carry several policy files (`lavamoat/webpack/{mv2,mv3}/{beta,experimental,flask,main}/policy.json`). When the grant delta is identical across them, a single justification covers all — confirm the identity once. A grant that appears in one variant and not others is itself a question. @@ -148,17 +181,22 @@ droppable. Those observations are worth more than the grant inventory. Lead with ## Output **The capability → call-site table is the deliverable. Never dissolve it into prose.** -One row per grant, every grant, with its permalink and its removability in the row. A reviewer -scans the column, not paragraphs — 11 rows is denser and faster to read than three paragraphs -carrying the same 11 facts, so the table *is* the trimmed form. Prose around it is what gets cut. +One row per grant, every grant, with its permalink, its used surface and its breadth in the row. +A reviewer scans the column, not paragraphs — 11 rows is denser and faster to read than three +paragraphs carrying the same 11 facts, so the table *is* the trimmed form. Prose around it is +what gets cut. + +**Present findings; do not explain the mechanism.** Reviewers here know what a LavaMoat policy is, +what the bot does and what CI enforces. Restating it spends their attention on what they already +know. No paragraph on why call-site search is tautological, no methodology section — open on what +was found. ``` LavaMoat grants — <package> <old> -> <new> -<one or two lines: the call-site search is tautological on a generated policy; - the question is what can be dropped> +<one line naming the findings: which grant is wider than its use, which look stale> -| capability | package | call site | can it go? | +| capability | surface used | what it does | breadth | |---|---|---|---| | <cap> | <pkg> | <permalink, named by what the code does there> | yes — <the gate our usage never opens> | | <cap> | <pkg> | <permalink> · <second permalink if two sites> | at a cost — <what degrades> | diff --git a/domains/security/skills/supply-chain-audit/skill.md b/domains/security/skills/supply-chain-audit/skill.md index 8a8cb91c..1401095b 100644 --- a/domains/security/skills/supply-chain-audit/skill.md +++ b/domains/security/skills/supply-chain-audit/skill.md @@ -1,6 +1,6 @@ --- name: supply-chain-audit -description: Assess whether a dependency change is safe to take, across every detector that answers a different part of that question — Socket Security (malicious/anomalous package behavior, install scripts, new maintainers), `yarn npm audit` and advisories (known vulnerabilities), lockfile and manifest diffs (what actually changed, including transitive and resolution swaps), and LavaMoat policy grants (new capabilities, delegated to `lavamoat-policy-diligence`). Also covers the fronts no upstream scanner sees because they are things your own repo does to dependencies afterwards: yarn patches that modify dependency source at install, `resolutions` that force or stub versions, `npmAuditIgnoreAdvisories` suppression lists, CI actions riding mutable tags instead of pinned SHAs, and yarn plugins that execute at install. The falsifier is a lane whose finding is unaccounted for — a flagged package, an unresolved advisory, or a grant with no call site. Detection belongs to the tools; the job is disposition, and handing it to the humans who own the dependency. Triggers on /supply-chain-audit, or when asked whether a dependency bump is safe, to review a lockfile or package.json change, to triage a Socket or audit finding, or to assess supply-chain risk of a change. Callable by `evidence` as its supply-chain engine. +description: Assess whether a dependency change is safe to take, across every detector that answers a different part of that question — Socket Security (malicious/anomalous package behavior, install scripts, new maintainers), `yarn npm audit` and advisories (known vulnerabilities), lockfile and manifest diffs (what actually changed, including transitive and resolution swaps), and LavaMoat policy grants (new capabilities, delegated to `lavamoat-policy`). Also covers the fronts no upstream scanner sees because they are things your own repo does to dependencies afterwards: yarn patches that modify dependency source at install, `resolutions` that force or stub versions, `npmAuditIgnoreAdvisories` suppression lists, CI actions riding mutable tags instead of pinned SHAs, and yarn plugins that execute at install. The falsifier is a lane whose finding is unaccounted for — a flagged package, an unresolved advisory, or a grant with no call site. Detection belongs to the tools; the job is disposition, and handing it to the humans who own the dependency. Triggers on /mms-supply-chain-audit, or when asked whether a dependency bump is safe, to review a lockfile or package.json change, to triage a Socket or audit finding, or to assess supply-chain risk of a change. Callable by `evidence` as its supply-chain engine. maturity: experimental --- @@ -22,7 +22,7 @@ different question and is blind to the others, so a single green check is never | what actually changed? | lockfile / `package.json` diff | direct vs transitive; resolution swaps; version range widening | | known-vulnerable? | `yarn npm audit`, GitHub advisories, Dependabot | fixed-in version, or an explicit accepted-risk with reachability | | behaving maliciously or anomalously? | **Socket Security** | per-alert disposition — see below | -| new capability reached? | **LavaMoat** policy diff | **delegate to `lavamoat-policy-diligence`** | +| new capability reached? | **LavaMoat** policy diff | **delegate to `lavamoat-policy`** | | install-time code execution? | `allowScripts` in `package.json` (`@lavamoat/allow-scripts`) | a newly-`true` entry is a finding in its own right | | **is dependency source modified in-repo?** | **`.yarn/patches/*.patch`** | read the diff — see below | | **is a version being forced?** | **`resolutions`** in `package.json` | pinned below a fix? stubbed out? | @@ -114,10 +114,10 @@ say which lanes you ran and which you skipped, and why. read it, not as a unilateral assertion. This skill produces a justification for humans to act on; it does not approve anything. -## Capability containment → `lavamoat-policy-diligence` +## Capability containment → `lavamoat-policy` LavaMoat policy grants are a specialized lane with their own method and tooling. **Delegate to -`lavamoat-policy-diligence`** and fold its result in as this audit's capability-containment lane. +`lavamoat-policy`** and fold its result in as this audit's capability-containment lane. Do not restate that lane's question as "does each new capability have a call site" — the policy is generated from a real run, so it always does, and that check cannot fail. The lane's actual @@ -141,7 +141,7 @@ Supply-chain assessment — <package> <old> -> <new> (<direct|transitive|resol resolutions <forced/stubbed entries touched> | unchanged audit ignores <npmAuditIgnoreAdvisories added> → reason + retire-when | unchanged ci actions <third-party uses: added> → SHA-pinned? | unchanged - capabilities → lavamoat-policy-diligence: <removable: …; load-bearing: …> | no policy change + capabilities → lavamoat-policy: <removable: …; load-bearing: …> | no policy change lanes skipped <lane> — <why> Unresolved: <finding> — <what would settle it> | none ``` @@ -154,5 +154,5 @@ unresolved and what would settle it. ## Related -- `lavamoat-policy-diligence` — the capability-containment engine this skill delegates to. +- `lavamoat-policy` — the capability-containment engine this skill delegates to. - `evidence` — packages this skill's output as its [supply-chain evidence category](https://github.com/MetaMask/skills/blob/main/domains/pr-workflow/skills/evidence/references/evidence-catalog.md). From 609dd9c40600475b2aab664dc5af14e92f4fed22 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 05:49:40 -0400 Subject: [PATCH 123/135] End the diligence comment with an applicable override diff, and define its marker pair MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three additions to the output contract. A fenced `diff` against `lavamoat/webpack/<variant>/policy-override.json` for every narrowing or removal in the table. Prose like "could be narrowed to `node:url.fileURLToPath`" makes the reviewer translate it into JSON; a diff makes it a decision. The override is the file a human edits — the generated `policy.json` is regenerated and would lose the change. Dotted paths already work there: `copy-webpack-plugin>serialize-javascript` is granted `crypto.getRandomValues`, so the precedent is cited rather than the support asserted. A marker pair, `LAVAMOAT_DILIGENCE_START`/`_END`. `evidence/skill.md` already described this contract as having "its own header and marker pair" and no such pair was ever defined, so a re-run appended a second comment. Deliberately not `VALIDATION_RUN_*` — sharing that region would let an evidence re-run silently eat a diligence comment. And a sharper rule on runtime claims. A permalink witnesses a line; it does not witness what the author ran. An `npm pack` result, a grep over a tarball, a byte-comparison across policy files — those read as properties of the package and are properties of an unwitnessed local run. State them as the search or publish the output. Bare integers in prose fall under the same rule. --- .../security/skills/lavamoat-policy/skill.md | 56 +++++++++++++++---- 1 file changed, 44 insertions(+), 12 deletions(-) diff --git a/domains/security/skills/lavamoat-policy/skill.md b/domains/security/skills/lavamoat-policy/skill.md index 9f4634ac..bc0981a9 100644 --- a/domains/security/skills/lavamoat-policy/skill.md +++ b/domains/security/skills/lavamoat-policy/skill.md @@ -192,30 +192,62 @@ know. No paragraph on why call-site search is tautological, no methodology secti was found. ``` +<!-- LAVAMOAT_DILIGENCE_START --> +> <trial disclaimer, three lines, linking this skill's PR> + LavaMoat grants — <package> <old> -> <new> <one line naming the findings: which grant is wider than its use, which look stale> | capability | surface used | what it does | breadth | |---|---|---|---| -| <cap> | <pkg> | <permalink, named by what the code does there> | yes — <the gate our usage never opens> | -| <cap> | <pkg> | <permalink> · <second permalink if two sites> | at a cost — <what degrades> | -| <cap> | <pkg> | <permalink> | no | +| <cap> | <the members actually called, counted> | <what the code does, permalinked per boundary> | wider than its use — <narrowest path covering every call> | +| <cap> | <members> | <what it does> | <n of m members, and any property of the API worth stating> | +| <cap> | not referenced at any spelling | — | 🔍 candidate — <why it is now unreferenced> | …every grant gets a row… -Test for the "yes" rows: drop the grant, rebuild, run <suite>. -Note on <cap>: <what the reading turned up that bears on security> +<recommended override diff — see below> + +Test: <edit the resource, regenerate, run the suite>. +Loose ends: <what you could not settle, and what would make a row wrong> Removed: <cap>, <cap>. ← names only, no table, no justification column -Loose ends: <what you could not settle> -<identity across N policy files + observation artifact link> +<!-- LAVAMOAT_DILIGENCE_END --> +``` + +The marker pair is not decoration: a re-run replaces the region between them instead of appending +a second comment. It is deliberately *not* `VALIDATION_RUN_*` — this is a diligence artifact with +no verdict, and sharing that region would let an evidence re-run silently eat it. + +Order is the point: the lead names the findings, the table carries them, the diff makes them +actionable. Post it on the PR carrying the `policy.json` change, untagged. + +**End with a diff a reviewer can apply, not a description of one.** Every narrowing or removal in +the table gets a fenced `diff` block against the matching +`lavamoat/webpack/<variant>/policy-override.json`, because that is the file a human edits — the +generated `policy.json` is regenerated and would lose the change. Prose like "could be narrowed to +`node:url.fileURLToPath`" makes the reviewer translate; a diff makes it a decision. + +```diff + "resources": { ++ "sass-loader": { ++ "builtin": { ++ "node:url.fileURLToPath": true, ++ "node:url.pathToFileURL": true, ++ "node:url": false ++ } ++ }, ``` -Order is the point: the lead frames the question, the table answers it, and a reviewer hits the -removable rows immediately. Post it on the PR carrying the `policy.json` change, untagged. +Dotted paths already work in these files — `copy-webpack-plugin>serialize-javascript` is granted +`crypto.getRandomValues`, not bare `crypto`. Cite that precedent rather than asserting support. -**Runtime claims need a runtime artifact.** "Byte-identical across all 8 policy files" is an -observation, not something a `/blob/` link witnesses — publish the check output (JSON) and link -it. The `pr-evidence-gate` hook enforces this and will block the post otherwise; it is right to. +**Runtime claims need a runtime artifact.** A permalink witnesses what a line says; it does not +witness what *you ran*. "The tarball's complete specifier set is X", "byte-identical across all 8 +policy files", any grep or `npm pack` result — those are claims about a local run, and the reader +cannot check them. Publish the output and link it, or state the claim as the search it was +("searched N files, found no match") rather than as a property of the package. Bare integers in +prose need the same treatment: a reader who cannot trace "14 call sites" to something shown is +being asked to take it on trust. ## Worked example — extension#42867 (@sentry/browser 8.33.1 → 10.38.0) From 38316eee4d3ab28ba8e749fc1381546296708179 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 05:50:22 -0400 Subject: [PATCH 124/135] Give the diligence format a gate with `attest-gate.sh --diligence` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A diligence comment renders no verdict and deliberately does not use the Validation Run envelope. That exemption meant it was checked by nothing: this gate only knew the Validation Run shape, and `pr-evidence-gate.py` by design does not trip on a body claiming no verdict. So every rule the diligence skills state about their own output — including "runtime claims need a runtime artifact" — had no execution path. It showed. A lavamoat comment shipped with no marker pair, an `npm pack` specifier set no reader could fetch, and two bare integers traceable to nothing. `--diligence` swaps the four envelope checks for that contract's own — its marker pair, its header, permalinks pinned to a tag or SHA rather than a branch head, and a runtime claim check asking for the thing a `/blob/` link cannot witness. 3, 8 and 9 report SKIP with the reason rather than passing silently, since a check that cannot fail should not read as a check that passed. Everything downstream of the envelope is shared, because those defects are shared. Run against the comment that prompted this, it fails 1, 5 and 13 and passes the rest. --- .../skills/evidence/scripts/attest-gate.sh | 57 ++++++++++++++++++- 1 file changed, 54 insertions(+), 3 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 5fbed0ef..61d3728a 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -16,16 +16,17 @@ # 2 usage error set -uo pipefail -FILE="${1:-}"; REF=""; TARGET="" +FILE="${1:-}"; REF=""; TARGET=""; MODE="run" shift || true while [ $# -gt 0 ]; do case "$1" in --reference) REF="${2:-}"; shift 2 ;; --target) TARGET="${2:-}"; shift 2 ;; + --diligence) MODE="diligence"; shift ;; *) shift ;; esac done -[ -n "$FILE" ] || { echo "usage: attest-gate.sh <artifact.md> [--reference <file>] [--target <owner/repo#N>]" >&2; exit 2; } +[ -n "$FILE" ] || { echo "usage: attest-gate.sh <artifact.md> [--reference <file>] [--target <owner/repo#N>] [--diligence]" >&2; exit 2; } [ -f "$FILE" ] || { echo "attest-gate: not found: $FILE" >&2; exit 2; } FAILED=0 @@ -39,6 +40,25 @@ hasi() { grep -qiE "$1" "$FILE"; } # case-insensitive; a separate function bec echo "attest-gate: $FILE" echo +# A diligence comment (lavamoat-policy and its siblings) renders no verdict and deliberately +# does not use the Validation Run envelope — see "One comment per evidence kind" in +# references/evidence-publishing.md. That exemption used to mean it was checked by nothing at +# all: attest-gate only knew the Validation Run shape, and pr-evidence-gate.py by design does +# not trip on a body claiming no verdict. So every rule the diligence skills state about their +# own output had no execution path, and a comment shipped with an unwitnessed local `npm pack` +# result and untraceable integers. --diligence swaps the envelope checks for that contract's +# own; everything downstream of the envelope is shared, because those defects are shared. +if [ "$MODE" = diligence ]; then + has 'LAVAMOAT_DILIGENCE_START' && has 'LAVAMOAT_DILIGENCE_END' \ + && pass "1 marker pair" \ + || fail "1 marker pair" "no LAVAMOAT_DILIGENCE_START/_END — a re-run appends a duplicate instead of replacing, and the pair must not be VALIDATION_RUN_* or an evidence re-run would eat this region" + + hasre '^\*\*LavaMoat grants|^LavaMoat grants' \ + && pass "2 canonical header" \ + || fail "2 canonical header" "missing the 'LavaMoat grants — <package> <old> -> <new>' opener" + + printf ' SKIP %s\n' "3 verdict line — a diligence comment renders none, by contract" +else has 'VALIDATION_RUN_START' && has 'VALIDATION_RUN_END' \ && pass "1 marker pair" \ || fail "1 marker pair" "no VALIDATION_RUN_START/_END — a re-run appends a duplicate instead of replacing" @@ -50,14 +70,27 @@ has '## 🧪 Validation Run' \ hasre '^\*\*Verdict:\*\*.*\*\*Claim:\*\*' \ && pass "3 verdict line" \ || fail "3 verdict line" "no '**Verdict:** … — **Claim:** …' — valence is not legible at a glance" +fi # A run outside the repo's toolchain pins a different thing. A browser-memory lane # names "Firefox 153.0"; a repo lane names a head SHA and a lockfile hash. Both are # pins, and a check that only knows the second one fails every run of the first — # telling an author their pinned environment is unpinned. +# A diligence comment pins a read, not a run: its citations are permalinks, and the thing +# that can rot is a branch-head link drifting out from under the line it names. +if [ "$MODE" = diligence ]; then + if grep -qE 'https://github\.com/[^ )]+/blob/(main|master|develop|HEAD)/' "$FILE"; then + fail "4 citations pinned" "a permalink points at a branch head; it will drift off the line it cites. Pin a tag or a 40-char SHA" + elif grep -qE 'https://github\.com/[^ )]+/blob/[^/]+/' "$FILE"; then + pass "4 citations pinned" + else + fail "4 citations pinned" "no source permalink at all — the permalink IS the evidence here; a retyped 'it needs X' proves nothing about provenance" + fi +else hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `|[Ff]irefox [0-9]+\.[0-9]|[Cc]hrom(e|ium) [0-9]+\.|[Ss]afari [0-9]+\.|[Nn]ode v?[0-9]+\.[0-9]' \ && pass "4 environment pinned" \ || fail "4 environment pinned" "no head SHA, lockfile hash, or pinned toolchain/browser version" +fi # 5 — the one that matters, and it asks for a MEDIUM, not for better text. # @@ -77,7 +110,20 @@ hasre 'head `[0-9a-f]{7,}|sha256|node `v|yarn\.lock `|[Ff]irefox [0-9]+\.[0-9]|[ # tell: it looks reproducible and cannot be run. # An image, a re-executing link, or a hosted artifact — verification that does not route # through the author. `Produced by` and `evidence-artifacts/` are provenance, not this. -if ! hasre '!\[[^]]*\]\(https?://|<img [^>]*src="https?://|actions/runs/[0-9]|/gist\.|https?://[^ )]+\.(png|jpg|jpeg|gif|svg|txt|log|json)\b'; then +# In diligence mode the medium is the permalink, already required by check 4 — a reader +# clicks it and lands on the line. What a permalink cannot witness is what the AUTHOR RAN, +# and that is the defect this variant catches: an `npm pack` unpacked locally, a grep over a +# tarball, a byte-comparison across policy files. Those read as properties of the package +# and are actually properties of an unwitnessed local run. State them as the search +# ("searched N files, no match") or publish the output; do not assert them as fact. +if [ "$MODE" = diligence ]; then + if hasre "(complete|full) (specifier|import|require) set|byte-identical|identical across all|^Searched: .*tarball|npm pack" \ + && ! hasre 'actions/runs/[0-9]|/gist\.|https?://[^ )]+\.(txt|log|json)\b'; then + fail "5 runtime claims witnessed" "asserts a result only a local run could produce ($(grep -m1 -oiE '(complete|full) (specifier|import|require) set|byte-identical|identical across all|npm pack' "$FILE")) with nothing a reader can fetch. A /blob/ permalink witnesses a line, not your shell" + else + pass "5 runtime claims witnessed" + fi +elif ! hasre '!\[[^]]*\]\(https?://|<img [^>]*src="https?://|actions/runs/[0-9]|/gist\.|https?://[^ )]+\.(png|jpg|jpeg|gif|svg|txt|log|json)\b'; then fail "5 captured artifact" "no reader-verifiable capture — an image of the tool surface, a run link, or a hosted artifact. A fenced block is the author\'s transcription, whatever produced it" # No separate attribution test: a hosted artifact the reader fetches is its own # attribution, and requiring `Produced by` on top of it only fails runs whose @@ -114,6 +160,10 @@ else pass "7 no process narration" fi +if [ "$MODE" = diligence ]; then + printf ' SKIP %s\n' "8 verdict is earned — no verdict rendered" + printf ' SKIP %s\n' "9 verdict matches artifact — no verdict rendered" +else if hasi '\*\*Verdict:\*\*.*proven' && ! hasre 'Produced by |actions/runs|evidence-artifacts/'; then fail "8 verdict is earned" "claims 'proven' with no execution artifact — reading yields 'unverified'" else @@ -130,6 +180,7 @@ if printf '%s' "$HDR" | grep -q 'proven' && [ -n "$BODY" ]; then else pass "9 verdict matches artifact" fi +fi # 10 — the positive counterpart to check 6. A run succeeds by putting concerns in front # of a reviewer, so an artifact that floats nothing has reported only what it happened to From 76ecd0a9c0268f0aa1d731c7783688796f303d75 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 06:02:59 -0400 Subject: [PATCH 125/135] Remove private-repo and personal references from a public skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This repository is public. Naming a private repository here discloses its existence, its owner and roughly its contents to every reader — and a prohibition naming it ("do not re-host to X, it is private") discloses exactly as much as a recommendation would. Four such references were doing that, and the guidance survives without them: the rule is audience-reachability, which is stated directly rather than by example. Two memory-file citations offered as "source of truth" pointed into a private repo, so a reader was told to follow a rule whose justification they cannot open. The reasoning is inlined; the pointer is gone. The publish-surface snippet hardcoded a GitHub username, which decided the destination for whoever ran it. Now derived from `gh api user --jq .login`, and the surrounding prose is second-person rather than first — a shared skill has no "my PRs". `/attest` is no longer linked to a personal repository. That leaves it named but not resolvable, which is honest and is the smaller problem; the workflow depending on a command nobody else has is tracked separately. --- .../references/evidence-publishing.md | 34 +++++++++---------- .../evidence/references/lane-assertions.md | 2 +- domains/pr-workflow/skills/evidence/skill.md | 4 +-- 3 files changed, 20 insertions(+), 20 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md index 72ce8da4..65b88b4d 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md @@ -20,14 +20,15 @@ https://majorlift-artifacts-share.s3.us-west-1.amazonaws.com/public/metamask/pr- Anonymous `GetObject` is allowed under `public/*`; bucket listing is not, so the prefix is not browsable — link individual files, and don't promise readers an index. -**Do NOT re-host to `MajorLift/metamask-extension-skills`.** It is a **personal private** repo: -every raw link to it returns 404 for every reader but its owner. That was the previous target -here, and this file simultaneously said links to it were unreachable — guidance that instructed -you to publish dead links. Verified live in a published artifact. +**Do not re-host to a personal repo.** A personal private repo returns 404 for every reader but +its owner, so every artifact link published from one is dead on arrival. That was the previous +target here, and this file simultaneously said such links were unreachable — guidance that +instructed you to publish dead links. Verified live in a published artifact. -The test is **audience-reachability, not public-vs-private.** A `MetaMask/*` org repo is private -but readable by colleagues, so an internal-audience link to one is fine. A `MajorLift/*` personal -repo is unreachable by colleagues *and* by the public, so it fails for every audience. +The test is **audience-reachability, not public-vs-private.** An org repo may be private and still +readable by colleagues, so an internal-audience link to one is fine. A personal repo is unreachable +by colleagues *and* by the public, so it fails for every audience. Re-host to an org-owned +destination, or to the bucket above. - Path convention: `pr-<n>/<run-id>/<artifact-name>` keeps runs from colliding. - **Verify unauthenticated before shipping**: `curl -s -o /dev/null -w "%{http_code}"` on each @@ -116,20 +117,20 @@ Screenshots block (injected into `### After`, or appended under `### Screenshots ## Step 3 — Choose the surface by ownership, then publish -**Publish surface depends on my relationship to the PR** (see exogram -`evidence-publish-surface-by-ownership`). Determine it FIRST: +**Publish surface depends on your relationship to the PR.** Determine it FIRST: ```bash PR=<n>; REPO=MetaMask/metamask-extension -SURFACE=$(gh pr view "$PR" --repo "$REPO" --json author,commits --jq ' - if .author.login=="MajorLift" then "body" - elif ([.commits[] | select(.authors[].login=="MajorLift") - | select([.authors[].login] | map(select(.!="MajorLift" and .!="Copilot" and (test("claude|anthropic")|not))) | length == 0)] | length) > 0 +ME=$(gh api user --jq .login) +SURFACE=$(gh pr view "$PR" --repo "$REPO" --json author,commits --jq --arg me "$ME" ' + if .author.login==$me then "body" + elif ([.commits[] | select(.authors[].login==$me) + | select([.authors[].login] | map(select(.!=$me and .!="Copilot" and (test("claude|anthropic")|not))) | length == 0)] | length) > 0 then "comment" else "skip" end') ``` -- `body` — I authored the PR → upsert into the PR body (below). Validation is - part of my own claim. +- `body` — you authored the PR → upsert into the PR body (below). Validation is + part of your own claim. - `comment` — not author but I have a solo commit (no HUMAN co-author) → post a `gh pr comment` under the canonical `## 🧪 Validation Run` header. Never edit someone else's PR body. @@ -231,7 +232,6 @@ The common loop — a run refutes a claim, the author pushes a fix, `/evidence` - New head → **new hosted artifact directory keyed to the fix commit** (`pr-<n>/fix-<sha>/`), commit-pinned raw URLs; never overwrite a prior run's published files. - Residuals the fix intentionally leaves get their own row/section — don't round a fixed-with-residual claim up to fully proven. -Source of truth: `exogram-core/memory/evidence-revalidation-delta-reports.md`. ## Lead with a lane-status ledger (no silent absence) @@ -293,7 +293,7 @@ contract. `hooks/pr-evidence-gate.py` enforces the canonical literal only on bod trip it, which is the tell that the two are different artifacts rather than one with a different skin. -**Per-scenario presentation (2026-07-21):** the same applies one level down — when the evidence spans multiple test scenarios (flag-on vs flag-off, control vs treatment in an A/B falsifier, numbered manual-testing steps), give each scenario its **own sub-section**: a heading naming the scenario in observation terms, one line on what it tests plus its verdict, and that scenario's artifacts co-located under it. Never bunch all scenarios' artifacts into one large evidence dump — the reviewer verifies "under condition X, artifact shows Y" one condition at a time, and a merged block destroys that mapping even when every artifact is real. For long artifact sets use a `<details>` block *per scenario*, not a merge. (Preference: exogram-core `memory/evidence-present-scenarios-separately.md`; instance #44610.) +**Per-scenario presentation (2026-07-21):** the same applies one level down — when the evidence spans multiple test scenarios (flag-on vs flag-off, control vs treatment in an A/B falsifier, numbered manual-testing steps), give each scenario its **own sub-section**: a heading naming the scenario in observation terms, one line on what it tests plus its verdict, and that scenario's artifacts co-located under it. Never bunch all scenarios' artifacts into one large evidence dump — the reviewer verifies "under condition X, artifact shows Y" one condition at a time, and a merged block destroys that mapping even when every artifact is real. For long artifact sets use a `<details>` block *per scenario*, not a merge. (Instance: #44610.) ## Artifact contract (ADR-0058 alignment) diff --git a/domains/pr-workflow/skills/evidence/references/lane-assertions.md b/domains/pr-workflow/skills/evidence/references/lane-assertions.md index 89622327..7ba6dca7 100644 --- a/domains/pr-workflow/skills/evidence/references/lane-assertions.md +++ b/domains/pr-workflow/skills/evidence/references/lane-assertions.md @@ -23,4 +23,4 @@ Maps each evidence-catalog lane to a declarative assertion form, so a Claim Card | F7 i18n | static: `verify-locales` exit 0 | out-of-band | | F8 runtime containment | `Object.isFrozen(Object.prototype)`; scuttled global throws + exception resolves; `typeof SNOW` | **yes** — `Runtime.evaluate`, but only against the SHIPPED build variant (dev is unscuttled, test's exception list is wider) | -**Takeaway.** UI-state and runtime-metric lanes (A/B-visual, C1–C3/C6, F3/F5) map cleanly to CDP recipe assertions — ADR-0058's sweet spot. Static (D, F7), test-runner (B3, C5, F1), and dashboard (E) lanes are **out-of-band**: the recipe should *reference* them as proof targets without executing them. That out-of-band reference is precisely the **non-UI scaling gap** MajorLift's review of #173 flagged — a recipe schema that admits out-of-band assertion references (not only CDP actions) closes it. This table is the proposed taxonomy to contribute back (ITERATION item 10). +**Takeaway.** UI-state and runtime-metric lanes (A/B-visual, C1–C3/C6, F3/F5) map cleanly to CDP recipe assertions — ADR-0058's sweet spot. Static (D, F7), test-runner (B3, C5, F1), and dashboard (E) lanes are **out-of-band**: the recipe should *reference* them as proof targets without executing them. That out-of-band reference is precisely the **non-UI scaling gap** flagged in review of decisions#173 — a recipe schema that admits out-of-band assertion references (not only CDP actions) closes it. This table is the proposed taxonomy to contribute back (ITERATION item 10). diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index 4f3f0b77..b879f483 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -326,7 +326,7 @@ Check 5 is the one that matters and the easiest to slip past: if every character is one the operator typed, the run published an assertion. Pass `--reference <showcase>` to compare capture density against a known-good artifact. -This is phase 0 of [`/attest`](https://github.com/MajorLift/Reprise); phases 1 and 2 dispatch +This is phase 0 of `/attest`; phases 1 and 2 dispatch `/outframe ‖ /missing ‖ /press` then `/trim` to fresh instances, because those passes cannot be self-run — the author is positionally the wrong reader. @@ -619,7 +619,7 @@ Three adjacent things; keep the boundary clear so they compose instead of collid - **AEP** — governed *fleet orchestration*: sandboxes, Temporal, autonomous runs at scale. The heavy engine. - **ADR-0058 recipes** ([decisions#173](https://github.com/MetaMask/decisions/pull/173)) — a *dev-machine inner-loop* proof artifact: a declarative per-PR recipe run against the live app over CDP, emitting `summary.json`/`trace.json`/manifest. -- **evidence** (this skill) — the *claim→evidence methodology + taxonomy* both draw on. The Claim Card is the bridge from a PR's claim to the right proof target; the [evidence catalog](references/evidence-catalog.md) is the lane vocabulary; [lane-assertions.md](references/lane-assertions.md) maps each lane to a recipe assertion (and flags the out-of-band, non-UI lanes — the gap MajorLift's #173 review raised). +- **evidence** (this skill) — the *claim→evidence methodology + taxonomy* both draw on. The Claim Card is the bridge from a PR's claim to the right proof target; the [evidence catalog](references/evidence-catalog.md) is the lane vocabulary; [lane-assertions.md](references/lane-assertions.md) maps each lane to a recipe assertion (and flags the out-of-band, non-UI lanes — the gap raised in review of decisions#173). evidence is the one a human drives; it can dispatch an AEP run or author a recipe as its capture step. From fbf89095488cdcd73708ff65633303d7c91e4ba3 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 06:13:39 -0400 Subject: [PATCH 126/135] Take the artifact bucket and test fork out of the published text MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The re-hosting section named a personal S3 bucket, its region, its prefix layout, and stated that anonymous `GetObject` is enabled under `public/*`. That is a live unauthenticated endpoint advertised, with its structure, to every reader of a public repository — a larger disclosure than the repository names removed alongside it, and one that reads as configuration rather than as a secret, which is why it survived two passes over this file. Now `EVIDENCE_BUCKET` and `EVIDENCE_REGION` from the environment. The requirements the bucket must satisfy — anonymous GetObject under `public/*`, listing disabled — are stated, because those are the load-bearing part; the name never was. The G5 lane likewise named a private test fork, which carried both the org and a personal handle. Now "your own test fork". --- .../evidence/references/evidence-catalog.md | 2 +- .../references/evidence-publishing.md | 20 ++++++++++--------- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md index 103ab9b3..adca4077 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-catalog.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-catalog.md @@ -267,7 +267,7 @@ COOKIE_NAME=grafana_session COOKIE_VALUE="$sess" COOKIE_DOMAIN=<host> \ - **G2. Coverage delta** — `yarn test:unit:coverage` → `coverage/unit/` (and `yarn test:unit:webpack:coverage`); `codecov.yml`. Proves the new code is exercised. - **G3. Automated-reviewer output** — independent bot (e.g. cursor[bot]) found nothing blocking. Complements, never replaces, behavior evidence. - **G4. Manual reproduction steps** — human-followable steps that reproduce the fixed behavior; populates the PR template's Manual testing steps. -- **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-<X>=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: push to a branch literally named **`main`** (or `stable`) on `consensys-test/metamask-extension-test-majorlift` — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets on the fork (`INFURA_PROJECT_ID`, `TEST_SRP_*` for benchmarks; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo.. +- **G5. CI-workflow change, run on a test fork** — a CI-YAML-only PR usually **cannot exercise the workflow it edits**: identical build output ⇒ builds reused from base ⇒ `needs-<X>=false` ⇒ the workflow is *skipped* (`get-requirements.yml:654`). Escape: push to a branch literally named **`main`** (or `stable`) on your own test fork of the repo — `IS_RUN_EVERYTHING_BRANCH` (line 48) disables `find-reusable-builds` (line 310), so the workflow runs; `IS_CROSS_REPO_PR` is false inside the fork. Requires the workflow's secrets on the fork (`INFURA_PROJECT_ID`, `TEST_SRP_*` for benchmarks; `vars.`-gated Sentry/AWS steps skip cleanly) and a fork sync first. **State fork-scope in the published evidence** — it proves the workflow logic, not a run on the canonical repo.. - **G6. CI job-duration delta** — compare job wall-clock across arms in the Actions UI or `gh run view`. **Falsifier: build reuse.** `get-requirements.yml` skips jobs when build output matches base, so a measured "speedup" is often a skipped job — confirm each arm actually ran the work before comparing. Runner class and queue time vary independently of the change; report job time, not wall-clock from push. Pairs with `D7`, which measures the same change on a machine you control. --- diff --git a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md index 65b88b4d..2b452e7d 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md @@ -10,15 +10,17 @@ Canonical source for the format: `~/Code/metamask/metamask-autonomous-engineerin Control-plane artifact URLs (`localhost:3000/v1/runs/:id/artifacts/:name`) won't render on GitHub. Re-host each artifact and link the hosted URL. -**Host: the S3 bucket `majorlift-artifacts-share`, prefix `public/`.** +**Host: an S3 bucket you configure, prefix `public/`.** Set `EVIDENCE_BUCKET` and +`EVIDENCE_REGION` in your environment; this file does not name a bucket, because a bucket name +published here is an anonymously-readable endpoint advertised to everyone who reads it. ``` -s3://majorlift-artifacts-share/public/metamask/pr-<n>/<run-id>/<artifact-name> -https://majorlift-artifacts-share.s3.us-west-1.amazonaws.com/public/metamask/pr-<n>/<run-id>/<artifact-name> +s3://$EVIDENCE_BUCKET/public/metamask/pr-<n>/<run-id>/<artifact-name> +https://$EVIDENCE_BUCKET.s3.$EVIDENCE_REGION.amazonaws.com/public/metamask/pr-<n>/<run-id>/<artifact-name> ``` -Anonymous `GetObject` is allowed under `public/*`; bucket listing is not, so the prefix is not -browsable — link individual files, and don't promise readers an index. +The bucket must allow anonymous `GetObject` under `public/*` and must **not** allow listing, so +the prefix is not browsable — link individual files, and don't promise readers an index. **Do not re-host to a personal repo.** A personal private repo returns 404 for every reader but its owner, so every artifact link published from one is dead on arrival. That was the previous @@ -28,7 +30,7 @@ instructed you to publish dead links. Verified live in a published artifact. The test is **audience-reachability, not public-vs-private.** An org repo may be private and still readable by colleagues, so an internal-audience link to one is fine. A personal repo is unreachable by colleagues *and* by the public, so it fails for every audience. Re-host to an org-owned -destination, or to the bucket above. +destination, or to the configured bucket. - Path convention: `pr-<n>/<run-id>/<artifact-name>` keeps runs from colliding. - **Verify unauthenticated before shipping**: `curl -s -o /dev/null -w "%{http_code}"` on each @@ -36,8 +38,8 @@ destination, or to the bucket above. ```bash RUN_ID=<id>; PR=<n>; CP=localhost:3000 -BUCKET=majorlift-artifacts-share -BASE="https://$BUCKET.s3.us-west-1.amazonaws.com" +BUCKET="$EVIDENCE_BUCKET" +BASE="https://$BUCKET.s3.$EVIDENCE_REGION.amazonaws.com" for name in <artifactName1> <artifactName2>; do curl -fsS "$CP/v1/runs/$RUN_ID/artifacts/$name" -o "/tmp/$name" key="public/metamask/pr-$PR/$RUN_ID/$name" @@ -306,7 +308,7 @@ To stay interoperable with the recipe-based verification system (MetaMask/decisi - [ ] Each lane passed the [trustworthiness gate](evidence-trustworthiness.md) (shows the claimed surface, signal > noise, could-have-failed) - [ ] Multi-scenario evidence rendered **per scenario** (own heading + verdict + co-located artifacts), not bunched into one block - [ ] **Automated-process voice, no first person** — published validation output never says "I ran/captured/verified"; attribute to the process ("Automated validation ran…", "the harness captured…") so readers know the evidence is machine-generated, not a manual account under the author's name -- [ ] Every image/GIF re-hosted to `majorlift-artifacts-share/public/…`; no localhost/local-path URLs in the body +- [ ] Every image/GIF re-hosted to the configured bucket under `public/…`; no localhost/local-path URLs in the body - [ ] **Every published link curl'd unauthenticated and returning 200** — never a personal private repo - [ ] Work cited by **PR link** rather than tracking-ticket id, unless the ticket's own content (an RCA, a spec) is the referent - [ ] Narrative scrubbed of username/paths/internal hosts From 663420347b866075de35be2a3e43fe7cb01ad063 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 06:25:30 -0400 Subject: [PATCH 127/135] Restore what the privacy scrub broke: a working jq filter and bucket setup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removing the hardcoded username left `gh pr view --jq --arg me "$ME"`, which is not a thing gh supports — its built-in filter takes no --arg and the command dies with "accepts at most 1 arg(s)". Piped to real jq instead, and checked against both branches of the logic: a PR authored by someone else resolves to "skip", one authored by the caller to "body". Replacing the named bucket with `EVIDENCE_BUCKET` removed a working default and put nothing in its place, so the section told you to configure a bucket without saying what "conforming" meant. The policy is now stated: anonymous `s3:GetObject` under `public/*`, public-access blocks off for that bucket, `s3:ListBucket` to nobody. With a note that an org-owned bucket beats a personal one, since artifact links outlive their publisher. --- .../references/evidence-publishing.md | 23 +++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md index 2b452e7d..d86337b3 100644 --- a/domains/pr-workflow/skills/evidence/references/evidence-publishing.md +++ b/domains/pr-workflow/skills/evidence/references/evidence-publishing.md @@ -20,7 +20,24 @@ https://$EVIDENCE_BUCKET.s3.$EVIDENCE_REGION.amazonaws.com/public/metamask/pr-<n ``` The bucket must allow anonymous `GetObject` under `public/*` and must **not** allow listing, so -the prefix is not browsable — link individual files, and don't promise readers an index. +the prefix is not browsable — link individual files, and don't promise readers an index. If you +do not have one, that is the whole policy: + +```json +{ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Principal": "*", + "Action": "s3:GetObject", + "Resource": "arn:aws:s3:::YOUR-BUCKET/public/*" + }] +} +``` + +with `BlockPublicPolicy` and `RestrictPublicBuckets` disabled on that bucket and +`s3:ListBucket` granted to nobody. An org-owned bucket is preferable to a personal one: artifact +links outlive the person who published them. **Do not re-host to a personal repo.** A personal private repo returns 404 for every reader but its owner, so every artifact link published from one is dead on arrival. That was the previous @@ -124,7 +141,9 @@ Screenshots block (injected into `### After`, or appended under `### Screenshots ```bash PR=<n>; REPO=MetaMask/metamask-extension ME=$(gh api user --jq .login) -SURFACE=$(gh pr view "$PR" --repo "$REPO" --json author,commits --jq --arg me "$ME" ' +# Piped to jq rather than `gh --jq`: gh's built-in filter takes no --arg, and passing one +# fails with "accepts at most 1 arg(s)". +SURFACE=$(gh pr view "$PR" --repo "$REPO" --json author,commits | jq -r --arg me "$ME" ' if .author.login==$me then "body" elif ([.commits[] | select(.authors[].login==$me) | select([.authors[].login] | map(select(.!=$me and .!="Copilot" and (test("claude|anthropic")|not))) | length == 0)] | length) > 0 From d5fa3a549b7670d9d90cde1e91cb3afd3ef527e7 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 06:45:47 -0400 Subject: [PATCH 128/135] Enforce the evidence rules where the model cannot route around them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A results section reached a public PR whose entire content was hand-typed to look like terminal output. Three independent things had to hold for that, and all three did. The gate is model-invoked, so it can be skipped: the publish and the gate ran as two statements rather than one chain, and the verdict was read after the write. The hook that fires on the publish call carried a SECOND, narrower copy of the rules — keyed on verdict tokens — so a comment rendering no verdict satisfied neither copy. Two rule sets means the weaker one governs whatever falls between them. The hook now delegates to `attest-gate.sh`: one rule set, invoked by construction rather than by choice, in the mode the body's markers imply. It fails CLOSED once it has identified a body it is about to publish — an enforcement point that waves things through when it cannot find its rules is not one. And check 5 in `--diligence` had been rewritten as a phrase denylist ("npm pack", "complete specifier set"), which is precisely the regression its own comment records as having shipped four times: every property of plaintext is forgeable by whatever emits the plaintext. It is a medium test again — if the artifact shows a command or a run result, it owes the reader something fetchable. `/blob/` links are excluded, because a permalink to a `.json` file satisfied a naive extension test and was the specific reason the hand-typed section passed. Comment-update URLs carry the comment id, not the issue's, so check 12 was asking whether pull #5177261620 was open. Resolved through the API instead. Four-arm verified: blocks the exact command and body that shipped; ignores `ls`; ignores a `gh` read with no body write; refuses when the gate is unreachable. --- .../skills/evidence/hooks/pr-evidence-gate.py | 95 ++++++++++++++++++- .../skills/evidence/scripts/attest-gate.sh | 21 +++- 2 files changed, 110 insertions(+), 6 deletions(-) diff --git a/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py b/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py index 2bcd4651..03ac744e 100755 --- a/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py +++ b/domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py @@ -13,13 +13,17 @@ Each class below implements a numbered item of `references/evidence-trustworthiness.md`. Contract: reads PreToolUse JSON on stdin. Exit 0 = allow. Exit 2 = block -(stderr shown to the model). Fails OPEN on anything it cannot parse, so it -never bricks unrelated Bash commands. +(stderr shown to the model). Fails OPEN on anything it cannot parse, so it never +bricks unrelated Bash commands — but once it has identified a body it is going to +publish, it fails CLOSED: if attest-gate.sh cannot be found or run, the write is +refused rather than waved through. """ import json import os import re +import subprocess import sys +import tempfile def _out_allow(): @@ -63,6 +67,7 @@ def main(): _out_allow() # can't read it -> don't block; nothing to scan violations = _scan(body) + violations += _run_attest_gate(body, cmd) if not violations: _out_allow() @@ -90,6 +95,9 @@ def main(): NEEDS = { + "attest-gate": "the check named above to pass — run scripts/attest-gate.sh yourself to iterate", + "gate-missing": "attest-gate.sh on disk; refusing to publish a body nothing verified", + "gate-error": "attest-gate.sh to run successfully; refusing to publish unverified", "verdict": "an inspectable ARTIFACT (https:// permalink, /blob/<sha>/, or a *.test.ts ref)", "observation": "an OBSERVATION artifact (screenshot/recording/log/JSON/permalink) — " "a /blob/ code link witnesses code, not runtime behavior", @@ -240,6 +248,89 @@ def _extract_body(cmd): ) + +# ── attest-gate delegation ──────────────────────────────────────────────────── +# The rules live in attest-gate.sh. This hook used to carry a second, narrower copy +# of them — keyed on verdict tokens — and a diligence comment that renders no verdict +# satisfied neither the copy here nor the copy there. Two rule sets means the weaker +# one governs whatever falls between them, which is how a results section of hand-typed +# terminal output reached a public PR under both gates. +# +# So: one rule set, invoked at the one point the model cannot route around. The model +# runs the gate by choice; this runs it by construction. +def _gv(kind, token, snippet): + """attest-gate findings, in the shape the reporter already renders.""" + return {"kind": kind, "token": token, "snippet": snippet} + + +def _repo_pr_from_cmd(cmd): + """owner/repo#N for check 12. A comment-update URL carries the COMMENT id, not the + issue's — reading it as a PR number asks the gate whether pull #5177261620 is open, + which 404s and reports as 'destination unknown'. So resolve it.""" + m = re.search(r"(?:--repo\s+|github\.com/|repos/)([\w.-]+/[\w.-]+)", cmd) + repo = m.group(1) if m else "" + if not repo: + return "" + c = re.search(r"issues/comments/(\d+)", cmd) + if c: + try: + out = subprocess.run( + ["gh", "api", f"repos/{repo}/issues/comments/{c.group(1)}", + "--jq", ".issue_url"], + capture_output=True, text=True, timeout=30) + n = re.search(r"/issues/(\d+)\s*$", out.stdout.strip()) + return f"{repo}#{n.group(1)}" if n else "" + except Exception: # noqa: BLE001 + return "" + n = re.search(r"(?:issues|pulls?)/(\d+)|\bpr\s+(?:comment|edit|create)\s+(\d+)", cmd) + num = next((g for g in (n.groups() if n else ()) if g), "") + return f"{repo}#{num}" if num else "" + + +def _find_gate(): + here = os.path.dirname(os.path.abspath(__file__)) + for cand in ( + os.path.join(here, "..", "scripts", "attest-gate.sh"), + os.path.join(here, "attest-gate.sh"), + os.path.expanduser("~/.claude/skills/mms-evidence/scripts/attest-gate.sh"), + os.environ.get("ATTEST_GATE", ""), + ): + if cand and os.path.isfile(cand): + return os.path.abspath(cand) + return "" + + +def _run_attest_gate(body, cmd): + gate = _find_gate() + if not gate: + # Fails CLOSED. An enforcement point that waves things through when it cannot + # find its rules is not an enforcement point; the whole reason this exists is + # that the model-invoked path was skippable. + return [_gv("gate-missing", "attest-gate.sh not found", + "Set ATTEST_GATE to its path, or install mms-evidence.")] + mode = ["--diligence"] if "LAVAMOAT_DILIGENCE_START" in body else [] + target = _repo_pr_from_cmd(cmd) + with tempfile.NamedTemporaryFile("w", suffix=".md", delete=False) as fh: + fh.write(body) + path = fh.name + try: + argv = ["bash", gate, path] + mode + (["--target", target] if target else []) + proc = subprocess.run(argv, capture_output=True, text=True, timeout=120) + except Exception as exc: # noqa: BLE001 - any failure to run it is a failure to verify + os.unlink(path) + return [_gv("gate-error", str(exc), "attest-gate.sh could not be run.")] + os.unlink(path) + if proc.returncode == 0: + return [] + out = proc.stdout.splitlines() + fails = [] + for i, ln in enumerate(out): + if ln.strip().startswith("FAIL"): + detail = out[i + 1].strip() if i + 1 < len(out) else "" + fails.append(_gv("attest-gate", ln.strip()[6:].strip(), detail)) + return fails or [_gv("attest-gate", f"exit {proc.returncode}", proc.stdout[-200:])] + + def _scan(body): # Strip bot-generated summary block — not our claim. body = re.sub(r"<!--\s*CURSOR_SUMMARY\s*-->.*?<!--\s*/CURSOR_SUMMARY\s*-->", diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 61d3728a..019274da 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -117,11 +117,24 @@ fi # and are actually properties of an unwitnessed local run. State them as the search # ("searched N files, no match") or publish the output; do not assert them as fact. if [ "$MODE" = diligence ]; then - if hasre "(complete|full) (specifier|import|require) set|byte-identical|identical across all|^Searched: .*tarball|npm pack" \ - && ! hasre 'actions/runs/[0-9]|/gist\.|https?://[^ )]+\.(txt|log|json)\b'; then - fail "5 runtime claims witnessed" "asserts a result only a local run could produce ($(grep -m1 -oiE '(complete|full) (specifier|import|require) set|byte-identical|identical across all|npm pack' "$FILE")) with nothing a reader can fetch. A /blob/ permalink witnesses a line, not your shell" + # A permalink is the medium for a CITATION — a reader clicks it and lands on the line. It is + # not the medium for anything you RAN. The first version of this branch tested for phrases + # ("npm pack", "complete specifier set") and passed an artifact whose entire results section + # was hand-typed to look like terminal output, because none of those words appeared in it. + # That is the regression the block below already documents as having shipped four times: + # every property of plaintext is forgeable by whatever emits the plaintext. So the test is + # the same one, on the same terms — if the artifact shows a command or a run result, it owes + # the reader something fetchable. + # `/blob/` is a CITATION, never a capture — it witnesses a line in a file, not a run. + # Excluding it matters: a permalink to `policy-override.json` ends in `.json` and satisfied + # a naive extension test, so an artifact whose entire results section was hand-typed passed + # on the strength of a source link. + if hasre '^\$ |^ *\$ |\bexit [0-9]|\bexit=[0-9]' \ + && ! grep -qE 'actions/runs/[0-9]|/gist\.|!\[[^]]*\]\(https?://' "$FILE" \ + && ! grep -E 'https?://[^ )]+\.(txt|log|json)\b' "$FILE" | grep -qv '/blob/'; then + fail "5 captured artifact" "shows a command or a run result with nothing a reader can fetch — a fenced block is your transcription, whatever produced it. Publish the log/gist/run and link it" else - pass "5 runtime claims witnessed" + pass "5 captured artifact" fi elif ! hasre '!\[[^]]*\]\(https?://|<img [^>]*src="https?://|actions/runs/[0-9]|/gist\.|https?://[^ )]+\.(png|jpg|jpeg|gif|svg|txt|log|json)\b'; then fail "5 captured artifact" "no reader-verifiable capture — an image of the tool surface, a run link, or a hosted artifact. A fenced block is the author\'s transcription, whatever produced it" From 27be7aeeb0a02f25df84eadfc51116432ad172d3 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 07:59:16 -0400 Subject: [PATCH 129/135] =?UTF-8?q?Add=20`attest`=20=E2=80=94=20the=20publ?= =?UTF-8?q?ish=20gate,=20ported=20from=20a=20personal=20command?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `mms-evidence` named `/attest` as the gate its output passes and shipped no such thing. Anyone installing the skill got phase 0 as a script and a reference to a command only its author had, which is half a publish path for everyone else. Phase 0 is `attest-gate.sh`, already here. Phase 1 is three briefs sent to fresh instances — frame, coverage, and how it reads to a stranger — written out in `references/dispatched-passes.md` so dispatch does not depend on commands that live elsewhere. The check table in `references/phase-0-checks.md` is generated from the gate rather than retyped, so it cannot drift from what runs. Two things the port makes explicit that the original left to discipline. The gate must be the same shell chain as the publish, because running both and reading the verdict after the write is how a blocked artifact reached a public PR. And softening a check to fit the case in hand is called out as an anti-pattern: if the new version could be satisfied by better prose alone, it is no longer the check. --- .../attest/references/dispatched-passes.md | 41 +++++++ .../attest/references/phase-0-checks.md | 35 ++++++ domains/pr-workflow/skills/attest/skill.md | 103 ++++++++++++++++++ .../skills/evidence/scripts/attest-gate.sh | 7 +- domains/pr-workflow/skills/evidence/skill.md | 2 +- 5 files changed, 186 insertions(+), 2 deletions(-) create mode 100644 domains/pr-workflow/skills/attest/references/dispatched-passes.md create mode 100644 domains/pr-workflow/skills/attest/references/phase-0-checks.md create mode 100644 domains/pr-workflow/skills/attest/skill.md diff --git a/domains/pr-workflow/skills/attest/references/dispatched-passes.md b/domains/pr-workflow/skills/attest/references/dispatched-passes.md new file mode 100644 index 00000000..0583920e --- /dev/null +++ b/domains/pr-workflow/skills/attest/references/dispatched-passes.md @@ -0,0 +1,41 @@ +# Phase 1 — the three dispatched briefs + +Send each to a **fresh instance** with the artifact and nothing else: not the transcript, not +your reasoning, not what you expect it to find. Context is what you are testing for. An instance +that knows what you meant will read what you meant. + +Run them concurrently — they are independent, and sequencing lets the first one's findings frame +the others. + +## outframe — contest the frame + +> You are reading a finished set of findings you did not produce. Do not check whether the +> findings are correct. Ask what claim was chosen and what a different framing makes visible: +> what question would a reader with different priorities have asked of the same material, what +> does the chosen frame make it impossible to notice, and which of the findings only look +> significant because of how the problem was cut. Return findings the framing hid, not a +> critique of the writing. + +## missing — contest the coverage + +> You are auditing a completed run for what it did not do. Enumerate: a modality that was not +> run, a claim asserted but not verified, a source cited but not read, a case the method +> structurally cannot reach. For each, say what running it would cost and what it could change. +> Do not restate what the run found. Absence is the deliverable. + +## press — read it as the stranger + +> You are the reviewer this lands in front of, with no context and a decision to make. Read only +> the artifact. Say what you would have to take on trust, which number you could not check if you +> wanted to, what reads as a measurement but is a sentence, and anything that assumes you were +> present for work you were not. Flag register slips: hedging that reads as concealment, +> confidence that outruns the evidence, and any place the author's process shows through. + +## Reading the returns + +A finding from any pass that invalidates the claim is `BLOCKED`. A finding that qualifies it is +`ATTESTED WITH` — and the caveat goes **into the published artifact**, not just into the verdict, +or the reader never sees it. + +Disagreement between passes is signal, not noise: `press` clearing something `outframe` flagged +usually means the artifact reads well and is framed wrong, which is the more dangerous state. diff --git a/domains/pr-workflow/skills/attest/references/phase-0-checks.md b/domains/pr-workflow/skills/attest/references/phase-0-checks.md new file mode 100644 index 00000000..994ad33c --- /dev/null +++ b/domains/pr-workflow/skills/attest/references/phase-0-checks.md @@ -0,0 +1,35 @@ +# Phase 0 — what each check catches + +Generated from the checks in `mms-evidence/scripts/attest-gate.sh`; that script is the +authority. Each entry exists because a run shipped without it. + +| # | check | run mode | diligence mode | +|---|---|---|---| +| 1 | marker pair | ✓ | ✓ | +| 2 | canonical header | ✓ | ✓ | +| 3 | verdict line | ✓ | ✓ | +| 4 | citations pinned | ✓ | ✓ | +| 5 | captured artifact | ✓ | ✓ | +| 6 | no prescriptions | ✓ | ✓ | +| 7 | no process narration | ✓ | ✓ | +| 8 | verdict is earned | ✓ | ✓ | +| 9 | verdict matches artifact | ✓ | ✓ | +| 10 | floats something for review | ✓ | ✓ | +| 11 | disclaimer present and early | ✓ | ✓ | +| 12 | destination is open | ✓ | ✓ | +| 13 | figures trace to an exhibit | ✓ | ✓ | + +Checks 1–4 differ by mode: in `--diligence` they test that contract's own marker pair, its +header, and that citations are pinned to a tag or SHA rather than a branch head, and the +verdict-line check reports SKIP because a diligence artifact renders none. Checks 8 and 9 SKIP +for the same reason. Everything from 5 down is shared, because those defects are shared. + +**Check 5 is the one that matters, and it asks for a medium.** Every earlier version tested a +property of the plaintext — does it carry a marker, does the command contain a placeholder — and +each caught one defect and missed the next, because every property of plaintext is forgeable by +whatever emits the plaintext. Four runs shipped that way. A `/blob/` permalink is a citation and +does not satisfy it: it witnesses a line in a file, never a run. + +**Check 12 tests the destination**, which no property of the text reveals. Across one register of +published runs, 22 of 27 comments went to pull requests that had already merged — median 22 days +after the merge, gate-clean every time. diff --git a/domains/pr-workflow/skills/attest/skill.md b/domains/pr-workflow/skills/attest/skill.md new file mode 100644 index 00000000..e5af6d89 --- /dev/null +++ b/domains/pr-workflow/skills/attest/skill.md @@ -0,0 +1,103 @@ +--- +name: attest +description: The gate an evidence artifact passes before it is published to a pull request, issue or shared tracker. Two halves that do not substitute for each other — a mechanical pass that greps for the properties a reader needs (marker pair, pinned environment, a captured artifact rather than typed prose, a destination that is still open) and a dispatched pass sent to fresh instances that contest the framing, the coverage, and how it reads to a stranger. The author is the wrong checker: they remember running the check, and the memory supplies the provenance the text lacks. Verdicts are attested, attested with named caveats, blocked, or not a run — the last being common and legitimate, because a run that could not execute has produced nothing to publish. Triggers on mms-attest, or before posting any evidence, validation or diligence output to a public surface. +maturity: experimental +--- + +# /mms-attest + +The gate an evidence artifact passes before it leaves your hands. Use before posting any +`/mms-evidence` or diligence output to a pull request, issue, or shared tracker. + +## The author is the wrong reader, and the wrong checker + +A validation run claims something was measured. Its characteristic failure is not a wrong number +— it is **prose that reads like a measurement**. An operator who ran the check cannot see this, +because they remember running it; the memory supplies the provenance the text lacks, before the +eye registers that it was missing. + +This is not hypothetical. A run in this workflow shipped a results section whose commands, exit +codes and "reached 100%" were typed by hand, while the real logs sat unpublished on disk. The +author had the skill installed that forbids exactly that. + +So the gate has two halves, and neither substitutes for the other. + +**The mechanical half is not advisory.** Marker presence, a pinned environment, whether any +fenced block is a tool's output rather than the author's transcription, whether the destination +is still open — all greppable. Anything checkable is checked before a model is asked for +judgement, because a model asked "is this good evidence?" answers from inside the frame that +produced it. + +**The dispatched half is positional.** Contesting the frame, the coverage, and the reading cannot +be self-run, for the same reason an author cannot proofread their own sentence for a word their +eye supplies. + +## Phase 0 — mechanical, no model + +``` +scripts/attest-gate.sh <artifact.md> --target <owner/repo#N> +scripts/attest-gate.sh <artifact.md> --target <owner/repo#N> --diligence +``` + +Thirteen checks; every one a hard fail. `--diligence` swaps the four Validation-Run envelope +checks for a no-verdict contract's own and shares everything downstream. See +[references/phase-0-checks.md](references/phase-0-checks.md) for what each check exists to catch +and the run that caused it to be written. + +**Run it as the same command that publishes, or it is a log line.** The gate and the write must +be one chain — `gate && publish`. Running both and reading the verdict afterwards is how a +blocked artifact reaches a public PR. The `hooks/pr-evidence-gate.py` PreToolUse hook enforces +this independently of your discipline, and fails closed; phase 0 is what you run to iterate +before it does. + +## Phase 1 — dispatched, three lenses + +| pass | reads for | returns | +|---|---|---| +| **outframe** | the frame — what claim was chosen, and what a different framing makes visible | findings the framing hid | +| **missing** | coverage — modality not run, claim unverified, source unread | the gap list | +| **press** | the text as it ships, as the stranger who has to act on it | leak and register findings | + +Dispatch to fresh instances is the mechanism, not an optimisation: a self-run frame check is +composed inside the frame it is meant to test. Briefs in +[references/dispatched-passes.md](references/dispatched-passes.md). + +Skipping a pass is allowed. Silently skipping it is not — name it as skipped in the verdict. + +## Phase 2 — shape + +Front-load the verdict, cut anything that does not change what the reader does, keep every +artifact and move only its placement. Shape only, after content is settled — a shape pass that +reaches content is how a capability table gets dissolved into paragraphs and the comment's +payload disappears. + +## Verdict + +``` +ATTESTED phase 0 clean, no blocking finding from phase 1 +ATTESTED WITH publishable, with named caveats carried INTO the artifact +BLOCKED phase 0 failure, or a phase 1 finding that invalidates the claim +NOT A RUN nothing was measured; there is no artifact to publish +``` + +`NOT A RUN` is legitimate and common. A run that could not execute its check produced no +evidence, and publishing the attempt with a disclaimer is worse than publishing nothing — the +disclaimer reads as hedging and the figure is kept anyway. + +## Anti-patterns + +| Bad | Good | +|---|---| +| Running phase 1 to decide phase 0 | Mechanical checks first; cheap and unarguable | +| Self-running the dispatched passes | Dispatch, or skip and say it was skipped | +| Attesting your own run | The gate is positional; an author attesting themselves attests nothing | +| Treating phase 0 items as advisory | Every one is a hard fail | +| `ATTESTED WITH` as a soft pass | The caveat goes *into the published artifact*, not just the verdict | +| Softening a check to fit the case in hand | If the new version could be satisfied by better prose alone, it is no longer the check | + +## Related + +- `mms-evidence` — produces the artifact this gates +- `mms-instrument-check` — prove the instrument fires before its output counts +- `mms-unmeasured-join` — audit the inference between the facts +- `mms-scope-of-search` — what a negative result is a fact about diff --git a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh index 019274da..4168e740 100755 --- a/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh +++ b/domains/pr-workflow/skills/evidence/scripts/attest-gate.sh @@ -167,7 +167,12 @@ else pass "6 no prescriptions" fi -if hasi "I originally|correction to my earlier|filed by me|hard to calibrate|I withdraw|my earlier comment"; then +# Drafting history is the author's, not the reader's: a reader who never saw the earlier +# version learns nothing from being told it existed, and the byline may not be yours. +# The list grew after a comment shipped a '### Correction:' section retracting its own +# previous revision in place — right instinct, wrong surface. Retract by restating the +# finding correctly; the account of how it changed belongs in a postmortem. +if hasi "I originally|correction to my earlier|filed by me|hard to calibrate|I withdraw|my earlier comment|earlier revision|previous revision|an earlier version of this|is withdrawn|that claim was wrong|^#{1,4} *Correction[: ]|corrected below|see the correction"; then fail "7 no process narration" "contains first-person process commentary — the reader did not see the earlier draft, and the byline may not be yours" else pass "7 no process narration" diff --git a/domains/pr-workflow/skills/evidence/skill.md b/domains/pr-workflow/skills/evidence/skill.md index b879f483..78ae160a 100644 --- a/domains/pr-workflow/skills/evidence/skill.md +++ b/domains/pr-workflow/skills/evidence/skill.md @@ -326,7 +326,7 @@ Check 5 is the one that matters and the easiest to slip past: if every character is one the operator typed, the run published an assertion. Pass `--reference <showcase>` to compare capture density against a known-good artifact. -This is phase 0 of `/attest`; phases 1 and 2 dispatch +This is phase 0 of `mms-attest`; phases 1 and 2 dispatch `/outframe ‖ /missing ‖ /press` then `/trim` to fresh instances, because those passes cannot be self-run — the author is positionally the wrong reader. From bbf746e3baba2c9cbcda3df3927f2893c3f3b5ca Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 07:59:35 -0400 Subject: [PATCH 130/135] Require the removal test to exercise the capability, not the app MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A three-arm build on extension#44727 established nothing. Denying sass-loader every URL and path builtin still compiled the extension, exactly as granting all of them did — so "it still builds" was the expected result either way and carried no information. Most grants are not on the startup path; that is usually why they look removable. So a build or a boot with the grant removed shows only that startup did not need it. The test has to name the scenario that actually executes the read — the error path formatting a span URL, the source-map write, the importer resolving a relative `@use`, the flag that turns the feature on — and run that. Two preconditions before either arm is believed, both cheap and both skipped on that run: confirm the effective policy really changed by merging the override into the base and printing the resource, and confirm a fully-denied arm actually fails. If denying everything passes, the grant is not enforced on that path, and "this suite does not arbitrate this grant" is the finding. --- .../security/skills/lavamoat-policy/skill.md | 27 ++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/domains/security/skills/lavamoat-policy/skill.md b/domains/security/skills/lavamoat-policy/skill.md index bc0981a9..851a1d53 100644 --- a/domains/security/skills/lavamoat-policy/skill.md +++ b/domains/security/skills/lavamoat-policy/skill.md @@ -45,7 +45,32 @@ don't get paragraphs. > **Falsifier.** A grant you called load-bearing that a build with it removed still passes. > The test is cheap and it is the only thing that settles the question: drop the grant from the -> resource, rebuild, run the relevant e2e. Propose it; the policy owners run it. +> resource, rebuild, run the relevant e2e. +> +> **Run it yourself whenever the steps are clear enough to run.** A removal candidate handed +> over untested asks the reviewer to do the work that would settle it, and most of them will not +> — so the finding sits. A build that passes with the capability removed converts `🔍 candidate` +> into a demonstrated non-breaking reduction, and that is a different object: it can be merged +> rather than considered. Publish the run as the evidence, not the conclusion drawn from it. +> +> Hand the test over only when you genuinely cannot run it — a variant needing credentials you +> do not have, an e2e suite the environment cannot host, a policy whose regeneration needs CI's +> toolchain. Say which of those it is; "propose the test" as a default is the failure mode this +> replaces. +> +> **Exercise the capability, not the app.** A build that compiles, or an app that boots, with the +> grant removed shows only that nothing on the startup path needed it. Most grants are not on the +> startup path — that is usually *why* they look removable — so "it still builds" is the null you +> should expect either way, and it is not evidence. Name the scenario that would actually execute +> the read: the error path that formats a `span.url`, the source-map write, the importer resolving +> a relative `@use`, the config flag that turns the feature on. Then run that. +> +> **And prove the arms differ before believing either.** Two things, both cheap. That the +> effective policy really changed — merge the override into the base and print the resource, +> rather than assuming an edit took. And that a fully-denied arm actually *fails*. If denying +> everything still passes, the grant is not enforced on that path, and no arrangement of arms on +> that path can tell you anything. Report that: "this suite does not arbitrate this grant" is a +> real finding, and more useful than a green you cannot cash. **Corollary — the reading is where the real findings come from.** Locating each call site means reading the code that uses the capability, and that is when genuine issues surface: unbounded work From 34b935c953dda2cb416629ee3a2f81b4146e8207 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 08:00:47 -0400 Subject: [PATCH 131/135] Name the installed command in `react-render-delta`'s description MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The installer emits `mms-react-render-delta`; the description advertised `/react-render-delta`, which resolves to nothing. Caught by the check #99 adds — this branch predates it and only fails once combined. --- domains/performance/skills/react-render-delta/skill.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/domains/performance/skills/react-render-delta/skill.md b/domains/performance/skills/react-render-delta/skill.md index 919c17c3..7b50c00b 100644 --- a/domains/performance/skills/react-render-delta/skill.md +++ b/domains/performance/skills/react-render-delta/skill.md @@ -1,6 +1,6 @@ --- name: react-render-delta -description: Prove a React rendering or memoization change actually reduced work, with a delivery gate and a reported band. Covers re-render counts (why-did-you-render), selector recomputes (reselect's real `.recomputations()` API), and A/B arms toggled at a FIXED commit rather than across a merge boundary. The falsifier is an arm whose treatment never reached the built bundle — a null from undelivered treatment is indistinguishable from a null from a small effect and reports as the second. Triggers on /react-render-delta, or when asked to prove a component stopped over-rendering, measure selector recomputation, validate a memoization/React Compiler change, run a render-count A/B, or interpret a re-render benchmark. Callable by `evidence` as its React render & selector proof engine. +description: Prove a React rendering or memoization change actually reduced work, with a delivery gate and a reported band. Covers re-render counts (why-did-you-render), selector recomputes (reselect's real `.recomputations()` API), and A/B arms toggled at a FIXED commit rather than across a merge boundary. The falsifier is an arm whose treatment never reached the built bundle — a null from undelivered treatment is indistinguishable from a null from a small effect and reports as the second. Triggers on /mms-react-render-delta, or when asked to prove a component stopped over-rendering, measure selector recomputation, validate a memoization/React Compiler change, run a render-count A/B, or interpret a re-render benchmark. Callable by `evidence` as its React render & selector proof engine. maturity: experimental --- From ea4f5018bbc04a7dc3c78340f92a3fb015bc6aea Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 08:00:58 -0400 Subject: [PATCH 132/135] Name the installed command in `debug`'s description The installer emits `mms-debug`; the description advertised `/debug`. --- domains/pr-workflow/skills/debug/skill.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/domains/pr-workflow/skills/debug/skill.md b/domains/pr-workflow/skills/debug/skill.md index 1a6458a7..2a35b42c 100644 --- a/domains/pr-workflow/skills/debug/skill.md +++ b/domains/pr-workflow/skills/debug/skill.md @@ -1,6 +1,6 @@ --- name: debug -description: Locate the cause of a symptom you cannot yet explain — a crash, a leak, a flake, a production error spike, a number that moved. The sibling of evidence: where evidence is handed a claim and looks for the observation that would falsify it, this is handed a symptom and must generate the hypothesis first, then kill it. Classifies the symptom into a defect class, routes to the engine skill that owns that class (memory-leak, race-condition-repro, react-render-proof, sentry-grafana-correlation, extension-errors-debugging, tsc-blindspots, supply-chain-audit), and holds the investigation to the same evidence bar evidence applies — an instrument that cannot fail is not evidence, a null needs its sensitivity stated, and a finding is scoped to what the change introduced versus what pre-existed. Stops when the cause is located or the class is excluded, not when a plausible story is available. Triggers on /debug, or when asked to debug, diagnose, or investigate a symptom, find why something is slow, leaking, flaky, or erroring, chase a production alert to its cause, or reproduce a bug that cannot be reproduced by hand. +description: Locate the cause of a symptom you cannot yet explain — a crash, a leak, a flake, a production error spike, a number that moved. The sibling of evidence: where evidence is handed a claim and looks for the observation that would falsify it, this is handed a symptom and must generate the hypothesis first, then kill it. Classifies the symptom into a defect class, routes to the engine skill that owns that class (memory-leak, race-condition-repro, react-render-proof, sentry-grafana-correlation, extension-errors-debugging, tsc-blindspots, supply-chain-audit), and holds the investigation to the same evidence bar evidence applies — an instrument that cannot fail is not evidence, a null needs its sensitivity stated, and a finding is scoped to what the change introduced versus what pre-existed. Stops when the cause is located or the class is excluded, not when a plausible story is available. Triggers on /mms-debug, or when asked to debug, diagnose, or investigate a symptom, find why something is slow, leaking, flaky, or erroring, chase a production alert to its cause, or reproduce a bug that cannot be reproduced by hand. maturity: experimental --- From 42d69a4901d7c55913e8daebc417dedd1cffa2ae Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 08:01:10 -0400 Subject: [PATCH 133/135] Name the installed command in `memory-leak`'s description The installer emits `mms-memory-leak`; the description advertised `/memory-leak`. --- domains/stability/skills/memory-leak/skill.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/domains/stability/skills/memory-leak/skill.md b/domains/stability/skills/memory-leak/skill.md index e59966b9..1250a56e 100644 --- a/domains/stability/skills/memory-leak/skill.md +++ b/domains/stability/skills/memory-leak/skill.md @@ -1,6 +1,6 @@ --- name: memory-leak -description: Find and investigate memory leaks / retention issues in JavaScript/TypeScript. Two phases. (1) Static identification from a diff — enumerate the retention primitives the change introduces (event listeners, timers, pending-request registries, subscriptions, module singletons, growing collections), pair every acquire with its release site, and scope findings to what the diff adds versus what pre-exists. (2) Runtime investigation, only for a primitive that cannot be paired statically — DevTools/CDP heap snapshots over N cycles, the retainer graph, detached-node count, and a falsifying lifecycle test. Leads with the cheap static read (the retention review a reviewer already performs) and escalates to a heap snapshot only where the read is inconclusive. Triggers on /memory-leak, or when asked to find or investigate a memory leak, check listener/subscription/timer cleanup, review a diff for retention, or take and read a heap snapshot. Callable by `evidence` as its memory-leak engine. +description: Find and investigate memory leaks / retention issues in JavaScript/TypeScript. Two phases. (1) Static identification from a diff — enumerate the retention primitives the change introduces (event listeners, timers, pending-request registries, subscriptions, module singletons, growing collections), pair every acquire with its release site, and scope findings to what the diff adds versus what pre-exists. (2) Runtime investigation, only for a primitive that cannot be paired statically — DevTools/CDP heap snapshots over N cycles, the retainer graph, detached-node count, and a falsifying lifecycle test. Leads with the cheap static read (the retention review a reviewer already performs) and escalates to a heap snapshot only where the read is inconclusive. Triggers on /mms-memory-leak, or when asked to find or investigate a memory leak, check listener/subscription/timer cleanup, review a diff for retention, or take and read a heap snapshot. Callable by `evidence` as its memory-leak engine. maturity: experimental --- From 43b5e920d05b6463f655b5665ebae790b90d2dbc Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 08:01:24 -0400 Subject: [PATCH 134/135] Name the installed command in `race-condition-repro`'s description The installer emits `mms-race-condition-repro`; the description advertised `/race-condition-repro`. --- domains/stability/skills/race-condition-repro/skill.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/domains/stability/skills/race-condition-repro/skill.md b/domains/stability/skills/race-condition-repro/skill.md index 47b8de16..72cf2ca2 100644 --- a/domains/stability/skills/race-condition-repro/skill.md +++ b/domains/stability/skills/race-condition-repro/skill.md @@ -1,6 +1,6 @@ --- name: race-condition-repro -description: Prove an ordering guarantee under concurrency — that when B arrives during A's pending window, A is canceled, or completes first, or the two commit in a defined order. Covers race conditions, retries, cancellation, supersession, debounce/throttle, locks, queues, and async state machines, where correctness IS the interleaving rather than a value. Builds a deterministic interleaving harness (fake timers advanced into the pending window, concurrent launch, microtask stepping) and asserts each guarantee separately, including asymmetric ones where two paths deliberately differ. The falsifier is a test that never interleaved — operations run to completion in sequence exercise no race and produce a vacuous green indistinguishable from a real pass, so the proof obligation is to show the interleaving occurred, not that the assertion passed. Triggers on /race-condition-repro, or when asked to prove a race condition is fixed, test cancellation or supersession, validate retry or debounce ordering, write a deterministic interleaving test, or check whether a concurrency test actually exercises the race. Callable by `evidence` as its deterministic-interleaving engine, and named by `falsifying-test` as its sibling for ordering bugs. +description: Prove an ordering guarantee under concurrency — that when B arrives during A's pending window, A is canceled, or completes first, or the two commit in a defined order. Covers race conditions, retries, cancellation, supersession, debounce/throttle, locks, queues, and async state machines, where correctness IS the interleaving rather than a value. Builds a deterministic interleaving harness (fake timers advanced into the pending window, concurrent launch, microtask stepping) and asserts each guarantee separately, including asymmetric ones where two paths deliberately differ. The falsifier is a test that never interleaved — operations run to completion in sequence exercise no race and produce a vacuous green indistinguishable from a real pass, so the proof obligation is to show the interleaving occurred, not that the assertion passed. Triggers on /mms-race-condition-repro, or when asked to prove a race condition is fixed, test cancellation or supersession, validate retry or debounce ordering, write a deterministic interleaving test, or check whether a concurrency test actually exercises the race. Callable by `evidence` as its deterministic-interleaving engine, and named by `falsifying-test` as its sibling for ordering bugs. maturity: experimental --- From 995f3c83f3ab5008169c31a3e18db97725e0f3e0 Mon Sep 17 00:00:00 2001 From: Jongsun Suh <jongsun.suh@icloud.com> Date: Tue, 4 Aug 2026 08:01:36 -0400 Subject: [PATCH 135/135] Name the installed command in `tsc-blindspots`'s description The installer emits `mms-tsc-blindspots`; the description advertised `/tsc-blindspots`. --- domains/typescript/skills/tsc-blindspots/skill.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/domains/typescript/skills/tsc-blindspots/skill.md b/domains/typescript/skills/tsc-blindspots/skill.md index e8006f05..a62730d9 100644 --- a/domains/typescript/skills/tsc-blindspots/skill.md +++ b/domains/typescript/skills/tsc-blindspots/skill.md @@ -16,7 +16,7 @@ description: >- enum lookups, calls made optional so a throw becomes a silent no-op. Use when reviewing a JS→TS migration, a PR that hand-writes types for values that already have them, a "rename-only" refactor, or any PR claiming a change is mechanical. - Triggers on /tsc-blindspots, or on phrases like "validate this TypeScript + Triggers on /mms-tsc-blindspots, or on phrases like "validate this TypeScript migration", "is this type right", "does this type match the real shape", "why didn't CI catch this type", "derive vs define", and "what can tsc not check".