From 470133b7a7f5d3ca2d99cbe9e543f5664d19f285 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9s=20Pe=C3=B1a?= Date: Wed, 29 Jul 2026 06:03:14 -0600 Subject: [PATCH] feat(audit): ENF audits whether a gate BINDS, not only whether it exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `findHookCoverageGaps` answers "does a hook exist for this rule?" — presence, and the easy half. A gate can be present, correctly named, wired into a config, reviewed and merged, and assert nothing at all. Adds `findHookEfficacyGaps`, which answers the other question, against the seven shapes of a vacuous gate. Shapes 1 and 3 are about the SET of hooks and stay in the coverage verifier; shapes 2 and 4-7 are about EACH hook and are new: 2 looks at the adjacent property asserts-adjacent-property, unresolved-symbol 4 correct and binds nothing non-blocking-exit-code, vacuous-on-empty-input, no-block-assertion 5 correct, blocking, and too late enforced-too-late 6 registered where nothing reads registered-where-nothing-reads 7 the incentive is inverted inverted-incentive Every shape came from a live repository. Three worth naming: - Exit code. Only exit 2 blocks a PreToolUse call; exit 1 is reported and then proceeded past. A repo found twelve hooks denying on exit 1, one of them guarding writes against production. Each was correct, reviewed, and inert. - Registry. A hook wired into `hooks.json` when the harness reads `settings.json` never runs, and its presence in *a* registry reads as covered. - Shape 7 is new and no coverage audit can see it, because the gate exists, looks at the right thing, and binds. A doc validator escalated to ERROR only when a record was `accepted`, degrading `draft`/`review` to a warning — so complying cost more than not complying, and readers were trained to stay lenient. The gate was not weak. It pointed the wrong way. Two shapes are not decidable from a file listing and the detector says so. Shape 2 is checked through its correlate — a suite asserting only the exit code pins nothing about WHAT the gate said — and that gap is labelled `proxy`. Shape 7 needs a reader, because which branch is cheaper depends on the repo. An observation left `undefined` produces no gap: the verifier reports what was looked up, never what was assumed. Refusing to encode the un-automatable is the point. A verifier that guessed shape 7 would be a gate that looks at the adjacent property — shape 2, in the tool built to find shape 2. The four efficacy `high`s share one property the `medium`s do not: each produces a gate that looks like coverage from every listing. A `medium` gap is a gate that is weak; a `high` gap is a gate that is misleading, and the second is worse because it stops anyone from looking again. 20 new cases; the 6 existing coverage cases are untouched and still pass. `vitest run src/audit` — 80 passed across 13 files. Co-Authored-By: Claude Opus 5 (1M context) --- commands/audit-enforcement-hooks.md | 37 ++- references/detectors/enforcement-hooks.md | 125 +++++++- .../enforcement-hooks-efficacy.test.ts | 209 +++++++++++++ .../verifiers/enforcement-hooks-efficacy.ts | 284 ++++++++++++++++++ 4 files changed, 643 insertions(+), 12 deletions(-) create mode 100644 src/audit/verifiers/enforcement-hooks-efficacy.test.ts create mode 100644 src/audit/verifiers/enforcement-hooks-efficacy.ts diff --git a/commands/audit-enforcement-hooks.md b/commands/audit-enforcement-hooks.md index db7962c..ce7cb62 100644 --- a/commands/audit-enforcement-hooks.md +++ b/commands/audit-enforcement-hooks.md @@ -16,14 +16,35 @@ detector profile (`references/detectors/enforcement-hooks.md`) against $ARGUMENTS (default: the current repo). The engine owns the full flow (preflight → scope → detect → verify → cure-map → -emit). Deterministic verification uses `findHookCoverageGaps` from -`src/audit/verifiers/enforcement-hooks.ts` — it takes the `EnforcementConfig` -assembled from Stage 2 (which repo-level hooks exist; are the toolkit hooks -inherited; is there a rules config for them to enforce; which structural rules -the other five audits recommend) and returns one gap per rule that holds -(`no-repo-hooks`, `no-toolkit-hooks`, `no-rules-config`, -`rule-without-hook:`), sorted by `code` — so the result is deterministic -and diffable. +emit). Deterministic verification runs **two** verifiers, because coverage and +efficacy are different questions: + +`findHookCoverageGaps` from `src/audit/verifiers/enforcement-hooks.ts` asks +**does a gate exist for this rule?** — shapes 1 and 3 of the seven vacuous-gate +shapes. + +`findHookEfficacyGaps` from `src/audit/verifiers/enforcement-hooks-efficacy.ts` +asks **does the gate that exists actually bind?** — shapes 2 and 4-7: it looks at +the adjacent property, it denies on an exit code that does not deny, it passes on +an empty input set, it is registered in a file the harness never loads, it names +an escape hatch that does not exist, it runs only in CI, or its incentive is +inverted so complying costs more than not complying. + +A repo with full coverage and zero efficacy gaps has gates that are present AND +binding, which is the only combination worth calling enforcement. + +The coverage verifier takes the `EnforcementConfig` assembled from Stage 2 +(which repo-level hooks exist; are the toolkit hooks inherited; is there a +rules config for them to enforce; which structural rules the other five audits +recommend) and returns one gap per rule that holds (`no-repo-hooks`, +`no-toolkit-hooks`, `no-rules-config`, `rule-without-hook:`), sorted by +`code` — so the result is deterministic and diffable. + +The efficacy verifier takes an `EfficacyConfig` (one +`HookObservation` per gate, plus the registry file the harness actually reads) +and returns gaps stamped with the `shape` they instance and a `confidence` of +`direct` or `proxy`. An observation left `undefined` produces no gap: the +verifier reports what was looked up, never what was assumed. This family is the **meta-audit that closes the loop**: it does not re-detect schema, contract, domain, layering, or migration drift — it audits whether the diff --git a/references/detectors/enforcement-hooks.md b/references/detectors/enforcement-hooks.md index 812e090..f27486b 100644 --- a/references/detectors/enforcement-hooks.md +++ b/references/detectors/enforcement-hooks.md @@ -49,6 +49,57 @@ actually installed to enforce it — detection that will silently drift back. --- +## The other half: a gate can exist and still bind nothing + +Everything above asks **"does a hook exist for this rule?"** — a question about +presence, and the easy half. A gate can be present, correctly named, wired into +a config, reviewed and merged, and assert nothing at all. + +**The seven shapes of a vacuous gate.** Shapes 1 and 3 are questions about the +SET of hooks and are what the coverage check above already answers. Shapes 2 and +4-7 are questions about EACH hook, and each was observed in a live repository. + +| # | Shape | The question to ask | +| - | ----- | ------------------- | +| 1 | It does not look | Is any gate covering this rule? | +| 2 | It looks at the adjacent property | What proposition does it assert — and is that the proposition the rule needs? | +| 3 | It was deleted | Does the rule outlive its gate? | +| 4 | It is correct and binds nothing | Wrong exit code, empty input set, or no test that it ever denies? | +| 5 | It is correct, blocking, and too late | Does a write-time twin exist, or does CI catch what write-time should have? | +| 6 | It is registered where nothing reads | Is it in the file the harness actually loads? | +| 7 | Its incentive is inverted | Is the compliant state cheaper than the non-compliant one? | + +Three of these are worth spelling out, because each cost a real repo months. + +**Shape 4 — the exit code.** In the Claude Code PreToolUse contract only **exit +2** blocks. Exit 1 is an error the harness reports and then proceeds past. A repo +found twelve hooks written to deny on `exit 1`, including one guarding writes +against production. Each was correct, reviewed, and inert. + +**Shape 6 — the registry.** A hook wired into `.claude/hooks/hooks.json` when the +harness reads `.claude/settings.json` never runs. Its presence in *a* registry +file reads as covered, which is why it survives review. + +**Shape 7 — the inverted incentive.** The newest and the one no coverage audit +can see, because the gate exists, looks at the right thing, and binds. A doc +validator escalated to ERROR **only when a record's status was `accepted`**, +degrading `draft` and `review` to a warning. Complying cost more than not +complying, so records stayed unsettled — and every reader who noticed was trained +to stay in the lenient branch. The gate was not weak. It pointed the wrong way. + +**Two of the seven are not decidable from a file listing, and the detector says +so rather than guessing.** Shape 2 is checked through its strongest correlate — +a gate whose tests assert only the exit code has nothing pinning WHAT it said, so +one that silently stopped emitting its message still passes — and that gap is +labelled `proxy`. Shape 7 needs a reader, because "which branch is cheaper" +depends on what the branches cost in that repo. + +Refusing to encode the un-automatable is the point. A verifier that guessed +shape 7 would be a gate that looks at the adjacent property — shape 2, in the +tool built to find shape 2. + +--- + ## Stage 2 — LLM detection prompt Run this prompt against the repo (its `.claude/hooks/` + `hooks.json`, its rules @@ -78,9 +129,38 @@ config, and the confirmed findings of the OTHER five families): > observation without an anchor is invalid — drop it. Flag the rules that have no > backing hook — those are the detection-without-enforcement gaps. +Then, for **each hook found in step 1**, record the facts the binding question +needs. Every one is a lookup, not an inference from the file name: + +> 5. **`registeredIn`** — every file that registers this hook, AND +> `harnessRegistryFile`, the one the harness actually loads. Shape 6 is only +> decidable against that second value; without it, every registration looks +> equally real. +> 6. **`intendsToBlock`** and **`blockingExitCode`** — does it deny or only warn, +> and with what code. A warn-mode gate is not vacuous for being non-blocking; +> that is its design. +> 7. **`testAssertsBlock`** and **`testAssertsMessage`** — does its suite prove a +> violating input is denied, and does it assert the message TEXT? A suite that +> checks only the exit code stays green if the gate stops saying anything. +> 8. **`failsOnEmptyInput`** — does it fail when its own allowlist or config is +> empty? A gate that iterates nothing and exits 0 passes every run while +> checking nothing. +> 9. **`namedSymbols`** — every symbol the gate NAMES and depends on: an escape +> label, a config path, a rule key, an issue ID. For each, record whether it +> **resolves against the system that owns it, by a command that was run**, and +> which system was asked. A gate offering an escape hatch that is not there +> offers no escape. +> 10. **`hasWriteTimeTwin`** — for a CI-enforced rule, does a write-time twin +> exist? Leave `undefined` if you did not check; an unanswered question is not +> a finding. +> 11. **`severityFavorsCompliance`** — is the compliant state at least as cheap as +> the non-compliant one? Read the gate's severity branches and answer. +> `undefined` if nobody read them. + Output: an `EnforcementConfig` object (`repoHooks`, `toolkitHooksPresent`, -`rulesConfigPresent`, `structuralRules`) plus the anchors backing each field, -ready for the deterministic verify in Stage 3. +`rulesConfigPresent`, `structuralRules`) **and** an `EfficacyConfig` +(`hooks: HookObservation[]`, `harnessRegistryFile`), plus the anchors backing +every field, ready for the deterministic verify in Stage 3. --- @@ -111,11 +191,33 @@ inspectable from the repo). includes - a fully-covered setup (≥1 repo hook, toolkit hooks present, rules config present, every structural rule backed by a hook) → `[]`. -3. **Reconcile** the verifier gaps with the Stage 2 observations: each gap must +3. **Call the efficacy verifier** — coverage and efficacy are different + questions and both must run: + ```ts + import { findHookEfficacyGaps } from "../../src/audit/verifiers/enforcement-hooks-efficacy"; + + const efficacy = findHookEfficacyGaps(efficacyConfig); + // -> [{ code: "non-blocking-exit-code:pre-write-x", shape: 4, confidence: "direct", detail: "..." }] + ``` + One gap per condition that holds, each stamped with the `shape` it instances + and a `confidence` of `direct` or `proxy`, sorted by `code`: + - registered outside `harnessRegistryFile` → `registered-where-nothing-reads:` (shape 6) + - PreToolUse denial on an exit code other than 2 → `non-blocking-exit-code:` (shape 4) + - passes on an empty input set → `vacuous-on-empty-input:` (shape 4) + - blocking with no test that it denies → `no-block-assertion:` (shape 4) + - tests assert the exit code only → `asserts-adjacent-property:` (shape 2, **proxy**) + - names a symbol that does not resolve → `unresolved-symbol::` (shape 2) + - CI-only with `hasWriteTimeTwin === false` → `enforced-too-late:` (shape 5) + - `severityFavorsCompliance === false` → `inverted-incentive:` (shape 7) + + An `undefined` observation produces no gap. The verifier reports what was + looked up, never what was assumed. + +4. **Reconcile** the verifier gaps with the Stage 2 observations: each gap must carry the anchor of the signal that produced it (the absent `.claude/hooks/`, the missing rules config, or the confirmed finding whose `hook` cure is unbacked). Drop any gap whose backing observation has no anchor. -4. **Stamp `confidence`:** `confirmed` when `findHookCoverageGaps` returns the +5. **Stamp `confidence`:** `confirmed` when `findHookCoverageGaps` returns the gap AND its backing observation resolves to an anchor (a real `.claude/hooks/` listing, a real rules config path, a real cross-referenced finding-ID); `probable` when a signal is only partially observable (e.g. toolkit-hook @@ -137,6 +239,21 @@ alone"*). Never truncate silently. | `no-rules-config` — no rules config for the hooks to enforce | `high` | | `no-toolkit-hooks` — toolkit enforcement hooks not inherited (Cure 4b) | `medium` | | `rule-without-hook:` — a recommended rule has no backing hook | `medium` | +| `registered-where-nothing-reads:` — inert, and reads as covered | `high` | +| `non-blocking-exit-code:` — denies on a code that does not deny | `high` | +| `unresolved-symbol::` — names something that does not exist | `high` | +| `inverted-incentive:` — complying costs more than not complying | `high` | +| `vacuous-on-empty-input:` — green while checking nothing | `medium` | +| `no-block-assertion:` — no test proves it ever denies | `medium` | +| `asserts-adjacent-property:` — **proxy**, needs a human read | `medium` | +| `enforced-too-late:` — CI catches what write-time should have | `medium` | + +The four efficacy `high`s share one property that the `medium`s do not: each +produces a gate that **looks like coverage from every listing**. An inert hook, +a denial on the wrong exit code, a phantom symbol and an inverted incentive all +survive review, appear in the inventory, and report green. A `medium` gap is a +gate that is weak; a `high` gap is a gate that is misleading, and the second is +worse because it stops anyone from looking again. `no-repo-hooks` and `no-rules-config` are `high` because each removes the enforcement layer wholesale: with no hooks and nothing for them to enforce, every diff --git a/src/audit/verifiers/enforcement-hooks-efficacy.test.ts b/src/audit/verifiers/enforcement-hooks-efficacy.test.ts new file mode 100644 index 0000000..215751a --- /dev/null +++ b/src/audit/verifiers/enforcement-hooks-efficacy.test.ts @@ -0,0 +1,209 @@ +import { describe, it, expect } from "vitest"; +import { + findHookEfficacyGaps, + VACUOUS_GATE_SHAPES, + type EfficacyConfig, + type HookObservation, +} from "./enforcement-hooks-efficacy"; + +/** + * Each case below is a shape observed in a live repository, reduced to its + * observation. The point of the suite is not that the function returns an + * array — it is that a gate which LOOKS fine in every listing still produces a + * gap when the binding question is asked. + */ + +/** A gate that binds: right event, right exit code, tested for both block and message. */ +const binding: HookObservation = { + name: "pre-write-ownership", + event: "PreToolUse", + registeredIn: [".claude/settings.json"], + intendsToBlock: true, + blockingExitCode: 2, + testAssertsMessage: true, + testAssertsBlock: true, + failsOnEmptyInput: true, + severityFavorsCompliance: true, +}; + +const base: EfficacyConfig = { + hooks: [binding], + harnessRegistryFile: ".claude/settings.json", +}; + +const only = (c: EfficacyConfig) => findHookEfficacyGaps(c).map((g) => g.code); +const withHook = (patch: Partial): EfficacyConfig => ({ + ...base, + hooks: [{ ...binding, ...patch }], +}); + +describe("a gate that binds produces no gaps", () => { + it("returns [] for the fully-binding observation", () => { + expect(findHookEfficacyGaps(base)).toEqual([]); + }); +}); + +describe("shape 6 — registered where nothing reads", () => { + it("flags a hook registered only in a file the harness does not load", () => { + const gaps = findHookEfficacyGaps( + withHook({ registeredIn: [".claude/hooks/hooks.json"] }), + ); + expect(gaps.map((g) => g.code)).toEqual([ + "registered-where-nothing-reads:pre-write-ownership", + ]); + expect(gaps[0].shape).toBe(6); + expect(gaps[0].confidence).toBe("direct"); + }); + + it("flags a hook registered nowhere at all", () => { + expect(only(withHook({ registeredIn: [] }))).toContain( + "registered-where-nothing-reads:pre-write-ownership", + ); + }); + + it("stays silent when the harness registry file is unknown — an unanswerable question is not a finding", () => { + const gaps = findHookEfficacyGaps({ + hooks: [{ ...binding, registeredIn: [".claude/hooks/hooks.json"] }], + harnessRegistryFile: undefined, + }); + expect(gaps).toEqual([]); + }); +}); + +describe("shape 4 — correct and binds nothing", () => { + it("flags a PreToolUse denial on exit 1, which does not block", () => { + const gaps = findHookEfficacyGaps(withHook({ blockingExitCode: 1 })); + expect(gaps.map((g) => g.code)).toEqual([ + "non-blocking-exit-code:pre-write-ownership", + ]); + expect(gaps[0].shape).toBe(4); + }); + + it("does NOT flag exit 1 on a warn-mode gate — non-blocking is its design, not its defect", () => { + expect( + only(withHook({ intendsToBlock: false, blockingExitCode: 1 })), + ).toEqual([]); + }); + + it("does not flag exit 1 outside PreToolUse, where the contract differs", () => { + expect(only(withHook({ event: "PreCommit", blockingExitCode: 1 }))).toEqual( + [], + ); + }); + + it("flags a gate that passes clean when its input set is empty", () => { + expect(only(withHook({ failsOnEmptyInput: false }))).toEqual([ + "vacuous-on-empty-input:pre-write-ownership", + ]); + }); + + it("flags a blocking gate with no test proving it ever denies", () => { + expect(only(withHook({ testAssertsBlock: false }))).toEqual([ + "no-block-assertion:pre-write-ownership", + ]); + }); +}); + +describe("shape 2 — asserts the adjacent property", () => { + it("flags exit-code-only tests, and labels the check a proxy", () => { + const gaps = findHookEfficacyGaps(withHook({ testAssertsMessage: false })); + expect(gaps.map((g) => g.code)).toEqual([ + "asserts-adjacent-property:pre-write-ownership", + ]); + expect(gaps[0].confidence).toBe("proxy"); + expect(gaps[0].detail).toMatch(/^PROXY/); + }); + + it("flags a symbol the gate names that does not resolve", () => { + const gaps = findHookEfficacyGaps( + withHook({ + namedSymbols: [ + { symbol: "openspec-exempt", resolves: false, owningSystem: "the repo label set" }, + { symbol: "hardening", resolves: true }, + ], + }), + ); + expect(gaps.map((g) => g.code)).toEqual([ + "unresolved-symbol:pre-write-ownership:openspec-exempt", + ]); + expect(gaps[0].detail).toContain("the repo label set"); + }); + + it("a resolving symbol produces nothing", () => { + expect( + only(withHook({ namedSymbols: [{ symbol: "hardening", resolves: true }] })), + ).toEqual([]); + }); +}); + +describe("shape 5 — correct, blocking, and too late", () => { + it("flags a CI-only rule with no write-time twin", () => { + const gaps = findHookEfficacyGaps( + withHook({ event: "CI", hasWriteTimeTwin: false, blockingExitCode: 1 }), + ); + expect(gaps.map((g) => g.code)).toEqual(["enforced-too-late:pre-write-ownership"]); + expect(gaps[0].shape).toBe(5); + }); + + it("does not flag a CI rule that HAS a write-time twin", () => { + expect( + only(withHook({ event: "CI", hasWriteTimeTwin: true, blockingExitCode: 1 })), + ).toEqual([]); + }); + + it("stays silent when the twin is unknown rather than guessing", () => { + expect( + only(withHook({ event: "CI", hasWriteTimeTwin: undefined, blockingExitCode: 1 })), + ).toEqual([]); + }); +}); + +describe("shape 7 — the incentive is inverted", () => { + it("flags a gate whose strictest outcome is reserved for the compliant state", () => { + const gaps = findHookEfficacyGaps( + withHook({ severityFavorsCompliance: false }), + ); + expect(gaps.map((g) => g.code)).toEqual(["inverted-incentive:pre-write-ownership"]); + expect(gaps[0].shape).toBe(7); + expect(gaps[0].detail).toContain("complying costs more"); + }); + + it("stays silent when nobody has made the observation — this shape is read, not computed", () => { + expect(only(withHook({ severityFavorsCompliance: undefined }))).toEqual([]); + }); +}); + +describe("determinism and shape coverage", () => { + it("sorts by code so runs are diffable", () => { + const gaps = findHookEfficacyGaps({ + hooks: [ + { ...binding, name: "z-hook", failsOnEmptyInput: false }, + { ...binding, name: "a-hook", testAssertsMessage: false }, + ], + harnessRegistryFile: ".claude/settings.json", + }); + expect(gaps.map((g) => g.code)).toEqual([ + "asserts-adjacent-property:a-hook", + "vacuous-on-empty-input:z-hook", + ]); + }); + + it("one hook can be several shapes at once", () => { + const gaps = findHookEfficacyGaps( + withHook({ + registeredIn: [".claude/hooks/hooks.json"], + blockingExitCode: 1, + failsOnEmptyInput: false, + testAssertsBlock: false, + testAssertsMessage: false, + severityFavorsCompliance: false, + }), + ); + expect(new Set(gaps.map((g) => g.shape))).toEqual(new Set([2, 4, 6, 7])); + }); + + it("names all seven shapes, including the one the coverage verifier owns", () => { + expect(Object.keys(VACUOUS_GATE_SHAPES)).toHaveLength(7); + expect(VACUOUS_GATE_SHAPES[7]).toContain("complying costs more"); + }); +}); diff --git a/src/audit/verifiers/enforcement-hooks-efficacy.ts b/src/audit/verifiers/enforcement-hooks-efficacy.ts new file mode 100644 index 0000000..ae43503 --- /dev/null +++ b/src/audit/verifiers/enforcement-hooks-efficacy.ts @@ -0,0 +1,284 @@ +/** + * Enforcement-hook EFFICACY — the seven shapes of a vacuous gate. + * + * `enforcement-hooks.ts` answers **"does a hook exist for this rule?"**. That is + * a question about presence, and it is the easy half. This file answers the + * other one: + * + * Does the hook that exists actually BIND? + * + * A gate can be present, correctly named, wired into a config, reviewed and + * merged — and assert nothing. Every shape below was observed in a live + * repository, which is why each carries a mechanical question rather than a + * warning. A shape with no question attached is a lecture; a shape with a + * question is a check. + * + * --------------------------------------------------------------------------- + * THE SEVEN SHAPES + * --------------------------------------------------------------------------- + * 1. It does not look. covered by `findHookCoverageGaps` + * 2. It looks at the adjacent property. + * 3. It was deleted. covered by `findHookCoverageGaps` + * 4. It is correct and binds nothing. + * 5. It is correct, blocking, and too late. + * 6. It is registered where nothing reads. + * 7. Its incentive is inverted. + * + * Shapes 1 and 3 are questions about the SET of hooks, so they stay in the + * coverage verifier. Shapes 2 and 4-7 are questions about EACH hook, so they + * need a per-hook observation and live here. + * + * --------------------------------------------------------------------------- + * WHY SOME SHAPES ARE PROXIES AND SAY SO + * --------------------------------------------------------------------------- + * Shape 2 — "does the hook assert the proposition it claims?" — is not + * decidable from a file listing. What IS decidable is its strongest known + * correlate: a hook whose tests check only the exit code has no assertion about + * WHAT it said, so a hook that silently stopped emitting its message still + * passes. That is a proxy, it is labelled a proxy in the gap detail, and it is + * the reason this verifier reports `confidence` rather than a verdict. + * + * Shape 7 is the newest and the least automatable: a gate whose lenient branch + * is cheaper than its strict one trains people to stay lenient. The observation + * `severityFavorsCompliance` must be made by a reader, because "which branch is + * cheaper" depends on what the branches cost in that repo. The verifier records + * the answer; it does not compute it. + * + * Refusing to encode the un-automatable is the point. A verifier that guessed + * shape 7 would be a gate that looks at the adjacent property — shape 2, in the + * tool built to find shape 2. + */ + +/** The lifecycle event a gate runs on. */ +export type HookEvent = + | "PreToolUse" + | "PostToolUse" + | "SessionStart" + | "UserPromptSubmit" + | "PreCommit" + | "PrePush" + | "CI" + | "other"; + +/** A symbol a gate NAMES and therefore depends on: a label, a path, a config key, an issue ID. */ +export interface NamedSymbol { + symbol: string; + /** Did it resolve against the system that owns it, by a command that was run? */ + resolves: boolean; + /** Which system was asked. Recorded so a `false` can be re-checked rather than re-argued. */ + owningSystem?: string; +} + +/** + * One gate, as observed. Every field is a fact someone looked up — not an + * inference from the file name. + */ +export interface HookObservation { + name: string; + event: HookEvent; + + /** + * Files that register this hook. Shape 6 compares this against the file the + * harness actually reads: a hook registered only in a settings file nobody + * loads is inert, and it is inert in a way that reads as covered. + */ + registeredIn: string[]; + + /** Does this gate intend to DENY, or only to warn? A warn-mode gate is not vacuous for being non-blocking. */ + intendsToBlock: boolean; + + /** + * The exit code the gate uses to deny. Shape 4: in the Claude Code PreToolUse + * contract only exit 2 blocks — exit 1 is an error the harness reports and + * then proceeds past. A blocking gate on exit 1 is correct, reviewed, and + * inert. + */ + blockingExitCode?: number; + + /** Do its tests assert the message TEXT, or only the exit code? Proxy for shape 2. */ + testAssertsMessage: boolean; + + /** Do its tests assert that a violating input is actually blocked? */ + testAssertsBlock: boolean; + + /** + * Does the gate fail when its own input set is empty? Shape 4: a gate that + * iterates an empty allowlist and exits 0 passes every run while checking + * nothing, and its green is indistinguishable from real coverage. + */ + failsOnEmptyInput: boolean; + + /** Symbols this gate names and depends on. Any that does not resolve is a phantom. */ + namedSymbols?: NamedSymbol[]; + + /** + * For a rule enforced in CI: does a write-time twin exist? Shape 5 — a + * violation caught in CI is caught after it was written, copied and opened as + * a PR. Correct, blocking, and too late. + */ + hasWriteTimeTwin?: boolean; + + /** + * Shape 7. `true` when the COMPLIANT state is at least as cheap as the + * non-compliant one. `false` means the gate punishes compliance — the + * strictest outcome is reserved for the state you want people to reach. + * Observed by a reader, never computed here. + */ + severityFavorsCompliance?: boolean; +} + +export interface EfficacyConfig { + hooks: HookObservation[]; + /** + * The registry file the harness actually reads. Shape 6 is only decidable + * against this: without it, every registration looks equally real. + */ + harnessRegistryFile?: string; +} + +export interface EfficacyGap { + code: string; + detail: string; + /** Which of the seven shapes this gap is an instance of. */ + shape: 2 | 4 | 5 | 6 | 7; + /** `proxy` when the check is a correlate rather than the proposition itself. */ + confidence: "direct" | "proxy"; +} + +/** + * Find the gaps where a gate exists but does not bind. + * + * Complements `findHookCoverageGaps`, which owns shapes 1 and 3. A repo with + * full coverage and zero efficacy gaps has gates that are present AND binding — + * which is the only combination worth calling enforcement. + * + * Output is sorted by `code`, so it is deterministic and diffable across runs. + */ +export function findHookEfficacyGaps(c: EfficacyConfig): EfficacyGap[] { + const gaps: EfficacyGap[] = []; + const PRE_TOOL_BLOCKING_EXIT = 2; + + for (const h of c.hooks) { + // --- Shape 6: registered where nothing reads -------------------------- + if (c.harnessRegistryFile && !h.registeredIn.includes(c.harnessRegistryFile)) { + gaps.push({ + code: `registered-where-nothing-reads:${h.name}`, + shape: 6, + confidence: "direct", + detail: + `"${h.name}" is registered in [${h.registeredIn.join(", ") || "nothing"}] ` + + `but the harness reads "${c.harnessRegistryFile}". The gate is inert, ` + + `and its presence in a registry file reads as covered.`, + }); + } + + // --- Shape 4a: a blocking gate on a non-blocking exit code ------------ + if ( + h.intendsToBlock && + h.event === "PreToolUse" && + h.blockingExitCode !== undefined && + h.blockingExitCode !== PRE_TOOL_BLOCKING_EXIT + ) { + gaps.push({ + code: `non-blocking-exit-code:${h.name}`, + shape: 4, + confidence: "direct", + detail: + `"${h.name}" intends to deny but exits ${h.blockingExitCode}. Only exit ` + + `${PRE_TOOL_BLOCKING_EXIT} blocks a PreToolUse call; anything else is ` + + `reported and then proceeded past.`, + }); + } + + // --- Shape 4b: passes vacuously when its own input is empty ----------- + if (!h.failsOnEmptyInput) { + gaps.push({ + code: `vacuous-on-empty-input:${h.name}`, + shape: 4, + confidence: "direct", + detail: + `"${h.name}" exits clean when it has nothing to check. An empty run ` + + `that passes is indistinguishable from real coverage.`, + }); + } + + // --- Shape 4c: no test proves it ever blocks -------------------------- + if (h.intendsToBlock && !h.testAssertsBlock) { + gaps.push({ + code: `no-block-assertion:${h.name}`, + shape: 4, + confidence: "direct", + detail: + `No test for "${h.name}" proves a violating input is denied. Its suite ` + + `would stay green if the gate stopped denying.`, + }); + } + + // --- Shape 2: asserts the adjacent property (proxy) ------------------- + if (!h.testAssertsMessage) { + gaps.push({ + code: `asserts-adjacent-property:${h.name}`, + shape: 2, + confidence: "proxy", + detail: + `PROXY — tests for "${h.name}" assert an exit code but never the message ` + + `text, so nothing pins WHAT it asserts. Read the gate and name the ` + + `proposition it checks; confirm that is the proposition the rule needs.`, + }); + } + + // --- Shape 2/3 variant: it names something that does not exist -------- + for (const s of h.namedSymbols ?? []) { + if (!s.resolves) { + gaps.push({ + code: `unresolved-symbol:${h.name}:${s.symbol}`, + shape: 2, + confidence: "direct", + detail: + `"${h.name}" names "${s.symbol}"${s.owningSystem ? ` and ${s.owningSystem} does not have it` : ", which does not resolve"}. ` + + `A gate offering an escape hatch that is not there offers no escape, ` + + `and a rule keyed on a phantom fires on nothing.`, + }); + } + } + + // --- Shape 5: correct, blocking, and too late ------------------------- + if (h.event === "CI" && h.hasWriteTimeTwin === false) { + gaps.push({ + code: `enforced-too-late:${h.name}`, + shape: 5, + confidence: "direct", + detail: + `"${h.name}" runs only in CI with no write-time twin. It catches the ` + + `violation after it was written, copied and opened as a PR.`, + }); + } + + // --- Shape 7: the incentive is inverted ------------------------------- + if (h.severityFavorsCompliance === false) { + gaps.push({ + code: `inverted-incentive:${h.name}`, + shape: 7, + confidence: "direct", + detail: + `"${h.name}" reserves its strictest outcome for the COMPLIANT state, so ` + + `complying costs more than not complying. The gate is not weak — it ` + + `points the wrong way, and every reader who notices is trained to stay ` + + `in the lenient branch.`, + }); + } + } + + return gaps.sort((a, b) => a.code.localeCompare(b.code)); +} + +/** The seven shapes, as the question each one asks. Exported so the report can print them. */ +export const VACUOUS_GATE_SHAPES: Record = { + 1: "It does not look — no gate covers the rule.", + 2: "It looks at the adjacent property — it asserts something true and beside the point.", + 3: "It was deleted — the rule survives, the gate does not.", + 4: "It is correct and binds nothing — wrong exit code, empty input, or no test that it ever denies.", + 5: "It is correct, blocking, and too late — CI catches what write-time should have.", + 6: "It is registered where nothing reads — present in a file the harness does not load.", + 7: "Its incentive is inverted — complying costs more than not complying.", +};