feat(audit): ENF audits whether a gate BINDS, not only whether it exists - #52
feat(audit): ENF audits whether a gate BINDS, not only whether it exists#52lapc506 wants to merge 1 commit into
Conversation
`findHookCoverageGaps` answers "does a hook exist for this rule?" — presence,
and the easy half. A gate can be present, correctly named, wired into a config,
reviewed and merged, and assert nothing at all.
Adds `findHookEfficacyGaps`, which answers the other question, against the seven
shapes of a vacuous gate. Shapes 1 and 3 are about the SET of hooks and stay in
the coverage verifier; shapes 2 and 4-7 are about EACH hook and are new:
2 looks at the adjacent property asserts-adjacent-property, unresolved-symbol
4 correct and binds nothing non-blocking-exit-code, vacuous-on-empty-input,
no-block-assertion
5 correct, blocking, and too late enforced-too-late
6 registered where nothing reads registered-where-nothing-reads
7 the incentive is inverted inverted-incentive
Every shape came from a live repository. Three worth naming:
- Exit code. Only exit 2 blocks a PreToolUse call; exit 1 is reported and then
proceeded past. A repo found twelve hooks denying on exit 1, one of them
guarding writes against production. Each was correct, reviewed, and inert.
- Registry. A hook wired into `hooks.json` when the harness reads
`settings.json` never runs, and its presence in *a* registry reads as covered.
- Shape 7 is new and no coverage audit can see it, because the gate exists,
looks at the right thing, and binds. A doc validator escalated to ERROR only
when a record was `accepted`, degrading `draft`/`review` to a warning — so
complying cost more than not complying, and readers were trained to stay
lenient. The gate was not weak. It pointed the wrong way.
Two shapes are not decidable from a file listing and the detector says so.
Shape 2 is checked through its correlate — a suite asserting only the exit code
pins nothing about WHAT the gate said — and that gap is labelled `proxy`.
Shape 7 needs a reader, because which branch is cheaper depends on the repo. An
observation left `undefined` produces no gap: the verifier reports what was
looked up, never what was assumed.
Refusing to encode the un-automatable is the point. A verifier that guessed
shape 7 would be a gate that looks at the adjacent property — shape 2, in the
tool built to find shape 2.
The four efficacy `high`s share one property the `medium`s do not: each produces
a gate that looks like coverage from every listing. A `medium` gap is a gate
that is weak; a `high` gap is a gate that is misleading, and the second is worse
because it stops anyone from looking again.
20 new cases; the 6 existing coverage cases are untouched and still pass.
`vitest run src/audit` — 80 passed across 13 files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![dojo-code-reviewer[bot]](https://avatars.githubusercontent.com/in/3092887?s=60&v=4){kind=small}
There was a problem hiding this comment.
💬 Review Comments
Comments — 0 blockers, 12 P2. Confidence: 1.00/5.00.
Walkthrough
Governance Warning
main branch directly. Per standard GitFlow practices, feature branches should target develop before merging to main. This warning is informational only and does not block the review verdict.
Review Walkthrough
This PR introduces the enforcement hook efficacy verifier (findHookEfficacyGaps) to identify "vacuous gates" that appear covered but fail to bind. It implements checks for five of the seven vacuous-gate shapes: registration mismatch, non-blocking exit codes, empty input bypass, lack of test assertions, unresolved symbols, delayed CI enforcement, and inverted incentives.
Scope of Review
I reviewed the newly added verifier file (src/audit/verifiers/enforcement-hooks-efficacy.ts), its corresponding unit test suite (src/audit/verifiers/enforcement-hooks-efficacy.test.ts), and the updated markdown documentation detailing the seven shapes and stage-2 observations (references/detectors/enforcement-hooks.md and commands/audit-enforcement-hooks.md).
Safety Rationale
This PR is safe to merge because the new verifier code is self-contained, does not modify any existing production code or execution paths, and passes all its newly introduced unit tests.
Verdict
Comment — 0 blockers, 12 P2.
🟡 P2 — Major
src/audit/verifiers/enforcement-hooks-efficacy.ts:67— 🟡 P2 (major) — Makeresolvesoptional to prevent false-positive gap findings when symbol resolution is not observed/unknown. Ifresolvesisundefined, it should produce no gaps under the 'An observation left undefined produces no gap' contract.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:87— 🟡 P2 (major) — MakeregisteredInoptional to allow omitting registration info without triggering a false gap or causing runtime type errors when parsing dynamic or incomplete observation data.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:90— 🟡 P2 (major) — MakeintendsToBlockoptional to correctly handle the case where it is unknown or not observed.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:101— 🟡 P2 (major) — MaketestAssertsMessageoptional so that an unobserved or unknown message test assertion state does not trigger a false-positiveasserts-adjacent-propertygap.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:104— 🟡 P2 (major) — MaketestAssertsBlockoptional so that an unobserved/unknown block test assertion state does not trigger a false-positiveno-block-assertiongap.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:111— 🟡 P2 (major) — MakefailsOnEmptyInputoptional so that an unobserved/unknown empty-input failure behavior does not trigger a false-positivevacuous-on-empty-inputgap.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:153— 🟡 P2 (major) — Guard the registry file check to prevent a runtime TypeError ifh.registeredInis undefined, and ensure it produces no gaps when unobserved.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:167— 🟡 P2 (major) — Use an explicit strict boolean equality check forh.intendsToBlockso thatundefinedvalues do not inadvertently pass.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:184— 🟡 P2 (major) — Use an explicit strict check forfailsOnEmptyInput === falseso thatundefined(unobserved) values are ignored and do not trigger a false gap.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:195— 🟡 P2 (major) — Use explicit strict checks forintendsToBlock === trueandtestAssertsBlock === falseto properly respectundefined(unobserved) inputs.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:206— 🟡 P2 (major) — Use an explicit strict check fortestAssertsMessage === falseso thatundefined(unobserved) values do not trigger a false-positive gap.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:218— 🟡 P2 (major) — Use an explicit strict check fors.resolves === falseto ensure an unobserved resolution status does not trigger a false gap.
[pass 1]
Total findings: 12 compliance (12 total)
| export interface NamedSymbol { | ||
| symbol: string; | ||
| /** Did it resolve against the system that owns it, by a command that was run? */ | ||
| resolves: boolean; |
There was a problem hiding this comment.
🟡 P2 (major) — Make resolves optional to prevent false-positive gap findings when symbol resolution is not observed/unknown. If resolves is undefined, it should produce no gaps under the 'An observation left undefined produces no gap' contract.
[pass 1]
| */ | ||
| registeredIn: string[]; | ||
|
|
||
| /** Does this gate intend to DENY, or only to warn? A warn-mode gate is not vacuous for being non-blocking. */ |
There was a problem hiding this comment.
🟡 P2 (major) — Make registeredIn optional to allow omitting registration info without triggering a false gap or causing runtime type errors when parsing dynamic or incomplete observation data.
[pass 1]
| /** Does this gate intend to DENY, or only to warn? A warn-mode gate is not vacuous for being non-blocking. */ | ||
| intendsToBlock: boolean; | ||
|
|
||
| /** |
There was a problem hiding this comment.
🟡 P2 (major) — Make intendsToBlock optional to correctly handle the case where it is unknown or not observed.
[pass 1]
| /** Do its tests assert the message TEXT, or only the exit code? Proxy for shape 2. */ | ||
| testAssertsMessage: boolean; | ||
|
|
||
| /** Do its tests assert that a violating input is actually blocked? */ |
There was a problem hiding this comment.
🟡 P2 (major) — Make testAssertsMessage optional so that an unobserved or unknown message test assertion state does not trigger a false-positive asserts-adjacent-property gap.
[pass 1]
| /** Do its tests assert that a violating input is actually blocked? */ | ||
| testAssertsBlock: boolean; | ||
|
|
||
| /** |
There was a problem hiding this comment.
🟡 P2 (major) — Make testAssertsBlock optional so that an unobserved/unknown block test assertion state does not trigger a false-positive no-block-assertion gap.
[pass 1]
| gaps.push({ | ||
| code: `registered-where-nothing-reads:${h.name}`, | ||
| shape: 6, | ||
| confidence: "direct", |
There was a problem hiding this comment.
🟡 P2 (major) — Use an explicit strict boolean equality check for h.intendsToBlock so that undefined values do not inadvertently pass.
[pass 1]
| ) { | ||
| gaps.push({ | ||
| code: `non-blocking-exit-code:${h.name}`, | ||
| shape: 4, |
There was a problem hiding this comment.
🟡 P2 (major) — Use an explicit strict check for failsOnEmptyInput === false so that undefined (unobserved) values are ignored and do not trigger a false gap.
[pass 1]
|
|
||
| // --- Shape 4b: passes vacuously when its own input is empty ----------- | ||
| if (!h.failsOnEmptyInput) { | ||
| gaps.push({ |
There was a problem hiding this comment.
🟡 P2 (major) — Use explicit strict checks for intendsToBlock === true and testAssertsBlock === false to properly respect undefined (unobserved) inputs.
[pass 1]
| } | ||
|
|
||
| // --- Shape 4c: no test proves it ever blocks -------------------------- | ||
| if (h.intendsToBlock && !h.testAssertsBlock) { |
There was a problem hiding this comment.
🟡 P2 (major) — Use an explicit strict check for testAssertsMessage === false so that undefined (unobserved) values do not trigger a false-positive gap.
[pass 1]
| } | ||
|
|
||
| // --- Shape 2: asserts the adjacent property (proxy) ------------------- | ||
| if (!h.testAssertsMessage) { |
There was a problem hiding this comment.
🟡 P2 (major) — Use an explicit strict check for s.resolves === false to ensure an unobserved resolution status does not trigger a false gap.
[pass 1]
1 participant
Problem
findHookCoverageGapsanswers "does a hook exist for this rule?" — presence,and the easy half. A gate can be present, correctly named, wired into a config,
reviewed and merged, and assert nothing at all.
The ENF family calls itself "the meta-audit that closes the loop", but the loop
it closes is coverage. A repo can score zero coverage gaps and have an
enforcement layer that binds nothing.
What this adds
findHookEfficacyGaps— the other question, against the seven shapes of avacuous gate. Shapes 1 and 3 are about the SET of hooks and stay in the
coverage verifier. Shapes 2 and 4-7 are about EACH hook and are new:
asserts-adjacent-property,unresolved-symbolnon-blocking-exit-code,vacuous-on-empty-input,no-block-assertionenforced-too-lateregistered-where-nothing-readsinverted-incentiveEvery shape came from a live repository
Shape 4 — the exit code. Only exit 2 blocks a PreToolUse call; exit 1 is
an error the harness reports and then proceeds past. A repo found twelve
hooks written to deny on
exit 1, one of them guarding writes againstproduction. Each was correct, reviewed, and inert.
Shape 6 — the registry. A hook wired into
.claude/hooks/hooks.jsonwhen theharness reads
.claude/settings.jsonnever runs. Its presence in a registryfile reads as covered, which is why it survives review.
Shape 7 — new, and invisible to any coverage audit, because the gate exists,
looks at the right thing, and binds. A doc validator escalated to ERROR only
when a record's status was
accepted, degradingdraftandreviewto awarning. Complying cost more than not complying, so records stayed unsettled and
every reader who noticed was trained to stay in the lenient branch. The gate was
not weak. It pointed the wrong way.
Two shapes are not automatable, and the detector says so
Shape 2 — "does the gate assert the proposition it claims?" — is not decidable
from a file listing. What is decidable is its strongest correlate: a suite that
checks only the exit code pins nothing about WHAT the gate said, so one that
silently stopped emitting its message still passes. That gap is stamped
confidence: "proxy"and its detail starts withPROXY.Shape 7 needs a reader, because which branch is cheaper depends on what the
branches cost in that repo. The verifier records the answer; it does not compute
it.
An observation left
undefinedproduces no gap. The verifier reports whatwas looked up, never what was assumed.
Severity
The four efficacy
highs share one property themediums do not: each producesa gate that looks like coverage from every listing. An inert hook, a denial
on the wrong exit code, a phantom symbol and an inverted incentive all survive
review, appear in the inventory, and report green.
A
mediumgap is a gate that is weak. Ahighgap is a gate that ismisleading, and the second is worse because it stops anyone from looking again.
Verification
The new suite asserts the behaviours that make the verifier honest, not just
that it returns an array: a warn-mode gate is not flagged for being
non-blocking; an unknown harness registry file produces no shape-6 finding
rather than a guess; an unread severity branch produces no shape-7 finding;
and one hook can instance several shapes at once.
Files
src/audit/verifiers/enforcement-hooks-efficacy.tssrc/audit/verifiers/enforcement-hooks-efficacy.test.tsreferences/detectors/enforcement-hooks.mdcommands/audit-enforcement-hooks.mdfindHookCoverageGapsand its six tests are untouched — coverage and efficacyare different questions and both run.
Created by Claude Code on behalf of @kvttvrsis