Skip to content

feat(audit): ENF audits whether a gate BINDS, not only whether it exists #52

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lapc506 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(audit): ENF audits whether a gate BINDS, not only whether it exists #52

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 29 additions & 8 deletions commands/audit-enforcement-hooks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,35 @@ detector profile (`references/detectors/enforcement-hooks.md`) against
$ARGUMENTS (default: the current repo).

The engine owns the full flow (preflight → scope → detect → verify → cure-map →
emit). Deterministic verification uses `findHookCoverageGaps` from
`src/audit/verifiers/enforcement-hooks.ts` — it takes the `EnforcementConfig`
assembled from Stage 2 (which repo-level hooks exist; are the toolkit hooks
inherited; is there a rules config for them to enforce; which structural rules
the other five audits recommend) and returns one gap per rule that holds
(`no-repo-hooks`, `no-toolkit-hooks`, `no-rules-config`,
`rule-without-hook:<rule>`), sorted by `code` — so the result is deterministic
and diffable.
emit). Deterministic verification runs **two** verifiers, because coverage and
efficacy are different questions:

`findHookCoverageGaps` from `src/audit/verifiers/enforcement-hooks.ts` asks
**does a gate exist for this rule?** — shapes 1 and 3 of the seven vacuous-gate
shapes.

`findHookEfficacyGaps` from `src/audit/verifiers/enforcement-hooks-efficacy.ts`
asks **does the gate that exists actually bind?** — shapes 2 and 4-7: it looks at
the adjacent property, it denies on an exit code that does not deny, it passes on
an empty input set, it is registered in a file the harness never loads, it names
an escape hatch that does not exist, it runs only in CI, or its incentive is
inverted so complying costs more than not complying.

A repo with full coverage and zero efficacy gaps has gates that are present AND
binding, which is the only combination worth calling enforcement.

The coverage verifier takes the `EnforcementConfig` assembled from Stage 2
(which repo-level hooks exist; are the toolkit hooks inherited; is there a
rules config for them to enforce; which structural rules the other five audits
recommend) and returns one gap per rule that holds (`no-repo-hooks`,
`no-toolkit-hooks`, `no-rules-config`, `rule-without-hook:<rule>`), sorted by
`code` — so the result is deterministic and diffable.

The efficacy verifier takes an `EfficacyConfig` (one
`HookObservation` per gate, plus the registry file the harness actually reads)
and returns gaps stamped with the `shape` they instance and a `confidence` of
`direct` or `proxy`. An observation left `undefined` produces no gap: the
verifier reports what was looked up, never what was assumed.

This family is the **meta-audit that closes the loop**: it does not re-detect
schema, contract, domain, layering, or migration drift — it audits whether the
Expand Down
125 changes: 121 additions & 4 deletions references/detectors/enforcement-hooks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,57 @@ actually installed to enforce it — detection that will silently drift back.

---

## The other half: a gate can exist and still bind nothing

Everything above asks **"does a hook exist for this rule?"** — a question about
presence, and the easy half. A gate can be present, correctly named, wired into
a config, reviewed and merged, and assert nothing at all.

**The seven shapes of a vacuous gate.** Shapes 1 and 3 are questions about the
SET of hooks and are what the coverage check above already answers. Shapes 2 and
4-7 are questions about EACH hook, and each was observed in a live repository.

| # | Shape | The question to ask |
| - | ----- | ------------------- |
| 1 | It does not look | Is any gate covering this rule? |
| 2 | It looks at the adjacent property | What proposition does it assert — and is that the proposition the rule needs? |
| 3 | It was deleted | Does the rule outlive its gate? |
| 4 | It is correct and binds nothing | Wrong exit code, empty input set, or no test that it ever denies? |
| 5 | It is correct, blocking, and too late | Does a write-time twin exist, or does CI catch what write-time should have? |
| 6 | It is registered where nothing reads | Is it in the file the harness actually loads? |
| 7 | Its incentive is inverted | Is the compliant state cheaper than the non-compliant one? |

Three of these are worth spelling out, because each cost a real repo months.

**Shape 4 — the exit code.** In the Claude Code PreToolUse contract only **exit
2** blocks. Exit 1 is an error the harness reports and then proceeds past. A repo
found twelve hooks written to deny on `exit 1`, including one guarding writes
against production. Each was correct, reviewed, and inert.

**Shape 6 — the registry.** A hook wired into `.claude/hooks/hooks.json` when the
harness reads `.claude/settings.json` never runs. Its presence in *a* registry
file reads as covered, which is why it survives review.

**Shape 7 — the inverted incentive.** The newest and the one no coverage audit
can see, because the gate exists, looks at the right thing, and binds. A doc
validator escalated to ERROR **only when a record's status was `accepted`**,
degrading `draft` and `review` to a warning. Complying cost more than not
complying, so records stayed unsettled — and every reader who noticed was trained
to stay in the lenient branch. The gate was not weak. It pointed the wrong way.

**Two of the seven are not decidable from a file listing, and the detector says
so rather than guessing.** Shape 2 is checked through its strongest correlate —
a gate whose tests assert only the exit code has nothing pinning WHAT it said, so
one that silently stopped emitting its message still passes — and that gap is
labelled `proxy`. Shape 7 needs a reader, because "which branch is cheaper"
depends on what the branches cost in that repo.

Refusing to encode the un-automatable is the point. A verifier that guessed
shape 7 would be a gate that looks at the adjacent property — shape 2, in the
tool built to find shape 2.

---

## Stage 2 — LLM detection prompt

Run this prompt against the repo (its `.claude/hooks/` + `hooks.json`, its rules
Expand Down Expand Up @@ -78,9 +129,38 @@ config, and the confirmed findings of the OTHER five families):
> observation without an anchor is invalid — drop it. Flag the rules that have no
> backing hook — those are the detection-without-enforcement gaps.

Then, for **each hook found in step 1**, record the facts the binding question
needs. Every one is a lookup, not an inference from the file name:

> 5. **`registeredIn`** — every file that registers this hook, AND
> `harnessRegistryFile`, the one the harness actually loads. Shape 6 is only
> decidable against that second value; without it, every registration looks
> equally real.
> 6. **`intendsToBlock`** and **`blockingExitCode`** — does it deny or only warn,
> and with what code. A warn-mode gate is not vacuous for being non-blocking;
> that is its design.
> 7. **`testAssertsBlock`** and **`testAssertsMessage`** — does its suite prove a
> violating input is denied, and does it assert the message TEXT? A suite that
> checks only the exit code stays green if the gate stops saying anything.
> 8. **`failsOnEmptyInput`** — does it fail when its own allowlist or config is
> empty? A gate that iterates nothing and exits 0 passes every run while
> checking nothing.
> 9. **`namedSymbols`** — every symbol the gate NAMES and depends on: an escape
> label, a config path, a rule key, an issue ID. For each, record whether it
> **resolves against the system that owns it, by a command that was run**, and
> which system was asked. A gate offering an escape hatch that is not there
> offers no escape.
> 10. **`hasWriteTimeTwin`** — for a CI-enforced rule, does a write-time twin
> exist? Leave `undefined` if you did not check; an unanswered question is not
> a finding.
> 11. **`severityFavorsCompliance`** — is the compliant state at least as cheap as
> the non-compliant one? Read the gate's severity branches and answer.
> `undefined` if nobody read them.

Output: an `EnforcementConfig` object (`repoHooks`, `toolkitHooksPresent`,
`rulesConfigPresent`, `structuralRules`) plus the anchors backing each field,
ready for the deterministic verify in Stage 3.
`rulesConfigPresent`, `structuralRules`) **and** an `EfficacyConfig`
(`hooks: HookObservation[]`, `harnessRegistryFile`), plus the anchors backing
every field, ready for the deterministic verify in Stage 3.

---

Expand Down Expand Up @@ -111,11 +191,33 @@ inspectable from the repo).
includes
- a fully-covered setup (≥1 repo hook, toolkit hooks present, rules config
present, every structural rule backed by a hook) → `[]`.
3. **Reconcile** the verifier gaps with the Stage 2 observations: each gap must
3. **Call the efficacy verifier** — coverage and efficacy are different
questions and both must run:
```ts
import { findHookEfficacyGaps } from "../../src/audit/verifiers/enforcement-hooks-efficacy";

const efficacy = findHookEfficacyGaps(efficacyConfig);
// -> [{ code: "non-blocking-exit-code:pre-write-x", shape: 4, confidence: "direct", detail: "..." }]
```
One gap per condition that holds, each stamped with the `shape` it instances
and a `confidence` of `direct` or `proxy`, sorted by `code`:
- registered outside `harnessRegistryFile` → `registered-where-nothing-reads:<hook>` (shape 6)
- PreToolUse denial on an exit code other than 2 → `non-blocking-exit-code:<hook>` (shape 4)
- passes on an empty input set → `vacuous-on-empty-input:<hook>` (shape 4)
- blocking with no test that it denies → `no-block-assertion:<hook>` (shape 4)
- tests assert the exit code only → `asserts-adjacent-property:<hook>` (shape 2, **proxy**)
- names a symbol that does not resolve → `unresolved-symbol:<hook>:<symbol>` (shape 2)
- CI-only with `hasWriteTimeTwin === false` → `enforced-too-late:<hook>` (shape 5)
- `severityFavorsCompliance === false` → `inverted-incentive:<hook>` (shape 7)

An `undefined` observation produces no gap. The verifier reports what was
looked up, never what was assumed.

4. **Reconcile** the verifier gaps with the Stage 2 observations: each gap must
carry the anchor of the signal that produced it (the absent `.claude/hooks/`,
the missing rules config, or the confirmed finding whose `hook` cure is
unbacked). Drop any gap whose backing observation has no anchor.
4. **Stamp `confidence`:** `confirmed` when `findHookCoverageGaps` returns the
5. **Stamp `confidence`:** `confirmed` when `findHookCoverageGaps` returns the
gap AND its backing observation resolves to an anchor (a real `.claude/hooks/`
listing, a real rules config path, a real cross-referenced finding-ID);
`probable` when a signal is only partially observable (e.g. toolkit-hook
Expand All @@ -137,6 +239,21 @@ alone"*). Never truncate silently.
| `no-rules-config` — no rules config for the hooks to enforce | `high` |
| `no-toolkit-hooks` — toolkit enforcement hooks not inherited (Cure 4b) | `medium` |
| `rule-without-hook:<rule>` — a recommended rule has no backing hook | `medium` |
| `registered-where-nothing-reads:<hook>` — inert, and reads as covered | `high` |
| `non-blocking-exit-code:<hook>` — denies on a code that does not deny | `high` |
| `unresolved-symbol:<hook>:<symbol>` — names something that does not exist | `high` |
| `inverted-incentive:<hook>` — complying costs more than not complying | `high` |
| `vacuous-on-empty-input:<hook>` — green while checking nothing | `medium` |
| `no-block-assertion:<hook>` — no test proves it ever denies | `medium` |
| `asserts-adjacent-property:<hook>` — **proxy**, needs a human read | `medium` |
| `enforced-too-late:<hook>` — CI catches what write-time should have | `medium` |

The four efficacy `high`s share one property that the `medium`s do not: each
produces a gate that **looks like coverage from every listing**. An inert hook,
a denial on the wrong exit code, a phantom symbol and an inverted incentive all
survive review, appear in the inventory, and report green. A `medium` gap is a
gate that is weak; a `high` gap is a gate that is misleading, and the second is
worse because it stops anyone from looking again.

`no-repo-hooks` and `no-rules-config` are `high` because each removes the
enforcement layer wholesale: with no hooks and nothing for them to enforce, every
Expand Down
Loading