Teach the policy author and publish skills to write Jev semantic checks

README.md

@@ -26,8 +26,8 @@ whole repository does.
 | Skill | What it does | Source of truth |
 |---|---|---|
 | [`failproofai`](skills/failproofai/) | **The main skill.** A complete, standalone FailproofAI product package: explain the product, set up a machine, operate the local runtime and daemon, use FailproofAI Cloud, understand and author policies, publish GitHub policy packs, instrument custom agents, work with evaluators, inspect every command, and troubleshoot by symptom. It also routes to focused sibling skills when they are installed. | Maintained here. Not synced from anywhere - edit in this repo. |
-| [`failproofai-policy-author`](skills/failproofai-policy-author/) | Turn what agents keep doing wrong into enforcement for [failproofai](https://github.com/FailproofAI/failproofai) - triage a `failproofai audit` or FailproofAI Cloud findings, convert a CLAUDE.md/AGENTS.md into policies, or take a plain complaint ("agents keep force-pushing") and enforce it. Checks the shipped builtins and their params before writing anything, since most requests are one line of config; knows which of the 12 supported agent CLIs actually enforce a given event, so it never ships a deny the harness discards; tests every policy it authors. | Maintained here. Not synced from anywhere - edit in this repo. |
-| [`failproofai-policy-publish`](skills/failproofai-policy-publish/) | The publishing companion to policy authoring - take tested policies, build an installable pack, publish its release assets to GitHub with `failproofai publish`, preview it, and verify the consumer path with `failproofai policies add <owner>/<repo>`. Cloud fleet rollout remains part of `fp-cloud-cli`. | Maintained here. Not synced from anywhere - edit in this repo. |
+| [`failproofai-policy-author`](skills/failproofai-policy-author/) | Turn what agents keep doing wrong into enforcement for [failproofai](https://github.com/FailproofAI/failproofai) - triage a `failproofai audit` or FailproofAI Cloud findings, convert a CLAUDE.md/AGENTS.md into policies, or take a plain complaint ("agents keep force-pushing") and enforce it. Checks the shipped builtins and their params before writing anything, since most requests are one line of config; knows which of the 12 supported agent CLIs actually enforce a given event, so it never ships a deny the harness discards; writes Jev semantic checks and reviewable policies for calls no regex can decide; tests every policy it authors. | Maintained here. Not synced from anywhere - edit in this repo. |
+| [`failproofai-policy-publish`](skills/failproofai-policy-publish/) | The publishing companion to policy authoring - take tested policies, build an installable pack, publish its release assets to GitHub with `failproofai publish`, preview it, and verify the consumer path with `failproofai policies add <owner>/<repo>`, including packs that carry Jev semantic checks. Cloud fleet rollout remains part of `fp-cloud-cli`. | Maintained here. Not synced from anywhere - edit in this repo. |
 | [`fp-cloud-cli`](skills/fp-cloud-cli/) | Operate FailproofAI Cloud with `fp`: inspect telemetry, evals and usage; triage issues and audits; manage keys, users, orgs, and settings; publish Cloud policy versions; deploy them to fleet machines; observe enforcement; promote or roll back. Global options go **before** the command: `fp --json sessions`, not `fp sessions --json`. | Synced from `FailproofAI/failproofai` → `fp-cloud-cli/skill/`. Do **not** hand-edit here. |
 | [`failproofai-sdk`](skills/failproofai-sdk/) | Make an AI agent report what it did - plan which points in the agent loop to record, write the instrumentation with the Python (`failproofai_sdk`) or TypeScript/JavaScript (`@failproofai/sdk`) SDK - a framework adapter or a hand-built loop - thread session/agent identity through it, and verify the events actually land. Also runs your own evaluator worker (the eval pod) in either language. For an agent loop that is **not** one of the 12 supported CLIs. | Synced from `FailproofAI/failproofai` → `sdk/python/skill/`. Do **not** hand-edit here. |
 | [`failproofai-eval-brainstorm`](skills/failproofai-eval-brainstorm/) | Work out **what is worth measuring** about an agent's production runs, from the sessions it actually produced - scan the population, confirm the signal is really in the telemetry (a measurement over a payload key nobody emits does not fail, it scores every session identically and looks like it works), check it separates good runs from bad, and converge on two to four proposals. Each one ends in the plain-English prompt that authors it. It stops there: composing, backtesting and deploying the evaluation is the dashboard's eval authoring page. | Synced from `FailproofAI/agenteye` → `agent/skills/failproofai-eval-brainstorm/` (private). Do **not** hand-edit here. |
@@ -175,7 +175,7 @@ skills/                         ← this repo
     │   └── agents/openai.yaml
     ├── failproofai-policy-author/
     │   ├── SKILL.md
-    │   ├── references/          ← api · builtins · cloud · harnesses · patterns
+    │   ├── references/          ← api · builtins · cloud · harnesses · jev · patterns
     │   │                           rules-files · traps
     │   ├── scripts/             ← runnable helpers (test a policy, sync a reference)
     │   └── agents/openai.yaml

skills/failproofai-policy-author/SKILL.md

@@ -5,7 +5,7 @@ description: |-
 
   Trigger when the user wants to:
   • act on an audit — turn `failproofai audit` findings into fixes, or ask which policies work;
-  • stop a recurring behaviour, in plain words or as "write a policy that blocks X";
+  • stop a recurring behaviour, in plain words or as "write a policy that blocks X" (regex or Jev);
   • enforce a rules file — make a CLAUDE.md / AGENTS.md real instead of advisory;
   • enable an existing builtin — usually the right answer, checked first;
   • work from FailproofAI Cloud — findings, hooks that fail or over-deny, backtesting a draft.
@@ -323,6 +323,9 @@ Run the attribution above first — a `DEAD` finding is not Bucket A, B or C; it
 **Bucket A — a builtin covers it and is off.** Do not write code. Add the short name to
 `enabledPolicies` in `.failproofai/policies-config.json`. This is the cheapest and most
 maintainable fix, and it is the right answer for most `source: "builtin"` findings.
+**Once any pack is installed on the machine** (a Jev pack included), `enabledPolicies` is no
+longer read at all: switch the builtin on in the pack instead,
+`failproofai policies add FailproofAI/policies --policy <name>` (`references/traps.md` §7).
 
 **Bucket B — a builtin covers it and is already on.** No action. Report it so the user knows
 the finding is historical, not ongoing.
@@ -453,6 +456,12 @@ and parameters. If one matches, enabling it beats writing a new file every time.
 Many builtins take `params` (allowlists, thresholds, protected branches) that go in the
 `policyParams` map — a parameterized builtin often covers a case that looks custom.
 
+If the complaint is that a builtin is **noisy** (`block-kubectl` denying `kubectl get`), try
+its `allowPatterns` / `allowPaths` param first: that trades nothing. The other fix is Jev:
+15 builtins ship **reviewable**, and Jev clears them on calls it judges harmless once it runs
+in `enforce` mode (*Jev: when no string decides it*). The price is that forged consent clears
+them too.
+
 Then check the project's **existing custom policies** — `ls .failproofai/policies/` and
 read their `name`/`description` lines. Coverage is not only builtins: a hand-written policy
 may already enforce exactly what you were about to author, and a duplicate means two
@@ -527,6 +536,43 @@ This is the highest-frequency failure in the whole system — see `references/tr
 
 See `references/patterns.md` for worked examples per event type.
 
+### Jev: when no string decides it
+
+Some concerns are not in the command. `rm -rf build/` that the user asked for and `rm -rf ~`
+that slipped into a plan; `prisma migrate deploy` against localhost and against production.
+A regex that blocks all of them gets disabled; one that allows them enforces nothing.
+**Jev** is failproofai's semantic evaluator: it answers yes/no questions about the call
+against what the human typed. **Read `references/jev.md` before writing either half** — it
+has the field rules, a complete pack entry, the budget, and the local test loop.
+
+The shape, in brief:
+
+- **Two tiers.** The regex is the hard floor; Jev judges above it on `PreToolUse` and
+  `PermissionRequest` only. Jev can **deny** or **instruct** through a check that fires, and
+  can **clear** only a **reviewable** policy's verdict. A hard deny is final. When Jev is not
+  configured, is in `shadow`, is `off`, or does not answer, the regex result applies, so
+  anything that must hold everywhere needs a regex floor.
+- **Reviewable** is two fields on `customPolicies.add`: `authority: "reviewable"` and
+  `reviewedBy: ["<check>"]` (check names, not policy names). The verdict clears only when every
+  named check was asked and none found the concern without the user asking. The test to apply
+  is **"once this clears, is there anything left that can deny?"**, not "can the reviewer keep
+  this block". A check that is asked but does not model a shape answers "no concern" and
+  **clears it silently**; an instruct-only reviewer can never deny; and an agent with a shell
+  can forge the user's consent. Keep irreversible, privilege and remote-code rules hard.
+- **A semantic check** is `semanticPolicies.add({ name, title, appliesTo, mode,
+  userCanOverride, probes, guidance })`: questions, no `fn`. Every probe must hold for it to
+  fire, so **every probe states the harmful claim**, true for the harmful call and false for
+  the harmless one.
+- **It only works in a pack.** In `.failproofai/policies/` it is never asked, and a
+  FailproofAI Cloud-managed policy ignores all Jev fields and is always hard. **Jev checks
+  belong in packs**, published with `failproofai-policy-publish`, with `minCliVersion` ≥
+  `1.0.8-beta.0`. A pack's checks join the 16 built-in ones, cannot reuse their names, share
+  a ~9,100-character question budget beside them, and are not asked for an `observe` pack.
+- **Test both tiers.** `test-policy.mjs --policy` tests the floor alone (no Jev).
+  `failproofai publish <file> --dry-run` validates the pack. Then `failproofai jev setup
+  … --mode shadow` (its bring-your-own-key default is `enforce`), `jev test`, `jev status`.
+  Use a **vague** prompt to see a check decide: naming the operation reads as consent.
+
 ### Verify it actually fires
 
 Loading and execution are both fail-open — a broken policy is indistinguishable from a
@@ -775,11 +821,13 @@ failproofai policies --list
 | Half | The question it answers | Skill |
 |---|---|---|
 | author | *what is the rule, and does it decide correctly?* | this one |
-| publish | *how does this tested policy become a versioned GitHub pack others can install?* | `failproofai-policy-publish` |
+| publish | *how does this tested policy, or a Jev check, become a versioned GitHub pack others can install?* | `failproofai-policy-publish` |
 | cloud rollout | *which fleet machines run a Cloud policy version, and what did it block?* | `fp-cloud-cli` |
 
 Everything past a proven local file is the deploy half: minting a version with
-`fp policies publish` (which **deploys nothing** on its own), choosing `enforce` vs
+`fp policies publish` (which **deploys nothing** on its own, and makes a Cloud-managed
+policy that never reads Jev fields: strip `semanticPolicies.add`, `authority` and
+`reviewedBy` first, and ship the Jev half as a pack), choosing `enforce` vs
 `observe`, `fp fleet deploy`, rollback, and reading `fp guardrails` to see the rule fire on
 real traffic. Those are shipped commands — if you find yourself about to say deployment is
 "dashboard work" or "not exposed by the CLI", that is wrong, and it tells the reader to stop

skills/failproofai-policy-author/references/api.md

@@ -5,6 +5,7 @@
 - [The policy object](#the-policy-object) · [Context](#context) · [Decisions](#decisions)
 - [Events](#events) · [Filtering by tool](#filtering-by-tool)
 - [Execution model](#execution-model) · [Configuration](#configuration)
+- [Jev fields](#jev-fields)
 
 > Source pointers below are paths inside the failproofai package. In a project that
 > installed it, they live under `node_modules/failproofai/`; in a source checkout,
@@ -13,9 +14,12 @@
 Everything here is exported from `src/index.ts` — that file is the entire public surface:
 
 ```ts
-export { customPolicies, getCustomHooks, clearCustomHooks } from "./hooks/custom-hooks-registry";
+export { customPolicies, semanticPolicies, getCustomHooks, getSemanticRegistrations,
+  clearCustomHooks } from "./hooks/custom-hooks-registry";
 export { allow, deny, instruct } from "./hooks/policy-helpers";
-export type { PolicyContext, PolicyResult, CustomHook, PolicyDecision, PolicyFunction } from "./hooks/policy-types";
+export type { PolicyContext, PolicyResult, CustomHook, PolicyDecision, PolicyFunction,
+  PolicyAuthority, SemanticPolicyDeclaration, SemanticProbeDeclaration,
+  SemanticToolClass } from "./hooks/policy-types";
 ```
 
 ## The policy object
@@ -30,6 +34,8 @@ export interface CustomHook {
     events?: HookEventType[];
   };
   fn: (ctx: PolicyContext) => PolicyResult | Promise<PolicyResult>;
+  authority?: "hard" | "reviewable";   // absent = hard. See *Jev fields*
+  reviewedBy?: string[];               // semantic check names
 }
 ```
 
@@ -181,3 +187,17 @@ is no per-policy enabled/disabled object, and omission means off.
 
 Merged across three scopes, in precedence order: project `{cwd}/.failproofai/` → local →
 global `~/.failproofai/` (`hooks-config.ts`, grep `readMergedHooksConfig`).
+
+## Jev fields
+
+Two additions to the surface, both covered in full in `jev.md`:
+
+- `authority` / `reviewedBy` on `customPolicies.add`: whether the Jev semantic evaluator may
+  clear this policy's verdict, and through which checks. Honoured for your own local files;
+  for a pack policy the manifest decides (`failproofai publish` copies them there); a
+  FailproofAI Cloud-managed policy ignores them and is always hard (`policy-types.ts`, grep
+  `interface CustomHook`).
+- `semanticPolicies.add(decl)`: a Jev check, a question set with no `fn`
+  (`policy-types.ts`, grep `interface SemanticPolicyDeclaration`). Read only by
+  `failproofai publish`, so it takes effect **only in a pack**. `getSemanticRegistrations()`
+  returns what is declared, for tests; `clearCustomHooks()` clears both registries.

skills/failproofai-policy-author/references/cloud.md

@@ -232,7 +232,15 @@ plus a config entry, and that is the whole story for one machine. It is **not**
 for a fleet: `fp policies publish`, `fp fleet deploy` and `fp guardrails` are shipped commands
 that carry the same rule to every machine and show it firing, and they need `policies:write`.
 That Cloud rollout path belongs to `fp-cloud-cli` — hand off rather than assuming the local
-edit is all there is. Then prove both local edits took effect, because neither is self-evident:
+edit is all there is.
+
+**A Cloud-managed policy has no Jev half.** It is always hard, whatever `authority` and
+`reviewedBy` say, and a `semanticPolicies.add` in it is never asked. So strip all three
+before `fp policies publish`. When the finding needs a judgment no string decides, ship that
+half as a Jev check in a policy **pack** (`jev.md`), beside the hard Cloud policy or instead
+of it.
+
+Then prove both local edits took effect, because neither is self-evident:
 
 ```bash
 export SKILL_DIR=/path/to/skills/failproofai-policy-author   # this skill's own folder