Skip to content

Teach the policy author and publish skills to write Jev semantic checks - #23

Closed
chhhee10 wants to merge 4 commits into
mainfrom
feat/jev-policy-skills
Closed

chhhee10 wants to merge 4 commits into
mainfrom
feat/jev-policy-skills

Conversation

@chhhee10

Copy link
Copy Markdown
Member

Teaches the two policy skills to write and publish Jev (semantic) checks, next to the regex/JS policies they already cover. Everything is checked against failproofai 1.0.8-beta.0 (FailproofAI/failproofai#833): every command and refusal message quoted here was run and captured from that build.

failproofai-policy-author

  • New references/jev.md covers:
    • when to use Jev over a regex ("asked for" vs "slipped in");
    • the two tiers (regex is a hard floor, Jev judges above it);
    • what reviewable means, and the rule for marking a policy reviewable: is there anything left that can deny;
    • the check fields, the noul question format, the thresholds, and what Jev is shown;
    • writing probes as claims;
    • a complete pack entry that installs and runs;
    • Jev fields count only in packs (FailproofAI Cloud-managed policies never read them);
    • budget, reserved names, observe/--cli packs, and minCliVersion;
    • local testing, with prompt-wording traps;
    • the 16 built-in checks and 15 reviewable built-ins.
  • SKILL.md gets a short "Jev: when no string decides it" section that points to it.
  • traps.md §7 was wrong and is corrected: enabledPolicies is ignored once any pack is installed. New §10 lists seven ways Jev fails quietly.
  • api.md and cloud.md are updated. scripts/test-policy.mjs now runs regex only, so a configured Jev can't hide what the floor decides.

failproofai-policy-publish

  • New "Packs that carry Jev checks" section covers:
    • what publish refuses, and the exact messages;
    • the dry-run output;
    • the question budget (9,101 characters for a pack from outside FailproofAI);
    • the minCliVersion it writes (1.0.8-beta.0);
    • the rollback reminder;
    • what someone installing the pack sees in policies add, policies show and jev status.

Notes

  • Needs failproofai 1.0.8-beta.0 or later (#833). Merge this after that release is out.
  • One line will follow as its own commit: #833 is tightening when consent may clear a call with several targets, and the skill will state the final rule.

🤖 Generated with Claude Code

chhhee10 and others added 4 commits September 27, 2026 10:53
Add references/jev.md: when a concern needs Jev rather than a regex, the
two tiers (regex floor, Jev deny/instruct/clear), why reviewable is
dangerous ("is there anything left that can deny"), the exact shape of a
semanticPolicies.add check, how to phrase probes as harmful claims, a
complete two-tier pack entry that builds on 1.0.8-beta.0, where Jev
fields count (pack only; Cloud-managed policies are always hard), the
shared question budget and reserved names, the local test loop, and the
16 built-in checks with the 15 reviewable builtins.

SKILL.md gains a short Jev section pointing there, a noisy-builtin note,
and the caveat that enabledPolicies stops being read once any pack is
installed. traps.md §7 is corrected for that and §10 lists Jev's quiet
failures. test-policy.mjs now forces the legacy evaluator so a configured
Jev cannot mask what the regex floor decides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pack is the only way a Jev check reaches a machine. Document what
`failproofai publish` validates and refuses for one (built-in check
names from outside FailproofAI, the 9,101-character question budget,
reviewedBy naming undeclared checks, alwaysOn, minCliVersion below
1.0.8-beta.0), the dry-run lines to expect, the Jev-only rollback
reminder, installing a dry run from a local mirror, and what consumers
see in `policies add`, `policies show` and `jev status`. Discovery now
mentions semanticPolicies.add. Every message quoted was produced by the
1.0.8-beta.0 CLI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e command keeps the floor

Matches FailproofAI/failproofai#833 at 0adc9212.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UhTaConsTkbuxm14w6fye
chhhee10 added a commit that referenced this pull request Sep 30, 2026
Folds in what #23 had and this branch did not, rewritten for failproofai 1.0.9, so
one PR covers Jev from authoring to publishing.

failproofai-policy-publish:
- New "Packs that carry Jev checks" section: the 1.0.9 dry-run lines, what publish
  refuses, the 9,101-character budget, minCliVersion, the rollback reminder, and
  what policies show / policies add print for a pack's checks.
- publishing.md: the refusal messages as 1.0.9 prints them, the budget, reserved
  and contested names, the FailproofAI/ namespace, minCliVersion, installing a dry
  run locally, and observe / --cli packs.
- Discovery also finds semanticPolicies.add files.

failproofai-policy-author:
- Consent rules decideV1 checks in code: op_requested clears a shell command only
  when every target is named, task_step does not soften one the user named only in
  part, and a command the scan cannot read whole (any $ expansion, globs, brace
  expansion, heredocs, eval, sh -c) is never cleared or softened.
- Reviewable clearing: in 1.0.9 a warning nobody consented to keeps the floor, and
  one from a deny-mode check cancels every clear on the call (combine.ts,
  jev-review.ts). The text said a warning counted as a clear.

README: both rows mention Jev.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chhhee10

Copy link
Copy Markdown
Member Author

Superseded by #24. #24 now carries the publish-side Jev content from this PR (the policy-publish section, the refusal messages, the budget, minCliVersion, and what consumers see) and the consent rule from e4756b1. All of it was re-checked against failproofai 1.0.9 and rewritten where 1.0.9 changed it. Closing this one so there's a single PR.

@chhhee10 chhhee10 closed this Sep 30, 2026
SiddarthAA pushed a commit that referenced this pull request Sep 30, 2026
… 1.0.9 (#24)

* failproofai-policy-author: bring in Jev, corrected for failproofai 1.0.9

The skill here predated Jev and had drifted from the product's copy. Merge
that copy in three ways from this repo's first import (byte-identical to it),
with its content corrected for 1.0.9: the npm package ships no Jev checks,
reviewedBy counts only for installed checks, publish holds a third-party pack
to 9,101 characters, --min-cli-version defaults to 1.0.8-beta.0, and Hermes'
native plugin makes instruct block once per model response while user_said
stays empty.

Kept from this repo: routing to failproofai-policy-publish and fp-cloud-cli,
the policy backtest sections, and fleet-tool-coverage's canonicalNames fix.
builtins.md and policy-events.json regenerated against 1.0.9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Publish Jev packs end to end, and correct reviewable clearing for 1.0.9

Folds in what #23 had and this branch did not, rewritten for failproofai 1.0.9, so
one PR covers Jev from authoring to publishing.

failproofai-policy-publish:
- New "Packs that carry Jev checks" section: the 1.0.9 dry-run lines, what publish
  refuses, the 9,101-character budget, minCliVersion, the rollback reminder, and
  what policies show / policies add print for a pack's checks.
- publishing.md: the refusal messages as 1.0.9 prints them, the budget, reserved
  and contested names, the FailproofAI/ namespace, minCliVersion, installing a dry
  run locally, and observe / --cli packs.
- Discovery also finds semanticPolicies.add files.

failproofai-policy-author:
- Consent rules decideV1 checks in code: op_requested clears a shell command only
  when every target is named, task_step does not soften one the user named only in
  part, and a command the scan cannot read whole (any $ expansion, globs, brace
  expansion, heredocs, eval, sh -c) is never cleared or softened.
- Reviewable clearing: in 1.0.9 a warning nobody consented to keeps the floor, and
  one from a deny-mode check cancels every clear on the call (combine.ts,
  jev-review.ts). The text said a warning counted as a clear.

README: both rows mention Jev.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant