From 5b4ea936da730a199d63cd92242796c42809b101 Mon Sep 17 00:00:00 2001 From: sepehr-safari Date: Sat, 19 Sep 2026 20:42:55 +0300 Subject: [PATCH] feat: describe what 0.3.0 added, and what it does not claim The page described 0.2.0. Three things it said are now either incomplete or wrong for the version npm serves. The tool-list section said the comparison notices a description changed and never reads it. That is still true by default and is now a choice: reading them is its own switch, off unless someone turns it on, because turning it on sends every new or changed description to the model backend. The section says that plainly, with the policy snippet, rather than describing a capability without its cost. It also says the question behind that is not one the published numbers cover. No set in the benchmark asks it, its threshold is a judgement rather than a figure read off a curve, and a reader who meets a new capability next to a table of measurements will otherwise assume the table covers it. "Not a reader of tool descriptions" is retired, since it stopped being true for anyone who turns the switch on. In its place, the limit that now matters more: every threshold here was measured against one model, screening through a general gateway is supported, and those numbers look identical and mean something else. Shadow to enforce now names the two commands that make it a decision rather than a scroll. `report` leads with how many decisions enforcement would have stopped and did not, and `replay` answers the question that section was always asking without saying how, which is how to pick a different threshold without guessing at it. --- src/data/benchmark.ts | 2 +- src/pages/index.astro | 39 ++++++++++++++++++++++++++++++++------- 2 files changed, 33 insertions(+), 8 deletions(-) diff --git a/src/data/benchmark.ts b/src/data/benchmark.ts index ef109c0..c2b68c6 100644 --- a/src/data/benchmark.ts +++ b/src/data/benchmark.ts @@ -24,7 +24,7 @@ export const RUN = { * start claiming to come from a run nobody made. This one says what a reader * would install today. They were the same version once and are not any more. */ -export const CURRENT_VERSION = '0.2.0'; +export const CURRENT_VERSION = '0.3.0'; /** The thresholds the package actually ships with, from src/policy/schema.ts. */ export const SHIPPED = { diff --git a/src/pages/index.astro b/src/pages/index.astro index b8070de..b494b0f 100644 --- a/src/pages/index.astro +++ b/src/pages/index.astro @@ -75,10 +75,10 @@ const description = screening can be turned off per server for content that has to stay put.
  • - Not a reader of tool descriptions. The tool list is compared against the one a server - first advertised, so a description that changes is caught. Nothing asks whether a - description was hostile to begin with, so a server that arrives already carrying an - instruction is caught only when it changes. + Not calibrated outside one model. Every threshold here was chosen against measured + numbers from Jev. Screening through a general gateway is supported and returns numbers that + look the same and mean something else, so the results below say nothing about that. Read + your own log before letting one of those decide anything.
    • @@ -175,6 +175,21 @@ agent-chaperone hook post # after it returns, before the model reads itagent-chaperone trust <server>{' '} accepts it, by forgetting what was recorded so that the next list is the one to expect.

    +

    + Reading the descriptions, rather than only noticing that one changed, is a separate switch and + it is off by default. Turning it on asks whether a description goes past describing its own + tool and tries to steer the agent, and it sends every new or changed description to the model + backend in order to ask, which is why nobody acquires it by upgrading. +

    + 
    {`servers:
+  filesystem:
+    screen_tool_descriptions: true`}
    +

    + That question is not one the numbers below cover. No set in the benchmark asks it, so its + threshold is a judgement rather than a figure read off a curve, and the notice it prints says + as much rather than stating a verdict. A description nothing could read, because it was too + long to judge or the screen could not run, is reported as unread rather than passed over. +

    @@ -256,10 +271,20 @@ agent-chaperone hook post # after it returns, before the model reads it - 
    agent-chaperone log
    + 
    agent-chaperone report
    +

    + That leads with the count of decisions enforcement would have stopped and did not, which is + the question shadow mode is actually asking. Among them is whatever work you wanted done.{' '} + agent-chaperone log prints them one per line, and show opens one. +

    +

    + If you disagree with a line, move the threshold rather than the mode. You do not have to guess + at the new number: replay decides again over what was already judged, under a + policy you are considering, and says which way each decision moves. +

    + 
    agent-chaperone replay --policy candidate.yaml

    - If you disagree with a line, move the threshold rather than the mode. When the log stops - surprising you, set mode: enforce in the policy file. + When the log stops surprising you, set mode: enforce in the policy file.