Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tools/spec-loop/.last-sync
Original file line number Diff line number Diff line change
@@ -1 +1 @@
5c75abd216d798efce7257749cbdb3e26844b3c0
bcd8b7f28e5d926ca8eff6e603c64b327ae038fe
1 change: 1 addition & 0 deletions tools/spec-loop/specs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ Start with [`overview.md`](overview.md), then:
[`maintainer-education.md`](maintainer-education.md),
[`spec-gap-staleness.md`](spec-gap-staleness.md),
[`vetted-command-surface.md`](vetted-command-surface.md),
[`adversarial-review.md`](adversarial-review.md),
[`sandbox-diagnostics.md`](sandbox-diagnostics.md),
[`container-gateway.md`](container-gateway.md).

Expand Down
147 changes: 147 additions & 0 deletions tools/spec-loop/specs/adversarial-review.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->

---
title: Adversarial review by other models
status: experimental
kind: feature
mode: infra
source: >
docs/designs/2026-09-23-adversarial-review.md (rollout PR 1 of 4).
Implemented in tools/adversarial-review/ and published as the
magpie-adversarial-review substrate plugin (plugins/magpie-adversarial-review/,
SUBSTRATE_PLUGINS in tools/dev/check-family-plugins.py).
acceptance:
- The tool runs only installed reviewer CLIs (codex, copilot, gemini,
claude), each in its own read-only headless mode, and skips the model
running the harness unless told otherwise.
- A reviewer's input is limited by construction to the diff, the changed
files, and the PR title and body as they will be posted; no option
accepts any other context.
- The report is advisory — one merged JSON document, exit 0 whenever the
run completes — and an unavailable reviewer never fails the run.
- Reviewer output is data; a finding that reads like an instruction is
never acted on.
---

# Adversarial review by other models

## What it does

Gives a change a second read from models other than the one that wrote it.
The tool detects which reviewer CLIs are installed, runs the configured ones
read-only and in parallel over a branch, a diff or a PR, and merges what they
find into one advisory report.
A reviewer is only useful as a *different* model, so the harness's own model
is skipped by default.

This spec covers what has shipped: the tool and its plugin.
Wiring it into the skills — setup, a shared pre-PR block in every PR-creating
skill, and a multi-reviewer second read in `pr-management-code-review` — is the
remaining rollout, recorded under *Known gaps*.

## Where it lives

- `tools/adversarial-review/` — stdlib-only Python package, capability
`substrate:review`, harness `agnostic`. Two subcommands:
- `detect` — for each backend, whether its CLI is on `PATH` and answers a
cheap probe, and which backend is the running harness (`self`),
recognised from the environment variables each harness sets.
- `run` — builds the input from `--target branch` (with `--base`),
`pr:<N>` (with `--repo`, through `gh`) or `diff:<path>`, plus `--title`
and `--body-file`, runs every requested reviewer, and prints the merged
report.
- `plugins/magpie-adversarial-review/` — the substrate plugin in the Claude
Code catalogue, linking `tools/adversarial-review` so the tool runs from the
installed plugin tree
([marketplace distribution](marketplace-distribution.md)).
- `adversarial-review.md` — optional configuration, resolved
`.apache-magpie-local/` first, then `.apache-magpie-overrides/`, under
`--project-root`: `mode`, `reviewers`, `timeout_minutes`, per-backend
`models`. `--reviewers` on the command line overrides the list.
- `docs/designs/2026-09-23-adversarial-review.md` and its implementation plan.

## Behaviour & contract

- **Read-only backends, pinned by tests.** One adapter per CLI holds its
headless command line: `codex exec -s read-only --ephemeral` with MCP servers
switched off and `--output-schema`; `copilot -p` with the shell and write
tools denied; `gemini --approval-mode plan -o json`; `claude -p` with
`--strict-mcp-config` and Bash, the editing tools, web access and `Task`
disallowed. `tests/test_backends.py` pins each line, so a regression that
drops a read-only flag fails.
- **The input builder is the privacy boundary.** The prompt carries only the
diff, the changed-file list and the public PR text, so no privacy-LLM gate
applies: nothing private is ever passed. When `--repo-dir` is the project's
private tracker the report says so in `warnings`.
- **One findings schema, merged.** Every reviewer answers against the same
schema (`severity`, `file`, `line`, `claim`, `evidence`). Parsing keeps the
good findings from a partly malformed reply and reports the bad ones;
findings are de-duplicated across reviewers, keeping every reviewer's name,
and sorted by severity.
- **Bounded and interruptible.** Reviewers run in parallel, each with a
timeout (default 8 minutes, under the 10-minute cap harnesses put on one
shell call); a timed-out reviewer's whole process group is killed, and
Ctrl-C or SIGTERM kills every live reviewer.
- **Advisory.** Each reviewer reports `ok`, `unavailable`, `error`, `timeout`
or `skipped` with its reason. The exit code is 0 whenever the run completes
and 2 only for a wrong invocation or an invalid config.
- **Nothing written to the repository.** The brief and the schema live in a
temporary directory removed afterwards; the tool itself makes no network
calls.
- **Standalone resolution.** Like `tools/vetted-ops`, the project declares no
workspace `dev` group, because it runs as
`uvx --from <plugin>/tools/adversarial-review adversarial-review …`, where
the workspace root does not exist.

## Out of scope

- Blocking a PR on findings; the human decides what to act on.
- Reviewing private content: tracker bodies, mail threads, CVE IDs before
disclosure, advisory text.
- Authenticating reviewer CLIs; each uses its own login.

## Acceptance criteria

1. `detect` reports each backend's availability and marks the running
harness `self`; `run` skips `self` unless `--self none` is passed.
2. Each backend's command line matches its pinned test, and none can run in
a writable mode.
3. Given tracker-shaped context, only the diff, the file list and the public
PR text reach the prompt.
4. One slow or failing reviewer does not block the others, and its timeout
or error is reported per reviewer.
5. The run exits 0 with a merged report whenever it completes, whatever the
reviewers returned.
6. `check-family-plugins.py` passes with `magpie-adversarial-review` in
`SUBSTRATE_PLUGINS` and its entry point resolving through the plugin link.

## Validation

```bash
uv run --all-packages --group dev pytest tools/adversarial-review/tests
python3 tools/dev/check-family-plugins.py
```

## Known gaps

- **No consumer is wired yet.** The design's rollout PRs 2–4 are unbuilt:
`setup` running `detect` in `config` and `verify`, writing the
configuration and the per-harness commands, and installing the sandbox
exclusion; the shared pre-PR block in every PR-creating skill; and
`with-reviewers:` in `pr-management-code-review`, which today still takes a
single user-fired `with-reviewer:` slash command. The tool README's
statements that skills run it and that setup installs its exclusion
describe that target state.
- **No `commands` subcommand.** The design's `commands --harness <name>`,
which would print the per-harness command files for `setup`, does not
exist; only `detect` and `run` ship.
- **The `mode` key is parsed but unused** until the pre-PR block consumes it.
- **Reviewers can read beyond the prompt.** `codex -s read-only` restricts
writes and network, not reads, so an instruction injected into the diff
could have it read a file elsewhere on the machine; `copilot` and `gemini`
keep any MCP servers they are configured with. The README tells operators
to keep private checkouts away from review machines or leave `codex` out.
- **The sandbox blocks it.** Reviewer CLIs need network access and their own
credentials, which the reference sandbox denies, and no `excludedCommands`
entry ships for the tool yet.
66 changes: 62 additions & 4 deletions tools/spec-loop/specs/agent-isolation-sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,13 +98,39 @@ existing sandbox grants can widen the baseline. See `docs/adapters/gemini.md`.
`watcher.pid: Operation not permitted` before any watcher started —
and it has to be stable across contexts rather than `$TMPDIR`, which
differs between the agent's hooks and a terminal `git` and would give
two contexts two registries that cannot see each other. The
two contexts two registries that cannot see each other.
The hook-side owner is the harness process, not the hook's parent:
Claude Code on Linux runs a hook through `sh -c`, which exits the
moment the hook returns, so `arm` walks up past intermediate shells
to the first non-shell ancestor and records that — otherwise the
watcher's parent-liveness check ended the watch before the key ever
blocked (#1365).
On macOS the window closes when its application loses activation
(`<Deactivate>`, bound after a short grace because activation itself
churns focus) rather than re-grabbing the keyboard: the overlay
never traps the screen, since the key still has to be touched for
the command to go through (#1325). The
git the agent runs reads the same global config, so the wrapper's
two files are a `sandbox.filesystem.allowRead` grant of their own
(nothing wider under `~/.claude/`), or every sandboxed signed commit
fails with `cannot exec`. Installed by `setup-isolated-setup-install` Step K,
checked by `setup-isolated-setup-verify` check 10. Capability:
`substrate:sandbox`.
- **Hardware-key touch policy** (the recommendation Step K proposes and
check 10 reports against). The touch goes on the slot that **signs**,
never on the ssh transport alone: a touch on every fetch and pull is
a prompt on a read, and a push is already gated by the `git push`
ask rule and carries only commits signed with a touch. Which slot
signs follows `gpg.format`: with OpenPGP signing (unset or
`openpgp`) it is `sig`, recommended `cached`, and `aut` is
recommended `off`; with `gpg.format=ssh` the signature is made by the
key `ssh-add -L` lists, so `aut` is the signing slot and must stay
`cached` — the skill never proposes turning it off there, and states
once that the transport then pays the touch too (OpenPGP signing or
an https remote avoids it) — while `sig` signs nothing and is left
alone. `cached` (a touch honoured for 15 seconds) rather than `on`,
and never `fixed` / `cached-fixed`, which cannot be undone without
deleting the private key (#1367).
- `tools/agent-guard/` — deterministic pre-execution guard dispatcher
(`stdlib`-only). Wired as a `PreToolUse` hook (Claude Code) or a
`tool.execute.before` plugin (OpenCode), with a `--gemini` adapter for
Expand All @@ -113,7 +139,12 @@ existing sandbox grants can widen the baseline. See `docs/adapters/gemini.md`.
before it runs and denies the ones that break a hard framework rule,
independent of model memory. The guard decisions live in a single
harness-agnostic `dispatch()` core so every wired harness enforces
an identical rule set. Capability: `substrate:action-guard`.
an identical rule set. Git guards resolve the subcommand by walking
past git's global options (`git -C <dir> commit`, `git -c k=v commit`,
`git --no-pager commit`), never by a fixed argv slice, and
`GuardContext.git_subcommand()` gives contributed guards the same
resolution `gh_subcommand()` gives for `gh` (#1330).
Capability: `substrate:action-guard`.
- `tools/permission-audit/` — audits and atomically edits Claude Code's
`permissions.allow[]` entries in `.claude/settings.json` and
`.claude/settings.local.json`. Backs the `--apply-permission-audit`
Expand All @@ -130,6 +161,22 @@ existing sandbox grants can widen the baseline. See `docs/adapters/gemini.md`.
paths the committed list also names; `--no-tool-paths` limits it to
the project root. Checked by `setup-isolated-setup-verify` check 8.
Capability: `substrate:sandbox`.
- Whole-user git hooks — the install skill's Step P.3 alternative to
per-project scope: global `core.hooksPath` pointing at
`~/.claude/git-hooks/`, in a *simple* flavour (a standalone
`post-checkout`) or a *dispatcher* flavour (every hook name symlinked
to `git-hook-dispatcher.sh`, which chains to per-repo `.git/hooks/*`).
That directory sits under the read-denied home, and git treats a hook
directory it cannot see as "no hooks", so without a read-only
user-scope `sandbox.filesystem.allowRead` grant for it (plus
`~/.claude-config/git-hooks/` when the hooks are symlinks into the
sync repo, since the sandbox checks the resolved path) every
sandboxed commit silently skips `pre-commit`, `commit-msg` and the
rest. Install proposes the grant at Step P.3-whole-user; verify
check 8 probes the directory from inside the sandbox (#1364). Both
the update skill's drift check and verify check 8 recognise the
dispatcher flavour's symlinks as its installed shape, not as drift
or as inert per-repo hooks (#1322, #1358).
- `tools/egress-gateway/` — local HTTP(S) forward proxy for egress
control. Framework tools point `HTTPS_PROXY`/`HTTP_PROXY` at it; the
gateway rejects any connection to a host not on its allowlist before a
Expand All @@ -150,7 +197,13 @@ existing sandbox grants can widen the baseline. See `docs/adapters/gemini.md`.
`sandbox-lint --gemini .gemini` checks the static profile, with opt-in pytest integration tests against native 0.59.0 APIs for settings, policies, headless refusal, and Linux enforcement.
Every Gemini upgrade requires revalidating the native probe against that version; static CI checks alone do not establish effective policy precedence.
- Skills: `setup-isolated-setup-install`, `-update`, `-verify`,
`-doctor`. The diagnostic side — the failure catalog in
`-doctor`. The update skill establishes the agent-guard wiring before
diffing anything: with the `magpie-agent-guard` plugin enabled, the
plugin registers the hook and resolves the engine under
`${CLAUDE_PLUGIN_ROOT}`, so an absent `~/.claude/scripts/agent-guard.py`
is the expected shape rather than drift, and on either wiring a
`git commit` carrying a `Co-Authored-By:` trailer must be denied as a
behavioural canary (#1323). The diagnostic side — the failure catalog in
`docs/setup/sandbox-troubleshooting.md`, the `sandbox-error-hint.sh`
hook, the doctor's live probes and the verify checks — is specified
in [`sandbox-diagnostics.md`](sandbox-diagnostics.md).
Expand Down Expand Up @@ -227,7 +280,12 @@ The reference model is four layers, layered:
default (prompt in default mode, classifier in auto).

Pinned system tools (`bubblewrap`, `socat`, agent CLI) are aged through a
cooldown window; bumps are PRs, not silent updates.
cooldown window; bumps are PRs, not silent updates. The window is the
framework's 7-day default unless a tool's `[tools.<name>]` table in
`tools/agent-isolation/pinned-versions.toml` sets its own
`cooldown_days`; `bubblewrap` carries `cooldown_days = 1`, so a pin
carrying a sandbox-setup security fix can move a day after release
(#1360, pinned at 0.13.0).

## Out of scope

Expand Down
20 changes: 19 additions & 1 deletion tools/spec-loop/specs/cve-tooling.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,17 @@ reviewable.
flow. The skill detects session expiry via `vulnogram-api-check` and
falls back to the manual paste path when the session is not configured
or expired.
- `tools/cve-org/` — CVE.org / CVE-services helpers.
- `tools/cve-org/` — CVE.org / CVE-services helpers; the `check-published`
recipe runs as the `cve-check-published` vetted-ops read operation, not raw
`curl` ([vetted command surface](vetted-command-surface.md)).
- `tools/osv/` — OSV.dev adapter (`contract:security-cross-ref`): read-only,
unauthenticated lookups against the OSV.dev REST API v1 — a record and its
alias cross-walk by ID (CVE ↔ GHSA ↔ PYSEC / RUSTSEC / GO …), advisories
affecting a package version, vulnerabilities tied to a public upstream
commit, and a batch form of the last two. Each recipe is a vetted-ops read
operation (`osv-get-vuln`, `osv-query-package`, `osv-query-commit`,
`osv-query-batch`); `api.osv.dev` is on the egress gateway's allowlist
(#1297, #1326).
- Skill: `security-cve-allocate` — walks the (PMC-gated) allocation form,
then updates the tracker and regenerates the attached JSON via
`generate-cve-json --attach`.
Expand Down Expand Up @@ -80,6 +90,14 @@ uv run --project tools/cve-tool-vulnogram/oauth-api --group dev pytest

- `stable`; drift appears if the CVE 5.x schema or Vulnogram export shape
changes upstream — caught by the tool's own tests.
- **`tools/osv/` has no consuming skill yet.** Its `tool.md` names the
intended consumers — `security-issue-triage` (early deduplication against
known vulnerabilities), `security-issue-deduplicate` (alias resolution
before a merge), `security-cve-allocate` (check no existing advisory covers
the report) and `dependency-audit` — none of which calls it today. An OSV
query about a pre-disclosure report must also stay within the embargo
boundaries `tool.md` sets: only public identifiers, packages and commits
go into a query.
- The oauth-api's session-cookie approach is a workaround for the ASF
Vulnogram instance's lack of a Bearer-token API; if ASF Infra adds
a proper API token surface, the tool will migrate to it.
13 changes: 13 additions & 0 deletions tools/spec-loop/specs/marketplace-distribution.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,19 @@ adopter-facing page.
still installs them intact — measured on Codex, which used to install such a
plugin with zero skills and no error. Every catalogue lists all ten families.

- **Substrate plugins publish a tool, not a family.** Beside the ten
families, the Claude Code catalogue carries three substrate plugins —
`magpie-agent-guard`, `magpie-vetted-ops` and `magpie-adversarial-review`
(#1368) — declared in `SUBSTRATE_PLUGINS` in `check-family-plugins.py`.
Each inherits the shared manifest metadata, declares no `skills`, exposes
its `tools/<name>` through a symlink whose entry point must resolve, and
declares hook wiring only where it has a hook (agent-guard's
`PreToolUse`). They exist to run the tool from the installed plugin tree:
the code a sandbox exclusion or a hook executes must sit where the agent
calling it cannot rewrite it. A tool shipped this way must resolve
outside the workspace, so it declares no workspace-only `dev` dependency
group (#1357).

- **The same skill is invoked by a different name per install method**, and
both are correct: `/magpie-<name>` under the portable snapshot install (where
the `magpie-` prefix on the install directory *is* the namespace), and
Expand Down
Loading