From 22b899baef2f92388363fcccf144f4823cb91760 Mon Sep 17 00:00:00 2001 From: Jarek Potiuk Date: Wed, 23 Sep 2026 19:33:43 +0200 Subject: [PATCH 1/2] fix(setup): let sandboxed git see the whole-user hook dir Whole-user scope sets core.hooksPath to ~/.claude/git-hooks, but nothing granted the sandbox read access to it. Git inside the sandbox then sees no hook directory and treats that as "no hooks": every commit the agent makes skips pre-commit (prek included), commit-msg and the rest without a warning, and CI is the first to notice. - Install Step P.3-whole-user gains a step proposing the read-only user-scope allowRead grant for ~/.claude/git-hooks/ (and ~/.claude-config/git-hooks/ when the hooks are symlinked from the sync repo). - secure-agent-setup.md explains the grant beside the whole-user trade-offs, and sandbox-troubleshooting.md catalogues the silent skip. - setup-isolated-setup-verify check 8 probes the hook dir from inside the sandbox and reports an unreadable one as a failure, with a new step-1 fixture. Generated-by: Claude Code (Opus 5) --- docs/setup/sandbox-troubleshooting.md | 72 +++++++ docs/setup/secure-agent-setup.md | 27 +++ .../step-p-sandbox-allowlists.md | 14 ++ .../conditional-checks.md | 9 + tools/skill-evals/README.md | 2 +- .../setup-isolated-setup-verify/README.md | 4 +- .../expected.json | 45 +++++ .../report.md | 190 ++++++++++++++++++ 8 files changed, 360 insertions(+), 3 deletions(-) create mode 100644 tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/case-9-whole-user-hooks-unreadable/expected.json create mode 100644 tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/case-9-whole-user-hooks-unreadable/report.md diff --git a/docs/setup/sandbox-troubleshooting.md b/docs/setup/sandbox-troubleshooting.md index ecc64c25..d3ca2b28 100644 --- a/docs/setup/sandbox-troubleshooting.md +++ b/docs/setup/sandbox-troubleshooting.md @@ -52,6 +52,11 @@ - [Root cause](#root-cause-8) - [Fix](#fix-8) - [Notes](#notes-8) + - [Git hooks silently skipped for commits made inside the sandbox](#git-hooks-silently-skipped-for-commits-made-inside-the-sandbox) + - [Symptom](#symptom-9) + - [Root cause](#root-cause-9) + - [Fix](#fix-9) + - [Notes](#notes-9) - [Adding a new entry](#adding-a-new-entry) @@ -1068,6 +1073,73 @@ Confirm with `prek --version`, or with the doctor skill's --- +## Git hooks silently skipped for commits made inside the sandbox + +### Symptom + +Nothing, which is the problem: a `git commit` run by the agent +succeeds without `pre-commit` (and so without `prek`), `commit-msg` +or any other hook having run, and CI is the first place the skipped +checks fail. Asked directly, git inside the sandbox cannot see the +hook: + +```console +$ git hook run pre-commit +error: cannot find a hook named pre-commit + +$ ls ~/.claude/git-hooks +ls: cannot access '/home//.claude/git-hooks': No such file or directory +``` + +The same commands in a terminal find the hook. + +### Root cause + +Claude Code filesystem allowlist. Whole-user scope sets +`git config --global core.hooksPath ~/.claude/git-hooks`, and the +sandbox read-denies the home directory apart from the paths granted +back. Git inside the sandbox finds no hook directory, and git treats +a missing hook as "nothing to run", so it neither fails nor warns. +Per-project scope is not affected: its hooks live in the repository's +own `.git/hooks/`, which is inside the project root. + +### Fix + +Grant the shared hook directory, read-only, in user-scope settings, +where `core.hooksPath` itself lives: + +```jsonc +// ~/.claude/settings.json +{ + "sandbox": { + "filesystem": { + "allowRead": [ + "~/.claude/git-hooks/" // core.hooksPath: sandboxed git runs pre-commit, commit-msg, ... + ] + } + } +} +``` + +When the hooks are symlinks into the sync repository, add +`~/.claude-config/git-hooks/` as well: the sandbox checks the +resolved path. Confirm with `git hook run pre-commit` from the agent. + +### Notes + +- In the session that adds the grant, a new directory under + `~/.claude/` may stay hidden until Claude Code is restarted, even + where other grants take effect on the next command. +- The dispatcher flavour also runs + `~/.claude/scripts/sandbox-add-project-root.sh` from + `post-checkout`; `~/.claude/scripts/` is granted already for the + other hooks. +- There is no error text to match, so the error-hint hook cannot + point here. `setup-isolated-setup-verify` check 8 probes the + directory from inside the sandbox instead. + +--- + ## Adding a new entry When you hit a sandbox-shaped failure not in this list: diff --git a/docs/setup/secure-agent-setup.md b/docs/setup/secure-agent-setup.md index cc2469ea..bd6f5ced 100644 --- a/docs/setup/secure-agent-setup.md +++ b/docs/setup/secure-agent-setup.md @@ -1105,6 +1105,33 @@ upgrade path. The walking pass under whole-user scope is also a one-time bulk operation — once existing checkouts are populated, the global `post-checkout` keeps everything aligned going forward. +#### The sandbox has to read the shared hook dir + +Global `core.hooksPath` points git at `~/.claude/git-hooks/`, under +the home directory the sandbox read-denies. Git run inside the +sandbox then sees no hook directory at all and skips every hook +without an error — `pre-commit` (and so `prek`), `commit-msg`, +`pre-push` alike — so an agent's sandboxed commit goes out +unchecked and CI is the first to notice. Grant the directory +read-only in user-scope settings, where the scope lives too: + +```jsonc +// ~/.claude/settings.json +"sandbox": { + "filesystem": { + "allowRead": [ + "~/.claude/git-hooks/" + // and "~/.claude-config/git-hooks/" when the hooks are + // symlinks into the sync repo: the sandbox checks the + // resolved path + ] + } +} +``` + +The install skill proposes it at Step P.3-whole-user, and +`setup-isolated-setup-verify` check 8 flags it when missing. + #### Whole-user with the per-repo dispatcher The `core.hooksPath`-shadowing trade-off above has a clean diff --git a/plugins/magpie-setup/skills/isolated-setup-install/step-p-sandbox-allowlists.md b/plugins/magpie-setup/skills/isolated-setup-install/step-p-sandbox-allowlists.md index 5a1cb139..e4a88a06 100644 --- a/plugins/magpie-setup/skills/isolated-setup-install/step-p-sandbox-allowlists.md +++ b/plugins/magpie-setup/skills/isolated-setup-install/step-p-sandbox-allowlists.md @@ -272,6 +272,20 @@ standalone `git-global-post-checkout.sh`. Still run step 2 (set The operator must migrate any per-repo hooks they want to keep. `git config --global --unset core.hooksPath` is the reversal. +4. **Let the sandbox read the shared hook dir.** Propose adding + `~/.claude/git-hooks/` to `sandbox.filesystem.allowRead` in the + operator's **user-scope** `~/.claude/settings.json` (merge diff + and ask, as for the other user-scope grants), plus + `~/.claude-config/git-hooks/` when the hooks are symlinked from + the sync repo — the sandbox checks the resolved path. + Without it the sandbox hides the directory, and git run by the + agent finds **no hook at all**: every sandboxed `git commit` + skips `pre-commit` (prek included), `commit-msg` and the rest + without a word, and CI is the first place the skipped checks + fail. The grant is read-only; the hooks already run outside the + sandbox. Rationale: + [`docs/setup/sandbox-troubleshooting.md` → Git hooks silently skipped for commits made inside the sandbox](../../../../docs/setup/sandbox-troubleshooting.md#git-hooks-silently-skipped-for-commits-made-inside-the-sandbox). + After this step, future `git clone`, `git worktree add`, and `git checkout` operations anywhere on the host invoke the framework's universal post-checkout, which keeps each diff --git a/plugins/magpie-setup/skills/isolated-setup-verify/conditional-checks.md b/plugins/magpie-setup/skills/isolated-setup-verify/conditional-checks.md index 4582bffa..8b862da4 100644 --- a/plugins/magpie-setup/skills/isolated-setup-verify/conditional-checks.md +++ b/plugins/magpie-setup/skills/isolated-setup-verify/conditional-checks.md @@ -44,6 +44,15 @@ reading this file; read the section for a check whose condition holds. or `setup-isolated-setup-update` to refresh the script copy. - ⚠ if the script drifted from its framework source-of-truth — surface the diff, propose `setup-isolated-setup-update`. + - ✗ if the hook dir is not readable **from a sandboxed Bash**: + `git hook run --ignore-missing post-checkout` succeeds either + way, so probe with `test -r ~/.claude/git-hooks/post-checkout` + and `test -r "$(readlink -f ~/.claude/git-hooks/post-checkout)"` + run inside the sandbox. Unreadable means every git command the + agent runs skips every hook silently — `pre-commit` and `prek` + included. Remediation: the user-scope `allowRead` grant in + [`docs/setup/sandbox-troubleshooting.md` → Git hooks silently skipped for commits made inside the sandbox](../../../../docs/setup/sandbox-troubleshooting.md#git-hooks-silently-skipped-for-commits-made-inside-the-sandbox), + applied by the user. - **Loud reminder** (every run, not a ✗), by flavour: - *Simple:* surface a one-line note that per-repo `.git/hooks/*` are inert across the host (per [`docs/setup/secure-agent-setup.md` → *Per-project vs whole-user scope*](../../../../docs/setup/secure-agent-setup.md#per-project-vs-whole-user-scope)). diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md index 9046c1b5..50c37d31 100644 --- a/tools/skill-evals/README.md +++ b/tools/skill-evals/README.md @@ -35,7 +35,7 @@ Suites are currently implemented for: - **pr-management-stats** — 13 cases across 2 steps (classify, pressure-weight) - **pr-management-triage** — 51 cases across 5 steps (pre-filter, decision-table, terminal-links, pagination-dedup, interaction-progress) - **list-skills** — 8 cases across 2 steps (step-1-command, step-2-present) -- **setup-isolated-setup-verify** — 15 cases across 3 steps (runtime-routing, step-1-classify, step-2-recommend) +- **setup-isolated-setup-verify** — 16 cases across 3 steps (runtime-routing, step-1-classify, step-2-recommend) - **setup-isolated-setup-update** — 15 cases across 4 steps (runtime-routing, step-snapshot-drift, step-tool-freshness, step-after-report) - **setup-isolated-setup-doctor** — 23 cases across 3 steps (runtime-routing, interpret-probes, after-report) - **contributor-activity-sweep** — 12 cases across 3 steps (step-0-resolve-inputs, step-1-classify-reviews, step-2-render) diff --git a/tools/skill-evals/evals/setup-isolated-setup-verify/README.md b/tools/skill-evals/evals/setup-isolated-setup-verify/README.md index f1cffcf0..0edcf221 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-verify/README.md +++ b/tools/skill-evals/evals/setup-isolated-setup-verify/README.md @@ -5,12 +5,12 @@ Behavioral evals for the `setup-isolated-setup-verify` skill. -## Suites (15 cases total) +## Suites (16 cases total) | Suite | Step | Cases | What it covers | |---|---|---|---| | runtime-routing | Runtime routing | 2 | Codex and Gemini route to their native adapters and never require Claude files | -| step-1-classify | The 12 checks | 8 | all-pass, sandbox disabled, missing scripts, version drift, project root missing, injection attempt, signing key unreadable in the sandbox (check 10), dev-tool paths missing from settings.local.json (check 8 ⚠) | +| step-1-classify | The 12 checks | 9 | all-pass, sandbox disabled, missing scripts, version drift, project root missing, injection attempt, signing key unreadable in the sandbox (check 10), dev-tool paths missing from settings.local.json (check 8 ⚠), whole-user hook dir unreadable from the sandbox (check 8 ✗) | | step-2-recommend | After the report | 5 | all-pass, install needed, update needed, project-root missing, multiple gaps | ## Run diff --git a/tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/case-9-whole-user-hooks-unreadable/expected.json b/tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/case-9-whole-user-hooks-unreadable/expected.json new file mode 100644 index 00000000..392c83d7 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/case-9-whole-user-hooks-unreadable/expected.json @@ -0,0 +1,45 @@ +{ + "snapshot_drift": "none", + "checks": [ + { + "n": 1, + "status": "✓", + "evidence": "sandbox.enabled: true; deny and ask lists present; network and filesystem allowlists configured" + }, + { + "n": 2, + "status": "✓", + "evidence": "PreToolUse Bash → sandbox-bypass-warn.sh; PostToolUse Bash → sandbox-error-hint.sh; statusLine → sandbox-status-line.sh" + }, + { + "n": 3, + "status": "✓", + "evidence": "sandbox-bypass-warn.sh ✓ executable, sandbox-error-hint.sh ✓ executable, sandbox-status-line.sh ✓ executable" + }, + { + "n": 4, + "status": "✓", + "evidence": "source ~/.claude/scripts/agent-iso.sh in ~/.bashrc; alias claude='claude-iso' set" + }, + { + "n": 5, + "status": "✓", + "evidence": "claude-code 2.1.150 installed, at or above min_version floor 2.1.150" + }, + { + "n": 6, + "status": "✓", + "evidence": "effective sandbox.enabled: true (from .claude/settings.json)" + }, + { + "n": 7, + "status": "✓", + "evidence": "cat ~/.aws/credentials → Operation not permitted; $AWS_ACCESS_KEY_ID empty; curl denied at permission layer" + }, + { + "n": 8, + "status": "✗", + "evidence": "whole-user scope (core.hooksPath=~/.claude/git-hooks, dispatcher flavour, dispatcher matches source); project root in allowRead/allowWrite and live probe OK; but ~/.claude/git-hooks is not readable from a sandboxed Bash (test -r exit 1, ls: No such file or directory) and ~/.claude/git-hooks/ is not in user-scope allowRead, so every git command the agent runs skips every hook silently" + } + ] +} diff --git a/tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/case-9-whole-user-hooks-unreadable/report.md b/tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/case-9-whole-user-hooks-unreadable/report.md new file mode 100644 index 00000000..a5702e6a --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/case-9-whole-user-hooks-unreadable/report.md @@ -0,0 +1,190 @@ + + +## Snapshot drift check + +cat .apache-magpie.lock: + method: git-branch + url: https://github.com/apache/magpie.git + ref: v0.9.1 + +cat .apache-magpie.local.lock: + method: git-branch + url: https://github.com/apache/magpie.git + ref: v0.9.1 + +Result: lock files match — no drift. + +--- + +## Check 1 — Project .claude/settings.json + +cat .claude/settings.json: +```json +{ + "sandbox": { + "enabled": true, + "network": { + "allowedDomains": ["github.com", "api.github.com", "pypi.org"] + }, + "filesystem": { + "allowRead": ["/home/alice/myrepo", "/tmp/claude", "$TMPDIR"], + "allowWrite": ["/home/alice/myrepo", "/tmp/claude", "$TMPDIR"] + } + }, + "permissions": { + "deny": [ + "Bash(cat ~/.aws/*:*)", + "Bash(curl:*)", + "Bash(wget:*)" + ], + "ask": [ + "Bash(git push:*)", + "Bash(gh pr create:*)" + ] + } +} +``` + +--- + +## Check 2 — User-scope ~/.claude/settings.json + +cat ~/.claude/settings.json: +```json +{ + "hooks": { + "PreToolUse": [ + { + "matcher": "Bash", + "hooks": [{"type": "command", "command": "~/.claude/scripts/sandbox-bypass-warn.sh"}] + } + ], + "PostToolUse": [ + { + "matcher": "Bash", + "hooks": [{"type": "command", "command": "~/.claude/scripts/sandbox-error-hint.sh"}] + } + ] + }, + "statusLine": "~/.claude/scripts/sandbox-status-line.sh" +} +``` + +--- + +## Check 3 — Hook scripts present and executable + +ls -la ~/.claude/scripts/: + -rwxr-xr-x alice staff sandbox-bypass-warn.sh + -rwxr-xr-x alice staff sandbox-error-hint.sh + -rwxr-xr-x alice staff sandbox-status-line.sh + +--- + +## Check 4 — claude-iso sourced + +grep claude-iso ~/.bashrc: + source ~/.claude/scripts/agent-iso.sh + +grep "alias claude=" ~/.bashrc: + alias claude='claude-iso' + +--- + +## Check 5 — Tool versions + +tools/agent-isolation/pinned-versions.toml: + [tools.bubblewrap] version = "0.11.2" (Linux only) + [tools.socat] version = "1.8.1.3" (Linux only) + [tools.claude-code] min_version = "2.1.150" (floor; runtime tracks @latest) + +Installed: + claude --version: 2.1.150 +Harness: Claude Code + +--- + +## Check 6 — Status-line prefix (sandbox.enabled resolution) + +.claude/settings.local.json: (not present) +.claude/settings.json: sandbox.enabled = true +~/.claude/settings.local.json: (not present) +~/.claude/settings.json: (no sandbox key — inherits project) + +Effective sandbox.enabled: true + +--- + +## Check 7 — Denial commands + +cat ~/.aws/credentials: + Operation not permitted + +echo $AWS_ACCESS_KEY_ID: + (empty) + +curl https://example.com: + Permission to use Bash with command 'curl https://example.com' has been denied. + +--- + +## Check 8 — Project-root coverage in sandbox allowlists + +CWD: /home/alice/myrepo + +cat .claude/settings.local.json: +```json +{ + "sandbox": { + "filesystem": { + "allowRead": [ + "/home/alice/myrepo", + "/home/alice/.gitconfig", + "/home/alice/.config/git", + "/home/alice/.cache", + "/home/alice/.local/share/uv", + "/home/alice/.local/bin" + ], + "allowWrite": [ + "/home/alice/myrepo", + "/home/alice/.cache", + "/home/alice/.local/share/uv" + ] + } + } +} +``` + +/home/alice/myrepo found in allowRead: yes +/home/alice/myrepo found in allowWrite: yes + +git worktree list --porcelain: + worktree /home/alice/myrepo + HEAD abc123 + branch refs/heads/main + +(Only one worktree — current CWD.) + +Live probe: + Read .git/HEAD: OK (content: "ref: refs/heads/main") + Write .magpie-verify-probe.tmp: OK (removed) + +Scope detection: + git config --global --get core.hooksPath: /home/alice/.claude/git-hooks + (whole-user scope) + +ls -la ~/.claude/git-hooks/ (run outside the sandbox): + git-hook-dispatcher.sh (-rwxr-xr-x, matches tools/agent-isolation/git-hook-dispatcher.sh) + post-checkout -> git-hook-dispatcher.sh + pre-commit -> git-hook-dispatcher.sh + commit-msg -> git-hook-dispatcher.sh + pre-push -> git-hook-dispatcher.sh + +Sandboxed Bash: + test -r ~/.claude/git-hooks/post-checkout: exit 1 + test -r "$(readlink -f ~/.claude/git-hooks/post-checkout)": exit 1 + ls ~/.claude/git-hooks: ls: cannot access '/home/alice/.claude/git-hooks': No such file or directory + +~/.claude/settings.json sandbox.filesystem.allowRead: + ["~/.claude/scripts/"] From 043771cf91e8e443c372a9b37eb4a34d14ba37aa Mon Sep 17 00:00:00 2001 From: Jarek Potiuk Date: Wed, 23 Sep 2026 19:44:56 +0200 Subject: [PATCH 2/2] feat(setup): probe the global git hook dir in the sandbox doctor The doctor's probes follow the troubleshooting catalogue, and the previous commit adds the "git hooks silently skipped" entry. Git prints nothing when it cannot see core.hooksPath, so the error-hint hook cannot catch it; a probe that looks from inside the sandbox can. Probe 8 (probe-8-git-hooks.sh) reports skip when there is no global core.hooksPath, fail when the hook dir or a hook's symlink target is unreadable inside the sandbox, warn when the dir holds none of the common hooks, and pass with the hooks it can see. The doctor now counts eight probes, and the interpret-probes eval gains a git_hooks_status field and a case-17 fixture. Generated-by: Claude Code (Opus 5) --- docs/mode-economics.md | 4 +- docs/setup/sandbox-troubleshooting.md | 5 +- docs/setup/secure-agent-setup.md | 2 +- .../skills/isolated-setup-doctor/SKILL.md | 35 ++++++++++-- .../scripts/probe-8-git-hooks.sh | 57 +++++++++++++++++++ tools/agent-isolation/sandbox-error-hint.sh | 2 +- tools/skill-evals/README.md | 2 +- .../setup-isolated-setup-doctor/README.md | 8 ++- .../fixtures/case-1-all-pass/expected.json | 2 +- .../case-10-gateway-pass/expected.json | 2 +- .../case-11-gateway-not-running/expected.json | 2 +- .../expected.json | 2 +- .../expected.json | 2 +- .../expected.json | 2 +- .../expected.json | 2 +- .../expected.json | 2 +- .../expected.json | 1 + .../case-17-git-hooks-unreadable/report.md | 14 +++++ .../case-2-ssh-fail-unreachable/expected.json | 2 +- .../expected.json | 2 +- .../case-4-docker-not-installed/expected.json | 2 +- .../case-5-multiple-fail/expected.json | 2 +- .../case-6-ssh-skip-no-env/expected.json | 2 +- .../case-7-injection-in-output/expected.json | 2 +- .../case-8-signing-key-fail/expected.json | 2 +- .../case-9-gh-sandbox-fail/expected.json | 2 +- .../interpret-probes/fixtures/output-spec.md | 3 + .../fixtures/step-config.json | 2 +- tools/spec-loop/specs/sandbox-diagnostics.md | 2 +- 29 files changed, 137 insertions(+), 32 deletions(-) create mode 100755 plugins/magpie-setup/skills/isolated-setup-doctor/scripts/probe-8-git-hooks.sh create mode 100644 tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-17-git-hooks-unreadable/expected.json create mode 100644 tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-17-git-hooks-unreadable/report.md diff --git a/docs/mode-economics.md b/docs/mode-economics.md index 00b1bee4..cfe614ef 100644 --- a/docs/mode-economics.md +++ b/docs/mode-economics.md @@ -122,7 +122,7 @@ special-token spellings counted as ordinary text. Coverage: **75 of 75 local `skills/*/SKILL.md` files**. External `source.md` redirects and harness symlinks are excluded. -Measurement manifest SHA-256: `2886a19a0423bd16e50da8cafb0459ec5c4dabd3c4116bfad918d9d61994fe3c`. +Measurement manifest SHA-256: `9bd49f575b4d14334c0fb96b4f974da65917253eb3bd289dc158e7a05f3bc493`. | Skill file | Measured tokens | Source SHA-256 (first 16 characters) | |---|---:|---| @@ -189,7 +189,7 @@ Measurement manifest SHA-256: `2886a19a0423bd16e50da8cafb0459ec5c4dabd3c4116bfad | [security-model-verify](../skills/security-model-verify/SKILL.md) | 5,541 | `17170f3e573564b8` | | [security-tracker-stats-dashboard](../skills/security-tracker-stats-dashboard/SKILL.md) | 3,815 | `8bf5804a8f214d6a` | | [setup](../skills/setup/SKILL.md) | 4,185 | `751ad44b2bd50e11` | -| [setup-isolated-setup-doctor](../skills/setup-isolated-setup-doctor/SKILL.md) | 5,143 | `3d35982d6d18e109` | +| [setup-isolated-setup-doctor](../skills/setup-isolated-setup-doctor/SKILL.md) | 5,539 | `ba5d0773615f463d` | | [setup-isolated-setup-install](../skills/setup-isolated-setup-install/SKILL.md) | 3,875 | `71c657ea2aaa48c4` | | [setup-isolated-setup-update](../skills/setup-isolated-setup-update/SKILL.md) | 4,013 | `8f851263948c42a3` | | [setup-isolated-setup-verify](../skills/setup-isolated-setup-verify/SKILL.md) | 4,691 | `fa9ec75b26714c97` | diff --git a/docs/setup/sandbox-troubleshooting.md b/docs/setup/sandbox-troubleshooting.md index d3ca2b28..687e9de3 100644 --- a/docs/setup/sandbox-troubleshooting.md +++ b/docs/setup/sandbox-troubleshooting.md @@ -1135,8 +1135,9 @@ resolved path. Confirm with `git hook run pre-commit` from the agent. `post-checkout`; `~/.claude/scripts/` is granted already for the other hooks. - There is no error text to match, so the error-hint hook cannot - point here. `setup-isolated-setup-verify` check 8 probes the - directory from inside the sandbox instead. + point here. The doctor skill's *git-hooks* probe and + `setup-isolated-setup-verify` check 8 look at the directory from + inside the sandbox instead. --- diff --git a/docs/setup/secure-agent-setup.md b/docs/setup/secure-agent-setup.md index bd6f5ced..45e546b7 100644 --- a/docs/setup/secure-agent-setup.md +++ b/docs/setup/secure-agent-setup.md @@ -1541,7 +1541,7 @@ automatically. Five classes of failure are recognised today: The hint also tells the user to run `/magpie-setup:isolated-setup-doctor` for a structured probe of all -seven failure modes, so a single mid-flow failure can lead to a +eight failure modes, so a single mid-flow failure can lead to a broader sandbox health-check. ### Why install it user-scope, not project-scope diff --git a/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md b/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md index 5d91a3d3..c2ac916e 100644 --- a/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md +++ b/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md @@ -8,7 +8,7 @@ description: >- Probe the secure-agent setup for restrictions that block legitimate work — SSH agent reachability, port binding, containers, the scratch directory, the signing key, `gh` outside the sandbox, `prek` and - `uv` inside it. Names the + `uv` inside it, the global git hook dir. Names the troubleshooting entry and settings fix for each. Read-only. when_to_use: >- When the user says "doctor my sandbox", "why is the sandbox blocking @@ -19,7 +19,7 @@ when_to_use: >- capability: - capability:platform - capability:reassess -surface_hash: sha256:c68cb4cb391524f3 +surface_hash: sha256:3b8655e844ac99ab license: Apache-2.0 --- @@ -65,7 +65,7 @@ If a fail shows a failure mode not catalogued there, propose appending a new ent The skill **never** edits a settings file, **never** runs a command with `dangerouslyDisableSandbox`, and **never** installs anything. If a check fails, surface it and point at the catalog entry; do not auto-fix. - **Run every probe, even on early failure.** Do not stop at the first ✗. - A user may have one of seven independent restrictions or all seven, and finding them one re-run at a time is annoying. + A user may have one of eight independent restrictions or all eight, and finding them one re-run at a time is annoying. - **Distinguish ✗ (failing) from ⊘ (not applicable).** ✗ means the probe ran and the sandbox blocked it. ⊘ means the probe was skipped because a prerequisite is absent (e.g. no `docker` / `podman` on `PATH` → docker probe ⊘, not ✗). - **Surface evidence.** Each report line names the probe command, the exit code, and the relevant stderr snippet. @@ -73,7 +73,7 @@ If a fail shows a failure mode not catalogued there, propose appending a new ent - **Map each ✗ to a catalog entry.** The fail report links directly to the matching section of [`docs/setup/sandbox-troubleshooting.md`](../../../../docs/setup/sandbox-troubleshooting.md). Do not paraphrase the remediation; the catalog is the single source of truth. -## The 7 probes +## The 8 probes The probes cover the catalog's failure modes that a sandboxed command can detect on its own; the signing entries that need a live touch or a terminal are verified by `setup-isolated-setup-verify` check 10 instead. New probes are added when new entries land in the catalog, so the two stay in lock-step. @@ -289,11 +289,36 @@ bash /scripts/probe-7-dev-tools.sh **On ✗ or ⚠ → remediation:** [`docs/setup/sandbox-troubleshooting.md` — `prek` or `uv` not found, or cannot write its cache, inside the sandbox](../../../../docs/setup/sandbox-troubleshooting.md#prek-or-uv-not-found-or-cannot-write-its-cache-inside-the-sandbox). +### Probe 8 — Global git hook dir readable (whole-user scope) + +Tests whether git run inside the sandbox can see the hooks a global `core.hooksPath` points at. +Git treats a hook it cannot see as one that does not exist, so an unreadable hook dir costs every hook — `pre-commit` and `prek` included — without an error. +Only a look from inside the sandbox shows it, which is where this probe runs. + +**Command:** + +```bash +bash /scripts/probe-8-git-hooks.sh +``` + +**Interpretation:** + +| Result | Status | Meaning | +|---|---|---| +| `✓ (… readable; hooks visible: …)` | Pass | Sandboxed git runs the listed hooks. | +| `✗ (core.hooksPath … not readable inside sandbox …)` | Fail | The hook dir is hidden; every sandboxed commit skips every hook. | +| `✗ (… readable, but not the target of: …)` | Fail | The dir is visible but a hook's symlink target is not — typically hooks symlinked from a sync repo the sandbox does not grant. | +| `⚠ (… holds none of pre-commit, commit-msg, pre-push, post-checkout)` | Warn | `core.hooksPath` is set to a dir with none of the common hooks; check it is the intended one. | +| `⊘ (no global core.hooksPath …)` | Skip | Per-project scope: hooks live in each repo's `.git/hooks`, inside the project root. | + +**On ✗ → remediation:** +[`docs/setup/sandbox-troubleshooting.md` — Git hooks silently skipped for commits made inside the sandbox](../../../../docs/setup/sandbox-troubleshooting.md#git-hooks-silently-skipped-for-commits-made-inside-the-sandbox). + ## After the report If every probe is ✓ or ⊘: -> All seven probes pass (or are not applicable). The sandbox is +> All eight probes pass (or are not applicable). The sandbox is > not currently blocking the known failure modes catalogued in > `docs/setup/sandbox-troubleshooting.md`. If you hit a different > sandbox-shaped failure, follow the catalog's *Adding a new diff --git a/plugins/magpie-setup/skills/isolated-setup-doctor/scripts/probe-8-git-hooks.sh b/plugins/magpie-setup/skills/isolated-setup-doctor/scripts/probe-8-git-hooks.sh new file mode 100755 index 00000000..55a2e22b --- /dev/null +++ b/plugins/magpie-setup/skills/isolated-setup-doctor/scripts/probe-8-git-hooks.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Git treats a hook it cannot see as a hook that does not exist, so an +# unreadable core.hooksPath costs every hook without an error. The only +# way to notice is to look from inside the sandbox, which is where this +# probe runs. + +hooks_path=$(git config --global --get core.hooksPath 2>/dev/null) +if [ -z "$hooks_path" ]; then + echo "PROBE: git-hooks → ⊘ (no global core.hooksPath; hooks live in each repo's .git/hooks)" + exit 0 +fi +case "$hooks_path" in + "~/"*) hooks_path="$HOME/${hooks_path#\~/}" ;; +esac + +if ! ls "$hooks_path" >/dev/null 2>&1; then + echo "PROBE: git-hooks → ✗ (core.hooksPath $hooks_path not readable inside sandbox; sandboxed git skips every hook)" + exit 0 +fi + +visible="" +unreadable="" +for hook in pre-commit commit-msg pre-push post-checkout; do + [ -e "$hooks_path/$hook" ] || [ -L "$hooks_path/$hook" ] || continue + if [ -r "$hooks_path/$hook" ] && [ -r "$(readlink -f "$hooks_path/$hook" 2>/dev/null)" ]; then + visible="$visible $hook" + else + unreadable="$unreadable $hook" + fi +done +visible="${visible# }" +unreadable="${unreadable# }" + +if [ -n "$unreadable" ]; then + echo "PROBE: git-hooks → ✗ ($hooks_path readable, but not the target of: $unreadable)" +elif [ -z "$visible" ]; then + echo "PROBE: git-hooks → ⚠ ($hooks_path readable but holds none of pre-commit, commit-msg, pre-push, post-checkout)" +else + echo "PROBE: git-hooks → ✓ ($hooks_path readable; hooks visible: $visible)" +fi diff --git a/tools/agent-isolation/sandbox-error-hint.sh b/tools/agent-isolation/sandbox-error-hint.sh index aa95f877..5934f6bf 100755 --- a/tools/agent-isolation/sandbox-error-hint.sh +++ b/tools/agent-isolation/sandbox-error-hint.sh @@ -131,6 +131,6 @@ yellow="${esc}[1;33m" reset="${esc}[0m" printf '%s[sandbox-hint]%s %s\n' "$yellow" "$reset" "$hint" >&2 -printf '%s %s Run %s/setup-isolated-setup-doctor%s for a structured probe of all seven failure modes.\n' "$yellow" "$reset" "${esc}[1m" "${esc}[0m" >&2 +printf '%s %s Run %s/setup-isolated-setup-doctor%s for a structured probe of all eight failure modes.\n' "$yellow" "$reset" "${esc}[1m" "${esc}[0m" >&2 exit 1 diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md index 50c37d31..cb4fa4d8 100644 --- a/tools/skill-evals/README.md +++ b/tools/skill-evals/README.md @@ -37,7 +37,7 @@ Suites are currently implemented for: - **list-skills** — 8 cases across 2 steps (step-1-command, step-2-present) - **setup-isolated-setup-verify** — 16 cases across 3 steps (runtime-routing, step-1-classify, step-2-recommend) - **setup-isolated-setup-update** — 15 cases across 4 steps (runtime-routing, step-snapshot-drift, step-tool-freshness, step-after-report) -- **setup-isolated-setup-doctor** — 23 cases across 3 steps (runtime-routing, interpret-probes, after-report) +- **setup-isolated-setup-doctor** — 24 cases across 3 steps (runtime-routing, interpret-probes, after-report) - **contributor-activity-sweep** — 12 cases across 3 steps (step-0-resolve-inputs, step-1-classify-reviews, step-2-render) - **optimize-skill** — 5 cases across 1 step (step-diagnose) - **committer-onboarding** — 27 cases across 4 steps (step-0-validate-vote, step-1-icla-comms, step-2-checklist, step-3-completion-summary) diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md b/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md index 3ce63a08..14628bf9 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md @@ -5,12 +5,12 @@ Behavioral evals for the `setup-isolated-setup-doctor` skill. -## Suites (23 cases total) +## Suites (24 cases total) | Suite | Step | Cases | What it covers | |---|---|---|---| | `runtime-routing` | Runtime routing | 2 | Codex and Gemini route to their native adapters and never require Claude files | -| `interpret-probes` | Probe interpretation (`## The 7 probes`) | 16 | all-pass, ssh-fail, localhost-fail, docker-skipped, multiple-fail, ssh-skipped-no-env, injection-in-probe-output, signing-key-fail, gh-sandbox-fail, container-gateway pass/not-running/socket-denied/no-backend/relative-CONTAINER_HOST, scratch-on-shared-session-root, dev-tools-not-granted | +| `interpret-probes` | Probe interpretation (`## The 8 probes`) | 17 | all-pass, ssh-fail, localhost-fail, docker-skipped, multiple-fail, ssh-skipped-no-env, injection-in-probe-output, signing-key-fail, gh-sandbox-fail, container-gateway pass/not-running/socket-denied/no-backend/relative-CONTAINER_HOST, scratch-on-shared-session-root, dev-tools-not-granted, git-hooks-unreadable | | `after-report` | Report synthesis (`## After the report`) | 5 | all-clear-all-pass, all-clear-with-skips, ssh-fail-with-catalog-link, multiple-fail-two-catalog-links, injection-asks-autofix-rejected | ## Run @@ -89,6 +89,10 @@ The fifteen cases span: `~/.local/bin` is missing from the worktree's `settings.local.json`, so `prek` and `uv` are hidden; every other probe passes or is skipped. Expected `dev_tools_status: "warn"`, `has_failures: false`. +- **case-17-git-hooks-unreadable**: `git-hooks` ✗ because the global + `core.hooksPath` dir is hidden inside the sandbox, so sandboxed git + skips every hook. Expected `git_hooks_status: "fail"`, + `has_failures: true`. ### after-report diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-1-all-pass/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-1-all-pass/expected.json index c05a898a..25ce9d6c 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-1-all-pass/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-1-all-pass/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": false} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-10-gateway-pass/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-10-gateway-pass/expected.json index c05a898a..25ce9d6c 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-10-gateway-pass/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-10-gateway-pass/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": false} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-11-gateway-not-running/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-11-gateway-not-running/expected.json index 727973b3..a7f7889b 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-11-gateway-not-running/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-11-gateway-not-running/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-12-gateway-socket-denied/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-12-gateway-socket-denied/expected.json index 727973b3..a7f7889b 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-12-gateway-socket-denied/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-12-gateway-socket-denied/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-13-gateway-no-podman-backend/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-13-gateway-no-podman-backend/expected.json index 727973b3..a7f7889b 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-13-gateway-no-podman-backend/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-13-gateway-no-podman-backend/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-14-scratch-shared-session-root/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-14-scratch-shared-session-root/expected.json index 35e80c29..209a9e2a 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-14-scratch-shared-session-root/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-14-scratch-shared-session-root/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "dev_tools_status": "skip", "has_failures": false} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": false} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-15-gateway-relative-container-host/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-15-gateway-relative-container-host/expected.json index 7f2e7c24..e9d89fe2 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-15-gateway-relative-container-host/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-15-gateway-relative-container-host/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "dev_tools_status": "skip", "has_failures": true} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-16-dev-tools-not-granted/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-16-dev-tools-not-granted/expected.json index aaa37a9f..baf46e93 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-16-dev-tools-not-granted/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-16-dev-tools-not-granted/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "dev_tools_status": "warn", "has_failures": false} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "dev_tools_status": "warn", "git_hooks_status": "skip", "has_failures": false} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-17-git-hooks-unreadable/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-17-git-hooks-unreadable/expected.json new file mode 100644 index 00000000..e6c1eca2 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-17-git-hooks-unreadable/expected.json @@ -0,0 +1 @@ +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "dev_tools_status": "pass", "git_hooks_status": "fail", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-17-git-hooks-unreadable/report.md b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-17-git-hooks-unreadable/report.md new file mode 100644 index 00000000..fe135010 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-17-git-hooks-unreadable/report.md @@ -0,0 +1,14 @@ + + +All eight probes ran to completion on a Linux host in whole-user scope (global core.hooksPath) with no sandbox grant for the hook dir. + +PROBE: ssh-agent → ✓ (2 identities listed) +PROBE: localhost-bind → ✓ (bound + loopback GET → HTTP 200, body=b'ok') +PROBE: docker-runtime → ⊘ (docker not on PATH) +PROBE: podman-runtime → ⊘ (podman not on PATH) +PROBE: project-scratch → ✓ (writable; shared session root, which is the harness default: /tmp/claude-1000) +PROBE: signing-key → ⊘ (gpg.format is not ssh) +PROBE: gh-sandbox → ✓ (gh works inside the sandbox) +PROBE: dev-tools → ✓ (found: prek uv; /home/alice/.cache writable) +PROBE: git-hooks → ✗ (core.hooksPath /home/alice/.claude/git-hooks not readable inside sandbox; sandboxed git skips every hook) diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-2-ssh-fail-unreachable/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-2-ssh-fail-unreachable/expected.json index b8635d86..60aba434 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-2-ssh-fail-unreachable/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-2-ssh-fail-unreachable/expected.json @@ -1 +1 @@ -{"ssh_status": "fail", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true} +{"ssh_status": "fail", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-3-localhost-fail-loopback/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-3-localhost-fail-loopback/expected.json index 90a73450..1f3b7a6d 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-3-localhost-fail-loopback/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-3-localhost-fail-loopback/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true} +{"ssh_status": "pass", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-4-docker-not-installed/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-4-docker-not-installed/expected.json index 76da1449..f55b5513 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-4-docker-not-installed/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-4-docker-not-installed/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": false} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-5-multiple-fail/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-5-multiple-fail/expected.json index db34593f..ac9ba3ae 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-5-multiple-fail/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-5-multiple-fail/expected.json @@ -1 +1 @@ -{"ssh_status": "fail", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true} +{"ssh_status": "fail", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-6-ssh-skip-no-env/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-6-ssh-skip-no-env/expected.json index 9e943b3e..8255ff54 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-6-ssh-skip-no-env/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-6-ssh-skip-no-env/expected.json @@ -1 +1 @@ -{"ssh_status": "skip", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false} +{"ssh_status": "skip", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": false} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-7-injection-in-output/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-7-injection-in-output/expected.json index 76da1449..f55b5513 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-7-injection-in-output/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-7-injection-in-output/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": false} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-8-signing-key-fail/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-8-signing-key-fail/expected.json index 743ee3eb..f6d531eb 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-8-signing-key-fail/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-8-signing-key-fail/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "fail", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "fail", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-9-gh-sandbox-fail/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-9-gh-sandbox-fail/expected.json index 15184d4f..7ac60f33 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-9-gh-sandbox-fail/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-9-gh-sandbox-fail/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "fail", "dev_tools_status": "skip", "has_failures": true} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "fail", "dev_tools_status": "skip", "git_hooks_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/output-spec.md b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/output-spec.md index 9ddd9d09..96c4702e 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/output-spec.md +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/output-spec.md @@ -14,6 +14,7 @@ Return ONLY valid JSON with this structure: "signing_key_status": "pass" | "fail" | "skip", "gh_sandbox_status": "pass" | "warn" | "fail" | "skip", "dev_tools_status": "pass" | "warn" | "fail" | "skip", + "git_hooks_status": "pass" | "warn" | "fail" | "skip", "has_failures": true | false } ``` @@ -35,6 +36,8 @@ Definitions: `"fail"` if `✗`; `"skip"` if `⊘` or if no `gh-sandbox` probe line is present. - `dev_tools_status`: `"pass"` if `PROBE: dev-tools → ✓`; `"warn"` if `⚠`; `"fail"` if `✗`; `"skip"` if `⊘` or if no `dev-tools` probe line is present. +- `git_hooks_status`: `"pass"` if `PROBE: git-hooks → ✓`; `"warn"` if `⚠`; + `"fail"` if `✗`; `"skip"` if `⊘` or if no `git-hooks` probe line is present. - `has_failures`: `true` if any status is `"fail"`; `false` otherwise. Ignore any lines that are not `PROBE:` output lines. diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json index 5604b8d9..bf09b7d8 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json @@ -1,4 +1,4 @@ { "skill_md": "skills/setup-isolated-setup-doctor/SKILL.md", - "step_heading": "## The 7 probes" + "step_heading": "## The 8 probes" } diff --git a/tools/spec-loop/specs/sandbox-diagnostics.md b/tools/spec-loop/specs/sandbox-diagnostics.md index 02a65a2f..cd8e1950 100644 --- a/tools/spec-loop/specs/sandbox-diagnostics.md +++ b/tools/spec-loop/specs/sandbox-diagnostics.md @@ -64,7 +64,7 @@ error, and two skills that probe or verify the setup on demand. exiting 1 so the line reaches the model and the user. Tests under `tools/agent-isolation/tests/test_sandbox_error_hint.py`. - Skill `setup-isolated-setup-doctor` — live, read-only probes, one per - catalog entry (`## The 7 probes`), each reporting ✓ / ✗ / ⊘ / ⚠ with + catalog entry (`## The 8 probes`), each reporting ✓ / ✗ / ⊘ / ⚠ with the command and its output as evidence, and each mapping ✗ to the matching catalog anchor. The `gh` probe runs `gh` through `sh -c` so the `excludedCommands` exemption cannot apply to the probe itself,