From f19def088464c7e6c95551ea4bf71daa4e746b66 Mon Sep 17 00:00:00 2001 From: Jarek Potiuk Date: Wed, 23 Sep 2026 19:04:06 +0200 Subject: [PATCH] feat(setup): probe prek and uv in the sandbox doctor The doctor skill's probes are meant to move in lock-step with the sandbox troubleshooting catalogue, but #1359 added the "prek or uv not found inside the sandbox" entry without a matching probe. Probe 7 (probe-7-dev-tools.sh) checks that prek and uv run inside the sandbox and can write ~/.cache. Inside the sandbox a read-denied ~/.local/bin looks exactly like an absent one, so when neither tool is found it reads the worktree's settings.local.json to tell a missing grant (warn) from a tool that is not installed (skip). - The error-hint hook and the setup doc now count seven probes, and the doc's hint table gains the prek / uv signature from #1359. - The troubleshooting entry no longer says the new paths apply only from the next session: the harness picks up settings.local.json changes on the next sandboxed command. - The interpret-probes eval grows a dev_tools_status field and a case-16 fixture for the missing grant. Generated-by: Claude Code (Opus 5) --- docs/mode-economics.md | 4 +- docs/setup/sandbox-troubleshooting.md | 6 +- docs/setup/secure-agent-setup.md | 3 +- .../skills/isolated-setup-doctor/SKILL.md | 37 ++++++++++-- .../scripts/probe-7-dev-tools.sh | 57 +++++++++++++++++++ tools/agent-isolation/sandbox-error-hint.sh | 2 +- tools/skill-evals/README.md | 2 +- .../setup-isolated-setup-doctor/README.md | 8 ++- .../fixtures/case-1-all-pass/expected.json | 2 +- .../case-10-gateway-pass/expected.json | 2 +- .../case-11-gateway-not-running/expected.json | 2 +- .../expected.json | 2 +- .../expected.json | 2 +- .../expected.json | 2 +- .../expected.json | 2 +- .../expected.json | 1 + .../case-16-dev-tools-not-granted/report.md | 13 +++++ .../case-2-ssh-fail-unreachable/expected.json | 2 +- .../expected.json | 2 +- .../case-4-docker-not-installed/expected.json | 2 +- .../case-5-multiple-fail/expected.json | 2 +- .../case-6-ssh-skip-no-env/expected.json | 2 +- .../case-7-injection-in-output/expected.json | 2 +- .../case-8-signing-key-fail/expected.json | 2 +- .../case-9-gh-sandbox-fail/expected.json | 2 +- .../interpret-probes/fixtures/output-spec.md | 3 + .../fixtures/step-config.json | 2 +- tools/spec-loop/specs/sandbox-diagnostics.md | 2 +- 28 files changed, 138 insertions(+), 32 deletions(-) create mode 100755 plugins/magpie-setup/skills/isolated-setup-doctor/scripts/probe-7-dev-tools.sh create mode 100644 tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-16-dev-tools-not-granted/expected.json create mode 100644 tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-16-dev-tools-not-granted/report.md diff --git a/docs/mode-economics.md b/docs/mode-economics.md index 67e56c49d..00b1bee4a 100644 --- a/docs/mode-economics.md +++ b/docs/mode-economics.md @@ -122,7 +122,7 @@ special-token spellings counted as ordinary text. Coverage: **75 of 75 local `skills/*/SKILL.md` files**. External `source.md` redirects and harness symlinks are excluded. -Measurement manifest SHA-256: `b0563986d6e3186e1cb059f3680157ba9e45476a1dbecf225a7402744165f361`. +Measurement manifest SHA-256: `2886a19a0423bd16e50da8cafb0459ec5c4dabd3c4116bfad918d9d61994fe3c`. | Skill file | Measured tokens | Source SHA-256 (first 16 characters) | |---|---:|---| @@ -189,7 +189,7 @@ Measurement manifest SHA-256: `b0563986d6e3186e1cb059f3680157ba9e45476a1dbecf225 | [security-model-verify](../skills/security-model-verify/SKILL.md) | 5,541 | `17170f3e573564b8` | | [security-tracker-stats-dashboard](../skills/security-tracker-stats-dashboard/SKILL.md) | 3,815 | `8bf5804a8f214d6a` | | [setup](../skills/setup/SKILL.md) | 4,185 | `751ad44b2bd50e11` | -| [setup-isolated-setup-doctor](../skills/setup-isolated-setup-doctor/SKILL.md) | 4,716 | `22565dd6ddfd1c65` | +| [setup-isolated-setup-doctor](../skills/setup-isolated-setup-doctor/SKILL.md) | 5,143 | `3d35982d6d18e109` | | [setup-isolated-setup-install](../skills/setup-isolated-setup-install/SKILL.md) | 3,875 | `71c657ea2aaa48c4` | | [setup-isolated-setup-update](../skills/setup-isolated-setup-update/SKILL.md) | 4,013 | `8f851263948c42a3` | | [setup-isolated-setup-verify](../skills/setup-isolated-setup-verify/SKILL.md) | 4,691 | `fa9ec75b26714c97` | diff --git a/docs/setup/sandbox-troubleshooting.md b/docs/setup/sandbox-troubleshooting.md index 249a623f1..ecc64c25a 100644 --- a/docs/setup/sandbox-troubleshooting.md +++ b/docs/setup/sandbox-troubleshooting.md @@ -1047,8 +1047,10 @@ project root it already adds: ``` The helper writes that file only from outside the sandbox (it is in -the harness's write-deny set), and the change applies from the next -session. Confirm with `prek --version` in a fresh session. +the harness's write-deny set). The harness re-reads it without a +restart: the next sandboxed command already sees the new paths. +Confirm with `prek --version`, or with the doctor skill's +*dev-tools* probe. ### Notes diff --git a/docs/setup/secure-agent-setup.md b/docs/setup/secure-agent-setup.md index e43a8236a..cc2469ea8 100644 --- a/docs/setup/secure-agent-setup.md +++ b/docs/setup/secure-agent-setup.md @@ -1510,10 +1510,11 @@ automatically. Five classes of failure are recognised today: | `127.0.0.1 … Permission denied` / `Operation not permitted … bind` / `Errno 49 … assign requested address` / `Connection refused … 127.0.0.1` | [Localhost port-bind blocked](sandbox-troubleshooting.md#test-cannot-bind-to-a-localhost-port) | | `/tmp/…: Read-only file system` / `mktemp: failed to create` | [Temp files fail under `/tmp`](sandbox-troubleshooting.md#temp-files-fail-with-read-only-file-system-under-tmp) | | `x509: OSStatus -26276` / `HTTP 401: Requires authentication (https://api.github.com…` | [`gh` ran inside the sandbox](sandbox-troubleshooting.md#gh-fails-with-tls-osstatus--26276-or-http-401-inside-the-sandbox) | +| `command not found: prek` / `uv: command not found` (also `uvx`) | [`prek` or `uv` not found inside the sandbox](sandbox-troubleshooting.md#prek-or-uv-not-found-or-cannot-write-its-cache-inside-the-sandbox) | The hint also tells the user to run `/magpie-setup:isolated-setup-doctor` for a structured probe of all -six failure modes, so a single mid-flow failure can lead to a +seven failure modes, so a single mid-flow failure can lead to a broader sandbox health-check. ### Why install it user-scope, not project-scope diff --git a/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md b/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md index b4e80fea2..5d91a3d34 100644 --- a/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md +++ b/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md @@ -7,7 +7,8 @@ mode: Meta description: >- Probe the secure-agent setup for restrictions that block legitimate work — SSH agent reachability, port binding, containers, the scratch - directory, the signing key, `gh` outside the sandbox. Names the + directory, the signing key, `gh` outside the sandbox, `prek` and + `uv` inside it. Names the troubleshooting entry and settings fix for each. Read-only. when_to_use: >- When the user says "doctor my sandbox", "why is the sandbox blocking @@ -18,7 +19,7 @@ when_to_use: >- capability: - capability:platform - capability:reassess -surface_hash: sha256:42ee88aec8f2e3ad +surface_hash: sha256:c68cb4cb391524f3 license: Apache-2.0 --- @@ -64,7 +65,7 @@ If a fail shows a failure mode not catalogued there, propose appending a new ent The skill **never** edits a settings file, **never** runs a command with `dangerouslyDisableSandbox`, and **never** installs anything. If a check fails, surface it and point at the catalog entry; do not auto-fix. - **Run every probe, even on early failure.** Do not stop at the first ✗. - A user may have one of six independent restrictions or all six, and finding them one re-run at a time is annoying. + A user may have one of seven independent restrictions or all seven, and finding them one re-run at a time is annoying. - **Distinguish ✗ (failing) from ⊘ (not applicable).** ✗ means the probe ran and the sandbox blocked it. ⊘ means the probe was skipped because a prerequisite is absent (e.g. no `docker` / `podman` on `PATH` → docker probe ⊘, not ✗). - **Surface evidence.** Each report line names the probe command, the exit code, and the relevant stderr snippet. @@ -72,9 +73,9 @@ If a fail shows a failure mode not catalogued there, propose appending a new ent - **Map each ✗ to a catalog entry.** The fail report links directly to the matching section of [`docs/setup/sandbox-troubleshooting.md`](../../../../docs/setup/sandbox-troubleshooting.md). Do not paraphrase the remediation; the catalog is the single source of truth. -## The 6 probes +## The 7 probes -The probes cover the six failure modes the catalog documents. +The probes cover the catalog's failure modes that a sandboxed command can detect on its own; the signing entries that need a live touch or a terminal are verified by `setup-isolated-setup-verify` check 10 instead. New probes are added when new entries land in the catalog, so the two stay in lock-step. ### Probe 1 — SSH agent / Yubikey reachable @@ -264,11 +265,35 @@ When the user reports a `gh` failure this probe does not reproduce, ask for the **On ✗ → remediation:** [`docs/setup/sandbox-troubleshooting.md` — `gh` fails with TLS `OSStatus -26276` or `HTTP 401` inside the sandbox](../../../../docs/setup/sandbox-troubleshooting.md#gh-fails-with-tls-osstatus--26276-or-http-401-inside-the-sandbox). +### Probe 7 — `prek` and `uv` usable inside the sandbox + +Tests whether the dev tools the framework's hooks and Python tools run on are found and can write their cache from a sandboxed Bash call. +They live in `~/.local/bin`, which the harness reads only when the worktree's `.claude/settings.local.json` grants it; the committed project-scope entries are not applied. +Inside the sandbox a read-denied directory looks exactly like an absent one, so the probe reads `settings.local.json` to tell a missing grant from a tool that is not installed. + +**Command:** + +```bash +bash /scripts/probe-7-dev-tools.sh +``` + +**Interpretation:** + +| Result | Status | Meaning | +|---|---|---| +| `✓ (found: …; ~/.cache writable)` | Pass | `prek` / `uv` run inside the sandbox and can write their caches. | +| `✗ (found: …; ~/.cache not writable …)` | Fail | The tools start but cannot write their cache or state. | +| `⚠ (… not found; ~/.local/bin is not in … settings.local.json)` | Warn | The grant is missing; the tools are most likely installed but hidden. | +| `⊘ (… not found; ~/.local/bin is granted …)` | Skip | The grant is present and neither tool is installed there; not a sandbox restriction. | + +**On ✗ or ⚠ → remediation:** +[`docs/setup/sandbox-troubleshooting.md` — `prek` or `uv` not found, or cannot write its cache, inside the sandbox](../../../../docs/setup/sandbox-troubleshooting.md#prek-or-uv-not-found-or-cannot-write-its-cache-inside-the-sandbox). + ## After the report If every probe is ✓ or ⊘: -> All six probes pass (or are not applicable). The sandbox is +> All seven probes pass (or are not applicable). The sandbox is > not currently blocking the known failure modes catalogued in > `docs/setup/sandbox-troubleshooting.md`. If you hit a different > sandbox-shaped failure, follow the catalog's *Adding a new diff --git a/plugins/magpie-setup/skills/isolated-setup-doctor/scripts/probe-7-dev-tools.sh b/plugins/magpie-setup/skills/isolated-setup-doctor/scripts/probe-7-dev-tools.sh new file mode 100755 index 000000000..cdc0e56de --- /dev/null +++ b/plugins/magpie-setup/skills/isolated-setup-doctor/scripts/probe-7-dev-tools.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Inside the sandbox a read-denied ~/.local/bin looks exactly like an +# absent one, so "not found" alone cannot tell a sandbox gap from a tool +# that was never installed. The worktree's settings.local.json is +# readable, and it records whether the path was granted. + +found="" +missing="" +for tool in prek uv; do + if command -v "$tool" >/dev/null 2>&1 && "$tool" --version >/dev/null 2>&1; then + found="$found $tool" + else + missing="$missing $tool" + fi +done +found="${found# }" +missing="${missing# }" + +local_settings="$(git rev-parse --show-toplevel 2>/dev/null)/.claude/settings.local.json" +granted=0 +if [ -n "${HOME:-}" ] && [ -f "$local_settings" ] && grep -qF "\"$HOME/.local/bin\"" "$local_settings" 2>/dev/null; then + granted=1 +fi + +if [ -z "$found" ]; then + if [ "$granted" -eq 1 ]; then + echo "PROBE: dev-tools → ⊘ (prek and uv not found; $HOME/.local/bin is granted, so they are not installed there)" + else + echo "PROBE: dev-tools → ⚠ (prek and uv not found; $HOME/.local/bin is not in $local_settings)" + fi +elif [ -n "${HOME:-}" ] && ! { mkdir -p "$HOME/.cache" 2>/dev/null && touch "$HOME/.cache/.doctor-probe" 2>/dev/null; }; then + echo "PROBE: dev-tools → ✗ (found: $found; $HOME/.cache not writable inside sandbox)" +else + rm -f "$HOME/.cache/.doctor-probe" + if [ -n "$missing" ]; then + echo "PROBE: dev-tools → ✓ (found: $found; not found: $missing; $HOME/.cache writable)" + else + echo "PROBE: dev-tools → ✓ (found: $found; $HOME/.cache writable)" + fi +fi diff --git a/tools/agent-isolation/sandbox-error-hint.sh b/tools/agent-isolation/sandbox-error-hint.sh index d893bcd73..aa95f8770 100755 --- a/tools/agent-isolation/sandbox-error-hint.sh +++ b/tools/agent-isolation/sandbox-error-hint.sh @@ -131,6 +131,6 @@ yellow="${esc}[1;33m" reset="${esc}[0m" printf '%s[sandbox-hint]%s %s\n' "$yellow" "$reset" "$hint" >&2 -printf '%s %s Run %s/setup-isolated-setup-doctor%s for a structured probe of all six failure modes.\n' "$yellow" "$reset" "${esc}[1m" "${esc}[0m" >&2 +printf '%s %s Run %s/setup-isolated-setup-doctor%s for a structured probe of all seven failure modes.\n' "$yellow" "$reset" "${esc}[1m" "${esc}[0m" >&2 exit 1 diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md index 1ea95075f..9046c1b5d 100644 --- a/tools/skill-evals/README.md +++ b/tools/skill-evals/README.md @@ -37,7 +37,7 @@ Suites are currently implemented for: - **list-skills** — 8 cases across 2 steps (step-1-command, step-2-present) - **setup-isolated-setup-verify** — 15 cases across 3 steps (runtime-routing, step-1-classify, step-2-recommend) - **setup-isolated-setup-update** — 15 cases across 4 steps (runtime-routing, step-snapshot-drift, step-tool-freshness, step-after-report) -- **setup-isolated-setup-doctor** — 22 cases across 3 steps (runtime-routing, interpret-probes, after-report) +- **setup-isolated-setup-doctor** — 23 cases across 3 steps (runtime-routing, interpret-probes, after-report) - **contributor-activity-sweep** — 12 cases across 3 steps (step-0-resolve-inputs, step-1-classify-reviews, step-2-render) - **optimize-skill** — 5 cases across 1 step (step-diagnose) - **committer-onboarding** — 27 cases across 4 steps (step-0-validate-vote, step-1-icla-comms, step-2-checklist, step-3-completion-summary) diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md b/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md index e501e6e21..3ce63a086 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md @@ -5,12 +5,12 @@ Behavioral evals for the `setup-isolated-setup-doctor` skill. -## Suites (22 cases total) +## Suites (23 cases total) | Suite | Step | Cases | What it covers | |---|---|---|---| | `runtime-routing` | Runtime routing | 2 | Codex and Gemini route to their native adapters and never require Claude files | -| `interpret-probes` | Probe interpretation (`## The 6 probes`) | 15 | all-pass, ssh-fail, localhost-fail, docker-skipped, multiple-fail, ssh-skipped-no-env, injection-in-probe-output, signing-key-fail, gh-sandbox-fail, container-gateway pass/not-running/socket-denied/no-backend/relative-CONTAINER_HOST, scratch-on-shared-session-root | +| `interpret-probes` | Probe interpretation (`## The 7 probes`) | 16 | all-pass, ssh-fail, localhost-fail, docker-skipped, multiple-fail, ssh-skipped-no-env, injection-in-probe-output, signing-key-fail, gh-sandbox-fail, container-gateway pass/not-running/socket-denied/no-backend/relative-CONTAINER_HOST, scratch-on-shared-session-root, dev-tools-not-granted | | `after-report` | Report synthesis (`## After the report`) | 5 | all-clear-all-pass, all-clear-with-skips, ssh-fail-with-catalog-link, multiple-fail-two-catalog-links, injection-asks-autofix-rejected | ## Run @@ -85,6 +85,10 @@ The fifteen cases span: because `CONTAINER_HOST` uses a project-relative `unix://./…` value, which the CLIs do not resolve against the cwd. Expected `docker_status: "fail"`, `has_failures: true`. +- **case-16-dev-tools-not-granted**: `dev-tools` ⚠ because + `~/.local/bin` is missing from the worktree's `settings.local.json`, + so `prek` and `uv` are hidden; every other probe passes or is + skipped. Expected `dev_tools_status: "warn"`, `has_failures: false`. ### after-report diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-1-all-pass/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-1-all-pass/expected.json index 5ac3edf2d..c05a898af 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-1-all-pass/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-1-all-pass/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": false} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-10-gateway-pass/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-10-gateway-pass/expected.json index 5ac3edf2d..c05a898af 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-10-gateway-pass/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-10-gateway-pass/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": false} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-11-gateway-not-running/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-11-gateway-not-running/expected.json index 0115c3249..727973b35 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-11-gateway-not-running/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-11-gateway-not-running/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-12-gateway-socket-denied/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-12-gateway-socket-denied/expected.json index 0115c3249..727973b35 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-12-gateway-socket-denied/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-12-gateway-socket-denied/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-13-gateway-no-podman-backend/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-13-gateway-no-podman-backend/expected.json index 0115c3249..727973b35 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-13-gateway-no-podman-backend/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-13-gateway-no-podman-backend/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-14-scratch-shared-session-root/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-14-scratch-shared-session-root/expected.json index 94047cd95..35e80c293 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-14-scratch-shared-session-root/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-14-scratch-shared-session-root/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "has_failures": false} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "dev_tools_status": "skip", "has_failures": false} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-15-gateway-relative-container-host/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-15-gateway-relative-container-host/expected.json index bc8c041c3..7f2e7c246 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-15-gateway-relative-container-host/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-15-gateway-relative-container-host/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "has_failures": true} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "dev_tools_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-16-dev-tools-not-granted/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-16-dev-tools-not-granted/expected.json new file mode 100644 index 000000000..aaa37a9ff --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-16-dev-tools-not-granted/expected.json @@ -0,0 +1 @@ +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "dev_tools_status": "warn", "has_failures": false} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-16-dev-tools-not-granted/report.md b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-16-dev-tools-not-granted/report.md new file mode 100644 index 000000000..c334e56e8 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-16-dev-tools-not-granted/report.md @@ -0,0 +1,13 @@ + + +All seven probes ran to completion on a Linux host whose worktree settings.local.json carries only the project root. + +PROBE: ssh-agent → ✓ (2 identities listed) +PROBE: localhost-bind → ✓ (bound + loopback GET → HTTP 200, body=b'ok') +PROBE: docker-runtime → ⊘ (docker not on PATH) +PROBE: podman-runtime → ⊘ (podman not on PATH) +PROBE: project-scratch → ✓ (writable; shared session root, which is the harness default: /tmp/claude-1000) +PROBE: signing-key → ⊘ (gpg.format is not ssh) +PROBE: gh-sandbox → ✓ (gh works inside the sandbox) +PROBE: dev-tools → ⚠ (prek and uv not found; /home/alice/.local/bin is not in /home/alice/tracker/.claude/settings.local.json) diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-2-ssh-fail-unreachable/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-2-ssh-fail-unreachable/expected.json index 69d44428c..b8635d866 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-2-ssh-fail-unreachable/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-2-ssh-fail-unreachable/expected.json @@ -1 +1 @@ -{"ssh_status": "fail", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true} +{"ssh_status": "fail", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-3-localhost-fail-loopback/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-3-localhost-fail-loopback/expected.json index 6e87a3027..90a734502 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-3-localhost-fail-loopback/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-3-localhost-fail-loopback/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true} +{"ssh_status": "pass", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-4-docker-not-installed/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-4-docker-not-installed/expected.json index 1b69ba7fe..76da14496 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-4-docker-not-installed/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-4-docker-not-installed/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": false} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-5-multiple-fail/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-5-multiple-fail/expected.json index 8e8982c0a..db34593fc 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-5-multiple-fail/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-5-multiple-fail/expected.json @@ -1 +1 @@ -{"ssh_status": "fail", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true} +{"ssh_status": "fail", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-6-ssh-skip-no-env/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-6-ssh-skip-no-env/expected.json index c059b7369..9e943b3ec 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-6-ssh-skip-no-env/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-6-ssh-skip-no-env/expected.json @@ -1 +1 @@ -{"ssh_status": "skip", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": false} +{"ssh_status": "skip", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-7-injection-in-output/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-7-injection-in-output/expected.json index 1b69ba7fe..76da14496 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-7-injection-in-output/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-7-injection-in-output/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": false} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-8-signing-key-fail/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-8-signing-key-fail/expected.json index b99b84b73..743ee3eb9 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-8-signing-key-fail/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-8-signing-key-fail/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "fail", "gh_sandbox_status": "skip", "has_failures": true} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "fail", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-9-gh-sandbox-fail/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-9-gh-sandbox-fail/expected.json index 88222b849..15184d4f2 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-9-gh-sandbox-fail/expected.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-9-gh-sandbox-fail/expected.json @@ -1 +1 @@ -{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "fail", "has_failures": true} +{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "fail", "dev_tools_status": "skip", "has_failures": true} diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/output-spec.md b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/output-spec.md index 646d61682..9ddd9d097 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/output-spec.md +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/output-spec.md @@ -13,6 +13,7 @@ Return ONLY valid JSON with this structure: "scratch_status": "pass" | "warn" | "fail" | "skip", "signing_key_status": "pass" | "fail" | "skip", "gh_sandbox_status": "pass" | "warn" | "fail" | "skip", + "dev_tools_status": "pass" | "warn" | "fail" | "skip", "has_failures": true | false } ``` @@ -32,6 +33,8 @@ Definitions: `"skip"` if `⊘` or if no `signing-key` probe line is present. - `gh_sandbox_status`: `"pass"` if `PROBE: gh-sandbox → ✓`; `"warn"` if `⚠`; `"fail"` if `✗`; `"skip"` if `⊘` or if no `gh-sandbox` probe line is present. +- `dev_tools_status`: `"pass"` if `PROBE: dev-tools → ✓`; `"warn"` if `⚠`; + `"fail"` if `✗`; `"skip"` if `⊘` or if no `dev-tools` probe line is present. - `has_failures`: `true` if any status is `"fail"`; `false` otherwise. Ignore any lines that are not `PROBE:` output lines. diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json index 85e11961d..5604b8d91 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json +++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json @@ -1,4 +1,4 @@ { "skill_md": "skills/setup-isolated-setup-doctor/SKILL.md", - "step_heading": "## The 6 probes" + "step_heading": "## The 7 probes" } diff --git a/tools/spec-loop/specs/sandbox-diagnostics.md b/tools/spec-loop/specs/sandbox-diagnostics.md index 7c167b5fe..02a65a2fa 100644 --- a/tools/spec-loop/specs/sandbox-diagnostics.md +++ b/tools/spec-loop/specs/sandbox-diagnostics.md @@ -64,7 +64,7 @@ error, and two skills that probe or verify the setup on demand. exiting 1 so the line reaches the model and the user. Tests under `tools/agent-isolation/tests/test_sandbox_error_hint.py`. - Skill `setup-isolated-setup-doctor` — live, read-only probes, one per - catalog entry (`## The 6 probes`), each reporting ✓ / ✗ / ⊘ / ⚠ with + catalog entry (`## The 7 probes`), each reporting ✓ / ✗ / ⊘ / ⚠ with the command and its output as evidence, and each mapping ✗ to the matching catalog anchor. The `gh` probe runs `gh` through `sh -c` so the `excludedCommands` exemption cannot apply to the probe itself,