Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/mode-economics.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ special-token spellings counted as ordinary text.
Coverage: **75 of 75 local `skills/*/SKILL.md` files**.
External `source.md` redirects and harness symlinks are excluded.

Measurement manifest SHA-256: `b0563986d6e3186e1cb059f3680157ba9e45476a1dbecf225a7402744165f361`.
Measurement manifest SHA-256: `2886a19a0423bd16e50da8cafb0459ec5c4dabd3c4116bfad918d9d61994fe3c`.

| Skill file | Measured tokens | Source SHA-256 (first 16 characters) |
|---|---:|---|
Expand Down Expand Up @@ -189,7 +189,7 @@ Measurement manifest SHA-256: `b0563986d6e3186e1cb059f3680157ba9e45476a1dbecf225
| [security-model-verify](../skills/security-model-verify/SKILL.md) | 5,541 | `17170f3e573564b8` |
| [security-tracker-stats-dashboard](../skills/security-tracker-stats-dashboard/SKILL.md) | 3,815 | `8bf5804a8f214d6a` |
| [setup](../skills/setup/SKILL.md) | 4,185 | `751ad44b2bd50e11` |
| [setup-isolated-setup-doctor](../skills/setup-isolated-setup-doctor/SKILL.md) | 4,716 | `22565dd6ddfd1c65` |
| [setup-isolated-setup-doctor](../skills/setup-isolated-setup-doctor/SKILL.md) | 5,143 | `3d35982d6d18e109` |
| [setup-isolated-setup-install](../skills/setup-isolated-setup-install/SKILL.md) | 3,875 | `71c657ea2aaa48c4` |
| [setup-isolated-setup-update](../skills/setup-isolated-setup-update/SKILL.md) | 4,013 | `8f851263948c42a3` |
| [setup-isolated-setup-verify](../skills/setup-isolated-setup-verify/SKILL.md) | 4,691 | `fa9ec75b26714c97` |
Expand Down
6 changes: 4 additions & 2 deletions docs/setup/sandbox-troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -1047,8 +1047,10 @@ project root it already adds:
```

The helper writes that file only from outside the sandbox (it is in
the harness's write-deny set), and the change applies from the next
session. Confirm with `prek --version` in a fresh session.
the harness's write-deny set). The harness re-reads it without a
restart: the next sandboxed command already sees the new paths.
Confirm with `prek --version`, or with the doctor skill's
*dev-tools* probe.

### Notes

Expand Down
3 changes: 2 additions & 1 deletion docs/setup/secure-agent-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -1510,10 +1510,11 @@ automatically. Five classes of failure are recognised today:
| `127.0.0.1 … Permission denied` / `Operation not permitted … bind` / `Errno 49 … assign requested address` / `Connection refused … 127.0.0.1` | [Localhost port-bind blocked](sandbox-troubleshooting.md#test-cannot-bind-to-a-localhost-port) |
| `/tmp/…: Read-only file system` / `mktemp: failed to create` | [Temp files fail under `/tmp`](sandbox-troubleshooting.md#temp-files-fail-with-read-only-file-system-under-tmp) |
| `x509: OSStatus -26276` / `HTTP 401: Requires authentication (https://api.github.com…` | [`gh` ran inside the sandbox](sandbox-troubleshooting.md#gh-fails-with-tls-osstatus--26276-or-http-401-inside-the-sandbox) |
| `command not found: prek` / `uv: command not found` (also `uvx`) | [`prek` or `uv` not found inside the sandbox](sandbox-troubleshooting.md#prek-or-uv-not-found-or-cannot-write-its-cache-inside-the-sandbox) |

The hint also tells the user to run
`/magpie-setup:isolated-setup-doctor` for a structured probe of all
six failure modes, so a single mid-flow failure can lead to a
seven failure modes, so a single mid-flow failure can lead to a
broader sandbox health-check.

### Why install it user-scope, not project-scope
Expand Down
37 changes: 31 additions & 6 deletions plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ mode: Meta
description: >-
Probe the secure-agent setup for restrictions that block legitimate
work — SSH agent reachability, port binding, containers, the scratch
directory, the signing key, `gh` outside the sandbox. Names the
directory, the signing key, `gh` outside the sandbox, `prek` and
`uv` inside it. Names the
troubleshooting entry and settings fix for each. Read-only.
when_to_use: >-
When the user says "doctor my sandbox", "why is the sandbox blocking
Expand All @@ -18,7 +19,7 @@ when_to_use: >-
capability:
- capability:platform
- capability:reassess
surface_hash: sha256:42ee88aec8f2e3ad
surface_hash: sha256:c68cb4cb391524f3
license: Apache-2.0
---

Expand Down Expand Up @@ -64,17 +65,17 @@ If a fail shows a failure mode not catalogued there, propose appending a new ent
The skill **never** edits a settings file, **never** runs a command with `dangerouslyDisableSandbox`, and **never** installs anything.
If a check fails, surface it and point at the catalog entry; do not auto-fix.
- **Run every probe, even on early failure.** Do not stop at the first ✗.
A user may have one of six independent restrictions or all six, and finding them one re-run at a time is annoying.
A user may have one of seven independent restrictions or all seven, and finding them one re-run at a time is annoying.
- **Distinguish ✗ (failing) from ⊘ (not applicable).** ✗ means the probe ran and the sandbox blocked it.
⊘ means the probe was skipped because a prerequisite is absent (e.g. no `docker` / `podman` on `PATH` → docker probe ⊘, not ✗).
- **Surface evidence.** Each report line names the probe command, the exit code, and the relevant stderr snippet.
"Looks blocked" is not useful; "ssh-add -l → rc=2 → `Could not open a connection to your authentication agent`" is.
- **Map each ✗ to a catalog entry.** The fail report links directly to the matching section of [`docs/setup/sandbox-troubleshooting.md`](../../../../docs/setup/sandbox-troubleshooting.md).
Do not paraphrase the remediation; the catalog is the single source of truth.

## The 6 probes
## The 7 probes

The probes cover the six failure modes the catalog documents.
The probes cover the catalog's failure modes that a sandboxed command can detect on its own; the signing entries that need a live touch or a terminal are verified by `setup-isolated-setup-verify` check 10 instead.
New probes are added when new entries land in the catalog, so the two stay in lock-step.

### Probe 1 — SSH agent / Yubikey reachable
Expand Down Expand Up @@ -264,11 +265,35 @@ When the user reports a `gh` failure this probe does not reproduce, ask for the
**On ✗ → remediation:**
[`docs/setup/sandbox-troubleshooting.md` — `gh` fails with TLS `OSStatus -26276` or `HTTP 401` inside the sandbox](../../../../docs/setup/sandbox-troubleshooting.md#gh-fails-with-tls-osstatus--26276-or-http-401-inside-the-sandbox).

### Probe 7 — `prek` and `uv` usable inside the sandbox

Tests whether the dev tools the framework's hooks and Python tools run on are found and can write their cache from a sandboxed Bash call.
They live in `~/.local/bin`, which the harness reads only when the worktree's `.claude/settings.local.json` grants it; the committed project-scope entries are not applied.
Inside the sandbox a read-denied directory looks exactly like an absent one, so the probe reads `settings.local.json` to tell a missing grant from a tool that is not installed.

**Command:**

```bash
bash <skill-dir>/scripts/probe-7-dev-tools.sh
```

**Interpretation:**

| Result | Status | Meaning |
|---|---|---|
| `✓ (found: …; ~/.cache writable)` | Pass | `prek` / `uv` run inside the sandbox and can write their caches. |
| `✗ (found: …; ~/.cache not writable …)` | Fail | The tools start but cannot write their cache or state. |
| `⚠ (… not found; ~/.local/bin is not in … settings.local.json)` | Warn | The grant is missing; the tools are most likely installed but hidden. |
| `⊘ (… not found; ~/.local/bin is granted …)` | Skip | The grant is present and neither tool is installed there; not a sandbox restriction. |

**On ✗ or ⚠ → remediation:**
[`docs/setup/sandbox-troubleshooting.md` — `prek` or `uv` not found, or cannot write its cache, inside the sandbox](../../../../docs/setup/sandbox-troubleshooting.md#prek-or-uv-not-found-or-cannot-write-its-cache-inside-the-sandbox).

## After the report

If every probe is ✓ or ⊘:

> All six probes pass (or are not applicable). The sandbox is
> All seven probes pass (or are not applicable). The sandbox is
> not currently blocking the known failure modes catalogued in
> `docs/setup/sandbox-troubleshooting.md`. If you hit a different
> sandbox-shaped failure, follow the catalog's *Adding a new
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Inside the sandbox a read-denied ~/.local/bin looks exactly like an
# absent one, so "not found" alone cannot tell a sandbox gap from a tool
# that was never installed. The worktree's settings.local.json is
# readable, and it records whether the path was granted.

found=""
missing=""
for tool in prek uv; do
if command -v "$tool" >/dev/null 2>&1 && "$tool" --version >/dev/null 2>&1; then
found="$found $tool"
else
missing="$missing $tool"
fi
done
found="${found# }"
missing="${missing# }"

local_settings="$(git rev-parse --show-toplevel 2>/dev/null)/.claude/settings.local.json"
granted=0
if [ -n "${HOME:-}" ] && [ -f "$local_settings" ] && grep -qF "\"$HOME/.local/bin\"" "$local_settings" 2>/dev/null; then
granted=1
fi

if [ -z "$found" ]; then
if [ "$granted" -eq 1 ]; then
echo "PROBE: dev-tools → ⊘ (prek and uv not found; $HOME/.local/bin is granted, so they are not installed there)"
else
echo "PROBE: dev-tools → ⚠ (prek and uv not found; $HOME/.local/bin is not in $local_settings)"
fi
elif [ -n "${HOME:-}" ] && ! { mkdir -p "$HOME/.cache" 2>/dev/null && touch "$HOME/.cache/.doctor-probe" 2>/dev/null; }; then
echo "PROBE: dev-tools → ✗ (found: $found; $HOME/.cache not writable inside sandbox)"
else
rm -f "$HOME/.cache/.doctor-probe"
if [ -n "$missing" ]; then
echo "PROBE: dev-tools → ✓ (found: $found; not found: $missing; $HOME/.cache writable)"
else
echo "PROBE: dev-tools → ✓ (found: $found; $HOME/.cache writable)"
fi
fi
2 changes: 1 addition & 1 deletion tools/agent-isolation/sandbox-error-hint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,6 @@ yellow="${esc}[1;33m"
reset="${esc}[0m"

printf '%s[sandbox-hint]%s %s\n' "$yellow" "$reset" "$hint" >&2
printf '%s %s Run %s/setup-isolated-setup-doctor%s for a structured probe of all six failure modes.\n' "$yellow" "$reset" "${esc}[1m" "${esc}[0m" >&2
printf '%s %s Run %s/setup-isolated-setup-doctor%s for a structured probe of all seven failure modes.\n' "$yellow" "$reset" "${esc}[1m" "${esc}[0m" >&2

exit 1
2 changes: 1 addition & 1 deletion tools/skill-evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Suites are currently implemented for:
- **list-skills** — 8 cases across 2 steps (step-1-command, step-2-present)
- **setup-isolated-setup-verify** — 15 cases across 3 steps (runtime-routing, step-1-classify, step-2-recommend)
- **setup-isolated-setup-update** — 15 cases across 4 steps (runtime-routing, step-snapshot-drift, step-tool-freshness, step-after-report)
- **setup-isolated-setup-doctor** — 22 cases across 3 steps (runtime-routing, interpret-probes, after-report)
- **setup-isolated-setup-doctor** — 23 cases across 3 steps (runtime-routing, interpret-probes, after-report)
- **contributor-activity-sweep** — 12 cases across 3 steps (step-0-resolve-inputs, step-1-classify-reviews, step-2-render)
- **optimize-skill** — 5 cases across 1 step (step-diagnose)
- **committer-onboarding** — 27 cases across 4 steps (step-0-validate-vote, step-1-icla-comms, step-2-checklist, step-3-completion-summary)
Expand Down
8 changes: 6 additions & 2 deletions tools/skill-evals/evals/setup-isolated-setup-doctor/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@

Behavioral evals for the `setup-isolated-setup-doctor` skill.

## Suites (22 cases total)
## Suites (23 cases total)

| Suite | Step | Cases | What it covers |
|---|---|---|---|
| `runtime-routing` | Runtime routing | 2 | Codex and Gemini route to their native adapters and never require Claude files |
| `interpret-probes` | Probe interpretation (`## The 6 probes`) | 15 | all-pass, ssh-fail, localhost-fail, docker-skipped, multiple-fail, ssh-skipped-no-env, injection-in-probe-output, signing-key-fail, gh-sandbox-fail, container-gateway pass/not-running/socket-denied/no-backend/relative-CONTAINER_HOST, scratch-on-shared-session-root |
| `interpret-probes` | Probe interpretation (`## The 7 probes`) | 16 | all-pass, ssh-fail, localhost-fail, docker-skipped, multiple-fail, ssh-skipped-no-env, injection-in-probe-output, signing-key-fail, gh-sandbox-fail, container-gateway pass/not-running/socket-denied/no-backend/relative-CONTAINER_HOST, scratch-on-shared-session-root, dev-tools-not-granted |
| `after-report` | Report synthesis (`## After the report`) | 5 | all-clear-all-pass, all-clear-with-skips, ssh-fail-with-catalog-link, multiple-fail-two-catalog-links, injection-asks-autofix-rejected |

## Run
Expand Down Expand Up @@ -85,6 +85,10 @@ The fifteen cases span:
because `CONTAINER_HOST` uses a project-relative `unix://./…` value,
which the CLIs do not resolve against the cwd. Expected
`docker_status: "fail"`, `has_failures: true`.
- **case-16-dev-tools-not-granted**: `dev-tools` ⚠ because
`~/.local/bin` is missing from the worktree's `settings.local.json`,
so `prek` and `uv` are hidden; every other probe passes or is
skipped. Expected `dev_tools_status: "warn"`, `has_failures: false`.

### after-report

Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": false}
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false}
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": false}
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false}
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true}
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true}
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true}
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true}
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true}
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true}
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "has_failures": false}
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "dev_tools_status": "skip", "has_failures": false}
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "has_failures": true}
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "fail", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "dev_tools_status": "skip", "has_failures": true}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "pass", "dev_tools_status": "warn", "has_failures": false}
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->

All seven probes ran to completion on a Linux host whose worktree settings.local.json carries only the project root.

PROBE: ssh-agent → ✓ (2 identities listed)
PROBE: localhost-bind → ✓ (bound + loopback GET → HTTP 200, body=b'ok')
PROBE: docker-runtime → ⊘ (docker not on PATH)
PROBE: podman-runtime → ⊘ (podman not on PATH)
PROBE: project-scratch → ✓ (writable; shared session root, which is the harness default: /tmp/claude-1000)
PROBE: signing-key → ⊘ (gpg.format is not ssh)
PROBE: gh-sandbox → ✓ (gh works inside the sandbox)
PROBE: dev-tools → ⚠ (prek and uv not found; /home/alice/.local/bin is not in /home/alice/tracker/.claude/settings.local.json)
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"ssh_status": "fail", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true}
{"ssh_status": "fail", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true}
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"ssh_status": "pass", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true}
{"ssh_status": "pass", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true}
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": false}
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false}
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"ssh_status": "fail", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true}
{"ssh_status": "fail", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true}
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"ssh_status": "skip", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": false}
{"ssh_status": "skip", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false}
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": false}
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": false}
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "fail", "gh_sandbox_status": "skip", "has_failures": true}
{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "fail", "gh_sandbox_status": "skip", "dev_tools_status": "skip", "has_failures": true}
Loading