[yummybomb]

Summary

pi cuts any MCP result over 20KB to its start and end, saves the full text to a file, and tells the model to read that file. The bench runs pi with --no-builtin-tools, so agents could not open the file. A WebMCP tool list is often over the limit: four Amazon tools come to about 43KB. That left the tool names and refs in the cut-out middle. Some models recover by listing the tools through Playwright, and others fall back to the page.

This PR adds pi's read tool to allowlist.txt, for both arms. With read, an agent could also open the bench's own files: intents and their rules, run.json, other trials' transcripts, and .env. So agents on Kernel browsers now run in a bubblewrap sandbox. It needs no sudo and no setup step.

What the sandbox contains:

• /usr, read-only.
• The /etc files DNS and TLS need, and no others: /etc can hold credentials, such as /etc/hypeman/env on Hypeship VMs.
• Its own processes only, in a separate process namespace.
• A fresh directory per agent, bound over an empty /tmp. It holds the agent's mcp.json, pi config and TMPDIR, which is where pi saves full outputs.
• The network, which is shared. Your home, the checkout, other trials and the rest of /tmp are not visible.

How it runs:

• The agent gets only KERNEL_API_KEY, the variables of its model's provider (ANTHROPIC_* for anthropic/...; providers whose keys are named otherwise are mapped, such as GEMINI_* for google/...), the proxy variables pi's HTTP client reads, and NODE_EXTRA_CA_CERTS. They are passed through the environment, never on the command line. The values of keys and proxy URLs are redacted from transcripts, and sandboxed transcripts leave out pi's streamed message_update events, since a key can be split across their deltas; message_end holds each whole message.
• If Node or pi is installed outside /usr, such as under ~/.nvm or ~/.local, only Node's binary and pi's package directory are mounted, read-only, and pi runs as node <pi's cli.js>. The rest of the install prefix stays hidden.
• Sandbox directories are created under /tmp, whatever the host's TMPDIR.
• Killing bwrap with SIGTERM or SIGKILL also kills pi, so the runner's caps and shutdown work as before. pi itself is killed rather than sent SIGTERM, so its print-mode SIGTERM cleanup (closing MCP sessions) does not run; the runner deletes the browsers either way. Each agent's directory is removed afterward, and on shutdown after the browsers are deleted.
• Smoke gate: the run stops before any agent if bwrap cannot create a sandbox, which happens when it isn't installed or unprivileged user namespaces are disallowed. It also stops if a sandbox can see the bench directory or any entry in your home that isn't on a mount path, or if node <pi's cli.js> --version fails in a sandbox or differs from pi --version, as when pi is a standalone binary or a version manager's shim.
• Scope: the sandbox applies when the allowlist has a built-in tool, which is the Kernel-browser allowlist. Local runs (allowlist.local.txt) still run pi unsandboxed and without read.

Limits

• Linux only, and it needs bwrap (apt install bubblewrap) with unprivileged user namespaces. Some distributions restrict those with AppArmor.
• The agent can read /usr and the bound /etc files, and its own /proc/self/environ, which holds its own keys. Their values are redacted from the transcript if it does.

Testing

• node --test "bench/**/*.test.mjs": 108 pass.
  • test/sandbox.test.mjs covers:
    • only the allowed variables reach the agent (GH_TOKEN does not), including a provider whose key is not named after it (google/ gets GEMINI_API_KEY);
    • the proxy variables reach the agent;
    • only long values of key, token, secret and proxy URL variables are redacted;
    • pi under a ~/.local-style prefix mounts only its package and Node's binary;
    • nothing from home is bound;
    • the paths the gate checks;
    • a real-bwrap test, skipped where bwrap cannot run. In it, the agent can write its TMPDIR and read its mcp.json, but cannot see a file elsewhere in /tmp or /etc/hostname, and sees at most three processes. It runs a stand-in pi installed under a prefix and cannot see the prefix's other files. A second real-bwrap test gets the stand-in's version from a sandbox and fails on a shell shim in front of it.
  • test/agent.test.mjs: the runner redacts the given strings from the transcript, including a key split across streamed deltas, which it leaves out with their message_update events.
  • test/run.test.mjs: the run stops before any browser if bwrap fails, if a sandbox can see the bench, or if pi is a shim the sandbox cannot run with Node. A stub bwrap runs the existing tests end to end through the sandbox path.
• Manual: the real pi 1.1.0, copied under a fake ~/.local, ran --version in the sandbox with only its package mounted, and the gate passed with ~/.local/share hidden.
• Manual, on a Hypeship VM: pi ran inside the sandbox and reached the network. /home, /etc/hypeman/env, /etc/chromium.d/apikeys and other /tmp files were invisible. SIGTERM and SIGKILL to bwrap both ended the command.
• Live run of anthropic/claude-haiku-5-5 --thinking medium on two Amazon intents (product details and first search result, PR Add Amazon shopping and guarded retail cart tools #56's tools, 2 trials with tools only plus 3 pairs each, 2 slots, stealth):
  • The smoke gate saw read declared, and all 14 trials were correct.
  • In 7 of 8 treatment trials Haiku read pi's saved tool list from its sandbox's TMPDIR, with no read errors, and 7 of 8 invoked a tool. In earlier runs without read, the same intents had 40% and 60% of treatment trials invoking a tool.
  • Treatment was 0.6× and 0.9× control's speed, since Haiku's control finishes these in 3–5 turns.
  • No sandbox directories were left in /tmp.