fix(evals): remove synthetic system prompt hints, install skills natively per harness (AI-1034)

[claude]

Requested via Slack thread

Closes AI-1034 (linear) and #164.

An eval is supposed to measure what an agent does out of the box. The harness was undercutting that in two ways: it prepended a system prompt of its own invention to every agent, and it installed skills where only one of the three CLI harnesses could find them.

Before

Every agent — Claude Code, Codex, OpenCode, and the in-process ai-sdk one — received a harness-authored system prompt:

• Task framing describing a bash tool and files_* tools. Those tools only exist for ai-sdk; the CLI harnesses never see them and work the workspace with their own built-ins, so the text named tools they do not have.
• "End your turn with a short summary…", i.e. coaching on stopping behaviour — one of the things the evals score.
• An ## Available skills block listing each skill and instructing the agent to files_read .claude/skills/<name>/SKILL.md. Again an ai-sdk-only tool, and for Codex a path it cannot see at all.

Codex and OpenCode have no system-prompt flag, so all of that was glued onto the user prompt — the task the model reads.

Skills were installed with a flagless skills add. With no agent CLI installed yet, that CLI falls back to every one of the 71 agents it knows: ~53 stray roots in the workspace (.adal, .factory, .windsurf, … plus non-dotted data/ and skills/). The workspace is exported into run artifacts and is what gets scored. Only .claude/skills was ever verified, and only .claude/skills was advertised — but Codex does not read .claude/skills, so Codex runs effectively had no skills, whatever the injected listing claimed.

After

• Claude Code, Codex, OpenCode receive no harness system prompt at all. Nothing is staged, --append-system-prompt-file is omitted, and nothing is prepended to the user prompt. The user prompt is exactly the eval's prompt.
• ai-sdk is unchanged in substance — it is the one harness with no system prompt of its own, so it still gets task framing, the tool-surface description, and either the skills listing (local-stack, files_read) or the load_skill tool (tools mode). The two turn-ending sentences are gone for it too.
• Skills land where each harness natively looks for them: .claude/skills/ for Claude Code, .agents/skills/ for Codex and OpenCode. Each CLI then discovers, advertises and loads them through its own mechanism — Codex injects its own <skills_instructions> block, OpenCode exposes its skill tool. Skills are genuinely available to Codex for the first time.
• The workspace is clean: two skill directories instead of ~53 roots.
• The assembled system prompt is recorded in results/<experiment>/<eval>.json, so what an agent was told is now verifiable from the artifacts.

How

packages/core/src/agents/engine.ts treats the system prompt as optional: empty means no staged file and systemPromptPath: undefined. The three runners drop their respective mechanisms for passing it. Prompt assembly moves out of run-eval.ts (an entry script that calls main() at import, so untestable) into apps/framework/harness/system-prompt.ts, keyed on exp.agent.id; every block is ai-sdk-only, so a CLI harness assembles to ''. buildSkillsPrompt gains the same agent guard buildToolSurfaceAddendum already has.

packages/sandbox/src/skills.ts names the harnesses explicitly:

skills add /tmp/skills-src --agent claude-code codex opencode --skill '*' --copy --yes

Two details are load-bearing and covered by tests. --agent is variadic, so the source directory must come before it — skills add --agent codex <dir> swallows <dir> as an agent name and dies with Missing required argument: source — and --skill terminates the agent list. --copy stays because symlink mode skips any agent whose top-level directory does not already exist.

All three agents are installed unconditionally rather than threading exp.agent.id down to the installer. The three ids collapse to two directories, an unused one costs a few kilobytes of copied files, and this keeps a single code path. Naming them explicitly also removes a latent time bomb: the flagless fallback is install-order dependent, and had an agent CLI been installed before skills, it would have silently stopped producing .claude/skills.

The post-install check now verifies every agent scope, so a skill missing from one fails loudly rather than leaving that harness quietly skill-less.

Skill-usage scoring is unaffected — the extractor regex matches any <anything>skills/<name>/SKILL.md path, so it handles .agents/skills/… as readily as .claude/skills/….

Verification

Automated: pnpm format:check clean; @supabase-evals/framework 6/6, @supabase-evals/core 111/111, @supabase-evals/sandbox 49/49; pnpm typecheck clean. New coverage asserts that a CLI harness assembles to an empty prompt even with skills installed, that ai-sdk still gets its listing, that the install command carries all three agents in the correct argument order, and that a skill missing from any agent scope throws.

Two pre-existing failures on main are untouched and unrelated: the framework smoke test (needs OPENAI_API_KEY) and apps/web's eval-results.test.ts > describes each experiment the same way on every one of its runs (a data problem in the on-hold results JSON).

Not verified here: no Docker daemon, so packages/sandbox/test/docker.test.ts — including the updated multi-scope install assertions — did not run, and no real eval was executed. The CLI behaviour the change relies on was established by executing the pinned binaries directly (skills@1.5.11, @openai/codex@0.138.0, opencode-ai@1.18.5): codex debug prompt-input finds .agents/skills and not .claude/skills, and opencode serve + GET /skill finds both.

Prompt-level verification is now possible for the first time. The assembled systemPrompt is recorded in the run artifacts — it previously was not, for any CLI harness — so a single narrow dispatch is enough to confirm the prompt an agent actually received:

gh workflow run eval-refresh.yml \
  --ref fix/remove-eval-prompt-hints \
  -f eval=<eval-id> \
  -f experiments=<experiment-name> \
  -f suite=benchmark \
  -f experiment_suite=benchmark \
  -f runs=1 \
  -f timeout_sec=720 \
  -f merge=true \
  -f commit_to_branch=false

A single eval × experiment pair takes roughly 4–6 minutes. Then read systemPrompt out of results/<experiment>/<eval>.json — it should be "" for claude-code, codex and opencode — and confirm the skills the agent loaded came from its own harness listing.

Expected impact on numbers

This changes what every CLI-harness agent is told, so benchmark numbers are expected to move — in both directions. Codex in particular gains working skills where it previously had none. Published numbers are not comparable across this change until a full results refresh lands.

Results JSON is deliberately untouched in this PR: a refresh is already in flight, and this branch must not conflict with it. Refresh after merge, then compare.

Open item

The team AI Policy doc could not be reviewed: the Notion page is not shared with the "Claude Tag General Agent Access" integration.

---

Generated by Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
evals	Ignored	Preview	Aug 7, 2026 4:46pm