J: integrated gauntlet — native composition to a CI-green PR, and where it stops

[devin-ai-integration]

Summary

Workstream J composes A/C/E/F/K's winning configurations into one temporary agent version and runs a
broad, ambiguous "the revenue report can't be trusted" ticket end to end. It reached a CI-green PR
(#14, base swarm/j-gauntlet-base) with two human interventions — one intended (ask_human for an
undecidable rounding policy), one not (budget_reached fired one tool call before the push and needed
sessions.update(budget=…) plus a nudge). Coordinator roster + /mnt/memory + Skill + custom tool +
Linear/GitHub MCP + self-hosted EnvironmentWorker composed with no conflicts on the first attempt.

Findings: context/findings/J-integrated-gauntlet.md. Two results worth reading even if you skip the rest:

• A subagent poisoned the shared Memory Store. The read-only explorer, seeing the parent's edits
  appear underneath it, concluded the parent had fabricated the rounding policy (it cannot see the
  parent's ask_human exchange) and wrote that accusation into the shared store. The parent adjudicated,
  rejected it, kept the child's accurate findings, and corrected the entry before finishing. Native
  parent-side adjudication is the only defence.
• Cost, not capability, is the ceiling. $716 list / 6.4 M cache-read tokens for one ~15-minute
  ticket (parent $413, explorer $255, reviewer $47 — per-thread accounting is class A and genuinely
  ahead of parity). The Anthropic balance was then exhausted mid-review (session.error
billing_error), so only the full arm ran: no-memory, no-subagents, chaos, compaction survival
  and mid-run steering are recorded as untested, not classed. §7 forbids buying credits, so this is
  a master decision.

New class: budget_reached is recoverable — raising the budget plus one user.message continues the
same session with history, roster and sandbox workspace intact (git HEAD and /workspace/skills/
verified live in sb-aMJS9gYX270pQQEsi22IVa). Confirmed unchanged: dead-worker re-dispatch is class D —
the signed webhook replay against an idle session returned {"spawned":[]} and produced zero events.

Shared-file change, called out per §7: runtime/pyproject.toml gains norecursedirs = ["experiments"]
(3 lines incl. comment). The J fixture is deliberately red until an agent fixes it, and without this
pytest -c runtime/pyproject.toml collects it from the repo root and the repo suite fails. The fixture's
own CI check still runs it explicitly. No other shared file touched; packages/provision untouched — the
gauntlet reads CLEVIN_SYSTEM_PROMPT out of agent-definition.ts so the experiment cannot drift from the
shipped definition.

Reproduction

uv run --project runtime python experiments/J/gauntlet.py --arm full --budget 900
uv run --project runtime python experiments/J/gauntlet.py --arm chaos --chaos-after 600
uv run --project runtime python experiments/J/gauntlet.py --resume-file gauntlet-full-<id>.json --budget 900

Every run writes a self-contained evidence JSON under experiments/J/evidence/; the three from this run
are committed. Cleanup ledger is in the findings file — all temporary agents archived, Skill deleted (after
fixing a driver bug: a Skill cannot be deleted while versions exist), Memory Store contents captured then
archived, Modal sandbox terminated, HUM-16 cancelled. Nothing production-side was mutated.

Link to Devin session: https://app.devin.ai/sessions/b1ea5c7293a9456794a56c86335376dc
Open in Devin Desktop: https://app.devin.ai/desktop/session/b1ea5c7293a9456794a56c86335376dc?variant=devin
Requested by: @hrabbani

---

Devin Review

Status	Commit
⚪ Not started	—

Run Devin Review

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring