Repository navigation
Make verify-open-pstack agent-run: headless native sessions, machine-checked assertions, no operator prompts #136
Description
Problem
verify-open-pstack (#90, PR #111) requires a human operator at an interactive terminal. For every selected feature, the operator answers five readline prompts: surface, action, observed, artifact paths and PASS <feature>. The verifier also launches the Claude Code and Codex TUIs through /usr/bin/script. That design fails in two ways.
- Human load. The registry's
sharedrule selects all 54 skills plussetup,runner,shipped-toolsandproject-skillfor any change underplugins/pstack/agents/**orpoteto-mode/references/**. That is 116 features and about 580 typed answers for feat(models): make GPT-6.1 Sol the Sol default and admit ultra (#102) #115 or fix(pstack): preload poteto-mode in Claude poteto-agent (#49) #114. No part of this needs a human. The project wants agents to run the gate. - Brittleness under an agent. On fix(verify): map repository-root attribution files as nonRuntime (#134) #135, an agent drove the verifier through tmux, and the run needed only one feature per harness. Even so, the agent had to read the screen and send keystrokes to get past three dialogs the verifier does not describe: Claude's folder-trust dialog, Codex's folder-trust dialog and Codex's hook review. One
PASSwas typed twice by racing writers. The rawscriptcapture is escape-code soup. The usable evidence came from the session JSONL files.
The AGENTS.md rule this verifier enforces came from #7, a Codex duplicate-command bug that unit tests missed. It requires a live test of the changed behavior on the installed candidate in each affected harness. Its "self-reports do not satisfy" clause bars a model's claim that it worked. It does not bar an agent from running and judging the test.
Outcome
An agent can run verify.sh run --pr N end to end, with no TTY and no human, and get the same exact-SHA guarantees.
Acceptance criteria
- Each harness runs headless on the real installed candidate. Claude uses
claude -p --plugin-dir <exact-head>/plugins/pstackwith structured JSON output. Codex usescodex exec --jsonwith the exact-head plugin installed in the run-ownedCODEX_HOME. No/usr/bin/scriptand no TUI. - Each feature document defines the exercise prompt and a machine-checked assertion over the native session record, such as a
Skilltool call forpstack:<skill>, a runner receipt's model and effort, a restored setup snapshot, or adoctor.jsonwithresult: pass. Surface, action and observed come from the run and the assertion, not from a model's success claim. - No readline prompts remain. The
PASS <feature>step and operator-typed fields are removed. A failed assertion fails the run and publishes nothing. - Kept unchanged: exact head and base pinning, provenance checks, setup snapshot and byte-exact restoration, private mode-0700 evidence, one structured comment plus
live-gateon the exact SHA, and no PR state changes. - The SKILL.md text no longer requires an operator, and says an agent may run and judge the gate.
- The delivery PR passes its own
--self-testrun from its candidate head, driven by an agent with no TTY.
Not in scope
The shared rule stays. Every change to a shared file still selects all consuming skills.
Surfaced while delivering #114 and #115. Related: #90, #134.
Activity
ericlitman commentedon Oct 4, 2026
Duplicate of #112. The evidence and the 2026-10-04 decisions are on #112.