Skip to content

Headless mode for verify-open-pstack: live-gate without a person in the loop #112

Closed
#137
Closed
Headless mode for verify-open-pstack: live-gate without a person in the loop#112
#137

Description

@ericlitman
ericlitman
opened on Oct 2, 2026

Goal

Let verify-open-pstack (from #90, PR #111) produce the AGENTS.md installed-candidate live proof and live-gate for a PR with no person in the loop. The operator does a one-time login setup. After that, no per-PR login or interactive session is needed.

Why

#111 ships a supervised recorder. For every PR, the operator has to:

  • log in to Claude Code and Codex from scratch, because each run creates new isolated homes
  • drive a native session by hand in each harness
  • answer per-feature prompts and type PASS <feature>

The verifier can't run from an agent session or in CI. It keeps the operator as the bottleneck that #90 was meant to remove. This was surfaced by the operator's review of #111 at 20afcc2 on 2026-10-02.

Scope

In:

  • A persistent isolated login home per harness. Its location and setup are documented. The operator logs in once. It is separate from the daily install, and each run reuses only its credentials, never its plugin or configuration state.
  • A headless mode. For each feature the PR touches, the verifier drives the installed candidate non-interactively: claude -p with --plugin-dir for Claude Code, and codex exec against the isolated install for Codex. It asserts on observable effects, such as files written, tool calls in transcripts, receipts, and exit codes, never on the model's own success claim. Each feature document in features/ gets a headless recipe with its assertions.
  • Self-test in headless mode: each harness invokes the project skill and produces a passing child doctor.json.
  • A fallback: the existing supervised path stays available for features that have no headless recipe. Such a feature makes the run need an operator. It never passes silently.
  • Runnable by an agent. The operator session can run verify.sh run --pr N --headless from a non-TTY shell. It publishes the evidence comment and live-gate and marks the PR ready, with the same exact-head and compensation rules as today.

Out:

  • running in CI or in Factory sandboxes, which have no subscriptions
  • changes to .mergify.yml or .github/workflows/ (MASTRA-762)
  • weaker-model fallback or implicit timeouts

Acceptance criteria

  • The tests cover headless recipe selection, assertion failure, a missing recipe that routes to the supervised path and blocks a headless pass, credential reuse without state reuse, and non-TTY execution.
  • Bun tests, strict typecheck, static invariants, and plugin validation pass. The diff touches nothing under plugins/pstack/**.
  • Live: after the one-time login, the operator session runs verify.sh run --pr <this PR> --self-test --headless from a non-TTY shell. It posts exact-head evidence and live-gate=success and marks the PR ready. Mergify then queues and merges it with no hand-posted status.

Source refs

Activity

  1. ericlitman commented on Oct 3, 2026

    @ericlitman
    ericlitman
    on Oct 3, 2026
    OwnerAuthor

    Scope update (2026-10-02): subscription rotation through cswap, and login lifetime

    The operator rotates between several Claude subscriptions as each one drains. The headless verifier has to work with that, not with a single fixed login. Findings from the live runs on PR #111 decide part of the design:

    • Claude Code isolates credentials per config folder. With CLAUDE_CONFIG_DIR set, Claude stores the login in a Keychain item named Claude Code-credentials-<sha256(config dir)[:8]>, next to the daily item. The real HOME and USER must be kept. A fake HOME breaks Keychain lookup and brings up macOS's "Reset To Defaults" dialog. A missing USER stores the login under account "unknown".
    • cswap (claude-swap, installed at 0.18.1, 0.26.0 available) already manages this. cswap run <account> --no-share -- <claude args> launches Claude Code with CLAUDE_CONFIG_DIR set to a persistent per-account profile. It seeds that profile with the stored account's credentials and shares none of the user's settings, skills, commands, or agents. cswap --list --json reports each account's quota and token state, and --strategy best|next-available picks an account by remaining headroom.

    Added to scope:

    • Claude lanes run through cswap run <account> --no-share --, with the candidate flags: --plugin-dir <candidate>/plugins/pstack, --settings <run settings>, --setting-sources project, and -p for headless. The verifier never runs its own Claude login.
    • The account is chosen once per run, before any exercise. By default it is the account with the most remaining quota, from cswap --list --json. An explicit --claude-account NUM|EMAIL override takes precedence. The receipt and evidence comment record the account (email only), and the run never switches accounts mid-run. A mid-run quota failure fails the run with a named reason: no retry on another account, and no weaker model.
    • Doctor checks: cswap is present, the chosen account's token works (claude auth status under that profile), and its quota isn't exhausted. It fails closed with the exact cswap command to fix it.
    • Codex: state how Codex lanes authenticate. That means one persistent CODEX_HOME per Codex subscription, with an auth.json in each, chosen the same way. Also decide whether Codex rotation is needed. Open question for the operator.
    • Keychain hygiene: runs made with the Add exact-head project verification for Claude Code and Codex [#90] #111 per-run config folders left Claude Code-credentials-<hash> items behind, some with account "unknown". Document that, and don't create new per-run items. Only the operator removes old items, never the verifier.

    Login lifetime: a cswap profile keeps a refresh token, so Claude renews access by itself. The operator logs in again only when a provider revokes or expires a refresh token, and then adds the account to cswap again (cswap --add-account). Doctor surfaces that state before a run starts.

  2. ericlitman commented on Oct 3, 2026

    @ericlitman
    ericlitman
    on Oct 3, 2026
    OwnerAuthor

    Confirmed design (2026-10-03): caam shallow profiles plus CLAUDE_CONFIG_DIR

    This supersedes the cswap-based account handling in the 10-02 scope update. Codex rotation is required too, and cswap is Claude-only. Everything below was measured on the operator Mac (pro14) and on studio2.

    Accounts. caam 0.1.22 holds three Claude Max and three Codex Pro accounts on each host: eric@healthspanners.com, eric@litman.org, eric@mobilyze.com.

    • Accounts must be added with caam add <tool> <name>. Codex over SSH uses --device-code.
    • Codex's own /login signs out the previous session on the server, which revokes the saved copy of that login. That produced 401s on two vault copies whose tokens hadn't expired.
    • One Claude profile was saved while still signed in to another account. Its name said eric@healthspanners.com, but caam's identity check reported eric@litman.org.

    How each lane runs:

    • caam shallow-profile create <run-name> --from-vault <tool>/<account> --base <run-root>
    • Claude Code: caam shallow-spawn <run-name> --base <run-root> -- env CLAUDE_CONFIG_DIR=<run-root>/<run-name>/.claude claude -p ...
    • Codex: caam shallow-spawn <run-name> --base <run-root> -- codex exec ... (caam sets CODEX_HOME for the profile).

    CLAUDE_CONFIG_DIR is mandatory. Without it, a shallow Claude profile reports the profile's account in claude auth status, but sends real requests with the daily login's keychain token, because the profile links ~/Library back to the real home. Measured: the same eric@litman.org profile returned "weekly limit", which was the daily account's quota, without the variable, and OK with it.

    Results with this pattern. All six accounts made a real headless call (OK) on pro14 and on studio2. On studio2 the calls ran over SSH with the login keychain locked (security find-generic-password -w exit 36). So file-backed shallow profiles need no keychain access. studio2 is the host for unattended runs.

    Account choice, once per run:

    • Rank Claude accounts by the weekly window, secondary_window.used_percent in caam limits claude --format json, not by the 5-hour primary_window. caam's own limits --rank / --model modes are an alternative if they agree with that. Codex has primary_window only.
    • Before choosing, assert that each profile's resolved identity (caam list --json, identity.email) matches its name. A mismatched profile fails the run.
    • Record the chosen accounts (email only) in the receipt and evidence comment. Never switch accounts mid-run. A quota failure mid-run fails the run with its reason.

    Isolation.

    • Shallow profiles link the user's skills, MCP servers and settings back to the real home. That suits daily rotation, but not verification.
    • Keep the candidate isolation from Add exact-head project verification for Claude Code and Codex [#90] #111: --plugin-dir <candidate>, --settings <run settings>, --setting-sources project. Give each run its own shallow profiles under a run-owned --base, and delete them after the run, keeping the evidence.
    • Confirm in tests that user-scope skills and MCP servers don't reach the candidate session under these flags. If they do, copy only the credential file into a run-owned config folder and use --from-file.

    Tokens across hosts. Each host logs in separately, rather than exporting one host's vault to the other. Two hosts sharing one refresh token can break each other on refresh.

    Doctor checks before every run:

    • caam is present
    • all expected profiles exist
    • each identity matches its profile name
    • each token is usable (caam limits returns no error)
    • at least one account per tool has headroom

    On failure, doctor prints the exact caam add command that fixes it.

  3. ericlitman commented on Oct 3, 2026

    @ericlitman
    ericlitman
    on Oct 3, 2026
    OwnerAuthor

    Requirement from PR #111's Unfret review (2026-10-03): candidate code must not reach operator credentials

    Unfret's review of #111 at faca0ff (run panel:faca0ff6…:y-TffNRbK8z:9vRNbnzwkCY) raised three critical findings. They apply equally to this design, because caam shallow profiles also link back to the real HOME:

    1. Setup exercises touch daily user files.
    2. Stored GitHub credentials (~/.config/gh, git credential helpers) are readable by candidate code.
    3. External provider children fall back to the daily Claude and Codex configuration.

    Acceptance additions:

    • Exposure. Candidate-driven processes see only the chosen account's credential file and the run-owned config folder. They can't read ~/.config/gh, ~/.ssh, git credential helpers, the daily ~/.claude or ~/.codex, or other shallow-profile identities.
      • The verifier's own GitHub publishing stays in the trusted parent process, never in a candidate child.
      • Keychain access stays limited to what Claude needs. The approach that keeps macOS Keychain lookup working (the real HOME and USER with CLAUDE_CONFIG_DIR) must be squared with this, for example a run-owned HOME whose Library points only at the keychain folder. Measure that on the Mac before choosing.
    • Tests. Assert, for each of the three findings, that a candidate child can't read a planted sentinel in each of those locations.
    • Other findings. Review the remaining 11 findings on Add exact-head project verification for Claude Code and Codex [#90] #111 for anything this design inherits: tree checks after the plugin installs its own dependencies, feature mapping for bootstrap changes, and redaction safety.

  4. ericlitman commented on Oct 4, 2026

    @ericlitman
    ericlitman
    on Oct 4, 2026
    OwnerAuthor

    Measured 2026-10-04: credential copies and rotating refresh tokens (corrected)

    Correction. I first said the daily Codex login on the same account invalidated this copy. That's wrong: the daily Codex login on pro14 is eric@litman.org, and caam's litman copy still works after the daily login's 13:02 UTC refresh.

    What's observed. On pro14, caam's vault copy of eric@healthspanners.com's Codex login, saved 2026-10-03 03:06 UTC, worked in a verification run shortly after and returned 401 ("token expired or invalid") at about 02:00 UTC on 10-04. Its access token's own expiry was still days away. The litman and mobilyze copies went through the same runs and still work. On studio2, all three accounts' separate logins work.

    The cause is unknown. The candidates are a refresh inside a per-run copy, which would rotate the shared refresh token, or a sign-in or refresh of that account somewhere else.

    Design implications, valid whatever the cause:

    • Treat any copy of a credential as possibly invalidated by another copy's refresh.
    • Prefer one persistent verifier home per account, so refreshes persist in the only copy the verifier uses.
    • Doctor must detect an invalidated copy (caam limits returns unauthorized) and fail with the exact caam add command.
    • Whether an account that's active in the daily login also has to be excluded is unproven. Decide it once the cause is measured.

    Also measured: with the real HOME and run-owned CLAUDE_CONFIG_DIR and CODEX_HOME, headless claude -p and codex exec children wrote nothing to the daily ~/.claude, ~/.codex, or ~/.claude.json. setup-pstack still hard-codes daily paths until #120.

  5. ericlitman commented on Oct 4, 2026

    @ericlitman
    ericlitman
    on Oct 4, 2026
    OwnerAuthor

    Product constraint (2026-10-03): no dependence on one maintainer's setup

    open-pstack ships to outside users and contributors. The headless verifier must work for any contributor with Claude Code and Codex installed. The default is persistent per-account config dirs that the contributor logs into once with the harnesses' normal login (CLAUDE_CONFIG_DIR=<dir> claude auth login, CODEX_HOME=<dir> codex login) and names to the verifier.

    Account rotation stays generic: the contributor lists several such dirs, and the verifier picks by remaining quota when the harness exposes it. caam is at most an optional adapter that resolves profiles to those dirs. No requirement, doc, or error message names caam, studio2, the Factory, or Linear.

    The earlier caam-specific comments on this issue are measurements and one possible adapter, not the design.

  6. ericlitman commented on Oct 4, 2026

    @ericlitman
    ericlitman
    on Oct 4, 2026
    OwnerAuthor

    Measured 2026-10-04: copied OAuth credentials don't survive

    All three caam vault Claude copies on pro14 (eric@healthspanners.com, eric@litman.org, eric@mobilyze.com), saved about 25 hours earlier, now fail with "OAuth session expired and could not be refreshed" (caam limits claude: unauthorized for all three). Earlier, one Codex copy failed the same way.

    Whatever the exact rotation mechanics, credential files copied out of a harness's own config dir go stale. Their access tokens expire, and their refresh tokens are invalidated by refreshes elsewhere.

    Design consequence for #112: use persistent per-account config dirs that the harness itself owns and refreshes in place. That's CLAUDE_CONFIG_DIR=<dir> / CODEX_HOME=<dir>, logged in once with the normal login. The verifier points candidate runs at a run-owned dir only for state, and it never copies refresh tokens into short-lived copies. A real design question remains: how do you isolate candidate state while sharing a live, refreshing credential? One option is that the verifier runs the harness with the persistent account dir as its config, and gives candidates only a run-owned working directory and plugin dir. That needs to be measured: does Claude Code or Codex write candidate state into the config dir?

  7. ericlitman commented on Oct 4, 2026

    @ericlitman
    ericlitman
    on Oct 4, 2026
    OwnerAuthor

    Scope update (2026-10-04): no human operator, and evidence from the first agent-driven run

    Decision (Eric, 2026-10-04). The gate must not require a human to answer per-feature prompts. An agent runs it and judges it. The supervised fallback in the original scope stays only as a path an agent drives, not a human requirement. A feature with no headless recipe fails the run with a named reason and does not ask a person.

    What main already settled. Since #111 merged (67c08fa), the verifier reuses the operator's existing Claude Code and Codex logins. Codex gets a run-owned CODEX_HOME whose auth.json links to the daily one. This replaces the per-account login-home design discussed above. The credential parts of this issue are settled unless the headless change reopens them.

    Evidence from #135 (2026-10-04). An agent drove verify.sh run --pr 135 --self-test through tmux, and it passed with live-gate=success on d1fdc62. The run needed one feature per harness, and still showed why the terminal path does not hold up under an agent:

    • The agent had to read the screen and send keystrokes for three dialogs the verifier does not describe: Claude's folder-trust dialog, Codex's folder-trust dialog and Codex's hook review.
    • One PASS project-skill was typed twice by racing writers.
    • The script capture surface.raw is escape-code soup. The usable evidence was the Claude session JSONL and the Codex rollout JSONL.
    • Codex's workspace-write sandbox blocked the state/codex/self-test output path, so the doctor wrote under TMPDIR instead. This is the gap already recorded on Add exact-head project verification for Claude Code and Codex [#90] #111.

    Why it matters now. #114 and #115 each select the registry's shared feature, which means 54 skill invocations plus setup, runner, shipped-tools and project-skill in each harness. That is 116 features and about 580 prompts per PR on the current path.

    Intent of the rule. The AGENTS.md live-test rule came from #7 (2eb7fa7), a Codex duplicate-command bug that unit tests missed. "Self-reports do not satisfy" bars a model's claim that a change works. It does not bar an agent from running and judging the test. This issue's existing requirement already covers that: assert on tool calls, receipts and files.

    #136 duplicated this issue and is closed.

  8. ericlitman
    added 2 commits that reference this issue on Oct 4, 2026

    test(verify): let the stub session run before writing its fixture eff…

    241b1ba

    fix(verify): run setup last and let automate-me write its fixture ski…

    5a996d8
  9. mergify
    added a commit that references this issue on Oct 4, 2026

    Merge pull request #137 from ericlitman/fix/issue-112-headless-verifier

    d6c95c4
  10. mergify
    closed this as completedin #137on Oct 4, 2026
  11. ericlitman
    added 3 commits that reference this issue on Oct 4, 2026

    fix(verify): run the runner recipe's launcher in the foreground (#112)

    967a70b

    fix(verify): restore an operator-linked CLAUDE.md as the same link (#112

    e4d5303

    fix(verify): let the setup check write the harness config folder (#112)

    26c1912
  12. andrewfree
    added 3 commits that reference this issue on Oct 4, 2026

    fix(verify): run the live gate headless with machine-checked recipes (e…

    ef6cd9b

    fix(verify): exercise babysit headless against a local fixture (ericl…

    fd1bbe3

    Merge upstream ericlitman#137 for the headless verifier

    4da061d
  13. ericlitman
    added 3 commits that reference this issue on Oct 4, 2026

    fix(verify): keep headless skill sessions in the foreground (#112)

    8e66b76

    fix(verify): match runner and log.sh calls by file name (#112)

    d89fb58

    fix(verify): run the Claude setup case in bypass mode (#112)

    c3f8e45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions