Skip to content

fix(code): make codex-review run on current codex-cli (ISS-11082) - #207

Merged
peterulsteen merged 6 commits into
mainfrom
fix/iss-11082-codex-review-flag
Oct 1, 2026
Merged

peterulsteen merged 6 commits into
mainfrom
fix/iss-11082-codex-review-flag

Conversation

@peterulsteen

@peterulsteen peterulsteen commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes ISS-11082. code:codex-review failed on every round with a bare CODEX_FAILED:codex exited with code 2. /plan-with-codex uses the same script, so it failed too.

  • run_codex_review.sh passed --full-auto to codex exec and codex exec resume. codex-cli 0.147 removed that flag (Remove legacy --full-auto handling from codex exec openai/codex#36054), so codex now exits 2 with error: unexpected argument '--full-auto' found.
  • The script now passes -c sandbox_mode=read-only -c approval_policy=never, which both subcommands accept. -s read-only does not work here: codex exec resume rejects -s, so every resumed round would drop to a fresh session.
  • --full-auto also set approval to never. codex exec defaults to never, except when the user's config sets approvals_reviewer = "auto_review", in which case the user's approval_policy applies (codex-cli 0.159.3, codex-rs/exec/src/lib.rs:757-790). The explicit -c approval_policy=never keeps the old behavior: with auto_review and on-request configured, exec and resume report approval: never on 0.154.0 and 0.159.3.
  • codex's stderr is no longer sent to /dev/null. CODEX_FAILED now ends with one line saying why codex failed, taken from the first of these that exists:
    1. the last turn.failed message in the --json stream (or, if none, the last error message), because codex writes turn failures to stdout;
    2. the first stderr line that starts with error, which covers clap and config errors;
    3. the last stderr line, skipping codex's Reading ... from stdin... banner.
  • The full stderr goes to the script's stderr, now also on both CODEX_EMPTY paths (exit 0 with no agent message). When a resume fails, the message now names the cause before the script falls back to a fresh session.
  • plugins/code/hooks/plan-review.sh gets the same change: codex exec -c sandbox_mode=read-only -c approval_policy=never, with stderr appended to the hook's debug log instead of /dev/null (review, Mike).
  • debate-loop.sh now prints the CODEX_FAILED reason with printf '%s' instead of echo -e. The reason is now codex's own text, and echo -e read the \c in C:\cache as "stop output", which cut the message short.

Compatibility

No version detection is needed. I probed every locally installed codex-cli build with --help, and the sandbox argument works across the whole range. The -c approval_policy=never override was probed on 0.154.0 and 0.159.3 only; it uses the same generic -c key=value mechanism as the sandbox override.

codex-cli exec --full-auto exec -c sandbox_mode=read-only exec resume -c sandbox_mode=read-only exec resume -s read-only
0.136.0 to 0.146.0 (12 builds) accepted accepted accepted rejected
0.154.0, 0.158.0 rejected accepted accepted rejected

The reviewer's sandbox is now read-only instead of --full-auto's workspace-write. Nothing needs to write: Codex only reads the plan and the code, and the script writes the feedback file from the JSON stream. The callers (debate-loop.sh, plan-with-codex.md) match CODEX_FAILED:* by prefix, so the longer reason does not break them.

Versions

  • code 1.16.3 -> 1.16.4

Testing

  • New file plugins/code/tools/python/test_run_codex_review.py runs the script against a fake codex that rejects flags the way codex-cli 0.147+ does, and runs plan-review.sh against the same fake. All 12 tests pass at dbfe326. Review-round counterfactuals: dropping the approval flag fails the first-round, resume and fallback tests; dropping either CODEX_EMPTY stderr fails its case; restoring --full-auto or 2>/dev/null in the hook fails the hook tests.
  • I checked each test by breaking the code it covers and running only that test. Each one failed:
Mutation Test that failed, and what it saw
restore --full-auto both round tests: CODEX_FAILED:codex exited with code 2: error: unexpected argument '--full-auto' found
use -s read-only first round: config pair missing; resumed round: assert 2 == 1, a silent fresh-session fallback; resume-fallback test: wrong cause
restore 2>/dev/null CLI-rejection, stderr-error, last-line and resume-fallback tests
unanchored error match picks the ... ERROR codex_core::mcp ... tracing line
drop -m1 the token spills onto a second stdout line
drop the error-line preference reports ... try '--help'.
use the first line instead of the last reports WARNING: a notice
drop the banner filter reports Reading additional input from stdin...
skip the JSON stream reports the tracing line
prefer error over turn.failed reports Reconnecting... 1/5
drop the resume reason Codex session resume failed, starting fresh session...
  • Real runs used codex-cli 0.158.0 with gpt-6-astra.
    • origin/main (ed495653) prints CODEX_FAILED:codex exited with code 2.
    • This branch returns a verdict and a session id in round 1, and round 2 resumes that session with no fallback. Both were run at 2e01f2d, before the review round.
    • An unsupported model now reports CODEX_FAILED:codex exited with code 1: {"type":"error","status":400,...,"message":"The 'bogus-model-xyz' model is not supported when using Codex with a ChatGPT account."}}.
    • Running codex exec -c sandbox_mode=read-only directly reports sandbox: read-only, even though the user config sets workspace-write.
  • Gates:
    • uv run ruff check . is clean.
    • uv run pyright reports 0 errors.
    • uv run pytest plugins/ at the final head dbfe326: 2191 passed, 3 skipped. CI at dbfe326 is green.
  • Review: two reviewers read the pinned diff and found no BLOCKING or HIGH issues.
    • Fixed: the removal version, the JSON-stream cause, the three unpinned behaviors, echo -e in the consumer, the empty (), and the fake codex's banner order.
    • Kept as-is: full stderr on failure goes to the diagnostic channel (not trimmed to 50 lines).

Follow-ups (not in this PR)

  • plugins/code/hooks/plan-review.sh still uses gpt-5.3-codex-spark, which codex rejects for ChatGPT-account auth (now visible in the hook's debug log). The hook is not registered in hooks.json, so nothing runs it today.
  • The default --codex-model gpt-5.3-codex in run_codex_review.sh and debate-loop.sh is rejected for ChatGPT-account codex auth (The 'gpt-5.3-codex' model is not supported when using Codex with a ChatGPT account., found in this machine's ~/.closedloop-ai/plan-with-codex/ logs).

🤖 Generated with Claude Code

peterulsteen and others added 3 commits September 29, 2026 08:46
- Replace --full-auto, which codex-cli 0.154+ rejects with exit 2,
  with -c sandbox_mode=read-only in run_codex_review.sh. Both
  `codex exec` and `codex exec resume` accept it; -s does not work
  on resume.
- Capture codex stderr instead of discarding it. CODEX_FAILED now
  carries one stderr line after the exit code, the full stderr goes
  to the script's stderr, and a failed resume names its cause.
- Add test_run_codex_review.py, which runs the script against a fake
  codex that rejects flags the way codex-cli 0.154+ does.
- Bump code plugin to 1.15.1.

Testing: pytest test_run_codex_review.py (4 passed); real
round 1 + resumed round 2 against codex-cli 0.158.0 returned verdicts;
flag matrix probed on codex-cli 0.136.0 through 0.158.0.

Risks: the reviewer now runs read-only instead of workspace-write.
It only reads the plan and code; the script writes the feedback file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ISS-11082)

- Add a codex-review test where codex exits 1 with only its
  'Reading additional input from stdin...' banner on stderr, as it
  does when an API error arrives in the JSON stream.

Testing: pytest test_run_codex_review.py (5 passed)

Risks: None identified

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Read turn failures from codex's JSON stream first; codex writes
  them to stdout under --json, not to stderr.
- Match stderr lines that start with 'error', so an ERROR tracing
  line is not reported as the cause; keep the last-line fallback.
- Omit the empty parentheses when a failed resume has no reason.
- Print the reason in debate-loop.sh with printf '%s', not echo -e.
- Correct the removal version to codex-cli 0.147 (#36054).
- Tests: resume fallback, one-line token, last-line fallback,
  anchored match, JSON-stream turn failure.

Testing: pytest test_run_codex_review.py (8 passed)

Risks: None identified

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@mikeangstadt mikeangstadt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good fix, and the mutation table in the description is the kind of test evidence I wish every PR shipped with. Three non-blocking notes inline; the plan-review.sh one is the only one I'd consider pulling into this PR.

Comment thread plugins/code/skills/codex-review/scripts/run_codex_review.sh
Comment thread plugins/code/skills/codex-review/scripts/run_codex_review.sh Outdated
Comment thread plugins/code/skills/codex-review/scripts/run_codex_review.sh
peterulsteen and others added 3 commits October 1, 2026 17:05
- plugin.json: main moved code to 1.16.3; this branch now bumps it to
  1.16.4 (one PATCH bump over main)
- CHANGELOG.md: keep main's code v1.16.3 to v1.15.1 entries; this
  branch's entry moves from v1.15.1 (now taken by main) to v1.16.4 at
  the top

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- run_codex_review.sh passes -c approval_policy=never. codex exec
  defaults approval to never but drops that default when the user's
  config sets approvals_reviewer = "auto_review" (exec/src/lib.rs
  build_exec_config at rust-v0.159.3).
- run_codex_review.sh writes codex's stderr to its own stderr on both
  CODEX_EMPTY paths, as it already does for CODEX_FAILED.
- hooks/plan-review.sh replaces --full-auto with
  -c sandbox_mode=read-only -c approval_policy=never and appends
  codex's stderr to its debug log instead of /dev/null.
- Tests: approval pair on exec, resume and fallback calls; stderr on
  both CODEX_EMPTY paths; hook argv and hook stderr logging.

Testing: pytest test_run_codex_review.py (12 passed); each new
assertion fails when its code change is reverted. Real codex-cli
0.159.3 and 0.154.0 runs report approval: on-request with an
auto_review config and approval: never with the new flag.

Risks: None identified

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- CHANGELOG.md: keep main's code-review v3.10.3 entry below this
  branch's code v1.16.4 entry (newest-first)
- plugin.json: code is still 1.16.3 on main, so this branch's 1.16.4
  remains its single PATCH bump

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@peterulsteen
peterulsteen merged commit b9a1f25 into main Oct 1, 2026
7 checks passed
@peterulsteen
peterulsteen deleted the fix/iss-11082-codex-review-flag branch October 1, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants