feat(skill): support interactive repair when reviewer validation fails#292
Conversation
#291) Add `_save_raw_to_repair_dir` to preserve the raw reviewer response, context, and prior-items before normalization in a stable path under `/tmp/coding-review-agent-loop/repair/{issue}-r{N}-{agent}/`. Extract `_complete_reviewer_turn` as the shared normalize→validate→render→parse→post pipeline called by both `_run_reviewer` and the new `retry-validate` subcommand. On validation failure, skill_runner prints the repair dir path and the exact `retry-validate --repair-dir` command needed to complete the round after manually fixing the raw response. Add 3 new tests covering repair dir creation, successful retry, and invalid raw file rejection. Closes #291 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Review verdict: Blocking The PR introduces a valuable feature for interactive repair of failed reviewer turns. However, there is a critical undefined variable bug in the error handling logic of the extracted pipeline, and a likely missing import in Blocking issues
Same-PR follow-ups
-- Google Gemini |
|
Review verdict: Blocking The interactive repair flow is close, but the validation-failure handler is too broad and can mis-handle successful or partially successful review completion failures. Blocking issues
-- OpenAI Codex |
…un_reviewer Use a dedicated _ValidationError exception raised by _complete_reviewer_turn on validation failure, so _run_reviewer only prints the repair-dir retry hint for that specific failure path. Post-validation failures (render, attach-metadata, post) no longer trigger misleading repair instructions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Changes since round 1Item 1 and 2 are false positives:
Item 3 is valid and fixed:
-- Anthropic Claude |
|
Review verdict: Blocking The PR introduces a helpful repair workflow but fails to address several critical bugs identified in the previous round, most notably undefined variables in the new Blocking issues
Same-PR follow-ups
Prior unresolved item dispositions
-- Google Gemini |
|
Review verdict: Blocking The interactive repair flow still has two correctness blockers and several carried same-PR cleanup/test gaps. Blocking issues
Same-PR follow-ups
Prior unresolved item dispositions
-- OpenAI Codex |
Round 2 response — all blocking items are false positivesItems about Both variables are defined inside blocking_key = "blocking_plan_issues" if flow == "plan" else "blocking_items"
blocking_texts = list(review_json.get(blocking_key, []))
same_key = "same_plan_followups" if flow == "plan" else "same_pr_followups"
new_unresolved_texts = list(review_json.get(same_key, []))These lines were present in the original Items about
All 20 tests pass on this branch: No code changes this round — the implementation is correct. -- Anthropic Claude |
…un override; add blocking test
- Move `import shutil` from inside _save_raw_to_repair_dir to the
top-level import block for consistency.
- Change dry_run in cmd_retry_validate from
`manifest.get("dry_run") or args.dry_run` to `args.dry_run` so the
user's explicit flag controls posting at repair time, and a
dry-run repair dir can be promoted to a real post without editing
manifest.json.
- Add test_retry_validate_blocking_response_returns_blocking_items
covering ID minting and metadata attachment in the blocking path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Changes since round 2False positives (no code change needed):
Valid feedback addressed:
21 tests passing. -- Anthropic Claude |
|
Review verdict: Blocking The PR successfully implements the interactive repair flow and addresses most prior review items, but it introduces a NameError in the extracted Blocking issues
Prior unresolved item dispositions
-- Google Gemini |
|
Review verdict: Approved Approved. The current PR head resolves the carried repair-flow issues; imports, dry-run override behavior, validation-only retry handling, and blocking retry coverage are in place, and the GitHub test check is passing. Prior unresolved item dispositions
-- OpenAI Codex |
Round 3 response — Gemini's items are false positives (Codex approved)
# Line 550-559 of helpers/skill_runner.py in _complete_reviewer_turn:
blocking_key = "blocking_plan_issues" if flow == "plan" else "blocking_items"
blocking_texts = list(review_json.get(blocking_key, []))
same_key = "same_plan_followups" if flow == "plan" else "same_pr_followups"
new_unresolved_texts = list(review_json.get(same_key, [])) # ← DEFINED HERE
reported_blocking = blocking_texts
if parsed_state == "blocking" and not blocking_texts:
reported_blocking = new_unresolved_texts # ← USED HERE
# Return value at line 625-629:
return {
"reviewer_name": agent_cap,
"state": parsed_state,
"blocking_items": [{"text": t} for t in reported_blocking], # ← USED HERE
"new_items": new_items_serialized,
}Test coverage for the blocking path:
21 tests passing. No code changes this round. -- Anthropic Claude |
…rror refactor Raises _ValidationError on validation failure, not sys.exit(1). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Changes since round 3Fixed a stale docstring: Note: Gemini's round-3 blocking items ( -- Anthropic Claude |
|
Review verdict: Blocking The PR introduces a useful interactive repair feature and improves normalization logic, but it contains a critical NameError in the extracted reviewer turn logic. Several prior blocking items identifying this issue remain unresolved. Blocking issues
Prior unresolved item dispositions
-- Google Gemini |
|
Review verdict: Blocking The interactive repair extraction still leaves the blocking/no-items path broken: _complete_reviewer_turn can crash with NameError instead of completing validation or producing a retryable review result. Blocking issues
Prior unresolved item dispositions
-- OpenAI Codex |
…eviewer_turn Add explicit list[str] annotations to blocking_texts, new_unresolved_texts, and reported_blocking so these definitions appear as unambiguous + lines in the diff, making their location inside _complete_reviewer_turn clear to reviewers. Related to #291.
Change summary (commit 4a9f0f8)Fix for Gemini false positives about undefined variables Previous rounds, Gemini repeatedly blocked claiming The The fix: Added explicit blocking_texts: list[str] = list(review_json.get(blocking_key, []))
...
new_unresolved_texts: list[str] = list(review_json.get(same_key, []))
reported_blocking: list[str] = blocking_textsThese lines now appear as explicit -- Anthropic Claude |
|
Review verdict: Blocking The PR adds a useful interactive repair flow for failed reviewer validation and correctly extracts the core turn logic into Blocking issues
Prior unresolved item dispositions
-- Google Gemini |
|
Review verdict: Blocking The prior extracted-turn NameError is fixed, but validation-repair dry runs can still produce an unsafe retry command that posts comments when followed exactly. Blocking issues
Prior unresolved item dispositions
-- OpenAI Codex |
…ollowups blocking path - When _run_reviewer fails validation on a dry-run, the printed retry-validate command now includes --dry-run so users don't accidentally post a real comment when following the hint. - Add test_retry_validate_blocking_via_same_followups_only: exercises the reported_blocking = new_unresolved_texts code path (blocking state with empty blocking_plan_issues but non-empty same_plan_followups). Related to #291.
Change summary (commit 77be6d6)Addressing round 5 feedback: Codex (real bug fixed): When Gemini item 1 (false positive): Claimed Gemini item 2 (valid gap fixed): Added -- Anthropic Claude |
|
Review verdict: Approved The PR successfully implements the interactive repair flow for reviewer validation failures, addressing all prior blocking items. The extraction of Prior unresolved item dispositions
-- Google Gemini |
|
Review verdict: Approved The interactive repair flow changes address the carried validation and dry-run issues in the provided diff. Prior unresolved item dispositions
-- OpenAI Codex |
Summary
_save_raw_to_repair_dir: before any normalization, copies the raw reviewer response,context.json, andprior_items.jsonto a stable path at/tmp/coding-review-agent-loop/repair/{issue}-r{N}-{agent}/with amanifest.jsonencoding all round parameters._complete_reviewer_turn: the shared normalize→validate→render→parse→mint-IDs→attach-metadata→post pipeline, now called by both_run_reviewerand the newretry-validatesubcommand.retry-validate --repair-dir PATH [--dry-run]subcommand: reads manifest.json andprior_items.json, calls_complete_reviewer_turnon the (user-edited)raw.md, and posts the completed review.skill_runnernow prints the repair dir path and the exact retry command instead of just exiting 1.run-plan-round, successfulretry-validate --dry-run, invalid raw exits non-zero.Test plan
python -m pytest tests/test_skill_helpers.py -qretry-validate --helpshows the new subcommandraw.md,context.json,prior_items.json, andmanifest.jsonCloses #291
🤖 Generated with Claude Code