Repository navigation
Conversation
…part 1) The review agent (design, 'Review rounds' step 1) answers one round as JSON: lane D's Claude review command now carries an answer schema, as planning's does, with planning's model budget. prompts/review.md and core/review-prompt.ts build the round's request with untrusted text in escaped data blocks, and validate the answer, splitting a problem that names several plan items into one per item. scripts/record-review.ts records real Claude output, which test/review-recorded.test.ts replays; a Docker test proves the command and schema reach the vendor. The runner does not run review rounds yet (part 2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd 1 findings (#201 part 1) - Refuse answer text with NUL or unpaired surrogates, and answers whose problems would not fit the next round's prompt; keep that room in every round's prompt. - Give the next round only the previous round's problems. - Keep a problem's check text only on items that have that check entry. - Name the review in schema errors; share planning's root removal and output check. - Add the Docker test to the scheduling test; sweep with the real docker and a limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ound 2 findings (#201 part 1) - Reserve room on the next round's largest prompt: the longest round number and every declared path listed as changed; the last round keeps none. - Refuse blank summaries and details; let a problem name up to 30 items, as a plan has. - Test the byte arithmetic to the boundary, and the review command's lack of a shell. - Bring the T9 rows, architecture and adapter notes up to date for the review schema. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dings (#201 part 1) - The review agent reads the declared files; without changed files the next round's prompt is this one's but for its round number, so the reservation is exact. - Round 1 takes no earlier problems; a clearer attempt ID refusal; no dead check. - The review's host roots get their own prefix; comments and T9 rows say what is tested. - The reservation test runs with ASCII text too, so a 2-byte error fails it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#201 part 1) Review round 4 found that the 12 KiB answer budget refused schema-valid answers (AGENTS.md: valid bounded input must not fail only because a budget is smaller). The earlier problems are a hint, so the next round's prompt now lists as many as fit, in order, and says how many it left out. The prompt may use the review's whole 64 KiB model budget less its schema. validateReviewAnswer checks an answer without rebuilding its prompt. The CI guide covers recording the review fixtures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d fix review round 5 findings (#201 part 1) - When the prompt would not fit, it leaves out earlier problems, the issue's comments, its description, the lessons, then the plan's change texts, each with a note. Only a plan too large without its change texts is refused; the plan block may use the whole budget. - The earlier-problems fit lists every one when all fit, and drops the section when none does. - The same problem twice for one item is one problem. - A Claude review captured with approved argv is refused: it has no tool to run them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…6 findings (#201 part 1) - The plan's items, intents, files and checks always go in; then, while they fit, the change texts, earlier problems, the issue's description, lessons and comments. A note names what was left out, including when no earlier problem fits. - Only a size refusal counts as not fitting; other refusals surface. - At most 10 problems of at most 10 items each per round. - The known limit (a plan too large without change texts) is documented. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… findings (#201 part 1) - The smallest prompt is the plan and the prompt's own text; codeboost's notes go in next, so round 2 never refuses a plan round 1 accepted. - Text a data block cannot hold is refused wherever it is. - decodeClaudeAnswer reads planning and review answers from structured_output, tested. - Comment and doc wording follow the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Notes are kept one by one when not all fit, and tried again once parts are in, so a part is left out unannounced only when the plan leaves no room for its note. - The template's earlier-problems section no longer leaves a blank-line gap. - The command-check dispatch test uses the runner vendor, as production does; the execute launcher's schema is tested as refused for a review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ings (#201 part 1) - The smallest prompt keeps room for every note in its longest form and for the longest round number, so notes always go in, a note never pushes out the part it names, and every later round of a plan fits whenever round 1 did. - Docs say which duties are part 2's, and that the round limit is 3 by default. - Clearer turn-limit advice and test names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gs (#201 part 1) - A test stands in for setup and the supervisor to check the decoder the Claude start call uses, per phase. - Tests for the earlier-problems-before-description order and NUL in check text. - The comment-count reserve is the real count; the docs name the issue-change exception. - Clearer refusal text; the policy test imports the execute schema; the Docker test's Git keeps the developer's configuration out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dings (#201 part 1) - The smallest prompt already holds every note of a part it leaves out; only the round number and the earlier-problems note change between rounds, so only they are reserved. - A test pins that only a size refusal counts as not fitting. - The start call's decoding is tested for fix too; doc wording follows the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reserve room for the issue description and lessons notes, give the zero-listed earlier-problems note its heading, and pad tests with schema-valid plans. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d 13 (#201 part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ew round 14 (#201 part 1) Round 1 keeps room for what can change between rounds; a later round no longer refuses for that room, so a comment added between rounds does not blame the plan. Test both the room round 1 keeps and the later round's own check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Validation can drift from the captured round, and missing Claude recordings leave required replay and CI gates failing.
2 open issues
What changed in this PR
Adds the review-agent foundation for #201: bounded prompts, validated answers, and read-only Claude execution. The runner’s review loop, persistence, merge blocking, and UI remain in part 2.
Changes:
- Builds escaped, size-bounded review prompts and validates problems against plan items.
- Extends Claude’s structured-answer handling to the review phase.
- Adds recording tools, regression coverage, Docker scheduling, and implementation documentation.
| File | Description |
|---|---|
vitest.config.ts |
Schedules review Docker tests serially. |
test/vitest-config.test.ts |
Checks review Docker-test scheduling. |
test/review-recorded.test.ts |
Requires and validates real Claude recordings. |
test/review-prompt.test.ts |
Tests prompt budgets, validation, and review helpers. |
test/fixtures/review/recording-inputs.ts |
Defines recording inputs and provenance hashes. |
test/agent-supervisor.test.ts |
Uses questions for schema-free cancellation testing. |
test/agent-review.test.ts |
Adds a real-Docker review invocation probe. |
test/agent-policy.test.ts |
Tests review schemas, limits, and command restrictions. |
test/agent-launch.test.ts |
Uses questions for schema-free launch testing. |
test/agent-adapter.test.ts |
Tests review schema loading and answer decoding. |
test/agent-adapter-decode.test.ts |
Verifies decoding through the adapter’s start path. |
test/agent-adapter-cleanup.test.ts |
Keeps cleanup fixtures schema-free. |
scripts/record-review.ts |
Captures genuine Claude review answers. |
runner/review-agent.ts |
Adds read-only review execution helpers. |
runner/question-container.ts |
Allows review in shared read-only infrastructure. |
runner/production.ts |
Exports the execute schema for compatibility tests. |
runner/planning-provider.ts |
Shares cleanup and structured-output helpers. |
prompts/review.md |
Defines plan-based review instructions. |
prompts/review-answer.schema.json |
Defines bounded review-answer JSON. |
docs/implementation/review-rounds.md |
Documents part 1 and part 2 integration decisions. |
docs/implementation/ci.md |
Documents recording and replay procedures. |
docs/implementation/agent-isolation.md |
Updates review isolation and schema guarantees. |
docs/architecture.md |
Records the review-agent foundation and remaining scope. |
core/review-prompt.ts |
Builds prompts and validates per-item problems. |
core/planning-author.ts |
Supports configurable data-block limits and size errors. |
agents/policy.ts |
Adds structured review commands and restrictions. |
agents/adapters/claude.ts |
Loads review schemas and decodes structured answers. |
.github/workflows/ci.yml |
Excludes review Docker tests from parallel execution. |
.github/workflows/agent-isolation.yml |
Updates cleanup-time accounting comments. |
Review effort: Balanced
Comment on lines
+15
to
+17
| it('has a Claude recording for each round', () => { | ||
| // Fails until scripts/record-review.ts has been run with credentials. Synthetic fixtures do not satisfy it. | ||
| expect(names).toEqual([...REVIEW_RECORDING_NAMES]); |
Contributor
Author
There was a problem hiding this comment.
Agreed, and the assertion stays on. The recordings need a real Claude token, so the repository owner must run scripts/record-review.ts. Once both files are committed, the gates will be rerun on that head. Until then this PR is not ready to merge. I am leaving this thread open until the recordings land.
🤖 Addressed by Claude Code
…view (#201 part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ound 1, and pad tests with schema-valid change texts, from review round 18 (#201 part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…19 (#201 part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…om review round 20 (#201 part 1) The blocks over 32 KiB that remain are marked as inputs the loader refuses, kept to show prepareReview does not rely on that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…om review round 22 (#201 part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…output, from review round 23 (#201 part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… part 1) The review prompt now carries the plan's base commit and each changed file's git diff, fitted like the other parts (as many as fit, in path order, with a fixed note for the rest), and asks the agent to report changes no item's intent covers. runner/review-diffs.ts makes the diffs with hardened Git. The recording repository commits the code before and after the plan, and plants an unrelated edit in a declared file. The recording script deletes its temporary file when a write fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndary (#201 part 1) The runner prepares a review inside the server, so the per-file Git calls must not block it: they now run as child processes with a signal, keep at most 32 KiB of stdout and drain stderr. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pass --text so an in-tree attribute cannot hide a diff, keep only a file's own section when Git's prefix match adds files under it, stop reading diffs once they fill a whole prompt, and keep Git out of submodules as AGENTS.md asks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ted, from review round 26 (#201 part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ble paths, and keep renames into a directory visible, from review round 27 (#201 part 1) Adds an AGENTS.md rule for Git's pathspec prefix match, which caused findings in review rounds 25, 26 and 27. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… name, from review round 28 (#201 part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bytes and build them lazily, from review round 29 (#201 part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…as not shown, from review round 30 (#201 part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…diffs in the architecture, from review round 31 (#201 part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…art 1) AGENTS.md asks that no volume of benign messages can push a failure out of a bounded buffer; a warning split between chunks is still found. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ound 32 (#201 part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, and remove only a temporary file the recording opened, from review round 33 (#201 part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, from review round 34 (#201 part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cribe the planted P2 defect accurately (#201 part 1) The first real recording (Claude Code 2.1.281, on fb3f536) found the planted unrelated edit in round 1 but missed P2's unmet check entry until round 2 was told about it. Review round 35 found the fixture's description of that defect inaccurate. Both change the prompts, so the recordings must be made again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Part 1 of #201: the review agent. Part 2 (the runner's review rounds, Store v20, the merge blocker and the screen) is built on branch
feat/201-review-rounds, stacked on this one. It is not pushed yet: it opens as a stacked PR once its own review loop is clean on this branch's final head. The user chose this two-part split.What this adds
prompts/review.mdandprompts/review-answer.schema.json. The review answer for one round is JSON: at most 10 problems, each naming up to 10 plan items.core/review-prompt.ts.prepareReviewbuilds one round's request within the review's 64 KiB model budget, less the schema.git difffrom the base (as many as fit, in path order), the previous round's problems, the issue's description, the lessons, then the issue's comments. A note names each part left out.validateReviewAnswerchecks the answer:It splits a problem that names several items into one per item, in plan order, and drops repeats. A copy keeps its
checktext only when that text is one of the item's owncheck:entries.STRUCTURED_PHASES= planning and review. A Claude review gets--json-schemaand--max-turns 50, as planning does.decodeClaudeAnswerreads planning and review answers fromstructured_output, and every other phase's answer fromresult.runner/review-diffs.ts.reviewDiffs(repository, base, head, signal)lists every changed file with hardened Git, as child processes that a signal stops and whose output is bounded.core.precomposeunicodeoff, and with literal pathspecs.--textmeans an in-tree attribute the agent wrote cannot hide a diff. Each file's diff matches only that file, so type changes and files replaced by a directory show exactly their own change, and a rename shows as its rename and its edits. If Git skips rename detection (it does when the unpaired deletes times the adds pass its limit of 1,000 squared),reviewDiffsrefuses withReviewDiffsRefusedinstead of hiding a renamed file's edits.runner/review-agent.ts. The read-only review feature: its own root prefix, Codex refused, and turn-limit advice.scripts/record-review.tsrecords real Claude output for rounds 1 and 2 of a fixture task with two planted defects: a new test that does not do what its check asks (it callslabel(3)once, not twice), and an unrelated edit that weakens a check in a declared file, which only the diffs show. The fixture commits the code before and after the plan with a fixed Git identity, so the commits and diffs are the same on every run.test/review-recorded.test.tsreplays the recordings through validation. It refuses stale recordings (by the hash of the prompt, the schema, the files and the argv).test/agent-review.test.ts(real Docker) proves that a review request leaves the container through the vendor proxy: with an invalid token it fails with the vendor's authentication error. The schema itself reaching the command is covered by unit tests (agent-policy,review-prompt), not by this test.docs/implementation/review-rounds.md(new);agent-isolation.md(the T9 rows),ci.mdandarchitecture.md(updated).Acceptance (#201)
test/review-recorded.test.ts. Blocked: the recordings are not committed yet. The user records them with their own token (below). Until then this test fails, so thetestandreal-dockerchecks are red.test/agent-review.test.tsand the existing Docker tests. CI result below.Recording (needs the user)
The script writes
test/fixtures/review/recorded/claude-review-round1.jsonandclaude-review-round2.json. They then need committing. Nothing from Claude's output has been verified yet. The first recording also shows whether Claude Code's--json-schemaaccepts every keyword the schema uses (pattern, the length limits, string-or-null).Known limit and follow-up
The plan schema allows a plan too large to review in one prompt, even without its change texts: 30 items, each with 10 checks of up to 400 characters.
prepareReviewrefuses such a plan, and part 2 then sends the task to needs human after its items have run. Follow-up #209 tracks it: either review such a plan in batches of items, or check before its items run that it can be reviewed.Follow-up #210 moves the planning and review prompts from a command argument to stdin, as AGENTS.md asks; today the budget keeps them under the OS argument limit.
Screen files
This PR touches no screen files (
web/public/*,web/server.ts). Part 2, not yet opened, changes about 25 lines ofweb/public/app.jsand about 28 lines ofweb/server.ts. #200 is changing the same files.Validation (head
7a71286, the last fully validated head; later heads are validated again before the next Copilot request)Each gate is reported separately.
review-recordedfails only its "has a Claude recording for each round" check, as expected.review-recorded(expected: no recordings yet) and a 15 s timeout inpre-merge.test.ts> "keeps waiting while GitHub still reports the pre-push head…". That file is untouched by this PR, and it passes alone (112/112). Recorded as machine load, not a product failure.tsc --noEmit)test/agent-review.test.tson the real Docker daemon7a71286(1/1)test/real-dockerReview record
Base
0913c8fthroughout. Each round was a fresh independent agent that reviewed the full diff. It had not written the change.ccdc72680a053880a05384dced6a4dced6aac392acac392ac1a7c82e1a7c82e70b275c70b275c5c7aae75c7aae7bb5716fbb5716feacb92ceacb92c9f46ebc9f46ebc596d8c3596d8c3bd411f1bd411f11d5b0611d5b061794ddb7794ddb7f7f8167f7f81678be6f128be6f127a712867a712867a71286f10f918f10f9180a9e5000a9e500ea90109and this bodyea90109282555f: every test issue has a title and fits in 32 KiB; the two blocks over 32 KiB that remain are marked as inputs the loader refuses282555fdd3ef41, #209, and this bodydd3ef4101fb52f(docs), #21001fb52fe8078dd: built from Claude Code 2.1.281's recorded envelope; audit rows reclassifiede8078dd4890ba84890ba8.gitattributesthe agent writes could hide its diffs; Git's prefix match put a directory's files into the diff of a file it replaced; diffs were read for every changed file with no bound;--ignore-submodules=nonewent against AGENTS.mdcb5d178(asynchronous, stoppable),a4144f5a4144f579764d079764d06dead1a6dead1adad493adad493aE2BIG); building them scanned every path up frontb932fa7b932fa7core.precomposeunicodemade a decomposed name's pathspec miss it (an empty or another file's diff); round 29's audit row cited the stdin rule as covering an exception to it; a type comment gave an old reason for a null diffa09d05eand this bodya09d05erunner/review-diffs.ts5e44ff25e44ff20cd1c9c: stderr is searched as it arrives, with carry-over between chunks; tested with 20,000 hints and a split warning0cd1c9c0cd1c9c; thereviewDiffscomment said no in-tree diff attributes apply, but a built-in driver attribute still changes hunk headers022798fand this body022798f47bc62dand this body47bc62dfb3f536and this bodyfb3f536f97854bfb3f536(Claude Code 2.1.281, by the user)isLabelled()edit but not P2's unmet check entry; round 2, told about it, found both. The replay test refused round 1, as it should.f97854b: the prompt now asks the agent to go through eachcheckentry and report any the code does not meet as written. Not committed: the recordings answer the old prompt, so they must be made again.Self-review was done before each independent round. Round 17 is the clean independent pass that allows the Copilot request on
7a71286.Review-lesson audit
Every finding is classified. The rules cited are in CLAUDE.md ("Review readiness") unless AGENTS.md is named.
vitest-config.test.tspattern, testedassertPlantest on the paddingvalidateReviewAnswerscripts/record-planning.tshas the same pattern; it is outside this PR.--text, and tested with a* -diffattribute--ignore-submodules=none--ignore-submoduleson the command line"; now=all, tested with a gitlink-Mwith both paths, tested with a large renamed file and a one-line editTextDecoderdrops a leading U+FEFF by default; nowignoreBOM, tested with a BOM-prefixed name next to the plain onegit diffreads pathspecs only from its arguments (it has no--pathspec-from-file), so the agent-chosen names are bounded to 64 KiB of arguments, far under the OS limits, with a null diff past it. Tested.core.precomposeunicode=falseon every call, and an empty diff is null; tested with an NFD name next to its NFC twinreviewDiffsrefuse (ReviewDiffsRefused), tested with a lowered limitchangeandfrom, tested with a large rename and a large deleteEEXISTnow leaves itNew rules: one, in AGENTS.md, from round 27 (Git pathspec prefix match). Every other finding is covered by an existing rule or is specific to this feature.
🤖 Generated with Claude Code