fix(workbench): give the workbench a ref so reviewer clones succeed - #565
Conversation
Prepare builds the workbench with init + fetch-by-SHA + checkout --detach, so
the directory ends up with FETCH_HEAD and no refs/ at all. Git does not treat
such a directory as a repository, so the per-reviewer
git clone --no-hardlinks <workbench> <reviewer-workspace>
fails with "repository does not exist" for every agent that needs a workspace.
The failure is silent and actively misleading. A reviewer that cannot start is
recorded as incomplete_failed and contributes **zero findings**, which the
rollup then renders as a clean review. On a real PR this produced "0 findings"
across four of five reviewers -- only the one agent that reads the diff without
a checkout actually ran -- and the result read as an all-clear.
Writing refs/heads/cr-review-head at the head SHA after checkout makes the
workbench a valid, clonable repository. HEAD stays detached, so nothing else
about the pinned checkout changes.
The regression test asserts the property that matters -- that the workbench can
actually be cloned the way a reviewer workspace is created -- and fails without
the fix.
piekstra-dev
left a comment
There was a problem hiding this comment.
Automated PR Review
Reviewed commit: 992b1f3e4931
Profile: reviewer - Posting as: piekstra-dev
Summary
| Reviewer | Findings |
|---|---|
| go:implementation-tests | 0 |
| harness-engineering:repo-health | 0 |
| security:code-auditor | 0 |
| architecture:solid | 2 |
architecture:solid (2 findings)
Major - internal/workbench/workbench.go:154
U-L1 (behavioral contract):
Preparenow establishes a new postcondition — the workbench is clonable because it hasrefs/heads/cr-review-head— but the reuse fast path does not check it.reusable(workbench.go:189-226) validates metadata,commitPresentfor base/head, andverifyClean; none of those touchrefs/, andmetadataSchemaVersionis unchanged at 2 with schema 1 still accepted (workbench.go:204). A workbench built by a pre-fix binary therefore passes the gate verbatim,Preparereturns early, and everygit clone --no-hardlinksinprepareReviewerWorkspace(workbench.go:416) still fails with "repository does not exist". That is precisely the silent zero-findings outcome this PR exists to fix, and it hits the most likely user: someone resuming the run whose reviewer fleet just died (NewestCompatibleIncompleteRunreuses the existing run's artifact dir, includingworkbench/).Both paths out of
Prepareshould guarantee the same postcondition. Concrete fix: either add the ref check to the reuse gate — e.g. adeps.gitCommand(ctx, repoDir, "rev-parse", "--verify", workbenchHeadRef+"^{commit}")alongside the existingcommitPresentcalls, or fold it intoverifyCleanso build and reuse share one clonability assertion — or bumpmetadataSchemaVersionto 3 and stop accepting 1/2 inreusable, forcing a rebuild of workbenches that predate the ref. The check-based option is cheaper and keeps durable task fingerprints stable, which the schema-1 reuse note in docs/checkout-native-review-contract.md:106-109 explicitly cares about. Whichever you pick, the existingTestPrepareLeavesWorkbenchClonableshould get a sibling that prepares a workbench, deletesrefs/heads/cr-review-head, callsPrepareagain, and asserts the result is clonable.
Minor - internal/workbench/workbench.go:147
U-G1 (new durable surface gets a documented contract): the workbench's ref set is now load-bearing shared state —
workbench/repo/is a durable run artifact whose shape docs/checkout-native-review-contract.md:44-80 and 90-109 specify, and the reviewer-clone consumer depends on it. The ref is described only in this inline comment. Worth an amendment to that doc's workbench-layout notes ('workbench/repo/is a clean pinned checkout at the PR head SHA, carryingrefs/heads/cr-review-headso the per-reviewer clone succeeds'), so the next change toPreparedoes not re-lose it.The same amendment should settle a second question this line decides implicitly: only the head gets a ref, and
git clonetransfers only ref-reachable objects, so the base commit thatensureCommitdeliberately fetches (workbench.go:134) and thatreusableasserts is present (workbench.go:215) does not survive into reviewer workspaces whenever base is not an ancestor of head — the normal case for a branch cut from an older main. Reviewers cannot then rungit log base..HEADorgit diff <base>locally. This is defensible either way: reviewers are handed the provider-generateddiff.patchand nothing in the pipeline reads base from the workspace, so it may simply be out of the workbench's job. If you want the base reachable, the fix is one moreupdate-ref refs/heads/cr-review-base <base SHA>next to this call; if you do not, say so in the doc so the omission reads as a decision rather than an oversight. Evidence that would change this to a real defect: any reviewer prompt or workspace tool that resolves the base SHA insideworkspaceRepo.
Reviewer Coverage
go:implementation-tests— complete (broad); skipped: none; constraints: noneharness-engineering:repo-health— complete (broad); skipped: none; constraints: nonesecurity:code-auditor— complete (broad); inspected 1 assigned file (2 inspected across reviewers):internal/workbench/workbench.go; skipped: none; constraints: nonearchitecture:solid— complete (broad); inspected 1 assigned file (2 inspected across reviewers):internal/workbench/workbench.go; skipped: none; constraints: Only the assigned file internal/workbench/workbench.go was reviewed as a finding surface; workbench_test.go, runartifact, pipeline, and docs/checkout-native-review-contract.md were read for context only. Sandbox deniedgo testandgitinvocations in the reviewer workspace, so the fix and the reuse-path reasoning were verified by code reading rather than by running the suite. The PR's second stated symptom (a fleet-wide reviewer failure rendering as a clean review in the rollup) lives outside the assigned file and was not reviewed.
Inspected files (2)
internal/workbench/workbench.gointernal/workbench/workbench_test.go
0 PR discussion threads considered. 0 summarized; 0 resolved.
Completed in 5m 13s | $3.20 | claude-sonnet-5, claude-opus-5 | cr dev
| Field | Value |
|---|---|
| Model | claude-sonnet-5, claude-opus-5 |
| Reviewers | go:implementation-tests, harness-engineering:repo-health, security:code-auditor, architecture:solid |
| Engine | claude_cli · claude-sonnet-5, claude-opus-5 |
| Reviewed by | cr · piekstra-dev |
| Duration | 5m 13s wall · 7m 54s compute |
| Cost | $3.20 |
| Tokens | 116 in / 31.9k out |
Per-workstream usage
orchestrator-selection— claude-sonnet-5- In: 6
- Out: 2.6k
- Cache read: 66.7k
- Cache create: 17.7k
- Cost: $0.17
- Duration: 30s
go:implementation-tests— claude-sonnet-5- In: 14
- Out: 4.5k
- Cache read: 246.6k
- Cache create: 46.2k
- Cost: $0.42
- Duration: 1m 00s
harness-engineering:repo-health— claude-sonnet-5- In: 26
- Out: 5.7k
- Cache read: 568.1k
- Cache create: 48.4k
- Cost: $0.55
- Duration: 1m 15s
security:code-auditor— claude-sonnet-5- In: 12
- Out: 2.2k
- Cache read: 183.7k
- Cache create: 24.5k
- Cost: $0.24
- Duration: 35s
architecture:solid— claude-opus-5- In: 52
- Out: 16.0k
- Cache read: 1.1M
- Cache create: 64.9k
- Cost: $1.60
- Duration: 3m 48s
orchestrator-rollup— claude-sonnet-5- In: 6
- Out: 965
- Cache read: 101.3k
- Cache create: 31.2k
- Cost: $0.23
- Duration: 43s
…e ref Review raised that Prepare's new postcondition -- the workbench is clonable because it carries a head ref -- was established only on the build path, while the reuse fast path returned early without checking it. Adding the assertion is right regardless, so it is here. But the specific scenario in the finding does not appear reachable: a workbench missing refs/ entirely is already rejected, because commitPresent and verifyClean shell out to git and fail in a directory git does not consider a repository. I tried several ways to construct a reuse that reaches a reviewer clone with a broken workbench and could not; deleting only the ref leaves refs/ in place, which git still accepts and clones fine. So the check stays as belt-and-braces for the narrower case (refs/ present, head ref gone) and to stop a future change to those checks quietly dropping clonability -- and the comment says exactly that rather than claiming to fix a failure I could not reproduce. No test accompanies it: a test that passes with the change reverted verifies nothing, and I would rather ship the guard honestly labelled than a green assertion that discriminates nothing. Also documents the ref in the checkout contract, per the same review. That includes the consequence the reviewer surfaced: only the head is given a ref, so the base commit does not reach reviewer workspaces when base is not an ancestor of head. Recorded as a decision with the one-line fix if it ever needs to change.
|
Both findings addressed in Major (reuse path): the assertion is added — both exits from But I could not reproduce the failure as described, and I want to be honest about that rather than let the commit imply otherwise. A workbench missing So the check ships as belt-and-braces for the narrower case ( If you can construct the reuse-reaches-clone case, I'll happily add the test and drop the hedging in the comment. Minor (documentation): done. Worth noting: this PR was reviewed by a |
piekstra-dev
left a comment
There was a problem hiding this comment.
Automated PR Review
Reviewed commit: d4b668f4d386
Profile: reviewer - Posting as: piekstra-dev
Summary
| Reviewer | Findings |
|---|---|
| go:implementation-tests | 1 |
| harness-engineering:repo-health | 0 |
| security:code-auditor | 0 |
| architecture:solid | 0 |
go:implementation-tests (1 finding)
Major - internal/workbench/workbench_test.go:724
The reuse-path fix (workbench.go:230, refPresent check against workbenchHeadRef) has no test proving it, even though it guards the exact silent-zero-findings scenario this PR exists to fix. TestPrepareReusesValidV1WorkbenchWithoutRewritingMetadata only exercises reuse of a workbench that already has the ref (built by this same fixed code in the same test run), so it can't distinguish 'refPresent correctly gates reuse' from 'refPresent is a no-op'. Nothing in the suite deletes refs/heads/cr-review-head from an otherwise-valid, metadata-matching workbench and asserts Prepare rebuilds it into a clonable state rather than reusing it verbatim -- which is precisely the pre-fix-binary/resumed-run scenario the reuse check was added for. Add a sibling to TestPrepareLeavesWorkbenchClonable that prepares a workbench, runs
update-ref -d refs/heads/cr-review-head(or equivalent) against the resulting repo to simulate a workbench built before this fix, calls Prepare again with an unchanged Request, and asserts (a) Prepare did not skip rebuilding via the reuse fast path and (b) the resulting workbench is clonable via the samegit clone --no-hardlinksassertion already used in TestPrepareLeavesWorkbenchClonable. Without this, a future refactor of reusable() or refPresent() that silently breaks the gate would pass the full suite.
Reviewer Coverage
go:implementation-tests— complete (constrained); inspected 2 assigned files (3 inspected across reviewers):internal/workbench/workbench.go,internal/workbench/workbench_test.go; skipped: none; constraints: noneharness-engineering:repo-health— complete (constrained); skipped: none; constraints: nonesecurity:code-auditor— complete (constrained); inspected 1 assigned file (3 inspected across reviewers):internal/workbench/workbench.go; skipped: none; constraints: nonearchitecture:solid— complete (constrained); inspected 1 assigned file (3 inspected across reviewers):internal/workbench/workbench.go; skipped: none; constraints: Both findings from the prior round are resolved at d4b668f: the reuse gate now asserts the head ref (workbench.go:230), and the doc amendment records the ref plus the deliberate head-only decision (checkout-native-review-contract.md:77-85). Findings are limited to the assigned file internal/workbench/workbench.go; workbench_test.go and docs/checkout-native-review-contract.md changed in this PR and were read for context only. Sandbox deniedgo testandgitinvocations from this workspace, so the reuse-path and clonability reasoning is from code reading, not from running the suite.
Inspected files (3)
docs/checkout-native-review-contract.mdinternal/workbench/workbench.gointernal/workbench/workbench_test.go
0 PR discussion threads considered. 0 summarized; 0 resolved.
Completed in 2m 22s | $3.43 | claude-sonnet-5, claude-opus-5 | cr dev
| Field | Value |
|---|---|
| Model | claude-sonnet-5, claude-opus-5 |
| Reviewers | go:implementation-tests, harness-engineering:repo-health, security:code-auditor, architecture:solid |
| Engine | claude_cli · claude-sonnet-5, claude-opus-5 |
| Reviewed by | cr · piekstra-dev |
| Duration | 2m 22s wall · 5m 05s compute |
| Cost | $3.43 |
| Tokens | 80 in / 20.4k out |
Per-workstream usage
go:implementation-tests— claude-sonnet-5- In: 22
- Out: 6.3k
- Cache read: 849.3k
- Cache create: 75.2k
- Cost: $0.80
- Duration: 1m 23s
harness-engineering:repo-health— claude-sonnet-5- In: 22
- Out: 5.9k
- Cache read: 825.2k
- Cache create: 82.8k
- Cost: $0.83
- Duration: 1m 23s
security:code-auditor— claude-sonnet-5- In: 12
- Out: 1.4k
- Cache read: 224.8k
- Cache create: 39.1k
- Cost: $0.32
- Duration: 23s
architecture:solid— claude-opus-5- In: 18
- Out: 5.2k
- Cache read: 644.4k
- Cache create: 74.2k
- Cost: $1.19
- Duration: 1m 27s
orchestrator-rollup— claude-sonnet-5- In: 6
- Out: 1.7k
- Cache read: 112.9k
- Cache create: 37.3k
- Cost: $0.28
- Duration: 27s
piekstra-dev
left a comment
There was a problem hiding this comment.
Automated PR Review
Reviewed commit: 0f6dfb29f1d3
Profile: reviewer - Posting as: piekstra-dev
Summary
| Reviewer | Findings |
|---|---|
| go:implementation-tests | 0 |
| harness-engineering:repo-health | 0 |
| security:code-auditor | 0 |
| architecture:solid | 0 |
Reviewer Coverage
go:implementation-tests— complete (constrained); inspected 2 assigned files (3 inspected across reviewers):internal/workbench/workbench.go,internal/workbench/workbench_test.go; skipped: none; constraints: noneharness-engineering:repo-health— complete (constrained); skipped: none; constraints: nonesecurity:code-auditor— complete (constrained); inspected 1 assigned file (3 inspected across reviewers):internal/workbench/workbench.go; skipped: none; constraints: nonearchitecture:solid— complete (constrained); inspected 1 assigned file (3 inspected across reviewers):internal/workbench/workbench.go; skipped: none; constraints: All three prior findings are resolved at 0f6dfb2: reuse gate asserts the head ref (workbench.go:230), the doc records the ref and the head-only decision, and TestPrepareRestoresHeadRefOnReuse covers the gate's rejection branch. Findings are scoped to the assigned file internal/workbench/workbench.go; workbench_test.go and docs/checkout-native-review-contract.md also changed and were read for context only. Sandbox deniedgo testandgitexecution from this workspace, so the diff was judged by code reading rather than by running the suite. The comment at workbench.go:225-229 claims a workbench with no refs/ is already rejected by commitPresent/verifyClean. Consistent with the PR's own findings and the author's reproduction attempt, but not independently verified here since git could not be run.
Inspected files (3)
docs/checkout-native-review-contract.mdinternal/workbench/workbench.gointernal/workbench/workbench_test.go
0 PR discussion threads considered. 0 summarized; 0 resolved.
Completed in 1m 39s | $3.42 | claude-sonnet-5, claude-opus-5 | cr dev
| Field | Value |
|---|---|
| Model | claude-sonnet-5, claude-opus-5 |
| Reviewers | go:implementation-tests, harness-engineering:repo-health, security:code-auditor, architecture:solid |
| Engine | claude_cli · claude-sonnet-5, claude-opus-5 |
| Reviewed by | cr · piekstra-dev |
| Duration | 1m 39s wall · 2m 25s compute |
| Cost | $3.42 |
| Tokens | 52 in / 8.6k out |
Per-workstream usage
go:implementation-tests— claude-sonnet-5- In: 12
- Out: 2.0k
- Cache read: 532.4k
- Cache create: 87.5k
- Cost: $0.71
- Duration: 28s
harness-engineering:repo-health— claude-sonnet-5- In: 16
- Out: 2.6k
- Cache read: 688.2k
- Cache create: 96.8k
- Cost: $0.83
- Duration: 38s
security:code-auditor— claude-sonnet-5- In: 6
- Out: 743
- Cache read: 144.9k
- Cache create: 48.1k
- Cost: $0.34
- Duration: 12s
architecture:solid— claude-opus-5- In: 12
- Out: 2.8k
- Cache read: 502.9k
- Cache create: 92.3k
- Cost: $1.24
- Duration: 54s
orchestrator-rollup— claude-sonnet-5- In: 6
- Out: 537
- Cache read: 125.0k
- Cache create: 40.3k
- Cost: $0.29
- Duration: 11s
The previous commit added the reuse-path check without a test and said so, because the test I wrote passed with the change reverted. Review pointed out the reason, and it was a flaw in my assertion rather than in the scenario: I was asserting the reused workbench is clonable, which does not discriminate, since deleting only the ref leaves refs/ in place and git still clones from it happily. The discriminating assertion is that the ref comes back. Without the gate, Prepare takes the reuse path and the ref stays deleted; with it, the workbench is rebuilt and the ref resolves to the head SHA again. Confirmed red with the gate removed and green with it. Comment updated to describe what the check actually catches -- refs/ present but the head ref absent, which every other reuse check accepts -- rather than hedging about a scenario I had not managed to construct.
b490d78 to
4cfc86c
Compare
piekstra-dev
left a comment
There was a problem hiding this comment.
Automated PR Review
Reviewed commit: 4cfc86cd4b9a
Profile: reviewer - Posting as: piekstra-dev
Summary
| Reviewer | Findings |
|---|---|
| go:implementation-tests | 0 |
| harness-engineering:repo-health | 0 |
| security:code-auditor | 0 |
| architecture:solid | 0 |
Reviewer Coverage
go:implementation-tests— complete (constrained); inspected 2 assigned files (3 inspected across reviewers):internal/workbench/workbench.go,internal/workbench/workbench_test.go; skipped: none; constraints: noneharness-engineering:repo-health— complete (constrained); skipped: none; constraints: nonesecurity:code-auditor— complete (constrained); inspected 1 assigned file (3 inspected across reviewers):internal/workbench/workbench.go; skipped: none; constraints: nonearchitecture:solid— complete (constrained); inspected 1 assigned file (3 inspected across reviewers):internal/workbench/workbench.go; skipped: none; constraints: All three prior findings remain resolved: build path sets the ref (workbench.go:154), the reuse gate asserts it (workbench.go:230), TestPrepareRestoresHeadRefOnReuse covers the rejection branch, and the contract doc records the head-only ref decision. Findings are scoped to the assigned file internal/workbench/workbench.go; workbench_test.go and docs/checkout-native-review-contract.md also changed and were read for context only. Sandbox deniedgo testandgitexecution from this workspace, so the diff was judged by code reading rather than by running the suite. The comment at workbench.go:225-229 claims a workbench with no refs/ is already rejected by commitPresent/verifyClean. Plausible and author-verified, but not independently confirmed here since git could not be run. internal/workbench/workbench.go is unchanged at 4cfc86c relative to the previously reviewed 0f6dfb2 (still +31 -2); only the test file grew, refactoring the clone assertion into cloneWorkbench/gitCommandSucceeds helpers.
Inspected files (3)
docs/checkout-native-review-contract.mdinternal/workbench/workbench.gointernal/workbench/workbench_test.go
0 PR discussion threads considered. 0 summarized; 0 resolved.
Completed in 1m 23s | $3.83 | claude-sonnet-5, claude-opus-5 | cr dev
| Field | Value |
|---|---|
| Model | claude-sonnet-5, claude-opus-5 |
| Reviewers | go:implementation-tests, harness-engineering:repo-health, security:code-auditor, architecture:solid |
| Engine | claude_cli · claude-sonnet-5, claude-opus-5 |
| Reviewed by | cr · piekstra-dev |
| Duration | 1m 23s wall · 2m 00s compute |
| Cost | $3.83 |
| Tokens | 48 in / 7.1k out |
Per-workstream usage
go:implementation-tests— claude-sonnet-5- In: 12
- Out: 1.9k
- Cache read: 593.3k
- Cache create: 100.1k
- Cost: $0.81
- Duration: 29s
harness-engineering:repo-health— claude-sonnet-5- In: 12
- Out: 2.1k
- Cache read: 645.7k
- Cache create: 109.9k
- Cost: $0.88
- Duration: 30s
security:code-auditor— claude-sonnet-5- In: 6
- Out: 668
- Cache read: 162.6k
- Cache create: 56.8k
- Cost: $0.40
- Duration: 11s
architecture:solid— claude-opus-5- In: 12
- Out: 2.0k
- Cache read: 593.4k
- Cache create: 108.2k
- Cost: $1.43
- Duration: 38s
orchestrator-rollup— claude-sonnet-5- In: 6
- Out: 443
- Cache read: 129.1k
- Cache create: 43.3k
- Cost: $0.31
- Duration: 10s
…566) ## The problem Zero findings and "did not run" are the same number and opposite meanings. One says the code was examined and is clean; the other says nothing was examined. The summary table rendered both as `0`. On a real PR, this is what a run looked like where **four of five reviewers failed to start**: ``` | Reviewer | Findings | | security:code-auditor | 0 | | rust:implementation-tests | 0 | | architecture:solid | 0 | | policies:conventions | 0 | | documentation:docs | 1 | ``` Only `documentation:docs` actually ran — it is the one agent that needs no repo checkout. The failures *were* recorded, but as `incomplete_failed` in a coverage section well below the headline table. The summary is the part that gets read, pasted, and acted on, and it said all-clear. I nearly merged on that reading, twice. ## Scope **The outcome logic is already correct** — `hasIncompleteReviewerCoverage` coerces approval away when coverage is incomplete, so the PR was never actually approved. This change is purely about what a human reads. ## The fix A reviewer whose coverage status did not produce a result renders as `⚠️ did not run` instead of a count, reusing the existing `coverageResultProduced` helper. Reviewers **absent** from coverage keep their number, so an unknown status cannot turn a genuine zero into a phantom failure. That case has its own test. ## Tests Two tests, both confirmed to **fail with the change reverted**: - a failed reviewer is not rendered as `0`, while a genuinely clean reviewer keeps its honest `0` in the same table - unknown coverage leaves the count untouched Full suite green. ## Related [#565](#565) fixes the *cause* of that particular fleet failure (the workbench has no `refs/`, so per-reviewer clones fail). This one fixes the reporting, which matters for any reviewer failure regardless of cause — the two are independent.
The symptom
On a real PR, four of five reviewers reported 0 findings and the rollup rendered it as a clean review. They had not found nothing — they had never run:
A reviewer that cannot start contributes zero findings, so a total failure of the reviewer fleet is indistinguishable from an all-clear. That is the part worth fixing regardless of the cause.
The cause
Preparebuilds the workbench withinit→fetch --no-tags <remote> <sha>→checkout --detach. Fetching by SHA writes onlyFETCH_HEAD, and a detached checkout writes no ref, so the resulting.githas norefs/:Git does not consider a directory without
refs/a repository, so the per-reviewergit clone --no-hardlinks <workbench> <workspace>fails withrepository does not exist. Confirmed by hand: copying that workbench and writing a single ref makes the identical clone succeed.The fix
git update-ref refs/heads/cr-review-head <head SHA>after the checkout. HEAD stays detached, so the pinned-checkout semantics are unchanged; the repo simply becomes clonable.Test
TestPrepareLeavesWorkbenchClonableasserts the head is reachable through a real ref and then performs the actualgit clone --no-hardlinksa reviewer workspace uses. Verified it fails without the fix and passes with it. Full suite green.