Skip to content

fix(review): bot pull requests are reviewed, edits cancel no review, conventions read live labels - #18

Merged
soydiloreto merged 5 commits into
mainfrom
fix/review-bots-concurrency-live-labels
Sep 30, 2026
Merged

soydiloreto merged 5 commits into
mainfrom
fix/review-bots-concurrency-live-labels

Conversation

@soydiloreto

@soydiloreto soydiloreto commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

📝 What changes

  • Pull requests the organisation's App opens get reviewed. claude-review.yml passes allowed_bots: dependabot[bot],<the App's slug> (from the token step, as issue-repro.yml already does). The Claude action refuses any bot actor not listed, and it compares names without the [bot] suffix.
  • An edit's review never cancels, is never cancelled, and pays nothing. It runs in a concurrency group of its own run; a push keeps its per-pull-request group and still cancels the review of the push before it (that check sits on an older commit and blocks nothing). On a commit the review already read it reuses the verdict, as before. On a commit its push's run is still reviewing, it waits (up to 50 minutes, longer than a review job can run; its own job gets 60) for that run's record of the commit and reuses its verdict, blocking included, with the edited description compared to the one the review read; when no record comes it fails, because a green check without a verdict would replace the push run's check under the same name. A capped review is tried before that wait and replays its last verdict. So an edit and a push on one commit pay for one review.
  • The conventions read live labels, and wait for the review's reading. A step reads the labels and the comments through the REST API (a warning, and the event's labels, when it cannot). conventions.sh finds the review App's last record (review-bot input, only that login's comment, only a 40-hex commit) and compares the title's type with the type:* label unless the review is still to read the head commit and is not capped: the record now carries max (the cap), so a capped review, which reads no new commit, is compared against at once. Any record it cannot trust means the comparison runs. Seven new --test cases: a mismatch on the reviewed commit fails; a record for another commit waits; a record someone else posted, a capped review (count 3 of 3) and a record whose sha is not a commit all compare (and fail), while count 3 of a cap of 8 still waits; with paginated comments, the last page's record counts.
  • README.md says all three.

💡 Why

Three things blocked Offload's pull requests this week with nothing wrong in the code. #26, a weekly learnings proposal opened by dilux-bot, got "did not finish" twice in 17 seconds: "Workflow initiated by non-human actor: dilux-bot (type: Bot). Add bot to allowed_bots list". On #26 again, the push-run review and the edit-run review landed a second apart on one commit; one was cancelled, and the cancelled check kept the pull request BLOCKED although the other passed, until it was re-run by hand. On #30 the title was changed to feat while the label was still the old review's type:perf: the conventions failed, the review then set type:feat, and neither a re-run (which replays the old event's labels) nor anything else cleared it until the description was edited.

🧪 How I tested it

  • bash scripts/conventions.sh --test (the seven new cases included) and every other script's --test.
  • The jq that reads the last review record, against a sample of the comments' JSON.
  • The workflows parse as YAML; the allowed_bots comparison read in the pinned action's source (src/github/validation/actor.ts at 8cf3482: lower-cased, [bot] stripped).

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

soydiloreto and others added 4 commits September 30, 2026 03:09
…conventions read live labels

Three things the last week of Offload pull requests ran into. The review of a pull request the organisation's App opened (the weekly review-learnings proposals) ended without a verdict: the Claude action refuses any bot actor not in allowed_bots, which listed only Dependabot; it now lists the App's slug too. An edit of the title or description and a push landing on one commit shared the review's concurrency group with cancel-in-progress, so one cancelled the other and the cancelled check blocked the merge though the other passed; an edit now reviews in its own group and cancels nothing. The conventions read the labels from the event, so a re-run replayed the old type:* label; they now read the labels as they are, and compare the title's type with the label only once the review has read the commit, so a push that changes the title waits for the review's new reading instead of failing on the old label.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ws compare; edits spend nothing

The local review of the first version found that the conventions trusted a review record in anyone's comment (a fake one switched the title check off, and a sha with a newline could write into GITHUB_OUTPUT), and that a capped review never writes a new record, which would switch the check off for good. The record is now read in conventions.sh from the REST comments, only from the review App's login, only with a 40-hex commit; the review writes its cap (max) into the record, and a capped review is compared against at once. An edit's review runs in a group of its own run, so a newer edit cannot cancel a queued one, and on a commit the review has not read it spends nothing (pending), so an edit and a push on one commit pay for one review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… instead of passing

The local review found that the pending mode ended green without a verdict, under the same check name as the push's review: editing the title after a blocking or unfinished push review would have turned the required check green. An edit on a commit its push's run is still reviewing now waits for that run's record and reuses its verdict (blocking included, and the edited description compared with the one the review read), and fails when none comes; a capped review is tried first and replays its verdict. The README paragraph is re-wrapped, and the conventions test the last page of paginated comments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… counterpart under the cap

The case passed a cap through printf's %s with its backslashes doubled, so the record was invalid JSON and the comparison ran for that reason, not because of the cap. The cap is now built inside rec, and a record under a cap of 8 waits where one at 3 of 3 compares. The README tables list review-bot, COMMENTS_FILE and REVIEW_BOT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread .github/workflows/claude-review.yml Outdated
@dilux-bot dilux-bot Bot added risk:high Set by the Claude review complexity:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review labels Sep 30, 2026
@dilux-bot

dilux-bot Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

✍️ The description changed since this review, so nobody has checked it against the code: no auto-merge until a push or the review:full label brings a new review.

Claude review · risk high · complexity high · type fix

New commit: an edit now waits up to 50 minutes (100 × 30 s), and its job gets timeout-minutes: 60 against the push review job's 30. That fixes the earlier major: a push review that runs long no longer loses the race to a red edit check. The job-level timeout-minutes expression is valid, and the 60-minute limit still leaves room for the steps before the loop. The other half of that finding is still open, now as a minor and worse than before. The loop does not check whether a push run exists. When there is none (the push review failed, was cancelled or ended with "did not finish"), every edit waits 50 minutes and then fails. The four earlier minors are untouched: the header in claude-review.yml, the header in conventions.yml, and the two conventions.sh edge cases. The description gets one thing wrong: it still says an edit waits "up to 25 minutes", and the code now waits up to 50.

✍️ The description does not match the code (see the summary): no auto-merge until it does.

  • minor .github/workflows/claude-review.yml:238: The edit wait does not check whether a push run exists: if the push review failed or was cancelled, every edit burns 50 runner minutes, then fails
  • minor .github/workflows/claude-review.yml:14: Header step 2 still says an edit only reuses the verdict. It omits the 50-minute wait for the push's verdict and that the App's PRs are reviewed
  • minor .github/workflows/conventions.yml:13: Header usage example does not show the new review-bot input or that labels and comments are read live
  • minor scripts/conventions.sh:466: With review:full set and the cap reached, the conventions compare at once against the old label though a new review will relabel; the check can stay red until re-run
  • minor scripts/conventions.sh:464: Records written before this change have no max, so 5 is assumed: a caller with another max-auto-reviews misjudges whether the review is capped until the next review

Policy floor: high (touches high-risk paths: .github/workflows/claude-review.yml, .github/workflows/conventions.yml, README.md, scripts/conventions.sh). Reviewed 3454d08 (since a7779d5; review 2 of 5 automatic). Author trusted for auto-merge: true.

🤖 AI review · claude-opus-5-5 (Anthropic) · $0.21, 6 turns

…nger than that review can run

The review on GitHub found that the edit's 25-minute wait was no longer than a push's review job (30 minutes, its Claude step alone 25), so an edit could fail with nothing wrong. The edit now waits up to 50 minutes, and an edit's job gets 60.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dilux-bot dilux-bot Bot added risk:high Set by the Claude review complexity:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review and removed risk:high Set by the Claude review complexity:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review labels Sep 30, 2026
@soydiloreto
soydiloreto merged commit 5d0f851 into main Sep 30, 2026
11 checks passed
@soydiloreto
soydiloreto deleted the fix/review-bots-concurrency-live-labels branch September 30, 2026 03:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

complexity:high Set by the Claude review risk:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant