fix(bench): bucket competitors under release-dry-run for a like-for-like head-to-head#188
Merged
Merged
Conversation
…ike head-to-head Closes #187
BryanFRD
enabled auto-merge (squash)
July 20, 2026 19:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #187
The competitive head-to-head compared different operations: competitors ran their dry-run release planning while their result was filed under the
checkbucket, so the site showedferrflow check(a lighter read-only planning check) against the competitors' full--dry-run.This changes the empty-command fallback so a competitor's single dry-run invocation is bucketed as
release-dry-run— the same key asferrflow release --dry-run. The site'sbuildGridis data-driven (it shows whichever command the competitors carry), so the displayed comparison automatically becomesrelease --dry-runfor every tool: compute next version + notes, write nothing.ferrflow still benchmarks
check(used only by the separately-labelled warm-cache stat), and itsrelease --dry-runcell already existed — it just never had competitors in the same column until now.No behaviour change to the compare/derive scripts; bats fixtures use arbitrary keys and are unaffected. Effective once FerrFlow bumps its pinned
FerrLabs/Benchmarksref and the next bench run refreshes the site data.