fix(benchmarks): wait for scripted event delivery - #6487
Conversation
Signed-off-by: Salman Mohammed <smohammed@squareup.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f5c90d1613
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Salman Mohammed <smohammed@squareup.com>
philazar
left a comment
There was a problem hiding this comment.
lgtm. Agent reviewer found no bugs and I didnt find anything either
wpfleger96
left a comment
There was a problem hiding this comment.
Reviewed the exact head and traced the receipt contract through the benchmark runtime, completed-batch delivery accounting, the completed-before-control race, and successful non-cancelling steer acknowledgments. The scripted-event gate now requires every expected event receipt plus a terminal turn, while ordinary solo tasks keep the existing first-turn completion behavior. CI is green, and a mutation run that restored the premature ends > 0 shortcut failed both new regressions (2 failed, 85 passed), confirming the tests observe the fix at the intended seam.
…uctions-prompt * origin/main: fix(benchmarks): wait for scripted event delivery (#6487) Polish mobile channel navigation and message sends (#6488) Clarify huddle message destination (#6496) feat(archive): add observer-frame retention schema and gated DB adapter (#5719) fix(desktop): restore human barge-in over agent TTS in huddles (#6431) chore(release): release Buzz Desktop version 0.5.18 (#6489) fix(desktop): simplify duplicate agent provenance (#6401) test(benchmarks): expand Buzz-native dataset (#6448) fix(desktop): sender names in notifications + macOS click-through routing (#6427) docs: clarify two-layer moderation ownership (#6481) Fix mobile thread tail and iOS channel header (#6399) chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392) polish(desktop): finish Projects navigation and context chrome (#6429) fix(desktop): clarify add agents channel action (#6374) Repair stale large channel roster snapshots (#6251) Signed-off-by: Salman Mohammed <smohammed@squareup.com>
* origin/main: (88 commits) docs(nips): comprehensive NIP-FI — core + claimable profiles (EDGE/LIFECYCLE/DELEG/CONF) (#5946) fix(benchmarks): wait for scripted event delivery (#6487) Polish mobile channel navigation and message sends (#6488) Clarify huddle message destination (#6496) feat(archive): add observer-frame retention schema and gated DB adapter (#5719) fix(desktop): restore human barge-in over agent TTS in huddles (#6431) chore(release): release Buzz Desktop version 0.5.18 (#6489) fix(desktop): simplify duplicate agent provenance (#6401) test(benchmarks): expand Buzz-native dataset (#6448) fix(desktop): sender names in notifications + macOS click-through routing (#6427) docs: clarify two-layer moderation ownership (#6481) Fix mobile thread tail and iOS channel header (#6399) chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392) polish(desktop): finish Projects navigation and context chrome (#6429) fix(desktop): clarify add agents channel action (#6374) Repair stale large channel roster snapshots (#6251) feat(desktop-messages): show compact Buzz link metadata (#6252) feat(workflows): reply in-thread from send_message action (#6178) perf(desktop): split discover_acp_providers into cheap and forced paths (#6330) fix(desktop): restore recent channel sorting (#6402) ... Co-authored-by: Bradley Axen <baxen@squareup.com> Signed-off-by: Bradley Axen <baxen@squareup.com> # Conflicts: # crates/buzz-acp/src/base_prompt.md
* origin/main: (65 commits) fix(composer): preserve caret when inserting mentions mid-message (#6531) chore(deps): update rust crate async-trait to v0.1.92 (#6094) chore(deps): update dependency sonner to v2.0.8 (#6093) chore(deps): update rust crate http-body-util to v0.1.4 (#5452) chore(deps): update rust crate http to v1.4.2 (#5451) chore(deps): update rust crate futures-util to v0.3.33 (#5448) chore(deps): update rust crate futures to v0.3.33 (#5445) chore(deps): update dependency @tauri-apps/api to v2.11.1 (#5444) chore(deps): update ubuntu:24.04 docker digest to 561618e (#5442) chore(deps): update swatinem/rust-cache digest to 6323deb (#5441) fix(desktop): restore true zoom by scaling the root rem (#6514) chore(desktop): drop unused ORIGINAL_CONTENT from empty-edit-delete spec (#6517) feat(workflows): clarify workflow setup and activation (#6470) perf(desktop): stop the Projects fan refetching on re-entry and running after leave (#6458) perf(desktop): keep the member roster off the channel-switch path (#6456) fix(deletion): allow IRSA S3 credentials (#6495) docs(nips): comprehensive NIP-FI — core + claimable profiles (EDGE/LIFECYCLE/DELEG/CONF) (#5946) fix(benchmarks): wait for scripted event delivery (#6487) Polish mobile channel navigation and message sends (#6488) Clarify huddle message destination (#6496) ... Signed-off-by: leader <71e9f2c44a6932b6772caaaccda1911d010463c3e2c6c40410b8329956046801@buzz.block.builderlab.xyz> Co-authored-by: Tom Brow <tomb@block.xyz> Signed-off-by: Tom Brow <tomb@block.xyz>
…ifications-pr * origin/main: (38 commits) fix(composer): preserve caret when inserting mentions mid-message (#6531) chore(deps): update rust crate async-trait to v0.1.92 (#6094) chore(deps): update dependency sonner to v2.0.8 (#6093) chore(deps): update rust crate http-body-util to v0.1.4 (#5452) chore(deps): update rust crate http to v1.4.2 (#5451) chore(deps): update rust crate futures-util to v0.3.33 (#5448) chore(deps): update rust crate futures to v0.3.33 (#5445) chore(deps): update dependency @tauri-apps/api to v2.11.1 (#5444) chore(deps): update ubuntu:24.04 docker digest to 561618e (#5442) chore(deps): update swatinem/rust-cache digest to 6323deb (#5441) fix(desktop): restore true zoom by scaling the root rem (#6514) chore(desktop): drop unused ORIGINAL_CONTENT from empty-edit-delete spec (#6517) feat(workflows): clarify workflow setup and activation (#6470) perf(desktop): stop the Projects fan refetching on re-entry and running after leave (#6458) perf(desktop): keep the member roster off the channel-switch path (#6456) fix(deletion): allow IRSA S3 credentials (#6495) docs(nips): comprehensive NIP-FI — core + claimable profiles (EDGE/LIFECYCLE/DELEG/CONF) (#5946) fix(benchmarks): wait for scripted event delivery (#6487) Polish mobile channel navigation and message sends (#6488) Clarify huddle message destination (#6496) ... Signed-off-by: Tom Brow <tomb@block.xyz>
Why
The Buzz-native runtime could stop a solo agent after one turn even when a scripted follow-up event was queued or already running. That could turn a harness timing race into a benchmark failure.
Follow-up to #6448.
What
DONE:message.Testing
pytest -q tests/test_container_runtime.py -k "solo_turn_end or scripted_events"— 3 passedruff checkon the three touched Python filesruff format --checkon the three touched Python filesbuzz-native-solo-luna.yamlHarbor run, one attempt per affected task:cross-thread-requests— reward 1.0; all 6 task-specific checks passedinterleaved-agent-reports— reward 1.0; all 7 task-specific checks passedRisk
Low. This only changes completion detection for solo benchmark tasks with scripted events. Ordinary solo tasks retain immediate completion after their first turn ends.
Update Aug 21, 13:07 EDT: Replaced the fixed settle delay with exact delivery receipts after review feedback.
interleaved-agent-reportsscored 1.0.cross-thread-requestscompleted both calculations and thread isolation but Luna omitted the user mention on ALPHA, so that model-output dimension scored 0.Generated with Codex