Skip to content

fix(benchmarks): wait for scripted event delivery - #6487

Merged
salman1993 merged 2 commits into
mainfrom
smohammed/wait-for-scripted-turns
Aug 21, 2026
Merged

fix(benchmarks): wait for scripted event delivery#6487
salman1993 merged 2 commits into
mainfrom
smohammed/wait-for-scripted-turns

Conversation

@salman1993

@salman1993 salman1993 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Why

The Buzz-native runtime could stop a solo agent after one turn even when a scripted follow-up event was queued or already running. That could turn a harness timing race into a benchmark failure.

Follow-up to #6448.

What

  • Treat solo tasks with scripted events separately from ordinary one-turn tasks.
  • Wait until turn counts and authored message IDs remain idle and unchanged for several polls before stopping the agent.
  • Remove the message-count shortcut that could stop an active turn.
  • Add focused regressions for a delayed follow-up turn and an already-running turn with a DONE: message.

Testing

  • pytest -q tests/test_container_runtime.py -k "solo_turn_end or scripted_events" — 3 passed
  • ruff check on the three touched Python files
  • ruff format --check on the three touched Python files
  • Local buzz-native-solo-luna.yaml Harbor run, one attempt per affected task:
    • cross-thread-requests — reward 1.0; all 6 task-specific checks passed
    • interleaved-agent-reports — reward 1.0; all 7 task-specific checks passed
    • 2 completed trials, 0 exceptions, 56 seconds total

Risk

Low. This only changes completion detection for solo benchmark tasks with scripted events. Ordinary solo tasks retain immediate completion after their first turn ends.


Update Aug 21, 13:07 EDT: Replaced the fixed settle delay with exact delivery receipts after review feedback.

  • ACP now records the event IDs delivered by each completed turn; the runtime also recognizes existing successful-steer receipts.
  • Scripted trials stop only after every expected event ID is acknowledged and no turn is active. Missing receipts wait for the trial budget instead of producing partial evidence.
  • TDD regression delayed delivery beyond the old five-poll boundary and failed before the implementation; the receipt parser and Rust receipt format are pinned independently.
  • Focused verification: 31 container-runtime tests passed, 6 ACP delivery tests passed, plus Ruff, rustfmt, and Clippy.
  • Receipt-gated Luna run: 2 completed trials and 0 exceptions. interleaved-agent-reports scored 1.0. cross-thread-requests completed both calculations and thread isolation but Luna omitted the user mention on ALPHA, so that model-output dimension scored 0.
  • Risk remains low: the ACP production change adds delivery receipt logging and restores a missing turn-end log on a completed-before-control race; it does not change queue dispatch behavior.

Generated with Codex

Signed-off-by: Salman Mohammed <smohammed@squareup.com>
@salman1993
salman1993 marked this pull request as ready for review August 21, 2026 16:42
@salman1993
salman1993 requested a review from a team as a code owner August 21, 2026 16:42

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f5c90d1613

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread benchmarks/harbor-buzz-orchestra/src/harbor_buzz_orchestra/container_runtime.py Outdated
Signed-off-by: Salman Mohammed <smohammed@squareup.com>
@salman1993 salman1993 changed the title fix(benchmarks): wait for scripted turns to settle fix(benchmarks): wait for scripted event delivery Aug 21, 2026

@philazar philazar left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. Agent reviewer found no bugs and I didnt find anything either

@salman1993
salman1993 enabled auto-merge (squash) August 21, 2026 19:59

@wpfleger96 wpfleger96 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the exact head and traced the receipt contract through the benchmark runtime, completed-batch delivery accounting, the completed-before-control race, and successful non-cancelling steer acknowledgments. The scripted-event gate now requires every expected event receipt plus a terminal turn, while ordinary solo tasks keep the existing first-turn completion behavior. CI is green, and a mutation run that restored the premature ends > 0 shortcut failed both new regressions (2 failed, 85 passed), confirming the tests observe the fix at the intended seam.

@salman1993
salman1993 merged commit 0254255 into main Aug 21, 2026
32 checks passed
@salman1993
salman1993 deleted the smohammed/wait-for-scripted-turns branch August 21, 2026 20:09
salman1993 added a commit that referenced this pull request Aug 21, 2026
…uctions-prompt

* origin/main:
  fix(benchmarks): wait for scripted event delivery (#6487)
  Polish mobile channel navigation and message sends (#6488)
  Clarify huddle message destination (#6496)
  feat(archive): add observer-frame retention schema and gated DB adapter (#5719)
  fix(desktop): restore human barge-in over agent TTS in huddles (#6431)
  chore(release): release Buzz Desktop version 0.5.18 (#6489)
  fix(desktop): simplify duplicate agent provenance (#6401)
  test(benchmarks): expand Buzz-native dataset (#6448)
  fix(desktop): sender names in notifications + macOS click-through routing (#6427)
  docs: clarify two-layer moderation ownership (#6481)
  Fix mobile thread tail and iOS channel header (#6399)
  chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392)
  polish(desktop): finish Projects navigation and context chrome (#6429)
  fix(desktop): clarify add agents channel action (#6374)
  Repair stale large channel roster snapshots (#6251)

Signed-off-by: Salman Mohammed <smohammed@squareup.com>
baxen pushed a commit that referenced this pull request Aug 21, 2026
* origin/main: (88 commits)
  docs(nips): comprehensive NIP-FI — core + claimable profiles (EDGE/LIFECYCLE/DELEG/CONF) (#5946)
  fix(benchmarks): wait for scripted event delivery (#6487)
  Polish mobile channel navigation and message sends (#6488)
  Clarify huddle message destination (#6496)
  feat(archive): add observer-frame retention schema and gated DB adapter (#5719)
  fix(desktop): restore human barge-in over agent TTS in huddles (#6431)
  chore(release): release Buzz Desktop version 0.5.18 (#6489)
  fix(desktop): simplify duplicate agent provenance (#6401)
  test(benchmarks): expand Buzz-native dataset (#6448)
  fix(desktop): sender names in notifications + macOS click-through routing (#6427)
  docs: clarify two-layer moderation ownership (#6481)
  Fix mobile thread tail and iOS channel header (#6399)
  chore(deps): pin earshot below 1.2.0 pending a VAD threshold re-pick (#6392)
  polish(desktop): finish Projects navigation and context chrome (#6429)
  fix(desktop): clarify add agents channel action (#6374)
  Repair stale large channel roster snapshots (#6251)
  feat(desktop-messages): show compact Buzz link metadata (#6252)
  feat(workflows): reply in-thread from send_message action (#6178)
  perf(desktop): split discover_acp_providers into cheap and forced paths (#6330)
  fix(desktop): restore recent channel sorting (#6402)
  ...

Co-authored-by: Bradley Axen <baxen@squareup.com>
Signed-off-by: Bradley Axen <baxen@squareup.com>

# Conflicts:
#	crates/buzz-acp/src/base_prompt.md
brow added a commit that referenced this pull request Aug 22, 2026
* origin/main: (65 commits)
  fix(composer): preserve caret when inserting mentions mid-message (#6531)
  chore(deps): update rust crate async-trait to v0.1.92 (#6094)
  chore(deps): update dependency sonner to v2.0.8 (#6093)
  chore(deps): update rust crate http-body-util to v0.1.4 (#5452)
  chore(deps): update rust crate http to v1.4.2 (#5451)
  chore(deps): update rust crate futures-util to v0.3.33 (#5448)
  chore(deps): update rust crate futures to v0.3.33 (#5445)
  chore(deps): update dependency @tauri-apps/api to v2.11.1 (#5444)
  chore(deps): update ubuntu:24.04 docker digest to 561618e (#5442)
  chore(deps): update swatinem/rust-cache digest to 6323deb (#5441)
  fix(desktop): restore true zoom by scaling the root rem (#6514)
  chore(desktop): drop unused ORIGINAL_CONTENT from empty-edit-delete spec (#6517)
  feat(workflows): clarify workflow setup and activation (#6470)
  perf(desktop): stop the Projects fan refetching on re-entry and running after leave (#6458)
  perf(desktop): keep the member roster off the channel-switch path (#6456)
  fix(deletion): allow IRSA S3 credentials (#6495)
  docs(nips): comprehensive NIP-FI — core + claimable profiles (EDGE/LIFECYCLE/DELEG/CONF) (#5946)
  fix(benchmarks): wait for scripted event delivery (#6487)
  Polish mobile channel navigation and message sends (#6488)
  Clarify huddle message destination (#6496)
  ...

Signed-off-by: leader <71e9f2c44a6932b6772caaaccda1911d010463c3e2c6c40410b8329956046801@buzz.block.builderlab.xyz>

Co-authored-by: Tom Brow <tomb@block.xyz>
Signed-off-by: Tom Brow <tomb@block.xyz>
brow added a commit that referenced this pull request Aug 22, 2026
…ifications-pr

* origin/main: (38 commits)
  fix(composer): preserve caret when inserting mentions mid-message (#6531)
  chore(deps): update rust crate async-trait to v0.1.92 (#6094)
  chore(deps): update dependency sonner to v2.0.8 (#6093)
  chore(deps): update rust crate http-body-util to v0.1.4 (#5452)
  chore(deps): update rust crate http to v1.4.2 (#5451)
  chore(deps): update rust crate futures-util to v0.3.33 (#5448)
  chore(deps): update rust crate futures to v0.3.33 (#5445)
  chore(deps): update dependency @tauri-apps/api to v2.11.1 (#5444)
  chore(deps): update ubuntu:24.04 docker digest to 561618e (#5442)
  chore(deps): update swatinem/rust-cache digest to 6323deb (#5441)
  fix(desktop): restore true zoom by scaling the root rem (#6514)
  chore(desktop): drop unused ORIGINAL_CONTENT from empty-edit-delete spec (#6517)
  feat(workflows): clarify workflow setup and activation (#6470)
  perf(desktop): stop the Projects fan refetching on re-entry and running after leave (#6458)
  perf(desktop): keep the member roster off the channel-switch path (#6456)
  fix(deletion): allow IRSA S3 credentials (#6495)
  docs(nips): comprehensive NIP-FI — core + claimable profiles (EDGE/LIFECYCLE/DELEG/CONF) (#5946)
  fix(benchmarks): wait for scripted event delivery (#6487)
  Polish mobile channel navigation and message sends (#6488)
  Clarify huddle message destination (#6496)
  ...

Signed-off-by: Tom Brow <tomb@block.xyz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants