Skip to content

fix(audio-quality): flush the build before the first row - #5807

Merged
kixelated merged 16 commits into
mainfrom
quest/m1/audio-jitter-target/opus-isolated-target
Oct 17, 2026
Merged

kixelated merged 16 commits into
mainfrom
quest/m1/audio-jitter-target/opus-isolated-target

Conversation

@kixelated

@kixelated kixelated commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The nightly's Opus near-zero isolated row reaches a 100 to 800 ms target with roughly one-second underrun episodes, while the plain row holds 60 ms. #5617 traced it to 100 to 320 ms main-thread long tasks that delay arrivals, and to sampling gaps in the CPU profile, but did not find the cause.

Root cause

It was never the isolated ring. That row just happens to be the matrix's first row, and it starts straight after run.sh's cold cargo build.

  • Every failing row on record was the first row after a fresh build. Every passing row was not. That includes the controls in test(audio): capture isolated Opus target stalls #5617, where the passing plain,isolated harness ran second, after a no-op build.
  • The long tasks come in clusters roughly 10.3 s apart (about 2, 12 and 23 s in, give or take a few seconds) and stop after about 30 s. This holds in the control, GC and CPU runs alike.
  • In an ARM experiment, plain was the first row after a cold build, and it failed the same way: 540 ms target, 2.2 episodes/min, long-task clusters at 5.7, 15.9 and 26.2 s. /proc/meminfo showed 800 MB dirty when the page loaded, flushed in 100 to 280 MB writeback bursts at exactly those moments (vmstat bo up to 278 MB/s, wa up to 17%). Once the backlog drained, about 30 s in, the long tasks stopped. In the same run, isolated-first with the build flushed beforehand held 60 ms with no long tasks.

Fix

run.sh runs sync after its builds and before the first row. The verification run used the nightly's original order (isolated first) straight after a 167 s cold build. The flush took 2.8 s. Both rows held a 60 ms target with zero underruns and no long task over 100 ms.

Impact

  • Public API: none. Wire: none. Harness only.
  • No regression test: reproducing it needs a cold build on a slow disk. The nightly row is the regression net once its budgets are tightened.
  • Quests: deletes opus-isolated-target.md and adds Opus near-zero budgets, which waits on seven post-fix nightlies before re-measuring the ceilings, following the AAC bursty pattern. watch.md no longer requires it.

Validation

  • just check passes (includes quest check), and shellcheck on run.sh is clean.
  • ARM experiment and verification runs are linked above. Their temporary workflow was removed before this PR.

Decisions

Both settled by the maintainer:

  1. Where the flush lives. The sync after builds stays in test/audio-quality/run.sh only. A shared harness_* helper is a follow-up only if another timing harness turns out to build and then grade straight away.
  2. Quest shape. The budget re-measure is its own waiting quest, opus-near-zero-budgets.md, not folded into aac-bursty-budgets.md, so it does not ride a different fix's nightly window.

Branch history

The branch carries #5617's stale pre-squash commits beneath a merge -s ours. The PR diff is clean (the five files above), and a squash merge drops them.

Follow-ups

  • Opus near-zero budgets: tighten once seven post-fix nightlies have run (quest added here).
  • Possibly check whether other timed harnesses (native-play, the drills) grade right after a cold build.

(Written by Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 15 commits October 15, 2026 13:49
Co-Authored-By: GPT-6 <noreply@openai.com>
Temporary branch-only diagnostic workflow; restore Nightly before the final quest PR.

Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Temporary branch-only experiment workflow; removed before the PR is ready.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…olated ring

Deletes opus-isolated-target and leaves the budget re-measure waiting on post-fix nightlies; drops the temporary experiment workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Outcome: root-caused and fixed. The near-zero stall follows the first row after a cold build, not the isolated ring. Periodic writeback of about 800 MB of dirty build output froze the page's main thread every ~10 s for the first ~30 s. The ARM experiment reproduced it on the plain ring when that row went first. With sync added to run.sh, the original isolated-first order held 60 ms with zero underruns after a cold build.

Left as a draft for two small decisions, each with a recommendation in the body: where the flush lives, and whether the budget re-measure stays a separate waiting quest (quest/m1/audio-jitter-target/opus-near-zero-budgets.md). Checks: just check and quest check pass, shellcheck is clean, and both ARM runs are linked in the body.

(Written by Opus 5.5)

@kixelated
kixelated marked this pull request as ready for review October 17, 2026 00:23
@coderabbitai

coderabbitai Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 03dba97d-c0d3-4c61-b385-d5f8763fc6f7
📥 Commits

Reviewing files that changed from the base of the PR and between 3789ee2 and b2bde3a.

📒 Files selected for processing: 6
  • quest/m1/audio-jitter-target/README.md
  • quest/m1/audio-jitter-target/opus-isolated-target.md
  • quest/m1/audio-jitter-target/opus-near-zero-budgets.md
  • quest/m1/audio-jitter-target/watch.md
  • quest/m1/browser-rename-check.md
  • test/audio-quality/run.sh
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-17T00:24:44.831455Z 84608c0 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated

Copy link
Copy Markdown
Collaborator Author

Merging via /quest-merge.

  • Change: test/audio-quality/run.sh runs sync after its builds, so the first graded row no longer competes with writeback of the cold build's dirty pages. Harness only; no public API or wire impact.
  • Quests: deletes opus-isolated-target.md; adds the waiting opus-near-zero-budgets.md to re-measure ceilings after seven post-fix nightlies.
  • Decisions (settled by the maintainer): flush stays local to run.sh; budget re-measure is its own quest.
  • Review: Codex completed on head 84608c0 with no findings, plus a thumbs-up. CI green.
  • Squash merge (the queue's method) drops test(audio): capture isolated Opus target stalls #5617's stale pre-squash commits under the merge -s ours.

Follow-up worth considering: check whether other timed harnesses (native-play, the drills) grade right after a cold build.

(Written by Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 17, 2026
Keep main's Audio rename quest in the epic's Required list, drop the
finished Opus isolated-ring line, and point browser-rename-check's
related link at the PR instead of the deleted quest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated added this pull request to the merge queue Oct 17, 2026
Merged via the queue into main with commit 091e38b Oct 17, 2026
8 checks passed
@kixelated
kixelated deleted the quest/m1/audio-jitter-target/opus-isolated-target branch October 17, 2026 04:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant