Repository navigation
flaky: just test ts loses most of the first video group when it is evicted as Old just after subscribing (4 of 51 CI runs) #5565
Description
The TS workflow's "TS compliance" step (just test ts) fails intermittently on its video-content check:
"check": "video-content",
"detail": "decoded pictures are not an intact suffix of the source"
4 of the 51 ts.yml runs between 2026-10-14 19:57Z and 2026-10-15 03:55Z failed this way, on four unrelated branches, two of which change nothing in moq-net, moq-relay or moq-mux's consumer:
- 38634848576,
fix/tokio-windows-refused-dial: onlyrs/moq-tokio/src/connection.rs, on3dabaafce - 38657903232, fix(ts)!: run the export one delay behind its source #5553: the TS export, its
--delaydefault andtest/ts, on323fda277 - 38662560784,
quest/m1/papercuts-rs - 38677047938,
quest/m1/cache-max-age(fix(net)!: age superseded groups by wall clock as well as media time #5548, since merged)
All four have the same signature in the failure artifact's sub.log:
WARN moq_mux::container::consumer: current group evicted; skipping to next buffered group track="0.avc3" group=1 error=Hang(Moq(Stream(Old)))
and the same timing and loss:
| run | video subscribe after publisher start | evicted after subscribe | source pictures in the capture |
|---|---|---|---|
| 38634848576 | 1.974 s | 16.9 ms | 25-33, 37, 50-499 |
| 38657903232 | 1.980 s | 20.6 ms | 25-37, 41, 50-499 |
| 38662560784 | 1.986 s | 19.8 ms | 25-33, 50-499 |
| 38677047938 | 1.979 s | 19.4 ms | 25-29, 33, 50-499 |
The clip has a group a second at 25 fps, so the subscriber starts on group 1 (pictures 25-49) within about 20 ms of group 2's start, receives its first few pictures, loses it as Old, and skips to group 2; the check admits only a suffix of the source, so it fails.
Not reproduced locally: 15 of 15 runs pass on main (3abf0161c, macOS arm64, debug) and 5 of 5 on #5553. #5548 (superseded groups also aged by wall clock) merged after the last of these runs, so whether it changes the rate is not known. Not diagnosed: I would look at what marks the group being read Old when its successor arrives that soon after the subscription, but I have not.
(Written by Claude Opus 5.5)
Activity
kixelated commentedon Oct 15, 2026
Root cause, from the #5553 run's artifacts (38657903232): the export's live-edge start subscribes with a zero budget, and every hop applies that budget to the start group itself.
ExportSource::live()(rs/moq-mux/src/container/source.rs, from feat(moq-mux)!: fixed-delay jitter buffer and per-PID T-STD admission for TS export #4645) subscribes withmax_delay = 0so the subscription starts at the newest group, then raises it to the export's budget withset_max_delayonce the first frame is read.max_delay: 0nsin bothrelay.logandpub.logfor0.avc3(publisher at 00:12:01.8836).poll_driftedmeasured group 1's reach (group 2's start) at zero behind the edge. Zero is>=a zero budget, so the publisher reset group 1's streamOld. The raised budget had not crossed both hops yet.video-contentfailed.The window is the time the first frame takes to come back plus the time the update takes to go out through the relay. A run fails when its subscribe lands within that window before a keyframe, and this clip has one keyframe a second. Nothing on the failing branches touches it: the race has been on
mainsince #4645.Retrying or raising timeouts won't fix this. "Start at the newest group" has to stop being expressed as a budget, so the export's real budget applies from the SUBSCRIBE on.
Relative(0)..in subscribe ranges is exactly that. Until it lands, the alternative without a wire change is to subscribe with the real budget and drop locally whatever comes before the newest group. That costs up to a delay of backlog on every join.(Written by Claude Opus 5.5)
kixelated commentedon Oct 15, 2026
This is hypothesis 1 (first candidate) of quest/m1/test-flakes-2/ts-video-content.md, now confirmed from the logs above. The quest recommends keeping the zero-budget live-edge pick and making the start group atomic: the export admits or drops a group whole. That also handles this case, because group 1 arrives truncated by
Oldand is dropped whole, and the capture starts intact at group 2. My earlier note about replacing the zero budget is the alternative the quest already weighed and rejected for its join latency.(Written by Claude Opus 5.5)
Merge origin/main into claude/quest-import-e3b6cb
quest: import moq-dev#5521, moq-dev#5563, moq-dev#5564 and moq-dev#5565…
fix(ts): start an export on a whole group and count every drop