From ddc68c66e8bef7c74fe6d9831139f4d2251c72f9 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Sat, 17 Oct 2026 09:20:28 -0700 Subject: [PATCH 1/2] quest: plan the TS export's remaining MP2 join hole Co-Authored-By: Claude Opus 5.5 --- quest/m1/test-flakes-2/README.md | 1 + quest/m1/test-flakes-2/ts-mp2-join-hole.md | 41 ++++++++++++++++++++++ quest/m2/audio-join-budget.md | 3 ++ 3 files changed, 45 insertions(+) create mode 100644 quest/m1/test-flakes-2/ts-mp2-join-hole.md diff --git a/quest/m1/test-flakes-2/README.md b/quest/m1/test-flakes-2/README.md index 2ba320a6b16..80209fa9c89 100644 --- a/quest/m1/test-flakes-2/README.md +++ b/quest/m1/test-flakes-2/README.md @@ -35,6 +35,7 @@ Public API: none. Wire: none. ## Required +- [TS MP2 join hole](/quest/m1/test-flakes-2/ts-mp2-join-hole.md) - the TS export skips no MPEG-1 audio group right after its live join, fixed at the cause - [Auth refresh mocks time](/quest/m1/test-flakes-2/auth-refresh-mock-time.md) - moq-relay's auth refresh and fixed-lease tests measure a grant's life on one mocked clock - [uring teardown close](/quest/m1/test-flakes-2/uring-teardown-close.md) - moq-uring's published-close teardown test holds under load, fixed at its cause - [Relay burst drill](/quest/m1/test-flakes-2/relay-burst-impaired.md) - the impaired transport-peer burst drill never aborts its subscription while the link is up diff --git a/quest/m1/test-flakes-2/ts-mp2-join-hole.md b/quest/m1/test-flakes-2/ts-mp2-join-hole.md new file mode 100644 index 00000000000..601c32518a5 --- /dev/null +++ b/quest/m1/test-flakes-2/ts-mp2-join-hole.md @@ -0,0 +1,41 @@ +# [M] TS export loses no MP2 group after its live join + +## Goal + +TS validation's "TS full multiplex at its own rate" step (`just test ts --full`) passes +reliably because the exporter never skips an MPEG-1 audio group right after it joins. +The fix lands at the cause, with a deterministic regression test that fails without it. + +## Plan + +Seen 2026-10-17 on #5857's TS validation run 39042472016 (head 754a5af, `main` with +#5854 plus a test-only change). PID 0x101 lost 96 ms in one 120 ms step at 2.390 s, +right after the capture's first MP2 frame. The exporter logged it itself, so this is +real loss, not one of the ffprobe ghost holes #5854 fixed in `holes.py`: +`media skipped upstream ... track="2.mp2" at=2.389977s skipped=120ms`. + +What #5854 already rules out: the SUBSCRIBE carried `max_delay: 450ms`, so the +zero-budget race it fixed (and that `a_live_start_loses_no_group_the_publisher_bursts` +reproduces) is not this. The skip was declared about 60 ms after `subscribe started`, +far inside the budget, so a budget expiry can't explain it either. The subscriber +starts before the publisher, so the relay held no backlog at the join. MP2 is +published as one group per 24 ms frame, about five per PES, and 120 ms is about one +PES. + +Suspicion, unverified: #5854's live latch (`container::Consumer::live`) starts at the +newest group *announced* when any frame first arrives, and drops what lies below it. +A group's announcement and its frames race on separate streams. So the latch may pick a +start that later reads as a hole against groups still in flight, or that misjudges a +burst. + +First step: trace the consumer's skip and latch decisions against that run's artifact +(the `ts-failure` upload, retained until about 2026-10-24) to find which rule declared +the gap. Then reproduce it deterministically over `moq_net::transport::memory`, the way +#5857's test does, before choosing a fix. + +Public API: none expected. Wire: none expected. + +## Related + +- [Audio join budget](/quest/m2/audio-join-budget.md) - moq-audio's live join may share the same race; whichever lands first should check the other +- [TS fidelity and recovery](/quest/m1/ts-reliability/README.md) - loss after a late join is part of the TS contract diff --git a/quest/m2/audio-join-budget.md b/quest/m2/audio-join-budget.md index 9d039df4423..907d5d394a0 100644 --- a/quest/m2/audio-join-budget.md +++ b/quest/m2/audio-join-budget.md @@ -20,3 +20,6 @@ Reproduce it the same way first: a publisher bursting one-frame groups over budget from the start and keep the warmup reach local, as the export now picks its live edge in the consumer instead of on the wire. +## Related + +- [TS MP2 join hole](/quest/m1/test-flakes-2/ts-mp2-join-hole.md) - the TS export still loses MP2 groups after its live join; it may share this race From 1c70e2b39aef525368a473a5d4f7b6e90148f0a6 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Sat, 17 Oct 2026 09:24:02 -0700 Subject: [PATCH 2/2] quest: keep the run's evidence and don't rule out a media-time skip Co-Authored-By: Claude Opus 5.5 --- quest/m1/test-flakes-2/ts-mp2-join-hole.md | 41 ++++++++++++++-------- 1 file changed, 26 insertions(+), 15 deletions(-) diff --git a/quest/m1/test-flakes-2/ts-mp2-join-hole.md b/quest/m1/test-flakes-2/ts-mp2-join-hole.md index 601c32518a5..f12983dea00 100644 --- a/quest/m1/test-flakes-2/ts-mp2-join-hole.md +++ b/quest/m1/test-flakes-2/ts-mp2-join-hole.md @@ -14,24 +14,35 @@ right after the capture's first MP2 frame. The exporter logged it itself, so thi real loss, not one of the ffprobe ghost holes #5854 fixed in `holes.py`: `media skipped upstream ... track="2.mp2" at=2.389977s skipped=120ms`. -What #5854 already rules out: the SUBSCRIBE carried `max_delay: 450ms`, so the -zero-budget race it fixed (and that `a_live_start_loses_no_group_the_publisher_bursts` -reproduces) is not this. The skip was declared about 60 ms after `subscribe started`, -far inside the budget, so a budget expiry can't explain it either. The subscriber -starts before the publisher, so the relay held no backlog at the join. MP2 is -published as one group per 24 ms frame, about five per PES, and 120 ms is about one -PES. +Evidence kept from that run's `ts-failure` artifact, which expires about 2026-10-24 +and holds only INFO logs, not the consumer's decisions: + +- Publisher instance up at 15:04:31.10. The subscriber subscribed to `2.mp2` at + 15:04:33.082 with `max_delay: 450ms`, `start_group: None`, `order: Descending`. The + relay forwarded the same upstream at 15:04:33.083. +- The skip was logged at 15:04:33.143, about 60 ms after subscribing. The capture's + first MP2 frame is at 2.366 s against a source starting at 1.070 s. Video starts at + 2.080 s, AC-3 at 2.515 s, both without holes. +- MP2 is published as one group per 24 ms frame, about five per PES, so the 120 ms + step is about one PES. + +What #5854 already rules out: the zero-budget race it fixed (reproduced by +`a_live_start_loses_no_group_the_publisher_bursts`), since this SUBSCRIBE carried the +real budget. What it doesn't rule out: the consumer measures `max_delay` in media time +(`max_timestamp - next_start`), not wall time, so 60 ms after subscribing is no bound. +The publisher had run about two seconds before the join, so its cache could hand the +budget a backlog spanning that much media time even though the relay held none. Suspicion, unverified: #5854's live latch (`container::Consumer::live`) starts at the newest group *announced* when any frame first arrives, and drops what lies below it. -A group's announcement and its frames race on separate streams. So the latch may pick a -start that later reads as a hole against groups still in flight, or that misjudges a -burst. - -First step: trace the consumer's skip and latch decisions against that run's artifact -(the `ts-failure` upload, retained until about 2026-10-24) to find which rule declared -the gap. Then reproduce it deterministically over `moq_net::transport::memory`, the way -#5857's test does, before choosing a fix. +A group's announcement and its frames race on separate streams, and the backlog +arrives beside the live groups. So the latch, or the budget's skip rules right after +it, may misjudge a burst that is still in flight. + +First step: reproduce it deterministically over `moq_net::transport::memory`, the way +#5857's test does. Use a publisher holding a backlog when the live export joins, then +bursting a PES of one-frame groups. Trace which skip or latch rule declares the gap +before choosing a fix. Public API: none expected. Wire: none expected.