Repository navigation
A real contribution steps its timebase every day or two, unflagged: decoders ride through, moq import and moq export end #7102
Description
What happens
We carry a live broadcast contribution into moq import srt on two independent paths (a 1+1 pair),
on a build of main from mid-October. In 90 hours the source switched timebase three times: back
16,841 s, forward 16,845 s two days later, back 16,844.6 s a day and a half after that. Both paths
saw each switch in the same second (the captured one in the same millisecond), so it is upstream of
both.
We captured the last one. At a single packet the whole multiplex changes source:
- every PID breaks its continuity counter once;
- PCR, and every PES PID's PTS and DTS, step back together by 16,844.35–16,844.65 s;
- no packet sets
discontinuity_indicator; - PAT, PMT, NIT and SDT are byte-identical on both sides, versions and CRCs alike;
- the new video joins mid-GOP, with its first keyframe 80 ms later in decode order.
It looks like a failover between two identically configured encoders whose clocks are not
synchronised. Strictly the stream is non-conformant: ISO/IEC 13818-1 requires the flag at a timebase
change, and TR 101 290 counts its absence as a priority-2 error (2.3b, PCR_discontinuity_indicator_error). We are reporting it to the
contribution's operator. But it is the stream as delivered, and it recurs.
What each receiver does with it
| Receiver | Backward switch | Forward switch |
|---|---|---|
moq import srt / ts |
both legs exit: frame timestamp … is below the previous group's start (#4543's rule) | carries on |
moq export ts on each subscriber |
ends with the broadcast | exits: the source's clock runs +172.5 ppm off ours, past the 30 ppm the output's may follow |
| Programme lost at a subscriber | 9.8 s with systemd restarting the import; 2.8 s at best, the capture replayed on main with a supervisor republishing at once under a new epoch |
8.0 s |
| ffmpeg 9.0.2 decoding the same capture | flags timestamp discontinuity, re-offsets, decodes on; errors until the next keyframe; output continuous | — (not captured) |
The same contribution feeds a large conventional distribution of IRDs and players, and as far as
we know none of these switches caused an outage there.
Because both legs see the same bytes, the 1+1 pair fails together: redundancy does not help. On
main at 4fdbad719 the import's rule (container/mod.rs, consumer.rs) and the export's clock
bound (jitter.rs) are unchanged.
Why ask again
#4582 asked for an in-process republish of a flagged rewind and was closed because restarting a
broadcast inside the importer is the wrong abstraction. #4543's premise is that a rewound timebase
is new content. This feed tests both:
- it is not new content: one service, the same tables and programme, switched between redundant
encoders. That is the case 13818-1's discontinuity model exists for, and receivers infer the
missing flag; - it is unflagged, so even A signalled backward TS discontinuity ends the import: republish the rest as a new broadcast in the same process #4582's ask would not have covered it;
- the forward half ends every subscriber's export, which no rewind rule explains;
- the cost is asymmetric: ffmpeg shows under half a second of damaged picture, while the MoQ chain
loses seconds of programme at every subscriber, both legs at once; - and the cost is the restart's, not the splice's. Replayed on
main(66af72315) with the import
restarted at once (export ts --delay 1s --follow 30s --stitch), a subscriber loses 2.8 s: 1.2 s of the old
encode, which the export held and drops when the broadcast ends, then 1.6 s of the new one,
while the restarted import holds its catalog about 2 s for its first mux-rate window and the
export joins at the group live by then. The mid-GOP join costs nothing of its own.
We understand why the rule exists: timestamps derived from the input alone, and group starts that
never go backwards. The options below try to keep that.
Options, least invasive first
- Export: treat an implausible clock step as a discontinuity, not drift. A step no source clock
could produce between two PCRs (2.3b's bound is 100 ms) re-locks the output with a marked break
instead of exiting. That fixes the forward half at every subscriber, with no importer change. - Import: infer the flag. When PCR and every timed PID step together beyond that bound, treat
it as signalled. The forward case then publishes with break markers, as a flagged forward jump
already does. - Backward: keep the broadcast, mark the break. This is the symmetric treatment of the forward
case and the decision we would ask you to reopen. One shape that keeps timestamps verbatim would be
a timeline generation in the catalog: group starts are monotonic within a generation, and
consumers treat a new generation as a break. You will know better than we do whether that fits
the hang draft. - If the rule stays, ship the supervisor.
moq import tsandmoq import srtwould republish
in-process under a fresh epoch at a rewind (theimport.restart(broadcast)call planned on A signalled backward TS discontinuity ends the import: republish the rest as a new broadcast in the same process #4582 before its quest was deleted in quest: audit the whole tree (2026-10-08) #5058), so each
deployment does not write its own. The docs would also say that real contributions trigger it.
As measured, a supervisor still pays both parts of the 2.8 s, unless a restarted import can
skip the catalog hold and the export can release what it holds of the old instance.
We can't share the capture itself: it is a broadcaster's content. We can provide a synthetic
reproducer, a public clip with the same whole-multiplex, unflagged step, if that would help.
(Written by Claude Opus 5.5)
Activity
kixelated commentedon Oct 28, 2026
Sorry can you explain the core problem? Why did the source jump timebase so dramatically? Is this MoQ's problem to fix, or something upstream?
kixelated commentedon Oct 28, 2026
16,844 is a VERY strange number to post. It sounds like an overflow somewhere to me?
kixelated commentedon Oct 28, 2026
Thanks for the detailed capture and measurements. Here's where we landed, planned in #7124.
Root cause. The two encoders share a frequency reference: once corrected for where the step lands in the GOP, the offset holds at ~16,844.5 s, a few ppm over days. But their timestamps start ~4 h 40 m apart, and the changeover after them sets no
discontinuity_indicator. So the fix belongs upstream: time-lock the encoders (source-timecode or epoch locking), or have the switch flag the break. The export's "+172.5 ppm" isn't a clock measurement. An unmarked step leaves the export's startup offset in place, so it bails on the first 2 s rate reading over 30 ppm.What changes in MoQ (all in m1, under
quest/m1/ts-reliability/):step-restart: two encoders are two broadcasts. An unflagged whole-multiplex step either way (forward beyond ~10 s, or a non-wrap step back) or any rewind ends the instance.moq import ts/srtthen republishes in-process under a fresh epoch. The supervisor lives in the CLI, not moq-mux. feat(moq-mux)!: importers publish stream timestamps with monotonic group starts; the catalog clock maps them to wall #4543's rule stays, and a catalog timeline generation was rejected.export-reacquire:moq export tsre-acquires an unmarked step instead of exiting, for non-moq TS sources.restart-catalog: a restarted import publishes its catalog at once by reusing the previous mux-rate measurement.stitch-catch-upquest drops the old instance's tail.A synthetic public reproducer of the whole-multiplex unflagged step would be very welcome for the tests.
(Written by Claude Opus 5.5)
kixelated commentedon Oct 28, 2026
The plan from #7124 is in the merge queue. In short: the source switched between two redundant encoders whose clocks started ~4 h 40 m apart, and it didn't flag the switch. That is an upstream misconfiguration, but MoQ will tolerate it. An unflagged step or a rewind republishes under a new epoch, and the export re-acquires the clock instead of exiting.
Three quests track it:
quest/m1/ts-reliability/step-restart.md: an unflagged step or a rewind ends the import instance, and themoq import ts/srtsupervisor republishes under a fresh epoch on the same SRT connection. This one closes the issue.quest/m1/ts-reliability/export-reacquire.md: the TS export re-acquires after an unmarked step instead of bailing on a bogus drift reading.quest/m1/ts-reliability/restart-catalog.md: a restarted instance reuses its predecessor's mux-rate measurement, so it publishes its catalog at once. Start it after step-restart.To donate tokens, run
/quest-start <branch>with one of these branches:quest/m1/ts-reliability/step-restartquest/m1/ts-reliability/export-reacquirequest/m1/ts-reliability/restart-catalog(Written by Claude Opus 5.5)
t0ms commentedon Oct 28, 2026
correct
tolerating this fault and avoiding the exit sounds like the right outcome, thanks
quest: import moq-dev#7067, moq-dev#7088, moq-dev#7101 and moq-dev#7102…
t0ms commentedon Oct 29, 2026
A synthetic reproducer of the whole-multiplex unflagged step, as asked:
t64-switch.py. It needs only Python 3 and FFmpeg with libx264, and no capture.It makes two encodes of one synthetic programme: H.264 at 25 Hz, MP2 and AC-3, under PAT, PMT, SDT and NIT, at 6 Mb/s CBR. The encodes run on clocks 16,844.5 s apart, with their GOPs out of phase, and are interleaved packet by packet as a broadcast multiplexer does. They are then cut together at one packet, where:
random_access_indicatorand holds non-IDR slices, and its first keyframe is 80 ms later in decode order, as on the real feed;discontinuity_indicator.t64-switch.py out.ts backsteps every clock back by 16,844.70–16,845.00 s, andforwardsteps it forward by 16,844.00–16,844.30 s. The lower clock sits at 33,155.5 s, so a fresh import doesn't read the step back as a wrap forward.STEP,CUTandKEYare environment knobs.On
mainat66af72315(before #7144 and #7165), each stimulus was played in real time intomoq import tsunder a supervisor that restarts it at once, with twomoq export ts --delay 1s --follow 30s --stitch:frame timestamp 33185806666 µs is below the previous group's start 50030568000 µs, the live feed's error. Each export loses 2.48–2.52 s of programme, against 2.8 s for the real capture replayed the same way. Both exports exit 0, and the two are byte-identical.the source's clock runs +40847.1 ppm off ours, past the 30 ppm the output's may follow; the live feed read +172.5 ppm.The stimulus in #7144's
step_test.rsplays one clip twice. This one differs where the real switch did: it cuts mid-PES and mid-GOP, and its streams step by amounts up to 0.3 s apart. It doesn't reproduce teletext or a TDT.If you measure loss on the export's output: the export authors the DTS of each group's IDR and the picture after it, 2 and 1 ticks below the IDR's PTS. With B-pictures, match pictures on PTS.
(Written by Claude Opus 5.5)