fix(hang)!: CMAF decoders time samples from the frame timestamp

doc/concept/hang.md

@@ -139,6 +139,13 @@ The `container.kind` on each rendition says how frames are framed:
 A consumer skips renditions with a kind it doesn't recognize and carries them
 through when republishing the catalog.
 
+A CMAF frame's timestamp is its fragment's earliest presentation time, and the
+broadcast timeline: `tfdt` only places the samples relative to each other. A
+publisher can move a passthrough track to another timeline by changing the frame
+timestamp alone, without rewriting the fragment. An untimed track's frames carry
+no timestamp, so their samples present at `tfdt`. A timed CMAF track declares the
+init segment's `mdhd` timescale, so the frame timestamp lands on a media tick.
+
 A Legacy or LOC video publisher can close the last frame's duration with an empty
 codec payload whose timestamp is that frame's exclusive end. Consumers treat it
 as metadata and never pass it to a decoder. This lets a group close immediately

doc/setup/upgrade.md

@@ -230,6 +230,17 @@ These land with the next breaking release, not the 2026-09-23 train.
     On a moq-transport track with `TIMESCALE`, an object without a Timestamp
     is malformed. A new subscriber with no start on an untimed track starts at
     the latest group.
+- **CMAF decodes at the frame timestamp.** A fragment's earliest sample
+  presents at its moq-net frame timestamp; `tfdt` only orders the samples. In
+  TypeScript, `Container.Format.decode` and `Cmaf.decodeDataSegment` take that
+  timestamp: pass `frame.timestamp` alongside `frame.payload`, or `undefined`
+  for an untimed frame, whose samples present at `tfdt`.
+  `Cmaf.decodeTimestamp` is gone; the frame timestamp is the fragment's time.
+  moq-mux's CMAF `Wire::write` refuses a track whose timescale isn't the init's
+  `mdhd` timescale with `fmp4::Error::TimescaleMismatch`. `import::Track` and
+  `TrackStream` accept a CMAF rendition's track at that timescale; a track you
+  create yourself declares it with `track::Info::with_timescale`. Both decoders refuse a `trun` whose
+  `data_offset` doesn't start at the next sample in the `mdat`.
 
 ## Wire
 

drafts/draft-lcurley-moq-hang.md

@@ -609,7 +609,12 @@ For example, h.264 with no `description` field would be annex.b encoded, while h
 For a text track, the remainder is the cue in the track's declared `format` (for example a `WEBVTT` segment).
 
 ## cmaf
-Each frame is a complete fragmented MP4 fragment (`moof`+`mdat`), carrying its own timestamps.
+Each frame is a complete fragmented MP4 fragment (`moof`+`mdat`).
+The frame timestamp is the presentation time of the fragment's earliest sample; the fragment's own timestamps only place its samples relative to that sample.
+A consumer MUST present each sample at the frame timestamp plus its offset from the earliest sample, even when `tfdt` says otherwise.
+A frame on an untimed track has no timestamp, so its samples present at the fragment's own timestamps.
+A timed track's timescale MUST equal the init segment's `mdhd` timescale, so the frame timestamp carries the earliest sample's time exactly.
+A fragment's runs MUST lie back to back in the `mdat`, in `trun` order, with each `data_offset` counting from the first byte of the `moof`.
 Audio samples MUST be marked as sync samples, including samples inside a group.
 A sync sample does not declare an audio group boundary; the publisher chooses those boundaries.
 
@@ -1082,6 +1087,7 @@ This document has no IANA actions.
 
 - Defined encoder `jitter` as flush lateness above the rendition's own recent minimum, replacing fixed frame-duration hints; container batches retain media-span estimates.
 - Clarified that CMAF audio samples are sync samples independently of publisher group boundaries.
+- Timed CMAF samples from the frame timestamp, which is the fragment's earliest presentation time; `tfdt` is only relative within the fragment, and places the samples of an untimed frame. A timed CMAF track counts in its `mdhd` ticks.
 - Clarified that container importers can estimate jitter from batch media spans without measuring input wait time.
 - Specified the `jitter` field's computation: the publisher's own structure rather than the network, rounded up to whole milliseconds, never `0` (a consumer treats `0` as absent), and never lowered once advertised.
 - For video, an empty codec payload is the exclusive end of the frame before it. A video publisher SHOULD end each group with one when the exclusive end is known. Audio retains its terminal-trimming marker before codec drain packets.

js/hang/src/container/cmaf/decode.ts

@@ -298,37 +298,29 @@ function extractAudioSpecificConfig(esds: Uint8Array): Uint8Array | undefined {
 	return undefined;
 }
 
-/**
- * Extract just the base media decode time from a data segment (moof + mdat).
- * This is a lighter-weight function when you only need the timestamp.
- *
- * @param segment - The moof + mdat data
- * @param init - Parsed init segment (provides timescale)
- * @returns The base media decode time in microseconds
- */
-export function decodeTimestamp(segment: Uint8Array, init: InitSegment): Time.Micro {
-	const boxes = readIsoBoxes(toArrayBuffer(segment), { readers: DATA_READERS }) as ParsedIsoBox[];
-
-	// Find moof > traf > tfdt for base media decode time
-	const tfdt = findBox(boxes, isBoxType<TrackFragmentBaseMediaDecodeTimeBox & ParsedIsoBox>("tfdt"));
-	const baseDecodeTime = tfdt?.baseMediaDecodeTime ?? 0;
-
-	// Convert to microseconds
-	return ((baseDecodeTime * 1_000_000) / init.timescale) as Time.Micro;
-}
-
 /**
  * Parse a data segment (moof + mdat) to extract raw samples.
  *
  * Sample duration/size/flags fall back through trun → tfhd → trex (init segment)
  * per ISO/IEC 14496-12 §8.8.7. The init segment's trex defaults are required for
  * fragments where the encoder only set them once in moov (e.g. gstreamer passthrough).
  *
+ * The moq-net `timestamp` is the broadcast timeline: the fragment's earliest sample presents at
+ * it. `tfdt` and the composition offsets only place the samples relative to each other, since a
+ * publisher may move a passthrough track to another timeline without rewriting the payload.
+ * An untimed frame (`undefined`) has no broadcast time, so its samples present at the time
+ * `tfdt` gives them.
+ *
  * @param segment - The moof + mdat data
  * @param init - Parsed init segment (provides timescale and trex defaults)
+ * @param timestamp - The moq-net frame timestamp carrying this segment, if its track is timed
  * @returns Array of decoded samples
  */
-export function decodeDataSegment(segment: Uint8Array, init: InitSegment): Sample[] {
+export function decodeDataSegment(
+	segment: Uint8Array,
+	init: InitSegment,
+	timestamp: Time.Timestamp | undefined,
+): Sample[] {
 	// Cast to ParsedIsoBox[] since the library's return type changes with readers
 	const boxes = readIsoBoxes(toArrayBuffer(segment), { readers: DATA_READERS }) as ParsedIsoBox[];
 
@@ -342,9 +334,12 @@ export function decodeDataSegment(segment: Uint8Array, init: InitSegment): Sampl
 	const defaultSize = tfhd?.defaultSampleSize ?? init.defaultSampleSize;
 	const defaultFlags = tfhd?.defaultSampleFlags ?? init.defaultSampleFlags;
 
-	// Find moof > traf > trun for sample info
-	const trun = findBox(boxes, isBoxType<TrackRunBox & ParsedIsoBox>("trun"));
-	if (!trun) {
+	// Find moof > traf > trun for sample info. A traf may split its samples across several runs,
+	// which continue one decode timeline and one mdat.
+	const traf = findBox(boxes, isBoxType<ParsedIsoBox>("traf"));
+	// biome-ignore lint/suspicious/noExplicitAny: ISO box structure varies
+	const truns = ((traf as any)?.boxes ?? []).filter(isBoxType<TrackRunBox & ParsedIsoBox>("trun"));
+	if (truns.length === 0) {
 		throw new Error("No trun box found in data segment");
 	}
 
@@ -361,70 +356,96 @@ export function decodeDataSegment(segment: Uint8Array, init: InitSegment): Sampl
 		throw new Error("No data in mdat box");
 	}
 
+	// Samples are read from the mdat front to back, so each run must start where the previous one
+	// ended. A run's dataOffset counts from the moof's first byte (CMAF's default-base-is-moof).
+	if (tfhd?.baseDataOffset !== undefined) {
+		throw new Error("tfhd base_data_offset is unsupported: CMAF data offsets count from the moof");
+	}
+	let position = 0;
+	let moofStart: number | undefined;
+	let mdatDataStart: number | undefined;
+	for (const box of boxes) {
+		const size = box.largesize ?? box.size;
+		if (box.type === "moof") moofStart ??= position;
+		if (box === mdat) mdatDataStart = position + size - mdatData.byteLength;
+		position += size;
+	}
+	if (moofStart === undefined || mdatDataStart === undefined || mdatDataStart < moofStart) {
+		throw new Error("mdat must follow the moof");
+	}
+	const dataStart = mdatDataStart - moofStart;
+
 	const samples: Sample[] = [];
+	const ptss: number[] = [];
 
-	// trun.dataOffset is an offset from the base data offset (typically moof start) to the first sample.
-	// For simple CMAF segments where moof is followed immediately by mdat, this equals moof.size + 8.
-	// Since mdat.data is the mdat payload (excluding the 8-byte header), we need to compute the
-	// offset within mdatData. For now, we assume samples start at the beginning of mdat.data
-	// when dataOffset is not specified or when it points to the start of mdat payload.
-	// TODO: For complex cases with base_data_offset in tfhd, this needs additional handling.
 	let dataOffset = 0;
 	let decodeTime = baseDecodeTime;
 
-	for (let i = 0; i < trun.sampleCount; i++) {
-		const sample: TrackRunSample = trun.samples[i] ?? {};
+	for (const trun of truns) {
+		if (trun.dataOffset !== undefined && trun.dataOffset !== dataStart + dataOffset) {
+			throw new Error(
+				`trun data_offset ${trun.dataOffset} doesn't start at the next sample (${dataStart + dataOffset})`,
+			);
+		}
 
-		const sampleSize = sample.sampleSize ?? defaultSize;
-		const sampleDuration = sample.sampleDuration ?? defaultDuration;
+		for (let i = 0; i < trun.sampleCount; i++) {
+			const sample: TrackRunSample = trun.samples[i] ?? {};
 
-		// Validate sample size - must be positive to produce valid data
-		if (sampleSize <= 0) {
-			throw new Error(`Invalid sample size ${sampleSize} for sample ${i} in trun`);
-		}
+			const sampleSize = sample.sampleSize ?? defaultSize;
+			const sampleDuration = sample.sampleDuration ?? defaultDuration;
 
-		// Duration 0 is valid for single-sample CMAF fragments where duration
-		// is implicit. Negative duration would indicate corrupt data.
-		if (sampleDuration < 0) {
-			throw new Error(`Invalid sample duration ${sampleDuration} for sample ${i} in trun`);
-		}
+			// Validate sample size - must be positive to produce valid data
+			if (sampleSize <= 0) {
+				throw new Error(`Invalid sample size ${sampleSize} for sample ${i} in trun`);
+			}
 
-		// Bounds check before slicing to prevent reading past mdat data
-		if (dataOffset + sampleSize > mdatData.length) {
-			throw new Error(
-				`Sample ${i} would overflow mdat: offset=${dataOffset}, size=${sampleSize}, mdatLength=${mdatData.length}`,
-			);
+			// Duration 0 is valid for single-sample CMAF fragments where duration
+			// is implicit. Negative duration would indicate corrupt data.
+			if (sampleDuration < 0) {
+				throw new Error(`Invalid sample duration ${sampleDuration} for sample ${i} in trun`);
+			}
+
+			// Bounds check before slicing to prevent reading past mdat data
+			if (dataOffset + sampleSize > mdatData.length) {
+				throw new Error(
+					`Sample ${i} would overflow mdat: offset=${dataOffset}, size=${sampleSize}, mdatLength=${mdatData.length}`,
+				);
+			}
+
+			const sampleFlags =
+				i === 0 && trun.firstSampleFlags !== undefined
+					? trun.firstSampleFlags
+					: (sample.sampleFlags ?? defaultFlags);
+			const compositionOffset = sample.sampleCompositionTimeOffset ?? 0;
+
+			// Extract sample data
+			const data = new Uint8Array(mdatData.slice(dataOffset, dataOffset + sampleSize));
+			dataOffset += sampleSize;
+
+			ptss.push(decodeTime + compositionOffset);
+			const duration = Math.round((sampleDuration * 1_000_000) / init.timescale);
+
+			// Check if keyframe (sample_is_non_sync_sample flag is bit 16)
+			// If flag is 0, treat as keyframe for safety. Audio never reports one: every
+			// audio sample is a sync sample, and the group start is the consumer's to mark.
+			const keyframe = init.kind === "video" && (sampleFlags === 0 || (sampleFlags & 0x00010000) === 0);
+
+			// Set below, once the earliest presentation time is known.
+			samples.push({ data, timestamp: 0, keyframe, duration });
+
+			decodeTime += sampleDuration;
 		}
+	}
 
-		const sampleFlags =
-			i === 0 && trun.firstSampleFlags !== undefined
-				? trun.firstSampleFlags
-				: (sample.sampleFlags ?? defaultFlags);
-		const compositionOffset = sample.sampleCompositionTimeOffset ?? 0;
-
-		// Extract sample data
-		const data = new Uint8Array(mdatData.slice(dataOffset, dataOffset + sampleSize));
-		dataOffset += sampleSize;
-
-		// Calculate presentation timestamp in microseconds
-		// PTS = (decode_time + composition_offset) * 1_000_000 / timescale
-		const pts = decodeTime + compositionOffset;
-		const timestamp = Math.round((pts * 1_000_000) / init.timescale);
-		const duration = Math.round((sampleDuration * 1_000_000) / init.timescale);
-
-		// Check if keyframe (sample_is_non_sync_sample flag is bit 16)
-		// If flag is 0, treat as keyframe for safety. Audio never reports one: every
-		// audio sample is a sync sample, and the group start is the consumer's to mark.
-		const keyframe = init.kind === "video" && (sampleFlags === 0 || (sampleFlags & 0x00010000) === 0);
-
-		samples.push({
-			data,
-			timestamp,
-			keyframe,
-			duration,
-		});
-
-		decodeTime += sampleDuration;
+	// A loop, not `Math.min(...ptss)`: a long fragment's sample count can exceed the argument limit.
+	let earliest = 0;
+	if (timestamp !== undefined) {
+		earliest = Number.POSITIVE_INFINITY;
+		for (const pts of ptss) earliest = Math.min(earliest, pts);
+	}
+	const anchor = timestamp?.asMicros() ?? 0;
+	for (const [i, sample] of samples.entries()) {
+		sample.timestamp = Math.round(anchor + ((ptss[i] - earliest) * 1_000_000) / init.timescale);
 	}
 
 	return samples;

js/hang/src/container/cmaf/format.test.ts

@@ -0,0 +1,125 @@
+import { expect, test } from "bun:test";
+import { Time } from "@moq/net";
+import type { InitSegment } from "./decode.ts";
+import { Format } from "./format.ts";
+
+const TIMESCALE = 90_000;
+const INIT: InitSegment = {
+	timescale: TIMESCALE,
+	trackId: 1,
+	kind: "video",
+	defaultSampleDuration: 0,
+	defaultSampleSize: 0,
+	defaultSampleFlags: 0,
+};
+
+function box(type: string, ...parts: Uint8Array[]): Uint8Array {
+	const size = 8 + parts.reduce((sum, part) => sum + part.byteLength, 0);
+	const out = new Uint8Array(size);
+	const view = new DataView(out.buffer);
+	view.setUint32(0, size);
+	out.set(new TextEncoder().encode(type), 4);
+	let offset = 8;
+	for (const part of parts) {
+		out.set(part, offset);
+		offset += part.byteLength;
+	}
+	return out;
+}
+
+function u32(...values: number[]): Uint8Array {
+	const out = new Uint8Array(values.length * 4);
+	const view = new DataView(out.buffer);
+	for (const [i, value] of values.entries()) view.setInt32(i * 4, value);
+	return out;
+}
+
+type Run = [duration: number, cts: number][];
+
+/** A moof+mdat at `tfdt` ticks whose runs list one-byte samples in decode order. */
+function fragment(tfdt: number, ...runs: Run[]): Uint8Array {
+	const count = runs.reduce((sum, run) => sum + run.length, 0);
+	const moof = (dataOffset: number) => {
+		let offset = dataOffset;
+		const truns = runs.map((run) => {
+			// Version 1 (signed CTS): data-offset, sample-duration, sample-size, and CTS present.
+			const trun = box("trun", u32(0x01000b01, run.length, offset), ...run.map(([d, cts]) => u32(d, 1, cts)));
+			offset += run.length;
+			return trun;
+		});
+		return box(
+			"moof",
+			box("mfhd", u32(0, 0)),
+			box(
+				"traf",
+				box("tfhd", u32(0x020000, 1)),
+				box("tfdt", u32(0x01000000, Math.floor(tfdt / 2 ** 32), tfdt % 2 ** 32)),
+				...truns,
+			),
+		);
+	};
+	const header = moof(0);
+	return new Uint8Array([...moof(header.byteLength + 8), ...box("mdat", new Uint8Array(count))]);
+}
+
+// The frame timestamp is the broadcast timeline: a passthrough fragment whose `tfdt` still
+// carries its source PTS decodes at the frame timestamp, keeping its B-frame order.
+test("CmafFormat times samples from the frame timestamp", () => {
+	const source = 3_600 * TIMESCALE;
+	// I, P, B in decode order, presenting at +0, +2, and +1 frames.
+	const segment = fragment(source, [
+		[3000, 3000],
+		[3000, 6000],
+		[3000, 0],
+	]);
+
+	const frames = new Format(INIT).decode(segment, Time.Timestamp.fromMillis(10_000));
+	expect(frames.map((f) => f.timestamp)).toEqual([10_000_000, 10_066_667, 10_033_333] as Time.Micro[]);
+});
+
+// A fragment can open on a sample that presents after a later one (open-GOP leading pictures,
+// or a cut mid-GOP). The earliest presentation time is the anchor, not the first sample.
+test("CmafFormat anchors the earliest presentation time", () => {
+	const segment = fragment(0, [
+		[3000, 6000],
+		[3000, 0],
+	]);
+
+	const frames = new Format(INIT).decode(segment, Time.Timestamp.fromMillis(5_000));
+	expect(frames.map((f) => f.timestamp)).toEqual([5_033_333, 5_000_000] as Time.Micro[]);
+});
+
+// Runs continue one decode timeline, so the anchor is the earliest sample across all of them.
+test("CmafFormat anchors the earliest sample across runs", () => {
+	const segment = fragment(0, [[3000, 6000]], [[3000, 0]]);
+
+	const frames = new Format(INIT).decode(segment, at(3000));
+	expect(frames.map((f) => f.timestamp)).toEqual([66_667, 33_333] as Time.Micro[]);
+});
+
+// An untimed track's frames carry no broadcast time, so their samples present at `tfdt`.
+test("CmafFormat times an untimed fragment from its tfdt", () => {
+	const segment = fragment(TIMESCALE, [
+		[3000, 3000],
+		[3000, 6000],
+		[3000, 0],
+	]);
+
+	const frames = new Format(INIT).decode(segment, undefined);
+	expect(frames.map((f) => f.timestamp)).toEqual([1_033_333, 1_100_000, 1_066_667] as Time.Micro[]);
+});
+
+// Samples are read front to back, so a run that skips bytes is refused rather than sliced wrong.
+test("CmafFormat refuses a run that doesn't start at the next sample", () => {
+	const segment = fragment(0, [[3000, 0]], [[3000, 0]]);
+	const view = new DataView(segment.buffer);
+	// The second trun's data_offset follows its type, version/flags, and sample count.
+	const trun = segment.findLastIndex((_, i) => new TextDecoder().decode(segment.subarray(i, i + 4)) === "trun");
+	view.setInt32(trun + 12, view.getInt32(trun + 12) + 1);
+
+	expect(() => new Format(INIT).decode(segment, undefined)).toThrow(/data_offset/);
+});
+
+function at(ticks: number): Time.Timestamp {
+	return new Time.Timestamp(ticks, Time.Timescale(TIMESCALE));
+}