From 2fa0670a5eb150ae013529e61f9d32bae1da1ead Mon Sep 17 00:00:00 2001 From: Rhainland Date: Mon, 5 Oct 2026 22:23:40 +0200 Subject: [PATCH 1/4] fix(hls): anchor injected subtitle renditions to the origin's timestamps External subtitles injected as WebVTT renditions on a remote HLS source (#316) appeared early when the origin's media timestamps do not start at the playlist's time 0. An ffmpeg MPEG-TS transcode cut with -copyts -max_delay 5000000 puts the video 10 s after source time, and every cue showed 10 s early; direct play was in sync. The served .vtt was whole-program cues in source time with no X-TIMESTAMP-MAP, so AVPlayer pinned cue time 0 to MPEG-TS timestamp 0 while it placed the A/V on the playlist timeline from the segments' own timestamps. After building the rewritten master, the proxy now reads the head of the segment the load opens on (the one AVPlayer fetches first, so an on-demand transcoder is not sent to another position): the first video PES PTS for MPEG-TS, tfdt over the track's mdhd timescale for fMP4. The anchor is that timestamp minus the segment's playlist start, and the renditions are served with X-TIMESTAMP-MAP=MPEGTS:,LOCAL:00:00:00.000 and unchanged cue times. The probe runs in the background under one deadline that covers all of its reads (the segment head and, for fMP4, the init segment), through the authorizing relay when the load has one, and the .vtt handler waits for it inside its existing wait. An anchor near zero, a byte-range or encrypted playlist, an unresolvable EXT-X-DEFINE reference, or a failed probe keeps the plain WebVTT. Tests cover a +10 s MPEG-TS origin (MPEGTS:900000 with unchanged cues), a zero-offset origin and a failed probe (plain body), start-segment selection with EXT-X-DEFINE substitution, fMP4 tfdt parsing, the negative-anchor map, the authorized probe's headers and range, and an fMP4 probe whose segment arrives late and whose init segment stalls returning within the budget. --- CHANGELOG.md | 1 + .../AetherEngine/AetherEngine+Loading.swift | 8 +- .../Native/RemoteHLSSubtitleProvider.swift | 84 ++++- .../Native/RemoteHLSSubtitleProxy.swift | 48 ++- .../Native/RemoteHLSTimestampAnchor.swift | 354 ++++++++++++++++++ .../AetherEngine/Network/HLSOriginRelay.swift | 16 +- .../Subtitles/WebVTTBuilder.swift | 25 +- .../RefreshableHLSAuthorizationTests.swift | 10 +- .../RemoteHLSSubtitleProxyTests.swift | 211 +++++++++++ docs/architecture.md | 1 + docs/formats.md | 1 + 11 files changed, 734 insertions(+), 25 deletions(-) create mode 100644 Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index c91fd1c81..d8155d062 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,7 @@ the public-API contract. ### Fixed +- External subtitles on a remote HLS source no longer appear early when the origin's media timestamps do not start at the playlist's time 0. The WebVTT renditions the engine injects (#316) carried no `X-TIMESTAMP-MAP`, so AVPlayer pinned cue time 0 to timestamp 0, and an ffmpeg MPEG-TS transcode cut with `-copyts -max_delay 5000000` showed every cue 10 s early. The engine now reads the first media timestamp of the segment the load opens on, off the load path, and serves the renditions with a matching `X-TIMESTAMP-MAP`; an origin already on its playlist timeline, or one it cannot read, keeps the plain WebVTT. - The software video decoder no longer runs more than 16 frame threads. It used one per core, and each frame thread holds back one decoded frame, so a 32-core Mac waited for 31 frames before showing the first one after a load or seek (about 3 s at 10 fps). FFmpeg also warns above 16 threads. Hosts with 16 or fewer cores keep their current thread count. - A paused video no longer starts playing by itself. When the player item died while paused (`failedToPlayToEndTime`), the recovery reload bypassed the pause guard and called `play()` on the fresh item. The reload now keeps a pause made before the item died, whether it came through the engine, AVKit, Control Center or PiP, and mounts the item paused at the same position. - A dead item's recovery no longer restarts the title from where the session was first opened. When AVPlayer refused the recovery item's master (`-11868`), the media fallback reloaded at the first mount's start position, so a title opened from its beginning restarted at 0:00. The fallback now reloads where the refused item was placed. Upstream #621. diff --git a/Sources/AetherEngine/AetherEngine+Loading.swift b/Sources/AetherEngine/AetherEngine+Loading.swift index f79aae82e..4d22f4fc4 100644 --- a/Sources/AetherEngine/AetherEngine+Loading.swift +++ b/Sources/AetherEngine/AetherEngine+Loading.swift @@ -499,7 +499,8 @@ extension AetherEngine { // rewritten master, without them the relay stands alone. A required authorization relay // that cannot start throws rather than returning nil. let playbackURL = try await prepareRemoteHLSStandIn( - originURL: url, options: options, expectedGeneration: bypassGeneration) ?? url + originURL: url, options: options, startPosition: startPosition, + expectedGeneration: bypassGeneration) ?? url // With a relay in front, the item AVPlayer fails is a loopback 502 and the refused handshake // happened out of its sight, so the classification has to be able to ask the side that made it. if let relay = remoteHLSSubtitleProxy?.server.relay { @@ -575,6 +576,7 @@ extension AetherEngine { @MainActor private func prepareRemoteHLSStandIn(originURL: URL, options: LoadOptions, + startPosition: Double?, expectedGeneration: UInt64) async throws -> URL? { let requiresAuthorizationRelay = options.httpRequestAuthorization != nil let mayNeedRelay = EngineTLS.serverTrustEvaluator != nil @@ -600,7 +602,9 @@ extension AetherEngine { let preparation = Task { await RemoteHLSSubtitleProxy.prepare( originURL: originURL, tracks: tracks, httpHeaders: options.httpHeaders, - needsRelay: needsRelay, httpRequestAuthorization: options.httpRequestAuthorization) + needsRelay: needsRelay, httpRequestAuthorization: options.httpRequestAuthorization, + // Picks the segment the subtitle timestamp probe reads: the one this load opens on. + startPosition: startPosition) } remoteHLSPreparationTask = preparation let result = await withTaskCancellationHandler { diff --git a/Sources/AetherEngine/Native/RemoteHLSSubtitleProvider.swift b/Sources/AetherEngine/Native/RemoteHLSSubtitleProvider.swift index fc54a1f13..065348108 100644 --- a/Sources/AetherEngine/Native/RemoteHLSSubtitleProvider.swift +++ b/Sources/AetherEngine/Native/RemoteHLSSubtitleProvider.swift @@ -32,6 +32,20 @@ final class RemoteHLSSubtitleProvider: HLSSegmentProvider, @unchecked Sendable { private let fillLock = NSLock() private var fillTask: Task? + /// Where cue time 0 sits on the origin's media timestamps (`RemoteHLSTimestampAnchor`). Unprobed + /// and failed probes both serve the plain body; a running probe holds the `.vtt` answer inside the + /// same wait as an unfinished store, because AVPlayer keeps the first answer for the session. + private enum TimestampAnchor { + case unprobed + case probing + case resolved(Double?) + } + /// Guards `timestampAnchor` and `anchorTask`: resolved on the probe's task, read on the server's + /// connection thread, cancelled from the engine's teardown. + private let anchorLock = NSLock() + private var timestampAnchor = TimestampAnchor.unprobed + private var anchorTask: Task? + /// How long a `.vtt` fetch waits for its store to finish. AVPlayer fetches a whole-program VOD /// subtitle segment ONCE and never re-fetches it, so serving early means serving truncated for the /// rest of the session; the fill normally completes long before the rendition is ever selected. @@ -72,7 +86,7 @@ final class RemoteHLSSubtitleProvider: HLSSegmentProvider, @unchecked Sendable { /// machinery the loopback path uses (#266), so a container holding several subtitle streams is read /// once and a single bad stream index cannot blank its siblings. func startFill() { - cancelFill() + cancelDecode() let jobs = Self.fillJobs(tracks: tracks, stores: stores, defaultHeaders: defaultHeaders) guard !jobs.isEmpty else { return } let task = Task.detached(priority: .utility) { [jobs] in @@ -86,7 +100,17 @@ final class RemoteHLSSubtitleProvider: HLSSegmentProvider, @unchecked Sendable { fillLock.unlock() } + /// Teardown: stops the decode and a timestamp probe still in flight. func cancelFill() { + cancelDecode() + anchorLock.lock() + let probe = anchorTask + anchorTask = nil + anchorLock.unlock() + probe?.cancel() + } + + private func cancelDecode() { fillLock.lock() let task = fillTask fillTask = nil @@ -94,6 +118,50 @@ final class RemoteHLSSubtitleProvider: HLSSegmentProvider, @unchecked Sendable { task?.cancel() } + /// Runs `probe` once, off the load path, and anchors every rendition this provider serves to its + /// answer. The proxy starts it right after the build, so the segment it reads is the one the + /// origin is producing for the player's first request anyway. + func startTimestampAnchorProbe(_ probe: @escaping @Sendable () async -> Double?) { + anchorLock.lock() + timestampAnchor = .probing + anchorLock.unlock() + let task = Task.detached(priority: .utility) { [weak self] in + let anchor = await probe() + self?.resolveTimestampAnchor(anchor) + } + anchorLock.lock() + anchorTask = task + anchorLock.unlock() + } + + private func resolveTimestampAnchor(_ anchor: Double?) { + anchorLock.lock() + timestampAnchor = .resolved(anchor) + anchorLock.unlock() + } + + /// Wait for the started probe, the `awaitFill` of the anchor: for tests, not for the session. + func awaitTimestampAnchor() async { + await currentAnchorTask()?.value + } + + private func currentAnchorTask() -> Task? { + anchorLock.lock() + defer { anchorLock.unlock() } + return anchorTask + } + + /// Nil while unprobed, still probing, or when the probe had no answer: all mean the plain body. + private var anchorSnapshot: (pending: Bool, seconds: Double?) { + anchorLock.lock() + defer { anchorLock.unlock() } + switch timestampAnchor { + case .unprobed: return (false, nil) + case .probing: return (true, nil) + case .resolved(let seconds): return (false, seconds) + } + } + /// Wait for the started fill to finish. Nothing in the session waits for it, the `.vtt` handler /// polls the store instead; this exists so a test can assert on a finished store without racing /// the wall clock. `nativeSubtitleVTT`'s wait is a budget, and a budget loses under a saturated @@ -167,15 +235,25 @@ final class RemoteHLSSubtitleProvider: HLSSegmentProvider, @unchecked Sendable { /// Whole-program WebVTT on the origin item's timeline. The host can declare /// an upstream reanchor offset; unfinished stores must not be served because /// AVPlayer caches this response for the rest of the session. + /// + /// The cues stay in source time; an `X-TIMESTAMP-MAP` ties cue time 0 to where the origin's media + /// timestamps put it. A probe that has not answered inside the wait serves the plain body, which + /// is what this returned before the anchor existed. func nativeSubtitleVTT(ordinal: Int, segmentIndex: Int) -> NativeSubtitleVTTResponse { guard tracks.indices.contains(ordinal), segmentIndex == 0 else { return .missing } let store = stores[ordinal] let deadline = Date().addingTimeInterval(vttFillWaitSeconds) - while !store.isFinished, Date() < deadline { + while !store.isFinished || anchorSnapshot.pending, Date() < deadline { usleep(100_000) } guard store.isFinished else { return .pending } + let anchor = anchorSnapshot + if anchor.pending { + EngineLog.emit( + "[AetherEngine] #316: subs_\(ordinal) served before the timestamp probe answered, " + + "cue time 0 stays at timestamp 0", category: .engine) + } let cues = store.allCues() - return .ready(WebVTTBuilder.body(cues: cues)) + return .ready(WebVTTBuilder.body(cues: cues, timestampAnchorSeconds: anchor.seconds)) } } diff --git a/Sources/AetherEngine/Native/RemoteHLSSubtitleProxy.swift b/Sources/AetherEngine/Native/RemoteHLSSubtitleProxy.swift index 16e649de4..e1fcd6c85 100644 --- a/Sources/AetherEngine/Native/RemoteHLSSubtitleProxy.swift +++ b/Sources/AetherEngine/Native/RemoteHLSSubtitleProxy.swift @@ -20,7 +20,9 @@ import Foundation /// one variant for the program duration and the VOD verdict), a rewrite, a socket. Anything that does /// not line up, and the sidecars stay overlay-only exactly as before. A load is never failed over a /// subtitle feature. A refusal with a relay wanted still stands the relay up on its own, because the -/// media has nowhere else to go. +/// media has nowhere else to go. The one read that is not a playlist, the head of a media segment for +/// the renditions' `X-TIMESTAMP-MAP` (`RemoteHLSTimestampAnchor`), runs after the build, off the load +/// path. enum RemoteHLSSubtitleProxy { /// A standing stand-in: the caller plays `masterURL` and owns the teardown. @@ -56,7 +58,8 @@ enum RemoteHLSSubtitleProxy { tracks: [RemoteHLSSubtitleProvider.Track], httpHeaders: [String: String], needsRelay: Bool, - httpRequestAuthorization: HTTPRequestAuthorization? = nil) async -> Prepared? { + httpRequestAuthorization: HTTPRequestAuthorization? = nil, + startPosition: Double? = nil) async -> Prepared? { let needsRelay = needsRelay || httpRequestAuthorization != nil guard !Task.isCancelled else { return nil } guard !tracks.isEmpty || needsRelay else { return nil } @@ -64,7 +67,8 @@ enum RemoteHLSSubtitleProxy { do { let prepared = try await build( originURL: originURL, tracks: tracks, httpHeaders: httpHeaders, - needsRelay: needsRelay, httpRequestAuthorization: httpRequestAuthorization) + needsRelay: needsRelay, httpRequestAuthorization: httpRequestAuthorization, + startPosition: startPosition) EngineLog.emit( "[AetherEngine] #316: serving \(tracks.count) external subtitle rendition(s) over a " + "rewritten master at \(prepared.masterURL.absoluteString), media " @@ -126,7 +130,8 @@ enum RemoteHLSSubtitleProxy { tracks: [RemoteHLSSubtitleProvider.Track], httpHeaders: [String: String], needsRelay: Bool, - httpRequestAuthorization: HTTPRequestAuthorization?) async throws -> Prepared { + httpRequestAuthorization: HTTPRequestAuthorization?, + startPosition: Double?) async throws -> Prepared { let session = makeSession() defer { session.finishTasksAndInvalidate() } let preflightRelay = httpRequestAuthorization.map { @@ -138,8 +143,9 @@ enum RemoteHLSSubtitleProxy { let (body, finalURL) = try await fetchPlaylist(originURL, session: session, headers: httpHeaders, relay: preflightRelay) let parsed = try parse(body, at: finalURL) - let duration = try await programDuration(of: parsed, at: finalURL, - session: session, headers: httpHeaders, relay: preflightRelay) + let variant = try await mediaPlaylist(of: parsed, body: body, at: finalURL, + session: session, headers: httpHeaders, relay: preflightRelay) + let duration = variant.media.segments.reduce(0) { $0 + $1.duration } try Task.checkCancellation() let master = try RemoteHLSMasterRewrite.rewrite( @@ -176,6 +182,16 @@ enum RemoteHLSSubtitleProxy { // Decode up front: the rendition is fetched the moment the host selects it, and a whole-program // .vtt is fetched once and never again. provider.startFill() + // Anchor the renditions to the origin's media timestamps. Started now and not awaited: the + // segment it reads is the one the origin is producing for the player's first request, and the + // `.vtt` handler holds its answer for the probe the way it holds it for the decode. + if let target = RemoteHLSTimestampAnchor.target(mediaPlaylistBody: variant.body, media: variant.media, + at: variant.url, startPosition: startPosition) { + provider.startTimestampAnchorProbe { + await RemoteHLSTimestampAnchor.probe(target, headers: httpHeaders, + authorization: httpRequestAuthorization) + } + } return Prepared(server: server, provider: provider, masterURL: masterURL) } @@ -223,17 +239,19 @@ enum RemoteHLSSubtitleProxy { } } - /// Sum of the origin's own EXTINFs. A master is resolved through its first variant; the durations - /// are identical across variants, and one small GET buys both the length and the VOD verdict. - private static func programDuration(of playlist: HLSPlaylist, - at url: URL, - session: URLSession, - headers: [String: String], - relay: HLSOriginRelay?) async throws -> Double { + /// The media playlist the program duration (the sum of its EXTINFs) and the timestamp anchor are + /// read from. A master is resolved through its first variant; the durations are identical across + /// variants, and one small GET buys the length, the VOD verdict and the segment list. + private static func mediaPlaylist(of playlist: HLSPlaylist, + body: String, + at url: URL, + session: URLSession, + headers: [String: String], + relay: HLSOriginRelay?) async throws -> (media: HLSMediaPlaylist, body: String, url: URL) { switch playlist { case .media(let media): guard media.hasEndList else { throw Refusal.notVOD } - return media.segments.reduce(0) { $0 + $1.duration } + return (media, body, url) case .master(let master): guard let variant = master.variants.first, let variantURL = HLSPlaylistParser.resolve(uri: variant.uri, against: url) else { @@ -244,7 +262,7 @@ enum RemoteHLSSubtitleProxy { throw Refusal.unusablePlaylist("variant is not a media playlist") } guard media.hasEndList else { throw Refusal.notVOD } - return media.segments.reduce(0) { $0 + $1.duration } + return (media, body, finalURL) } } } diff --git a/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift b/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift new file mode 100644 index 000000000..de71c0fb9 --- /dev/null +++ b/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift @@ -0,0 +1,354 @@ +import Foundation + +/// Where the origin's media timestamps sit against its own playlist timeline, so the whole-program WebVTT +/// the #316 proxy injects lands on the picture. +/// +/// A WebVTT segment without `X-TIMESTAMP-MAP` maps cue time 0 to MPEG-2 timestamp 0 (RFC 8216 §3.5), while +/// AVPlayer places the A/V on the playlist timeline from the segments' own timestamps. An origin whose +/// media does not start at timestamp 0 therefore shows every injected cue early by exactly that offset. +/// Measured on an ffmpeg MPEG-TS transcode cut with `-copyts -max_delay 5000000`: the muxer adds twice +/// `max_delay`, the video sits 10 s after source time, and the sidecar ran 10 s ahead of the speech. +/// Default muxer settings still leave 1.4 s. +/// +/// The anchor is the first video timestamp of one media segment minus that segment's start on the +/// playlist timeline: the PES PTS for MPEG-TS, `tfdt` over the track's `mdhd` timescale for fMP4. It is +/// read once per session, off the load path, from the head of a single segment. +enum RemoteHLSTimestampAnchor { + + /// The one segment the probe reads, and where it sits on the playlist timeline. + struct Target: Equatable, Sendable { + let segmentURL: URL + /// The `EXT-X-MAP` in force for that segment; nil for MPEG-TS. + let initURL: URL? + let segmentStart: Double + } + + /// Anchors closer to zero than this keep the plain body. Well under what a viewer can see on a + /// subtitle, and it absorbs a first frame's composition offset in an fMP4 origin that already + /// carries source time. + static let negligibleSeconds = 0.1 + + /// Whole-probe budget: one deadline across every request the probe makes (the segment head and, + /// for fMP4, its init segment), on both the relay and the plain session path. Started when the + /// proxy is built, so it covers the origin producing the segment AVPlayer is about to ask for + /// anyway; it stays inside `RemoteHLSSubtitleProvider`'s `.vtt` wait so an answer is in before the + /// once-per-session rendition fetch gives up on it. + static let probeBudgetSeconds: TimeInterval = 20 + + /// Bytes read from the head of the media segment: PAT, PMT and the first video PES, or the `moof`. + static let segmentHeadBytes = 128 * 1024 + /// Ceiling for an fMP4 init segment, which carries the track timescales. + static let initSegmentBytes = 256 * 1024 + + // MARK: - Which segment + + /// The segment that holds `startPosition`, which is the one AVPlayer fetches first for this load. + /// That choice matters for an origin that transcodes on demand: the segment is being produced for + /// the player anyway, where asking for any other one (the first, say, on a resumed load) can make + /// such an origin restart its transcode there. + /// + /// Nil when the playlist gives no single trustworthy segment: a byte-range playlist (the URI then + /// names the whole file, not the segment), an encrypted segment, or a URI with a variable this + /// cannot resolve. + static func target(mediaPlaylistBody: String, media: HLSMediaPlaylist, at url: URL, + startPosition: Double?) -> Target? { + guard !media.segments.isEmpty, !media.hasUnsupportedEncryption, + !mediaPlaylistBody.contains("#EXT-X-BYTERANGE") else { return nil } + let position = max(0, startPosition ?? 0) + var index = 0 + var start = 0.0 + var elapsed = 0.0 + for (offset, segment) in media.segments.enumerated() { + if elapsed > position { break } + index = offset + start = elapsed + elapsed += segment.duration + } + let segment = media.segments[index] + guard segment.crypt == nil else { return nil } + + // The parser keeps neither EXT-X-DEFINE nor the EXT-X-MAP URI, so both come off the same lines + // it reads, in the same order (a URI line is every non-tag line). + let lines = mediaPlaylistBody + .split(separator: "\n", omittingEmptySubsequences: true) + .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) } + .filter { !$0.isEmpty } + let queryItems = URLComponents(url: url, resolvingAgainstBaseURL: false)?.queryItems ?? [] + var variables: [String: String] = [:] + var map: String? + var mapForSegment: String? + var uriLines = 0 + for line in lines { + if line.hasPrefix("#EXT-X-DEFINE:") { + if let name = HLSPlaylistParser.attribute("NAME", in: line), + let value = HLSPlaylistParser.attribute("VALUE", in: line) { + variables[name] = value + } else if let name = HLSPlaylistParser.attribute("QUERYPARAM", in: line), + let value = queryItems.first(where: { $0.name == name })?.value { + variables[name] = value + } + } else if line.hasPrefix("#EXT-X-MAP:") { + map = HLSPlaylistParser.attribute("URI", in: line) + } else if !line.hasPrefix("#") { + if uriLines == index { mapForSegment = map; break } + uriLines += 1 + } + } + guard let segmentURL = resolve(segment.uri, variables: variables, against: url) else { return nil } + var initURL: URL? + if media.hasMap { + guard let mapURI = mapForSegment, + let resolved = resolve(mapURI, variables: variables, against: url) else { return nil } + initURL = resolved + } + return Target(segmentURL: segmentURL, initURL: initURL, segmentStart: start) + } + + /// `{$name}` substitution (RFC 8216bis §4.3), then the usual relative resolution. A reference left + /// unresolved (an IMPORT from the master, a missing query parameter) yields nil rather than a URL + /// the origin would refuse. + private static func resolve(_ uri: String, variables: [String: String], against url: URL) -> URL? { + var substituted = uri + for (name, value) in variables { + substituted = substituted.replacingOccurrences(of: "{$\(name)}", with: value) + } + guard !substituted.contains("{$") else { return nil } + return HLSPlaylistParser.resolve(uri: substituted, against: url) + } + + // MARK: - Probe + + /// Reads the target's head (and its init segment for fMP4) and returns the anchor, or nil when it + /// is negligible or cannot be read. Nil means the plain WebVTT body, which is what the renditions + /// served before the anchor existed. A refreshable authorizer goes through a relay of its own, with + /// the same redirect and credential policy as the playlist preflight; anything else uses a session + /// on `EngineTLS`'s delegate, like the playlist reads. `budget` bounds the whole probe, not each + /// request: a URL session's timeouts apply per task, so two sequential reads against a stalled + /// origin would otherwise take twice as long. + static func probe(_ target: Target, headers: [String: String], + authorization: HTTPRequestAuthorization?, + budget: TimeInterval = probeBudgetSeconds) async -> Double? { + let deadline = Date().addingTimeInterval(budget) + let relay = authorization.map { + HLSOriginRelay(authorization: $0, authorizationTimeout: budget, resourceTimeout: budget, + deadline: deadline) + } + defer { relay?.stop() } + let session = relay == nil ? makeSession(timeout: budget) : nil + defer { session?.invalidateAndCancel() } + + @Sendable func head(_ url: URL, _ maximumBytes: Int) async throws -> Data { + if let relay { return try await relay.fetchHead(url, headers: headers, maximumBytes: maximumBytes) } + guard let session else { throw URLError(.cancelled) } + return try await fetchHead(url, headers: headers, maximumBytes: maximumBytes, session: session) + } + + do { + let (segment, initSegment) = try await beforeDeadline(deadline) { + let segment = try await head(target.segmentURL, segmentHeadBytes) + guard let initURL = target.initURL, LiveSegmentFormat.classify(segment) != .mpegts else { + return (segment, nil as Data?) + } + return (segment, try await head(initURL, initSegmentBytes)) + } + guard let anchor = anchorSeconds(segmentHead: segment, initSegment: initSegment, + segmentStart: target.segmentStart) else { + EngineLog.emit( + "[AetherEngine] #316: no media timestamp in the head of \(target.segmentURL.lastPathComponent), " + + "subtitle renditions keep cue time 0 at timestamp 0", category: .engine) + return nil + } + EngineLog.emit( + "[AetherEngine] #316: origin media sits \(String(format: "%.3f", anchor)) s from its playlist " + + "timeline (\(target.segmentURL.lastPathComponent) at \(String(format: "%.3f", target.segmentStart)) s)", + category: .engine) + return abs(anchor) < negligibleSeconds ? nil : anchor + } catch { + if !Task.isCancelled { + EngineLog.emit( + "[AetherEngine] #316: timestamp probe of \(target.segmentURL.lastPathComponent) failed (\(error)), " + + "subtitle renditions keep cue time 0 at timestamp 0", category: .engine) + } + return nil + } + } + + /// Runs `work` until `deadline`, then cancels it and throws `URLError(.timedOut)`. Cancelling the + /// task cancels the URL session task it is awaiting, and the relay stops on cancellation, so the + /// probe returns at the deadline whichever request is outstanding. + private static func beforeDeadline( + _ deadline: Date, _ work: @escaping @Sendable () async throws -> T + ) async throws -> T { + try await withThrowingTaskGroup(of: T?.self) { group in + group.addTask { try await work() } + group.addTask { + let remaining = deadline.timeIntervalSinceNow + if remaining > 0 { try await Task.sleep(nanoseconds: UInt64(remaining * 1_000_000_000)) } + return nil + } + defer { group.cancelAll() } + guard let first = try await group.next(), let value = first else { throw URLError(.timedOut) } + return value + } + } + + private static func makeSession(timeout: TimeInterval) -> URLSession { + let config = URLSessionConfiguration.ephemeral + config.timeoutIntervalForRequest = timeout + config.timeoutIntervalForResource = timeout + config.requestCachePolicy = .reloadIgnoringLocalCacheData + return URLSession(configuration: config, delegate: EngineTLS.sessionDelegate, delegateQueue: nil) + } + + /// Streams at most `maximumBytes` and abandons the rest, so an origin that ignores the range costs + /// the same as one that honours it. + private static func fetchHead(_ url: URL, headers: [String: String], maximumBytes: Int, + session: URLSession) async throws -> Data { + var request = URLRequest(url: url) + for (field, value) in headers { request.setValue(value, forHTTPHeaderField: field) } + request.setValue("bytes=0-\(maximumBytes - 1)", forHTTPHeaderField: "Range") + let (bytes, response) = try await session.bytes(for: request) + defer { bytes.task.cancel() } + if let http = response as? HTTPURLResponse, !(200..<300).contains(http.statusCode) { + throw URLError(.badServerResponse) + } + var data = Data() + data.reserveCapacity(min(maximumBytes, 64 * 1024)) + for try await byte in bytes { + data.append(byte) + if data.count >= maximumBytes { break } + } + return data + } + + // MARK: - Parsing + + /// First media timestamp of the segment, in seconds, minus the segment's playlist start. + static func anchorSeconds(segmentHead: Data, initSegment: Data?, segmentStart: Double) -> Double? { + if let pts = firstPESTimestamp90k(in: segmentHead) { + return Double(pts) / 90_000 - segmentStart + } + if let initSegment, let decodeTime = fragmentDecodeTimeSeconds(initSegment: initSegment, + fragment: segmentHead) { + return decodeTime - segmentStart + } + return nil + } + + /// PTS of the first video PES (stream_id 0xE0-0xEF) in an MPEG-TS head, or of the first audio PES + /// (0xC0-0xDF) when no video PES starts within it. Keyed on the PES stream_id, so no PMT is needed. + static func firstPESTimestamp90k(in data: Data) -> Int64? { + let packetSize = 188 + let bytes = [UInt8](data) + var audio: Int64? + var packet = 0 + while packet + packetSize <= bytes.count { + guard bytes[packet] == 0x47 else { return audio } // lost packet alignment + defer { packet += packetSize } + let end = packet + packetSize + let adaptationControl = (bytes[packet + 3] >> 4) & 0x03 + guard bytes[packet + 1] & 0x40 != 0, // payload_unit_start + adaptationControl == 1 || adaptationControl == 3 else { continue } + var pes = packet + 4 + if adaptationControl == 3 { pes += 1 + Int(bytes[pes]) } + guard pes + 14 <= end, + bytes[pes] == 0, bytes[pes + 1] == 0, bytes[pes + 2] == 1, + bytes[pes + 6] & 0xC0 == 0x80, // '10' marker of the optional PES header + bytes[pes + 7] & 0x80 != 0 else { continue } // PTS present + let p = pes + 9 + let pts = (Int64(bytes[p] >> 1) & 0x07) << 30 + | Int64(bytes[p + 1]) << 22 + | Int64(bytes[p + 2] >> 1) << 15 + | Int64(bytes[p + 3]) << 7 + | Int64(bytes[p + 4] >> 1) + switch bytes[pes + 3] { + case 0xE0...0xEF: return pts + case 0xC0...0xDF where audio == nil: audio = pts + default: break + } + } + return audio + } + + /// `tfdt` of the fragment's video track (or its first track) over that track's `mdhd` timescale. + static func fragmentDecodeTimeSeconds(initSegment: Data, fragment: Data) -> Double? { + let initBytes = [UInt8](initSegment) + var tracks: [UInt32: (timescale: UInt32, isVideo: Bool)] = [:] + guard let moov = boxes(initBytes, in: 0.. 0 else { continue } + let hdlr = mdiaBoxes.first(where: { $0.type == "hdlr" }) + let isVideo = hdlr.map { $0.payload.count >= 12 && fourCC(initBytes, $0.payload.lowerBound + 8) == "vide" } + ?? false + tracks[trackID] = (timescale, isVideo) + } + + let bytes = [UInt8](fragment) + guard let moof = boxes(bytes, in: 0..= 8, + let tfdt = trafBoxes.first(where: { $0.type == "tfdt" }), tfdt.payload.count >= 8, + let track = tracks[uint32(bytes, tfhd.payload.lowerBound + 4)] else { continue } + let decodeTime = bytes[tfdt.payload.lowerBound] == 1 && tfdt.payload.count >= 12 + ? uint64(bytes, tfdt.payload.lowerBound + 4) + : UInt64(uint32(bytes, tfdt.payload.lowerBound + 4)) + let seconds = Double(decodeTime) / Double(track.timescale) + if track.isVideo { return seconds } + if first == nil { first = seconds } + } + return first + } + + /// The complete boxes inside `range`; a box running past it (a truncated `mdat`) ends the walk. + private static func boxes(_ bytes: [UInt8], in range: Range) -> [(type: String, payload: Range)] { + var found: [(type: String, payload: Range)] = [] + var offset = range.lowerBound + while offset + 8 <= range.upperBound { + var size = Int(uint32(bytes, offset)) + var header = 8 + if size == 1 { + guard offset + 16 <= range.upperBound else { break } + let large = uint64(bytes, offset + 8) + guard large <= UInt64(Int.max) else { break } + size = Int(large) + header = 16 + } else if size == 0 { + size = range.upperBound - offset + } + guard size >= header, offset + size <= range.upperBound else { break } + found.append((fourCC(bytes, offset + 4), (offset + header)..<(offset + size))) + offset += size + } + return found + } + + /// A version-dependent 32-bit field of a full box: at `v0` for version 0, at `v1` for version 1. + private static func fullBoxField32(_ bytes: [UInt8], _ payload: Range, v0: Int, v1: Int) -> UInt32? { + guard !payload.isEmpty else { return nil } + let at = bytes[payload.lowerBound] == 1 ? v1 : v0 + guard payload.count >= at + 4 else { return nil } + return uint32(bytes, payload.lowerBound + at) + } + + private static func fourCC(_ bytes: [UInt8], _ offset: Int) -> String { + String(decoding: bytes[offset..<(offset + 4)], as: UTF8.self) + } + + private static func uint32(_ bytes: [UInt8], _ offset: Int) -> UInt32 { + bytes[offset..<(offset + 4)].reduce(0) { $0 << 8 | UInt32($1) } + } + + private static func uint64(_ bytes: [UInt8], _ offset: Int) -> UInt64 { + bytes[offset..<(offset + 8)].reduce(0) { $0 << 8 | UInt64($1) } + } +} diff --git a/Sources/AetherEngine/Network/HLSOriginRelay.swift b/Sources/AetherEngine/Network/HLSOriginRelay.swift index 534adabdf..46e023dd7 100644 --- a/Sources/AetherEngine/Network/HLSOriginRelay.swift +++ b/Sources/AetherEngine/Network/HLSOriginRelay.swift @@ -513,16 +513,28 @@ final class HLSOriginRelay: @unchecked Sendable { return body } + /// The first `maximumBytes` of a resource, asked for as a byte range. An origin that ignores the + /// range and announces more than the cap fails the read rather than streaming a whole segment; the + /// caller treats that as no answer. + func fetchHead(_ url: URL, headers: [String: String], maximumBytes: Int) async throws -> Data { + guard Self.originKey(for: url) != nil else { throw URLError(.unsupportedURL) } + guard maximumBytes > 0 else { throw URLError(.dataLengthExceedsMaximum) } + let body = try await fetchWhole(url, headers: headers, maximumBytes: maximumBytes, + range: "bytes=0-\(maximumBytes - 1)").body + try Task.checkCancellation() + return body + } + /// One successful body, read whole. Dispatching the synchronous pump keeps the caller's actor /// and cooperative executor free; cancellation stops the relay, which wakes the pump. private func fetchWhole(_ url: URL, headers: [String: String], - maximumBytes: Int? = nil) async throws -> Fetched { + maximumBytes: Int? = nil, range: String? = nil) async throws -> Fetched { try await withTaskCancellationHandler { try Task.checkCancellation() return try await withCheckedThrowingContinuation { continuation in DispatchQueue.global(qos: .userInitiated).async { [self] in let sink = Sink(head: { _, _, _, _ in false }, body: { _ in false }) - switch fetch(origin: url, headers: headers, range: nil, sink: sink, + switch fetch(origin: url, headers: headers, range: range, sink: sink, forceHold: true, maximumBytes: maximumBytes) { case .held(let fetched) where (200..<300).contains(fetched.status): continuation.resume(returning: fetched) diff --git a/Sources/AetherEngine/Subtitles/WebVTTBuilder.swift b/Sources/AetherEngine/Subtitles/WebVTTBuilder.swift index fa106b36f..e731f138c 100644 --- a/Sources/AetherEngine/Subtitles/WebVTTBuilder.swift +++ b/Sources/AetherEngine/Subtitles/WebVTTBuilder.swift @@ -6,8 +6,17 @@ import Foundation /// in the PiP layer); muxing timed text into the A/V fMP4 is non-conformant for HLS (see #55). enum WebVTTBuilder { /// `WEBVTT` header followed by one cue block per non-empty cue: `HH:MM:SS.mmm --> HH:MM:SS.mmm` + text. - static func body(cues: [(start: Double, end: Double, text: String)]) -> String { - var out = "WEBVTT\n\n" + /// + /// `timestampAnchorSeconds` is where cue time 0 sits on the origin's media timestamps (the remote-HLS + /// proxy's whole-program renditions, #316). Without it the header carries no `X-TIMESTAMP-MAP`, and + /// RFC 8216 then pins cue time 0 to timestamp 0. Cue times are written unchanged either way. + static func body(cues: [(start: Double, end: Double, text: String)], + timestampAnchorSeconds: Double? = nil) -> String { + var out = "WEBVTT\n" + if let anchor = timestampAnchorSeconds { + out += timestampMap(anchorSeconds: anchor) + "\n" + } + out += "\n" for cue in cues { let text = MovTextSampleBuilder.sanitize(cue.text) if text.isEmpty { continue } @@ -40,6 +49,18 @@ enum WebVTTBuilder { return out } + /// `X-TIMESTAMP-MAP` placing cue time 0 at `anchorSeconds` of the origin's media timestamps. A positive + /// anchor is the MPEG-2 value itself, wrapped to its 33 bits. A negative one would need a value below + /// zero, so it moves to the cue side instead (cue time |anchor| at timestamp 0), which says the same + /// thing without leaning on the player's rollover handling. + static func timestampMap(anchorSeconds: Double) -> String { + guard anchorSeconds >= 0 else { + return "X-TIMESTAMP-MAP=MPEGTS:0,LOCAL:\(timestamp(-anchorSeconds))" + } + let ticks = Int64((anchorSeconds * 90_000).rounded()) & ((Int64(1) << 33) - 1) + return "X-TIMESTAMP-MAP=MPEGTS:\(ticks),LOCAL:00:00:00.000" + } + private static func timestamp(_ seconds: Double) -> String { let total = max(0, seconds) let whole = Int(total) diff --git a/Tests/AetherEngineTests/RefreshableHLSAuthorizationTests.swift b/Tests/AetherEngineTests/RefreshableHLSAuthorizationTests.swift index 06800f903..9d533f1c1 100644 --- a/Tests/AetherEngineTests/RefreshableHLSAuthorizationTests.swift +++ b/Tests/AetherEngineTests/RefreshableHLSAuthorizationTests.swift @@ -46,7 +46,15 @@ struct RefreshableHLSAuthorizationTests { })) defer { prepared.tearDown() } #expect(prepared.servesSubtitleRenditions) - #expect(origin.requests.map { $0["authorization"] } == ["Bearer /master.m3u8", "Bearer /redirect", "Bearer /final/media.m3u8"]) + #expect(origin.requests.prefix(3).map { $0["authorization"] } + == ["Bearer /master.m3u8", "Bearer /redirect", "Bearer /final/media.m3u8"]) + // The renditions' timestamp probe runs after the build and reads the head of the segment the + // load opens on, through the same authorizer. + await prepared.provider?.awaitTimestampAnchor() + let probe = origin.requests.dropFirst(3) + #expect(probe.map { $0["path"] } == ["/final/segment.ts"]) + #expect(probe.first?["authorization"] == "Bearer /final/segment.ts") + #expect(probe.first?["range"] == "bytes=0-\(RemoteHLSTimestampAnchor.segmentHeadBytes - 1)") #expect(origin.requests.allSatisfy { $0["x-static-secret"] == nil }) } diff --git a/Tests/AetherEngineTests/RemoteHLSSubtitleProxyTests.swift b/Tests/AetherEngineTests/RemoteHLSSubtitleProxyTests.swift index 1a6f714cb..1fa9d9db9 100644 --- a/Tests/AetherEngineTests/RemoteHLSSubtitleProxyTests.swift +++ b/Tests/AetherEngineTests/RemoteHLSSubtitleProxyTests.swift @@ -194,6 +194,217 @@ struct RemoteHLSSubtitleProxyTests { #expect(ready.body.contains("After reanchor")) } + // MARK: - Timestamp anchor + + /// One TS packet opening a PES on PID 0x100 whose header carries only a PTS. + private static func pesPacket(pts90k: Int64, streamID: UInt8 = 0xE0) -> [UInt8] { + var packet: [UInt8] = [0x47, 0x41, 0x00, 0x10, 0x00, 0x00, 0x01, streamID, 0x00, 0x00, 0x80, 0x80, 0x05] + packet += [UInt8(0x21 | ((pts90k >> 29) & 0x0E)), + UInt8((pts90k >> 22) & 0xFF), + UInt8(((pts90k >> 14) & 0xFE) | 1), + UInt8((pts90k >> 7) & 0xFF), + UInt8(((pts90k << 1) & 0xFE) | 1)] + return packet + [UInt8](repeating: 0xFF, count: 188 - packet.count) + } + + /// A PAT ahead of the PES, as a muxer writes it: a section the probe has to step over. + private static func tsSegment(videoPTS90k: Int64) -> Data { + let pat: [UInt8] = [0x47, 0x40, 0x00, 0x10, 0x00, 0x00, 0xB0, 0x0D, 0x00, 0x01, 0xC1, 0x00, 0x00, + 0x00, 0x01, 0xF0, 0x00] + return Data(pat + [UInt8](repeating: 0xFF, count: 188 - pat.count) + + pesPacket(pts90k: videoPTS90k - 9000, streamID: 0xC0) + + pesPacket(pts90k: videoPTS90k)) + } + + /// A finished origin on disk: a master, a variant of five 6 s segments, and only the segment files + /// passed in, so a probe that reads any other segment fails. The sidecar has a cue at 20 s. + private static func diskOrigin(segments: [Int: Data]) throws -> (dir: URL, master: URL, srt: URL) { + let dir = FileManager.default.temporaryDirectory.appendingPathComponent("ae-anchor-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: dir.appendingPathComponent("v"), withIntermediateDirectories: true) + try "#EXTM3U\n#EXT-X-STREAM-INF:BANDWIDTH=800000\nv/media.m3u8\n" + .write(to: dir.appendingPathComponent("master.m3u8"), atomically: true, encoding: .utf8) + var media = "#EXTM3U\n#EXT-X-TARGETDURATION:6\n#EXT-X-MEDIA-SEQUENCE:0\n#EXT-X-PLAYLIST-TYPE:VOD\n" + for index in 0..<5 { media += "#EXTINF:6.000000,\nseg\(index).ts\n" } + media += "#EXT-X-ENDLIST\n" + try media.write(to: dir.appendingPathComponent("v/media.m3u8"), atomically: true, encoding: .utf8) + for (index, data) in segments { try data.write(to: dir.appendingPathComponent("v/seg\(index).ts")) } + let srt = dir.appendingPathComponent("en.srt") + try "1\n00:00:20,000 --> 00:00:22,000\nAnchored line\n\n".write(to: srt, atomically: true, encoding: .utf8) + return (dir, dir.appendingPathComponent("master.m3u8"), srt) + } + + /// Prepares the proxy over a disk origin opened at 13 s (segment 2, which starts at 12 s), waits for + /// the decode and the probe, and returns the served rendition. + private static func servedVTT(segments: [Int: Data]) async throws -> String { + let origin = try diskOrigin(segments: segments) + defer { try? FileManager.default.removeItem(at: origin.dir) } + let prepared = try #require(await RemoteHLSSubtitleProxy.prepare( + originURL: origin.master, tracks: [Self.track(100_000, url: origin.srt)], httpHeaders: [:], + needsRelay: false, startPosition: 13)) + defer { prepared.tearDown() } + let provider = try #require(prepared.provider) + await provider.awaitFill() + await provider.awaitTimestampAnchor() + let (status, body) = try await Self.get("/\(prepared.server.pathToken)/subs_0_0.vtt", + port: prepared.server.port) + #expect(status == 200) + #expect(body.contains("Anchored line")) + // Cue times stay in source time; only the header moves them. + #expect(body.contains("00:00:20.000 --> 00:00:22.000")) + return body + } + + @Test("An origin whose media runs 10 s ahead of its playlist anchors the rendition there", + .timeLimit(.minutes(2))) + func renditionAnchorsToOriginTimestamps() async throws { + // ffmpeg's MPEG-TS muxer with -copyts -max_delay 5000000: source time + 10 s. + let body = try await Self.servedVTT(segments: [2: Self.tsSegment(videoPTS90k: (12 + 10) * 90_000)]) + #expect(body.hasPrefix("WEBVTT\nX-TIMESTAMP-MAP=MPEGTS:900000,LOCAL:00:00:00.000\n\n")) + } + + @Test("An origin whose media sits on its playlist timeline keeps the plain body", .timeLimit(.minutes(2))) + func zeroOffsetKeepsPlainBody() async throws { + let body = try await Self.servedVTT(segments: [2: Self.tsSegment(videoPTS90k: 12 * 90_000)]) + #expect(body.hasPrefix("WEBVTT\n\n")) + #expect(!body.contains("X-TIMESTAMP-MAP")) + } + + @Test("A probe that cannot read the segment keeps the plain body", .timeLimit(.minutes(2))) + func probeFailureKeepsPlainBody() async throws { + // Only segment 0 exists: the probe asks for the segment the load opens on, and that one fails. + let body = try await Self.servedVTT(segments: [0: Self.tsSegment(videoPTS90k: 10 * 90_000)]) + #expect(body.hasPrefix("WEBVTT\n\n")) + #expect(!body.contains("X-TIMESTAMP-MAP")) + } + + @Test("The probe reads the segment the load opens on, with EXT-X-DEFINE variables substituted") + func probeTargetsTheStartSegment() throws { + let body = """ + #EXTM3U + #EXT-X-VERSION:8 + #EXT-X-DEFINE:NAME="q",VALUE="token=abc" + #EXT-X-TARGETDURATION:6 + #EXT-X-MAP:URI="init.mp4?{$q}" + #EXTINF:6.0, + seg_00000.m4s?{$q} + #EXTINF:6.0, + seg_00001.m4s?{$q} + #EXTINF:6.0, + seg_00002.m4s?{$q} + #EXT-X-ENDLIST + """ + let url = try #require(URL(string: "https://origin.test/hls/main.m3u8")) + guard case .media(let media) = try HLSPlaylistParser.parse(body) else { + Issue.record("not a media playlist") + return + } + let resumed = RemoteHLSTimestampAnchor.target(mediaPlaylistBody: body, media: media, at: url, + startPosition: 7.5) + #expect(resumed?.segmentURL.absoluteString == "https://origin.test/hls/seg_00001.m4s?token=abc") + #expect(resumed?.initURL?.absoluteString == "https://origin.test/hls/init.mp4?token=abc") + #expect(resumed?.segmentStart == 6) + let fromStart = RemoteHLSTimestampAnchor.target(mediaPlaylistBody: body, media: media, at: url, + startPosition: nil) + #expect(fromStart?.segmentURL.lastPathComponent == "seg_00000.m4s") + #expect(fromStart?.segmentStart == 0) + } + + private static func box(_ type: String, _ payload: [UInt8]) -> [UInt8] { + let size = UInt32(8 + payload.count) + return [UInt8(size >> 24), UInt8(size >> 16 & 0xFF), UInt8(size >> 8 & 0xFF), UInt8(size & 0xFF)] + + Array(type.utf8) + payload + } + + private static func be32(_ value: UInt32) -> [UInt8] { + [UInt8(value >> 24), UInt8(value >> 16 & 0xFF), UInt8(value >> 8 & 0xFF), UInt8(value & 0xFF)] + } + + private static func trak(id: UInt32, timescale: UInt32, handler: String) -> [UInt8] { + let tkhd = box("tkhd", [0, 0, 0, 0] + be32(0) + be32(0) + be32(id) + [UInt8](repeating: 0, count: 68)) + let mdhd = box("mdhd", [0, 0, 0, 0] + be32(0) + be32(0) + be32(timescale) + be32(0) + [0, 0, 0, 0]) + let hdlr = box("hdlr", [0, 0, 0, 0] + be32(0) + Array(handler.utf8) + [UInt8](repeating: 0, count: 13)) + return box("trak", tkhd + box("mdia", mdhd + hdlr)) + } + + private static func traf(id: UInt32, decodeTime: UInt64) -> [UInt8] { + let tfdt = box("tfdt", [1, 0, 0, 0] + be32(UInt32(decodeTime >> 32)) + be32(UInt32(decodeTime & 0xFFFF_FFFF))) + return box("traf", box("tfhd", [0, 0, 0, 0] + be32(id)) + tfdt) + } + + @Test("fMP4 anchors on the video track's tfdt over its own timescale") + func fragmentAnchorUsesVideoTrack() { + let initSegment = Data(Self.box("ftyp", Array("iso6".utf8) + Self.be32(0)) + + Self.box("moov", Self.trak(id: 2, timescale: 48_000, handler: "soun") + + Self.trak(id: 1, timescale: 12_800, handler: "vide"))) + let fragment = Data(Self.box("styp", Array("msdh".utf8) + Self.be32(0)) + + Self.box("moof", Self.box("mfhd", [0, 0, 0, 0] + Self.be32(3)) + + Self.traf(id: 2, decodeTime: 0) + Self.traf(id: 1, decodeTime: 22 * 12_800)) + + [0x00, 0x10, 0x00, 0x00] + Array("mdat".utf8)) // truncated mdat, as a ranged read leaves it + let anchor = RemoteHLSTimestampAnchor.anchorSeconds(segmentHead: fragment, initSegment: initSegment, + segmentStart: 12) + #expect(anchor == 10) + } + + #if os(macOS) + /// An fMP4 probe makes two sequential reads. The budget covers both: a URL session's timeouts are per + /// task, so a segment that arrives late followed by an init segment that never does used to take the + /// segment's wait plus a whole budget, past the provider's `.vtt` wait. + @Test("The probe budget bounds both fMP4 reads together", .timeLimit(.minutes(1))) + func probeBudgetCoversBothReads() async throws { + let budget: TimeInterval = 4 + let segmentDelay = 2.5 + let script = """ + import http.server, time + + class Handler(http.server.BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + + def log_message(self, *args): + pass + + def do_GET(self): + with open("requests.log", "a") as log: + log.write(self.path + "\\n") + if self.path.startswith("/init.mp4"): + time.sleep(60) + return + time.sleep(\(segmentDelay)) + body = b"\\x00\\x00\\x00\\x10stypmsdh\\x00\\x00\\x00\\x00" + self.send_response(200) + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) + print("READY", server.server_address[1], flush=True) + server.serve_forever() + """ + let launched = try #require(await PythonOrigin.launch(prefix: "aether-anchor-stall", script: script)) + defer { launched.stop() } + let base = "http://127.0.0.1:\(launched.port)" + let target = RemoteHLSTimestampAnchor.Target( + segmentURL: try #require(URL(string: "\(base)/seg_00000.m4s")), + initURL: try #require(URL(string: "\(base)/init.mp4")), segmentStart: 0) + + let started = Date() + let anchor = await RemoteHLSTimestampAnchor.probe(target, headers: [:], authorization: nil, + budget: budget) + let elapsed = Date().timeIntervalSince(started) + + #expect(anchor == nil) + // Per-task timeouts would end this at segmentDelay + budget (6.5 s). + #expect(elapsed < budget + 1.5, "probe took \(elapsed) s against a \(budget) s budget") + let log = try String(contentsOf: launched.workDir.appendingPathComponent("requests.log"), encoding: .utf8) + #expect(log.split(separator: "\n") == ["/seg_00000.m4s", "/init.mp4"]) + } + #endif + + @Test("A negative anchor moves to the cue side instead of wrapping the 33-bit timestamp") + func negativeAnchorUsesLocalTime() { + #expect(WebVTTBuilder.timestampMap(anchorSeconds: 10) == "X-TIMESTAMP-MAP=MPEGTS:900000,LOCAL:00:00:00.000") + #expect(WebVTTBuilder.timestampMap(anchorSeconds: -2.5) == "X-TIMESTAMP-MAP=MPEGTS:0,LOCAL:00:00:02.500") + } + // MARK: - Rendition metadata @Test("Renditions are numbered in subs_{ordinal} order and carry the host's own labels") diff --git a/docs/architecture.md b/docs/architecture.md index 2685ea960..1f0b183af 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -430,6 +430,7 @@ Sources/AetherEngine/ │ ├── RemoteHLSMasterRewrite.swift Declares host-supplied sidecar subtitles as legible renditions on the bypass by rewriting the origin master, since media selection on an HLS asset comes from the playlist and nowhere else (#316) │ ├── RemoteHLSSubtitleProxy.swift Stands a loopback origin in front of that remote master: two playlist GETs, a rewrite and a socket, and anything that does not line up keeps the plain origin URL (#316) │ ├── RemoteHLSSubtitleProvider.swift Serves only what the proxy injected (`/master.m3u8`, `/subs_N.m3u8`, `/subs_N_0.vtt`); the A/V variants still point at the origin, so no media byte moves off it (#316) +│ ├── RemoteHLSTimestampAnchor.swift Where the origin's media timestamps sit against its playlist timeline, read from the head of the segment the load opens on, so the injected WebVTT carries an `X-TIMESTAMP-MAP` instead of landing early by the origin's PTS offset (#316) │ ├── RemoteHLSReadinessDeadline.swift The terminal state the bypass never had: an origin that answers everything while AVFoundation can build no track from what it serves, which no failure signal ever reports (#334) │ ├── RerouteVerdictMemory.swift Bounded, expiring memory of master URLs whose carriage watchdog fired, so a retune after an ingest death relands on the working path instead of re-paying a full mount plus grace (#199) │ ├── SWClockAnchorPolicy.swift Where the SW demux loop anchors the synchronizer clock on the first decoded sample: verbatim for files and resumes (intrinsic A/V lead-in survives), at the sample for a mid-stream-joined source (#107) diff --git a/docs/formats.md b/docs/formats.md index fae44f750..faeb8c5e3 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -316,6 +316,7 @@ External subtitle files register with the engine and appear in `subtitleTracks` - **Selection.** `selectSubtitleTrack(index:)` and `selectSecondarySubtitleTrack(index:)` accept external ids and route onto the whole-file decode internally; `activeSubtitleTrackIndex` publishes the external id like any other selection. `removeExternalSubtitleTrack(id:)` unregisters (an active selection is cleared). - **Renditions.** Load-declared external tracks join the native WebVTT renditions (next section): their store is filled by one whole-file decode at load (one pass per container, covering every track pointing at it) and marked finished, and a finished store also backfills the fullscreen overlay instantly on select (no re-download; styled-ASS selections re-decode to keep raw markup). A store that could not be filled stays unfinished rather than serving a complete but blank `.vtt`. Tracks added after load are host-overlay only until the next load, because the rendition set is fixed in the master playlist at item creation. - **Renditions on the `nativeRemoteHLS` bypass** (#316). That path plays the origin playlist directly and has no loopback master to declare anything in, so a declared sidecar used to be dropped outright. For a VOD source the engine now fetches the origin master, absolutises every variant / audio / key URI against the origin, adds one `EXT-X-MEDIA:TYPE=SUBTITLES` per text sidecar (joining the origin's own subtitles group when it has one, so a variant's group choice cannot lose them) and serves that master from the loopback origin. The media never moves: AVPlayer fetches A/V from the origin exactly as before, which is the property the bypass exists for (E-AC-3 / Atmos passthrough). Renditions are declared `DEFAULT=NO,AUTOSELECT=NO` and without `FORCED`, the same discipline as the loopback master, so nothing self-engages; the track keeps its external id and selecting it drives `AVMediaSelection` rather than the overlay, so it survives PiP / AirPlay / an external display. Live playlists (no `EXT-X-ENDLIST`), bitmap sidecars (`.sup`), an unparseable or unrewritable playlist and an origin slower than the 5 s budget all keep the origin URL and overlay-only subtitles. +- **Timing of the bypass renditions.** The injected `.vtt` is the whole program with cue times in source time. Without `X-TIMESTAMP-MAP`, cue time 0 sits at media timestamp 0, so an origin whose media starts later on its own timestamps would show every cue early by that offset (an ffmpeg MPEG-TS transcode with `-copyts -max_delay 5000000` puts the video 10 s after source time). After building the master, the engine reads the head of the segment the load opens on (the one AVPlayer asks for first, so an on-demand transcoder is not sent elsewhere): the first video PES PTS for MPEG-TS, `tfdt` over the track timescale for fMP4. The anchor is that timestamp minus the segment's playlist start, and the rendition is served with `X-TIMESTAMP-MAP=MPEGTS:,LOCAL:00:00:00.000`. This runs off the load path; the `.vtt` answer waits for it like it waits for the decode. An anchor of about zero, a byte-range or encrypted playlist, or a probe that fails keeps the plain body. - **Preferences.** `preferredSubtitleLanguages` ranks external tracks together with embedded ones. A track added mid-session re-runs the preference and auto-activates on a match, but only while the host has made no explicit subtitle call (select / sidecar / clear) in the session, so a deliberate subtitles-off stays off. ### Track selection by language preference From d57205a6084a25f5447434dd7ab046732223cd6b Mon Sep 17 00:00:00 2001 From: Rhainland Date: Tue, 6 Oct 2026 01:43:24 +0200 Subject: [PATCH 2/4] fix(hls): stop crafted origin box sizes and timestamps from trapping Two inputs from a broken or malicious HLS origin could crash the player while the subtitle timestamp anchor was read or written. The MP4 box walk added a 64-bit extended box size to the current offset. A size of Int.max after any complete box passed the Int.max check and then overflowed on the addition. The walk now compares the size with the bytes left instead, so such a box ends the walk and the rendition keeps the plain WebVTT body. X-TIMESTAMP-MAP converted the anchor's 90 kHz tick count to Int64 before wrapping it to 33 bits. A version-1 tfdt of UInt64.max over a timescale of 1 is finite but far past Int64, and the conversion trapped. The ticks are now wrapped while still a Double. An anchor that is not finite (a playlist whose EXTINFs sum to NaN) is dropped by the probe, and timestampMap answers one with the identity map. --- .../Native/RemoteHLSTimestampAnchor.swift | 25 +++++++----- .../Subtitles/WebVTTBuilder.swift | 7 +++- .../RemoteHLSSubtitleProxyTests.swift | 38 +++++++++++++++++++ 3 files changed, 59 insertions(+), 11 deletions(-) diff --git a/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift b/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift index de71c0fb9..73026de72 100644 --- a/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift +++ b/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift @@ -223,16 +223,21 @@ enum RemoteHLSTimestampAnchor { // MARK: - Parsing - /// First media timestamp of the segment, in seconds, minus the segment's playlist start. + /// First media timestamp of the segment, in seconds, minus the segment's playlist start. Nil when + /// either side is not a finite number (a playlist whose EXTINFs sum to NaN), so the rendition keeps + /// the plain body. static func anchorSeconds(segmentHead: Data, initSegment: Data?, segmentStart: Double) -> Double? { + let timestamp: Double if let pts = firstPESTimestamp90k(in: segmentHead) { - return Double(pts) / 90_000 - segmentStart - } - if let initSegment, let decodeTime = fragmentDecodeTimeSeconds(initSegment: initSegment, - fragment: segmentHead) { - return decodeTime - segmentStart + timestamp = Double(pts) / 90_000 + } else if let initSegment, let decodeTime = fragmentDecodeTimeSeconds(initSegment: initSegment, + fragment: segmentHead) { + timestamp = decodeTime + } else { + return nil } - return nil + let anchor = timestamp - segmentStart + return anchor.isFinite ? anchor : nil } /// PTS of the first video PES (stream_id 0xE0-0xEF) in an MPEG-TS head, or of the first audio PES @@ -309,7 +314,9 @@ enum RemoteHLSTimestampAnchor { return first } - /// The complete boxes inside `range`; a box running past it (a truncated `mdat`) ends the walk. + /// The complete boxes inside `range`; a box running past it (a truncated `mdat`) ends the walk. The + /// size is checked against the bytes left rather than added to the offset: an origin's 64-bit size + /// can be anything up to `Int.max`, and the sum would trap. private static func boxes(_ bytes: [UInt8], in range: Range) -> [(type: String, payload: Range)] { var found: [(type: String, payload: Range)] = [] var offset = range.lowerBound @@ -325,7 +332,7 @@ enum RemoteHLSTimestampAnchor { } else if size == 0 { size = range.upperBound - offset } - guard size >= header, offset + size <= range.upperBound else { break } + guard size >= header, size <= range.upperBound - offset else { break } found.append((fourCC(bytes, offset + 4), (offset + header)..<(offset + size))) offset += size } diff --git a/Sources/AetherEngine/Subtitles/WebVTTBuilder.swift b/Sources/AetherEngine/Subtitles/WebVTTBuilder.swift index e731f138c..ad6a27cbd 100644 --- a/Sources/AetherEngine/Subtitles/WebVTTBuilder.swift +++ b/Sources/AetherEngine/Subtitles/WebVTTBuilder.swift @@ -52,12 +52,15 @@ enum WebVTTBuilder { /// `X-TIMESTAMP-MAP` placing cue time 0 at `anchorSeconds` of the origin's media timestamps. A positive /// anchor is the MPEG-2 value itself, wrapped to its 33 bits. A negative one would need a value below /// zero, so it moves to the cue side instead (cue time |anchor| at timestamp 0), which says the same - /// thing without leaning on the player's rollover handling. + /// thing without leaning on the player's rollover handling. The anchor comes from origin data, so the + /// tick count is wrapped while still a `Double` (a huge `tfdt` would not fit `Int64`), and a value + /// that is not finite gets the identity map, which is what the header-less body means. static func timestampMap(anchorSeconds: Double) -> String { + guard anchorSeconds.isFinite else { return "X-TIMESTAMP-MAP=MPEGTS:0,LOCAL:00:00:00.000" } guard anchorSeconds >= 0 else { return "X-TIMESTAMP-MAP=MPEGTS:0,LOCAL:\(timestamp(-anchorSeconds))" } - let ticks = Int64((anchorSeconds * 90_000).rounded()) & ((Int64(1) << 33) - 1) + let ticks = Int64((anchorSeconds * 90_000).rounded().truncatingRemainder(dividingBy: 8_589_934_592)) return "X-TIMESTAMP-MAP=MPEGTS:\(ticks),LOCAL:00:00:00.000" } diff --git a/Tests/AetherEngineTests/RemoteHLSSubtitleProxyTests.swift b/Tests/AetherEngineTests/RemoteHLSSubtitleProxyTests.swift index 1fa9d9db9..30f295b7a 100644 --- a/Tests/AetherEngineTests/RemoteHLSSubtitleProxyTests.swift +++ b/Tests/AetherEngineTests/RemoteHLSSubtitleProxyTests.swift @@ -405,6 +405,44 @@ struct RemoteHLSSubtitleProxyTests { #expect(WebVTTBuilder.timestampMap(anchorSeconds: -2.5) == "X-TIMESTAMP-MAP=MPEGTS:0,LOCAL:00:00:02.500") } + /// An extended-size box whose 64-bit size is `Int.max`, after a complete box, used to trap on + /// `offset + size` while walking an origin's init segment or fragment. + @Test("An extended box size past the data ends the box walk instead of trapping") + func oversizedExtendedBoxEndsTheWalk() { + let oversized: [UInt8] = Self.be32(1) + Array("moov".utf8) + + Self.be32(UInt32(UInt64(Int.max) >> 32)) + Self.be32(UInt32(UInt64(Int.max) & 0xFFFF_FFFF)) + let ftyp = Self.box("ftyp", Array("iso6".utf8) + Self.be32(0)) + let initSegment = Data(ftyp + Self.box("moov", Self.trak(id: 1, timescale: 12_800, handler: "vide"))) + let fragment = Data(Self.box("moof", Self.box("mfhd", [0, 0, 0, 0] + Self.be32(3)) + + Self.traf(id: 1, decodeTime: 22 * 12_800))) + #expect(RemoteHLSTimestampAnchor.anchorSeconds(segmentHead: fragment, initSegment: Data(ftyp + oversized), + segmentStart: 0) == nil) + #expect(RemoteHLSTimestampAnchor.anchorSeconds(segmentHead: Data(ftyp + oversized), initSegment: initSegment, + segmentStart: 0) == nil) + #expect(RemoteHLSTimestampAnchor.anchorSeconds(segmentHead: fragment, initSegment: initSegment, + segmentStart: 12) == 10) + } + + /// A version-1 `tfdt` of `UInt64.max` over a timescale of 1 is a finite anchor too large for `Int64` + /// ticks; it used to trap building the rendition's header. + @Test("An origin anchor too large for Int64 ticks wraps to 33 bits, and a non-finite one is dropped") + func hugeAnchorWrapsWithoutTrapping() { + let initSegment = Data(Self.box("moov", Self.trak(id: 1, timescale: 1, handler: "vide"))) + let fragment = Data(Self.box("moof", Self.traf(id: 1, decodeTime: .max))) + let anchor = RemoteHLSTimestampAnchor.anchorSeconds(segmentHead: fragment, initSegment: initSegment, + segmentStart: 0) + #expect(anchor == Double(UInt64.max)) + #expect(WebVTTBuilder.timestampMap(anchorSeconds: Double(UInt64.max)) + == "X-TIMESTAMP-MAP=MPEGTS:0,LOCAL:00:00:00.000") // 2^64 * 90000 is a multiple of 2^33 + #expect(WebVTTBuilder.timestampMap(anchorSeconds: 1e15) // 9e19 ticks mod 2^33 + == "X-TIMESTAMP-MAP=MPEGTS:3643277312,LOCAL:00:00:00.000") + for value in [Double.infinity, -.infinity, .nan] { + #expect(WebVTTBuilder.timestampMap(anchorSeconds: value) == "X-TIMESTAMP-MAP=MPEGTS:0,LOCAL:00:00:00.000") + } + #expect(RemoteHLSTimestampAnchor.anchorSeconds(segmentHead: fragment, initSegment: initSegment, + segmentStart: .nan) == nil) + } + // MARK: - Rendition metadata @Test("Renditions are numbered in subs_{ordinal} order and carry the host's own labels") From ae677a57f238381d948dec87ddb0829c393511a5 Mon Sep 17 00:00:00 2001 From: Rhainland Date: Fri, 9 Oct 2026 11:23:43 +0200 Subject: [PATCH 3/4] fix(hls): keep the plain subtitle body when the origin segment starts before its slot A Jellyfin-style origin restarts its transcode at the keyframe before the requested slot (-noaccurate_seek -copyts), so the segment the timestamp probe reads starts up to a keyframe interval before its playlist start and the probe measured a negative anchor. Its media timestamps are still source time, so the mapped rendition showed every cue early by the keyframe gap, and AE#616 then matched lines at a zero offset and left sourceTime ahead of the picture. Only a positive anchor is an offset of the media timestamps (a muxer delay), so negative anchors now keep the plain body and the lead stays AE#616's to measure. The +10 s MPEG-TS case is unchanged. --- CHANGELOG.md | 2 +- .../Native/RemoteHLSTimestampAnchor.swift | 21 +++++++--- .../RemoteHLSSubtitleProxyTests.swift | 40 +++++++++++++++++++ docs/formats.md | 2 +- 4 files changed, 58 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 945872612..59c09f16a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,7 +29,7 @@ the public-API contract. ### Fixed -- External subtitles on a remote HLS source no longer appear early when the origin's media timestamps do not start at the playlist's time 0. The WebVTT renditions the engine injects (#316) carried no `X-TIMESTAMP-MAP`, so AVPlayer pinned cue time 0 to timestamp 0, and an ffmpeg MPEG-TS transcode cut with `-copyts -max_delay 5000000` showed every cue 10 s early. The engine now reads the first media timestamp of the segment the load opens on, off the load path, and serves the renditions with a matching `X-TIMESTAMP-MAP`; an origin already on its playlist timeline, or one it cannot read, keeps the plain WebVTT. +- External subtitles on a remote HLS source no longer appear early when the origin's media timestamps do not start at the playlist's time 0. The WebVTT renditions the engine injects (#316) carried no `X-TIMESTAMP-MAP`, so AVPlayer pinned cue time 0 to timestamp 0, and an ffmpeg MPEG-TS transcode cut with `-copyts -max_delay 5000000` showed every cue 10 s early. The engine now reads the first media timestamp of the segment the load opens on, off the load path, and serves the renditions with a matching `X-TIMESTAMP-MAP`; an origin already on its playlist timeline, one whose segment starts before its playlist slot (a transcode restarted at the previous keyframe, which AE#616 corrects), or one it cannot read, keeps the plain WebVTT. - The software video decoder no longer runs more than 16 frame threads. It used one per core, and each frame thread holds back one decoded frame, so a 32-core Mac waited for 31 frames before showing the first one after a load or seek (about 3 s at 10 fps). FFmpeg also warns above 16 threads. Hosts with 16 or fewer cores keep their current thread count. - TrueHD Atmos rendered to APAC (`LoadOptions.objectAudioRendering`) no longer plays 42.7 ms ahead of the video. The bridge dropped the encoder's 2048 frames of priming and stamped the first content packet on the source position, but AVFoundation presents an APAC packet's audio 2048 frames before its timestamp, so every session ran early, at load and after every seek. The priming packets now stay in the stream and take the source position's timestamp. - Authorized native HLS uses the engine relay from the initial load, without forwarding origin credentials to the loopback asset. Optional subtitle playlist preparation shares the authorizer and has a bounded deadline across redirects and refreshes. diff --git a/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift b/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift index 46461615d..da2727c32 100644 --- a/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift +++ b/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift @@ -23,11 +23,22 @@ enum RemoteHLSTimestampAnchor { let segmentStart: Double } - /// Anchors closer to zero than this keep the plain body. Well under what a viewer can see on a - /// subtitle, and it absorbs a first frame's composition offset in an fMP4 origin that already - /// carries source time. + /// Anchors below this, negative ones included, keep the plain body. Well under what a viewer can + /// see on a subtitle, and it absorbs a first frame's composition offset in an fMP4 origin that + /// already carries source time. static let negligibleSeconds = 0.1 + /// The anchor the renditions are served with, given what the probe measured. Only a positive offset + /// is one of the media timestamps: a muxer delay (ffmpeg MPEG-TS adds 1.4 s by default, 10 s with + /// `-max_delay 5000000`) only ever moves them later. A segment whose media starts before its playlist + /// slot is an origin that restarted its transcode at the keyframe before the slot (Jellyfin: + /// `-noaccurate_seek -copyts`); its media timestamps are still source time, so the plain body + /// already lands on the frames, and AE#616 measures the lead of item time off it. Mapping that gap + /// would show every cue early by it and hide the lead from AE#616. + static func renditionAnchor(measured: Double) -> Double? { + measured < negligibleSeconds ? nil : measured + } + /// Whole-probe budget: one deadline across every request the probe makes (the segment head and, /// for fMP4, its init segment), on both the relay and the plain session path. Started when the /// proxy is built, so it covers the origin producing the segment AVPlayer is about to ask for @@ -119,7 +130,7 @@ enum RemoteHLSTimestampAnchor { // MARK: - Probe /// Reads the target's head (and its init segment for fMP4) and returns the anchor, or nil when it - /// is negligible or cannot be read. Nil means the plain WebVTT body, which is what the renditions + /// is negligible, negative (`renditionAnchor(measured:)`) or cannot be read. Nil means the plain WebVTT body, which is what the renditions /// served before the anchor existed. A refreshable authorizer goes through a relay of its own, with /// the same redirect and credential policy as the playlist preflight; anything else uses a session /// on `EngineTLS`'s delegate, like the playlist reads. `budget` bounds the whole probe, not each @@ -165,7 +176,7 @@ enum RemoteHLSTimestampAnchor { "[AetherEngine] #316: origin media sits \(String(format: "%.3f", anchor)) s from its playlist " + "timeline (\(target.segmentURL.lastPathComponent) at \(String(format: "%.3f", target.segmentStart)) s)", category: .engine) - return abs(anchor) < negligibleSeconds ? nil : anchor + return renditionAnchor(measured: anchor) } catch { if !Task.isCancelled { EngineLog.emit( diff --git a/Tests/AetherEngineTests/RemoteHLSSubtitleProxyTests.swift b/Tests/AetherEngineTests/RemoteHLSSubtitleProxyTests.swift index 2b02e31f0..57aa70d62 100644 --- a/Tests/AetherEngineTests/RemoteHLSSubtitleProxyTests.swift +++ b/Tests/AetherEngineTests/RemoteHLSSubtitleProxyTests.swift @@ -296,6 +296,46 @@ struct RemoteHLSSubtitleProxyTests { #expect(body.hasPrefix("WEBVTT\n\n")) #expect(!body.contains("X-TIMESTAMP-MAP")) } + + /// A Jellyfin-style origin restarts its transcode at the keyframe before the requested slot + /// (`-noaccurate_seek -copyts`): the segment at 12 s starts at media 10.5 s, and media time stays + /// source time. The probe reads -1.5 s, but that is the keyframe gap, not an offset of the media + /// timestamps. The rendition must keep cue time on media time, so the line lands on its frame and + /// AE#616 measures the 1.5 s by which AVPlayer's item time leads the picture. + @Test("A keyframe-restart origin keeps the plain body and leaves the lead to AE#616", .timeLimit(.minutes(2))) + func keyframeRestartLeavesLeadToCueClock() async throws { + let slot = 12.0 + let keyframe = 10.5 + let body = try await Self.servedVTT(segments: [2: Self.tsSegment(videoPTS90k: Int64(keyframe * 90_000))]) + #expect(body.hasPrefix("WEBVTT\n\n")) + #expect(!body.contains("X-TIMESTAMP-MAP")) + + // AVPlayer places the first loaded segment at its slot, so item time = media time + lead, and it + // shows a cue at media time cue + (MPEGTS - LOCAL) of the served map. + let lead = slot - keyframe + let cue = (start: 20.0, text: "Anchored line") + let shownAtItem = cue.start + Self.cueToMediaShift(body) + lead + var clock = RemoteHLSCueClock() + clock.setCues([cue]) + let landing = clock.observe(strings: [], itemTime: slot) + #expect(landing == nil) // the landing delivery teaches nothing + let measured = clock.observe(strings: [cue.text], itemTime: shownAtItem) + let offset = try #require(measured) + // The line shows on its own frame: the frame on screen at item t is source time t - lead. + #expect(abs((shownAtItem - lead) - cue.start) < 0.001) + // AE#616 measures that lead, so sourceTime (item less the offset) is the presented frame's. + #expect(abs(offset - lead) < 0.001) + } + + /// Seconds a served rendition's `X-TIMESTAMP-MAP` adds to cue time to reach media time; 0 without one. + private static func cueToMediaShift(_ body: String) -> Double { + guard let line = body.split(separator: "\n").first(where: { $0.hasPrefix("X-TIMESTAMP-MAP=") }), + let mpegts = line.firstRange(of: "MPEGTS:"), let local = line.firstRange(of: "LOCAL:") else { return 0 } + let ticks = Double(line[mpegts.upperBound...].prefix(while: \.isNumber)) ?? 0 + let parts = line[local.upperBound...].split(separator: ":").compactMap { Double($0) } + let localSeconds = parts.count == 3 ? parts[0] * 3600 + parts[1] * 60 + parts[2] : 0 + return ticks / 90_000 - localSeconds + } #endif @Test("The probe reads the segment the load opens on, with EXT-X-DEFINE variables substituted") diff --git a/docs/formats.md b/docs/formats.md index d4f7f8db6..474cef959 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -336,7 +336,7 @@ External subtitle files register with the engine and appear in `subtitleTracks` - **Selection.** `selectSubtitleTrack(index:)` and `selectSecondarySubtitleTrack(index:)` accept external ids and route onto the whole-file decode internally; `activeSubtitleTrackIndex` publishes the external id like any other selection. `removeExternalSubtitleTrack(id:)` unregisters (an active selection is cleared). - **Renditions.** Load-declared external tracks join the native WebVTT renditions (next section): their store is filled by one whole-file decode at load (one pass per container, covering every track pointing at it) and marked finished, and a finished store also backfills the fullscreen overlay instantly on select (no re-download; styled-ASS selections re-decode to keep raw markup). A store that could not be filled stays unfinished rather than serving a complete but blank `.vtt`. Tracks added after load are host-overlay only until the next load, because the rendition set is fixed in the master playlist at item creation. - **Renditions on the `nativeRemoteHLS` bypass** (#316). That path plays the origin playlist directly and has no loopback master to declare anything in, so a declared sidecar used to be dropped outright. For a VOD source the engine now fetches the origin master, absolutises every variant / audio / key URI against the origin, adds one `EXT-X-MEDIA:TYPE=SUBTITLES` per text sidecar (joining the origin's own subtitles group when it has one, so a variant's group choice cannot lose them) and serves that master from the loopback origin. The media never moves: AVPlayer fetches A/V from the origin exactly as before, which is the property the bypass exists for (E-AC-3 / Atmos passthrough). Renditions are declared `DEFAULT=NO,AUTOSELECT=NO` and without `FORCED`, the same discipline as the loopback master, so nothing self-engages; the track keeps its external id and selecting it drives `AVMediaSelection` rather than the overlay, so it survives PiP / AirPlay / an external display. Live playlists (no `EXT-X-ENDLIST`), bitmap sidecars (`.sup`), an unparseable or unrewritable playlist and an origin slower than the 5 s budget all keep the origin URL and overlay-only subtitles. -- **Timing of the bypass renditions.** The injected `.vtt` is the whole program with cue times in source time. Without `X-TIMESTAMP-MAP`, cue time 0 sits at media timestamp 0, so an origin whose media starts later on its own timestamps would show every cue early by that offset (an ffmpeg MPEG-TS transcode with `-copyts -max_delay 5000000` puts the video 10 s after source time). After building the master, the engine reads the head of the segment the load opens on (the one AVPlayer asks for first, so an on-demand transcoder is not sent elsewhere): the first video PES PTS for MPEG-TS, `tfdt` over the track timescale for fMP4. The anchor is that timestamp minus the segment's playlist start, and the rendition is served with `X-TIMESTAMP-MAP=MPEGTS:,LOCAL:00:00:00.000`. This runs off the load path; the `.vtt` answer waits for it like it waits for the decode. An anchor of about zero, a byte-range or encrypted playlist, or a probe that fails keeps the plain body. +- **Timing of the bypass renditions.** The injected `.vtt` is the whole program with cue times in source time. Without `X-TIMESTAMP-MAP`, cue time 0 sits at media timestamp 0, so an origin whose media starts later on its own timestamps would show every cue early by that offset (an ffmpeg MPEG-TS transcode with `-copyts -max_delay 5000000` puts the video 10 s after source time). After building the master, the engine reads the head of the segment the load opens on (the one AVPlayer asks for first, so an on-demand transcoder is not sent elsewhere): the first video PES PTS for MPEG-TS, `tfdt` over the track timescale for fMP4. The anchor is that timestamp minus the segment's playlist start, and the rendition is served with `X-TIMESTAMP-MAP=MPEGTS:,LOCAL:00:00:00.000`. This runs off the load path; the `.vtt` answer waits for it like it waits for the decode. An anchor of about zero or below it, a byte-range or encrypted playlist, or a probe that fails keeps the plain body. A negative anchor is an origin that restarted its transcode at the keyframe before the slot (Jellyfin, `-noaccurate_seek -copyts`): its media timestamps are still source time, so the plain body is on the frames, and AE#616 measures how far item time leads them. - **Preferences.** `preferredSubtitleLanguages` ranks external tracks together with embedded ones. A track added mid-session re-runs the preference and auto-activates on a match, but only while the host has made no explicit subtitle call (select / sidecar / clear) in the session, so a deliberate subtitles-off stays off. ### Track selection by language preference From 87ddcdb4dafb15718c13628cb1df5c5110101e6e Mon Sep 17 00:00:00 2001 From: Rhainland Date: Sun, 11 Oct 2026 23:35:24 +0200 Subject: [PATCH 4/4] docs(hls): write down where the positive-only subtitle anchor is inexact The probe reads the muxer delay minus the keyframe-restart gap as one number, so keeping only a positive measurement is exact when either is zero. Document the mixed case next to renditionAnchor(measured:), what it does to AE#616, why Silo's own HLS output does not reach it, and the follow-up that would read the delay directly. --- .../Native/RemoteHLSTimestampAnchor.swift | 38 +++++++++++++++---- docs/formats.md | 2 +- 2 files changed, 32 insertions(+), 8 deletions(-) diff --git a/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift b/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift index da2727c32..9a184dbf1 100644 --- a/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift +++ b/Sources/AetherEngine/Native/RemoteHLSTimestampAnchor.swift @@ -28,13 +28,37 @@ enum RemoteHLSTimestampAnchor { /// already carries source time. static let negligibleSeconds = 0.1 - /// The anchor the renditions are served with, given what the probe measured. Only a positive offset - /// is one of the media timestamps: a muxer delay (ffmpeg MPEG-TS adds 1.4 s by default, 10 s with - /// `-max_delay 5000000`) only ever moves them later. A segment whose media starts before its playlist - /// slot is an origin that restarted its transcode at the keyframe before the slot (Jellyfin: - /// `-noaccurate_seek -copyts`); its media timestamps are still source time, so the plain body - /// already lands on the frames, and AE#616 measures the lead of item time off it. Mapping that gap - /// would show every cue early by it and hide the lead from AE#616. + /// The anchor the renditions are served with, given what the probe measured: the measurement when it + /// is positive, otherwise nil (the plain body). + /// + /// On a `-copyts` origin the probe observes one number, `D - G`, but the map has to be `D` alone: + /// - `D` is the muxer delay, constant for the session, and it only moves timestamps later (ffmpeg + /// MPEG-TS adds 1.4 s by default and 10 s with `-max_delay 5000000`; fMP4 adds none). + /// - `G` is the keyframe-restart gap: how far before its playlist slot the first segment of a + /// started or restarted transcode begins, when the origin restarts at the keyframe before the slot + /// (Jellyfin: `-noaccurate_seek -copyts`). Only that first segment carries it. + /// + /// The rule is exact only when `D == 0` or `G == 0`. When `G == 0`, the positive measurement is `D`. + /// When `D == 0`, the media is already on source time, so the plain body lands on the frames and + /// AE#616 measures the lead of item time off it. Mapping a negative measurement instead would show + /// every cue early by `G` and hide that lead from AE#616. + /// + /// When both are nonzero, the result is wrong by an amount AE#616 cannot see, because AE#616 measures + /// item time against these same cues and its reference moves with the map: + /// - `0 < G < D`: the map is `D - G`, every cue shows `G` early, AE#616 measures about 0, and its + /// `sourceTime` also leads by `G`. + /// - `G > D`: the plain body shows every cue `D` early, AE#616 measures `G - D`, and its `sourceTime` + /// is off by `D`. + /// An MPEG-TS origin with a muxer delay that resumes by restarting at the previous keyframe is in + /// this case on every resumed load. Silo's own HLS output is not: it cuts MPEG-TS with + /// `-copyts -max_delay 5000000` (`D` = 10 s), but a re-encode seeks accurately to a segment boundary + /// with keyframes forced on boundaries, and a stream copy starts its playlist at the keyframe it + /// actually begins on, so `G` is 0 in both. + /// + /// A possible fix: `D` is a session constant and `G` affects only the first segment of a job, so + /// probing the segment after the start segment (same job, and AVPlayer requests it next) would read + /// `D` directly. That holds only if the origin forces keyframes on slot boundaries, which a stream + /// copy does not, so it would need a fallback there. static func renditionAnchor(measured: Double) -> Double? { measured < negligibleSeconds ? nil : measured } diff --git a/docs/formats.md b/docs/formats.md index 474cef959..beee587f8 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -336,7 +336,7 @@ External subtitle files register with the engine and appear in `subtitleTracks` - **Selection.** `selectSubtitleTrack(index:)` and `selectSecondarySubtitleTrack(index:)` accept external ids and route onto the whole-file decode internally; `activeSubtitleTrackIndex` publishes the external id like any other selection. `removeExternalSubtitleTrack(id:)` unregisters (an active selection is cleared). - **Renditions.** Load-declared external tracks join the native WebVTT renditions (next section): their store is filled by one whole-file decode at load (one pass per container, covering every track pointing at it) and marked finished, and a finished store also backfills the fullscreen overlay instantly on select (no re-download; styled-ASS selections re-decode to keep raw markup). A store that could not be filled stays unfinished rather than serving a complete but blank `.vtt`. Tracks added after load are host-overlay only until the next load, because the rendition set is fixed in the master playlist at item creation. - **Renditions on the `nativeRemoteHLS` bypass** (#316). That path plays the origin playlist directly and has no loopback master to declare anything in, so a declared sidecar used to be dropped outright. For a VOD source the engine now fetches the origin master, absolutises every variant / audio / key URI against the origin, adds one `EXT-X-MEDIA:TYPE=SUBTITLES` per text sidecar (joining the origin's own subtitles group when it has one, so a variant's group choice cannot lose them) and serves that master from the loopback origin. The media never moves: AVPlayer fetches A/V from the origin exactly as before, which is the property the bypass exists for (E-AC-3 / Atmos passthrough). Renditions are declared `DEFAULT=NO,AUTOSELECT=NO` and without `FORCED`, the same discipline as the loopback master, so nothing self-engages; the track keeps its external id and selecting it drives `AVMediaSelection` rather than the overlay, so it survives PiP / AirPlay / an external display. Live playlists (no `EXT-X-ENDLIST`), bitmap sidecars (`.sup`), an unparseable or unrewritable playlist and an origin slower than the 5 s budget all keep the origin URL and overlay-only subtitles. -- **Timing of the bypass renditions.** The injected `.vtt` is the whole program with cue times in source time. Without `X-TIMESTAMP-MAP`, cue time 0 sits at media timestamp 0, so an origin whose media starts later on its own timestamps would show every cue early by that offset (an ffmpeg MPEG-TS transcode with `-copyts -max_delay 5000000` puts the video 10 s after source time). After building the master, the engine reads the head of the segment the load opens on (the one AVPlayer asks for first, so an on-demand transcoder is not sent elsewhere): the first video PES PTS for MPEG-TS, `tfdt` over the track timescale for fMP4. The anchor is that timestamp minus the segment's playlist start, and the rendition is served with `X-TIMESTAMP-MAP=MPEGTS:,LOCAL:00:00:00.000`. This runs off the load path; the `.vtt` answer waits for it like it waits for the decode. An anchor of about zero or below it, a byte-range or encrypted playlist, or a probe that fails keeps the plain body. A negative anchor is an origin that restarted its transcode at the keyframe before the slot (Jellyfin, `-noaccurate_seek -copyts`): its media timestamps are still source time, so the plain body is on the frames, and AE#616 measures how far item time leads them. +- **Timing of the bypass renditions.** The injected `.vtt` is the whole program with cue times in source time. Without `X-TIMESTAMP-MAP`, cue time 0 sits at media timestamp 0, so an origin whose media starts later on its own timestamps would show every cue early by that offset (an ffmpeg MPEG-TS transcode with `-copyts -max_delay 5000000` puts the video 10 s after source time). After building the master, the engine reads the head of the segment the load opens on (the one AVPlayer asks for first, so an on-demand transcoder is not sent elsewhere): the first video PES PTS for MPEG-TS, `tfdt` over the track timescale for fMP4. The anchor is that timestamp minus the segment's playlist start, and the rendition is served with `X-TIMESTAMP-MAP=MPEGTS:,LOCAL:00:00:00.000`. This runs off the load path; the `.vtt` answer waits for it like it waits for the decode. An anchor of about zero or below it, a byte-range or encrypted playlist, or a probe that fails keeps the plain body. A negative anchor is an origin that restarted its transcode at the keyframe before the slot (Jellyfin, `-noaccurate_seek -copyts`): with no muxer delay its media timestamps are still source time, so the plain body is on the frames, and AE#616 measures how far item time leads them. The probe sees the muxer delay minus that restart gap as one number, so an origin with both (MPEG-TS with a muxer delay, resumed at the previous keyframe) is mapped wrong by one of them; `RemoteHLSTimestampAnchor.renditionAnchor(measured:)` documents the cases. Silo's own HLS output has no restart gap. - **Preferences.** `preferredSubtitleLanguages` ranks external tracks together with embedded ones. A track added mid-session re-runs the preference and auto-activates on a match, but only while the host has made no explicit subtitle call (select / sidecar / clear) in the session, so a deliberate subtitles-off stays off. ### Track selection by language preference