Repository navigation
NVENC H.264: publisher pli_count does not rise on SFU join/resubscribe (L4, 1.1.17/1.1.18) #1428
Description
Summary
On LiveKit Cloud + NVIDIA L4, publishing a 1024×1024 25 fps track with TrackPublishOptions.video_encoder = ENCODER_BACKEND_NVENC and video_codec = H264 (max_bitrate = 5_000_000) does not increment publisher OutboundRtpStreamStats.pli_count within 6 s of deterministic SFU keyframe triggers (new subscriber join, set_subscribed(false/true), simulcast layer switch).
This blocks us from treating NVENC as safe to ship: a prior real-avatar trial (same encoder, livekit-rtc 1.1.17) froze 16–36 s after the first inbound PLI (framesDecoded / bytesReceived flat; Python render loop kept running at 24.6 fps, no exception). We built an isolated repro to capture libwebrtc lines around those PLIs; the encoder stays up, but the PLI counters we were told to use as the trigger proof do not move.
Environment
- GPU: Modal L4 (
nvidia-smimemory.used ~194 MiB at t=0, ~437–444 MiB at t=110) - livekit-rtc 1.1.17 (FFI 0.12.76, rust-sdks
2d9f01a) and 1.1.18 (pip overlay) LIVEKIT_RTC_DEBUG=1;livekitlogger DEBUG; list-handler kept WARN+- Synthetic publisher:
rtc.VideoSource(1024, 1024)RGBA gradient + per-frame noise, 25 fps - Subscribers on a slim
livekit==1.1.18image,auto_subscribe=True, no avatar worker in the room (agent_nameis explicit on our pool) - Probe: https://github.com/splatchat/splatchat/blob/dev/server/modal_nvenc_probe.py (if that path is not yet pushed: the file lives in our tree as
server/modal_nvenc_probe.py)
Matrix (2 runs × 120 s per cell)
| livekit-rtc | simulcast | impl | 1024 held | libwebrtc ERROR | pli Δ @ T∈{20,40,60,80} + 6 s | verdict |
|---|---|---|---|---|---|---|
| 1.1.17 | off | NVIDIA H264 Encoder |
yes | none | missing on ≥1 trigger both runs | NO-TRIGGER |
| 1.1.17 | on | SimulcastEncoderAdapter (NVIDIA H264 Encoder × 3) |
yes (rid=f) | none | rid=f often 3→3 / 5→5 / 6→6 | NO-TRIGGER |
| 1.1.18 | off | NVIDIA H264 Encoder |
yes | none | same class | NO-TRIGGER |
| 1.1.18 | on | adapter × 3 | yes | none | same class | NO-TRIGGER |
Nvidia Decoder is supported. prints on boot. rtc_warn=0 on every run (no Failed EncodeFrame NvEncoder). Subscribers 2 and 3 had strictly increasing bytes_received. Subscriber 1’s only counter discontinuity is the expected reset after set_subscribed(False) @ 40 s / True @ 42 s.
With default simulcast=true, LiveKit still opens three NVENC sessions even though NvidiaH264EncoderImpl::GetEncoderInfo reports supports_simulcast=false (memory +249 MiB vs +243 MiB for a single session).
What we expected
LiveKit SFU keyFrameRequester (livekit/livekit#2197) should PLI the publisher on each new DownTrack / resubscribe / layer switch. We expected pli_count (and usually key_frames_encoded) to rise within ~6 s of T=20 (sub-2 join), T=40 (resubscribe), T=60 (sub-3 join), T=80 (LOW→HIGH when simulcasted).
Related
- rust-sdks fix nvenc SetRate #1297 (SetRate / dangling
encodeConfig) is already in 1.1.17 - rust-sdks CudaContext lifecycle memory fixes #1395 / Video source lifecycle memory fixes #1400 (CUDA context / fail-closed init) are in 1.1.18 — no change to this PLI-stats picture
- rust-sdks adding support to setting preferred encoder on video track publication #1126 (
video_encoderpreferred backend)
We are not claiming the isolated probe reproduced the original freeze. We are claiming we cannot certify the IDR/PLI path from sender stats, so we will not bake NVENC in production/staging until that path is observable or the freeze has a different root cause with logs.
Happy to attach the per-run JSON (impl, stats, rtc_log, gpu_mem, subscriber inbound series).
Activity