diff --git a/apps/media-server/Dockerfile b/apps/media-server/Dockerfile index 5763347374a..c09234479ff 100644 --- a/apps/media-server/Dockerfile +++ b/apps/media-server/Dockerfile @@ -10,6 +10,7 @@ COPY apps/media-server/package.json apps/media-server/bun.lock ./ RUN bun install --frozen-lockfile --production COPY apps/media-server/src ./src +COPY apps/media-server/capcodec.md ./capcodec.md RUN bun test src/__tests__/lib/audio-quality-policy.test.ts src/__tests__/lib/audio-quality.integration.test.ts \ && bun test src/__tests__/lib/audio-quality-formats.integration.test.ts src/__tests__/lib/audio-quality-benchmark.test.ts \ @@ -18,7 +19,7 @@ RUN bun test src/__tests__/lib/audio-quality-policy.test.ts src/__tests__/lib/au && bun test src/__tests__/lib/media-size.test.ts src/__tests__/lib/media-probe.integration.test.ts \ && bun test src/__tests__/lib/media-audio.integration.test.ts \ && bun test src/__tests__/lib/recording-verification.integration.test.ts src/__tests__/lib/job-manager.test.ts \ - && bun test src/__tests__/lib/media-transfer.test.ts \ + && bun test src/__tests__/lib/media-transfer.test.ts src/__tests__/lib/capcodec-encoder.test.ts \ && bun test src/__tests__/routes/recording-verification.test.ts \ && bun test src/__tests__/routes/video.test.ts \ && bun test src/__tests__/lib/media-routes-real-world.integration.test.ts \ diff --git a/apps/media-server/Dockerfile.standalone b/apps/media-server/Dockerfile.standalone index 00de144faab..cd413a925e6 100644 --- a/apps/media-server/Dockerfile.standalone +++ b/apps/media-server/Dockerfile.standalone @@ -10,6 +10,7 @@ COPY package.json bun.lock ./ RUN bun install --frozen-lockfile --production COPY src ./src +COPY capcodec.md ./capcodec.md ENV PORT=3456 EXPOSE 3456 diff --git a/apps/media-server/capcodec-benchmarks.md b/apps/media-server/capcodec-benchmarks.md new file mode 100644 index 00000000000..e6ae9ecc601 --- /dev/null +++ b/apps/media-server/capcodec-benchmarks.md @@ -0,0 +1,194 @@ +# capcodec vs x264 veryfast + +Measurements from the capcodec bench harness for the binary `CAP_MEDIA_VIDEO_ENCODER=capcodec` runs. This repository's CI does not regenerate them. A speed ratio above 1 means capcodec is faster. A BD-rate below 0 means capcodec uses fewer bits at the same quality. + +The 12-clip speed table is main `00fecf7` (read-ahead, CABAC, intra minIndex), retimed with other processes stopped. The 4-thread column ran capcodec with `--threads 4` and x264 with `-threads 0` (automatic). The 1-thread column set both to 1 thread. `--bframes 0` matches that encoder. Concurrent B-frames are the default after that measurement. Screen clips stay almost entirely P frames, so those speed rows still describe the default. full_motion was retimed with concurrent B-frames: 4-thread wall is 1.35-1.40x medium and 1.44-1.50x live (the same sessions measured the previous encoder at 1.09-1.10x and 1.19-1.21x). The other 11 clips were not retimed. full_motion quality below is the pre-concurrent-B bitstream. + +`rd-next` (`d0cce5e`) and SSIM round 3 (`b72e995`) are not in the binary this flag runs. The media server's `ultrafast` preset maps to capcodec `fast`. Job CRF and preset apply unless `CAPCODEC_CRF` or `CAPCODEC_PRESET` is set. + +| Metric | Result | +|---|---| +| Keyframe-only CPU | 1.06-1.76x at 1 and 4 threads; busy_ui uses 12% more bits in all-intra | +| Peak RSS, 4 threads | 34-308 MiB vs 141-1304 MiB on the speed rerun | +| Output delay | live 2 frames (5 with B-frames), medium 10 (steady 13), zerolatency 0; x264 17-19 frames at 4 threads | +| Startup | 0.9-1.2 ms vs 1.3-1.5 ms on a 64x64 frame | +| Decode CPU of the output | 0.58-1.54 s vs 1.59-2.40 s on the measured clips | +| Binary | 561,312 bytes at `00fecf7`; 609,152 bytes with concurrent B-frames. Minimal x264 build is 1.29 MB | +| Build CPU | about 12 s vs 36 s (minimal x264) | +| Bitrate error at Cap targets | two-pass 4.9% vs 5.5%; one-pass 7.7% vs 5.5% (one-pass rate control is not in this binary) | + +Quality measured 2026-10-09 on main `fff504d`. Speed remeasured the same day on main `00fecf7` (read-ahead, pin7 intra minIndex, CABAC table). Those commits match the `fff504d` bitstreams, so the quality tables still describe this encoder. Pending quality branch `rd-next` `d0cce5e`. All 12 corpus clips. + +How to reproduce (from the `capcodec/` directory in this repository): + +- Quality: `CAPCODEC_BIN_UNCHECKED= python3 bench/bench.py standard --quality-only --encoders capcodec-medium,capcodec-live,capcodec-live-zerolatency,x264-veryfast,x264-veryfast-zerolatency --out ` (1080p, 10 s per clip; code_4k is scaled to 1080p here). capcodec's CRF ladder runs 20-44 so its VMAF range covers x264's CRF 18-33; the overlap column shows how much of the two curves BD-rate compares (it was 15-24% on screen clips with the old 20-36 ladder). +- Speed: `bench/lane.sh start`, then `bench/lane.sh run "python3 bench/speedcmp.py --binary --clips all --modes medium,live,zerolatency --out "`. A fresh run matches each clip to x264 CRF 23 by VMAF, then times 5 shuffled rounds at 1 and 4 threads with other processes stopped. code_4k runs at native 4K for 5 s. The published table reused an earlier CRF match; that results file is not in the repository. +- Tables: `python3 bench/scoreboard.py --quality LABEL= --speed LABEL= --out SCOREBOARD.md`. + +### main fff504d: capcodec-medium vs x264-veryfast (BD-rate, negative = capcodec needs fewer bits) + +| clip | VMAF | PSNR-YUV | SSIM | VMAF range overlap | +|---|---|---|---|---| +| busy_ui | +5.4% lose | +5.1% lose | +14.8% lose | 52% | +| busy_ui_60fps | +2.6% lose | +2.0% lose | +16.8% lose | 74% | +| code_4k | -3.3% win | +1.0% lose | +9.2% lose | 72% | +| code_editor | -0.5% win | +4.9% lose | +7.3% lose | 76% | +| dark_mode | -17.2% win | -16.9% win | -11.9% win | 75% | +| dashboard | +7.9% lose | +12.3% lose | +25.9% lose | 59% | +| full_motion | -12.2% win | -10.0% win | +4.8% lose | 44% | +| idle | -8.8% win | -6.3% win | -3.4% win | 73% | +| slides | -18.0% win | +0.5% lose | -3.6% win | 66% | +| slow_typing | -0.8% win | +4.5% lose | +8.6% lose | 80% | +| text_scroll | -15.5% win | -14.8% win | -14.9% win | 88% | +| webcam_overlay | -20.0% win | -4.3% win | -4.0% win | 69% | + +19 wins, 17 losses across 12 clips x 3 metrics. + +### main fff504d: capcodec-live vs x264-veryfast (BD-rate, negative = capcodec needs fewer bits) + +| clip | VMAF | PSNR-YUV | SSIM | VMAF range overlap | +|---|---|---|---|---| +| busy_ui | +5.7% lose | +5.7% lose | +15.6% lose | 51% | +| busy_ui_60fps | +3.5% lose | +2.9% lose | +18.1% lose | 73% | +| code_4k | +0.2% lose | +5.3% lose | +13.4% lose | 72% | +| code_editor | +2.8% lose | +8.6% lose | +11.2% lose | 76% | +| dark_mode | -16.6% win | -16.2% win | -10.9% win | 75% | +| dashboard | +10.8% lose | +15.7% lose | +29.3% lose | 61% | +| full_motion | -7.8% win | -1.9% win | +12.0% lose | 43% | +| idle | -8.4% win | -6.2% win | -3.0% win | 72% | +| slides | -20.0% win | -1.5% win | -6.4% win | 65% | +| slow_typing | +1.1% lose | +4.8% lose | +9.1% lose | 78% | +| text_scroll | -15.1% win | -14.7% win | -14.8% win | 88% | +| webcam_overlay | -18.7% win | -3.8% win | -1.7% win | 67% | + +17 wins, 19 losses across 12 clips x 3 metrics. + +### main fff504d: capcodec-live-zerolatency vs x264-veryfast-zerolatency (BD-rate, negative = capcodec needs fewer bits) + +| clip | VMAF | PSNR-YUV | SSIM | VMAF range overlap | +|---|---|---|---|---| +| busy_ui | +1.6% lose | +2.3% lose | +6.7% lose | 23% | +| busy_ui_60fps | +2.0% lose | +1.9% lose | +9.2% lose | 30% | +| code_4k | -1.2% win | +2.4% lose | +7.9% lose | 36% | +| code_editor | -0.5% win | +8.0% lose | +5.8% lose | 36% | +| dark_mode | -22.3% win | -21.6% win | -21.3% win | 35% | +| dashboard | +0.2% lose | +2.3% lose | +7.4% lose | 27% | +| full_motion | -3.1% win | -1.0% win | +15.1% lose | 98% | +| idle | -17.2% win | -17.8% win | -15.6% win | 28% | +| slides | -30.8% win | -22.7% win | -20.8% win | 40% | +| slow_typing | -0.8% win | +3.8% lose | +4.8% lose | 37% | +| text_scroll | -16.3% win | -17.6% win | -20.7% win | 33% | +| webcam_overlay | -74.8% win | -64.5% win | -54.8% win | 42% | + +20 wins, 16 losses across 12 clips x 3 metrics. + +### rd-next d0cce5e: capcodec-medium vs x264-veryfast (BD-rate, negative = capcodec needs fewer bits) + +| clip | VMAF | PSNR-YUV | SSIM | VMAF range overlap | +|---|---|---|---|---| +| busy_ui | -17.0% win | -2.4% win | +6.5% lose | 65% | +| busy_ui_60fps | -18.8% win | -3.8% win | +6.4% lose | 42% | +| code_4k | -23.0% win | -1.4% win | +5.0% lose | 57% | +| code_editor | -22.7% win | -3.7% win | +2.5% lose | 57% | +| dark_mode | -35.0% win | -19.9% win | -13.5% win | 58% | +| dashboard | -12.3% win | +3.2% lose | +12.8% lose | 77% | +| full_motion | -21.6% win | -8.9% win | +1.5% lose | 52% | +| idle | -24.0% win | -11.8% win | -2.9% win | 66% | +| slides | -33.6% win | -13.3% win | -10.2% win | 81% | +| slow_typing | -22.4% win | -4.1% win | +2.2% lose | 43% | +| text_scroll | -31.1% win | -19.6% win | -14.3% win | 54% | +| webcam_overlay | -34.1% win | -14.0% win | +3.4% lose | 87% | + +27 wins, 9 losses across 12 clips x 3 metrics. + +### rd-next d0cce5e: capcodec-live vs x264-veryfast (BD-rate, negative = capcodec needs fewer bits) + +| clip | VMAF | PSNR-YUV | SSIM | VMAF range overlap | +|---|---|---|---|---| +| busy_ui | -17.3% win | -2.3% win | +6.5% lose | 64% | +| busy_ui_60fps | -18.9% win | -3.7% win | +6.5% lose | 42% | +| code_4k | -22.7% win | -1.3% win | +5.2% lose | 58% | +| code_editor | -22.1% win | -3.7% win | +2.7% lose | 57% | +| dark_mode | -34.4% win | -19.6% win | -13.0% win | 59% | +| dashboard | -12.4% win | +3.8% lose | +14.0% lose | 73% | +| full_motion | -18.0% win | -2.6% win | +6.8% lose | 51% | +| idle | -23.8% win | -11.7% win | -2.7% win | 67% | +| slides | -35.3% win | -15.6% win | -12.6% win | 83% | +| slow_typing | -21.3% win | -4.1% win | +2.2% lose | 43% | +| text_scroll | -31.0% win | -19.6% win | -14.3% win | 54% | +| webcam_overlay | -34.1% win | -14.9% win | +1.2% lose | 88% | + +27 wins, 9 losses across 12 clips x 3 metrics. + +### rd-next d0cce5e: capcodec-live-zerolatency vs x264-veryfast-zerolatency (BD-rate, negative = capcodec needs fewer bits) + +| clip | VMAF | PSNR-YUV | SSIM | VMAF range overlap | +|---|---|---|---|---| +| busy_ui | -21.3% win | -6.6% win | +1.0% lose | 48% | +| busy_ui_60fps | -18.9% win | -6.2% win | +3.0% lose | 64% | +| code_4k | -20.7% win | -0.8% win | +5.4% lose | 62% | +| code_editor | -22.6% win | -3.7% win | +2.8% lose | 62% | +| dark_mode | -37.4% win | -26.1% win | -22.1% win | 58% | +| dashboard | -21.6% win | -7.8% win | -0.1% win | 45% | +| full_motion | -15.6% win | -0.3% win | +7.4% lose | 62% | +| idle | -33.5% win | -25.2% win | -13.5% win | 46% | +| slides | -46.6% win | -35.0% win | -23.4% win | 58% | +| slow_typing | -21.3% win | -6.2% win | +0.5% lose | 70% | +| text_scroll | -32.6% win | -24.5% win | -20.8% win | 53% | +| webcam_overlay | -80.6% win | -64.2% win | -43.9% win | 58% | + +30 wins, 6 losses across 12 clips x 3 metrics. + +### main 00fecf7: speed at matched VMAF (x264 / capcodec, interleaved medians, other work paused) + +| clip | mode | VMAF | CPU 1t | wall 1t | CPU 4t | wall 4t | peak RSS 4t capcodec / x264 MiB | +|---|---|---|---|---|---|---|---| +| busy_ui | live | 94.5 | 7.42x win | 6.55x win | 8.25x win | 4.40x win | 46 / 377 | +| busy_ui_60fps | live | 93.2 | 8.20x win | 7.23x win | 9.21x win | 4.17x win | 46 / 374 | +| code_4k | live | 95.2 | 8.92x win | 7.96x win | 9.73x win | 4.58x win | 176 / 1304 | +| code_editor | live | 94.4 | 7.36x win | 6.52x win | 7.97x win | 4.49x win | 49 / 382 | +| dark_mode | live | 93.4 | 7.33x win | 6.54x win | 6.80x win | 4.59x win | 48 / 393 | +| dashboard | live | 94.4 | 3.86x win | 3.51x win | 4.25x win | 2.10x win | 51 / 384 | +| full_motion | live | 88.2 | 1.61x win | 1.54x win | 1.37x win | 1.21x win | 80 / 379 | +| idle | live | 95.3 | 9.18x win | 8.11x win | 10.19x win | 4.55x win | 44 / 379 | +| slides | live | 95.0 | 5.15x win | 4.64x win | 4.61x win | 3.24x win | 74 / 391 | +| slow_typing | live | 95.2 | 7.87x win | 6.88x win | 8.82x win | 4.15x win | 45 / 387 | +| text_scroll | live | 95.0 | 4.88x win | 4.40x win | 3.84x win | 3.10x win | 53 / 384 | +| webcam_overlay | live | 95.4 | 4.64x win | 4.21x win | 4.55x win | 3.46x win | 74 / 391 | +| busy_ui | medium | 94.5 | 7.30x win | 6.31x win | 7.77x win | 4.11x win | 77 / 371 | +| busy_ui_60fps | medium | 93.2 | 8.08x win | 7.13x win | 8.77x win | 4.09x win | 77 / 372 | +| code_4k | medium | 95.2 | 7.96x win | 7.13x win | 8.78x win | 3.89x win | 308 / 1279 | +| code_editor | medium | 94.4 | 7.01x win | 6.17x win | 7.88x win | 4.13x win | 83 / 379 | +| dark_mode | medium | 93.4 | 7.25x win | 6.48x win | 6.37x win | 4.24x win | 79 / 386 | +| dashboard | medium | 94.4 | 3.59x win | 3.21x win | 3.82x win | 1.75x win | 85 / 382 | +| full_motion | medium | 88.2 | 1.39x win | 1.33x win | 1.22x win | 1.06x win | 103 / 375 | +| idle | medium | 95.3 | 9.37x win | 8.30x win | 10.28x win | 4.54x win | 73 / 385 | +| slides | medium | 95.0 | 4.86x win | 4.40x win | 4.34x win | 2.90x win | 98 / 384 | +| slow_typing | medium | 95.2 | 7.94x win | 6.94x win | 8.26x win | 3.97x win | 76 / 384 | +| text_scroll | medium | 95.0 | 4.39x win | 3.88x win | 3.64x win | 2.89x win | 88 / 385 | +| webcam_overlay | medium | 95.4 | 4.32x win | 3.92x win | 4.20x win | 3.01x win | 99 / 386 | +| busy_ui | zerolatency | 96.0 | 6.79x win | 5.96x win | 6.92x win | 3.85x win | 34 / 144 | +| busy_ui_60fps | zerolatency | 95.5 | 7.75x win | 6.82x win | 7.96x win | 3.95x win | 34 / 150 | +| code_4k | zerolatency | 96.0 | 7.51x win | 6.58x win | 7.06x win | 3.92x win | 113 / 458 | +| code_editor | zerolatency | 95.8 | 6.32x win | 5.51x win | 6.38x win | 3.61x win | 34 / 146 | +| dark_mode | zerolatency | 95.5 | 6.51x win | 5.67x win | 5.00x win | 3.76x win | 34 / 151 | +| dashboard | zerolatency | 96.1 | 3.44x win | 3.08x win | 3.64x win | 2.17x win | 37 / 150 | +| full_motion | zerolatency | 88.1 | 1.65x win | 1.58x win | 1.21x win | 1.49x win | 46 / 142 | +| idle | zerolatency | 96.4 | 9.40x win | 8.32x win | 8.89x win | 4.40x win | 34 / 141 | +| slides | zerolatency | 96.2 | 4.84x win | 4.34x win | 3.86x win | 2.95x win | 39 / 149 | +| slow_typing | zerolatency | 96.4 | 7.15x win | 6.15x win | 7.16x win | 3.53x win | 34 / 148 | +| text_scroll | zerolatency | 96.3 | 3.90x win | 3.45x win | 3.12x win | 2.78x win | 38 / 149 | +| webcam_overlay | zerolatency | 96.2 | 5.17x win | 4.73x win | 4.60x win | 4.27x win | 39 / 142 | +- live 1 thread cpu: geometric mean 5.83x, min 1.61x (12/12 clips faster) +- live 1 thread wall: geometric mean 5.23x, min 1.54x (12/12 clips faster) +- live 4 threads cpu: geometric mean 5.87x, min 1.37x (12/12 clips faster) +- live 4 threads wall: geometric mean 3.46x, min 1.21x (12/12 clips faster) +- medium 1 thread cpu: geometric mean 5.53x, min 1.39x (12/12 clips faster) +- medium 1 thread wall: geometric mean 4.94x, min 1.33x (12/12 clips faster) +- medium 4 threads cpu: geometric mean 5.51x, min 1.22x (12/12 clips faster) +- medium 4 threads wall: geometric mean 3.15x, min 1.06x (12/12 clips faster) +- zerolatency 1 thread cpu: geometric mean 5.39x, min 1.65x (12/12 clips faster) +- zerolatency 1 thread wall: geometric mean 4.80x, min 1.58x (12/12 clips faster) +- zerolatency 4 threads cpu: geometric mean 4.90x, min 1.21x (12/12 clips faster) +- zerolatency 4 threads wall: geometric mean 3.26x, min 1.49x (12/12 clips faster) + diff --git a/apps/media-server/capcodec.md b/apps/media-server/capcodec.md new file mode 100644 index 00000000000..0a52ec84855 --- /dev/null +++ b/apps/media-server/capcodec.md @@ -0,0 +1,16 @@ +# capcodec in the media server + +The published media-server images install ffmpeg and do not include the capcodec executable. Leave `CAP_MEDIA_VIDEO_ENCODER` unset to keep libx264. + +To encode with capcodec: + +1. Build a capcodec binary for the image architecture (amd64 and arm64 are separate). +2. Mount it into the container, for example `/usr/local/bin/capcodec`. +3. Set `CAP_MEDIA_VIDEO_ENCODER=capcodec`. +4. Set `CAPCODEC_BIN` to that path. When it is on `PATH`, `capcodec` is enough. + +Optional overrides: `CAPCODEC_CRF` (0–51), `CAPCODEC_PRESET` (`live`, `fast`, `medium`, `slow`), `CAPCODEC_KEYINT`, and `CAPCODEC_NOISE`. Without those, the job's CRF and preset are used. `ultrafast` maps to `fast`. + +The server writes a fragmented MP4 (`.fmp4`) so capcodec flushes each GOP, then ffmpeg remuxes audio and faststart. The decode filter converts frames to limited-range BT.709, matching the labels the encoder writes. + +Benchmark tables against x264 veryfast are in `capcodec-benchmarks.md`. The encoder source is `capcodec/` at the repository root. diff --git a/apps/media-server/src/__tests__/lib/capcodec-encoder.test.ts b/apps/media-server/src/__tests__/lib/capcodec-encoder.test.ts new file mode 100644 index 00000000000..174c18837ac --- /dev/null +++ b/apps/media-server/src/__tests__/lib/capcodec-encoder.test.ts @@ -0,0 +1,177 @@ +import { describe, expect, test } from "bun:test"; +import { + assertCapcodecBinary, + buildCapcodecDecodeArgs, + buildCapcodecEncodeArgs, + capcodecOptionsForJob, + capcodecOptionsFromEnv, + fitEvenSize, + frameRateFraction, + remainingTimeoutMs, +} from "../../lib/capcodec-encoder"; + +const enabled = { CAP_MEDIA_VIDEO_ENCODER: "capcodec" }; + +describe("capcodec encoder integration", () => { + test("is disabled unless CAP_MEDIA_VIDEO_ENCODER is capcodec", () => { + expect(capcodecOptionsFromEnv({})).toBeNull(); + expect( + capcodecOptionsFromEnv({ CAP_MEDIA_VIDEO_ENCODER: "x264" }), + ).toBeNull(); + expect(capcodecOptionsForJob({ crf: 18, preset: "slow" }, {})).toBeNull(); + }); + + test("uses the job CRF and preset unless the environment sets them", () => { + expect( + capcodecOptionsForJob({ crf: 18, preset: "ultrafast" }, enabled), + ).toEqual({ + binary: "capcodec", + crf: 18, + preset: "fast", + keyint: 250, + noise: 3, + }); + expect( + capcodecOptionsForJob( + { crf: 18, preset: "slow" }, + { + ...enabled, + CAPCODEC_BIN: "/opt/capcodec", + CAPCODEC_CRF: "26.5", + CAPCODEC_PRESET: "live", + CAPCODEC_KEYINT: "120", + CAPCODEC_NOISE: "0", + }, + ), + ).toEqual({ + binary: "/opt/capcodec", + crf: 26.5, + preset: "live", + keyint: 120, + noise: 0, + }); + }); + + test("ignores blank or out-of-range encoder settings", () => { + expect( + capcodecOptionsForJob( + { crf: 20, preset: "medium" }, + { + ...enabled, + CAPCODEC_BIN: " ", + CAPCODEC_CRF: "80", + CAPCODEC_PRESET: "", + CAPCODEC_KEYINT: "0", + CAPCODEC_NOISE: "-1", + }, + ), + ).toEqual({ + binary: "capcodec", + crf: 20, + preset: "medium", + keyint: 250, + noise: 3, + }); + expect( + capcodecOptionsFromEnv({ + ...enabled, + CAPCODEC_CRF: "nope", + CAPCODEC_PRESET: "placebo", + }), + ).toMatchObject({ crf: 23, preset: "slow" }); + }); + + test("fits the media server's 1080p box like the libx264 scale filter", () => { + expect(fitEvenSize(3840, 2160, 1920, 1080)).toEqual({ + width: 1920, + height: 1080, + }); + expect(fitEvenSize(1280, 720, 1920, 1080)).toEqual({ + width: 1280, + height: 720, + }); + expect(fitEvenSize(2560, 1600, 1920, 1080)).toEqual({ + width: 1728, + height: 1080, + }); + expect(fitEvenSize(1001, 563, 1920, 1080)).toEqual({ + width: 1000, + height: 562, + }); + expect(() => fitEvenSize(0, 1080, 1920, 1080)).toThrow( + "Video dimensions must be positive", + ); + }); + + test("expresses frame rates as exact fractions", () => { + expect(frameRateFraction(30)).toBe("30/1"); + expect(frameRateFraction(23.976)).toBe("24000/1001"); + expect(frameRateFraction(29.97)).toBe("30000/1001"); + expect(frameRateFraction(59.94)).toBe("60000/1001"); + expect(frameRateFraction(12.5)).toBe("25/2"); + expect(frameRateFraction(Number.NaN)).toBe("30/1"); + expect(frameRateFraction(0.0001)).toBe("30/1"); + }); + + test("requires a mounted capcodec binary", async () => { + await expect(assertCapcodecBinary("/bin/true")).resolves.toBeUndefined(); + await expect(assertCapcodecBinary("/no/such/capcodec")).rejects.toThrow( + /does not include capcodec/, + ); + }); + + test("leaves the mux the time the encode did not use", () => { + expect(remainingTimeoutMs(10_000, 4_000)).toBe(6_000); + expect(remainingTimeoutMs(10_000, 10_000)).toBe(1); + expect(remainingTimeoutMs(10_000, 0)).toBe(10_000); + }); + + test("pipes raw frames from ffmpeg into capcodec", () => { + const decode = buildCapcodecDecodeArgs( + { inputPath: "in.webm", extraInputArgs: [] }, + 1920, + 1080, + "30/1", + ); + expect(decode.slice(-4)).toEqual(["yuv420p", "-progress", "pipe:2", "-"]); + expect(decode).toContain( + "scale=1920:1080:flags=bicubic:in_color_matrix=auto:out_color_matrix=bt709:in_range=auto:out_range=tv,format=yuv420p", + ); + expect(decode).toContain("rawvideo"); + const encode = buildCapcodecEncodeArgs( + { + binary: "capcodec", + crf: 23.5, + preset: "medium", + keyint: 250, + noise: 3, + }, + "out.mp4", + 1920, + 1080, + "30/1", + ); + expect(encode).toEqual([ + "capcodec", + "encode", + "--input", + "-", + "--width", + "1920", + "--height", + "1080", + "--fps", + "30/1", + "--crf", + "23.5", + "--preset", + "medium", + "--keyint", + "250", + "--noise", + "3", + "--output", + "out.mp4", + ]); + }); +}); diff --git a/apps/media-server/src/lib/capcodec-encoder.ts b/apps/media-server/src/lib/capcodec-encoder.ts new file mode 100644 index 00000000000..652c3c48766 --- /dev/null +++ b/apps/media-server/src/lib/capcodec-encoder.ts @@ -0,0 +1,376 @@ +import { constants } from "node:fs"; +import { access } from "node:fs/promises"; +import { file, type Subprocess, spawn } from "bun"; +import { registerSubprocess, terminateProcess } from "./subprocess"; + +const DEFAULT_CRF = 23; +const DEFAULT_KEYINT = 250; +const DEFAULT_NOISE = 3; +const CAPCODEC_PRESETS = new Set(["live", "fast", "medium", "slow"]); + +export interface CapcodecOptions { + binary: string; + crf: number; + preset: string; + keyint: number; + noise: number; +} + +export interface CapcodecEncodeJob { + inputPath: string; + outputPath: string; + width: number; + height: number; + fps: number; + maxWidth: number; + maxHeight: number; + extraInputArgs: string[]; + options: CapcodecOptions; + totalDurationUs: number; + onProgress?: (progress: number, message: string) => void; + abortSignal?: AbortSignal; +} + +export function mapCapcodecPreset(preset: string): string { + const normalized = preset.trim().toLowerCase(); + if ( + normalized === "ultrafast" || + normalized === "superfast" || + normalized === "veryfast" || + normalized === "faster" + ) { + return "fast"; + } + if ( + normalized === "slower" || + normalized === "veryslow" || + normalized === "placebo" + ) { + return "slow"; + } + if (CAPCODEC_PRESETS.has(normalized)) return normalized; + return "medium"; +} + +function parseCrf(raw: string | undefined): number | undefined { + if (raw === undefined || raw.trim() === "") return undefined; + const value = Number(raw); + if (!Number.isFinite(value) || value < 0 || value > 51) return undefined; + return value; +} + +function parseBoundedInt( + raw: string | undefined, + min: number, + max: number, +): number | undefined { + if (raw === undefined || raw.trim() === "") return undefined; + const value = Number(raw); + if (!Number.isInteger(value) || value < min || value > max) return undefined; + return value; +} + +function clampCrf(crf: number): number { + if (!Number.isFinite(crf)) return DEFAULT_CRF; + return Math.min(51, Math.max(0, crf)); +} + +export function capcodecOptionsFromEnv( + env: Record = process.env, +): CapcodecOptions | null { + if (env.CAP_MEDIA_VIDEO_ENCODER !== "capcodec") return null; + return { + binary: env.CAPCODEC_BIN?.trim() || "capcodec", + crf: parseCrf(env.CAPCODEC_CRF) ?? DEFAULT_CRF, + preset: mapCapcodecPreset(env.CAPCODEC_PRESET ?? "medium"), + keyint: parseBoundedInt(env.CAPCODEC_KEYINT, 1, 100_000) ?? DEFAULT_KEYINT, + noise: parseBoundedInt(env.CAPCODEC_NOISE, 0, 255) ?? DEFAULT_NOISE, + }; +} + +export function capcodecOptionsForJob( + job: { crf: number; preset: string }, + env: Record = process.env, +): CapcodecOptions | null { + const base = capcodecOptionsFromEnv(env); + if (!base) return null; + const presetOverride = env.CAPCODEC_PRESET?.trim(); + return { + ...base, + crf: parseCrf(env.CAPCODEC_CRF) ?? clampCrf(job.crf), + preset: presetOverride + ? mapCapcodecPreset(presetOverride) + : mapCapcodecPreset(job.preset), + }; +} + +export async function assertCapcodecBinary(binary: string): Promise { + const resolved = binary.includes("/") ? binary : Bun.which(binary); + if (resolved) { + try { + await access(resolved, constants.X_OK); + return; + } catch {} + } + throw new Error( + `capcodec binary "${binary}" is not executable. The media-server image does not include capcodec. Mount a binary and set CAPCODEC_BIN to its path before setting CAP_MEDIA_VIDEO_ENCODER=capcodec. See /app/capcodec.md.`, + ); +} + +export function remainingTimeoutMs( + timeoutMs: number, + elapsedMs: number, +): number { + if (!Number.isFinite(timeoutMs) || timeoutMs <= 0) return 1; + if (!Number.isFinite(elapsedMs) || elapsedMs <= 0) return timeoutMs; + return Math.max(1, timeoutMs - elapsedMs); +} + +export function fitEvenSize( + width: number, + height: number, + maxWidth: number, + maxHeight: number, +): { width: number; height: number } { + if (!(width > 0) || !(height > 0) || !(maxWidth > 0) || !(maxHeight > 0)) { + throw new Error("Video dimensions must be positive"); + } + const boxWidth = Math.min(maxWidth, width); + const boxHeight = Math.min(maxHeight, height); + const scale = Math.min(boxWidth / width, boxHeight / height); + const scaledWidth = Math.round(width * scale); + const scaledHeight = Math.round(height * scale); + return { + width: Math.max(2, Math.trunc(scaledWidth / 2) * 2), + height: Math.max(2, Math.trunc(scaledHeight / 2) * 2), + }; +} + +export function frameRateFraction(fps: number): string { + if (!Number.isFinite(fps) || fps <= 0) return "30/1"; + const ntsc = [24, 30, 60].find( + (base) => Math.abs(fps - (base * 1000) / 1001) < 0.01, + ); + if (ntsc !== undefined) return `${ntsc * 1000}/1001`; + const rounded = Math.round(fps * 1000); + if (!Number.isFinite(rounded) || rounded <= 0) return "30/1"; + const divisor = gcd(rounded, 1000); + return `${rounded / divisor}/${1000 / divisor}`; +} + +function gcd(a: number, b: number): number { + let x = Math.abs(a); + let y = Math.abs(b); + while (y !== 0) { + const t = y; + y = x % y; + x = t; + } + return x; +} + +function formatCrf(crf: number): string { + if (Number.isInteger(crf)) return String(crf); + return String(Math.round(crf * 1000) / 1000); +} + +export function buildCapcodecDecodeArgs( + job: Pick, + width: number, + height: number, + fps: string, +): string[] { + return [ + "ffmpeg", + "-hide_banner", + "-nostdin", + "-threads", + "2", + ...job.extraInputArgs, + "-i", + job.inputPath, + "-map", + "0:v:0", + "-vf", + `scale=${width}:${height}:flags=bicubic:in_color_matrix=auto:out_color_matrix=bt709:in_range=auto:out_range=tv,format=yuv420p`, + "-fps_mode", + "cfr", + "-r", + fps, + "-f", + "rawvideo", + "-pix_fmt", + "yuv420p", + "-progress", + "pipe:2", + "-", + ]; +} + +export function buildCapcodecEncodeArgs( + options: CapcodecOptions, + outputPath: string, + width: number, + height: number, + fps: string, +): string[] { + return [ + options.binary, + "encode", + "--input", + "-", + "--width", + String(width), + "--height", + String(height), + "--fps", + fps, + "--crf", + formatCrf(options.crf), + "--preset", + options.preset, + "--keyint", + String(options.keyint), + "--noise", + String(options.noise), + "--output", + outputPath, + ]; +} + +function parseOutTimeUs(line: string): number | null { + const match = line.match(/^out_time_us=(\d+)/); + if (!match?.[1]) return null; + return Number.parseInt(match[1], 10); +} + +function pushStderrLine(lines: string[], line: string): void { + lines.push(line); + if (lines.length > 50) lines.shift(); +} + +async function collectStderr( + stream: ReadableStream, + lines: string[], + onLine?: (line: string) => void, +): Promise { + const reader = stream.getReader(); + const decoder = new TextDecoder(); + let buffer = ""; + try { + while (true) { + const { done, value } = await reader.read(); + if (done) break; + buffer += decoder.decode(value, { stream: true }); + const parts = buffer.split("\n"); + buffer = parts.pop() ?? ""; + for (const line of parts) { + pushStderrLine(lines, line); + onLine?.(line); + } + } + buffer += decoder.decode(); + if (buffer.length > 0) { + pushStderrLine(lines, buffer); + onLine?.(buffer); + } + } finally { + reader.releaseLock(); + } +} + +function cancelled(): Error { + return new Error("Video processing was cancelled"); +} + +export async function encodeVideoWithCapcodec( + job: CapcodecEncodeJob, +): Promise { + if (job.abortSignal?.aborted) throw cancelled(); + await assertCapcodecBinary(job.options.binary); + if (job.abortSignal?.aborted) throw cancelled(); + const size = fitEvenSize(job.width, job.height, job.maxWidth, job.maxHeight); + const fps = frameRateFraction(job.fps); + let decoder: Subprocess | undefined; + let encoder: Subprocess | undefined; + let stopping: Promise | undefined; + const stop = () => { + if (!stopping) { + const procs = [decoder, encoder].filter( + (proc): proc is Subprocess => proc !== undefined, + ); + stopping = Promise.all(procs.map((proc) => terminateProcess(proc))).then( + () => undefined, + ); + } + return stopping; + }; + const onAbort = () => { + void stop(); + }; + try { + decoder = registerSubprocess( + spawn({ + cmd: buildCapcodecDecodeArgs(job, size.width, size.height, fps), + stdout: "pipe", + stderr: "pipe", + }), + ); + encoder = registerSubprocess( + spawn({ + cmd: buildCapcodecEncodeArgs( + job.options, + job.outputPath, + size.width, + size.height, + fps, + ), + stdin: decoder.stdout, + stdout: "ignore", + stderr: "pipe", + }), + ); + job.abortSignal?.addEventListener("abort", onAbort, { once: true }); + const decoderLines: string[] = []; + const encoderLines: string[] = []; + await Promise.all([ + collectStderr(decoder.stderr, decoderLines, (line) => { + const outTime = parseOutTimeUs(line); + if (outTime !== null && job.onProgress && job.totalDurationUs > 0) { + const progress = Math.min(100, (outTime / job.totalDurationUs) * 100); + job.onProgress(progress, `Encoding: ${Math.round(progress)}%`); + } + }), + collectStderr(encoder.stderr, encoderLines), + ]); + if (job.abortSignal?.aborted) throw cancelled(); + const [decoderExit, encoderExit] = await Promise.all([ + decoder.exited, + encoder.exited, + ]); + if (job.abortSignal?.aborted) throw cancelled(); + if (decoderExit !== 0 || encoderExit !== 0) { + const parts: string[] = []; + if (decoderExit !== 0) { + parts.push( + `FFmpeg decode for capcodec exited with code ${decoderExit}. Last stderr: ${decoderLines.slice(-10).join(" | ")}`, + ); + } + if (encoderExit !== 0) { + parts.push( + `capcodec exited with code ${encoderExit}. Last stderr: ${encoderLines.slice(-10).join(" | ")}`, + ); + } + throw new Error(parts.join(" ")); + } + if ((await file(job.outputPath).size) === 0) { + throw new Error("capcodec produced an empty output file"); + } + } catch (error) { + if (job.abortSignal?.aborted) throw cancelled(); + throw error; + } finally { + job.abortSignal?.removeEventListener("abort", onAbort); + await stop(); + } +} diff --git a/apps/media-server/src/lib/media-video.ts b/apps/media-server/src/lib/media-video.ts index 8f87ca0cec3..48158979327 100644 --- a/apps/media-server/src/lib/media-video.ts +++ b/apps/media-server/src/lib/media-video.ts @@ -3,6 +3,12 @@ import { mkdtemp, readdir, rm, writeFile } from "node:fs/promises"; import { isAbsolute, join } from "node:path"; import { setTimeout as sleep } from "node:timers/promises"; import { type BunFile, file, spawn } from "bun"; +import { + type CapcodecOptions, + capcodecOptionsForJob, + encodeVideoWithCapcodec, + remainingTimeoutMs, +} from "./capcodec-encoder"; import { uploadDriveResumable } from "./drive-resumable-upload"; import type { VideoMetadata } from "./job-manager"; import { @@ -1521,6 +1527,29 @@ export async function processVideo( metadata.duration, opts.timeoutMs, ); + const capcodec = + videoTranscode && !normalizeH264Level ? capcodecOptionsForJob(opts) : null; + if (capcodec) { + try { + await processVideoWithCapcodec( + inputPath, + metadata, + opts, + capcodec, + outputTempFile.path, + audioTranscode, + extraInputArgs, + extraOutputArgs, + processTimeoutMs, + onProgress, + abortSignal, + ); + return outputTempFile; + } catch (err) { + await outputTempFile.cleanup(); + throw err; + } + } const ffmpegArgs: string[] = [ "ffmpeg", "-threads", @@ -1662,6 +1691,91 @@ export async function processVideo( } } +async function processVideoWithCapcodec( + inputPath: string, + metadata: VideoMetadata, + opts: Required, + capcodec: CapcodecOptions, + outputPath: string, + audioTranscode: boolean, + extraInputArgs: string[], + extraOutputArgs: string[], + processTimeoutMs: number, + onProgress?: ProgressCallback, + abortSignal?: AbortSignal, +): Promise { + const videoOnly = await createTempFile(".fmp4"); + const controller = new AbortController(); + const forwardAbort = () => controller.abort(); + abortSignal?.addEventListener("abort", forwardAbort, { once: true }); + const startedAt = performance.now(); + const encodePromise = encodeVideoWithCapcodec({ + inputPath, + outputPath: videoOnly.path, + width: metadata.width, + height: metadata.height, + fps: metadata.fps, + maxWidth: opts.maxWidth, + maxHeight: opts.maxHeight, + extraInputArgs, + options: capcodec, + totalDurationUs: metadata.duration * 1_000_000, + onProgress, + abortSignal: controller.signal, + }); + try { + await withTimeout(encodePromise, processTimeoutMs, async () => { + controller.abort(); + await encodePromise.catch(() => undefined); + }); + const muxArgs = [ + "ffmpeg", + "-hide_banner", + "-nostdin", + "-i", + videoOnly.path, + ]; + if (metadata.audioCodec) { + muxArgs.push( + ...extraInputArgs, + "-i", + inputPath, + "-map", + "0:v:0", + "-map", + "1:a:0", + "-c:v", + "copy", + ); + if (audioTranscode) { + muxArgs.push("-c:a", "aac", "-b:a", opts.audioBitrate); + } else { + muxArgs.push("-c:a", "copy"); + } + } else { + muxArgs.push("-map", "0:v:0", "-c:v", "copy", "-an"); + } + muxArgs.push( + "-movflags", + "+faststart", + ...extraOutputArgs, + "-y", + outputPath, + ); + await runFfmpegCommand( + muxArgs, + remainingTimeoutMs(processTimeoutMs, performance.now() - startedAt), + abortSignal, + ); + if ((await file(outputPath).size) === 0) { + throw new Error("capcodec mux produced an empty output file"); + } + } finally { + abortSignal?.removeEventListener("abort", forwardAbort); + await videoOnly.cleanup(); + } +} + function getThumbnailTimestamp( duration: number, requestedTimestamp: number, diff --git a/biome.json b/biome.json index e37af6e1205..189cd4a7406 100644 --- a/biome.json +++ b/biome.json @@ -13,7 +13,14 @@ "!apps/desktop/src/utils/tauri.ts", "!apps/desktop/src/global.d.ts", "!apps/web/public/gif.worker.js", - "!packages/ui-solid/src/auto-imports.d.ts" + "!packages/ui-solid/src/auto-imports.d.ts", + "!capcodec/**/*.tov", + "!capcodec/**/*.py", + "!capcodec/**/*.md", + "!capcodec/**/*.sh", + "!capcodec/**/*.jsonl", + "!capcodec/**/*.pyc", + "!capcodec/capcodec.bundle" ], "ignoreUnknown": false }, diff --git a/capcodec/.gitignore b/capcodec/.gitignore new file mode 100644 index 00000000000..5cdca7f8573 --- /dev/null +++ b/capcodec/.gitignore @@ -0,0 +1,9 @@ +/build/ +/bench/out/ +__pycache__/ +*.pyc +*.yuv +*.264 +*.h264 +*.mkv +/tmp/ diff --git a/capcodec/CAP_BASELINES.md b/capcodec/CAP_BASELINES.md new file mode 100644 index 00000000000..95398295dd4 --- /dev/null +++ b/capcodec/CAP_BASELINES.md @@ -0,0 +1,1759 @@ +# Cap H.264 production encoding baselines + +This file lists the exact H.264 encoder settings used by each Cap surface, so a benchmark harness can reproduce them on Linux. + +## Sources and environment + +| Item | Value | +|---|---| +| Cap repo | `/workspace` on branch `main`, commit `d327051c1` (read only) | +| PR 2312 | `FETCH_HEAD` = `549c5afa9` (`origin/feature/web-editor-desktop-parity`), read with `git show FETCH_HEAD:` | +| Rust ffmpeg bindings | `ffmpeg-next`, CapSoftware fork rev `49db1fede112` | +| Verification ffmpeg | 6.1.1-3ubuntu5 | +| Verification libx264 | core 164 r3108 | +| Verification Chrome | Google Chrome 148.0.7778.96 at `/usr/local/bin/google-chrome`, headless | +| Verification CPU | `nproc` = 4 | + +Line numbers refer to `main` unless marked "PR". + +A value marked "verified" was observed in an encode on this VM, either in ffprobe output or in the x264 SEI options string. A value marked "source" was read from code but not run. Hardware encoders (VideoToolbox, NVENC, QSV, AMF, Media Foundation, AVFoundation) cannot run on this VM. + +## Contents + +0. Conventions and shared facts +1. Media server +2. Web app server-side ffmpeg +3. Render farm +4. Desktop recording +5. Desktop export +6. Web recorders on `main` (MediaRecorder) +7. Browser WebCodecs encoders on `main` +8. PR 2312 differences +9. Running browser surfaces in headless Chrome with Playwright +10. Ambiguities and assumptions +11. Summary table + +--- + +## 0. Conventions and shared facts + +### 0.1 Placeholders + +| Placeholder | Meaning | +|---|---| +| `W`, `H` | Output width and height | +| `FPS` | Frame rate | +| `B` | Target bitrate in bit/s | +| `M` | maxrate, always `B * 3 / 2` with integer division where Cap sets it | +| `K` | Keyframe interval in frames | +| `N` | `nproc` | + +All ffmpeg reproductions read raw input with: + +``` +-f rawvideo -pix_fmt yuv420p -s WxH -r FPS -i in.yuv +``` + +### 0.2 Shared bits-per-pixel (bpp) bitrate model + +Desktop export, desktop recording through `enc-ffmpeg`, the render farm, and the PR browser export all use this model. + +| Implementation | Location | +|---|---| +| `get_bitrate` | `crates/enc-ffmpeg/src/video/h264.rs` lines 1651-1658 | +| `bitrate` | `crates/render-farm/src/video.rs` lines 184-187 | +| `exportBitrate` (PR) | `packages/editor-solid-web/src/browser-export-estimate.ts` | + +The formula is: + +``` +B = W * H * ((max(FPS - 30, 0) * 0.6) + 30) * bpp +M = B * 3 / 2 (Rust only: encoder.set_max_bit_rate(bitrate * 3 / 2)) +``` + +How each implementation computes it: + +- **Rust:** bpp is an `f32`, widened to `f64`, and the result is truncated to `usize`. That gives slightly odd values, for example 29,859,841. +- **PR browser export:** uses JavaScript `f64` with `Math.round`. + +Worked values for the Rust model, shown as `B (M)`: + +| bpp | 1280x720@30 | 1920x1080@30 | 1920x1080@60 | 3840x2160@30 | +|---|---|---|---|---| +| 1.0 | 27,648,000 (41,472,000) | 62,208,000 (93,312,000) | 99,532,800 (149,299,200) | 248,832,000 (373,248,000) | +| 0.3 | 8,294,400 (12,441,600) | 18,662,400 (27,993,600) | 29,859,841 (44,789,761) | 74,649,602 (111,974,403) | +| 0.18 | 4,976,640 (7,464,960) | 11,197,440 (16,796,160) | 17,915,904 (26,873,856) | 44,789,761 (67,184,641) | +| 0.15 | 4,147,200 (6,220,800) | 9,331,200 (13,996,800) | 14,929,920 (22,394,880) | 37,324,801 (55,987,201) | +| 0.08 | 2,211,839 (3,317,758) | 4,976,639 (7,464,958) | 7,962,623 (11,943,934) | 19,906,559 (29,859,838) | +| 0.04 | 1,105,919 (1,658,878) | 2,488,319 (3,732,478) | 3,981,311 (5,971,966) | 9,953,279 (14,929,918) | + +Browser model (`Math.round`, `f64`), B only: + +| bpp | 1280x720@30 | 1920x1080@30 | 1920x1080@60 | 3840x2160@30 | +|---|---|---|---|---| +| 0.3 | 8,294,400 | 18,662,400 | 29,859,840 | 74,649,600 | +| 0.15 | 4,147,200 | 9,331,200 | 14,929,920 | 37,324,800 | +| 0.08 | 2,211,840 | 4,976,640 | 7,962,624 | 19,906,560 | +| 0.04 | 1,105,920 | 2,488,320 | 3,981,312 | 9,953,280 | + +### 0.3 Shared keyframe interval + +`crates/enc-ffmpeg/src/video/h264.rs` lines 1223-1239: + +``` +DEFAULT_KEYFRAME_INTERVAL_SECS = 2 +K = round(2 * FPS).max(1) +``` + +| FPS | K | +|---|---| +| 30 | 60 | +| 60 | 120 | + +The render farm uses `fps * 2` with integer fps (`crates/render-farm/src/video.rs` line 473; keyframe secs = 2 at line 629). That gives the same values. + +### 0.4 How Cap's x264 options reach libx264 (applies to every `enc-ffmpeg` and render-farm libx264 path) + +Options are passed as an AVDictionary to `open_as_with`, which calls `avcodec_open2`. Keys that libx264/AVCodecContext does not recognise are silently left unconsumed and ignored. + +| Key | Status | +|---|---| +| `preset`, `tune`, `crf`, `threads`, `bf`, `rc-lookahead`, `aq-mode`, `trellis`, `g`, `keyint_min` | Consumed | +| `b-adapt` | Ignored. The real option name is `b_strategy`. | +| `ref` | Ignored. The real option name is `refs`. | +| `subme` | Ignored. The real option name is `subq`. | +| `pix_fmt` | Ignored. It is not a codec option. | + +Effect, verified from the x264 SEI string: + +- The export/render-farm "fast" set produces `ref=1 subme=2`, which are the veryfast preset defaults. The intended `ref 2` never applies. +- `subme 2` and `trellis 0` happen to equal the veryfast defaults. +- `b-adapt` is irrelevant because `bf 0`. + +Two more x264 behaviours, both verified: + +- **`maxrate` is ignored.** Cap sets `rc_max_rate` (`M`) but never sets `rc_buffer_size`. x264 logs `VBV maxrate specified, but no bufsize, ignored`, so every Cap libx264 bitrate encode is plain ABR (`rc=abr`, no VBV). +- **`keyint_min` is clamped.** Cap sets `keyint_min = K`. x264 clamps it to `K/2 + 1`, so 60 becomes 31 and 120 becomes 61. + +### 0.5 Shared `enc-ffmpeg` encoder context + +Source: `crates/enc-ffmpeg/src/video/h264.rs` lines 775-816. + +**Threading:** `threads = available_parallelism()` via `set_threading(Config::count(...))`. + +**Colour tags:** +- colorspace BT709 +- color_range MPEG (limited) +- color_primaries BT709 +- color_trc BT709 + +**Rate control:** +- CRF mode: `bit_rate = 0`. +- Otherwise: `bit_rate = get_bitrate(out_w, out_h, fps, bpp)` and `max_bit_rate = bit_rate * 3 / 2`. + +**Timing:** +- Encoder time base is the input `VideoInfo.time_base`. +- Muxer stream time base is `1/90000` (lines 286 and 413). + +**Pixel format (lines 671-688):** +- Uses the input format if the encoder supports it. +- Otherwise YUV420P for libx264 and NV12 for the others. + +**Scaler (lines 627-657):** +- BICUBIC when resizing, FAST_BILINEAR otherwise. +- RGB input uses BT.709 coefficients, full-range in, limited-range out. + +**Encoder priority:** + +| Platform | Order | +|---|---| +| macOS | `h264_videotoolbox`, `libx264` | +| Windows, NVIDIA GPU | `h264_nvenc`, `h264_mf`, `h264_qsv`, `h264_amf`, `libx264` | +| Windows, AMD GPU | `h264_amf`, `h264_mf`, `h264_nvenc`, `h264_qsv`, `libx264` | +| Windows, Intel GPU | `h264_qsv`, `h264_mf`, `h264_nvenc`, `h264_amf`, `libx264` | +| Windows, other or unknown GPU | `h264_nvenc`, `h264_qsv`, `h264_amf`, `h264_mf`, `libx264` | +| Windows, GPU without hardware encoding | `libx264` | +| Linux with `/dev/nvidiactl` present | `h264_nvenc`, `libx264` | +| Linux otherwise | `libx264` | + +Sources: `get_default_encoder_priority` at lines 1133-1173 and `linux_encoder_priority` at lines 1124-1131. + +**Priority overrides:** +- `CAP_EXPORT_FORCE_SOFTWARE_ENCODER` set to `1`, `true`, `yes` or `on` forces `libx264` (line 1192). +- `requires_software_encoder` (line 1062) never applies to exports. For recordings it forces `libx264` when: + - the preset is `HighThroughput`, or + - on macOS/Windows, FPS is above 0.9 times the estimated hardware maximum. +- A CRF request always forces `libx264` (line 1241). +- Windows export with an AMD GPU uses `h264_amf`, `h264_mf`, `h264_nvenc`, `h264_qsv`, `libx264` (lines 1201-1221). + +**Fallback:** encoders are tried in priority order, and the first that opens is used. + +### 0.6 Option sets per encoder + +Source: `crates/enc-ffmpeg/src/video/h264.rs` lines 1258-1357, quoted literally. + +**`h264_videotoolbox`** + +| Mode | Options | +|---|---| +| Export | `realtime=false`, `profile=main`, `allow_sw=0` | +| Recording | `realtime=true`, `prio_speed=true`, `profile=main` | + +No `g` is set in either mode. + +**`h264_nvenc`** + +| Mode | Options | +|---|---| +| Export | `preset=p5`, `tune=hq`, `rc=vbr`, `spatial-aq=1`, `temporal-aq=1`, `b_ref_mode=middle`, `g=K` | +| Recording | `preset=p4`, `tune=ll`, `rc=vbr`, `spatial-aq=1`, `temporal-aq=1`, `g=K` | + +**`h264_qsv`** + +| Mode | Options | +|---|---| +| Export | `preset=medium`, `look_ahead=1`, `look_ahead_depth=20`, `g=K` | +| Recording | `preset=faster`, `look_ahead=1`, `g=K` | + +**`h264_amf`** + +| Mode | Options | +|---|---| +| Export | `quality=quality`, `rc=vbr_peak`, `g=K` | +| Recording | `quality=balanced`, `rc=vbr_latency`, `g=K` | + +**`h264_mf`** + +| Mode | Options | +|---|---| +| Export | `hw_encoding=true`, `scenario=0`, `quality=0`, `g=K` | +| Recording | `hw_encoding=true`, `scenario=4`, `quality=1`, `g=K` | + +**`libx264`**, which branch applies: + +| Condition | Options | +|---|---| +| CRF requested | `preset=slow`, `crf=`, `pix_fmt=yuv420p` (ignored) | +| Export, preset Slow | `preset=slow` | +| Export, preset Medium | `preset=medium` | +| Export, any other preset | `preset=veryfast`, `threads=N`, `bf=0`, `rc-lookahead=10`, `b-adapt=0` (ignored), `aq-mode=1`, `ref=2` (ignored), `subme=2` (ignored), `trellis=0` | +| Recording, Slow or Medium | `preset=veryfast`, `tune=zerolatency` | +| Recording, Ultrafast or HighThroughput | `preset=ultrafast`, `tune=zerolatency` | + +All `libx264` branches also set `g=K` and `keyint_min=K`. + +### 0.7 VideoToolbox and NVENC defaults that Cap inherits (source: FFmpeg 7.1) + +**VideoToolbox (`libavcodec/videotoolboxenc.c`):** +- `vt_defaults` only sets `b`, `qmin` and `qmax`, so `gop_size` keeps the AVCodecContext default of 12. +- Because Cap never sets `g` for VideoToolbox, `MaxKeyFrameInterval = 12` frames. This is not 2 s. +- `max_b_frames` defaults to 0, so `AllowFrameReordering = false` (no B-frames). +- `profile=main` maps to `kVTProfileLevel_H264_Main_AutoLevel`. +- `rc_max_rate` maps to `kVTCompressionPropertyKey_DataRateLimits` as `[M/8 bytes, 1 s]`. + +**NVENC:** +- `g` is set by Cap. +- `bf` stays at -1, meaning the preset decides. +- `refs` stays at 0 (default). +- `rc-lookahead` and `bufsize` are not set by Cap. + +### 0.8 Audio summary + +| Surface | Audio | +|---|---| +| Media server processVideo | AAC 128k | +| Media server edit | AAC 160k | +| Render farm | AAC 320 kbps, 48 kHz | +| Desktop `enc-ffmpeg` | AAC 320 kbps | + +The desktop value comes from `crates/enc-ffmpeg/src/audio/aac.rs` line 33: `OUTPUT_BITRATE: usize = 320 * 1000; // 128k`. The trailing comment is stale. + +--- + +## 1. Media server (`apps/media-server`) + +Image: `apps/media-server/Dockerfile` uses `FROM oven/bun:1.4.0` plus apt `ffmpeg`, with no version pinned. + +### 1.1 MS-TRANSCODE: `processVideo` video transcode + +**Location:** `apps/media-server/src/lib/media-video.ts`. + +| Lines | What | +|---|---| +| 162-172 | `DEFAULT_OPTIONS` | +| 47-50 | Level size constants | +| 1288-1302 | `needsVideoTranscode` | +| 1304-1306 | `needsAudioTranscode` | +| 1405-1420 | `pickMobileSafeH264Level` | +| 1483-1581 | `processVideo` | + +**Constants resolved from `DEFAULT_OPTIONS`:** + +| Option | Value | +|---|---| +| maxWidth | 1920 | +| maxHeight | 1080 | +| crf | 23 | +| preset | `"medium"` | +| audioBitrate | `"128k"` | +| videoBitrate | `"5M"` (defined, never used in any argument) | +| remuxOnly | false | +| normalizeH264Level | false | + +**Literal argument vector when `videoTranscode` is true:** + +``` +ffmpeg -threads 2 [extraInputArgs] -i + -c:v libx264 -preset medium -crf 23 + -vf "scale='min(1920,iw)':'min(1080,ih)':force_original_aspect_ratio=decrease,scale=trunc(iw/2)*2:trunc(ih/2)*2" + -pix_fmt yuv420p -level:v <4.2|5.1|5.2> + [-bf 0] # only when source codec is vp8 or vp9 + <-c:a aac -b:a 128k | -c:a copy | -an> + -movflags +faststart [extraOutputArgs] -progress pipe:2 -y .mp4 +``` + +**When it runs.** `needsVideoTranscode` is true when any of these hold: + +- source width > maxWidth, or height > maxHeight +- the codec is not `h264` +- the probed source H.264 level is above the target level + +**Level choice** (`pickMobileSafeH264Level`) uses the target size `(min(srcW, maxW), min(srcH, maxH))`: + +| Target size | Level | +|---|---| +| ≤2048x1088 | `4.2` | +| ≤4096x2304 | `5.1` | +| Larger | `5.2` | + +With the defaults every transcode is level 4.2. + +**Callers:** + +| Caller | Options passed | Effect | +|---|---|---| +| Web workflows (`process-video.ts`, admin reprocess) | No size or quality options | Defaults apply: a 1080p cap, level 4.2 | +| `/video/convert` (`routes/video.ts` line 594) | Source dimensions as max | No downscale; 4K gets level 5.1 | +| Multipart (`multipart.ts` line 977) | `remuxOnly: true` for web MP4 | Video and audio copied | + +**Audio:** +- Source audio is AAC: `-c:a copy`. +- Other source audio: `-c:a aac -b:a 128k`. +- No audio: `-an`. + +**Resilient mode.** The retry path adds these flags, built at `routes/video.ts` lines 842-847: + +| Position | Flags | +|---|---| +| Input | `-err_detect ignore_err -fflags +genpts+discardcorrupt` | +| Output | `-max_muxing_queue_size 1024` | + +**Encoder settings:** + +| Setting | Value | +|---|---| +| Encoder | libx264 | +| Rate control | CRF 23. No bitrate cap or VBV. | +| GOP | x264 defaults: keyint 250, keyint_min 25, scenecut 40 (verified) | +| B-frames, VP8/VP9 source | `-bf 0` (verified bframes=0) | +| B-frames, other sources | x264 medium default bframes=3 | +| Preset / tune | `medium`, no tune | +| x264 internals at medium | ref=3, subme=7, trellis=1, rc_lookahead=40 (verified) | +| Profile | High (x264 automatic, verified) | +| Level | Forced by `-level:v` (verified 42) | +| Pixel format | yuv420p | + +**Threads.** `-threads 2` is placed before `-i`, so it applies to the decoder only. The encoder uses ffmpeg's automatic count: verified threads=6 on 4 cores. + +**Colour.** No colour options are set, so tags are passed through from the decoder. Untagged sources stay untagged. + +**Scaling:** +- First: fit inside 1920x1080, preserving aspect ratio, never upscaling. +- Then: round both dimensions down to even. +- Scaler flags: swscale default (bicubic). + +**Frame rate:** +- No `-r`, no fps filter, no `-fps_mode`/`-vsync`. +- The ffmpeg CLI `fps_mode auto` picks CFR for the MP4 muxer, so VFR browser captures are duplicated/dropped to the rate ffmpeg guesses for the input. +- Verified: a VFR WebM tagged 30/1 came out as 90 frames at 30/1. + +**Container:** MP4 with `+faststart`. Audio, when transcoded, is AAC 128k. + +**Reproduction on Linux (verified):** + +``` +ffmpeg -y -threads 2 -f rawvideo -pix_fmt yuv420p -s WxH -r FPS -i in.yuv \ + -c:v libx264 -preset medium -crf 23 \ + -vf "scale='min(1920,iw)':'min(1080,ih)':force_original_aspect_ratio=decrease,scale=trunc(iw/2)*2:trunc(ih/2)*2" \ + -pix_fmt yuv420p -level:v 4.2 -an -movflags +faststart ms_transcode.mp4 +``` + +- **MS-TRANSCODE-VPX:** add `-bf 0` before `-an`. This is the variant that applies to Chrome WebM recordings on `main`. +- **3840x2160 input:** with default options, the output is 1920x1080 at level 4.2. + +### 1.2 MS-LEVELFIX: level rewrite without re-encode (dormant) + +**Location:** `media-video.ts` lines 1533-1539. + +``` +ffmpeg -threads 2 -i -c:v copy -bsf:v h264_metadata=level=<4.2|5.1|5.2> ... -movflags +faststart -progress pipe:2 -y +``` + +It runs only when `opts.normalizeH264Level` is true and the source fits inside the size limits with a level above the target. No caller sets `normalizeH264Level`, so this is not reached in production. A too-high level instead causes a full MS-TRANSCODE. + +### 1.3 Other media-server ffmpeg invocations (no H.264 encode) + +**Copy path:** `-c:v copy` in `processVideo`, used for H.264 sources within the size and level limits. + +**Container repair** (`media-video.ts` lines 1325-1361): + +``` +ffmpeg -threads 2 -err_detect ignore_err -fflags +genpts+igndts -i -c copy -y .mkv +``` + +**Edit concat** (`media-edit.ts` `buildConcatArgs`, line 207): + +``` +ffmpeg -hide_banner -y -f concat -safe 0 -i -map 0 -c copy -movflags +faststart +``` + +**Unused:** `buildTranscodeSegmentArgs` and `buildStreamCopySegmentArgs` in `media-edit.ts` have no callers. + +### 1.4 MS-EDIT: edited-video render (`renderEditedVideo`) + +**Location:** `apps/media-server/src/lib/media-edit.ts`. + +| Lines | What | +|---|---| +| 83-87 | `getOutputFps` | +| 151-205 | `buildTranscodeEditArgs` | +| 446-469 | `renderEditedVideo` | + +It is called from `routes/video.ts` line 1168. + +**Inputs and constants:** +- `fps` is the probed `metadata.fps`. +- `getOutputFps(fps) = min(120, max(1, round(fps*100)/100))`, or `DEFAULT_OUTPUT_FPS` = 30 when fps is missing or invalid. +- Up to `MAX_TRANSCODE_RANGES_PER_BATCH` = 4 ranges per ffmpeg run. Multiple batches are joined by the concat copy in section 1.3. + +**Literal arguments:** + +``` +ffmpeg -hide_banner -y + (-ss -t -i ) x ranges + -filter_complex "[i:v:0]fps=F,setpts=PTS-STARTPTS[vi];[i:a:0]asetpts=PTS-STARTPTS[ai];...;[v0][a0]...concat=n=R:v=1:a=1[v][a]" + -map [v] -c:v libx264 -preset fast -crf 18 -pix_fmt yuv420p -enc_time_base:v 1/F + (-map [a] -c:a aac -b:a 160k | -an) + -movflags +faststart +``` + +Without audio, each video filter is `[i:v:0]fps=F,setpts=PTS-STARTPTS[vi]` and the concat is `concat=n=R:v=1:a=0[v]`. + +**Encoder settings:** + +| Setting | Value | +|---|---| +| Encoder | libx264 | +| Rate control | CRF 18 | +| GOP | x264 default keyint 250, min 25 | +| B-frames | x264 fast default bframes=3, b_adapt=1 (verified has_b_frames=2) | +| Preset / tune | `fast`, no tune | +| x264 internals at fast | ref=2, subme=6, rc_lookahead=30 (verified) | +| Profile | High (verified) | +| Level | x264 automatic (verified 3.1 at 720p30) | +| Pixel format | yuv420p | +| Colour | Passed through from input | + +**Frame rate:** CFR at `F`, enforced by the `fps` filter, with `-enc_time_base:v 1/F`. + +**Container and audio:** MP4 with `+faststart`; AAC 160k. + +**Reproduction on Linux** (single range, no audio, exactly the filter string the code builds; verified): + +``` +ffmpeg -hide_banner -y -f rawvideo -pix_fmt yuv420p -s WxH -r FPS -i in.yuv \ + -filter_complex "[0:v:0]fps=FPS,setpts=PTS-STARTPTS[v0];[v0]concat=n=1:v=1:a=0[v]" \ + -map "[v]" -c:v libx264 -preset fast -crf 18 -pix_fmt yuv420p -enc_time_base:v 1/FPS \ + -an -movflags +faststart ms_edit.mp4 +``` + +--- + +## 2. Web app server-side ffmpeg + +### 2.1 WEB-LOOM-FFMPEG: Loom download conversion + +**Location:** `apps/web/lib/video-convert.ts`, used by `app/api/tools/loom-download/route.ts`. + +It first tries a remux (line 212): + +``` +ffmpeg -y -i -c copy -movflags +faststart +``` + +If the remux fails, it re-encodes (line 223): + +``` +ffmpeg -y -i -c:v libx264 -preset veryfast -pix_fmt yuv420p -c:a aac -b:a 128k -movflags +faststart +``` + +**Encoder settings:** + +| Setting | Value | +|---|---| +| Rate control | CRF 23 (libx264 default, no `-crf` given) | +| GOP | 250/25 | +| B-frames | veryfast default bframes=3 (verified has_b_frames=2) | +| Profile / level | High, automatic level (verified 3.1 at 720p30) | +| Frame rate | CLI auto, so CFR for MP4 | +| Colour | Passed through | +| Container | MP4 `+faststart` | +| Audio | AAC 128k | + +**Reproduction on Linux (verified):** + +``` +ffmpeg -y -f rawvideo -pix_fmt yuv420p -s WxH -r FPS -i in.yuv \ + -c:v libx264 -preset veryfast -pix_fmt yuv420p -an -movflags +faststart web_loom_ffmpeg.mp4 +``` + +--- + +## 3. Render farm + +### 3.1 Runtime configuration + +**Location:** `apps/render-farm/Dockerfile`. + +- Base image: Debian trixie, using Debian's ffmpeg. +- Lines 47-49: + ``` + ENV RF_ENCODER=h264_nvenc + ENV RF_NVENC_PRESET=p1 + ENV RF_REQUIRE_GPU=1 + ``` + +**Encoder selection and fallback:** +- In code, `RF_ENCODER` defaults to `"libx264"` (`crates/render-farm/src/video.rs` line 425). Production sets NVENC through the Dockerfile. +- There is no automatic NVENC to libx264 fallback. +- If NVENC is unavailable, `find_by_name` fails (`"{encoder_name} unavailable"`). +- `RF_REQUIRE_GPU` checks at `video.rs` lines 257 and 364 fail fast without a GPU. +- libx264 is used only when `RF_ENCODER` is overridden. + +**Job defaults** (`apps/render-farm/src/coordinator.ts` lines 374-384): + +| Field | Value | +|---|---| +| compression | `Maximum` | +| fps | 30 | +| resolution | `[1920, 1080]` | +| Worker threads per task | 8 (line 1067) | + +**Compression map** (`apps/render-farm/src/protocol.ts` `COMPRESSION_BPP`): + +| Compression | bpp | +|---|---| +| Maximum | 0.3 | +| Social | 0.15 | +| Web | 0.08 | +| Potato | 0.04 | + +### 3.2 RF-RENDER-NVENC and RF-RENDER-X264: chunked render encode + +**Location:** `crates/render-farm/src/video.rs` `encode`, lines 415-512. + +**Common settings, both encoders:** + +| Setting | Value | +|---|---| +| Input pixel format | NV12, or CUDA surfaces when NVENC and the renderer produced GPU frames (Linux) | +| Time base | `1/fps` | +| Frame rate | `fps/1` (CFR; frame-numbered) | +| Colour | BT709 colorspace, MPEG (limited) range, BT709 primaries, BT709 trc | +| Bitrate | `bit_rate = bitrate(W, H, fps, bpp)`, `max_bit_rate = B * 3 / 2` | +| Flags | `GLOBAL_HEADER` | +| K | `fps * 2` | + +**NVENC dictionary (production):** + +| Option | Value | +|---|---| +| preset | `RF_NVENC_PRESET` or `"p1"` | +| tune | `RF_NVENC_TUNE` or `"hq"` | +| rc | `constqp` with `qp=0` if tune is `"lossless"`, otherwise `vbr` | +| spatial-aq | `RF_NVENC_SAQ` or `"1"` | +| temporal-aq | `RF_NVENC_TAQ` or `"1"` | +| bf | `0` | +| g | K | +| forced-idr | `1` | + +NVENC does not get the thread setting. + +**libx264 dictionary (override only):** + +| Option | Value | +|---|---| +| preset | `veryfast` | +| threads | `8` | +| bf | `0` | +| rc-lookahead | `10` | +| b-adapt | `0` (ignored) | +| aq-mode | `1` | +| ref | `2` (ignored) | +| subme | `2` (ignored) | +| trellis | `0` | +| g | K | +| keyint_min | K | + +The context threading count is also 8. + +**Rate control:** +- NVENC: VBR with average B and max M. bufsize is not set, so NVENC's default VBV applies. +- libx264: ABR at B, maxrate ignored (verified `rc=abr bitrate=8294` at 720p30, Maximum). + +**GOP:** +- NVENC: K with forced IDR. Each HLS segment of `round(fps*2)` frames is one GOP. +- libx264: keyint K, keyint_min clamped to K/2+1 (verified 60/31), scenecut 40, no B-frames. + +**Profile and level:** +- NVENC: not set, so the encoder picks it. The preset decides profile; High is expected. +- libx264: High, automatic level (verified High). + +**Container.** The farm writes its own boxes rather than using ffmpeg's muxer: + +- **MP4** (`apps/render-farm/src/mp4.ts`): + - `buildHeader` (line 654) writes `ftyp` (major `isom`, minor `0x200`, brands `isom iso2 avc1 mp41`), then `moov`, `free`, `mdat`. The moov is up front. + - Video timescale is `fps*1000` and sample delta is 1000 (CFR). + - The `avc1` entry (line 584) carries `colr nclx 1/1/1/0` and `pasp`. +- **HLS** (`RF_HLS` is on unless it is `"0"`; `RF_HLS_SEGMENT_SECONDS` defaults to 2; `coordinator.ts` lines 94-95): + - Segments are fMP4 with `init.mp4` (line 1981) plus `styp+moof+mdat` `.m4s` files. + - Segment keys are `${prefix}/c${chunk}-p${firstPart}-${index}.m4s` (`hls.ts` line 40). + - `segmentFrames = round(fps*2)` (line 1087). + - The playlist (`fmp4.ts` lines 267-285) has `#EXT-X-VERSION:7`, `#EXT-X-TARGETDURATION:` = 4, `#EXT-X-MEDIA-SEQUENCE:0`, `#EXT-X-PLAYLIST-TYPE:EVENT`, `#EXT-X-INDEPENDENT-SEGMENTS`, `#EXT-X-MAP`, `#EXTINF:` and `#EXT-X-ENDLIST`. + +**Audio:** AAC at 320,000 bit/s and 48,000 Hz (`crates/render-farm/src/audio.rs`). + +**Worked bitrates at Maximum (bpp 0.3):** + +| Resolution | B | M | +|---|---|---| +| 1920x1080@30 (job default) | 18,662,400 | 27,993,600 | +| 1920x1080@60 | 29,859,841 | 44,789,761 | +| 1280x720@30 | 8,294,400 | | +| 3840x2160@30 | 74,649,602 | | + +Other compressions follow the table in section 0.2. + +**Reproduction on Linux.** + +RF-RENDER-X264 (verified; the x264 SEI matches `ref=1 subme=2 trellis=0 threads=8 keyint=60 keyint_min=31 rc_lookahead=10 rc=abr bitrate=8294`): + +``` +ffmpeg -y -f rawvideo -pix_fmt yuv420p -s WxH -r FPS -i in.yuv \ + -pix_fmt nv12 -c:v libx264 -preset veryfast -threads 8 -b:v B -maxrate M \ + -bf 0 -rc-lookahead 10 -aq-mode 1 -trellis 0 -g K -keyint_min K \ + -colorspace bt709 -color_primaries bt709 -color_trc bt709 -color_range tv \ + -an -movflags +faststart rf_render_x264.mp4 +``` + +- `-maxrate` is included because the production code sets it. It has no effect without `-bufsize`. +- The farm's own MP4 writer is approximated with `-movflags +faststart` (moov first). + +HLS approximation (verified to produce `#EXT-X-VERSION:7`, `PLAYLIST-TYPE:EVENT`, `INDEPENDENT-SEGMENTS`, `EXT-X-MAP init.mp4`). Replace the last line of the previous command with: + +``` + -f hls -hls_time 2 -hls_segment_type fmp4 -hls_playlist_type event \ + -hls_fmp4_init_filename init.mp4 -hls_flags independent_segments out/rf.m3u8 +``` + +This approximation differs from production in two ways: +- ffmpeg writes TARGETDURATION 2, while the farm writes 4. +- Segment names differ. + +RF-RENDER-X264-INTENDED is not production. It shows what the dictionary was meant to do: add `-refs 2 -subq 2 -b_strategy 0`. + +RF-RENDER-NVENC cannot run on this VM. On an NVIDIA host: + +``` +ffmpeg -y -f rawvideo -pix_fmt yuv420p -s WxH -r FPS -i in.yuv \ + -pix_fmt nv12 -c:v h264_nvenc -preset p1 -tune hq -rc vbr -b:v B -maxrate M \ + -spatial-aq 1 -temporal-aq 1 -bf 0 -g K -forced-idr 1 \ + -colorspace bt709 -color_primaries bt709 -color_trc bt709 -color_range tv \ + -an -movflags +faststart rf_render_nvenc.mp4 +``` + +### 3.3 RF-TRANSCODE: source preparation before rendering + +**Location:** `apps/render-farm/src/transcode.ts`. + +| Lines | What | +|---|---| +| 7-49 | `transcodeArgs` | +| 56 | `MAX_REMUX_KEYFRAME_GAP_SECONDS = 4` | +| 77-111 | `canRemux` | +| 113 | `remuxArgs` | + +`worker.ts` line 705 sets `TRANSCODE_ENCODER = process.env.RF_TRANSCODE_ENCODER ?? "h264_nvenc"`. `keyframeSeconds` is `TRANSCODE_KEYFRAME_SECONDS = 1` (`coordinator.ts` line 1551). + +**Remux path.** It is used when `canRemux` is true: +- codec h264 +- `yuv420p` +- `has_b_frames=0` +- every packet has pts == dts +- keyframe gaps ≤ 4 s, including the tail + +``` +ffmpeg -hide_banner -nostdin -y -loglevel error -progress pipe:1 -nostats -i -map 0:v:0 -an -c:v copy -movflags +faststart +``` + +**Transcode path, literal** (with `keyframeSeconds = 1`): + +``` +ffmpeg -hide_banner -nostdin -y -loglevel error -progress pipe:1 -nostats + [-hwaccel cuda] # NVENC only + -i -map 0:v:0 -an -fps_mode passthrough -pix_fmt yuv420p -c:v + NVENC: -preset p1 -rc vbr -cq 19 -b:v 0 + libx264: -preset veryfast -crf 18 + -bf 0 -force_key_frames "expr:gte(t,n_forced*1)" + [-forced-idr 1] # NVENC only + -movflags +faststart +``` + +**Encoder settings:** + +| Setting | NVENC (production default) | libx264 | +|---|---|---| +| Rate control | Constant-quality VBR, `-cq 19`, `-b:v 0` | CRF 18 | +| Preset | p1 | veryfast | +| Keyframes | Forced every 1 s, as IDR | Forced every 1 s, on top of x264 keyint 250 / min 25 / scenecut 40 | +| B-frames | None | None | + +- **Profile and level:** encoder automatic. libx264 verified High. +- **Frame rate:** `-fps_mode passthrough` keeps the source VFR timestamps. +- **Colour:** passed through. +- **Container:** MP4 `+faststart`, video only. +- **Hardware decode:** `-hwaccel cuda` decodes on the GPU but frames are downloaded (no `-hwaccel_output_format`), so `-pix_fmt yuv420p` applies. + +**Reproduction on Linux.** + +RF-TRANSCODE-X264 (verified keyframes at 0, 1, 2 s, bframes=0, High): + +``` +ffmpeg -hide_banner -nostdin -y -f rawvideo -pix_fmt yuv420p -s WxH -r FPS -i in.yuv \ + -map 0:v:0 -an -fps_mode passthrough -pix_fmt yuv420p -c:v libx264 -preset veryfast -crf 18 \ + -bf 0 -force_key_frames "expr:gte(t,n_forced*1)" -movflags +faststart rf_transcode_x264.mp4 +``` + +RF-TRANSCODE-NVENC cannot run on this VM: + +``` +ffmpeg -hide_banner -nostdin -y -hwaccel cuda -i in.mp4 -map 0:v:0 -an -fps_mode passthrough \ + -pix_fmt yuv420p -c:v h264_nvenc -preset p1 -rc vbr -cq 19 -b:v 0 -bf 0 \ + -force_key_frames "expr:gte(t,n_forced*1)" -forced-idr 1 -movflags +faststart rf_transcode_nvenc.mp4 +``` + +--- + +## 4. Desktop recording + +### 4.1 Common facts + +**Defaults** (`crates/recording/src/defaults.rs`): + +| Constant | Value | +|---|---| +| `DEFAULT_STUDIO_MAX_FPS` | 60 | +| `CAMERA_ACTIVE_STUDIO_MAX_FPS` | 30 | +| `DEFAULT_INSTANT_MODE_FPS` | 30 | +| `FREE_INSTANT_MODE_MAX_RESOLUTION` | 1280 | +| `PRO_INSTANT_MODE_MAX_RESOLUTION` | 1920 | +| `DEFAULT_CRASH_RECOVERY_RECORDING` | true, so fragmented recording is the default | +| `DEFAULT_OUT_OF_PROCESS_MUXER` | false | + +- Default studio quality is `Compatibility` if total RAM is under 16 GiB, else `Balanced`. `Ultra` is opt-in. +- Instant mode clamps the output to fit `(max, max*9/16)`: 1280x720 for free users, 1920x1080 for Pro (`instant_recording.rs` lines 719-734). + +**Timestamps (all `enc-ffmpeg` recording paths):** +- `VideoInfo.time_base = 1/1_000_000` (`crates/media-info/src/lib.rs` lines 47, 58, 89). +- `update_pts` (`crates/enc-ffmpeg/src/base.rs` lines 76-104) sets `pts = round(capture_ts_secs * 1e6) - first_pts`. +- So output is VFR, following capture timestamps. +- `frame_rate` is the capture rate. It is used only for bitrate and K. + +**Fragmented container.** `SegmentedVideoEncoder` (`crates/enc-ffmpeg/src/mux/segmented_stream.rs` lines 195-240) uses the ffmpeg `dash` muxer writing `dash_manifest.mpd`: + +| Muxer option | Value | +|---|---| +| `init_seg_name` | `init.mp4` | +| `media_seg_name` | `segment_$Number%03d$.m4s` | +| `seg_duration` | `` | +| `use_timeline` | `1` | +| `use_template` | `1` | +| `single_file` | `0` | +| `hls_playlist` | `1` | + +Output files, verified: `init.mp4`, `segment_001.m4s`, ..., `master.m3u8`, and `media_0.m3u8` (`#EXT-X-VERSION:6` with `PROGRAM-DATE-TIME`). + +### 4.2 Linux (X11 capture) + +**Location:** `crates/recording/src/capture_pipeline.rs` lines 374-430, and `crates/recording/src/output_pipeline/ffmpeg.rs`. + +**Studio:** `SegmentedVideoMuxer`, 2 s segments, preset `Medium` when quality is Ultra, otherwise `Ultrafast`. + +**Instant:** `SegmentedVideoMuxer`, 2 s segments, preset `Ultrafast`, output size clamped as in section 4.1. + +**bpp is always 0.3.** `start_encoder` (`ffmpeg.rs` lines 648-672) hardcodes `bpp: H264EncoderBuilder::QUALITY_BPP` (0.3) for every quality and for instant mode. Instant mode does not use 0.15 here, unlike macOS and Windows. `prefer_videotoolbox_hw_input` is false. + +**Camera:** `SegmentedVideoMuxer` with a 3 s segment (line 825). The muxer default is 3 s (line 471). + +**Encoder:** `/dev/nvidiactl` present gives `h264_nvenc` (recording set: p4, ll, vbr, AQ, `g=K`), otherwise `libx264`. + +**libx264 settings:** + +| Quality | Preset / tune | +|---|---| +| Balanced or Compatibility | `ultrafast` + `zerolatency` | +| Ultra (preset Medium) | `veryfast` + `zerolatency` | + +| Setting | Value | +|---|---| +| Rate control | ABR at B (bpp 0.3), maxrate ignored | +| GOP | keyint K, keyint_min clamped to K/2+1 | +| Scenecut | 0 (zerolatency) | +| B-frames | None | +| Pixel format | yuv420p, converted from capture format by swscale (BT.709, full-range in, limited out) | +| Colour | BT.709 tags | + +**Verified ultrafast/zerolatency output at 720p30:** +- Constrained Baseline, level 3.2 +- `cabac=0 ref=1 subme=0 trellis=0 8x8dct=0 sliced_threads=1 bframes=0 keyint=60 keyint_min=31 scenecut=0 rc=abr mbtree=0 aq=0` + +**Worked B at bpp 0.3:** + +| Resolution | B | +|---|---| +| 1280x720@30 | 8,294,400 | +| 1920x1080@30 | 18,662,400 | +| 1920x1080@60 | 29,859,841 | +| 3840x2160@30 | 74,649,602 | + +K is 60 at 30 fps and 120 at 60 fps. + +**Audio:** AAC 320 kbps via `enc-ffmpeg`. + +**Reproduction on Linux.** + +DESK-REC-X264 (verified): + +``` +mkdir -p out && ffmpeg -y -f rawvideo -pix_fmt yuv420p -s WxH -r FPS -i in.yuv \ + -c:v libx264 -preset ultrafast -tune zerolatency -threads N -b:v B -maxrate M -g K -keyint_min K \ + -pix_fmt yuv420p -colorspace bt709 -color_primaries bt709 -color_trc bt709 -color_range tv -an \ + -f dash -seg_duration 2 -use_timeline 1 -use_template 1 -single_file 0 -hls_playlist 1 \ + -init_seg_name init.mp4 -media_seg_name 'segment_$Number%03d$.m4s' out/dash_manifest.mpd +``` + +DESK-REC-X264-ULTRA: the same command with `-preset veryfast`. B stays at bpp 0.3 on Linux. + +Difference from production: the raw input is CFR. Production uses VFR capture timestamps in a 1/1e6 time base. + +### 4.3 macOS (cannot run on this VM) + +**Fragmented studio (default)** (`capture_pipeline.rs` lines 113-226): + +- Muxer: `MacOSFragmentedM4SMuxer`, or the out-of-process muxer when enabled. Both use `SegmentedVideoEncoder`. +- `prefer_videotoolbox_hw_input` is true for screen (`macos_fragmented_m4s.rs` line 295) and false for camera (line 682). +- Defaults (lines 185-187): 2 s segments, `Ultrafast`, bpp 0.3. + +| Quality | bpp | Preset | +|---|---|---| +| Ultra | 1.0 | Medium | +| Compatibility | 0.15 | Ultrafast | +| Balanced | 0.3 | Ultrafast | + +**DESK-REC-VT** (`h264_videotoolbox`, recording set): + +| Setting | Value | +|---|---| +| Options | `realtime=true`, `prio_speed=true`, `profile=main` | +| Bitrate | B; `DataRateLimits` = M/8 bytes per 1 s | +| GOP | 12 frames (FFmpeg default; Cap sets no `g`) | +| B-frames | None (frame reordering off) | +| Level | Main AutoLevel | +| Input | NV12 hardware surfaces, no scaling | +| Colour | BT.709 tags | +| Fallback | libx264 (section 0.6 recording set) when VideoToolbox fails to open or fps exceeds 0.9 times the estimated VideoToolbox maximum | + +Platform command (not runnable here): + +``` +ffmpeg -f rawvideo -pix_fmt nv12 -s WxH -r FPS -i in.nv12 -c:v h264_videotoolbox -realtime true -prio_speed true -profile:v main -b:v B -maxrate M -colorspace bt709 -color_primaries bt709 -color_trc bt709 -color_range tv out.mp4 +``` + +Production passes no `-g`, so the ffmpeg CLI default `gop_size` (12) also applies here. + +**Non-fragmented studio:** `AVFoundationMp4Muxer` (`crates/recording/src/output_pipeline/macos.rs` lines 326-352) builds `cap_enc_avfoundation::MP4Encoder` (`crates/enc-avfoundation/src/mp4.rs`). + +| Setting | Value | +|---|---| +| Settings base | `OutputSettingsAssistant::with_preset(h264_3840x2160)` (line 251) | +| Output height | `ensure_even(output_height or source height)` | +| Output width | Proportional, even (lines 260-268) | +| Bitrate | `AVVideoAverageBitRateKey` from the formulas below | +| `AVVideoAllowFrameReorderingKey` | `ultra && !instant` | +| `AVVideoExpectedSourceFrameRateKey` | fps | +| `AVVideoMaxKeyFrameIntervalKey` | `floor(0.75 * fps).max(1)` | +| Colour | `ITU_R_709_2` for transfer, primaries and matrix (lines 325-339) | +| Writer | `expects_media_data_in_real_time` true | +| Profile, level, entropy coding | AVFoundation defaults | + +Bitrate formulas (lines 1070-1109), with `pixels = W*H` and `f = min(fps, 60)/30`: + +| Profile | Formula | +|---|---| +| Balanced | `max(pixels*f*5, 8,000,000)` | +| Ultra | `clamp(pixels*f*10, 8,000,000, 120,000,000)` | +| Compatibility | `clamp(pixels*f*2.5, 2,500,000, 10,000,000)` | +| Instant | `1,500,000 + pixels/(1920*1080)*1,500,000 + f*500,000` | + +Worked values: + +| Profile | 1280x720@30 | 1920x1080@30 | 1920x1080@60 | 3840x2160@30 | +|---|---|---|---|---| +| Balanced | 8,000,000 | 10,368,000 | 20,736,000 | 41,472,000 | +| Ultra | 9,216,000 | 20,736,000 | 41,472,000 | 82,944,000 | +| Compatibility | 2,500,000 | 5,184,000 | 10,000,000 | 10,000,000 | +| Instant | 2,666,666 | 3,500,000 | 4,000,000 | 8,000,000 | + +Keyframe interval: 22 at 30 fps, 45 at 60 fps. + +**Instant mode:** +- Screen: `MacOSFragmentedM4SMuxer` with bpp 0.15 through VideoToolbox, 2 s segments. +- Camera-only instant: AVFoundation `init_instant_mode` (instant formula, no reordering). + +**Studio camera:** `AVFoundationCameraMuxer` (`macos.rs` lines 973-995), using the Compatibility profile when the compatibility flag is set, otherwise Balanced. Reordering is off. + +AVFoundation reproduction is a platform API (`AVAssetWriter` with the keys above). There is no ffmpeg equivalent. + +### 4.4 Windows (cannot run on this VM) + +**Fragmented studio (default)** (`capture_pipeline.rs` lines 247-369, `win_fragmented_m4s.rs`): + +- `SegmentedVideoEncoder` with 2 s segments (line 232). The camera uses 3 s (line 659). +- Hardware input is false. + +| Quality | bpp | Preset | +|---|---|---| +| Ultra | 1.0 | Medium | +| Other | 0.3 | Ultrafast | + +The encoder follows the Windows priority in section 0.5, with the recording option sets from section 0.6. + +NVENC example (platform command): + +``` +ffmpeg ... -c:v h264_nvenc -preset p4 -tune ll -rc vbr -spatial-aq 1 -temporal-aq 1 -g K -b:v B -maxrate M -colorspace bt709 -color_primaries bt709 -color_trc bt709 -color_range tv -f dash ... +``` + +**Instant:** 2 s segments, `Ultrafast`, bpp 0.15. + +**Non-fragmented studio:** `WindowsMuxer` (`output_pipeline/win.rs`). + +| Setting | Value | +|---|---| +| `frame_rate` | 30 | +| `bitrate_multiplier` | 0.3 for Ultra, else 0.15 | +| `fragmented` | false | +| `frag_duration_us` | 2,000,000 | + +- Encoder: `cap_enc_mediafoundation::H264Encoder` (`crates/enc-mediafoundation/src/video/h264.rs`). + - `MF_MT_AVG_BITRATE` (line 359) = `calculate_bitrate` (lines 693-696): `W*H*((max(fps-30,0)/2)+30)*multiplier`, with fps 30. + - `MF_MT_FRAME_RATE` = `frame_rate/1` (line 366). + - No GOP, profile or rate-control mode is set, so these are Media Foundation transform defaults. +- Frame pacing at `1/frame_rate` (`win.rs` line 340) gives CFR at 30 fps. + +Worked MF bitrates (fps fixed at 30): + +| Multiplier | 1280x720 | 1920x1080 | 1920x1080 (60 fps capture) | 3840x2160 | +|---|---|---|---|---| +| 0.3 | 8,294,400 | 18,662,400 | 18,662,400 | 74,649,600 | +| 0.15 | 4,147,200 | 9,331,200 | 9,331,200 | 37,324,800 | + +**Software fallback** (`win.rs` lines 229-243): `H264Encoder::builder(video_config)` with builder defaults (bpp 0.3, `Ultrafast`, recording mode, default priority). It is used when: +- software is forced +- the primary GPU is AMD (so AMD goes through `h264_amf` with `quality=balanced`, `rc=vbr_latency` first) +- Media Foundation init or validation fails + +**Camera:** Media Foundation multiplier 0.2 at the camera fps (`win.rs` line 763). + +--- + +## 5. Desktop export + +**Location:** `crates/export/src/mp4.rs`. + +**Compression map:** + +| Compression | bpp (lines 147-152) | CRF (lines 156-161) | +|---|---|---| +| Maximum | 0.3 | 24 | +| Social | 0.15 | 28 | +| Web | 0.08 | 32 | +| Potato | 0.04 | 36 | + +**`Mp4ExportSettings`** (lines 180-190): + +| Field | Notes | +|---|---| +| `fps` | | +| `resolution_base` | | +| `compression` | | +| `custom_bpp` | | +| `force_ffmpeg_decoder` | | +| `optimize_filesize` | serde default false | + +- `effective_bpp` (line 193) is `custom_bpp` if set, else the map. +- `VideoInfo` is NV12 with `time_base = 1/fps` (lines 365-367). Frames are numbered, so output is CFR. +- Builder chain: `with_bpp`, `with_export_priority` (`is_export = true`), `with_export_settings`, `with_external_conversion`, plus `with_crf(crf)` if `optimize_filesize`. +- The builder preset stays at its default `Ultrafast`, which selects the libx264 "fast" set. +- Container: `MP4File::init("output", path, self.optimize_filesize, ...)`. `+faststart` is written only when `optimize_filesize` is true (`crates/enc-ffmpeg/src/mux/mp4.rs` lines 44-90). Default exports have the moov at the end. +- Audio: AAC 320 kbps. +- macOS: `want_hw_input = !optimize_filesize && frame is a Surface`. + +**UI defaults** (`apps/desktop/src/routes/editor/ExportPage.tsx`): + +| Setting | Value | +|---|---| +| Format | Mp4 | +| fps | 30 (options 15/30/60, lines 79-83) | +| Resolution | 720p 1280x720 | +| Compression | Maximum | +| `optimizeFilesize` | false | +| Custom bpp slider | 0.02 to 0.5, step 0.01 (lines 1408-1410) | + +- Resolution options (`Header.tsx` lines 33-37): 720p 1280x720, 1080p 1920x1080, 4K 3840x2160. +- Output size is rounded: width up to a multiple of 4, height up to even (`crates/rendering/src/lib.rs` `get_output_size`, lines 3228-3242). + +### 5.1 DESK-EXPORT-X264-{Maximum,Social,Web,Potato}: default export on Linux, or any platform's software fallback + +Encoder settings: libx264 export "fast" set plus `g=K`, `keyint_min=K`, `threads=N`. + +| Setting | Value | +|---|---| +| Rate control | ABR at B, maxrate ignored | +| GOP | K, min K/2+1 | +| B-frames | None | +| Preset / tune | veryfast, no tune | +| Effective internals | ref=1, subme=2, trellis=0, rc_lookahead=10 | +| Profile / level | High, automatic level | +| Colour | BT.709 limited | +| Container | MP4 without faststart; track time base 1/90000 | + +Verified at 720p30 Maximum: SEI identical to the render farm except `threads=4`. + +Worked B per quality (Rust model, section 0.2): + +| Compression | 1280x720@30 | 1920x1080@30 | 1920x1080@60 | 3840x2160@30 | +|---|---|---|---|---| +| Maximum | 8,294,400 | 18,662,400 | 29,859,841 | 74,649,602 | +| Social | 4,147,200 | 9,331,200 | 14,929,920 | 37,324,801 | +| Web | 2,211,839 | 4,976,639 | 7,962,623 | 19,906,559 | +| Potato | 1,105,919 | 2,488,319 | 3,981,311 | 9,953,279 | + +K is 60 at 30 fps and 120 at 60 fps. The desktop UI also offers 15 fps, where K = 30. + +**Reproduction on Linux (verified):** + +``` +ffmpeg -y -f rawvideo -pix_fmt yuv420p -s WxH -r FPS -i in.yuv \ + -pix_fmt nv12 -c:v libx264 -preset veryfast -threads N -b:v B -maxrate M \ + -bf 0 -rc-lookahead 10 -aq-mode 1 -trellis 0 -g K -keyint_min K \ + -colorspace bt709 -color_primaries bt709 -color_trc bt709 -color_range tv \ + -an -video_track_timescale 90000 desk_export_x264.mp4 +``` + +### 5.2 DESK-EXPORT-CRF-{24,28,32,36}: "Optimize file size" + +CRF forces libx264. Dictionary: `preset=slow`, `crf=<24|28|32|36>`, `pix_fmt=yuv420p` (ignored), `g=K`, `keyint_min=K`. Threading is N through the context. No bitrate is set (`bit_rate = 0`). + +| Setting | Value | +|---|---| +| Rate control | CRF | +| GOP | K, min K/2+1 | +| B-frames | x264 slow default: bframes=3, b_adapt=1 | +| Preset | slow | +| Effective internals | ref=5, subme=8, trellis=2, rc_lookahead=50 | +| Profile | High | +| Colour | BT.709 limited | +| Container | MP4 with `+faststart` | + +Verified at 720p30, CRF 24: `ref=5 subme=8 trellis=2 threads=4 bframes=3 keyint=60 keyint_min=31 rc_lookahead=50 rc=crf crf=24.0`. + +Compression maps to CRF as Maximum 24, Social 28, Web 32, Potato 36. + +**Reproduction on Linux (verified):** + +``` +ffmpeg -y -f rawvideo -pix_fmt yuv420p -s WxH -r FPS -i in.yuv \ + -pix_fmt nv12 -c:v libx264 -preset slow -crf 24 -threads N -g K -keyint_min K \ + -colorspace bt709 -color_primaries bt709 -color_trc bt709 -color_range tv \ + -an -movflags +faststart -video_track_timescale 90000 desk_export_crf24.mp4 +``` + +### 5.3 Hardware export variants (cannot run on this VM) + +All variants use the same B and M as section 5.1. + +**DESK-EXPORT-VT (macOS default):** +- Options: `realtime=false`, `profile=main`, `allow_sw=0`; b B, maxrate M. +- GOP is 12 frames because no `g` is set. There are no B-frames. +- Input is NV12 surfaces. + +``` +ffmpeg ... -c:v h264_videotoolbox -realtime false -profile:v main -allow_sw 0 -b:v B -maxrate M ... +``` + +**DESK-EXPORT-NVENC** (Windows NVIDIA, or Linux with `/dev/nvidiactl`): + +``` +-c:v h264_nvenc -preset p5 -tune hq -rc vbr -spatial-aq 1 -temporal-aq 1 -b_ref_mode middle -g K -b:v B -maxrate M +``` + +**DESK-EXPORT-QSV:** + +``` +-c:v h264_qsv -preset medium -look_ahead 1 -look_ahead_depth 20 -g K -b:v B -maxrate M +``` + +**DESK-EXPORT-AMF:** + +``` +-c:v h264_amf -quality quality -rc vbr_peak -g K -b:v B -maxrate M +``` + +**DESK-EXPORT-MF:** + +``` +-c:v h264_mf -hw_encoding true -scenario 0 -quality 0 -g K -b:v B -maxrate M +``` + +--- + +## 6. Web recorders on `main` (MediaRecorder) + +### 6.1 BR-MR-MAIN: dashboard web recorder + +**Locations:** +- `packages/recorder-core/src/recorder-constants.ts` +- `packages/recorder-core/src/recorder-utils.ts` +- `apps/web/app/(org)/dashboard/caps/components/web-recorder-dialog/useWebRecorder.ts` + +**Capture constraints** (`recorder-constants.ts` lines 9-19): `{frameRate:{ideal:30}, width:{ideal:1920}, height:{ideal:1080}}`. These are ideal values, not exact. + +**MIME candidates:** + +| List | Candidates | +|---|---| +| `WEBM_MIME_TYPES` (lines 94-97), withAudio | `video/webm;codecs=vp9,opus`, `video/webm;codecs=vp8,opus` | +| `WEBM_MIME_TYPES`, videoOnly | `video/webm;codecs=vp9`, `video/webm;codecs=vp8`, `video/webm` | +| `MP4_MIME_TYPES` (lines 82-92), withAudio | `video/mp4;codecs="avc1.42E01E,mp4a.40.2"`, `video/mp4;codecs="avc1.4d401e,mp4a.40.2"` | +| `MP4_MIME_TYPES`, videoOnly | `video/mp4;codecs="avc1.42E01E"`, `video/mp4;codecs="avc1.4d401e"`, `video/mp4` | + +**Pipeline selection** (`selectRecordingPipelineFromSupport`, lines 140-190): + +1. WebM is supported and `preferStreamingUpload !== false`: mode `streaming-webm`. +2. Otherwise the first supported MP4 candidate: mode `buffered-raw`. +3. Otherwise WebM in mode `buffered-raw`. + +`shouldPreferStreamingUpload` (lines 105-130) returns false for iOS, Firefox and Safari, and true for Chromium brands. + +**Recorder options** (`getMediaRecorderOptions`, lines 36-41): + +``` +{ mimeType, videoKeyFrameIntervalDuration: 2_000 } +``` + +No `videoBitsPerSecond` is set, so the browser default applies (Chrome: 2.5 Mbps video, 128 kbps audio). It is constructed at `useWebRecorder.ts` line 1101. + +**Timeslice:** +- `streaming-webm`: `start(1000)` (`INSTANT_UPLOAD_REQUEST_INTERVAL_MS`, line 75). +- If that throws: `start()` with manual flushing. +- `buffered-raw`: `start(200)` (line 1141). + +**What each browser records:** + +| Browser | Output | +|---|---| +| Chrome / Edge | VP9 + Opus WebM. This is not H.264. | +| Safari | H.264 MP4, `avc1.42E01E,mp4a.40.2` (Constrained Baseline 3.0 requested; Safari's own encoder) | +| Firefox | First supported MP4 candidate, else WebM | + +`videoInstantCreate` is called with `videoCodec: "h264"` (line 1056). + +**Server step:** +- Chrome VP9 is transcoded by MS-TRANSCODE-VPX: libx264 medium, CRF 23, `-bf 0`, capped at 1080p, level 4.2, CFR, AAC 128k. +- Safari H.264 at or below 1080p and level 4.2 is copied. +- `recording-conversion.ts` `convertToMp4` (line 127) is unused. + +**Reproduction on Linux.** Chrome via Playwright: + +```js +new MediaRecorder(stream, { mimeType: "video/webm;codecs=vp9,opus", videoKeyFrameIntervalDuration: 2000 }) +recorder.start(1000) +``` + +Then apply MS-TRANSCODE-VPX to the WebM (use `-i rec.webm` in place of the rawvideo input). + +**BR-MR-MAIN-MP4 (Safari-path equivalent in Chrome):** `{ mimeType: 'video/mp4;codecs="avc1.42E01E"', videoKeyFrameIntervalDuration: 2000 }` with `start(200)`. Linux Chrome has no AAC MediaRecorder encoder, so the `mp4a.40.2` variants are unsupported there. + +### 6.2 BR-MR-EXT-MAIN: Chrome extension + +**Location:** `apps/chrome-extension/src/offscreen/recorder.ts`. + +- `new MediaRecorder(recordingStream, { mimeType: pipeline.mimeType })` at lines 944-946. +- No keyframe or bitrate options. +- `start(RECORDING_TIMESLICE_MS)` with `RECORDING_TIMESLICE_MS` = 1000 (lines 70, 1086). +- In Chrome this gives VP9/Opus WebM at the Chrome default bitrate, then MS-TRANSCODE-VPX. + +Reproduction: `new MediaRecorder(stream, { mimeType: "video/webm;codecs=vp9,opus" }); recorder.start(1000)`. + +### 6.3 BR-MR-TOOLS: SpeedController and TrimmingTool + +**Location:** `apps/web/components/tools/SpeedController.tsx` and `TrimmingTool.tsx`. + +- Source: `canvas.captureStream(60)` (SpeedController line 243, TrimmingTool line 408). +- Options: `{ mimeType: selectedMimeType, videoBitsPerSecond: 5000000 }` (SpeedController line 263, TrimmingTool line 446). +- Timeslice: `start(1000)` in SpeedController (line 296), `start(100)` in TrimmingTool (line 485). + +MIME list for MP4 input (non-MP4 input uses the same WebM entries first, then `video/mp4`): + +``` +"video/mp4", "video/webm;codecs=vp9,opus", "video/webm;codecs=vp8,opus", "video/webm;codecs=vp9", "video/webm;codecs=vp8", "video/webm" +``` + +Reproduction: `new MediaRecorder(canvas.captureStream(60), { mimeType: "video/mp4", videoBitsPerSecond: 5000000 })`. The codec Chrome picks for bare `video/mp4` is browser-dependent. + +--- + +## 7. Browser WebCodecs encoders on `main` + +### 7.1 BR-WC-REMOTION: Remotion `convertMedia` + +**Packages:** `@remotion/webcodecs` 4.0.484 and mediabunny 1.49.0, unpacked to `/tmp` for reading. + +**Callers:** +- `apps/web/app/s/[videoId]/_components/timeline/recording/convert-comment-media.ts` lines 25-32: `convertMedia({ container: "mp4", videoCodec: "h264", audioCodec: "aac" })` with `installAvcLevelClamp()` active. It skips conversion when the input is already MP4 or `canConvertToMp4InBrowser` (probe `avc1.42E01F` 1280x720 plus AAC) fails. +- `apps/web/components/tools/MediaFormatConverter.tsx` lines 650-657: `convertMedia({ container: "mp4", videoCodec: "h264" })` with no clamp. + +**VideoEncoderConfig built by Remotion:** + +| Field | Value | +|---|---| +| bitrate | Not set in Chrome. Safari: 3,000,000. | +| framerate | Source fps | +| hardwareAcceleration | Tries `prefer-hardware`, then `prefer-software` | +| Keyframes | Every 40 frames | +| Codec | `avc1.64` + `00` + level from Remotion's width/height/fps table (profile byte always 0x64) | + +**Level clamp.** `installAvcLevelClamp` (`avc-level-clamp.ts`) patches `VideoEncoder.prototype.configure` to replace the codec with `pickMobileSafeAvcCodec(width, height)`: + +| Size | Codec | +|---|---| +| ≤2048x1088 | `avc1.64002A` | +| ≤4096x2304 | `avc1.640033` | +| Larger | `avc1.640034` | + +**Chromium default bitrate** when none is given (`media/base/video_encoder.cc` `GetDefaultVideoEncodeBitrate`, source only): + +``` +2,000,000 * clamp(fps,1,300) * clamp(W*H,1,64M) / (1280*720*30) +``` + +The result is clamped to at least 10,000. + +**Worked values:** + +| | 1280x720@30 | 1920x1080@30 | 1920x1080@60 | 3840x2160@30 | +|---|---|---|---|---| +| Chromium default bitrate | 2,000,000 | 4,500,000 | 9,000,000 | 18,000,000 | +| Codec without clamp | `avc1.64001F` | `avc1.64002A` | `avc1.64002A` | `avc1.640034` | +| Codec with clamp | `avc1.64002A` | `avc1.64002A` | `avc1.64002A` | `avc1.640033` | + +**Container:** MP4 written by Remotion. Audio is AAC; Linux Chrome has no AAC AudioEncoder, so comment media with audio is left as WebM there. + +**Reproduction on Linux** (Playwright, Google Chrome): + +```js +new VideoEncoder(...).configure({ + codec: "avc1.64002A", // with clamp; "avc1.64001F" etc. without + width: W, + height: H, + framerate: FPS, + hardwareAcceleration: "prefer-software", +}) +``` + +Set `keyFrame: true` every 40th frame and leave `bitrate` unset. Headless Linux Chrome rejects `prefer-hardware` (verified), so Remotion's software retry is the path that runs. + +### 7.2 BR-WC-MEDIABUNNY-LOOM: Loom browser conversion + +**Location:** `apps/web/lib/loom-browser-conversion.ts` (same on `main` and the PR). + +DASH WebM path (lines 644-662): + +```js +new Mp4OutputFormat({ fastStart: false }) +new VideoSampleSource({ codec: "avc", bitrate: QUALITY_HIGH, hardwareAcceleration: "prefer-hardware", alpha: "discard" }) +new AudioSampleSource({ codec: "aac", bitrate: QUALITY_MEDIUM }) +``` + +Conversion path (lines 857-873): + +```js +Mp4OutputFormat({ fastStart: false }) +Conversion.init({ + video: { codec: "avc", bitrate: QUALITY_HIGH, hardwareAcceleration: "prefer-hardware" }, + audio: { codec: "aac", bitrate: QUALITY_MEDIUM }, +}) +``` + +**mediabunny 1.49.0 behaviour (source):** +- Quality factors: VERY_LOW 0.3, LOW 0.6, MEDIUM 1, HIGH 2, VERY_HIGH 4. +- Video bitrate: `ceil(3e6 * (W*H/2073600)^0.95 * 1.0(avc) * 2 / 1000) * 1000`. It does not depend on fps. +- `keyFrameInterval` defaults to 2 s, by timestamp. +- `latencyMode` defaults to quality. +- Codec is `avc1.6400LL` from mediabunny's AVC level table, chosen by macroblocks and bitrate. +- The config adds `avc: { format: "avc" }`. +- Timestamps are passed through from the source (VFR). +- AAC QUALITY_MEDIUM is 128,000 bit/s. + +**Worked values:** + +| | 1280x720 | 1920x1080 (any fps) | 3840x2160 | +|---|---|---|---| +| Video bitrate | 2,778,000 | 6,000,000 | 22,393,000 | +| Codec | `avc1.64001F` | `avc1.640028` | `avc1.640033` | + +**Container:** MP4 with the moov at the end (`fastStart: false`). + +**Reproduction on Linux:** + +```js +VideoEncoder.configure({ + codec: "avc1.640028", + width: 1920, + height: 1080, + bitrate: 6000000, + framerate: FPS, + hardwareAcceleration: "no-preference", + alpha: "discard", + avc: { format: "avc" }, +}) +``` + +`prefer-hardware` is unsupported in headless Linux Chrome (verified). Force a keyframe every 2 s of timestamp. + +--- + +## 8. PR 2312 differences (`FETCH_HEAD` 549c5afa9) + +The PR does not change: +- `crates/enc-ffmpeg` +- `crates/export/src/mp4.rs` +- `crates/render-farm/src/video.rs` +- the desktop capture pipelines + +### 8.1 BR-MR-PR-SCREEN, BR-MR-PR-CAMERA, BR-MR-PR-OVERLAY: web recorder + +**Location (PR):** `packages/recorder-core/src/recorder-encoding.ts`. + +**MIME candidates:** `recorder-constants.ts` lines 96-102 add: + +| List | Candidates | +|---|---| +| `STREAMING_MP4_MIME_TYPES`, withAudio | `video/mp4;codecs="avc1.64002A,mp4a.40.2"`, `video/mp4;codecs="avc1.64002A,opus"` | +| `STREAMING_MP4_MIME_TYPES`, videoOnly | `video/mp4;codecs="avc1.64002A"` | + +**Pipeline selection** (`recorder-utils.ts` lines 139-211): +- If `preferStreamingUpload !== false`: streaming MP4, else streaming WebM. Both use mode `"streaming"`. +- Otherwise buffered MP4 (`avc1.42E01E` list), else buffered WebM. +- `getMediaRecorderOptions` is removed. + +**Bitrate tiers** (`BITRATES`, lines 45-49), in bit/s for captures with ≤1280x720, ≤1920x1088, ≤2560x1600 and larger pixel counts: + +| Content | ≤720p | ≤1080p | ≤1600p | Larger | +|---|---|---|---|---| +| screen | 2,500,000 | 4,000,000 | 6,000,000 | 10,000,000 | +| camera | 2,600,000 | 4,500,000 | 6,500,000 | 10,000,000 | +| cameraOverlay | 2,000,000 | 3,500,000 | 5,000,000 | 8,000,000 | + +- `recordingBitrate` (lines 57-74): above 40 fps, `Math.round(base * 1.5)`. +- Missing width/height default to 1920x1080. Missing fps defaults to 30. + +**Recorder options** (`recorderOptions`, lines 87-117): + +``` +{ mimeType, videoBitsPerSecond: round(tier * bitrateScale * (vp8 ? 1.3 : 1)), videoKeyFrameIntervalDuration: 2000 } +``` + +- For a `video/mp4` MIME containing `avc1.xxxxxx`, the codec is rewritten to `pickMobileSafeAvcCodec(track W, H)`: `64002A`, `640033` or `640034`. The rewrite is kept only if `isTypeSupported`. +- The `avc1.42E01E` Safari candidate is also rewritten. +- W, H and fps come from `track.getSettings()`. + +**Quality settings** (`web-recorder-dialog/recording-quality.ts`, PR): + +| Setting | Value | +|---|---| +| Default | `screenHeight` 1080, `frameRate` 30, `cameraHeight` 1080, `level` standard | +| Screen height options | 1080 / 1440 / 2160 | +| fps options | 30 / 60 | +| Camera height options | 720 / 1080 | +| `qualityBitrateScale` | high = 1.6, standard = 1 | +| Storage | `localStorage` `"cap-web-recorder-quality"` | + +**Capture constraints** (`capture-streams.ts`): `{ frameRate: { ideal: q.frameRate ?? 30 }, width: { ideal: round(h*16/9) }, height: { ideal: h } }`. `DEFAULT_CAMERA_QUALITY` is 1080@30. + +**useWebRecorder (PR):** +- The display recorder uses `recorderOptions(pipeline.mimeType, mixedStream.getVideoTracks()[0], isSupported, qualityBitrateScale(quality), recordingMode === "camera" ? "camera" : "screen")` (lines 1753-1762 of the PR file). +- The separate camera recorder records a video-only stream with `"cameraOverlay"` (lines 1764-1775). +- `selectPairedCameraPipeline` (lines 77-93): if the display pipeline is buffered-raw MP4 (Safari), the camera uses `video/webm;codecs=vp8` in streaming mode, so the 1.3 VP8 scale applies. +- Timeslice: `start(1000)` for streaming, else `start(200)`. The camera recorder uses `start(1000)`. + +**Worked videoBitsPerSecond:** + +| Content | Scale | 1280x720@30 | 1920x1080@30 | 1920x1080@60 | 3840x2160@30 | +|---|---|---|---|---|---| +| screen | standard | 2,500,000 | 4,000,000 | 6,000,000 | 10,000,000 | +| screen | high (1.6) | 4,000,000 | 6,400,000 | 9,600,000 | 16,000,000 | +| camera | standard | 2,600,000 | 4,500,000 | 6,750,000 | 10,000,000 | +| camera | high (1.6) | 4,160,000 | 7,200,000 | 10,800,000 | 16,000,000 | +| cameraOverlay | standard | 2,000,000 | 3,500,000 | 5,250,000 | 8,000,000 | +| cameraOverlay | high (1.6) | 3,200,000 | 5,600,000 | 8,400,000 | 12,800,000 | +| cameraOverlay, VP8 (Safari camera) | standard | 2,600,000 | 4,550,000 | 6,825,000 | 10,400,000 | + +At 2560x1440@30 (the PR's 1440 option), screen standard is 6,000,000. + +**Observed Chrome 148 behaviour on Linux (verified, canvas source, 1280x720, 2,500,000 bit/s, `avc1.64002A`, 2 s keyframes):** + +| Property | Observed | +|---|---| +| Reported `recorder.mimeType` | `video/mp4;codecs=avc1.640020`. Chrome picked level 3.2 itself. | +| Profile | High | +| B-frames | `has_b_frames=0` | +| Colour tags | smpte170m, tv range | +| Container | Fragmented MP4: `ftyp`, `moov`, then `moof`/`mdat` per 1 s timeslice | +| Timestamps | VFR (avg 29.8 fps) | +| Keyframes | At 0, 2.03 and 4.06 s, plus extra encoder-chosen IDRs on scene changes | +| Achieved rate | 1.94 Mbit/s on synthetic content | +| Supported withAudio MIME | `avc1.64002A,opus` (no AAC) | + +**Server step:** +- PR media server: H.264 at or below 1080p and level 4.2 has its video copied; Opus audio is transcoded to AAC 128k. +- 1440p and 2160p captures are transcoded down to 1080p with MS-TRANSCODE. The PR `process-video` still passes no size options. +- Render-farm sources from these recordings usually take the remux path, because they have 2 s keyframes and no B-frames. + +**Reproduction on Linux** (Playwright, Google Chrome, fake-camera track from `in.y4m`; see section 9): + +```js +// BR-MR-PR-SCREEN at 1920x1080@30, standard +new MediaRecorder(stream, { mimeType: 'video/mp4;codecs="avc1.64002A,opus"', videoBitsPerSecond: 4000000, videoKeyFrameIntervalDuration: 2000 }); rec.start(1000); +// BR-MR-PR-CAMERA at 1920x1080@30: videoBitsPerSecond 4500000 +// BR-MR-PR-OVERLAY at 1920x1080@30: videoBitsPerSecond 3500000 +// 3840x2160: mimeType 'video/mp4;codecs="avc1.640033,opus"' (only if isTypeSupported), bitrate per table +``` + +### 8.2 Chrome extension (PR) + +`recorder.ts` lines 1318-1340 (PR): +- `recorderOptions(pipeline.mimeType, recordingStream.getVideoTracks()[0], isSupported, 1, request.mode === "camera" ? "camera" : "screen")` for the main track. +- `"cameraOverlay"` for the camera track. +- Timeslice 1000. + +The bitrates match the standard rows of the table in section 8.1. + +### 8.3 BR-MR-CLIP-PR: editor clip recorder + +**Location:** `apps/web/lib/editor-clip-recorder.ts`. + +- `new MediaRecorder(stream, { mimeType })`, with the MIME from `selectRecordingPipelineFromSupport`. +- No bitrate or keyframe option. +- `start(1000)`. +- In Chrome this records `avc1.64002A` streaming MP4 at the Chrome default (2.5 Mbps). + +Reproduction: `new MediaRecorder(stream, { mimeType: 'video/mp4;codecs="avc1.64002A,opus"' }); rec.start(1000)`. + +### 8.4 BR-WC-EXPORT-PR: browser local export (WebCodecs) + +**Locations (PR):** +- `packages/editor-solid-web/src/browser-export-worker.ts` lines 779-806 +- `browser-export-encoder.ts` +- `browser-export-estimate.ts` +- `browser-export-audio.ts` +- `browser-local-export.ts` + +**Job inputs:** +- fps is clamped to 1-120. +- `bitsPerPixel` is `custom_bpp` or `COMPRESSION_BPP`: Maximum 0.3, Social 0.15, Web 0.08, Potato 0.04. +- `optimize_filesize`, GIF and cursor-only exports are not handled locally and go to the server. + +**Constants:** + +| Constant | Safari | Other browsers | +|---|---|---| +| `HOLDS_QUALITY_FRAMES` | true | false | +| `KEYFRAME_SECONDS` | 5 | 2 | +| `BITRATE_SCALE` | 1.35 | 1 | + +**Config, literal:** + +```ts +const bitrate = Math.round(exportBitrate(width, height, job.fps, job.bitsPerPixel) * BITRATE_SCALE); +// guarded by mediabunny canEncodeVideo("avc", { width, height, bitrate }) +const encoderConfig: VideoEncoderConfig = { + codec: avcCodecString(width, height, bitrate), + width, height, bitrate, + framerate: job.fps, + latencyMode: HOLDS_QUALITY_FRAMES ? "realtime" : "quality", + hardwareAcceleration: "prefer-hardware", + alpha: "discard", + avc: { format: "avc" }, +}; +``` + +**Codec string** (`avcCodecString`): `avc1.6400LL`, where LL is the first entry of `AVC_LEVELS` with `ceil(W/16)*ceil(H/16) <= maxMacroblocks` and `bitrate <= maxBitrate`. The table mirrors mediabunny. + +**Frames and keyframes:** +- `VideoFrame` timestamp is `trunc(frame/fps*1e6)`, and duration is `trunc(1e6/fps)`. This is CFR. +- A keyframe is requested when `floor(frame/fps/KEYFRAME_SECONDS)` changes: every 2 s, or every 5 s on Safari. +- Up to 4 frames are queued in the encoder. + +**Container and audio:** +- `Mp4OutputFormat({ fastStart: file ? "reserve" : "in-memory" })`. Either way the moov is at the front. +- Packets are re-stamped `frame/fps`. +- Audio: 48 kHz stereo, AAC at `EXPORT_AUDIO_BITRATE` = 320,000 if `canEncodeAudio`, else Opus. + +**Fallback:** +- An error before the first progress message becomes `BrowserLocalExportUnavailable`. +- `tauri-bridge.ts` lines 248-271 then switches to the server export (section 8.6). +- In headless Linux Chrome, `isConfigSupported` with `prefer-hardware` returns unsupported (verified), so the PR as written falls back to the server there. + +**Worked values (non-Safari, Maximum 0.3):** + +| | 1280x720@30 | 1920x1080@30 | 1920x1080@60 | 3840x2160@30 | +|---|---|---|---|---| +| Bitrate | 8,294,400 | 18,662,400 | 29,859,840 | 74,649,600 | +| Codec | `avc1.64001F` | `avc1.640028` | `avc1.640029` | `avc1.640033` | + +Codecs at Social, Web and Potato: `64001F`, `640028`, `640028`, `640033`. + +Safari at Maximum (×1.35): 11,197,440 / 25,194,240 / 40,310,784 / 100,776,960. + +K is 60 at 30 fps and 120 at 60 fps (Safari: 150 at 30 fps). + +**Reproduction on Linux** (Playwright, Google Chrome; substitute `hardwareAcceleration`): + +```js +VideoEncoder.configure({ + codec: "avc1.640028", + width: 1920, + height: 1080, + bitrate: 18662400, + framerate: 30, + latencyMode: "quality", + hardwareAcceleration: "no-preference", // production: "prefer-hardware" + alpha: "discard", + avc: { format: "avc" }, +}) +// encode(frame, { keyFrame: Math.floor(i/30/2) !== Math.floor((i-1)/30/2) || i === 0 }) +``` + +**Observed Chrome 148 software path on Linux (verified, 1280x720, `avc1.640028` requested, 4,976,640 bit/s):** +- avcC `profile_idc` 100 with constraint byte `0x0C` (Constrained High). +- `level_idc` 31, not the requested 0x28. +- No B-frames. +- Requested keyframes were honoured, with extra encoder-inserted IDRs on scene changes. + +### 8.5 BR-WC-AUDIO2VIDEO-PR: audio-only to video + +**Location:** `apps/web/lib/audio-to-video.ts`. + +```js +// 1280x720 canvas +new Mp4OutputFormat({ fastStart: "in-memory" }) +new CanvasSource(canvas, { codec: "avc", bitrate: 150_000, keyFrameInterval: 60, latencyMode: "realtime" }) +// frameRate 1 +// audio: AAC (Opus if unsupported) at 160_000 +``` + +- Codec: `avc1.64001F` (mediabunny). +- Keyframe interval: 60 s, which is 60 frames at 1 fps. + +Reproduction: + +```js +VideoEncoder.configure({ + codec: "avc1.64001F", + width: 1280, + height: 720, + bitrate: 150000, + framerate: 1, + latencyMode: "realtime", + hardwareAcceleration: "no-preference", + avc: { format: "avc" }, +}) +``` + +Set `keyFrame` every 60th frame. + +### 8.6 Server export and web Save (PR) + +**Server export:** +- `apps/media-server/src/lib/editor-exports.ts` validates settings with zod (fps 1-60, x ≤ 3840, y ≤ 2160, `custom_bpp` ≤ 1, `optimize_filesize` optional). +- It spawns the native `export` binary (`crates/export/src/bin/web-editor.rs` `export()`, lines 512-566). +- That binary clamps fps to `1..=60`, caps size at 3840x2160, sets `force_ffmpeg_decoder = true`, and calls `Mp4ExportSettings.export`. +- So it is identical to section 5: on a Linux server, DESK-EXPORT-X264-* by default, DESK-EXPORT-CRF-* with optimize file size, and NVENC p5 if `/dev/nvidiactl` exists. + +**Web Save** (`apps/web/lib/render-farm-start.ts`): + +| Setting | Value | +|---|---| +| Resolution | `SAVE_RESOLUTION = [1920, 1080]` | +| fps | `clamp(round(fps ?? 30), 24, 60)` | +| Format | Mp4 | +| Compression | Maximum | +| `custom_bpp` | null | + +- Rendered by RF-RENDER-NVENC in production. +- At 30 fps: B = 18,662,400, M = 27,993,600, K = 60. At 60 fps: B = 29,859,841, K = 120. +- The fallback worker save uses the same `workerSaveSettings` without `optimize_filesize`, so it has no faststart. + +### 8.7 PR-PREVIEW-X264: live editor preview stream + +**Locations:** +- `crates/editor/src/bin/web-editor-service/preview_h264.rs` +- `web-editor-service.rs` (default preview_fps 60 at line 1255; bpp 18/100 at line 990; `{"bitrate":"low"}` gives 4/100 and `{"bitrate":"high"}` gives 18/100, lines 1053-1056) + +**Configuration:** +- `VideoInfo`: RGBA, `time_base 1/1_000_000`, `frame_rate fps/1`. +- Validation: fps `1..=60`, bpp `0.04..=0.18`. +- Builder: `.with_bpp(bpp).with_encoder_priority_override(&["libx264"]).build_standalone()`. +- Preset is the builder default `Ultrafast` in non-export mode, giving `preset=ultrafast`, `tune=zerolatency`, `g=K`, `keyint_min=K`. + +**Encoder settings:** + +| Setting | Value | +|---|---| +| Rate control | ABR at B, maxrate ignored | +| GOP | 2*fps, so 120 at the default 60 fps | +| Profile | Constrained Baseline (verified) | +| Colour | BT.709 limited, from RGBA via swscale | +| Output | Raw AVC packets with avcC description, sent to the browser decoder. There is no container. | + +**Worked B:** + +| Resolution | bpp 0.18 | bpp 0.04 | +|---|---|---| +| 1280x720@60 | 7,962,624 | 1,769,471 | +| 1920x1080@60 | 17,915,904 | 3,981,311 | + +**Reproduction on Linux** (verified at 720p30): + +``` +ffmpeg -y -f rawvideo -pix_fmt yuv420p -s WxH -r FPS -i in.yuv \ + -c:v libx264 -preset ultrafast -tune zerolatency -threads N -b:v B -maxrate M -g K -keyint_min K \ + -pix_fmt yuv420p -colorspace bt709 -color_primaries bt709 -color_trc bt709 -color_range tv \ + -an -f h264 preview.h264 +``` + +The production defaults are FPS=60, K=120 and bpp 0.18. + +### 8.8 PR media server and render farm changes + +**`media-video.ts` (PR):** +- `getOrientedLimits` swaps the max width and height for portrait sources, so the limit becomes 1080x1920. +- Level selection uses the long and short edge. +- `AUDIO_RUN_ARGS = ["-chunk_duration", "1000000"]` is appended after `+faststart` when audio is present, and in `muxMediaTracksToMp4`. +- Video encoder arguments are unchanged. + +**Render farm (PR):** +- `transcode.ts` only adds `downloadSource`. +- `worker.ts` line 721 keeps `RF_TRANSCODE_ENCODER ?? "h264_nvenc"`. +- `protocol.ts` adds a `CursorTask` that runs the external `cap-cursor-service` binary. Its encoding settings are not in this repo. + +--- + +## 9. Running browser surfaces in headless Chrome with Playwright + +1. Use Google Chrome, not Playwright's bundled Chromium, which lacks proprietary H.264/MP4 support: + ```js + chromium.launch({ executablePath: "/usr/local/bin/google-chrome", args: [...] }) + ``` + `channel: "chrome"` also works. +2. Feed the source frames: + - **MediaRecorder:** convert the raw input to y4m and use it as the fake camera: + ``` + ffmpeg -f rawvideo -pix_fmt yuv420p -s WxH -r FPS -i in.yuv -f yuv4mpegpipe in.y4m + ``` + Launch with `--use-fake-device-for-media-stream --use-fake-ui-for-media-stream --use-file-for-fake-video-capture=/abs/in.y4m`. Then call `getUserMedia({ video: { width: { exact: W }, height: { exact: H }, frameRate: { exact: FPS } } })`. The fake device loops the file. Content type changes only the bitrate tier, so the same track works for the screen, camera and overlay variants. + - **WebCodecs:** serve `in.yuv` over HTTP. Slice `W*H*3/2` bytes per frame and build: + ```js + new VideoFrame(buf, { format: "I420", codedWidth: W, codedHeight: H, timestamp: Math.trunc(i/FPS*1e6), duration: Math.trunc(1e6/FPS) }) + ``` +3. Replace `hardwareAcceleration: "prefer-hardware"` with `"no-preference"` or `"prefer-software"`. Headless Linux Chrome 148 reports `prefer-hardware` unsupported, while `no-preference` and `prefer-software` (OpenH264) work. This changes the encoder compared with production on Windows/macOS. +4. Linux Chrome supports neither AAC in MediaRecorder nor AAC in `AudioEncoder`. Use the `,opus` MIME variants, or video-only. +5. Avoid `--dump-dom --virtual-time-budget` for encoder probes. Virtual time stalls the encoders. Post results from the page to a local HTTP endpoint instead (used here: about 8 s per probe). + +--- + +## 10. Ambiguities and assumptions + +- **Media-server ffmpeg version:** `oven/bun:1.4.0` plus apt `ffmpeg` is unpinned. It is probably Debian's 5.1.x but not verified. Verification here used 6.1.1. The x264 presets and the `fps_mode auto` choice of CFR for MP4 are expected to match. +- **Media-server output frame rate:** for VFR WebM input, the output rate is whatever ffmpeg guesses (`r_frame_rate`, often the stream's nominal 30/1 or a 1000/1 time base fallback). The repro uses the raw input rate. +- **VideoToolbox GOP of 12** comes from the FFmpeg 7.1 source (AVCodecContext `gop_size` default, `vt_defaults`). It is not observed on a Mac. The FFmpeg version linked into the desktop app was not checked. +- **NVENC, QSV, AMF, Media Foundation and AVFoundation:** B-frames, profile, level, VBV and lookahead not set by Cap come from encoder or preset defaults. These were not observed. +- **Rust f32 model:** f32-to-f64 rounding makes some values 1-3 bit/s off the browser model. Benchmarks should use the Rust values for Rust surfaces. +- **getDisplayMedia and getUserMedia** use `ideal` constraints, so actual capture size and fps may differ. The PR bitrate uses `track.getSettings()`, so it follows the actual size. +- **Chrome MediaRecorder** chooses its own H.264 level (it reported `avc1.640020` for 720p) and inserts scene-change IDRs. On Windows/macOS Chrome uses platform hardware encoders, whose output will differ from Linux OpenH264. +- **Chromium default WebCodecs bitrate** comes from the Chromium source. The probe's achieved rate on synthetic content was dominated by scene-change IDRs, so it is not evidence either way. +- **The `video/mp4` MIME without codecs** (web tools) leaves the codec choice to the browser. +- **Instant-mode output size** on desktop is clamped to fit within (max, max*9/16). The free/Pro limit comes from the account, which is outside the encoder code. +- **`cap-cursor-service`** (PR cursor-only render) is an external binary whose encoding is unknown. +- **libx264 dictionary keys:** the "ignored" list is based on FFmpeg option names and on observed `ref=1` in the output. A different FFmpeg build linked into the desktop app could accept other aliases. This was not checked. + +--- + +## 11. Summary table + +| surface | encoder | rate control | GOP | preset/tune | profile/level | container | repro command id | +|---|---|---|---|---|---|---|---| +| Media server processVideo (non-H.264, >1080p or level too high) | libx264 | CRF 23, no cap | x264 default 250/25, scenecut; `-bf 0` for VP8/VP9 source, else 3 B-frames | medium / none | High / forced 4.2 (5.1, 5.2 above 2048x1088) | MP4 `+faststart`, CFR (auto), AAC 128k | MS-TRANSCODE, MS-TRANSCODE-VPX | +| Media server level fix (dormant) | copy + `h264_metadata` | n/a | source | n/a | level rewritten | MP4 `+faststart` | MS-LEVELFIX | +| Media server edit render | libx264 | CRF 18 | 250/25, 3 B-frames | fast / none | High / auto | MP4 `+faststart`, CFR via fps filter, AAC 160k | MS-EDIT | +| Web Loom download fallback | libx264 | CRF 23 (default) | 250/25, 3 B-frames | veryfast / none | High / auto | MP4 `+faststart`, AAC 128k | WEB-LOOM-FFMPEG | +| Render farm render (production) | h264_nvenc | VBR B, max 1.5B (bpp model, default Maximum 0.3) | 2*fps, forced IDR, bf 0 | p1 / hq, spatial+temporal AQ | encoder default | Custom MP4 (moov first) or fMP4 HLS 2 s, AAC 320k | RF-RENDER-NVENC | +| Render farm render (override) | libx264 | ABR B, maxrate ignored | 2*fps, min fps+1, bf 0 | veryfast (ref 1, rc-lookahead 10) | High / auto | as above | RF-RENDER-X264 | +| Render farm source transcode (production) | h264_nvenc | VBR CQ 19, b:v 0 | forced IDR every 1 s, bf 0 | p1 / none | encoder default | MP4 `+faststart`, VFR passthrough, no audio | RF-TRANSCODE-NVENC | +| Render farm source transcode (override) | libx264 | CRF 18 | forced every 1 s plus 250/25, bf 0 | veryfast / none | High / auto | MP4 `+faststart`, VFR passthrough | RF-TRANSCODE-X264 | +| Desktop recording Linux (Balanced/Compat/Instant) | libx264 (h264_nvenc p4/ll if `/dev/nvidiactl`) | ABR B at bpp 0.3, maxrate ignored | 2*fps, min fps+1, scenecut 0, bf 0 | ultrafast / zerolatency | Constrained Baseline / auto (3.2 at 720p30) | DASH fMP4 2 s + HLS playlists, VFR, AAC 320k | DESK-REC-X264 | +| Desktop recording Linux Ultra | libx264 | ABR B at bpp 0.3 | 2*fps, min fps+1, bf 0 | veryfast / zerolatency | auto | DASH fMP4 2 s | DESK-REC-X264-ULTRA | +| Desktop recording macOS fragmented (default) | h264_videotoolbox (libx264 fallback) | ABR B (bpp 0.3 / 1.0 Ultra / 0.15 Compat or Instant), DataRateLimits 1.5B per s | 12 frames (not set), no B-frames | realtime, prio_speed | Main / AutoLevel | DASH fMP4 2 s, VFR | DESK-REC-VT (not runnable) | +| Desktop recording macOS non-fragmented and camera | AVFoundation | AverageBitRate (Balanced/Ultra/Compat/Instant formulas) | floor(0.75*fps); reordering only Ultra | n/a | AVFoundation default | MP4 (AVAssetWriter) | DESK-REC-AVF (not runnable) | +| Desktop recording Windows fragmented (default) | nvenc/mf/qsv/amf, then libx264 | VBR B (bpp 0.3 / 1.0 Ultra / 0.15 Instant), max 1.5B | 2*fps | NVENC p4/ll; QSV faster; AMF balanced vbr_latency; MF scenario 4 | encoder default | DASH fMP4 2 s | DESK-REC-WIN-HW (not runnable) | +| Desktop recording Windows non-fragmented | Media Foundation | AVG_BITRATE, fps fixed 30, multiplier 0.15 (0.3 Ultra) | MFT default | MFT default | MFT default | MP4, CFR 30 | DESK-REC-WINMF (not runnable) | +| Desktop export (default, Linux or software) | libx264 | ABR B: 0.3 / 0.15 / 0.08 / 0.04 | 2*fps, min fps+1, bf 0 | veryfast (ref 1, rc-lookahead 10, trellis 0) | High / auto | MP4, moov at end, CFR, AAC 320k | DESK-EXPORT-X264-{Maximum,Social,Web,Potato} | +| Desktop export, Optimize file size | libx264 | CRF 24 / 28 / 32 / 36 | 2*fps, min fps+1, 3 B-frames | slow / none | High / auto | MP4 `+faststart` | DESK-EXPORT-CRF-{24,28,32,36} | +| Desktop export macOS | h264_videotoolbox | ABR B, DataRateLimits 1.5B | 12 frames (not set) | realtime false, allow_sw 0 | Main / AutoLevel | MP4, moov at end | DESK-EXPORT-VT (not runnable) | +| Desktop export Windows/Linux NVIDIA | h264_nvenc | VBR B, max 1.5B | 2*fps | p5 / hq, b_ref_mode middle, AQ | encoder default | MP4, moov at end | DESK-EXPORT-NVENC (not runnable) | +| Desktop export QSV / AMF / MF | h264_qsv / h264_amf / h264_mf | VBR(-peak) B, max 1.5B | 2*fps | medium la 20 / quality vbr_peak / scenario 0 quality 0 | encoder default | MP4, moov at end | DESK-EXPORT-QSV, -AMF, -MF (not runnable) | +| Web recorder main, Chrome | MediaRecorder VP9 (H.264 made by MS-TRANSCODE-VPX) | browser default 2.5 Mbps | 2 s requested | n/a | VP9 | WebM streaming, 1 s slices | BR-MR-MAIN | +| Web recorder main, Safari/Firefox | MediaRecorder H.264 | browser default | 2 s requested | n/a | `avc1.42E01E` / `4d401e` requested | MP4 buffered, 200 ms slices | BR-MR-MAIN-MP4 | +| Chrome extension main | MediaRecorder VP9 | browser default | browser default | n/a | VP9 | WebM, 1 s slices | BR-MR-EXT-MAIN | +| Web tools (speed, trim) | MediaRecorder | 5,000,000 | browser default | n/a | browser choice for `video/mp4` | MP4 or WebM | BR-MR-TOOLS | +| Comment media / format converter | WebCodecs via Remotion | none (Chromium default formula; Safari 3M) | 40 frames | prefer-hardware, then software | 0x64, level table; clamp to `64002A/640033/640034` for comments | MP4 (AAC) | BR-WC-REMOTION | +| Loom browser conversion | WebCodecs via mediabunny | QUALITY_HIGH (2.778M / 6M / 22.393M) | 2 s | prefer-hardware, latency quality | `6400LL` from mediabunny table | MP4 moov at end, AAC 128k | BR-WC-MEDIABUNNY-LOOM | +| PR web recorder screen/camera/overlay | MediaRecorder H.264 | tier bit/s × 1.5 above 40 fps × 1.6 high × 1.3 VP8 | 2 s requested | n/a | `avc1.64002A` / `640033` / `640034` (Chrome picks actual level) | fragmented MP4 streaming, 1 s slices | BR-MR-PR-SCREEN, -CAMERA, -OVERLAY | +| PR editor clip recorder | MediaRecorder H.264 | browser default | browser default | n/a | `avc1.64002A` | fragmented MP4, 1 s slices | BR-MR-CLIP-PR | +| PR browser export | WebCodecs | bpp model × (Safari 1.35) | 2 s (Safari 5 s) | prefer-hardware, latency quality (Safari realtime) | `avc1.6400LL` from table | MP4 moov first, AAC 320k or Opus | BR-WC-EXPORT-PR | +| PR audio-to-video | WebCodecs via mediabunny | 150,000 | 60 frames at 1 fps | latency realtime | `avc1.64001F` | MP4 moov first, audio 160k | BR-WC-AUDIO2VIDEO-PR | +| PR editor preview stream | libx264 | ABR bpp 0.18 (low 0.04) | 2*fps (120 at 60 fps), min fps+1 | ultrafast / zerolatency | Constrained Baseline / auto | raw AVC packets | PR-PREVIEW-X264 | +| PR server export | as desktop export | as desktop export | as desktop export | as desktop export | as desktop export | MP4 | DESK-EXPORT-X264-*, DESK-EXPORT-CRF-* | +| PR web Save | as render farm render, 1920x1080 Maximum, fps 24-60 | VBR B, max 1.5B | 2*fps | p1 / hq | encoder default | Custom MP4 or fMP4 HLS | RF-RENDER-NVENC | diff --git a/capcodec/DECISIONS.md b/capcodec/DECISIONS.md new file mode 100644 index 00000000000..be394d9eb20 --- /dev/null +++ b/capcodec/DECISIONS.md @@ -0,0 +1,77 @@ +# Decisions + +Choices made where the brief was ambiguous, newest last. Each entry: the decision, then why. + +## Project + +- **Repository name `capcodec`.** Nothing more specific was obvious. The encoder library, CLI and muxers are Tov; the benchmark harness and corpus tooling are Python 3 (they are tooling, not the product, and the brief's "no JavaScript/npm workarounds" rule is about gaps in Tov). +- **The Tov feature branch is delivered as a git bundle and patch series.** The token on this machine has no push access to `richiemcilroy/tov`, so `encoder-features` lives in the local clone and is exported to the artifacts directory. +- **Speed is the primary goal** (follow-up brief): the pitch is the fastest H.264 encoder for screen recordings at quality equal to x264 veryfast. Every design choice below that trades a little compression for a lot of speed is taken in that direction, and the headline metric is encode fps vs x264 veryfast at matched VMAF. +- **The OpenAI `math` repository was reviewed and is not used.** Its relevant results (an O(n (log n)^(1-1e-13)) exact DFT, `o(n log n)` Fourier circuits for unbounded lengths, matrix-multiplication exponent bounds) are asymptotic. H.264 fixes the transforms at 4x4 and 8x8 integer butterflies and SATD uses a 4x4/8x8 Hadamard, which already cost only additions and shifts; none of the results lowers the operation count at those sizes. Re-reviewed on 2026-10-07 (all 722 preprint titles, abstracts of the Fourier, tensor and matrix-multiplication papers): same conclusion. Speed comes from vector throughput, memory traffic, fewer evaluated candidates and threads, and from video-coding practice (global motion for screen content, weighted prediction, byte-oriented CABAC, decimation and fast skip), which is what the encoder applies. + +## Benchmark harness + +- **Colour conversion.** Lossless RGB captures are converted once to 8-bit 4:2:0 with BT.709 matrix, limited range, Lanczos scaling, and the VUI of every encoder is tagged BT.709/limited. This is what HD screen recordings and Cap's pipelines use. +- **GOP for reference encoders is 2 seconds** (`-g 2*fps`, `keyint_min 1 s`) so that keyframe cost matches Cap's 2 second chunking and HLS segments. Production Cap baselines keep their own production GOP. +- **Rate ladders.** x264: CRF 18/23/28/33. capcodec: QP 20/25/30/35 until rate control exists. Bitrate-driven encoders (OpenH264, browser encoders, hardware) use a ladder relative to an anchor, x264 veryfast's bitrate at CRF 23 on the same source, times 0.35/0.6/1/1.7/3. This keeps every curve overlapping the reference on clips whose bitrates differ by 100x (idle vs full motion). +- **BD-rate** uses piecewise cubic (PCHIP) interpolation of log-rate against quality over the overlapping quality range (the method of current codec evaluations), averaged over clips. PSNR is weighted YUV (6Y+U+V)/8, each component computed from the mean MSE over all frames, capped at 100 dB. +- **Matched-quality speed.** The target quality per clip is x264 veryfast at CRF 23. Each capcodec preset is re-encoded at the QP interpolated to the same VMAF, rounded towards higher quality, and both encoders are timed in serial runs using all cores. Single-thread CPU speedups are reported beside it. +- **Quality runs are single-threaded and parallel across cores; speed runs are serial and use each encoder's own threading.** CPU time and peak RSS come from `wait4` rusage. Production Cap reproductions keep their production threading. +- **VFR outputs are compared at the source frame rate**: decoded frames are mapped to source frames by presentation time (or by the encoder's frame map for elementary streams), repeating the last decoded frame, which is how a player shows a variable-frame-rate stream. +- **Regression gate**: smoke fails if capcodec-medium's BD-rate (PSNR-YUV vs x264 veryfast) is more than 1.0 percentage point worse than the committed `bench/best.json`, if its median single-thread fps drops more than 10%, or if any decode check fails. +- **Stage timing is batched.** Stages run as separate passes over each macroblock row (analysis, motion search, mode decision, transform/quant/reconstruction, entropy coding, deblocking), so the encoder reads the clock a few times per row instead of per macroblock. This costs well under 0.1% and makes the per-stage breakdown exact rather than sampled. +- **Change-proportional cost** is measured in single-threaded speed runs: static frame time (frames where nothing changed) against an explicit full-analysis run of the same clip (every macroblock forced through analysis, no frame dropping), plus the pixel-difference fallback without side info. +- **Colour tags go on the raw input, not the output.** Tagging only the output (`-colorspace bt709` after `-i`) makes ffmpeg insert a BT.601-to-BT.709 conversion in front of the encoder, so every ffmpeg-based encoder compressed a colour-shifted picture and was scored against the original (x264 lost about 5 dB PSNR-Y). Found by checking harness PSNR against a direct numpy computation; all ffmpeg adapters now tag the rawvideo input, and their cache identity carries a version marker so old encodes are discarded. +- **Production capcodec presets are the `--unchecked` build**, as Rust release builds are; `capcodec-medium-checked` (bounds and overflow checks on) runs beside them in every tier so that any memory-safety bug traps in the harness instead of corrupting output. +- **The OpenAI `math` repository** — see above; nothing applicable. + +- **Texture AQ off by default (2026-10-07).** The text-aware AQ lowered QP by 3 on text macroblocks and raised it by 2 on textured ones. A harness sweep (standard tier, five clips, `/tmp/std8`) showed the texture raise costs VMAF on video (full_motion BD +39.0% with it, +35.3% without) and slides (−6.9% vs −8.3%) and changes nothing on the screen clips, while the OCR tier showed the text reduction lowers character error on scrolling text by about a third at equal bitrate. Default: text −3 kept, texture raise 0 (`--aq-texture N` restores it). Refresh and lifetime AQ stay on: dropping refresh costs slides +9 points, dropping lifetime AQ costs busy_ui/code_editor +2.5 points. + +- **Benchmark builds target x86-64-v3 (AVX2).** `bench/build_encoder.sh` builds `build/capcodec` and `build/capcodec-unchecked` with `--target-cpu x86-64-v3` and keeps `build/capcodec-unchecked-baseline` for SSE2-only CPUs. x264 dispatches to AVX2/AVX-512 at run time on this machine, so comparing against a baseline build would understate capcodec; the AVX2 build is 10-18% faster (full_motion 23.6 -> 27.8 fps, text_scroll 154.7 -> 170.5 fps, identical output). On Arm (Apple silicon) the vectors compile to NEON either way. A shipping CLI would pick between the two builds at start-up. +- **Benchmark builds carry no unwind tables or symbols (Linux, without `-g`).** `bench/build_encoder.sh` adds `-fno-asynchronous-unwind-tables -fno-unwind-tables -Wl,-s` through `TOV_CFLAGS`, and `-fuse-ld=lld` when lld is installed: nothing unwinds a Tov program's stack and the instructions are unchanged, while the files shrink by 37.8 KB unchecked and 45.4 KB checked (460,240 and 608,256 bytes). A `-g` build keeps both for debuggers and profilers. + +- **Lookahead QP from macroblock lifetime, buffered in the CLI (2026-10-07).** `--lookahead N` (default 10 on fast, medium and slow; 0 on live, which must not add delay) reads N frames ahead and sets each macroblock's QP from how long its content stays, x264's macroblock tree adapted to screen content. The ring buffer and analysis live in the CLI (`src/lookahead.tov`, `enc.setLookahead()` before each `encodeFrame`) so the encoder's frame flow stays as it is for B-frame scheduling; with more than one thread the analysis runs on the read-ahead thread. Only changed macroblocks are analysed (side-info dirty rects and scroll rects, otherwise the pixel-difference test). Intra cost is the SAD against the block mean; inter cost is the best full-pel SAD over zero, the scroll hint, neighbour and previous-frame motion and a two-step diamond, on even rows. The kept fraction 1 - inter/intra carries each macroblock's cost back along its motion over the window (x264's propagate step), and content still unchanged at the window end is assumed to last to the horizon, weighted by kept^12 where it changed. QP offset -3 + 1.75 log2(span / lifetime), at most +8, span = min(keyint, 2 s). IDR frames skip lifetime AQ (it counts the same persistence twice); P frames keep it (full_motion 1.9 points better) and text AQ stays. Measured alternatives (standard tier, VMAF BD-rate, five clips): N=30 gains 0.1-0.25 points over N=10 and N=60 no more, for 3-6x the delay and memory; strength 1.5 or 2, base -2.5 and a +10 cap are worse; applying it to IDR frames only loses 2-8% on full_motion; ending unchanged content at the window edge halves the gain; text AQ off under the lookahead leaves -1.9% instead of about -4%, and a floor on text macroblocks' offsets is also worse; a wider search (steps 8 to 1) and a half-resolution model with DC/H/V intra costs leave full_motion at -0.7% and +0.6%. Without the 2 s span cap a 250-frame GOP lost 8% on full_motion; with it the loss is about 2%. +- **Benchmark builds target x86-64-v3 (AVX2).** `bench/build_encoder.sh` builds `build/capcodec` and `build/capcodec-unchecked` with `--target-cpu x86-64-v3` and keeps `build/capcodec-unchecked-baseline` for SSE2-only CPUs. x264 dispatches to AVX2/AVX-512 at run time on this machine, so comparing against a baseline build would understate capcodec; the AVX2 build is 10-18% faster (full_motion 23.6 -> 27.8 fps, text_scroll 154.7 -> 170.5 fps, identical output). On Arm (Apple silicon) the vectors compile to NEON either way. A shipping CLI would pick between the two builds at start-up. On x86-64 it also builds `build/capcodec-unchecked-v4` (`--target-cpu x86-64-v4`, AVX-512, identical output). The kernels stay 256-bit in both: Tov has no compile-time query of the target's vector width, and a 32-lane 16x16 SATD that is 10-20% faster under AVX-512 costs 3.4% more instructions and runs slower under AVX2 (register spills), while 64-byte SAD is no faster than the two-row 32-byte form on AVX-512. + +## B-frames + +- **One reference per list; adaptive `--bframes 3` by default (fast, medium and slow since `9bce84f`, live since `c6225e5`; 0 with `--zerolatency`).** Each B-frame is coded after its future anchor with L0 = previous anchor and L1 = next anchor (with the pyramid, the middle B-frame replaces one of them for the outer B-frames, see below). Macroblock types are B_Skip, B_Direct_16x16 (spatial direct with `direct_8x8_inference`, colZero from the anchor's vectors), and 16x16 L0, L1 and Bi; no B sub-partitions. POC type 0 is only used when B-frames are on, so `--bframes 0` streams stay byte-identical to the P-only encoder. Live first stayed at 0 because B-frames add up to `bframes` frames of latency and reordering; it uses them since `c6225e5` (see "Adaptive B-frames on every preset" below). The screen clips gain nothing from them (adaptive B usage keeps them within ±0.5% of P-only). See PROGRESS.md for the recommended settings. +- **B-pyramid (`--b-pyramid`, default on fast, medium and slow, off on live).** In runs of 2 or 3 the middle B-frame (index (n-1)/2) is coded first as a reference (`nal_ref_idc` 2) at the anchor's macroblock QP + 5 (`--b-ref-qp-offset`; 4 and 6 were 0.1-0.3 points worse), and the outer B-frames predict from it: the earlier ones with L1 = middle, the later ones with L0 = middle. The next anchor reorders L0 back to the previous anchor and frees the middle B-frame with MMCO 1, so the P chain and its references are unchanged; the decoded picture buffer grows by one frame, `max_num_reorder_frames` is 2 and fragmented MP4/HLS delay composition by two frame durations. full_motion VMAF/PSNR BD-rate against no pyramid (120 frames, CRF 22-34, with the B search gates below): medium −2.26/−0.66%, fast −2.31/−0.59%, slow −1.15/−0.18%, live b3 −2.69/−1.02%, for about 1.5% more instructions. The outer B-frames wait for the middle one's padded picture and half-pel tiles, so on 4 threads part of the run is serial (CPU +11%, wall −5% at load ~20 before the gates). Live keeps it off because it doubles the decoder's reorder depth. The lookahead does not credit the middle B-frame; it inherits the anchor's lookahead offset through the anchor's macroblock QP. +- **B-frames are not deblocked.** B slices signal `disable_deblocking_filter_idc` 1, including the pyramid's reference B-frame and with `--recon`. At the B QP the filter cost more bits than it gave back in VMAF: full_motion BD-rate −0.26/−0.16% (medium), −0.27/−0.13% (fast), −0.37/−0.26% (slow), −0.25/−0.07% (live b3) without it, and ffmpeg decodes full_motion 12% faster. +- **The B-frames of a run are coded concurrently and alongside the next anchor.** They read only the anchors (and the middle B-frame), so each gets its own reconstruction and per-macroblock state (the source is the padded pending buffer); workers claim rows from any frame of the run (a wavefront per frame) and whoever finishes a frame entropy-codes it, so the output does not depend on scheduling. With more than one thread and outside VBR the run is coded asynchronously while the next anchor is coded, and joined after that anchor's frame or at flush, which is how x264 scales B-frames: full_motion 4 threads wall medium b3 7.34 → 5.59 s at load ~12 (CPU unchanged). VBR joins the run before the next anchor because its rate control spends each B-frame's bits before choosing the anchor's QP. The price is memory: at 1080p each concurrent frame holds about 8.7 MB of macroblock state (6.8 MB of it coefficients) and a 3.5 MB reconstruction, and an asynchronous run keeps its three sources while the next three arrive, so 4 threads peak at 151.8 MiB on medium and 114.5 MiB on live b3 where main's sequential B-frames peaked at 95.5 and 65.3 MiB. With one worker the frames are coded one after another, share one slot and reconstruct into the idle frame (112.6 and 72.0 MiB against 91.1 and 60.7; the rest is the pyramid's reference frame). Entropy-coding B rows as they finish would let the coefficients live in a few rows instead of a frame. +- **Concurrent B runs on thread-efficiency's wavefront (merge of main `7189fa4`, 2026-10-08).** B rows run through the same wavefront row function as P rows but skip its row prep and per-row QP assignment: a run's macroblock QPs are set from the anchor's when it starts, and its sources were padded when they were queued. Each B-frame is entropy-coded whole by the worker that finishes its last row, so a run never takes the main thread, and the anchor coded while a run is in flight keeps the pipelined entropy coding. B workers do not claim half-pel tiles: the run computes both anchors' tiles in one parallel pass before its workers start, so the next anchor's on-demand claims on the same reference find every tile done; the pyramid's outer B-frames wait for the middle one with the adaptive spin-wait. The anchor scan peek runs when the B queue fills, which with asynchronous runs is while the previous run is in flight (13 of 14 peeks on full_motion without side info, 2 and 4 threads); it reads only the previous anchor's source and reconstruction, which the run also only reads, and writes its own buffers. Output is byte-identical to the branch before the merge with 1-4 threads, and ThreadSanitizer finds nothing in 43 scenarios. A run with one worker (one thread, or frames under 8 macroblock rows) is coded on the calling thread: spawning its only worker and joining it at once cost two thread handoffs per run, which at load 20-30 left the 1-thread full_motion encodes waiting on the scheduler (medium wall 29.7 → 11.1 s best of 5, 488 → 53 voluntary context switches; main 33.0 s and 18). Memory is unchanged by the merge; at 1080p (90 frames, CRF 24) live peaks at 80, 117 and 126 MiB on 1, 2 and 4 threads (main 67, 72, 73) and medium at 115, 150 and 159 MiB (main 92, 96, 97), `--zerolatency` at 42 and 47 MiB either way. Of live's +53 MiB on four threads, three B slots of macroblock state are about 26 MB (20 MB of it coefficients), three reconstructions about 10 MB and the asynchronous run's three sources, held while the next three arrive, about 9 MB; two threads already code the run's frames concurrently and pay almost as much. +- **Concurrent B runs with lookahead round 4 and the all-intra pool (merge of main `08b3b02`, 2026-10-09).** B candidates are buffered before analysis and never pass through the anchor encoder's `analyse`, so the encoder's source copy is current for the lookahead's change map only on an anchor coded right after an anchor (`heldAt === frameIndex - 1`); anchors after a B run compute their own codes, and the anchor scan peek's codes enter through `takeCodes`. Main's sequential B loop rebinds the encoder's planes to each B-frame with `bindSource`; a concurrent run instead slices each B-frame's source from the padded pending buffer, with the same layout, into its own `SharedState`, because the next anchor is coded on the encoder's planes while the run is in flight. B workers keep deblocking off and take the anchor's `filterRecon`, which is true whenever a B-frame can occur (keyint above 1). With `--keyint 1` and more than one thread main's frame-parallel intra pool codes every frame and no B path is reached; `flush()` drains the pool before the B queue. The 4x4 dead-zone tests (`lumaZero`, `chromaZero`) combine the branch's SAD gate (`6e2ef20`) with main's paired transform `fwd4x2`: a pair is transformed only when either block's SAD could reach outside the dead zone, and chroma tests both blocks of a pair in 8-lane vectors. All three variants are exact tests with the same output; callgrind (x86-64-v3, full_motion 31 frames CRF 24, one thread), against the SAD gate with single transforms and against main's ungated pairs: medium 4.090G / 4.137G -> 4.055G, live 3.652G / 3.701G -> 3.617G, `--bframes 0` 5.550G / 5.526G -> 5.513G instructions. In the benchmark lane full_motion's 4-thread wall clock against x264 veryfast at matched VMAF is 1.15x on medium and 1.32x on live (main 0.92x and 0.96x; PROGRESS.md). +- **Concurrent B runs on footprint's coefficient chunks and padding (merge of main `1859bac`, 2026-10-09).** Each B slot packs its macroblocks' coded 32-coefficient chunks into its own store, sized like the anchor's (the frame's worst case plus one partial block per thread and two spare), and a run resets the slot's top when it starts. B workers are fresh encoders per run, thread and frame, so their block cursors start empty; with one worker the run's frames share slot 0 and reset it after each frame's entropy coding. B macroblocks are nearly all cbp 0, so a slot touches little of its store, which is where most of the B-frames' memory went (6.8 MB of coefficients per concurrent 1080p frame). B targets and the pyramid's reference frame take the 32-pixel padding with tile guard rows, and the middle B-frame's half-pel tiles cover every tile row from the guard origin as `precomputeTiles` does. A run is joined after the next anchor's frame and before that anchor writes its co-located vectors, which are now written only while B-frames wait; an anchor coded while a run is in flight trims references only into the spare list and swaps half-plane buffers by reference, so nothing reconstructs into a frame the run reads before the join. Peak RSS (90 frames, CRF 26; main `1859bac` / `812670f` / merge, MiB): full_motion 1080p live 58 / 73 / 59, 63 / 108 / 85 and 64 / 112 / 89 on 1, 2 and 4 threads, medium 83 / 109 / 94, 87 / 141 / 116 and 89 / 144 / 120, zerolatency on 4 threads 39 / 47 / 39; busy_ui live 30 / 47 / 30 and 34 / 51 / 34 on 1 and 4 threads, medium 64 / 75 / 64 on 4; full_motion 1440p live 99 / 126 / 101 and 106 / 187 / 146, 4K live 214 / 274 / 218 and 229 / 416 / 315, 4K medium on 4 threads 327 / 543 / 439. The B-frames' extra over main at 1080p on 2 and 4 threads fell from 45-56 to 23-31 MiB. What remains of live's +25 MiB on four threads is mostly the run's three reconstructions (about 10 MB) and its three sources, held while the next three arrive (about 9 MB); medium's +11 MiB on one thread is the pyramid's reference frame with its half-pel planes. +- **No B-frame declarations for GOPs shorter than three frames.** A B run needs a P anchor after it inside its GOP, so `--keyint 1` and `2` never code one; `configureB` leaves B-frames off there, and the CLI configures them before the muxer, the lookahead and the pipelined path read the result. Before, the SPS of medium at keyint 1 declared POC type 0 with a 16-bit LSB, three reference and DPB frames, two reorder frames and `weighted_bipred_idc` 2 (8 bytes more per SPS, which an all-intra stream repeats on every frame; the slices were identical), and MP4, fMP4 and HLS carried a reorder delay. keyint 1 and 2 are now byte-identical to `--bframes 0`. keyint 3 and 4 still declare three B-frames and, on the pyramid presets, its reorder depth, though their runs are at most one and two frames long. +- **Concurrent B runs with main's read-ahead (merge of main `e20b7fe`, 2026-10-09).** Once the CLI waits for frame n, the read-ahead thread may reuse the ring slot of every earlier frame while an asynchronous run of B-frames before n is still being coded. The run reads only encoder-owned memory: its sources were copied out of the ring by `padInput` when they were submitted, and otherwise it reads the anchors' reconstructions and its own slots. The anchor-scan peek reads frame n+1's slot synchronously while frame n is submitted, when that slot cannot be reused. B workers receive the anchor's lifetime map, a CLI array that the reader rewrites when it takes frame n+2+extra, but never read it: only row prep (`assignMbQp`) and the all-intra pool do, and B rows skip row prep. ThreadSanitizer finds nothing in the 43 scenarios and in 14 more with the read-ahead running (0, 3 and 5 frames, 2-4 threads, 23-60 frames, side info, VBR, keyint 7, stdin, MP4 with `--recon`). The extra ring slots follow `enc.bframes`, so keyint 1 and 2 run as main's `--bframes 0`. Main's CABAC skip-run writer already takes B_Skip with its own contexts, and the merged `endSlice` keeps `nal_ref_idc` 2 for the pyramid's middle B-frame, which the auto-merged line would have marked non-reference. B configurations are byte-identical to `70ebc48` and to themselves with `--read-ahead 0`, 3 and 5 on 2 and 4 threads; P-only configurations are identical to main `e20b7fe`. In the benchmark lane, full_motion's 4-thread wall clock against x264 veryfast at matched VMAF rose from 1.16-1.25x to 1.35-1.40x on medium and from 1.38-1.39x to 1.44-1.50x on live (main 1.09-1.10x and 1.19-1.21x in the same sessions). The read-ahead's three extra raw frames add 9-10 MiB at 1080p and 38 MiB at 4K on 2 and 4 threads, as on main (90 frames, CRF 26; main `e20b7fe` / merge, MiB): full_motion 1080p live 58 / 59, 72 / 95 and 73 / 98 on 1, 2 and 4 threads, medium 82 / 94, 96 / 126 and 98 / 130; busy_ui on 4 threads live 43 / 43 and medium 73 / 73; full_motion 4K live on 4 threads 268 / 353. +- **Adaptive B usage (default; `--no-b-adapt` turns it off).** An input frame joins a B run only while at least half of its macroblocks change (side-info dirty rects, scroll hints excluded; without side info, the last anchor's changed macroblocks) and the last anchor itself moved without weighted prediction, scroll or global shift. Every other frame is coded at once as a P frame with its side info, and is dropped if unchanged, and a static frame ends a pending run. Screen recordings therefore keep P-mode behaviour: B-frames cannot use weighted prediction, scroll vectors or global shift, so the first rule tried ("B whenever the frame moves") lost +13.7% on slides and +8.3% on text_scroll. Anchors after a B run are coded without side info, because side info describes the change from the previous input frame and the anchor's reference is several frames back. +- **B QP is the anchor's macroblock QP + 6 (`--b-qp-offset`).** Offsets 2/4/6/8 were swept on full_motion (BD-rate keeps improving up to 12). At 6 the B-frames sit 0.8 dB PSNR below their anchors with a mean frame-to-frame PSNR step of 0.57 dB, which is less than x264 veryfast's own B/P swing (1.0 dB, 0.90 dB per step); 8 adds under one point. +- **Implicit weighted bi-prediction** (`weighted_bipred_idc` 2) whenever weighted prediction is enabled: B-frames at 1/3 and 2/3 of the anchor distance blend the anchors by POC distance (43/21), as x264's default `weightb` does. About 1% fewer bits at equal VMAF and 2% at equal PSNR on full_motion; no extra syntax. +- **Early B_Direct and gated B searches.** The presets with early exits (live, fast, medium) take B_Direct_16x16 without the L0/L1/Bi searches and intra when the direct prediction's SATD is under 96 lambda, or under 112 lambda (128 with the pyramid) when the left and top macroblocks are B_Skip or B_Direct, whose choice says the direct vectors fit the area. The 96 lambda gate alone was quality-neutral on full_motion (0.0% VMAF, −0.3% PSNR) for 13-14% less CPU; raising it for every macroblock lost 2.8% (128) and 18.5% (192). On the same presets quarter-pel refinement runs only on the list that is cheaper after half-pel (both when equal), and intra is skipped when the best inter cost is below 1.5 times the intra estimate (was 0.75). Together, against the 96 lambda gate with both lists refined: 6.6% fewer instructions on medium (10.0% with the pyramid), 7.3% (10.6%) on fast and 5.6% on live b3, with VMAF BD-rate −0.35/−0.46% (medium without/with pyramid), −0.42/−0.65% (fast) and −0.09% (live b3). With the pyramid a neighbour threshold of 144 ended 0.25 points worse than no gates (128: 0.46 points better); without it 128 ended +0.16% (medium), −0.01% (fast) and +0.32% (live b3) against no gates. Rejected: refining sub-pel only on lists whose integer cost is within 1-1.5 times the better one (−6.5% to −1.3% instructions for +1.3% to +0.2% BD-rate), bi-prediction only when the list costs are within 1.5-2 times (no change), half-pel only (1.8-8.5% BD-rate for 2-10% CPU), and capping diamond iterations (B searches converge earlier). Below macroblock QP 30 the neighbour threshold, the one-list quarter-pel and the Bi gate (that one on every preset) are off. They were tuned at CRF 22-34, where over 99% of B macroblocks end with cbp 0; at CRF 12-21 (Cap's 0.15 and 0.3 bpp bitrate targets on full_motion) residuals are coded, and medium was +1.9% VMAF / +3.4% PSNR BD-rate behind main. Removing each gate there: quarter-pel on both lists −1.0/−2.0 points, Bi always −1.0/−0.5, the neighbour threshold −0.5/−0.1, the intra skip 0.0, and deblocking B-frames +0.3/−0.1. With the QP 30 limit, CRF 12-21 against before: medium −1.95/−2.33% (now −0.16/+0.87% against main), fast −1.85/−2.24%, slow −0.85/−0.52%, live b3 −2.99/−2.95%, live b3 with the pyramid −3.23/−2.95%; CRF 22-34 within 0.1%. Instructions (31 frames) +1.7% at CRF 22 on medium and fast, 0.0% at CRF 24 on live b3, +7.9% (medium) and +4.2% (live b3) at CRF 16. A limit of 27 costs nothing at CRF 22 but keeps half the high-rate loss (+0.66/+1.50% against main). +- **B searches share the P search's cost shortcuts.** After merging intra-speed and me-speed, `searchB` uses the same visited-position map and, like `motionSearch`, compares half-pel candidates by SAD and (medium, macroblocks whose QP AQ did not lower) quarter-pel candidates by SAD, converting only the winner to SATD; on near text-free frames it uses the P search's video sub-pel settings (8 half-pel directions, SAD quarter-pel with 4 directions on live and 8 on fast). Unlike `motionSearch` it refines to quarter-pel even on presets whose P search stops at half-pel (on the cheaper list, see the previous item). Effect on full_motion: VMAF BD-rate within 0.2 points (medium b2 -26.6 -> -26.8% vs P-only), 5.1% fewer instructions for a medium b2 encode, 1.5% for live b2. +- **B macroblocks reuse the mode-decision prediction.** Direct, L0 and L1 winners are coded from the prediction built during the decision (the Bi trial is built into its own buffer) and test the 4x4 dead zone before transforming: over 99% of B macroblocks end with cbp 0. The closed-loop intra decision codes I4/I8 blocks through the same prediction buffer, so the direct prediction is saved around it when direct is the best inter mode. +- **Lifetime AQ counts streaks between analysed frames.** With B-frames only anchors are analysed, so change streaks and recency are measured against the previously analysed frame instead of `frameIndex - 1`; otherwise every continuously moving macroblock looked new at each anchor and lost lifetime AQ's +2 QP. +- **Containers.** Decode times are the sorted presentation times. Progressive MP4 stores non-negative ctts offsets and an edit list that removes the reorder delay; fragmented MP4 and HLS do the same with a constant delay of one frame duration (two with the pyramid, rounded up) and the edit list in the init segment, because the delay has to be known before the first fragment. Signed trun offsets (CMAF style) were tried first: ffmpeg's mov demuxer turns the first negative offset it meets into a stream-wide pts shift, so in HLS, where segments are read in order and adaptive B usage leaves early segments without B-frames, presentation times jumped by one frame at the first segment with B-frames. ffmpeg and Chrome (file playback and MSE) apply the edit list and present the first frame at 0. +- **Lookahead with B-frames (2026-10-08).** Since `9bce84f` fast, medium and slow use adaptive `--bframes 3` and the 10-frame lookahead together. The CLI keeps the lookahead ring and takes one map per input frame before `enc.submit()`; each buffered B candidate keeps its map in its `Pending` frame, so whichever picture becomes the anchor (at once, when a run fills, or on flush) is coded with the map of its own display index (`enc.takeLookahead()`; the ring logic stays in `src/lookahead.tov` and the CLI loop). B-frames inherit the anchor's macroblock QP + 6. Outside the pyramid they are non-reference, so propagation gives them nothing of their own, but setting their QP from the B offset alone (removing the anchor's lookahead offset) put full_motion at +9.3% vs main instead of -0.7%. Anchors are credited with the B-frames that predict from them through list 1, as x264's macroblock tree propagates B costs into both lists (list 0 is already the P-chain model): each B-frame of the run adds its bi-prediction weight (distance from the previous anchor over the anchor distance) times the co-located kept fraction from the anchor back to it, times 4, to the anchor's lifetime. Weights 0/0.5/1/2/3/4/6/8 on full_motion: -0.7/-1.2/-1.4/-2.1/-2.3/-2.5/-2.9/-2.9% VMAF BD-rate vs main (slides -12.4 to -12.7%, -12.5% at 8); webcam_overlay is flat (-8.7 to -9.0%), and the screen clips have no B runs. 4 keeps most of the gain short of the plateau. Offsets stay absolute, so CRF means less quality on video than before (full_motion at equal CRF: 23-29% fewer bits, 2-9 VMAF points below main; at the VMAF of x264 veryfast CRF 23 it needs CRF 22.4 where main needs 24.5) and more on screen content (busy_ui, code_editor, text_scroll: 4-14% more bits at 0.1-2.2 points more VMAF, CRF 31.9-32.9 where main needs 29.3-30.2). Holding the running mean of the offsets over every coded anchor macroblock since the start at or below zero (a positive mean is subtracted from each offset) keeps every clip at or above main's VMAF at every CRF (+2 to +25% bits). It leaves the screen clips byte-identical (their mean stays negative), moves full_motion from -2.5 to -2.8% (fast -2.8 to -3.2%, slow -1.8 to -2.5%) and slides from -12.7 to -11.7% (fast -12.6 to -11.9%, slow -15.0 to -14.1%), and its webcam_overlay gain (-8.9 to -12.1%) comes from main's CRF 20 point alone: without that point both are about +2%. It is not used: at equal CRF it costs 11% more single-thread CPU (full_motion +15%) because it codes video at higher quality, which fails the 5% CPU limit, although at matched VMAF it is no dearer than absolute offsets. A per-picture mean (+1.3% vs main over the five clips) and a two-sided running mean (-2.5%) give back most of the gain, and with `--bframes 0` the clamp loses on webcam_overlay (+4.1%), which keeps absolute offsets as on the P-only lookahead. Letting the lookahead's changed-macroblock count drive the adaptive B decision when there is no side info (otherwise the dirty rects already are its change map) lost 5% on a 3:2-pulldown copy of full_motion, where repeated frames left B runs, and is not used. +- **Measuring periodic GOPs.** VMAF with `n_subsample` 2 scores only even frames, which samples a periodic GOP unevenly (with one B-frame only anchors or only B-frames, with three the anchors and middle B-frames but never the outer ones), so it biased the b1/b3 offset sweeps by about 2 points. B-frame tuning used `n_subsample=1`; the harness's `n_subsample=5` is coprime with the 2-4 frame GOP periods. + +- **Adaptive B-frames on every preset, `--zerolatency` for strict real time (2026-10-08).** Live now defaults to `--bframes 3` like the other presets. Adaptive B only buffers frames while most of the frame moves, so screen recordings stay P-only and unchanged in quality, while full-motion video on live goes from +39.3% to -6.9% VMAF BD-rate vs x264 veryfast (PSNR +64.5% -> -3.7%, SSIM +80.6% -> +7.8%; `/tmp/std16`) for at most 3 frames (100 ms at 30 fps) of delay on moving content; x264 veryfast's default delay is about 13 frames. `--zerolatency` restores zero output delay (no B-frames, no lookahead) and is byte-identical to the previous live preset. A 4-frame lookahead on live gave a further -4.5 points mean (busy_ui +1.8%, code_editor +0.3%, slides -20.8%) for 4 more frames; it is left for the lookahead round that measures 1-4 frame windows. +- **Lookahead propagation on the read-ahead thread (2026-10-08).** With more than one thread, the thread that reads and analyses frame n+depth+1 also runs `take()` and the keep map for frame n+1 into double-buffered maps while frame n is coded (`22748f4`). The horizon it uses is the GOP position, periodic by display index; the main thread checks it against the encoder's and recomputes on a mismatch, so the propagation's visit tags come from an epoch counter shared by both. The propagation is a serial backward pass over the window and the read-ahead thread is idle once its read is done, so a pooled worker or `parallelFor` would only add a handoff. Main-thread instructions at 4 threads (medium, 60 frames): full_motion -9.1%, slides -29.7%; total instructions within 0.5%; output byte-identical to the main-thread version for any thread count. +- **Vector change map without side info (2026-10-08).** The lookahead's pixel-difference change map (the media server's `--noise 3 --no-sideinfo` path) compares 16- and 8-wide blocks with vectors: a pixel difference above the noise (lane maximum) or a block sum above the budget gives the same answer as the scalar `diffMb` (`17ebf21`). busy_ui at the media-server settings: 5.86G -> 3.52G instructions (the lookahead's cost over `--lookahead 0` from +83% to +9.6%); full_motion -1.9%. The encoder's own `mbDiffers` (`src/encoder.tov`) still calls the scalar `diffMb` on this path and is now 2.7G of busy_ui's 3.5G instructions. +- **Geometric tail on video frames without B-frames; no video cost model (2026-10-08).** On current main the media-server settings (medium with B-frames, `--keyint 250 --noise 3 --no-sideinfo`) no longer lose on full_motion: -1.6% VMAF BD-rate against `--lookahead 0` (busy_ui -8.3, code_editor -9.0, slides -9.9, text_scroll -1.4, webcam_overlay -14.6%). The loss is P-only: with `--bframes 0` full_motion is +2.9%. A video cost model on frames where most macroblocks change (SATD at the full-pel vector, 4 or 8 bilinear half-pel SATD positions, the best of i16 DC/V/H/plane SATD from source neighbours as intra cost, a step-8 diamond, an intra bias of 64) moved full_motion by -0.33 to +0.56% and webcam_overlay by -0.33 to +1.13% with B-frames, and gained 0.87% on P-only full_motion, for +2.7% (SATD and i16 intra costs) to +14% (with half-pel) instructions; strength 1.25 and 2.25, an offset cap of 4 and a 20-frame window gained at most 0.11% on either clip. The P-only loss is in the tail instead: content that changes in a video frame and is not seen again inside the window was credited n * q^12 frames up to the horizon (n frames left, q the kept fraction). Without B-frames a frame where at least half the macroblocks changed now uses q(1 - q^n)/(1 - q), the expected survival if every later frame keeps the same fraction (34 instead of 46 frames at q 0.98 with 58 left, 9 instead of 16 at 0.9, 1 instead of 0.01 at 0.5). VMAF BD-rate against the same lookahead without it: live `--lookahead 1/2/4` -0.46/-0.67/-0.81% (full_motion -3.2/-2.2/-1.0%, worst clip slides +0.46% at depth 1), medium `--bframes 0` -0.06% (worst +0.15%), P-only media-server settings -0.07% (full_motion -0.50%, worst slides +0.24%); full_motion is still +2.4% there against `--lookahead 0`. Ending that content at the window edge instead gains more on the long-GOP P-only setting (-0.82%) but loses on live depth 1 (+0.84%), on webcam_overlay at depth 2 (+0.71%) and with medium `--bframes 0` (+0.87%). With B-frames the geometric tail costs 0.5% on long-GOP slides (mean +0.05%), so fast, medium and slow keep the power law and stay byte-identical. +- **Live lookahead depth: default 0, `--lookahead 2` for desktop recording (2026-10-08).** Each frame of lookahead delays output by one frame (33 ms at 30 fps) and adds about 3 MB at 1080p (the raw frame in the ring). VMAF BD-rate against live (six clips, mean): depth 1 -5.9% but full_motion +1.9%; depth 2 -6.3% with every clip better (full_motion -1.0%); depth 4 -6.6% (webcam_overlay -12.6 instead of -10.9%); before the geometric tail depth 10 gave -6.3%. Callgrind instructions at CRF 26 over the six clips: -5.3/-4.9/-4.5% at depth 1/2/4 (text_scroll +3.5 to +3.8%, the rest -1.0 to -17.4%), because video and short-lived content get higher QPs; the analysis itself is 2.8% of text_scroll's and 8.7% of full_motion's instructions at depth 4. Live stays at 0 because it must not add delay; 2 is the point to offer when about 67 ms is acceptable, and 4 adds 0.24 points for another 67 ms and 6 MB. OCR character error at depth 2 and 4: text_scroll 1.81 -> 1.41/1.39%, slides 0.48 -> 0.45%, code_editor 4.70 -> 4.80/4.74%, busy_ui 3.68 -> 4.33/4.34%; busy_ui's text-region luma PSNR on the four OCR frames is -0.29 to +0.25 dB against live at matched bitrate (code_editor -0.26 to +1.06 dB), so the busy_ui difference is at least partly sampling noise but is not explained. +- **Lookahead memory stays a raw-frame ring (2026-10-08).** The +31 MB at 1080p is the ring of depth + 1 raw frames (one more with the read-ahead thread); the analysis keeps about 1.5 MB of per-macroblock state and no pixels. The ring is also the encoder's input delay: frame n is coded from its slot after n + depth has been read, so luma-only or half-resolution copies would add memory rather than replace any. Pointing the B-frame `Pending` copies at ring slots would need the ring to hold those frames for up to 3 more frames, the same memory. A ring of changed macroblocks only would shrink with static content, but peak RSS is set by full-change frames unless its allocation is shared with the encoder's buffers, and storing exact deltas needs a pixel compare (side-info dirty rects are conservative) of about 4.6M instructions per 1080p frame, +45% on live busy_ui; it is free only fused into the no-side-info change map. Tov's large typed arrays are resident only once touched, so the ring costs memory as it fills. A shorter window on medium saves 12 MB at depth 6 and 19 MB at depth 4 for +0.06% mean BD-rate, but full_motion loses 0.54 and 0.78%, so 10 stays. +- **Live reads 2 frames ahead by default (2026-10-08).** With adaptive B-frames already on, `--lookahead 2` on live takes VMAF BD-rate vs x264 veryfast from -4.3% to -8.7% mean (busy_ui +5.1 -> +1.7, code_editor +4.1 -> +0.7, slides -9.8 -> -22.1, text_scroll -14.0 -> -16.2, full_motion -6.9 -> -7.8; PSNR mean +1.0 -> -0.4, SSIM +4.2 -> +1.0; `/tmp/std17`) at matched-VMAF CPU within noise, for 67 ms of extra delay at 30 fps. full_motion PSNR/SSIM get slightly worse (-3.7 -> -1.9, +7.8 -> +12.0) and the lookahead agent measured a small busy_ui OCR change that it is investigating. `--zerolatency` keeps zero delay. +- **Change detection without side info: one vector pass, shared with the lookahead (2026-10-08).** At the media server's settings (medium, `--keyint 250 --noise 3`, no side info) the encoder compared every macroblock with the scalar `diffMb` (`mbDiffers`), 2.7G of busy_ui's 3.4G instructions over 60 frames on main `52510d7`. Four byte-identical steps replace it. (1) One vector change-map pass per frame (`49b82ed`), (2) two macroblocks per 32-byte luma row, the per-macroblock maxima from the halves of the mask and the SADs from the halves of `sadParts`, chroma one 16-byte U and V row each (`0ba6f14`). (3) The encoder takes the lookahead's map of the frame it codes (`572b08a`). The two compare different things: the lookahead frame n with frame n-1, the encoder frame n with its source copy, which only takes the macroblocks marked changed, so a macroblock that moves within the noise leaves the copy behind the input. The lookahead's map therefore has three codes (identical, changed, different within the noise) and lives in a ring of depth + 2 slots, because the read-ahead thread writes frame n + depth + 1 while n is coded; the encoder keeps a flag per macroblock for "the copy equals input frame `heldAt`", takes the map's code where the flag is set (16 flags at a time), compares the other macroblocks itself and keeps the flag where the code is identical or the macroblock is copied. Side-info frames, anchors after a B run and flushed anchors use the full pass. (4) Each pair of macroblocks is first tested for exact equality, four rows at a time with an early return (XOR and OR, then one mask), which skips the maxima and SADs on static content. Callgrind, 60 frames, one thread, millions of instructions: + +| clip | main `52510d7` | (1) | (2) | (3) | (4) | vs main | +|---|---|---|---|---|---|---| +| busy_ui | 3398.0 | 835.4 | 681.1 | 589.6 | 558.0 | -83.6% | +| code_editor | 3357.6 | 793.0 | 638.4 | 545.4 | 513.6 | -84.7% | +| slides | 3201.9 | 636.0 | 481.4 | 388.4 | 356.7 | -88.9% | +| full_motion | 12052.3 | 11941.1 | 11905.2 | 11925.7 | 11932.9 | -1.0% | +| webcam_overlay | 3722.6 | 1267.1 | 1117.3 | 1027.3 | 997.3 | -73.2% | + +The map is now the lookahead's only pass (70M of busy_ui's 558M; CABAC is 163M); with B-frames only 3 of full_motion's 17 anchors can take it, so there step (3) costs +0.17% and step (4) +0.06%. Without side info at noise 0 busy_ui goes from 956.8M on main to 563.9M and full_motion stays at 11725M. Rejected: loading all 16 rows before one exact test (0.3-0.9 points better on the screen clips, full_motion +0.23%; clang keeps the 16 row offsets in stack slots in both forms), a separate exact kernel for noise 0 (busy_ui +0.4%, full_motion -0.5%), and storing every code into an unzeroed map (+15% for the pass). Since main `7189fa4` the rows copy their changed macroblocks in the wavefront workers and the next-frame peeks (`diffFrame`) run the same vector pass, so the flags are set on the main encoder: from the codes in `analyse`, and for every macroblock the rows will copy once the frame is known to be coded. Against main `7189fa4` the merge executes -83.1/-84.3/-88.4/-1.0/-73.0% instructions at the media-server settings; wavefront frames take the peek's map, busy_ui's small changes stay off the wavefront and keep the lookahead's map at 2 threads. +- **Window tail for P-only GOPs longer than the lifetime span (2026-10-08).** Without B-frames and with `--keyint` above the 2 s span, content that a video frame replaces and that is not seen again inside the window now ends at the window edge instead of decaying geometrically (`7f60c22`). The gate keeps every case where the window edge lost (live and medium at their 2 s GOPs, above) on the geometric tail, so all preset defaults stay byte-identical. VMAF BD-rate at `--preset medium --bframes 0 --keyint 250 --noise 3 --no-sideinfo` against the geometric tail: busy_ui 0.00, code_editor 0.00, full_motion -0.50, slides -2.78, text_scroll -0.02, webcam_overlay -1.24% (mean -0.76%); against `--lookahead 0` full_motion goes from +2.37 to +1.67% and the mean from -8.66 to -9.31%; against x264 veryfast from -32.79 to -33.40%. OCR character error unchanged (text_scroll 1.33 -> 1.32%). Instructions at that setting: full_motion -0.63%, the screen clips +0.05 to +0.13%. What is left on full_motion is the same shape: content credited past the edge of a 10-frame window in a 250-frame GOP. +- **No text-aware lifetime rule for the live lookahead's busy_ui OCR result (2026-10-08).** With `--preset live --lookahead 2` busy_ui's character error is 4.33% against 3.68%. It is not text: the tokens that differ are icon glyphs that OCR reads as characters (tag icons fused with labels, "@upload" -> "e@upload", "e@camera" -> "ecamera"; calendar icons fused with dates, "OHOcti3", "Foct21"; priority bars read as "ail"; sidebar chevrons and badge counts), in all three text regions; the words next to them are read in both, the text-region luma PSNR is 0.4 to 1.9 dB higher with the lookahead at every CRF (-0.29 to +0.25 dB at matched bitrate), and the difference changes sign between CRFs (CER over 30 frames 2.49/2.55/3.02/4.32/6.92% without and 2.58/3.14/4.13/5.32/5.53% with it at CRF 20-36) and between regions at the same CRF. Most of that text does not change (the sidebar never does, the two board columns go through 8 and 3 states over the 30 frames), and most differences are one token read differently in all 30 frames, so each CRF point holds about a dozen independent renderings rather than 90. Capping the lookahead's QP raise at 0 or +2 on macroblocks the AQ classifier marks as text leaves busy_ui at 4.35/4.36% and gives up a third (-4.23% instead of -6.68% against live) or a fifth (-5.46%) of the BD-rate gain, so neither is used. diff --git a/capcodec/FINDINGS.md b/capcodec/FINDINGS.md new file mode 100644 index 00000000000..1f701fbab99 --- /dev/null +++ b/capcodec/FINDINGS.md @@ -0,0 +1,97 @@ +# Findings: capcodec against each Cap surface's encoder + +Status: 2026-10-07, interim (threads, the C ABI build and the WASM build are not in yet). Every number comes from the harness's standard tier: 1080p, 10 s of each clip, VMAF (every 5th frame), BD-rate over the overlapping quality range, `/tmp/std9` and `/tmp/std10` (results.csv, summary.json, report.html). Clips: busy_ui, code_editor, text_scroll, slides (screen recordings) and full_motion (a film playing full screen, the worst case). Negative BD-rate means capcodec needs fewer bits than the surface's encoder for the same VMAF. + +Speeds are single-core CPU frames per second for the harness's quality encodes (capcodec has no threads yet; the x264 surfaces run with `-threads 1` for these runs). Every capcodec stream decoded without error in JM and ffmpeg and matched the encoder's reconstruction bit for bit. + +The surfaces' settings come from CAP_BASELINES.md; hardware encoders (VideoToolbox, NVENC, Quick Sync, AMF, Media Foundation) cannot run on this Linux VM and are not compared. + +## Summary + +| Surface (today's encoder) | capcodec VMAF BD-rate | Bitrate capcodec needs at Cap's production point | Single-core speed, capcodec vs today | Verdict | +|---|---|---|---|---| +| Web recorder, PR 2312 (Chrome MediaRecorder H.264) | −51% (medium), −49% (live) | 0.15-0.66x | not comparable (browser) | capcodec wins on bits; needs the WASM build to replace it | +| Browser export, PR 2312 (WebCodecs) | −63% (medium), −59% (live) | 0.20-0.94x | not comparable (browser) | capcodec wins on bits; needs the WASM build | +| Desktop recording, Linux (x264 ultrafast zerolatency, ABR bpp 0.3) | −67% (medium), −63% (live) | 0.22-0.54x | ultrafast is faster on video (108 vs 23 fps), capcodec faster on screen content | capcodec wins on bits; speed depends on content | +| Desktop export and render-farm libx264 override (x264 veryfast ABR, no B-frames, 2 s GOP) | −34% (medium), −29% (live) | 0.29-0.73x on screen clips, 1.0x on video | 2.2-2.4x faster on screen clips, 0.4-0.5x on video | capcodec wins on screen content, ties on video bits but is slower there | +| Render-farm source transcode (x264 veryfast CRF 18, no B-frames, 1 s IDR) | −38% (medium), −32% (live) | 0.49-0.76x | 2.3x faster on screen clips, 0.5x on video | capcodec wins on screen content | +| Desktop export "Optimize file size" (x264 slow CRF 24-36, B-frames) | +27% (medium), +37% (live) | 1.2-1.3x on screen clips, 2x on video | 5x faster on screen clips | x264 slow wins on bits; capcodec is far cheaper | +| Media server transcode (x264 medium CRF 23, 250-frame GOP, B-frames) | +27% (medium, same GOP), +41% (live) | 1.28-1.38x on screen clips, 0.63x on scrolling text, 1.9x on video | 4-5x faster on screen clips, 1.1x on video | x264 medium wins on bits except scrolling text; capcodec costs a quarter of the CPU | +| Media server edit render (x264 fast CRF 18, 250-frame GOP) | +26% (medium, same GOP) | 1.25-1.44x, 0.61x on scrolling text | 4x faster on screen clips | as above | + +Matched-VMAF speed against x264 veryfast (wall clock, capcodec on one core, x264 on four), from `/tmp/std9`: capcodec-live busy_ui 1.62x, code_editor 1.58x, slides 0.86x, text_scroll 0.68x, full_motion 0.23x; geometric mean 0.81x (CPU time: 1.48x). + +## Per surface + +### Web recorder (PR 2312: Chrome MediaRecorder, `video/mp4;codecs="avc1.64002A"`, 4 Mbit/s at 1080p30) + +- **Does capcodec beat it?** Yes, by about half the bits at equal VMAF on every clip: busy_ui −47%, code_editor −48%, slides −75%, full_motion −34% (medium). Chrome's software H.264 encoder (OpenH264) has no screen-content tools, no CABAC and a fixed bitrate tier. +- **Speed and CPU.** Not comparable here: the browser encodes in real time at its own pace. The replacement path is the WASM build of capcodec running in the page, which waits on Tov's WASM target. +- **Where it loses.** Nowhere on quality; the integration is the open item. + +### Browser export (PR 2312: WebCodecs `avc1.6400LL`, bpp model) + +- **Does capcodec beat it?** Yes: −63% (medium) / −59% (live) BD-rate, text_scroll −80%. At Cap's production points capcodec needs 0.2x-0.94x of the bitrate for the same quality. +- **Speed.** Not comparable (headless Chrome software path); same WASM dependency as the recorder. + +### Desktop recording on Linux (x264 ultrafast, tune zerolatency, Constrained Baseline, ABR at bpp 0.3, 2 s DASH segments) + +- **Does capcodec beat it?** Yes: −67% (medium) / −63% (live) on the clips where the curves overlap (text_scroll −67%, full_motion −68%). Ultrafast disables CABAC, B-frames and most analysis. +- **Speed.** Ultrafast is the faster encoder on full-motion video (108 vs 23 fps on one core); on screen content capcodec's change-proportional coding is faster (280-330 fps on busy_ui/code_editor, ultrafast 220-240). +- **macOS and Windows** use VideoToolbox and GPU encoders, which cannot be measured here. + +### Desktop export (Linux default) and render-farm libx264 override (x264 veryfast ABR, bf 0, rc-lookahead 10, 2 s GOP) + +- **Does capcodec beat it?** Yes on screen content: busy_ui −32%, code_editor −31%, slides −48%, text_scroll −44% (medium). On full_motion medium is −16% and live +6%. +- **Production points.** At the export presets capcodec needs 0.29x (slides, Social) to 0.73x (code_editor, Potato) of the bitrate for the same quality; on full_motion about the same bitrate. +- **Speed.** 2.2-2.4x faster on screen clips on one core; about half the speed on full-motion video. +- **Integration.** Cap calls libx264 from Rust (crates/export); the drop-in needs the C ABI build (Tov `--lib`, in progress). + +### Render-farm source transcode (x264 veryfast CRF 18, bf 0, IDR every 1 s) + +- −38% (medium) BD-rate; 2.3x faster on screen clips, half the speed on video. Production renders use NVENC p1 on GPUs, which isn't measurable here. + +### Desktop export "Optimize file size" (x264 slow CRF 24/28/32/36, 3 B-frames) + +- **Does capcodec beat it?** No: +27% (medium) on average, +25% to +31% on screen clips and +78% on full_motion; it wins on text_scroll (−24%). +- **Cost.** capcodec is about 5x faster on one core on screen clips (280 vs 55 fps). +- **Where it loses.** x264 slow's RDO, trellis quantisation, B-frames and macroblock-tree rate control. + +### Media server transcode (x264 medium CRF 23, High 4.2, default 250-frame GOP, 3 B-frames) + +- **Like for like** (capcodec with `--keyint 250`): capcodec-medium needs 27% more bits on average: busy_ui +20%, code_editor +35%, slides +29%, full_motion +87%, and 36% fewer on text_scroll. With capcodec's default 2 s keyframes the gap is +95%, which is a GOP difference, not a coding one: on static screens keyframes dominate the bitrate. +- **Cost.** capcodec uses about a quarter of the CPU on screen clips (280-330 vs 65-72 fps on one core) and about the same on video. +- **End to end** (prototype in Cap branch `cursor/capcodec-integration-aa45`, draft PR #2429, 10 s 1080p typing clip from a VP9 upload): libx264 2.10 s, 309 KB, VMAF 92.9; capcodec 2.56 s, 531 KB, VMAF 93.7 (before this session's speed work and with 2 s keyframes). +- **Where it loses.** B-frames and macroblock-tree rate control (x264 veryfast without B-frames is +47.5% on full_motion; capcodec medium is +35%), and the noise of lossy uploads. + +## What is missing before capcodec can replace these encoders + +1. **Threads.** Wavefront coding is implemented and bit-identical for any thread count (`typed-planes` branch, commit `482440a`), but it is slower than one thread until Tov stops doing atomic reference counting on every typed-array field access (TOV_GAPS.md, in progress in Tov). Expected 2.5-3.5x wall clock on four cores for heavy frames. +2. **B-frames** for camera and video content (the media server, the render farm with webcam overlays) were x264's whole advantage on full_motion. They are now in, opt-in (`--bframes 1-3`, adaptive): on full_motion `--preset medium --bframes 3` needs 4.6% fewer bits than x264 veryfast for the same VMAF (P-only medium: +32.6%) and the screen clips are unchanged (standard tier, `/tmp/bf-post`, see PROGRESS.md). The surface comparisons above predate them. Still missing against x264 medium and slow: B sub-partitions, B-frames as references (pyramid), several references per list, and lookahead / macroblock-tree rate control. +3. **The C ABI build** for crates/export and the desktop app, and **the WASM build** for the browser recorder and export (both in progress in Tov). +4. **Hardware encoder comparisons** on macOS and Windows (VideoToolbox, NVENC, Media Foundation): run the harness's hardware adapters on those machines (see CAP_BASELINES.md section 9 and bench/harness/adapters/hardware.py). + +## Footprint and raw speed against x264 (2026-10-07) + +x264 built here from source (commit 0480cb0, `--enable-static`, nasm 2.16); capcodec main `ec8d3ac`. 4-core Sapphire Rapids VM, best of 3 interleaved runs on an idle machine. + +| | capcodec | x264 | +|---|---|---| +| Binary | 506 KB (498 KB stripped) | 2.50 MB (2.32 MB stripped) | +| Runtime dependencies | libc, libm | libc, libm | +| Clean build | 8.9 s (one C translation unit; 0.06 s when cached) | 17.1 s with `make -j4`, 59.7 s with `-j1`; needs nasm | +| Source | 8.8K non-blank lines of Tov, tests included | 79.7K lines of C, 60.1K lines of assembly | +| Startup, one 64x64 frame | 1.1 ms | 1.4 ms (veryfast) | +| Peak memory, 1080p | 46-53 MB | 152-249 MB (veryfast, 1-4 threads) | +| Output delay | 0 frames (N with `--bframes N`) | 3 B-frames + 10-frame lookahead (veryfast) | + +Since these measurements the fast, medium and slow presets use adaptive B-frames (`--bframes 3`) and read 10 frames ahead (`--lookahead`): up to 13 frames of output delay, about 31 MB more at 1080p, and a mean VMAF BD-rate against x264 veryfast of −3.4% (fast), −5.6% (medium) and −5.4% (slow) over busy_ui, code_editor, full_motion, slides and text_scroll; live is unchanged. At the media server's settings (medium, 250-frame GOP, `--noise 3` without side info) the lookahead gains on every clip, full_motion included (−1.6%), and since round 4 the change detection there is one vector pass per frame shared with the lookahead: single-thread instructions against main `52510d7` busy_ui −84%, code_editor −85%, slides −89%, webcam_overlay −73%, full_motion −1% (CPU at 300 frames on one thread −63%, −66%, −40%, −32% and −1%), with identical output. For desktop recording, `--preset live --lookahead 2` adds 2 frames of delay (67 ms at 30 fps) and 6 MB at 1080p for −6.3% VMAF BD-rate against live (every clip better) at 4.9% fewer instructions, and moves live from −0.4% to −4.5% against x264 veryfast over the six clips including webcam_overlay (PROGRESS.md). + +| Raw speed, 1080p | capcodec | x264 veryfast | +|---|---|---| +| busy_ui, 4 threads, no side info | 433 fps, 0.9 s CPU | 343 fps, 2.2 s CPU | +| full_motion, 1 thread (live CRF 30, 3.4 Mbit/s vs CRF 23, 3.7 Mbit/s) | 34.6 fps | 46.3 fps | +| full_motion, 4 threads | 65 fps, 11.4 s CPU | 115.5 fps, 6.6 s CPU | +| 30 keyframes (keyint 1), 1 / 4 threads | 20 / 32 fps | 65 / 210 fps | + +At similar bitrates x264's full-motion quality is much higher (VMAF BD-rate: capcodec-medium +35.5%, live +63.5%). Keyframe coding is the slowest path (scalar intra transforms, poor 4-thread scaling) but keyframe reuse hides it on typical screen recordings. diff --git a/capcodec/PROGRESS.md b/capcodec/PROGRESS.md new file mode 100644 index 00000000000..7ebe01a6f84 --- /dev/null +++ b/capcodec/PROGRESS.md @@ -0,0 +1,203 @@ +# Progress + +**Headline (main `1a1794a`; the 12-clip speed table in `SCOREBOARD.md` is still `00fecf7`, before concurrent B-frames, and `--bframes 0` matches that encoder): at matched VMAF that table is faster than x264 veryfast on every clip in every mode, on CPU and on wall clock, at 1 and at 4 threads (36/36). Geometric mean CPU 4.9-5.9x, wall 3.2-5.2x. Concurrent B-frames are now the default. On full_motion in the benchmark lane, 4-thread wall is 1.35-1.40x medium and 1.44-1.50x live (the same sessions measured the previous main at 1.09-1.10x and 1.19-1.21x). The other 11 clips were not retimed. Peak memory on the 12-clip run is 3-8x under x264. Quality on this encoder is the `fff504d` result (bitstreams match): level, 17-20 wins of 36 clip-metrics per mode. The pending quality branch `rd-next` wins VMAF on 12/12 clips and PSNR on 11/12, and still loses SSIM on 6-8 clips. SSIM round 3 (`ssim` `b72e995`, not on main; base is `rd-next` plus main `08b3b02`, 84 commits behind) lowers light flat and smooth IDR macroblocks above QP 34 and moves the 12-clip SSIM mean from about 0 to −1.4% on medium and live. It still loses SSIM on dashboard (+10.0% medium), busy UI, code, webcam, and full motion. Intra rounding and IDR rate-distortion (`intra-rd` `99822eb`, not on main) pass the 0.5% gate against main on all 12 clips in medium, live, and zerolatency. All-intra wins against x264 veryfast keyint 1 go from 7 to 12 of 36. The screen-clip SSIM and PSNR losses shrink and do not flip. All-intra CPU stays ahead of x264; full_motion is 1.01× and a round reaches 1.00×. Line and edge AQ (`dashboard` `4ca9f20`, on `rd-next`, not on main) cuts dashboard's loss to x264 veryfast from SSIM +12.8% to +6.2% and PSNR-YUV +3.2% to +1.7% in medium, and from +14.0% to +7.3% and +3.8% to +2.4% in live. Zerolatency dashboard wins all three metrics. No other clip gets worse by more than 0.5 against x264.** + +Threads: deterministic wavefront (rows claimed in order, each macroblock waits for the row above to pass its top-right neighbour), output byte-identical for any thread count. + +## Intra RD (not on main) + +`intra-rd` `99822eb` (rounding `719dab7` plus CABAC SSD choice among close I4/I8/I16 SATD candidates; `--no-intra-rd` matches the rounding commit). All-intra uses round-to-nearest above the first luma level, and textured or natural macroblocks use a 0.4 offset (chroma 0.36). In a normal GOP, IDRs keep main's rounding except lasting textured or natural macroblocks, which take the 0.4 offset. Against main, every clip in medium, live, and zerolatency is inside the 0.5% gate; the worst cells are live slides VMAF +0.28% and zerolatency webcam_overlay PSNR-Y +0.27%. Against x264, wins go from 19 to 20 (medium), 17 to 20 (live), and 19 to 22 (zerolatency). All-intra mean versus main is VMAF −13.2%, PSNR-YUV −1.0%, SSIM −1.7%. `tov test` 85/85, thread identity, TSan 0, `verify.py` 23/23. Not merged: it changes bitstreams, and full_motion all-intra speed is only 1.01× x264. + +## Dashboard lines (not on main) + +`dashboard` `4ca9f20` (docs `b74f140`) sits on `rd-next` `d0cce5e`. Macroblocks whose 16 rows or 16 columns are constant split into lines and edges. Edges take −4 QP in every frame. Lines on IDRs take −8, or −12 when the range is under 16; lines in P and B frames take −4. Flags: `--aq-stroke`, `--aq-line`, `--aq-faint-line`. full_motion stays byte-identical. The worst change outside dashboard against x264 is slow_typing medium VMAF +0.38. `tov test` 65/65, `verify.py` bit-exact on busy_ui and full_motion, 1 and 4 threads identical in 18 configurations. Instructions rise at most 0.38%. Not merged: medium and live dashboard still lose SSIM (+6.2%, +7.3%) and PSNR-YUV (+1.7%, +2.4%) to x264, and the base is `rd-next`. + +## Scoreboard vs x264 veryfast (goal: win every row) + +Quality: harness standard tier on 12 clips (1080p, 10 s) with capcodec's CRF ladder reaching 44 so both curves overlap (42-88%; the old ladder compared only 15-24% of screen-clip curves and flattered capcodec). Speed: `bench/speedcmp.py` in the benchmark lane (CRF matched to x264 CRF 23's VMAF, interleaved rounds, rest of the machine paused). Clip counts are wins out of 12. + +| Metric | main `fff504d` | quality branch `rd-next` | status | +|---|---|---|---| +| VMAF BD, medium / live / zerolatency | 8 / 6 / 7 | 12 / 12 / 12 (-12% to -81%) | win after rd-next | +| PSNR BD, medium / live / zerolatency | 5 / 6 / 6 | 11 / 11 / 12 | lose dashboard (+3.2% / +3.8%) | +| SSIM BD, medium / live / zerolatency | 5 / 5 / 6 | 4 / 4 / 6 | lose on rd-next: dashboard +12.8%, busy_ui +6.5%, busy_ui_60fps +6.4%, code_4k +5.0%, webcam_overlay +3.4%, code_editor +2.5%, slow_typing +2.2%, full_motion +1.5% (live +6.8%). `ssim` `9b329be` (not merged): medium 27/36 wins, SSIM mean −1.36% (dashboard +10.0, busy_ui +3.5, busy_ui_60fps +0.9, code_4k +3.8, code_editor +1.7, webcam +3.4, full_motion +1.5); zerolatency 31/36, slow_typing SSIM becomes a win | +| CPU at matched VMAF, 1 thread | 36/36 faster; geomean medium 5.53x, live 5.83x, zerolatency 5.39x; min 1.39x (full_motion medium) | | win | +| CPU at matched VMAF, 4 threads | 36/36; geomean 5.51x / 5.87x / 4.90x; min 1.21x (full_motion zerolatency) | | win | +| Wall at matched VMAF, 1 thread | 36/36; geomean 4.94x / 5.23x / 4.80x | | win | +| Wall at matched VMAF, 4 threads | 36/36; geomean 3.15x / 3.46x / 3.26x; min 1.06x (full_motion medium), live 1.21x, zerolatency 1.49x | | win (`bench/out/speedcmp-00fecf7`; other processes stopped, frozen median about 690); with the concurrent B-frames after merging `e20b7fe`, full_motion medium 1.35-1.40x and live 1.44-1.50x in the benchmark lane (main 1.09-1.10x and 1.19-1.21x in the same sessions) | +| Keyframe-only CPU at matched VMAF (`speedcmp.py --keyint1`, main `558481e`) | 1.07-1.75x at 1 thread, 1.06-1.76x at 4 (full_motion 1.07x / 1.06x, busy_ui 1.21x / 1.28x); 4-thread wall fps full_motion 149 vs 136 | | win on every clip; busy_ui needs 12% more bits than x264 at matched VMAF in all-intra | +| Peak memory, 4 threads | 34-308 MiB vs 141-1304 MiB on the `00fecf7` rerun | | win | +| Delay before output | live 2 frames (5 with B-frames), medium 10 (steady 13), zerolatency 0 | | win vs x264 at 4 threads (17) and auto (19); medium steady lag 13 vs 10 at 1 thread | +| Startup | 0.9-1.2 ms vs 1.3-1.5 ms (64x64 frame); first 1080p frame 30.5 vs 54.9 ms at default threads | | win; 1-thread medium first 1080p frame 48.9 vs 40.3 ms (keyframe speed) | +| Decode CPU of the output at matched VMAF | 0.58-1.54 s vs 1.59-2.40 s (dashboard, dark_mode, webcam_overlay, slow_typing, idle) | | win | +| Binary | 550 KB vs 1.29 MB (minimal x264 build) / 2.5 MB (CLI) | | win | +| Build CPU time | about 12 s vs 36.5 s (minimal x264) / 71 s (full) | | win | +| Bitrate accuracy at Cap targets (10 s, reachable) | two-pass 4.9% vs 5.5%; one-pass 7.7% vs 5.5% | | two-pass win, one-pass lose (rate-control round in progress) | + +## State + +| Milestone | State | +|---|---| +| 0 Benchmark harness | Done. Corpus: 12 clips including full-motion video (Big Buck Bunny, CC BY 3.0) and 4K; scripts and manifest in `bench/corpus/`, sources outside git. | +| 1 Minimal stream | Done | +| 2 Full intra | Done | +| 3 CABAC | Done; byte-oriented arithmetic coder, bit-exact with the spec engine (`src/cabac_ref.tov`, randomised equivalence tests) | +| 4 P frames | Done; global shift candidates (column/row profile matching) find slide push/cover transitions | +| 5 Screen-aware side info | Done; IDR frames reuse unchanged macroblocks of the previous IDR (decisions, coefficients and reconstruction) | +| 6 Text-aware AQ | Done (defaults under review) | +| 7 Rate control | Done; weighted prediction for fades added; lookahead QP from macroblock lifetime (`--lookahead`, 10 frames on fast/medium/slow) | +| 8 Speed | SIMD kernels (SAD, SATD, SA8D, half-pel, transforms, quantisation, deblocking, classification, hashing, chroma MC) and wavefront threads; IDR entropy and deblocking still serial | +| 9 Integration | Media server prototype on Cap branch `cursor/capcodec-integration-aa45` (draft PR #2429); C ABI and WASM wait on Tov | +| Muxers | Done: MP4 faststart, fragmented MP4, HLS | +| B-frames | Done: `--bframes 0-3` (default 3, 0 with `--zerolatency`), adaptive by default (`--no-b-adapt`), B-pyramid (default with the B-frames except on live, `--no-b-pyramid`; the middle B-frame of a run is a reference at the anchor's QP + 5), `--b-qp-offset` (6); one reference per list, B_Skip / B_Direct_16x16 / 16x16 L0, L1, Bi; the B-frames of a run are coded concurrently and alongside the next anchor; MP4, fMP4 and HLS (edit lists remove the reorder delay) | +| Read-ahead | On main. More than one thread already overlapped the next frame's read and lookahead (one extra ring slot) at `fff504d`. `f4781c5` (`cli/main.tov` only) reads further frames while a B run is coded (`--read-ahead`, default 3; off when B-frames are off). `a9b86d5` matched `/tmp/capcodec-main-latest18` in 459/459 configurations (verify.py 10/10, tov test 81/81, TSan 0 reports). A pin7 rebuild of `f4781c5` matched that baseline on busy_ui and full_motion for default, live, zerolatency, fast, `--bframes 0`, `--lookahead 0`, bitrate and `--keyint 1` at 1 and 4 threads (32/32; full_motion default and live also matched at 90 frames). `--read-ahead 0` and `5` are new flags; on these clips they hash the same as the default encode. Retimed on `00fecf7` with other processes stopped: full_motion 4-thread wall is 1.06x medium and 1.21x live (was 0.89x and 0.97x) | + +Every capcodec bitstream decodes without errors in JM 19 and ffmpeg and is bit-exact with the encoder's reconstruction (every harness tier, and `tools/verify.py` after each change). B-frames, after merging intra-speed and me-speed: 51 verify cases (bframes 1-3 x live/medium/slow x CABAC/CAVLC, keyint 7, stream ends 1-15 frames, ABR, constant QP, fixed pattern, no side info, adaptive runs on five screen clips, 300-frame encodes of four clips) are JM and ffmpeg bit-exact and identical for 1 and 4 threads; `--bframes 0` is byte-identical to main `dc8d9fb` (full_motion, busy_ui, slides x live/medium x 1/4 threads x CABAC/CAVLC, and MP4/fMP4/HLS); ffmpeg presentation and decode checks pass in all three containers (bframes 1-3, VFR drops), and Chrome 148 plays MP4, fMP4 and HLS (MSE) with B-frames in order. B-frames round 2 (speed, concurrent runs, pyramid; `6765509`): the 53-case verify matrix (the same kinds of cases as above, now 53) is JM and ffmpeg bit-exact and identical for 1 and 4 threads with and without the pyramid, every speed step that claims it is byte-identical (P-only, slow and screen clips where the decisions do not change), the reconstruction written with `--recon` matches ffmpeg's decode, and the container checks pass with reorder depth 2. After merging main `08b3b02` (`149f001`): B-frame configurations are byte-identical to the branch at `c12ee19` (36 configurations with 1-4 threads, and `--keyint 1` with the default B-frames), P-only configurations to main `08b3b02` (16 configurations with 1-4 threads, including the window tail at `--bframes 0 --keyint 250` and `300`, and all-intra), the 53-case matrix is JM and ffmpeg bit-exact and identical for 1 and 4 threads, ThreadSanitizer finds nothing in 43 scenarios, and the container checks and `tov test` pass. After merging main `1859bac` (`f1c6e24`): B-frame configurations are byte-identical to `812670f` (36 configurations with 1-4 threads, and footprint's odd sizes, synthetic pans, mixed content and live / zerolatency at 1080p, 1440p and 4K), P-only configurations to main `1859bac` (15 configurations with 1-4 threads), the 51-case matrix is JM and ffmpeg bit-exact and identical for 1 and 4 threads, ThreadSanitizer finds nothing in 43 scenarios, and the container checks and `tov test` pass. After merging main `e20b7fe` (`0c662ea`): B-frame configurations are byte-identical to `70ebc48` (the same 36 configurations and footprint sweeps) and to themselves with `--read-ahead 0`, 3 and 5 on 2 and 4 threads (10 configurations), P-only configurations to main `e20b7fe` (15 configurations with 1-4 threads), `--keyint 1` and `2` with the default B-frames to its `--bframes 0` (3 clips x 3 presets x 1-4 threads), the 51-case matrix is JM and ffmpeg bit-exact and identical for 1 and 4 threads, ThreadSanitizer finds nothing in 57 scenarios (14 of them with the read-ahead running), and the container checks and `tov test` (86) pass. + +## Numbers + +Standard tier, 1080p 10 s, VMAF BD-rate vs x264 veryfast (negative = fewer bits than x264 for the same VMAF), 2026-10-07 07:21, before weighted prediction: + +| clip | capcodec-live | capcodec-medium | +|---|---|---| +| busy_ui | +9.9% | +9.3% | +| code_editor | +10.5% | +7.5% | +| slides | +7.0% (was +65%) | +5.5% (was +57%) | +| text_scroll | −7.5% | −7.8% | +| full_motion | +67.7% | +39.0% | + +Since then: weighted prediction (slides −6.6% live / −6.9% medium, now fewer bits than x264 veryfast) and texture AQ off by default (capcodec-medium: busy_ui +9.4%, code_editor +7.5%, slides −8.3%, text_scroll −7.8%, full_motion +35.3%; mean +7.2%). + +Where full_motion loses: x264 veryfast's B-frames. Without them x264 is +47.5% on full_motion (and +0.0%/+2.2% on busy_ui/code_editor), so capcodec-medium's P-only +35.3% beats x264 veryfast without B-frames by about 12 points (`/tmp/std7`, `/tmp/std8`). capcodec's own B-frames close the gap (next section). + +### B-frames + +Standard tier, B-frames round 2 (2026-10-08, `/tmp/bf-r2`, branch `bframes` at `ca086f8`, which `6765509` reproduces byte for byte; variants via `CAPCODEC_VARIANTS`, side info on, lookahead on except on live). fast and medium are the defaults (three adaptive B-frames with the pyramid). VMAF BD-rate vs x264 veryfast, PSNR (YUV) in brackets: + +| clip | live | live b3 | live b3 pyramid | fast | medium | medium no pyramid | medium P-only | +|---|---|---|---|---|---|---|---| +| full_motion | +39.3% (+64.5%) | −8.5% (−4.4%) | −11.0% (−5.2%) | −14.3% (−7.9%) | −15.0% (−11.1%) | −12.7% (−10.3%) | +25.5% (+41.5%) | +| busy_ui | +5.1% | +5.2% | +5.2% | +1.7% | +1.3% | +1.3% | +1.3% | +| code_editor | +4.1% | +4.1% | +4.1% | −0.0% | −2.8% | −2.8% | −2.9% | +| slides | −9.8% | −9.8% | −9.8% | −21.1% | −20.9% | −20.9% | −20.7% | +| text_scroll | −14.0% | −14.0% | −14.0% | −16.3% | −16.1% | −16.1% | −16.2% | +| mean | +4.9% (+14.8%) | −4.6% (+0.9%) | −5.1% (+0.7%) | −10.0% (−1.3%) | −10.7% (−2.7%) | −10.3% (−2.5%) | −2.6% (+7.9%) | + +Against P-only of the same preset (VMAF/PSNR): full_motion medium −34.1/−37.8%, medium without the pyramid −32.4/−37.1%, live b3 −34.5/−41.7%, live b3 with the pyramid −36.2/−42.2%; the screen clips stay within ±0.1% because adaptive B usage codes them almost entirely as P frames. Main before round 2 (`0c7af6c`, same harness, full_motion, `/tmp/bf-r2-main`): medium −12.2% (−10.0%), live b3 −6.9% (−3.7%). Against main on `tune.py` (full_motion, 120 frames, CRF 22-34, VMAF/PSNR BD-rate): medium −2.94/−1.48%, medium without the pyramid −0.45/−0.34%, fast −4.08/−2.29%, slow −2.22/−1.21%, live b3 −1.50/−0.65%, live b3 with the pyramid −4.27/−1.68%. + +Matched-VMAF speed vs x264 veryfast CRF 23 (capcodec CRF from the harness: full_motion live 25, live b3 24, fast and medium 22, and main's medium 22 and live b3 24; busy_ui live 29, fast and medium 32; code_editor 31 and 32; slides 30; text_scroll 30 and 33), 300 frames, interleaved best of 3 per variant, 1 and 4 threads for both encoders, load average 21-27 on the 4-core VM (`/tmp/bfdbg/abtime_r2final.log`, `/tmp/bfdbg/abtime_r2screen.log`): + +| | live | live b3 | live b3 pyramid | fast | medium | medium no pyramid | main medium | main live b3 | +|---|---|---|---|---|---|---|---|---| +| full_motion, 1 thread, CPU | 1.04x | 1.37x | 1.32x | 1.18x | 1.11x | 1.07x | 0.91x | 1.01x | +| full_motion, 4 threads, CPU | 0.82x | 1.14x | 1.10x | 1.03x | 0.99x | 1.02x | 0.78x | 0.89x | +| full_motion, 4 threads, wall | 0.45x | 1.03x | 1.21x | 1.04x | 1.03x | 0.98x | 0.48x | 0.55x | +| 5 clips geometric mean, 1 thread, CPU | 3.77x | 3.84x | 3.84x | 3.56x | 3.47x | 3.49x | | | +| 5 clips geometric mean, 4 threads, CPU | 3.39x | 3.55x | 3.58x | 3.32x | 3.23x | 3.27x | | | +| 5 clips geometric mean, 4 threads, wall | 1.87x | 2.07x | 2.35x | 1.98x | 1.94x | 1.83x | | | + +Main was timed on full_motion only. x264's best 4-thread wall time on full_motion was 1.6x its CPU time (it got 0.63 cores at this load), so the 4-thread wall ratios mostly measure the load; the full_motion CPU ratios repeat within about ±3%. On the screen clips live and live b3 make the same decisions on nearly every frame (BD-rate within 0.1%), so their CPU differences of up to 9% (text_scroll, about 1 s encodes) are load noise. Callgrind (x86-64-v2, full_motion 31 frames CRF 24) against main: medium 7.516G → 5.664G instructions (−24.6%), live b3 7.116G → 5.126G (−28.0%); the steps are in DECISIONS.md under B-frames. Peak RSS at 1080p: medium 112.6 MiB on one thread and 151.8 MiB on four (main 91.1 and 95.5), live b3 72.0 and 114.5 MiB (main 60.7 and 65.3); the extra on four threads is the state of concurrently coded B-frames. + +Merging thread-efficiency (main `7189fa4`) took 5-8% off the 4-thread CPU time of the B-frame encodes (pipelined entropy, row prep inside the wavefront, adaptive spin-waits) and left 1-thread CPU unchanged (`/tmp/bfdbg/timing_m7.log`, `timing_m8.log`, timed at load 29-57). + +After merging main `08b3b02` (`149f001`), in the benchmark lane (`bench/lane.sh run "python3 bench/speedcmp.py ..."`: capcodec's CRF matched in quarter steps to the VMAF of x264 veryfast CRF 23, 300 frames with side info, five interleaved rounds, the rest of the machine paused for each timed run), two sessions for each binary (`/tmp/bfdbg/m9/lane-merged`, `lane-merged-r2`, `lane-main`, `lane-main-r2`). x264's median time over capcodec's (above 1 is faster), first / second session: + +| clip, mode | CPU, 1 thread | main | wall, 1 thread | main | CPU, 4 threads | main | wall, 4 threads | main | +|---|---|---|---|---|---|---|---|---| +| full_motion medium | 1.46 / 1.49x | 1.32 / 1.27x | 1.39 / 1.42x | 1.27 / 1.21x | 1.31 / 1.33x | 1.17 / 1.16x | 1.15 / 1.16x | 0.92 / 0.92x | +| full_motion live | 1.69 / 1.63x | 1.41 / 1.40x | 1.62 / 1.56x | 1.35 / 1.34x | 1.47 / 1.50x | 1.19 / 1.21x | 1.32 / 1.33x | 0.96 / 0.97x | +| full_motion zerolatency | 1.41 / 1.41x | 1.42 / 1.43x | 1.32 / 1.34x | 1.26 / 1.36x | 1.00 / 1.00x | 1.01 / 0.99x | 1.19 / 1.22x | 1.07 / 1.16x | +| webcam_overlay medium | 3.79 / 3.76x | 3.78 / 3.83x | 3.42 / 3.40x | 3.40 / 3.46x | 3.85 / 3.68x | 3.69 / 3.66x | 2.63 / 2.55x | 2.52 / 2.45x | +| webcam_overlay live | 4.09 / 4.08x | 4.16 / 4.00x | 3.09 / 3.68x | 3.76 / 3.62x | 3.84 / 3.93x | 4.09 / 3.90x | 2.32 / 2.75x | 2.90 / 2.76x | +| webcam_overlay zerolatency | 4.38 / 4.46x | 4.41 / 4.42x | 4.00 / 4.09x | 4.05 / 4.03x | 3.94 / 3.90x | 3.87 / 3.87x | 3.65 / 3.60x | 3.57 / 3.65x | +| busy_ui medium | 6.53 / 6.61x | 6.52 / 6.28x | 5.75 / 5.77x | 5.76 / 5.53x | 6.99 / 7.16x | 7.07 / 7.34x | 3.13 / 3.38x | 3.27 / 3.35x | +| busy_ui live | 6.70 / 6.73x | 6.81 / 6.70x | 5.78 / 5.91x | 6.01 / 5.88x | 7.91 / 7.71x | 7.52 / 7.45x | 3.79 / 3.56x | 3.57 / 3.51x | +| busy_ui zerolatency | 6.17 / 6.16x | 6.06 / 6.20x | 5.36 / 5.39x | 5.27 / 5.41x | 6.13 / 6.09x | 6.16 / 6.04x | 3.61 / 3.67x | 3.68 / 3.57x | + +full_motion's 4-thread wall clock, main's last speed loss on the scoreboard, is a win: medium 1.94 / 1.86 s against x264's 2.23 / 2.15 s (main 2.48 / 2.34 s), live 1.65 / 1.59 s against 2.18 / 2.11 s (main 2.29 / 2.14 s), from the concurrent and asynchronous B runs; with B-frames full_motion also takes 12-19% less CPU time than on main (round 2's B search and pyramid steps). The matched CRFs are the same for both binaries (full_motion 22.5 / 22.25 / 25.75 for medium / live / zerolatency, webcam_overlay 28.5 / 28.25 / 26.25, busy_ui 32.25 / 32.25 / 25). zerolatency output is byte-identical to main's, and on the screen clips adaptive B usage codes nearly every frame as P, so those rows differ only by the sessions' spread; webcam_overlay live's first session (wall 3.09x and 2.32x) had slow runs, and a 7-round lane recheck put the branch, `c12ee19` and main at 0.75, 0.76 and 0.78 s on one thread and 0.43 s each on four. Peak RSS at 1080p on 1 / 4 threads: full_motion medium 115 / 155 MiB (main 97 / 102, x264 253-260 / 373-385), live 82 / 125 (main 72-73 / 78), zerolatency 52 / 54 as on main; webcam_overlay medium 104 / 119-120 (main 91 / 96), live 70 / 88 (main 67 / 71-72). The extra is the concurrently coded B-frames' state (DECISIONS.md). + +After merging main `1859bac` (`f1c6e24`: footprint rounds 3-4, intra-speed round 2; B-frame output byte-identical to `812670f`), each concurrently coded B-frame packs its coefficients into its own chunk store, which halves the B-frames' extra memory. Peak RSS at 1080p on 1 / 2 / 4 threads (90 frames, CRF 26): full_motion live 59 / 85 / 89 MiB (main `1859bac` 58 / 63 / 64, `812670f` 73 / 108 / 112), medium 94 / 116 / 120 (main 83 / 87 / 89, `812670f` 109 / 141 / 144); zerolatency 39 on 4 threads and busy_ui 30-64 as on main; full_motion 4K live 218 / 315 on 1 / 4 threads (main 214 / 229, `812670f` 274 / 416). `--keyint 1` and `2` no longer declare B-frames in the SPS and are byte-identical to `--bframes 0`. In the benchmark lane (two sessions per binary, matched VMAF against x264 veryfast, ratios on 1 / 4 threads), full_motion medium takes 1.58-1.63x / 1.41-1.47x less CPU time and 1.45-1.55x / 1.16-1.25x less wall clock than x264 (main 1.31-1.32x / 1.19-1.22x and 1.26x / 0.94-0.97x), live 1.69-1.72x / 1.56-1.57x and 1.61-1.64x / 1.38-1.39x (main 1.45-1.49x / 1.24-1.30x and 1.39-1.43x / 1.01-1.02x); the 4-thread wall clock is 1.72-1.88 s on medium against x264's 2.15-2.17 s (main 2.16-2.31 s) and 1.48-1.49 s on live against 2.06 s (main 2.01-2.62 s). + +After merging main `e20b7fe` (`0c662ea`: read-ahead during B runs, pin7 CABAC round), in the benchmark lane (two sessions per binary), full_motion medium takes 1.69-1.71x / 1.48-1.49x less CPU time and 1.62-1.64x / 1.35-1.40x less wall clock than x264 veryfast on 1 / 4 threads (main `e20b7fe` 1.43-1.45x / 1.25-1.26x and 1.37-1.39x / 1.09-1.10x), live 1.88-1.93x / 1.60-1.66x and 1.80-1.85x / 1.44-1.50x (main 1.58-1.61x / 1.34-1.36x and 1.52-1.54x / 1.19-1.21x); the 4-thread wall clock is 1.51-1.53 s on medium against x264's 2.04-2.15 s (main 1.91-2.01 s) and 1.42-1.53 s on live against 2.13-2.20 s (main 1.71-1.81 s). Peak RSS at 1080p on 1 / 2 / 4 threads (90 frames, CRF 26): full_motion live 59 / 95 / 98 MiB (main 58 / 72 / 73), medium 94 / 126 / 130 (main 82 / 96 / 98); the read-ahead's three extra raw frames add 9-10 MiB on 2 and 4 threads to both, and 38 MiB at 4K (live on 4 threads 353, main 268). + +Recommended settings: the defaults (three adaptive B-frames with the pyramid on fast, medium and slow) wherever the reorder delay is acceptable (exports, uploads, the media server and render farm); live also defaults to three adaptive B-frames (no pyramid, 2 frames of lookahead), with `--zerolatency` for no reorder delay and the least memory. + +OCR (text-region character error, 3 frames per clip, `bench.py ocr`): the text AQ lowers character errors on scrolling text by about a third at equal bitrate; other clips are within noise. + +Lookahead (10 frames, on by default except live), VMAF BD-rate against the encoder without it (busy_ui, code_editor, full_motion, slides, text_scroll; mean): fast −1.6/−5.0/−1.3/−12.6/−4.5 (−5.0%), medium −1.7/−5.0/−0.7/−12.5/−4.5 (−4.9%), slow −1.4/−5.1/−1.7/−14.9/−4.9 (−5.6%). Against x264 veryfast, medium goes from +4.8% to +2.3% and slow from +6.8% to +2.0%. Single-thread instructions −3% to +4% per clip, 10 frames of delay, +31 MB at 1080p; OCR character error is unchanged on text_scroll and lower on code_editor. At equal CRF video gets fewer bits and long-lived screen content more (full_motion CRF 20 13.3 → 7.4 Mbit/s, text_scroll CRF 26 +13%). With 250-frame GOPs full_motion is still about 2% worse than without lookahead while the screen clips gain 3-18%. + +Lookahead with adaptive B-frames (the fast, medium and slow defaults since `9bce84f`; DECISIONS.md): every picture is coded with the lookahead map of its own display index, B-frames take the anchor's macroblock QP + 6, and an anchor gains lifetime from the B-frames that predict from it through list 1. VMAF BD-rate against main `9bce84f` (B-frames, no lookahead) of the same preset (busy_ui, code_editor, full_motion, slides, text_scroll; mean): fast −1.6/−5.0/−2.8/−12.6/−4.5 (−5.3%), medium −1.7/−5.0/−2.5/−12.7/−4.5 (−5.3%), slow −1.4/−5.1/−1.8/−15.0/−4.9 (−5.6%); without the list-1 credit medium full_motion is −0.7%. Against x264 veryfast the mean goes from +1.5% to −3.4% (fast), from −0.9% to −5.6% (medium) and from −0.4% to −5.4% (slow). webcam_overlay gains 8.9-11.6%, all of it from main's CRF 20 point (about +2% without it). Single-thread CPU at CRF 26 (best of 5, interleaved, loaded machine) over the six clips: fast +2.8%, medium +1.4%, slow +2.9%; callgrind instructions for medium −2.4/−2.0/+1.4/−0.6/+4.1% (+1.3%). The 5-10% on the shortest screen clips (0.4-1.2 s encodes) is about 8,000 more page faults from first touching the lookahead buffers plus noise; their user time is −3% to +3.5%. Up to 13 frames of output delay (10 lookahead + 3 B-frames), +31 MB at 1080p. OCR character error: busy_ui 3.96 → 4.00, code_editor 4.77 → 4.54, text_scroll 2.02 → 2.03; on slides 0.54 → 0.79 comes from icon and bullet glyphs and source misreads (the P-only lookahead gives 0.80) with equal or higher PSNR in the text regions. At equal CRF full_motion gets 23-29% fewer bits (2-9 VMAF points below main) and the screen clips 4-14% more; holding the mean offset at or below zero keeps main's quality at every CRF but costs 11% CPU at equal CRF (DECISIONS.md). + +Lookahead round 3 (2026-10-08, standard tier, six clips; DECISIONS.md): with more than one thread the read-ahead thread runs the propagation for the next frame (main-thread instructions -9.1% full_motion, -29.7% slides at 4 threads); the change map without side info is vectorised (media-server busy_ui -40% instructions); without B-frames, content that a video frame (at least half the macroblocks changed) replaces is assumed to decay geometrically to the horizon. fast, medium, slow, live and `--lookahead 0` are byte-identical to main. The media-server settings (medium, `--keyint 250 --noise 3 --no-sideinfo`) now gain on every clip against `--lookahead 0` (full_motion -1.6%, mean -7.5%); with `--bframes 0` full_motion is still +2.4% (+2.9% before the geometric tail). SATD, half-pel and i16 intra costs in the lookahead gained at most 0.9% (P-only full_motion) for +2.7 to +14% instructions. Live with a short lookahead (VMAF BD-rate against live; instructions at CRF 26 against live; peak memory at 1080p, one thread): + +| `--preset live` | busy_ui | code_editor | full_motion | slides | text_scroll | webcam_overlay | mean | vs x264 veryfast | instructions | delay at 30 fps | memory | +|---|---|---|---|---|---|---|---|---|---|---|---| +| `--lookahead 0` (default) | | | | | | | | -0.4% | | 0 | 41-52 MB | +| `--lookahead 1` | -3.4 | -3.3 | +1.9 | -16.5 | -4.0 | -9.8 | -5.9% | -3.8% | -5.3% | 33 ms | +3 MB | +| `--lookahead 2` | -3.3 | -3.5 | -1.0 | -15.2 | -4.1 | -10.9 | -6.3% | -4.5% | -4.9% | 67 ms | +6 MB | +| `--lookahead 4` | -3.3 | -3.9 | -1.2 | -14.4 | -4.1 | -12.6 | -6.6% | -4.6% | -4.5% | 133 ms | +12 MB | + +Live reads 2 frames ahead by default since `b0d85fb` (67 ms at 30 fps); `--zerolatency` keeps 0. OCR character error at depth 2: text_scroll 1.81 -> 1.41%, slides 0.48 -> 0.45%, code_editor 4.70 -> 4.80%, busy_ui 3.68 -> 4.33% with equal text-region PSNR (-0.29 to +0.25 dB at matched bitrate); the busy_ui rise is icon glyphs Tesseract fuses onto words, and the word-level `ocr_word_cer` stays at or below 0.7% for every clip from ~400 kbps. Medium at depth 6 or 4 saves 12 or 19 MB for +0.06% mean but loses 0.54 or 0.78% on full_motion; the ring of raw frames is the encoder's input delay, so the memory stays. + +Lookahead round 4 (2026-10-08, DECISIONS.md): the media server's settings (medium, `--keyint 250 --noise 3`, no side info) spent most of their time in the encoder's scalar per-macroblock change test. Change detection is now one vector pass per frame, two macroblocks per vector row with an exact-equality test first, and the encoder reuses the lookahead's map wherever its source copy is current. Every preset default and `--lookahead 0` is byte-identical to main `52510d7`, and after merging main `7189fa4` to it (270/270 gate runs identical to the pre-merge branch; instructions against `7189fa4` -83.1/-84.3/-88.4/-1.0/-73.0%). At the media-server settings against main `52510d7` (CPU is user + system time, best of 5 interleaved runs on a loaded 4-core VM; full_motion at one thread is the median of 10 paired runs, whose best-of-5 moved between -3 and +4%): + +| | busy_ui | code_editor | slides | full_motion | webcam_overlay | +|---|---|---|---|---|---| +| instructions, 60 frames, 1 thread | -83.6% | -84.7% | -88.9% | -1.0% | -73.2% | +| CPU, 300 frames, 1 thread | -62.6% | -66.1% | -40.1% | -0.6% | -32.1% | +| CPU, 300 frames, default threads | -61.1% | -58.8% | -36.2% | | -31.7% | + +P-only GOPs longer than 2 s (`--bframes 0 --keyint 250`) end content that a video frame replaces at the window edge: -0.76% mean VMAF BD-rate (slides -2.78, webcam_overlay -1.24, full_motion -0.50%, no clip worse), full_motion +1.67% against `--lookahead 0` (was +2.37%). The busy_ui OCR difference under the live lookahead is icon glyphs that OCR reads as characters, in mostly unchanging text, with higher text PSNR and a sign that flips between CRFs and regions; capping the lookahead's QP raise on text macroblocks does not change it and gives up a fifth to a third of the BD-rate gain, so live keeps reading 2 frames ahead as measured (its default since `b0d85fb`). + +Single-thread speed at 1080p (live preset, warm page cache): busy_ui ~590 fps (a third of it is the kernel copying the raw input file; a library integration would not pay that), code_editor ~450, text_scroll ~140, slides ~160, full_motion ~21.5. x264 veryfast on this VM: ~110-145 fps on one core, ~150-370 on four. + +Intra coding at equal QP (all-intra, AQ off): capcodec frames are 2.5% larger than x264 veryfast's with luma PSNR within 0.1 dB; chroma is ~1 dB lower because x264 uses chroma_qp_index_offset −2 (VMAF ignores chroma). + +Change-proportional cost (slow_typing, earlier measurement): static frame 0.18% of a full frame with side info; 1.7% with the pixel-difference fallback. + +## Speed rounds (profile-driven; each round worked on the top stage) + +| Round | Top stage | Change | Effect | +|---|---|---|---| +| 1-4 | ME, intra | sub-pel prediction, cached half-pel tiles, intra early exits, table-driven directional intra | full-change frame 265 → 125 ms | +| 5 | entropy (IDR) | byte-oriented CABAC | IDR entropy 31 → 14 ms | +| 6 | mode + tq (IDR) | IDR macroblock reuse | IDR ~95 → ~45 ms on typing/editor/UI clips (79-99% of macroblocks reused) | +| 7 | bookkeeping | one MV per macroblock, untouched skip state | 10-20% on motion-heavy clips | +| 8 | deblock | batched edges, flat-line skip | busy_ui deblock 38 → 27 ms total | +| 9 | ME/mode (video) | SIMD SAD/SATD/SA8D/half-pel/qpel | full_motion 10.3 → 16.1 fps | +| 10 | tq (video) | SIMD 4x4 transforms and quantisation | 16.1 → 18.6 fps | +| 11 | analysis | SIMD classification, row hashing, copies | 19.8 → 21.5 fps | + +Current full-motion profile (live, per frame): tq 16, motion 14, analysis 9.5, mode 6.7, deblock 3.9, entropy 2.8 ms. + +## Research applied + +- OpenAI math preprints (`~/work/openai-math`, 722 papers): the algorithmic results (DFT in O(n (log n)^(1−10⁻¹³)), complex matrix multiplication below 2.258, integer multiplication below n log n, finite tensor savings for Fourier circuits) are asymptotic and do not reduce work at H.264's 4x4/8x8 integer butterflies, which are already at minimal add counts; the bottleneck is vector throughput and memory traffic. Recorded in DECISIONS.md. +- Applied from video-coding practice: hash/profile-based global motion (screen content coding), byte-oriented CABAC (x264/FFmpeg style), weighted prediction for fades, x264-style decimation and fast skip, SIMD kernels. Tried and reverted: EPZS-style cursor and dominant-motion predictors for every macroblock (no gain on the corpus, slides +0.7%). + +## Tov + +`encoder-features` branch (all committed): binary fs I/O, `--unchecked`, typed arrays, SIMD vectors, threads and Atomics, borrowed typed-array field arguments (`976bb78`), loop hoisting of array fields (`dee1ec3`), module init order (`d8a0f81`), `Math.clz32`/`imul` (`eced999`), C library output `tov build --lib` (`5cb7538`: `libname.a/.so` + `name.h`, typed arrays borrowed zero-copy, opaque class handles, `name_last_error`), function-value adapters (`7b58f21`), `--target wasm32-wasi` for programs and libraries (`3bee8b1`; a thread-free capcodec build is byte-identical to native at 1.86x the CPU per frame). The installed compiler is `encoder-features` 20ea68f (from `~/work/tov-pin7`, installed 2026-10-09 07:50; capcodec builds byte-identically to 538446e; adds horizontal `min`/`max`/`minIndex`/`maxIndex` (`phminposuw`), vector records returned in registers, `mulHigh`/`mulWideLow`/`mulWideHigh`, `@inline`/`@noinline`, Buffer-style `readUInt32LE`/`writeUInt32BE`/... on `Uint8Array`; 538446e stays at `/usr/local/cargo/bin/tov.pin6-538446e`). Previously 538446e (from `~/work/tov-pin6`, installed 2026-10-08 23:50; against 64982fa: capcodec unchecked 620,816 -> 550,024 bytes, 0.8-1.8% fewer instructions, clean builds compiled as 16 parallel C units, `Math.select`, run-time `T.shuffle`, `pushFrom`, `Math.idiv`, `Target.vectorBits`, narrowing stores, `discard`; byte-identical encodes; 64982fa stays at `/usr/local/cargo/bin/tov.pin5-64982fa`). Previously 64982fa (from `~/work/tov-pin5`; 2.9-5.2% fewer capcodec instructions than 548ff6b, byte-identical output, multi-target `--target-cpu x86-64-v3,x86-64-v4` binaries, `--profile-generate`, module constants as C constants, smaller binaries): element borrows folded in (72b793b), serial threads on wasm (3b00bba), 512-bit vectors and concat/low/high, register-resident sums, mask/ctz/popcount, branch-free `?:`, borrowed typed-array locals, integer-quotient rounding, Atomics memory orders, Int64Array and 64-bit lanes, Channel, parallelFor options; capcodec builds byte-identically with it. See TOV_GAPS.md. + +## How to resume + +```sh +export PATH=$HOME/tools/bin:$PATH # static ffmpeg 8.1 (libx264, libopenh264, libvmaf), JM ldecod +cd ~/work/tov-pin2 && git log --oneline -1 && cargo +1.88.0 install --path crates/tov # elem-borrow 02a302a on top of encoder-features +cd ~/work/capcodec && bench/build_encoder.sh +python3 bench/bench.py smoke # < 60 s, regression gate +python3 bench/bench.py standard --encoders "capcodec-live,capcodec-medium,x264-veryfast" +python3 tools/profile.py full_motion busy_ui --presets live,medium # per-stage times +python3 tools/verify.py slides --binary build/capcodec-unchecked --preset medium # JM + ffmpeg bit-exact check +``` + +Backups (git bundles of capcodec and the Tov branch, PROGRESS.md) are written to `/cursor/stores/self/backups` by `tools/backup.sh`. + +## Next steps + +1. Parallel entropy coding for IDR frames (a few row-band slices, about +0.5% bits) and row-parallel deblocking: IDR frames are now the serial bottleneck on typing/editor/UI clips. +2. Full-motion efficiency: live preset is +68% BD vs medium's +39% without B-frames; stronger motion search where the frame is video-like, SIMD 8x8 transforms, intra in P frames. B-frames next steps: B sub-partitions (16x8, 8x16, 8x8 and direct 8x8), lookahead to choose the B run length per scene (and to credit the middle B-frame of a pyramid run), a second reference per list, and entropy coding of B rows as they finish so concurrent B-frames need a few rows of coefficients instead of a frame. +3. Screen-content BD-rate: tune I/P offset, refresh, lifetime AQ and AQ strength with harness variants; transient-content QP for animations. +4. C ABI build for crates/export, WASM build for the browser export, FINDINGS.md. +5. Lookahead: choose the B run length and reference structure from its lifetimes (the changed-macroblock count alone lost 5% on pulldown, DECISIONS.md); an opt-in quality-anchored CRF (the one-sided offset clamp) where CRF has to mean the same quality on video; P-only long GOPs, where full_motion still loses 1.7% (a per-macroblock tail from how long each block's motion has lasted, or intra refresh of the decayed content); a changed-macroblock ring sharing the encoder's buffers. +6. Media-server CPU after the change-detection round: CABAC (163M of busy_ui's 558M instructions over 60 frames, about 70M of it the first IDR frame), the change map (70M, nearly all of it the lookahead's pass, which already runs on the read-ahead thread with more than one thread), intra decisions (41M, mostly the IDR frame), and letting B-run anchors take the lookahead's map (full_motion: 3 of 17 anchors can today). diff --git a/capcodec/README.md b/capcodec/README.md new file mode 100644 index 00000000000..36500a5d900 --- /dev/null +++ b/capcodec/README.md @@ -0,0 +1,7 @@ +# capcodec + +Tov source for the H.264 screen encoder used when `CAP_MEDIA_VIDEO_ENCODER=capcodec`. This directory is the `main` snapshot (`517b7e6`). The full-motion page reads its media URL from `window.capVideoSrc`, which the corpus generator sets before load. Cap CI leaves it unbuilt. Mount a binary as described in `apps/media-server/capcodec.md`. + +`capcodec.bundle` stores every branch. Restore it with `git clone capcodec.bundle`. The media-server flag runs `main`. `rd-next`, `ssim`, `intra-rd`, and `dashboard` stay on their own branches. + +Benchmark notes keep the commit ids of the builds that were measured. In this bundle those commits are `76be2a6` (measured as `00fecf7`), `82b7840` (measured as `fff504d`), `42e1d3b` (measured as `d0cce5e`, branch `rd-next`), and `6610051` (measured as `b72e995`, branch `ssim`). diff --git a/capcodec/SCOREBOARD.md b/capcodec/SCOREBOARD.md new file mode 100644 index 00000000000..3c5f1a64006 --- /dev/null +++ b/capcodec/SCOREBOARD.md @@ -0,0 +1,177 @@ +# Scoreboard against x264 veryfast + +Quality measured 2026-10-09 on main `fff504d`. Speed remeasured the same day on main `00fecf7` (read-ahead, pin7 intra minIndex, CABAC table). Those commits match the `fff504d` bitstreams, so the quality tables still describe this encoder. Pending quality branch `rd-next` `d0cce5e`. All 12 corpus clips. + +How to reproduce: + +- Quality: `CAPCODEC_BIN_UNCHECKED= python3 bench/bench.py standard --quality-only --encoders capcodec-medium,capcodec-live,capcodec-live-zerolatency,x264-veryfast,x264-veryfast-zerolatency --out ` (1080p, 10 s per clip; code_4k is scaled to 1080p here). capcodec's CRF ladder runs 20-44 so its VMAF range covers x264's CRF 18-33; the overlap column shows how much of the two curves BD-rate compares (it was 15-24% on screen clips with the old 20-36 ladder). +- Speed: `bench/lane.sh start`, then `bench/lane.sh run "python3 bench/speedcmp.py --binary --clips all --modes medium,live,zerolatency --out "`. A fresh run matches each clip to x264 CRF 23 by VMAF, then times 5 shuffled rounds at 1 and 4 threads with other processes stopped. code_4k runs at native 4K for 5 s. The published table reused an earlier CRF match; that results file is not in the repository. +- Tables: `python3 bench/scoreboard.py --quality LABEL= --speed LABEL= --out SCOREBOARD.md`. + +### main fff504d: capcodec-medium vs x264-veryfast (BD-rate, negative = capcodec needs fewer bits) + +| clip | VMAF | PSNR-YUV | SSIM | VMAF range overlap | +|---|---|---|---|---| +| busy_ui | +5.4% lose | +5.1% lose | +14.8% lose | 52% | +| busy_ui_60fps | +2.6% lose | +2.0% lose | +16.8% lose | 74% | +| code_4k | -3.3% win | +1.0% lose | +9.2% lose | 72% | +| code_editor | -0.5% win | +4.9% lose | +7.3% lose | 76% | +| dark_mode | -17.2% win | -16.9% win | -11.9% win | 75% | +| dashboard | +7.9% lose | +12.3% lose | +25.9% lose | 59% | +| full_motion | -12.2% win | -10.0% win | +4.8% lose | 44% | +| idle | -8.8% win | -6.3% win | -3.4% win | 73% | +| slides | -18.0% win | +0.5% lose | -3.6% win | 66% | +| slow_typing | -0.8% win | +4.5% lose | +8.6% lose | 80% | +| text_scroll | -15.5% win | -14.8% win | -14.9% win | 88% | +| webcam_overlay | -20.0% win | -4.3% win | -4.0% win | 69% | + +19 wins, 17 losses across 12 clips x 3 metrics. + +### main fff504d: capcodec-live vs x264-veryfast (BD-rate, negative = capcodec needs fewer bits) + +| clip | VMAF | PSNR-YUV | SSIM | VMAF range overlap | +|---|---|---|---|---| +| busy_ui | +5.7% lose | +5.7% lose | +15.6% lose | 51% | +| busy_ui_60fps | +3.5% lose | +2.9% lose | +18.1% lose | 73% | +| code_4k | +0.2% lose | +5.3% lose | +13.4% lose | 72% | +| code_editor | +2.8% lose | +8.6% lose | +11.2% lose | 76% | +| dark_mode | -16.6% win | -16.2% win | -10.9% win | 75% | +| dashboard | +10.8% lose | +15.7% lose | +29.3% lose | 61% | +| full_motion | -7.8% win | -1.9% win | +12.0% lose | 43% | +| idle | -8.4% win | -6.2% win | -3.0% win | 72% | +| slides | -20.0% win | -1.5% win | -6.4% win | 65% | +| slow_typing | +1.1% lose | +4.8% lose | +9.1% lose | 78% | +| text_scroll | -15.1% win | -14.7% win | -14.8% win | 88% | +| webcam_overlay | -18.7% win | -3.8% win | -1.7% win | 67% | + +17 wins, 19 losses across 12 clips x 3 metrics. + +### main fff504d: capcodec-live-zerolatency vs x264-veryfast-zerolatency (BD-rate, negative = capcodec needs fewer bits) + +| clip | VMAF | PSNR-YUV | SSIM | VMAF range overlap | +|---|---|---|---|---| +| busy_ui | +1.6% lose | +2.3% lose | +6.7% lose | 23% | +| busy_ui_60fps | +2.0% lose | +1.9% lose | +9.2% lose | 30% | +| code_4k | -1.2% win | +2.4% lose | +7.9% lose | 36% | +| code_editor | -0.5% win | +8.0% lose | +5.8% lose | 36% | +| dark_mode | -22.3% win | -21.6% win | -21.3% win | 35% | +| dashboard | +0.2% lose | +2.3% lose | +7.4% lose | 27% | +| full_motion | -3.1% win | -1.0% win | +15.1% lose | 98% | +| idle | -17.2% win | -17.8% win | -15.6% win | 28% | +| slides | -30.8% win | -22.7% win | -20.8% win | 40% | +| slow_typing | -0.8% win | +3.8% lose | +4.8% lose | 37% | +| text_scroll | -16.3% win | -17.6% win | -20.7% win | 33% | +| webcam_overlay | -74.8% win | -64.5% win | -54.8% win | 42% | + +20 wins, 16 losses across 12 clips x 3 metrics. + +### rd-next d0cce5e: capcodec-medium vs x264-veryfast (BD-rate, negative = capcodec needs fewer bits) + +| clip | VMAF | PSNR-YUV | SSIM | VMAF range overlap | +|---|---|---|---|---| +| busy_ui | -17.0% win | -2.4% win | +6.5% lose | 65% | +| busy_ui_60fps | -18.8% win | -3.8% win | +6.4% lose | 42% | +| code_4k | -23.0% win | -1.4% win | +5.0% lose | 57% | +| code_editor | -22.7% win | -3.7% win | +2.5% lose | 57% | +| dark_mode | -35.0% win | -19.9% win | -13.5% win | 58% | +| dashboard | -12.3% win | +3.2% lose | +12.8% lose | 77% | +| full_motion | -21.6% win | -8.9% win | +1.5% lose | 52% | +| idle | -24.0% win | -11.8% win | -2.9% win | 66% | +| slides | -33.6% win | -13.3% win | -10.2% win | 81% | +| slow_typing | -22.4% win | -4.1% win | +2.2% lose | 43% | +| text_scroll | -31.1% win | -19.6% win | -14.3% win | 54% | +| webcam_overlay | -34.1% win | -14.0% win | +3.4% lose | 87% | + +27 wins, 9 losses across 12 clips x 3 metrics. + +### rd-next d0cce5e: capcodec-live vs x264-veryfast (BD-rate, negative = capcodec needs fewer bits) + +| clip | VMAF | PSNR-YUV | SSIM | VMAF range overlap | +|---|---|---|---|---| +| busy_ui | -17.3% win | -2.3% win | +6.5% lose | 64% | +| busy_ui_60fps | -18.9% win | -3.7% win | +6.5% lose | 42% | +| code_4k | -22.7% win | -1.3% win | +5.2% lose | 58% | +| code_editor | -22.1% win | -3.7% win | +2.7% lose | 57% | +| dark_mode | -34.4% win | -19.6% win | -13.0% win | 59% | +| dashboard | -12.4% win | +3.8% lose | +14.0% lose | 73% | +| full_motion | -18.0% win | -2.6% win | +6.8% lose | 51% | +| idle | -23.8% win | -11.7% win | -2.7% win | 67% | +| slides | -35.3% win | -15.6% win | -12.6% win | 83% | +| slow_typing | -21.3% win | -4.1% win | +2.2% lose | 43% | +| text_scroll | -31.0% win | -19.6% win | -14.3% win | 54% | +| webcam_overlay | -34.1% win | -14.9% win | +1.2% lose | 88% | + +27 wins, 9 losses across 12 clips x 3 metrics. + +### rd-next d0cce5e: capcodec-live-zerolatency vs x264-veryfast-zerolatency (BD-rate, negative = capcodec needs fewer bits) + +| clip | VMAF | PSNR-YUV | SSIM | VMAF range overlap | +|---|---|---|---|---| +| busy_ui | -21.3% win | -6.6% win | +1.0% lose | 48% | +| busy_ui_60fps | -18.9% win | -6.2% win | +3.0% lose | 64% | +| code_4k | -20.7% win | -0.8% win | +5.4% lose | 62% | +| code_editor | -22.6% win | -3.7% win | +2.8% lose | 62% | +| dark_mode | -37.4% win | -26.1% win | -22.1% win | 58% | +| dashboard | -21.6% win | -7.8% win | -0.1% win | 45% | +| full_motion | -15.6% win | -0.3% win | +7.4% lose | 62% | +| idle | -33.5% win | -25.2% win | -13.5% win | 46% | +| slides | -46.6% win | -35.0% win | -23.4% win | 58% | +| slow_typing | -21.3% win | -6.2% win | +0.5% lose | 70% | +| text_scroll | -32.6% win | -24.5% win | -20.8% win | 53% | +| webcam_overlay | -80.6% win | -64.2% win | -43.9% win | 58% | + +30 wins, 6 losses across 12 clips x 3 metrics. + +### main 00fecf7: speed at matched VMAF (x264 / capcodec, interleaved medians, other work paused) + +| clip | mode | VMAF | CPU 1t | wall 1t | CPU 4t | wall 4t | peak RSS 4t capcodec / x264 MiB | +|---|---|---|---|---|---|---|---| +| busy_ui | live | 94.5 | 7.42x win | 6.55x win | 8.25x win | 4.40x win | 46 / 377 | +| busy_ui_60fps | live | 93.2 | 8.20x win | 7.23x win | 9.21x win | 4.17x win | 46 / 374 | +| code_4k | live | 95.2 | 8.92x win | 7.96x win | 9.73x win | 4.58x win | 176 / 1304 | +| code_editor | live | 94.4 | 7.36x win | 6.52x win | 7.97x win | 4.49x win | 49 / 382 | +| dark_mode | live | 93.4 | 7.33x win | 6.54x win | 6.80x win | 4.59x win | 48 / 393 | +| dashboard | live | 94.4 | 3.86x win | 3.51x win | 4.25x win | 2.10x win | 51 / 384 | +| full_motion | live | 88.2 | 1.61x win | 1.54x win | 1.37x win | 1.21x win | 80 / 379 | +| idle | live | 95.3 | 9.18x win | 8.11x win | 10.19x win | 4.55x win | 44 / 379 | +| slides | live | 95.0 | 5.15x win | 4.64x win | 4.61x win | 3.24x win | 74 / 391 | +| slow_typing | live | 95.2 | 7.87x win | 6.88x win | 8.82x win | 4.15x win | 45 / 387 | +| text_scroll | live | 95.0 | 4.88x win | 4.40x win | 3.84x win | 3.10x win | 53 / 384 | +| webcam_overlay | live | 95.4 | 4.64x win | 4.21x win | 4.55x win | 3.46x win | 74 / 391 | +| busy_ui | medium | 94.5 | 7.30x win | 6.31x win | 7.77x win | 4.11x win | 77 / 371 | +| busy_ui_60fps | medium | 93.2 | 8.08x win | 7.13x win | 8.77x win | 4.09x win | 77 / 372 | +| code_4k | medium | 95.2 | 7.96x win | 7.13x win | 8.78x win | 3.89x win | 308 / 1279 | +| code_editor | medium | 94.4 | 7.01x win | 6.17x win | 7.88x win | 4.13x win | 83 / 379 | +| dark_mode | medium | 93.4 | 7.25x win | 6.48x win | 6.37x win | 4.24x win | 79 / 386 | +| dashboard | medium | 94.4 | 3.59x win | 3.21x win | 3.82x win | 1.75x win | 85 / 382 | +| full_motion | medium | 88.2 | 1.39x win | 1.33x win | 1.22x win | 1.06x win | 103 / 375 | +| idle | medium | 95.3 | 9.37x win | 8.30x win | 10.28x win | 4.54x win | 73 / 385 | +| slides | medium | 95.0 | 4.86x win | 4.40x win | 4.34x win | 2.90x win | 98 / 384 | +| slow_typing | medium | 95.2 | 7.94x win | 6.94x win | 8.26x win | 3.97x win | 76 / 384 | +| text_scroll | medium | 95.0 | 4.39x win | 3.88x win | 3.64x win | 2.89x win | 88 / 385 | +| webcam_overlay | medium | 95.4 | 4.32x win | 3.92x win | 4.20x win | 3.01x win | 99 / 386 | +| busy_ui | zerolatency | 96.0 | 6.79x win | 5.96x win | 6.92x win | 3.85x win | 34 / 144 | +| busy_ui_60fps | zerolatency | 95.5 | 7.75x win | 6.82x win | 7.96x win | 3.95x win | 34 / 150 | +| code_4k | zerolatency | 96.0 | 7.51x win | 6.58x win | 7.06x win | 3.92x win | 113 / 458 | +| code_editor | zerolatency | 95.8 | 6.32x win | 5.51x win | 6.38x win | 3.61x win | 34 / 146 | +| dark_mode | zerolatency | 95.5 | 6.51x win | 5.67x win | 5.00x win | 3.76x win | 34 / 151 | +| dashboard | zerolatency | 96.1 | 3.44x win | 3.08x win | 3.64x win | 2.17x win | 37 / 150 | +| full_motion | zerolatency | 88.1 | 1.65x win | 1.58x win | 1.21x win | 1.49x win | 46 / 142 | +| idle | zerolatency | 96.4 | 9.40x win | 8.32x win | 8.89x win | 4.40x win | 34 / 141 | +| slides | zerolatency | 96.2 | 4.84x win | 4.34x win | 3.86x win | 2.95x win | 39 / 149 | +| slow_typing | zerolatency | 96.4 | 7.15x win | 6.15x win | 7.16x win | 3.53x win | 34 / 148 | +| text_scroll | zerolatency | 96.3 | 3.90x win | 3.45x win | 3.12x win | 2.78x win | 38 / 149 | +| webcam_overlay | zerolatency | 96.2 | 5.17x win | 4.73x win | 4.60x win | 4.27x win | 39 / 142 | +- live 1 thread cpu: geometric mean 5.83x, min 1.61x (12/12 clips faster) +- live 1 thread wall: geometric mean 5.23x, min 1.54x (12/12 clips faster) +- live 4 threads cpu: geometric mean 5.87x, min 1.37x (12/12 clips faster) +- live 4 threads wall: geometric mean 3.46x, min 1.21x (12/12 clips faster) +- medium 1 thread cpu: geometric mean 5.53x, min 1.39x (12/12 clips faster) +- medium 1 thread wall: geometric mean 4.94x, min 1.33x (12/12 clips faster) +- medium 4 threads cpu: geometric mean 5.51x, min 1.22x (12/12 clips faster) +- medium 4 threads wall: geometric mean 3.15x, min 1.06x (12/12 clips faster) +- zerolatency 1 thread cpu: geometric mean 5.39x, min 1.65x (12/12 clips faster) +- zerolatency 1 thread wall: geometric mean 4.80x, min 1.58x (12/12 clips faster) +- zerolatency 4 threads cpu: geometric mean 4.90x, min 1.21x (12/12 clips faster) +- zerolatency 4 threads wall: geometric mean 3.26x, min 1.49x (12/12 clips faster) + diff --git a/capcodec/TOV_GAPS.md b/capcodec/TOV_GAPS.md new file mode 100644 index 00000000000..e7b8532464c --- /dev/null +++ b/capcodec/TOV_GAPS.md @@ -0,0 +1,155 @@ +# Tov gaps + +Features an encoder needs that Tov lacked, what was added on the `encoder-features` branch of Tov, and why. Each feature is general purpose, follows Tov's conventions, and ships with tests and spec updates. + +| # | Gap | Status | Commit | +|---|---|---|---| +| 1 | Binary file I/O (`readFileSync` only returned strings) | done | `9cfd397` feat(fs) | +| 2 | `--unchecked` kept bounds checks | done | `bce77c2` feat(unchecked) | +| 3 | Typed arrays (reference semantics, no copy-on-write, shareable) | done | `d4f036f` feat(typed-arrays) | +| 4 | SIMD vector types | done | `d65eb21` feat(simd) | +| 5 | Threads and Atomics | done | `386512e` feat(threads) | +| 6 | C ABI library output (`tov build --lib`) | planned | | +| 7 | WASM target | planned | | +| 8 | Field array stores re-check uniqueness on every store | open | | +| 9 | Module constants initialised by calls see imported constants empty | open | | +| 10 | Typed arrays passed from class fields retain/release atomically per call | done (arguments `976bb78`; element access `02a302a` on branch elem-borrow) | | + +## 1. Binary file I/O + +**Gap.** `node:fs` had only `readFileSync`/`writeFileSync` on strings. An encoder reads raw YUV frames (3 MB each at 1080p) from files and pipes and writes bitstreams. + +**Added.** Node's descriptor API: `openSync`, `closeSync`, `readSync(fd, &buffer, offset?, length?, position?)`, `writeSync(fd, buffer, offset?, length?, position?)`, `fstatSync`. Because Tov arrays are values, `readSync` takes its `u8[]` buffer `inout` (`&buffer`); to support that, standard-library natives can now declare an `inout` array parameter (`native_inout_param`), which code generation passes by pointer after making the buffer unique. Writes to stdout flush `console.log`'s buffer first so output never interleaves. Errors use Node's messages (`EBADF: bad file descriptor, read`), ranges are checked with Node's `RangeError` text. + +**Tests.** `tests/run/native_fs_bytes.tov`. **Spec.** §7a. + +## 2. `--unchecked` drops bounds checks + +**Gap.** `--unchecked` only made integer overflow wrap; every `xs[i]!` and `xs[i] = v` still compared against the length, which blocks vectorisation of pixel loops. + +**Added** (commit `bce77c2`): under `--unchecked`, `xs[i]!`, index assignment and compound assignment compile to bare loads/stores (out of bounds is undefined behaviour, as in C); `xs[i]` keeps its defined `T | undefined` result; sized-integer conversions keep the low bits and unary minus and `**` wrap. A 16x16 SAD over a 1080p frame: 0.92 ms instead of 1.5 ms. + +**Encoder measurement.** The production presets build with `--unchecked` (see DECISIONS.md); the checked build is kept for the harness's `capcodec-medium-checked` adapter and every unit test. + +## 3. Typed arrays + +**Gap.** Tov arrays are copy-on-write values, so a buffer held in a class field is made unique on every write, and nothing can be shared between threads or wrapped around memory owned by a C caller. + +**Added** (commit `d4f036f`): `Uint8Array`, `Int8Array`, `Uint16Array`, `Int16Array`, `Uint32Array`, `Int32Array`, `Float32Array`, `Float64Array`, `Uint8ClampedArray` with JavaScript semantics (shared references, fixed length, `subarray` views, `set`, `fill`, ...), one compare per element access (none unchecked), atomic reference counts, `readSync`/`writeSync` on `Uint8Array`. + +**Follow-up gap (10).** Passing a typed array held in a class field as an argument retains and releases it around every call, atomically. Porting the encoder to typed arrays (branch `typed-arrays-port` in capcodec, bit-identical output) made it 2x slower for this reason alone (transform/quant 1.9 ms → 8.6 ms per 1080p frame), so the encoder stays on `u8[]`/sized-number arrays. Fix to make in Tov: biased reference counting (plain increments until a typed array is shared with another thread) and/or not retaining field reads passed to callees that cannot reassign the field. + +## 4. SIMD vector types + +**Gap.** Every pixel kernel (SAD, SATD, interpolation, transforms, deblocking) was scalar; clang's auto-vectoriser does not vectorise loops over Tov arrays with 64-bit `int` arithmetic. + +**Added** (commit `d65eb21`): `i8x16 u8x16 i16x8 u16x8 i32x4 u32x4 f32x4` and their 256-bit twins as values compiled to the C compiler's vector extensions; `T.load`/`v.store` on typed arrays and sized-number arrays (one bounds check per access, none unchecked), widening loads, lane-wise arithmetic, masks and `select`, `absDiff`, `avg`, `u8x16.sad` (psadbw), `i16x8.madd` (pmaddwd), literal shuffles/swizzles, `widenLow/High`, `packSat`, `bitcast`, `convert`, and `tov build --target-cpu`. + +**Encoder use.** SAD, 4x4/8x8 SATD and SA8D, the 6-tap half-pel tiles, quarter-pel averaging, the 4x4 forward and inverse transforms with quantisation, macroblock classification, scroll row hashing, unchanged-macroblock tests, copies and weighted prediction run on vectors; each kernel has a unit test against its scalar definition and the encoder's output stayed byte-identical. Full-motion 1080p (live preset, one core): 10.3 → 21.5 fps across these rounds. + +## 8. Field array stores re-check uniqueness on every store + +**Gap.** `this.xs[i] = v` on an array held in a class field compiles to `tvg_at_mut(&self->f_xs, ...)`: a copy-on-write uniqueness check (load the buffer pointer, load its refcount) on every store, which the C compiler cannot hoist because the store may alias the refcount. A loop over 8160 macroblocks copying one field array into another (`prevMvx[mb] = mvx[mb]`) took 8% of a whole encode. + +**Workaround in the encoder.** Hot loops move into free functions taking the arrays `inout` (one uniqueness check per call), and per-macroblock state was shrunk (one motion vector per macroblock instead of sixteen). + +**Fix to make in Tov.** Hoist the uniqueness check and data pointer of a field array out of a loop when the loop body cannot make the array shared (no assignment of the field to another binding, no calls that receive it by value). + +## 9. Module constants initialised by calls see imported constants empty + +**Gap.** `const T: int[] = build()` at module level, where `build` reads a constant array exported by another module, ran before that module's constants were initialised, and trapped (`index 0 out of bounds for length 0`). + +**Workaround in the encoder.** Derived tables (CABAC state transitions, significance context maps, renormalisation shifts, 32-bit quantisation tables) are generated by `tools/gen_tables.py` as literals. + +**Fix to make in Tov.** Initialise modules in dependency order (imports first), as JavaScript does. + +## 10. Typed-array reference counting on shared data (threads) + +**Gap.** With threads, every atomic retain/release of a shared typed array bounces its header's cache line between cores. The encoder's wavefront was slower on 4 threads than on 1 (full_motion 11 vs 15 fps) because field reads passed as arguments (`sad(this.srcY, ...)`), element stores whose value read another field (`this.qpY[mb] = this.mbQp[mb]!`) and element reads whose index read a field (`this.changed[mby * this.mbw + mbx]!`) each took a count. + +**Added.** `976bb78` (Tov task): field reads passed to calls are lent without a count when no code the call can run replaces that field. `02a302a` (branch `elem-borrow`, by the encoder author, to be folded in with a test): an element access keeps the array alive only when its index or stored value may replace the field it was read from (Tov's `ta_element` used a syntactic "runs code" test that treated every field read and conversion call as running code). Remaining encoder-side rule: scratch buffers are typed arrays, because a call with an `inout` argument still retains its other typed-array arguments. + +**Effect.** 4-thread wavefront: full_motion 1080p 50.8 fps (one thread 25.0), text_scroll 295 (122); byte-identical output. + +## 11. Comparing a typed-array element with a sized-integer literal crashes code generation + +`ta[i]! === u8(0)` (or `i32(0)`) type-checks but fails in C generation with "internal compiler error: no conversion from `u8` to `int`" (C0003), reported at 1:1 of the entry file. Workaround: compare with an untyped literal (`=== 0`). Fix in Tov: either reject the comparison in the checker with a fix suggestion or convert. + +## 12-18. Found while vectorising intra coding (intra-speed branch, all worked around) + +12. **`.sum()` widens to 256-bit.** `d.abs().sum()` on an `i16x8` lowers to `__builtin_reduce_add` over eight 32-bit lanes (`vpmovsxwd %xmm,%ymm; vpaddd; vextracti128; ...`, 24 sites). Wanted: lane-width-preserving reductions, pmaddwd-style pair sums for 8/16-bit lanes, horizontal add and movemask primitives. Workaround: accumulate in 16-bit lanes, reduce once per block. +13. **A typed-array local copied from a field takes an atomic count** (`const t = this.c` emits `tv_ta_retain`/`tv_ta_release`), while indexing `this.c[i]!` reloads `self_->f_c->data` and `len` on every access because C is built with `-fno-strict-aliasing`. Wanted: borrow such locals when the field cannot be replaced in their scope (same analysis as gap 10), and keep the data pointer in a register. Workaround: pass arrays as parameters. +14. **`Atomics.wait`/`notify` share one process-global mutex and a linear waiter list**; `notify` locks it even with no waiters. Wanted: per-address (hashed) wait queues and a lock-free no-waiter fast path. Workaround: per-entry waiter counts in capcodec. +15. **The conditional operator lowers to `if`/`else`**, which clang kept as an unpredictable branch in the CABAC LPS select. Wanted: emit `c ? a : b` as a C conditional expression when both arms are pure. Workaround: mask arithmetic. +16. **No combine of two 128-bit vectors into a 256-bit one** (or split): the 16-lane deblock had to store two `i16x8` and reload an `i16x16`, defeating store-to-load forwarding. (Requested with the AVX-512 work: `concat`/`low`/`high`.) +17. **No 64-bit typed arrays or 64-bit vector lanes** (`BigInt64Array`-like `Int64Array`/`Uint64Array`, `i64x2`/`i64x4`): CABAC output uses 32-bit words with backward carry propagation instead. +18. **`parallelFor` hands out chunks of one index** and closures deep-copy captured objects; results come back through `join` or shared typed arrays. + +## 19-21. Found while pipelining P frames (thread-efficiency branch, worked around) + +19. **`Math.trunc(a / b)` on ints compiles to a float division plus a checked conversion call.** Wanted: an integer division operator or lowering `Math.trunc(int / int)` to C integer division (with the divide-by-zero check). Workaround: multiply-shift helpers for constant divisors. +20. **Every `Atomics` operation is sequentially consistent.** A wavefront progress publish needs only a release store and the wait an acquire load; each macroblock pays a full barrier. Wanted: `Atomics.storeRelease`/`loadAcquire` (or an ordering argument). +21. **Threads can't capture functions or class instances, and a running thread can't receive new typed arrays.** That forces per-frame spawning with a fresh worker `Encoder`, and rules out persistent workers unless reference frames live in a fixed pool of typed arrays. Wanted: a way to pass typed arrays to a running thread (e.g. a channel or a shared slot table) so a pool can live across frames. + +## 22-26. Found while reducing peak memory and binary size (footprint branch) + +Sizes are x86-64-v3 builds of the footprint branch (unchecked 460,240 bytes, checked 608,256) with Tov 548ff6b; effects of Tov changes were measured by patching the generated C through `CC`, with identical output. + +22. **Module integer constants are mutable globals set at start-up.** `const COEF_PER_MB = 416` becomes `static tv_int g7_COEF_PER_MB;`, assigned in `tvg_program`, so every use is a load, products and shifts by it aren't folded and `switch`es on `MB_*` or `CAT_*` can't become jump tables. Emitting the 77 literal-initialised ones as `static const tv_int g7_COEF_PER_MB = INT64_C(416);` (and folding constant expressions such as `BIN_TERM`, `BIG`, `COST_NA`): unchecked -9,568 bytes, checked -6,800, callgrind -1.5% (live busy_ui) to -7.1% (keyint 1 full_motion), -2.7% on medium full_motion. Not worked around: literals in the hot code would replace the names. +23. **Every `try` exit repeats the release of every live local.** `encodeCommand` has 23 error exits, each releasing up to ten strings, arrays and objects (139 release calls, 68 inlined `rl_K*`), an estimated 4-5 KB of its 20.7 KB of own code. Wanted: one cleanup chain per function (labels in reverse declaration order, each exit jumping to the label for its live set) or an outlined cold cleanup. +24. **The global symbol table stays without `-g`.** `-Wl,-x` drops local symbols, but the runtime's `tv_*` globals and their names cost 7,760 bytes; `-Wl,-s` drops them. Worked around in `bench/build_encoder.sh`. +25. **GNU ld pads the segments** (`-z separate-code`): linking with lld saves 10,376 bytes and `-Wl,-z,noseparate-code` with ld.bfd 4,096, code unchanged. Worked around in `bench/build_encoder.sh` (lld when installed). Unwind tables (20.5 KB unchecked, 28.7 KB checked) are already gone without `-g` in Tov 3369524; the script passes the flags until that Tov is installed. +26. **`fill` on a typed array of 16-, 32- or 64-bit elements calls `tv_ta_put_int` per element** (a switch on the element kind each time), where byte arrays get a `memset`. Wanted: a loop per element kind, `memset` for 0. + +## 27-32. Found while applying the 548ff6b features (tov-features branch, worked around) + +27. **No compile-time query of the target's vector width.** One source serves the x86-64-v3 and v4 builds, so the v4 build cannot select 512-bit kernels (a 32-lane SATD is 10-20% faster on v4 but slower on v3). Wanted: a constant such as `Target.vectorBits` (or `Target.hasAvx512`) usable in `if` so dead branches are removed per build, or per-function multiversioning. +28. **No narrow vector stores.** `v.store` writes every lane; nothing writes 8 bytes (the counterpart of `i16x8.loadU8`), so 8-pixel chroma rows, copies and pads stay scalar. Wanted: `v.storeLow(ta, i)` / `storeN(ta, i, lanes)` and narrowing stores (`i16x8.storeU8`). +29. **Lane lists can't be function parameters** (T0523), so helpers are duplicated per shuffle pattern. Wanted: `const` lane-list parameters (compile-time generic) or inlining-time constant propagation. +30. **Conditional or reassigned typed-array locals still take atomic counts** (`const p = c ? this.a : this.b`), so the 16-way branch in `predictQpel` stays. Wanted: borrow when every arm is a borrowable field path. +31. **Scalar fields are reloaded every iteration in loops that store to typed arrays** because the C is built with `-fno-strict-aliasing`; capcodec hoists `const mbw = this.mbw` by hand. Wanted: hoist loop-invariant scalar field reads when the loop only stores to typed-array elements (which can't alias object fields), or emit `restrict`. +32. **Integer division costs:** `Math.trunc(a / b)` with a constant divisor still adds a ±2^53 range check (about 10 instructions vs 2 for multiply-shift); a variable divisor is a double division; `%` is a checked `idiv`. Wanted: plain C integer division/modulo with only the zero check when both operands are ints. + +## 33-40. Found while speeding up keyframes (intra-speed2 branch, worked around) + +Measured with callgrind on all-intra encodes (3 frames, 1080p, one thread, matched CRF), as the machine has no hardware counters; every workaround kept the output byte-identical. Tov 538446e (installed during the round) closed two earlier gaps: `Target.vectorBits` / `Target.has` answer 27, and `storeLow` / `storeU8` answer 28 (8- and 4-pixel rows stored with them instead of lane-by-lane stores: 0.04 to 0.23% fewer instructions, commit `5e6ea66`). + +33. **No horizontal minimum or argmin.** The I4 mode decision needs the cheapest of nine costs and the first mode reaching it; the scalar loop over the costs took 11.4M instructions on full_motion. Worked around with a swizzle/min ladder, an `eq` against the minimum, `mask()` and `Math.ctz32` (part of `7566916`, -2.1% on full_motion). Wanted: `v.minLanes()` / `maxLanes()` and an index form (`phminposuw` does both for `u16x8` in one instruction). +34. **No shuffle by a runtime index vector** (answered by Tov 538446e's `T.shuffle(table, indices)`, with the branchless `Math.select` requested alongside; not used here yet). The I4 kernel stores every candidate prediction (five 16- or 32-byte stores per 4x4 block, 1.9M instructions on full_motion, 0.27% of the all-intra encode) so the caller can load the chosen mode's by offset; with `pshufb`/`vpermd` by a per-mode index row it would build only the chosen one from the edge vector, so 0.27% is the most it can save. +35. **No multiple return values (tuples are "later").** Kernels with several results write them into a typed-array scratch the caller reloads: the I4 kernel's nine costs plus best mode and cost in an `Int32Array(16)`, the I8 V/H/DC costs, the chroma plane parameters in an `Int16Array(32)` and its edges and DC quads in a 176-byte block. Wanted: returning a record or tuple of ints and vectors, kept in registers when the callee is inlined. +36. **No multiply-high for 16-bit lanes** (`pmulhw` / `pmulhuw` / `pmulhrsw`). capcodec's quantiser adds its rounding offset to the full 32-bit product (`(|c| * q + f) >> qbits`), so the 16-bit path uses `madd` against tables with the odd or even factors zeroed (two `pmaddwd` per 16 coefficients and a recombination) and the rest stays in 32-bit lanes (`vpmulld`). A multiply-high would serve x264's form (`((|c| + bias) * mf) >> 16`), which rounds differently, so it was not measured here. +37. **`mask()` of 16-bit lanes needs a pack first.** `i16x16.eq(...).mask()` lowers to a pack and lane fix-up before `vpmovmskb`; code4's zero test and coefficient count use the byte mask of the same comparison (`u8x32.bitcast`) and halve the popcount (with the `qp % 6` change below, `77e1dc1`: -0.3 to -0.6%). Wanted: `mask()` on 16-bit lanes as `vpmovmskb` plus `pext`, or a `maskBytes()`. +38. **`qp % 6` is a checked signed division** (gap 32's `%`), on every quantiser call: `qp - 6 * qpDiv6(qp)` with the existing multiply-shift `qpDiv6` instead (in `77e1dc1`). +39. **Module-level vector constants are mutable globals.** `const EVEN16 = i16x16(...)` becomes `static tvv_i16x16 g18_EVEN16;` assigned by an initialiser at start-up (as gap 22 for integers), so a `select` against one is an and/andn/or with a memory operand instead of a blend with a constant, and shuffles can't fold it. Wanted: `static const` vector initialisers for literal lanes. +40. **Profiling a split build.** Since programs over 384 KiB of C compile as parallel units, the debug information names temporary C files that are deleted after the build, so `callgrind_annotate` (and `perf annotate`) can't show source lines. `TOV_UNITS=1 tov build -g` restores it (the whole-program build runs 0.4% more instructions than the split one on full_motion). Wanted: keep the unit C files in the cache under stable names in `-g` builds. + +## 41-46. Found while profiling main with B-frames and the lookahead (tov-features branch, numbered 33-38 there) + +41. **The v4 build can't keep 512-bit types at 256 bits.** Refines 27: lane-wise `i32x16`/`u8x64` code lowers to two ymm operations on x86-64-v3 at the instruction count of two `i32x8`s (16-lane `quant4v`/`dequant4v`/`quant8v`/`dequant8v` and a 32-lane `nonzeroMask`: -0.01% to -0.12% instructions, CPU within noise), so one source can carry wide kernels for v3. On v4 the same code becomes sparse zmm instructions inside ymm code and was slower on this AVX512-FP16 Xeon (interleaved best of 7 and 9: all-intra +2.9% and +5.0%, full_motion medium +1.3% and +2.4%), so it was dropped. Four-row `u8x64` SADs (`sad16` and `sadAvg16`, one `vpsadbw` per four rows) are not neutral on v3 (full_motion medium +0.42%, live +0.12% instructions, from building the two ymm halves out of four 16-byte loads and reducing the wider accumulator) and on v4 CPU, best of 9, cost full_motion medium +4.6% (+11.6% with only `sad16` widened) for full_motion live -0.4% (-2.5%) and text_scroll live within noise, so no v4-only SAD module was kept either. Tov 9d70528 (`Target.vectorBits`) and ce80f99 (`--target-cpu x86-64-v3,x86-64-v4` in one binary) in ~/work/tov answer 27's query but are not in the installed 548ff6b compiler (`Target` is N0001 there). Wanted, next to 27's query: a build option (or per-function attribute) that lowers vectors wider than 256 bits as ymm halves while still using AVX-512VL/BW instructions on ymm (mask compares, `vpternlog`, `vpermt2*`), since clang's `-mprefer-vector-width=256` does not split explicit vector types. +42. **No permute with run-time lane indices.** `shuffle`/`swizzle` take literal lists only, so `pshufb`/`vpermd`/`vpermb` with an index vector (and v4's `vpcompressd`) are unreachable. The CABAC significance-map writer `residualBins` (3.7% of instructions on average, 5.5% on all-intra) could compact four positions' significance and last bins through a 16-entry index table, but stays scalar at about 8 instructions per coefficient position. Wanted: `T.permute(v, idx)` (`pshufb`/`vpermd`), `u8x16.lookup(table, idx)` and `v.compress(mask)`. +43. **No bulk append to a `u8[]` from a typed array.** Copying n bytes is n `push`es, each a `TVG_PUSH_C` with its capacity check: `escapeBytes` pushes the 16 bytes of an escape-free chunk one by one, and `endSlice` used to push every CABAC byte onto the slice header's array before escaping the whole array (removing that copy and escaping the coder's buffer in place: -0.9% to -1.3% instructions on screen content and all-intra). Wanted: `arr.pushFrom(ta, start, n)` (one reserve and a `memcpy`). +44. **Pushes skip their capacity check only for local arrays in loops bounded by a local or a literal.** `reserve_for_pushes` does not apply to `inout` array parameters (the `out` of `escapeBytes`, whose inner loop is `j < 16`) or to loops bounded by a field or `.length`, so those pushes stay `TVG_PUSH_C`/`TVG_PUSH_U`. Wanted: reserve before such loops too, for `inout` targets and loop-invariant field or length bounds. +45. **`subarray` allocates.** `ta.subarray(a, b)` creates a typed-array object (`tv_ta_subarray`), so `dst.set(src.subarray(s, s + 8), d)` is no option for short rows; `copyRect`'s 8- and 4-byte rows are written as all loads then all stores, which clang merges into one 8- or 4-byte load and store (the workaround for 28). Wanted: `dst.setFrom(d, src, s, n)` lowering to `memmove`, or `set(src.subarray(...))` without the allocation. +46. **No way to ask for a branchless select.** `c ? a : b` on ints becomes a C conditional, and whether clang emits `cmov` or a branch depends on the surrounding code: five variants of the CABAC bin loop with the same selects (`range = isLps ? rl : rm`, the renormalisation shift) compiled to between 0 and 2 `cmov`s and 22 to 24 conditional jumps. Wanted: `Math.select(c, a, b)` emitted with `__builtin_unpredictable` (or as mask arithmetic). + +## 47-50. Found with Tov 786b6c7 and in the CABAC bin loop (tov-features and tov-features-next branches, numbered 39-42 there) + +Instruction counts are callgrind totals of one-thread encodes (full_motion medium 30 frames and live 30, busy_ui and code_editor medium 90, text_scroll live 90, all-intra medium 10); `-g` and stripped builds of one source differ by under 110 instructions. + +47. **Field reads outside loops still compile differently from a hand-hoisted local, and nothing pins inlining.** 786b6c7 reads a number field once before a loop when nothing the loop runs can assign it, and removing the encoder's hand hoists, mostly before such loops (`const mbw = this.mbw`), was neutral (-0.05% to +0.12%). The lookahead reads `this.w` and `this.h` in straight-line code instead (`intraOf`, `keepOf`, and `candCost`, which `keepOf` calls up to 11 times per macroblock), where the rule does not apply; removing those hoists too cost full_motion +2.68% (medium) and +3.03% (live) instructions, other content +0.02% to +0.16%, all of it in `Lookahead_push` (+112.6M). `Lookahead_push` inlines `keepOf`, its `candCost` calls and `rowsSad`'s 8-row SAD loop; with the field reads clang made 17 copies of that loop instead of 13, and in four of them loop strength reduction kept sixteen row pointers and spilled them (73-75 instructions with 17-25 stack accesses per 8 rows, instead of 42 with none and the stride in a register). Workaround: the hoists stay in the lookahead. Wanted: the same single read for a field in straight-line code (until something that can assign it runs), so a hand-written local and the field read give the same C, and `@noinline`/`@inline` on functions and methods, so a small kernel such as `rowsSad` is compiled once instead of once per inlined call site. +48. **`Math.idiv` by a variable divisor perturbs the same function.** At the constant divisors (`qp / 6`, the quantiser rounding offsets) `Math.idiv` saves the range check (text_scroll live -0.12%). At the 15 variable-divisor sites (and one by 48) it cost full_motion +2.03% (medium) and +2.23% (live) instructions, other content +0.01% to +0.12%. The lookahead's divisions per macroblock (`mb / this.mbw`, the mean of a short bottom row, `keepOf`'s keep factor) take fewer instructions themselves (the keep factor 3.0M instead of 4.8M), but `Lookahead_push` grew by 85.6M through 39's mechanism (17 SAD loops instead of 13, four of them spilling). Workaround: variable divisors stay `Math.trunc(a / b)`, which 786b6c7 already lowers to the `f64` division because a 64-bit `idiv` has under half `divsd`'s throughput. Wanted: `Math.idiv` by a non-constant divisor lowered the same way when both operands are within ±2^53, so choosing the exact operator costs nothing. +49. **No branch-likelihood hint, so block layout follows how a test is phrased.** In the CABAC bin loop, `if (b >= 0) { decision } else if ...` made clang end the decision block with a `jmp` back to the shared word-flush test; matching plain decisions by `b >= 0 && b < SKIP_TERM`, which clang compiles to one unsigned compare, made the decision the loop's fall-through path, one instruction fewer per decision (all-intra -0.77%, where the new skip-run code never runs). Wanted: `if (Math.likely(c))` (or an `expect` form) emitted as `__builtin_expect`, so the hot path's layout does not depend on the form of its condition. Related to 46. +50. **Scalar shift semantics are undocumented and differ from JavaScript.** `<<`, `>>` and `>>>` on `int` shift 64 bits with the count taken modulo 64 (`1 << 32` is 4294967296, `1 << 64` is 1, and `1 << 41` is 2^41 where JavaScript gives 512); the spec states the modulo rule only for vector lanes. The CABAC table lookup relies on it (`e >> (b & 63)` picks a bin's next state out of one 64-bit table entry; the generated C masks the count again and clang folds both into one `sarx`). Wanted: the rule in the spec's operator section, next to the vector one. + +## 51-52. Found in the CABAC range update (tov-features branch, numbered 43-44 there) + +Same measurement setup as 47-50; the compiler is the installed Tov with clang 18.1.3. + +51. **A change inside one class moves inlining in unrelated encoder functions.** Two versions of the CABAC range update differ only inside `Cabac`: one keeps the LPS renormalisation shifts in spare bits of the existing table, the other in a second `Int64Array` field passed to `encodeBins` as one more parameter. Once closure and temporary names are renumbered, their generated C differs in 48 lines, all of them `Cabac` code: its struct, constructor, drop and print functions, `run`, `encodeBins`, and the field initialisation where the `Encoder` constructor allocates its `Cabac`. Clang nevertheless inlined differently: `diffFrame` (468 instructions) was inlined and `mbDiffers` (733) emitted out of line instead, `ensureTiles` went from 150 to 174 instructions and `computeMvpL` from 211 to 157. Relative to the parent commit, the first version changed full_motion medium by +0.84% instructions, all of it `Cabac_run` (+35.2M), and the second by -0.98% (live +0.99% and -0.98%). Between the two, `Cabac_run` differed by the expected -15.4M, while `ensureTiles` (-23.3M), `computeMvpL` (-4.3M), `codeMbAt` (-3.9M) and `entropyRow` (-3.0M) account for most of the remaining -61M. How the change reaches those functions (presumably through the inliner's cost budgets) was not traced. Consequence: a single change's instruction delta on full_motion carries perturbations of about ±1% that it did not cause, so steps are attributed per function from `-g` profiles. Wanted: as 47, `@noinline`/`@inline`, so the encoder's inlining does not move with unrelated edits. +52. **No branchless select, so an unpredictable select can become a branch.** `range = isLps ? lps << ls : mps << ms` compiles to two `cmov`s. The equivalent `range = (isLps ? lps : mps) << rs`, with `rs = isLps ? ls : ms` and `ls` read from a table, compiles to `test r8d, 0x100; je`, which branches on whether the bin is the LPS (close to random in CABAC) and executes the `ls` load only on that path. This is presumably LLVM's x86 cmov-conversion pass, which turns a `cmov` fed by a load into a branch. Workaround: the first form. Wanted: `Math.select(c, a, b)` that stays branchless, or an unpredictable hint lowered to `__builtin_unpredictable`, so latency-bound loops do not depend on how a select is written. Related to 49. + +## 53. Found while shortening the CABAC MPS renormalisation test (tov-features branch, numbered 45 there) + +Same setup as 51-52. + +53. **No way to keep an expression's association, so a constant can be moved onto the critical path.** In `encodeBins`, `ms = (range - 256 - lps) >>> 63`, next to `mps = range - lps`, is meant to subtract `lps` from `range - 256`, which is ready before the table lookup gives `lps`. Clang instead computes `mps` and derives `ms` from it (`sub r8, rbx; lea rdx, [r8-0x100]; shr rdx, 0x3f`), presumably to share the subtraction. That makes the MPS test three operations after `lps` instead of two, and puts one more cycle on the range chain that limits the bin loop. Reading the `-256` from memory once before the loop keeps the intended order (`add rcx, r12; sub rcx, r9; shr rcx, 0x3f`) at one more instruction per decision (all-intra 2,487,389,509 instead of 2,466,833,084 instructions). Yet that version is faster: interleaved 1-thread medians over 9 reps are busy_ui all-intra 2.224 s against 2.257 s, code_editor all-intra 2.088 s against 2.170 s and all-intra at QP 26 0.714 s against 0.728 s (A/A 1.006, 1.006, 0.991). Neither beat the committed `lps > range - 256 ? 1 : 0` (`cmp` and `setg`), so neither was kept. Wanted: an opaque-value hint (as Rust's `core::hint::black_box`, an empty `asm("" : "+r"(x))` in C) so a latency-bound loop can fix where a constant is applied without loading it from memory. diff --git a/capcodec/bench/bench.py b/capcodec/bench/bench.py new file mode 100644 index 00000000000..c64eb271faf --- /dev/null +++ b/capcodec/bench/bench.py @@ -0,0 +1,123 @@ +#!/usr/bin/env python3 +import argparse +import dataclasses +import json +import subprocess +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from harness import report # noqa: E402 +from harness.common import BENCH, MANIFEST, OUT_DIR, ROOT, Cache, ffmpeg_version, git_commit, log, ncores # noqa: E402 +from harness.runner import GATE_ENCODER, REFERENCE, TIERS, Bench, gate, machine_id, summarize # noqa: E402 + +BEST = BENCH / "best.json" +HISTORY = BENCH / "history.jsonl" + + +def committed_best() -> dict | None: + # A standalone checkout stores the file at bench/best.json. Inside Cap it is capcodec/bench/best.json. + for spec in ("HEAD:bench/best.json", "HEAD:capcodec/bench/best.json"): + r = subprocess.run(["git", "-C", str(ROOT), "show", spec], capture_output=True, text=True) + if r.returncode == 0 and r.stdout.strip(): + return json.loads(r.stdout) + if BEST.exists(): + return json.loads(BEST.read_text()) + return None + + +def main() -> int: + ap = argparse.ArgumentParser(description="capcodec benchmark harness") + ap.add_argument("tier", choices=sorted(TIERS)) + ap.add_argument("--encoders", help="comma-separated adapter names (prefix* allowed)") + ap.add_argument("--clips", help="comma-separated clip names") + ap.add_argument("--jobs", type=int, default=ncores()) + ap.add_argument("--no-cache", action="store_true") + ap.add_argument("--manifest", type=Path, default=MANIFEST) + ap.add_argument("--out", type=Path) + ap.add_argument("--update-best", action="store_true", help="write bench/best.json when the gate encoder improved") + ap.add_argument("--accept", action="store_true", help="write bench/best.json from this run unconditionally") + ap.add_argument("--no-history", action="store_true") + ap.add_argument("--list", action="store_true", help="list adapters and availability") + ap.add_argument("--quality-only", action="store_true", + help="skip speed, matched-speed and proportionality runs (speedcmp.py measures speed)") + args = ap.parse_args() + + tier = TIERS[args.tier] + if args.quality_only: + tier = dataclasses.replace(tier, speed=False, matched_key=None, proportionality=False) + out_dir = args.out or (OUT_DIR / tier.name) + out_dir.mkdir(parents=True, exist_ok=True) + encoders = args.encoders.split(",") if args.encoders else None + clips = args.clips.split(",") if args.clips else None + bench = Bench(tier, Cache(enabled=not args.no_cache), out_dir, encoders, clips, args.jobs, args.manifest) + if args.list: + for a in bench.adapters: + print(f"available {a.name:34s} {a.description}") + for n, why in sorted(bench.unavailable.items()): + print(f"unavailable {n:34s} {why}") + return 0 + if not bench.clips: + log(f"no clips available (manifest {args.manifest}); generate the corpus first") + return 2 + t0 = time.time() + rows = bench.run() + refs = [REFERENCE] + sorted({a.name for a in bench.adapters if getattr(a, "production", False)}) + refs = [r for r in refs if any(x["encoder"] == r for x in rows)] + summ = summarize(rows, tier, refs) + report.write_csv(rows, out_dir / "results.csv") + (out_dir / "summary.json").write_text(json.dumps(summ, indent=1, sort_keys=True, default=str)) + notes = [] + if any(a.family == "browser" for a in bench.adapters): + notes.append("Browser encoders run in headless Google Chrome on Linux, so MediaRecorder and WebCodecs use " + "Chrome's software H.264 path (OpenH264), not a hardware encoder.") + if bench.unavailable: + notes.append("Not available on this machine: " + ", ".join(f"{k} ({v})" for k, v in sorted(bench.unavailable.items()))) + meta = {"tier": tier.name, "commit": git_commit(), "machine": machine_id(), "ffmpeg": ffmpeg_version(), + "clips": len(bench.clips), "elapsed_s": f"{time.time() - t0:.0f}"} + ok, msgs = (True, []) + best = committed_best() + if tier.gate: + ok, msgs = gate(summ["summary"], best) + report.write_html(out_dir / "report.html", tier.name, rows, summ, refs, meta, out_dir / "crops", notes + msgs) + entry = {"time": time.strftime("%Y-%m-%dT%H:%M:%S"), "tier": tier.name, "commit": git_commit(), + "machine": machine_id(), "elapsed_s": round(time.time() - t0, 1), "clips": [c.name for c in bench.clips], + "summary": summ["summary"], "gate": {"ok": ok, "messages": msgs} if tier.gate else None} + if not args.no_history: + report.append_history(HISTORY, entry) + s = summ["summary"].get(GATE_ENCODER, {}) + cand = {"bd_psnr_yuv": s.get(f"bd_psnr_yuv_vs_{REFERENCE}"), "cpu_fps": s.get("cpu_fps_median"), + "commit": git_commit(), "tier": tier.name, "clips": [c.name for c in bench.clips]} + if tier.gate and cand["bd_psnr_yuv"] is not None: + better = best is None or ( + (best.get("bd_psnr_yuv") is None or cand["bd_psnr_yuv"] <= best["bd_psnr_yuv"]) + and (not best.get("cpu_fps") or (cand["cpu_fps"] or 0) >= best["cpu_fps"] * 0.97)) + if args.accept or (args.update_best and ok and better): + BEST.write_text(json.dumps(cand, indent=1, sort_keys=True) + "\n") + msgs.append(f"wrote {BEST}") + for e, s in sorted(summ["summary"].items()): + bits = [f"{k}={v:+.2f}%" if k.startswith("bd_") and not k.endswith("_n") else None + for k, v in s.items() if v is not None] + bits = [b for b in bits if b] + fps = s.get("cpu_fps_median") + sp = s.get("speed_wall_fps_median") + print(f"{e:34s} {' '.join(bits)} cpu-fps={'' if fps is None else f'{fps:.1f}'}" + f"{'' if sp is None else f' wall-fps={sp:.1f}'} decode_fail={s.get('decode_failures')} " + f"errors={s.get('errors')}") + for surf, t in sorted((s.get("vs_cap_targets") or {}).items()): + deltas = [f"{label}={t[k]:+.2f}" for label, k in (("dVMAF", "d_vmaf_mean"), ("dPSNR", "d_psnr_yuv_mean")) + if t.get(k) is not None] + if t.get("cpu_speedup_geomean"): + deltas.append(f"cpu x{t['cpu_speedup_geomean']:.2f}") + print(f"{'':34s} vs {surf} at Cap bpp targets ({t['n']}): |err| {t['our_abs_err_mean'] * 100:.1f}% " + f"(theirs {t['their_abs_err_mean'] * 100:.1f}%) {' '.join(deltas)}") + for m in msgs: + print(m) + print(f"report: {out_dir / 'report.html'}") + return 0 if ok else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/capcodec/bench/best.json b/capcodec/bench/best.json new file mode 100644 index 00000000000..75f9d0290ad --- /dev/null +++ b/capcodec/bench/best.json @@ -0,0 +1,7 @@ +{ + "bd_psnr_yuv": 1.5927990279928197, + "clips": ["code_editor", "text_scroll", "slides"], + "commit": "1c82bfaeb12e-dirty", + "cpu_fps": 291.64864116037273, + "tier": "smoke" +} diff --git a/capcodec/bench/build_encoder.sh b/capcodec/bench/build_encoder.sh new file mode 100755 index 00000000000..8315ca929d8 --- /dev/null +++ b/capcodec/bench/build_encoder.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")/.." +mkdir -p build +TOV="${TOV:-tov}" +CPU="${CAPCODEC_TARGET_CPU:-x86-64-v3}" +if [ "$(uname -s)" = Linux ] && [[ " $* " != *" -g "* ]]; then + FLAGS="-fno-asynchronous-unwind-tables -fno-unwind-tables -Wl,-s" + if command -v ld.lld >/dev/null; then + FLAGS="$FLAGS -fuse-ld=lld" + fi + export TOV_CFLAGS="$FLAGS ${TOV_CFLAGS:-}" +fi +"$TOV" build cli/main.tov -o build/capcodec --target-cpu "$CPU" "$@" +"$TOV" build cli/main.tov -o build/capcodec-unchecked --unchecked --target-cpu "$CPU" "$@" +"$TOV" build cli/main.tov -o build/capcodec-unchecked-baseline --unchecked "$@" +if [ "$(uname -m)" = x86_64 ]; then + "$TOV" build cli/main.tov -o build/capcodec-unchecked-v4 --unchecked --target-cpu x86-64-v4 "$@" +fi diff --git a/capcodec/bench/corpus/README.md b/capcodec/bench/corpus/README.md new file mode 100644 index 00000000000..bac8e853086 --- /dev/null +++ b/capcodec/bench/corpus/README.md @@ -0,0 +1,352 @@ +# Lossless screen-recording corpus + +Synthetic but app-like screen recordings for benchmarking the capcodec +screen-content H.264 encoder. Every clip is a real-time `x11grab` capture of +Chrome on a private Xvfb display, driven by scripted human-like input. The +captures are stored losslessly next to per-frame side info: dirty rects, +cursor position and scroll vectors. One clip, `full_motion`, is the worst +case for a screen encoder: a natural video (Big Buck Bunny) playing full +screen. + +Scripts, pages and the manifest live here (`bench/corpus/`). Captures and side +info go to `$CAPCODEC_CORPUS`, which defaults to `/home/ubuntu/work/corpus` and is +never committed. + +## Regenerating + +```sh +cd bench/corpus +python3 generate.py --list # clip table +python3 generate.py --clip slides # one clip (repeatable) +python3 generate.py --all # everything (~20 min) +python3 generate.py --all --missing # only clips without outputs +python3 generate.py --clip idle --sideinfo-only # recompute side info, previews, manifest entry +python3 generate.py --verify-lossless # codec round-trip check +``` + +Run long jobs inside tmux (`tmux new-session -d -s corpus-gen ...`). Each run +is idempotent: the clip's work dir (`$CAPCODEC_CORPUS/.work/`, which +holds the raw capture and the Chrome profile) is wiped, every output is +overwritten, and the clip's manifest entry is replaced. The work dir is +deleted again once the outputs are written. Clips are written to +`manifest.json` in `CLIPS` order. + +Environment: + +| variable | default | +|---|---| +| `CAPCODEC_CORPUS` | `/home/ubuntu/work/corpus` | +| `CAPCODEC_CACHE` | `/home/ubuntu/work/cache/capcodec-corpus` (downloaded source video, segment, match features; outside git) | +| `CAPCODEC_FFMPEG` | `/home/ubuntu/tools/bin/ffmpeg` (8.1) | +| `CAPCODEC_FFPROBE` | `ffprobe` next to `CAPCODEC_FFMPEG` | + +Requirements: Xvfb, xdotool, python3 with `python3-xlib`, `numpy` and +Playwright, and Google Chrome (`channel="chrome"`, tested with 148). The pages +use locally installed fonts (Noto Sans, Liberation, Inter, JetBrains Mono, +Bitstream Charter). `sudo -n renice` is used when available. Displays: `:99` at +1920x1080x24 is reused if it is already running (otherwise it is started), and +`:98` at 3840x2160x24 is started and stopped for `code_4k`. `:1` is refused. +Pages are `file://` URLs and all data (code, article text, board data) is +embedded in `pages/data/*.js`. The only network access is the one-time +download of the `full_motion` source video into `$CAPCODEC_CACHE`, which is +verified by sha256. Captures never touch the network. + +## Clips + +| name | size @ fps | s | page | content | +|---|---|---|---|---| +| slow_typing | 1920x1080@30 | 40 | `docs.html` | Docs-like editor, 15px text, human typing with pauses and BackSpace fixes | +| code_editor | 1920x1080@30 | 40 | `code.html` | VS Code Light+, 13px JetBrains Mono, typing with auto-indent, wheel/PageDown/caret | +| text_scroll | 1920x1080@30 | 40 | `article.html` | Wikipedia-like article: notches, flicks, eased smooth scroll, selection, hovers | +| slides | 1920x1080@30 | 40 | `slides.html` | deck advancing every ~3 s: cuts, fades, push/cover slide-ins, builds | +| busy_ui | 1920x1080@30 | 40 | `board.html` | dense issue board, constant curved pointer motion, menus, drags, scroll | +| dashboard | 1920x1080@30 | 40 | `dashboard.html` | 11-12px tables, KPI tiles, 1 s updates, animated charts and sparklines | +| dark_mode | 1920x1080@30 | 40 | `dark.html` | VS Code Dark+ and a terminal streaming logs, occasional typing | +| idle | 1920x1080@30 | 60 | `idle.html` | static desktop/PDF viewer, ticking clock, 3 toasts, 4 pointer moves | +| webcam_overlay | 1920x1080@30 | 40 | `slides.html?webcam=1` | slides with a 320px circular canvas webcam (gradient, figure, noise), dragged 4 times | +| busy_ui_60fps | 1920x1080@60 | 20 | `board.html` | short busy_ui variant at 60 fps | +| code_4k | 3840x2160@30 | 20 | `code.html` | DSF 2 on a 4K Xvfb, code typing and scrolling | +| full_motion | 1920x1080@30 | 30 | `video.html` | Big Buck Bunny 27.47-57.43 s full screen in a `