Skip to content

refactor!: one compressed track per name - #7046

Merged
kixelated merged 6 commits into
mainfrom
quest/m1/compress/migrate
Oct 28, 2026
Merged

kixelated merged 6 commits into
mainfrom
quest/m1/compress/migrate

Conversation

@kixelated

@kixelated kixelated commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Implements quest/m1/compress/migrate.md, and deletes that quest.

Problem

moq-net now compresses tracks marked compress: true. Even so, first-party compressed data still went out as an uncompressed track plus an application-compressed .z sibling (catalog.json.z, the moq-stats .z tracks). That meant duplicate tracks, duplicate codec state, and a flate library (moq-flate, @moq/flate, and the bindings' flate namespaces) in every stack.

Approach

  • One compressed track per name. catalog.json (moq-mux and js/publish), each moq-stats .json track, the <track>.hls record tracks, the CLI's stats.json and feedback report tracks, and the moq-c JSON tracks are created with compress: true. hang::Catalog::default_track_info() sets it.

  • Consumers read a compressed track like a plain one: the hang/moq-mux catalog consumers, js/watch, the moq-json and @moq/json decoders, and every binding's JSON consumer lose their compression options.

  • moq-stats: TrackPair became one Output per track and keeps merge-patch deltas with refresh(8). Measured over 600 ticks with moq-net's codec (raw DEFLATE level 6, sync flush per frame, fresh window per group), deltas save 35–82% over compressed snapshots:

    track refresh compressed snapshot compressed delta delta saves
    totals, 1 tier every 8 frames 148,595 B 44,170 B 70.3%
    totals, 4 tiers every 8 frames 251,627 B 97,790 B 61.1%
    prefix, 20 children every 8 frames 313,545 B 155,409 B 50.4%
    prefix, 200 children every 8 frames 1,050,155 B 678,139 B 35.4%
  • Drafts: draft-lcurley-moq-hang drops the compression field and catalog.json.z, and its Compression section now calls compression a track property that the transport carries. draft-lcurley-moq-e2ee drops the .z sibling but keeps compress-before-AEAD.

  • Quests: quest/m1/compress/README.md is sized [S] for its remaining end-to-end proof and guide.

Choices made without asking:

  • The MSF catalog track stays uncompressed, because third-party moq-transport readers consume it.
  • Empty #[non_exhaustive] config structs stay where the Rust API takes one (moq_json::stream::Config, window::ConsumerConfig, moq_stats::consume::Config). Empty FFI records, Go option structs, and C structs are removed instead, along with stream::Decoder::reset.
  • The hang binary catalog section stays. Only moq-mux's binary_* producer helpers go.
  • catalog::msf::Consumer doesn't reuse hang's escape check. MSF namespaces are absolute, so the check could never fire there.
  • Tests that measured compressed frame sizes through a local reader are deleted. Local readers now see inflated frames.

Impact

All breaking; each package's MIGRATE.md has the migration.

  • Wire / catalog: catalog.json.z and the moq-stats .z tracks are gone. Readers subscribe to catalog.json and the .json stats tracks, which moq-net compresses. hang's catalog compression field is removed (Rust, JS schema, draft).
  • hang: Catalog::COMPRESSED_NAME, TRACK_COMPRESSED, and the compression field are removed. default_track_info() sets compress: true.
  • moq-mux: CatalogFormat::HangZ, the moq_mux::binary module, and catalog::Producer::binary_* are removed.
  • moq-json / @moq/json: Config::compression, Compression, Desync, and stream::Decoder::reset are removed. @moq/json now depends on @moq/signals.
  • moq-stats: Encoding and TrackPair are removed. totals_track, presence_track, and prefix_track take no encoding. consume::Config is empty, and aggregate::Config loses encoding.
  • moq-flate, @moq/flate: deleted.
  • CLI / @moq/watch: the hangz catalog format is removed.
  • moq-ffi: the MoqFlate* objects and MoqJsonStreamConfig are removed.
  • moq-c: moq_binary_config and moq_publish_binary_* are removed.
  • Python, Kotlin, Swift, Go, Dart: the flate namespace and the JSON compression options are removed. The Dart bindings and the JS API reports are regenerated.

Alternatives

  • Snapshot-only moq-stats tracks. Simpler, but compressed deltas are 35–82% smaller (table above), so the deltas stay.
  • Compress the MSF catalog too. Rejected because third-party readers consume it.

Checks

  • cargo check --workspace --all-targets --all-features, cargo clippy -D warnings, and cargo test --all-features on the touched crates pass. JS biome, build with API reports, tests for the changed packages, and tsc for @moq/json pass. sh/dart/generate.sh, just drafts check, and quest check --base origin/main pass.
  • Not run locally: the Swift, Kotlin, Go, Python, and Dart wrapper suites and the interop lanes.

Follow-ups

  • Encoder heap on long-lived groups. A compressed group's encoder holds about 256 KiB (flate::ENCODER_HEAP) while the group is open. HLS record and stats tracks keep groups open, so each rendition's few-byte record track now costs 256 KiB. Options: a smaller window or memLevel for small groups, or drop the encoder between writes and re-prime it.
  • E2EE and compressed catalogs. default_track_info() now compresses catalog.json, so an encrypted catalog would be compressed as ciphertext, which the e2ee draft forbids. The m2 e2ee quests (quest/m2/net-wrap/e2ee.md, quest/m2/e2ee/rust-publish.md) must compress before AEAD.
  • Wire-size visibility. The moq-json telemetry example no longer reports wire bytes, because a local reader sees inflated frames. A moq-net helper or stats hook that exposes a group's stored size would restore it.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 5 commits October 28, 2026 00:01
Every first-party compressed track is now one name with compress: true:
catalog.json, the moq-stats tracks, HLS records, and the CLI's reports.
catalog.json.z, the stats .z siblings, hang's catalog compression field,
hangz, moq-flate, @moq/flate, moq-mux::binary, moq-json's Config::compression,
and the bindings' flate namespace are deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… API reports

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title One compressed track per name refactor!: one compressed track per name Oct 28, 2026
@kixelated
kixelated marked this pull request as ready for review October 28, 2026 09:15
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T21:01:35.461170Z 313c819 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: d7e43b4.

No actionable bugs found in the inspected changes. Moving compression into track metadata is a sound simplification: it removes duplicate codec state and sibling tracks while preserving catalog retention/priority, stream single-group failure semantics, and stats demand/refresh handling. Keeping merge-patch deltas and leaving MSF uncompressed are justified distinctions rather than unnecessary parallel implementations.

Verification limits: static review of the changed Rust/JS runtime paths, relevant surrounding code, tests, and C/FFI/wrapper API wiring, including an independent Rust pass. I did not run builds, tests, wrapper suites, or network interoperability checks; this is not a claim that those pass.

@kixelated
kixelated added this pull request to the merge queue Oct 28, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 28, 2026
Resolve conflicts with the binding publication wrapper (go README, carla runtime test) and keep both Unreleased MIGRATE entries in moq-c and moq-mux.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merged main into the branch (313c819) after the queue dropped it for a merge conflict:

  • go/wrapper/README.md and demo/carla/sim/tests/test_runtime.py: main's binding publication wrapper (CreateBroadcast returns a publication; the broadcast is publication.Broadcast()/.broadcast) combined with this PR's compressed TrackInfo and the info=None fake.
  • rs/moq-c/MIGRATE.md, rs/moq-mux/MIGRATE.md: kept both Unreleased entries (this PR's compression section above main's #7110 identity section).

just check origin/main passed locally, CI passed, and Codex reviewed the new head without findings. quest check is clean. Landing again.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit a7441f2 Oct 28, 2026
11 checks passed
@kixelated
kixelated deleted the quest/m1/compress/migrate branch October 28, 2026 22:16
michaelstuurman added a commit to michaelstuurman/moq that referenced this pull request Oct 29, 2026
moq-dev#7046 removed moq_json_stream_config, moq_binary_config and the binary
track functions, and slimmed moq_json_snapshot_config to `deltas`. The
data-track bitrate this branch added to them is dropped with them; the
audio and video hints stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated added a commit that referenced this pull request Oct 30, 2026
Main removed per-record compression (#7046), which most of
this branch's change was about, so the conflicted files take main's side
and the remaining change is redone on top.
kixelated added a commit that referenced this pull request Oct 30, 2026
Adopt #7046: the window track is compressed like the other moq-c JSON
tracks, `moq_json_window_config` keeps only `checkpoint_records`, and
`moq_consume_json_window` drops its `config` argument.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant