Skip to content

test: halve the slowest tests' CPU - #6987

Open
kixelated wants to merge 22 commits into
mainfrom
quest/m0/ci/slow-tests
Open

kixelated wants to merge 22 commits into
mainfrom
quest/m0/ci/slow-tests

Conversation

@kixelated

@kixelated kixelated commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

quest/m0/ci/slow-tests.md: the 103 test IDs frozen in moq-dev/moq#6914 took 137.83 CPU-seconds at baseline. The target is under 68.92 s, with every case and its coverage kept. Current main measures 100.55 s.

Result

Two full normal runs of this branch measure the cohort at 61.02 s and 67.49 s. Both are under 68.92 s. All 103 IDs ran and passed. Whole-suite test CPU fell from 206.2 s to 166.0 to 178.5 s.

Run-to-run noise is about 10%, mostly in the export timing tests under host contention. Run 2 clears the target by only 1.4 s, so a noisier host could put it over.

Module Tests Main s Run 1 s Run 2 s
moq-mux container::ts::export_timing_test 14 24.7 11.2 15.3
moq-mux container::ts::recovery_test 18 11.8 10.2 10.5
moq-net::integration routing_contracts 5 10.9 6.3 6.5
moq-quic 9 8.4 3.0 3.4
moq-net::integration mesh_withdraw 6 8.4 8.2 8.8
moq-mux jitter::tests 8 6.8 5.1 5.1
moq-json 8 6.4 1.8 2.1
moq-net 9 6.0 3.4 3.5
moq-stats 4 4.5 4.4 4.4
moq-net::integration catalog_churn 2 2.0 1.9 2.0
moq-tokio::integration 4 1.7 1.4 1.2
moq-audio 2 1.5 0.7 1.0
qmux 1 1.3 0.4 0.4
moq-video 1 1.2 0.2 0.2
moq-mux rate::tests::model 1 1.2 0.2 0.2
moq-pattern::integration 1 1.1 0.6 0.6
everything else 10 2.9 2.1 2.4

Changes

No test was deleted, ignored, or moved to nightly. No assertion or bound was loosened.

moq-mux

  • export_timing_test: the Live harness advances its own kio::clock::Deadlines queue instead of sleeping on the tokio paused clock, the same switch the jitter tests made in test(mux): reduce fixture CPU and checkpoint frozen cohort #6914.
    • A leg that reads after every frame now polls only once its export has been woken. 41k of 60k polls had nothing to wake them, and none returned a slot.
    • assert_same_packets compares 188 bytes at a time and only drops to bytes to name the offset that differs.
    • Every rendered slot's payload hash matched before and after across all 21 tests.
    • drain advances that queue 10 ms itself when nothing is due, instead of a kio::clock::timeout armed on a queue nothing else advances, which hung four tests once main's export stopped reporting the end there.
    • Frames now arrive on their exact microsecond. Tokio rounded sleeps up to the millisecond, so some slot send instants moved by up to about 21 ms.
  • export_timing_test: the three drifting-source cases measure 60 s of media instead of 300 s. See Decisions.
  • recovery_test:
    • x265 runs with pools=none:frame-threads=1, since it ignores -threads 1. This also makes the bitstream the same on every host.
    • The reference decode hashes pictures with murmur3 instead of MD5.
    • A run concatenates its TS output once instead of up to five times.
  • jitter: the drift and feed helpers drain due frames without collecting them. Steer::measure (production) skips the depth sort when only one track qualifies, because the depth only breaks ties between tracks.
    • The jitter_steering release bench: 1-track cells about 7% faster (1/300: 287 to 267 µs), 4, 16 and 64-track cells unchanged within noise.
  • rate model proptest uses RngAlgorithm::XorShift. ChaCha seeding on every weighted prop_oneof fork was 60 to 80% of the run. The case count, seed and strategies are unchanged.
  • fixture::Mux::packet builds into a [u8; 188], with identical bytes.

moq-net

  • routing_contracts reference model (test only): each identity's route is selected once per pass instead of up to seven times. Links live in a dense matrix, and per-peer views are restricted once per distinct interest.
    • handoff::loop_free skips identities that were never originated.
    • A temporary hash of every Net's full state, events, stats and queues was byte-identical before and after across all 56 routing_contracts tests.
  • front::tests::bounded_sequences_hold_the_invariants renders Debug only when an event emitted something other than Detach. It is the same check in contrapositive form, with cheap fields compared first.
  • a_busy_track_yields_to_its_siblings and a_busy_subscription_yields_to_its_siblings serve 2,000 groups instead of 10,000. See Decisions.

moq-json

  • The window encoder reads its group budget from a private max_group_bytes field, and the stream encoder its per-record limit from a private max_frame_bytes field (main's feat(net)!: shed group fronts past a 32 MB tail or under pool debt #6993 dropped the stream's group budget). Both default to MAX_CACHE_BYTES, so behavior is unchanged. This mirrors the snapshot encoder's existing field.
    • Their oversize tests use a 32 KiB limit and exercise the same refusal and roll branches. a_compressed_log_is_charged_its_inflated_size from the table below no longer exists on main.
  • The two codec property tests use XorShift, for the same reason as rate.

moq-quic

  • The sim payload() returns a slice of one shared 251-byte cycle instead of building 64 KiB byte by byte on every send attempt. Streams under 251 apart still differ at every offset.
  • subscribe() reads RUST_LOG and defaults to info instead of formatting every TRACE event. The per-step client and server spans are debug. The runs are deterministic, so a failure replays with RUST_LOG=trace. Main's subscribe_at is gone: its one caller, the probe runs, wanted info, which subscribe() now gives.

Elsewhere

  • qmux keep-alive: builds its 100 datagrams once instead of 200,000 times.
  • moq-audio decode corpus: replays 8 kHz PCM, since only a packet's duration feeds the frame term.
  • moq-video transcode: frames are an eighth as wide. Color::infer reads only the height.
  • moq-cli passthrough: joins its output with concat().
  • moq-relay mesh: reuses one reqwest::Client per mesh. Building one per 10 ms poll loaded the system trust roots each time.
  • moq-tokio zero-backoff: a 100 ms give-up window instead of 500 ms. The bug resets the window on every session, so any length catches it.
  • moq-hls (production): a file playlist builds no HTTP client, so it also skips the rind rustls install main added before building one. same_origin already kept every fetch on the entry's scheme, so it never used one, but building it parsed the system trust roots.
  • Cargo.toml: rubato builds at opt-level 3 in dev, as sonora already does. Its SIMD intrinsics are the whole resampler invariant test (0.9 to 0.14 s).

Decisions

Settled by the maintainer in a /quest-complete session, which accepted both recommendations.

  1. Measure the drifting-source cases over 60 s instead of 300 s (a_slow_source_clock_keeps_the_delay, a_fast_source_clock_keeps_the_delay, a_source_past_the_limit_is_counted; commit test(mux): measure a drifting source's rate over a minute). It saves about 6.5 s, without which the cohort misses the target.
    • Measured rates at 10 s are within 0.002 ppm of the 300 s values (29.99919 vs 29.99936; -30.00099 vs -30.00116; 399.84023 vs 399.84032), against bounds of 0.5 and 1 ppm. Neither RATE_WINDOW nor RESPONSE (both 600 s) fills by 300 s, so the steering stays in the same regime.
    • Margins at 60 s and 300 s: spread 15.7 vs 22.8 ms under a 187 ms bound, 0 drops, margin 498 vs 491 ms.
    • Tradeoff: 30 steering steps instead of 150, so fewer chances for the spread check to catch an anchor glitch.
    • Settled: keep.
  2. Serve 2,000 groups instead of 10,000 in the two busy-sibling yield tests (saves about 1.0 s).
    • Every poll is still checked against the 32-group budget, the sibling's turn and the stall bound. Two producer batches of 1,000 keep the backlog well past the budget, and the original bug shows on the first poll.
    • Settled: keep.

Also chosen while iterating:

  • XorShift draws different concrete proptest cases than ChaCha did, from the same distribution with the same seed and case count.
  • moq-quic test logs default to info.
  • The rubato opt-level follows the sonora precedent.

Impact

  • Public API: none. The moq-json budget field and the moq-hls client are private.
  • Wire: none.
  • Production behavior: unchanged. Steer::measure picks the same slope, and a file HLS import makes no HTTP request, as before.

Validation

  • quest gate nix develop -c just ci origin/main on cf952d0 (main 3164bce): every module passed but check flake, which fails on this host because the Nix builder cannot create user namespaces (unshare: write failed /proc/self/uid_map). test rs: 10,908 tests passed, 124 skipped.
  • Earlier: just check passed at 6d9931e (10,386 tests).
  • quest check --base origin/main passed.
  • Measured with the wrapper and command from test(mux): reduce fixture CPU and checkpoint frozen cohort #6914 (wrapper SHA-256 d16e86fb379a8d68b301f1471d5f135517faa5225f8fbaa1156f3e7405e5ef48), under just rs test --workspace --exclude moq-net-fuzz --exclude moq-quic-fuzz, zero retries. Same machine as the frozen baseline (Intel Core Ultra 7 270K Plus, 24 logical CPUs).
    • Main is fbeac8a9fd.
    • Runs 1 and 2 are this branch at 6d9931ef90; the later commit only removes the quest.
    • The host was shared, with a load average of about 10 during the runs.

Tried and dropped

  • select_nth_unstable_by for the jitter medians: the release bench regressed 15 to 25% at 300 floors. sort_unstable_by was neutral in release and cost 45% more in debug.
  • opt-level = 3 for serde_json: about 20% on the stats oversize tests, because its generic code is instantiated in the calling crates at opt-level 0.
  • Splitting the moq-net mock's Link::route by stream kind: at most 4% of the mesh tests, and it changes the wakeup order the pinned mesh counts depend on.
  • -aac_coder fast for the recovery clips: no change.

Follow-ups

  • Encode the recovery clips once per run instead of per test process. Each test pays an FFmpeg encode plus a reference decode, roughly 35 to 40% of recovery_test's CPU. Options: commit the clips (about 320 KB each), generate them in a nextest setup script, or keep a lock-guarded cache keyed by FFmpeg version.
  • Cheaper un-woken Export::poll_next. Track fill through moq-net's Subscriber::poll_group is about 24% of an export poll, and catalog polling through moq_json::snapshot::Consumer::poll_latest into Cursor::poll_next_group is 11 to 19%. This helps every export test and production. Needs a benchmark.
  • moq-net session cost in the random-mesh tests. AnnouncePrefix::poll_run, origin sync_route/requests_route, OriginScope::relative rebuilding Patterns on every call, and per-session setup (about 25%) dominate mesh_withdraw. Needs benchmarks.
  • moq-quic frame::Iter yields one Frame::Padding per padding byte. In debug, that move is about 30% of handshake CPU. Collapse runs, after checking what the frame_rx padding stats should count.
  • tracing-test's no-env-filter, enabled by moq-tokio, unifies across the build, so every crate formats at TRACE. That is 11 to 17% of the moq-tokio integration tests.
  • moq-stats oversize tests (3.3 s) push past the hard 32 MiB cache through a debug JSON pipeline. Production option: the snapshot encoder refuses an oversized plain snapshot before parsing it back, as it already does for deflate.
  • a_fill_refused_group_by_group_stays_linear (1.8 s) runs 100,000 refusals. A smaller count catches a plain quadratic but not a cheap per-refusal copy.
Per-test CPU seconds: frozen baseline, main, and two runs of this branch
Binary ID Frozen test ID Baseline Main Run 1 Run 2
moq-mux container::ts::export_timing_test::two_legs_render_a_drifting_source_the_same_way 5.64 5.81 3.77 4.47
moq-net::integration routing_contracts::handoff::sweep_handoff 4.51 4.38 2.61 2.64
moq-mux jitter::tests::the_clock_follows_a_drifting_source_for_a_day 4.94 3.59 2.53 2.53
moq-mux container::ts::export_timing_test::a_joiner_keeps_the_system_clock_in_tolerance 3.48 3.48 2.33 3.50
moq-mux container::ts::export_timing_test::a_fast_source_clock_keeps_the_delay 3.71 3.47 0.46 0.66
moq-mux container::ts::export_timing_test::a_source_past_the_limit_is_counted 3.18 3.43 0.49 0.78
moq-mux container::ts::export_timing_test::a_slow_source_clock_keeps_the_delay 3.15 3.43 0.46 0.53
moq-net::integration routing_contracts::handoff::sweep_requests_reach_a_quiet_origin 3.40 3.26 1.83 1.88
moq-net::integration mesh_withdraw::residual_random_meshes 3.32 3.08 3.00 3.22
moq-net::integration mesh_withdraw::capped_random_meshes_hunt_less 3.33 3.01 2.96 3.17
moq-net model::front::tests::bounded_sequences_hold_the_invariants 2.25 2.20 0.58 0.59
moq-quic tests::sim::random_workloads 2.13 2.03 0.72 0.74
moq-net::integration routing_contracts::sweep_forgetting_distances 2.06 1.99 1.10 1.14
moq-stats produce::tests::an_oversize_group_leaves_other_groups_readable 2.26 1.92 1.89 1.91
moq-net model::track::test::a_fill_refused_group_by_group_stays_linear 2.35 1.80 1.77 1.80
moq-quic tests::sim::replays_from_seed 1.83 1.79 0.17 0.22
moq-json window::encoder::test::a_header_larger_than_the_group_cache_is_rejected 1.37 1.39 0.00 0.00
moq-stats produce::tests::oversized_frame_ends_both_flavors 2.08 1.35 1.35 1.36
qmux session::tests::websocket_keep_alive_runs_during_continuous_output 1.35 1.30 0.37 0.37
moq-json window::encoder::test::an_op_that_would_evict_the_header_rolls_first 1.27 1.28 0.00 0.00
moq-video encode::encoder::tests::transcode_keeps_the_space_the_source_signalled 1.24 1.18 0.18 0.18
moq-mux rate::tests::model::invariants_hold_on_random_walks 1.15 1.15 0.20 0.19
moq-net::integration catalog_churn::catalog_churn_lite_07 1.14 1.13 1.11 1.12
moq-mux container::ts::recovery_test::hevc_keeps_radl_after_a_hole 1.13 1.11 0.82 0.86
moq-pattern::integration pattern_exhaustive::intersect_matches_exactly_what_both_match 1.09 1.09 0.57 0.57
moq-mux container::ts::recovery_test::hevc_trims_only_orphaned_rasl 1.06 1.05 0.84 0.86
moq-quic tests::probe::congested 0.97 0.99 0.39 0.56
moq-net::integration routing_contracts::sweep_forwarding_progress 0.94 0.93 0.51 0.52
moq-stats produce::tests::reheld_wire_presence_refreshes_two_relay_caches 1.70 0.90 0.86 0.87
moq-mux container::ts::recovery_test::a_skip_across_a_resolution_change_takes_the_new_parameters 0.84 0.89 0.75 0.81
moq-json window::properties::window_codec_property_matches_every_readers_logical_window 0.89 0.89 0.57 0.71
moq-mux container::ts::recovery_test::a_lost_discontinuity_marker_still_breaks_the_clock 0.84 0.88 0.75 0.79
moq-json snapshot::encoder::test::snapshot_codec_property_reconstructs_each_json_value 0.85 0.87 0.49 0.57
moq-audio resample::tests::duration_and_samples_are_invariant_across_partitions_and_epochs 0.87 0.87 0.14 0.13
moq-mux container::ts::recovery_test::a_declared_discontinuity_survives_loss_either_side 0.84 0.85 0.77 0.80
moq-net::integration catalog_churn::catalog_churn_lite_06 0.83 0.83 0.80 0.83
moq-mux container::ts::export_timing_test::frames_held_through_a_silence_drop_as_late 0.78 0.82 0.65 0.70
moq-quic tests::bbr_marking_versus_dropping 0.73 0.78 0.28 0.30
moq-quic tests::probe::borderline_skips_probe_rtt 0.68 0.74 0.30 0.31
moq-net::integration mesh_withdraw::partial_mesh_withdraw_then_republish 0.84 0.73 0.76 0.79
moq-net ietf::publisher::group_priority_test::a_busy_track_yields_to_its_siblings 0.73 0.72 0.15 0.16
moq-json stream::test::an_oversized_record_is_refused_and_the_log_continues 0.64 0.69 0.00 0.00
moq-mux container::ts::export_timing_test::a_skip_at_the_join_keeps_one_clock 0.87 0.69 0.48 0.82
moq-mux container::ts::export_timing_test::a_skip_while_running_keeps_one_clock 0.66 0.68 0.48 0.77
moq-mux jitter::tests::a_wandering_track_sent_latest_does_not_move_the_rate 1.14 0.67 0.63 0.63
moq-audio decode::consumer::tests::the_decode_path_conforms_to_the_corpus 0.62 0.67 0.56 0.83
moq-quic tests::probe::rises_to_target 0.64 0.62 0.27 0.29
moq-mux jitter::tests::closing_a_late_start_is_not_charged_to_the_rate 0.94 0.61 0.39 0.39
moq-mux container::ts::recovery_test::a_skipped_group_resumes_at_the_next 0.59 0.60 0.51 0.51
moq-mux container::ts::recovery_test::a_skip_across_the_wrap_keeps_the_clock 0.56 0.60 0.51 0.52
moq-mux container::ts::recovery_test::a_cut_group_keeps_what_it_delivered 0.60 0.59 0.49 0.52
moq-quic tests::sim::priority_on_a_clean_path 0.62 0.59 0.45 0.50
moq-mux container::ts::recovery_test::a_stalled_track_leaves_the_others_running 0.52 0.57 0.48 0.48
moq-net::integration mesh_withdraw::full_mesh_withdraw_retracts_once_lite04 0.60 0.56 0.55 0.58
moq-mux container::ts::recovery_test::skewed_audio_survives_a_video_skip 0.57 0.56 0.52 0.54
moq-mux container::ts::export_timing_test::a_track_sent_later_than_the_delay_loses_nothing 0.51 0.56 0.40 0.56
moq-mux container::ts::recovery_test::a_stall_drops_what_it_made_late 0.53 0.55 0.49 0.52
moq-mux container::ts::recovery_test::a_clean_path_loses_nothing 0.56 0.54 0.51 0.51
moq-mux container::ts::export_timing_test::a_cut_judges_held_video_when_the_loss_is_observed 0.89 0.54 0.39 0.63
moq-mux container::ts::recovery_test::a_congested_path_sheds_what_it_made_late 0.55 0.54 0.49 0.51
moq-net::integration mesh_withdraw::full_mesh_withdraw_retracts_once_lite06 0.61 0.53 0.52 0.58
moq-tokio::integration broadcast::zero_initial_backoff_still_gives_up_on_a_flapping_peer 0.48 0.53 0.13 0.12
moq-mux jitter::tests::a_late_start_within_the_drift_limit_never_fails 0.91 0.52 0.39 0.40
moq-mux container::ts::recovery_test::open_gop_trims_only_the_orphans 0.51 0.52 0.49 0.51
moq-json stream::test::a_compressed_log_is_charged_its_inflated_size 0.53 0.51 0.00 0.00
moq-mux container::ts::recovery_test::a_path_stall_drops_what_it_made_late 0.52 0.50 0.46 0.48
moq-mux container::ts::export_timing_test::the_video_held_at_the_end_goes_out 0.46 0.49 0.39 0.45
moq-mux container::ts::recovery_test::a_late_catalog_starts_on_a_whole_group 0.50 0.49 0.43 0.43
moq-mux container::ts::recovery_test::a_join_starts_on_a_whole_group 0.51 0.49 0.45 0.47
moq-net lite::publisher::tests::a_busy_subscription_yields_to_its_siblings 0.46 0.48 0.11 0.12
moq-quic tests::probe::overflow_keeps_probe_rtt 0.45 0.48 0.21 0.33
moq-net::integration fetch_fanout::sequential_viewers_share_the_relays_fetched_groups 0.49 0.48 0.46 0.47
moq-mux container::ts::recovery_test::the_graders_fail_what_they_should 0.42 0.45 0.43 0.44
moq-mux jitter::tests::a_spell_on_the_steady_track_keeps_the_rate_on_it 0.77 0.45 0.45 0.44
moq-net::integration mesh_withdraw::partial_mesh_withdraw_retracts_once 0.55 0.45 0.44 0.46
moq-tokio::integration receive_credit::bufferbloat_is_not_a_stuck_window 0.56 0.45 0.56 0.41
moq-mux container::ts::export_timing_test::a_successor_settles_held_video_across_a_skip_or_restart 0.41 0.45 0.35 0.52
moq-cli::bin/moq publish::tests::ts_passthrough_crosses_a_relay_through_a_flagged_jump 0.49 0.44 0.38 0.40
moq-mux container::ts::passthrough_test::a_long_gop_splits_at_the_byte_ceiling_between_objects 0.45 0.43 0.15 0.15
moq-mux container::ts::export_timing_test::video_held_for_most_of_the_delay_drops_nothing 0.39 0.42 0.27 0.48
moq-quic tests::bbr_burst_after_long_idle_keeps_learned_bandwidth 0.40 0.41 0.17 0.19
moq-mux container::ts::export_timing_test::live_audio_shares_its_pes 0.37 0.40 0.31 0.45
moq-net::integration routing_contracts::handoff::sweep_scoped_handoff 0.36 0.38 0.29 0.31
moq-mux jitter::tests::a_step_in_queueing_does_not_move_the_estimate 0.46 0.38 0.21 0.21
moq-tokio::integration broadcast::broadcast_rejoin_replays_a_current_warm_cache 1.05 0.38 0.36 0.38
moq-json snapshot::encoder::test::memo_matches_the_value_diff 0.37 0.37 0.36 0.40
moq-json snapshot::test::a_rejected_update_leaves_the_previous_value_readable 0.35 0.35 0.34 0.38
moq-cli::bin/moq hls::test::an_ended_playlist_announces_its_catalog 0.52 0.34 0.09 0.09
moq-relay::integration cluster_unknown::every_ingest_version_crosses_a_redundant_mesh 0.54 0.32 0.19 0.19
moq-tokio::integration receive_credit::lite_frame_of_many_windows_completes 0.57 0.32 0.36 0.31
moq-mux jitter::tests::start_up_latency_never_fails_a_full_multiplex 0.62 0.31 0.30 0.30
moq-mux jitter::tests::a_source_past_the_drift_limit_fails 0.38 0.29 0.16 0.17
moq-stats produce::tests::steady_drain_collects_without_allocating 0.44 0.29 0.28 0.28
moq-mux container::ts::export_test::scte35_fixtures_survive_roundtrip 0.37 0.25 0.26 0.43
moq-net model::track::test::an_empty_successor_still_lets_the_gaps_go 1.83 0.25 0.25 0.26
moq-net model::track::test::an_endless_untimed_track_with_gaps_stays_bounded 1.82 0.24 0.23 0.24
moq-hls export::tests::a_fresh_join_reads_a_bounded_number_of_records 0.67 0.21 0.21 0.20
moq-net::integration compress::a_start_group_arrives_whole 0.35 0.20 0.20 0.19
moq-net model::resume::test::a_resumed_reader_forgets_gaps_past_its_budget 1.46 0.12 0.12 0.18
moq-net model::track::test::an_endless_track_with_gaps_stays_bounded 7.40 0.12 0.10 0.10
moq-mux container::ts::import::test::a_signalled_reset_reaches_the_exported_clock 6.46 0.11 0.11 0.16
moq-mux container::ts::import::test::a_shared_forward_boundary_resets_once_per_rendition 6.52 0.07 0.08 0.08
moq-net model::track::test::an_untimed_sparse_feed_keeps_its_gaps 3.74 0.06 0.06 0.06
Total 137.83 100.55 61.02 67.49

Merge history

  • 36858e5 merges main 4b8e37b onto the land fixer's 31df580. Conflicts: main's moq_clock to kio::clock rename (moq-dev/moq#7027) in export_timing_test.rs (kept this PR's Deadlines harness, renamed) and in quest/m0/ci/slow-tests.md (kept this PR's deletion), and main's subscribe_at in moq-quic tests/util.rs (see Changes). five-minutes.md no longer waits on slow-tests.
  • 38896af fixes the drain hang above; a merge of main 3164bce had no conflicts, and cf952d0 updates the Import::new doc for it.

Review

  • Reviewer: Codex (quest review), gpt-6.1-sol, medium effort, session 01a19828.
  • Round 1 on 36858e5: no findings. Round 2 on 38896af: no findings. Round 3 on cf952d0: no findings.
  • Reviewed head: cf952d0.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 13 commits October 27, 2026 18:49
Quest-Claim: 6301895be7d657b9
Each case advances a `moq_clock::Deadlines` queue instead of the tokio paused
clock, and a receiver reading after every frame now reads only once its export
was woken, as the task polling it would. A read nothing woke never returned a
slot (41k of 60k reads in a five-minute run); every case renders the same
packets as before. Arrivals now land on their exact instant rather than tokio's
millisecond. Pair comparisons check a packet at a time before naming the byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With arrivals on the source's exact clock, the export's rate settles within
0.002 ppm of its five-minute value by 10 s at 30 and 400 ppm, far inside the
0.5 and 1 ppm the cases assert. Neither the rate window nor the pull-back
(600 s each) fills in five minutes either, so a minute runs the same steering.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The routing_contracts sweeps spend nearly all their time in the reference
model's per-message recompute. Pass the selected route into live/adopted
instead of selecting it again up to seven times per identity, restrict a
view once per peer interest instead of cloning it per peer, keep links in a
dense matrix and their grants in a set, and skip work whose result is
unused. Every sweep covers the same scenarios with identical model state,
events, and stats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A depth only chooses between tracks, so a lone track's steering step no
longer sorts its window for it. The drift and feed test helpers drain due
frames without collecting them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- recovery: x265 runs single-threaded (it sizes its pools by host cores and
  ignores -threads), FFmpeg hashes decoded pictures with murmur3 instead of
  MD5, and each run concatenates its output once.
- passthrough: the TS fixture builds packets in a fixed array and the
  readback concatenates payloads by slice instead of byte by byte.
- rate: the random-walk model uses proptest's XorShift RNG, since each op
  forks the runner's RNG and ChaCha dominated the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- moq-json: the window and stream encoders take their group budget from a
  field, as the snapshot encoder already does, so the oversize tests fill a
  32 KiB budget instead of serializing 32 MiB in a debug build.
- moq-json: the codec property tests draw cases with XorShift; seeding a
  ChaCha generator per union arm dominated case generation.
- moq-net: the front invariant walk renders the front only when an event
  that emitted actions leaves every cheap field unchanged.
- moq-net: the busy-track yield tests serve 2,000 groups instead of 10,000;
  every poll still faces a backlog past its budget.
- moq-pattern: the exhaustive intersect test matches each unordered pair once,
  since the commute assertion makes the mirrored pair the same union.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sim harness built a fresh 64 KiB payload byte by byte on every send
attempt, even a blocked one, and again to check each received chunk. Slice
one shared prime-length cycle instead, rotated per stream so streams still
differ at every offset.

`subscribe()` formatted every per-packet trace event at TRACE, which cost
more than the simulation itself. It now honors `RUST_LOG`, defaulting to
info so the tests' own events (the probe reports) still print, and the
per-step side spans are debug since they only label the stack's events.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- qmux: the keep-alive test builds its 100 datagrams once instead of
  200,000 times byte by byte.
- moq-audio: the corpus replay decodes 8 kHz PCM; only a packet's
  duration feeds the frame term, so the samples per packet don't matter.
- moq-video: the transcode color test uses narrow frames; only the height
  picks the guessed space, and the heights are unchanged.
- moq-cli: the passthrough test concatenates payload slices instead of
  collecting bytes one at a time.
- moq-relay: the mesh wait reuses one HTTP client instead of loading the
  system trust roots on every 10ms poll.
- moq-tokio: the zero-backoff test gives up after 100ms of redials rather
  than 500ms; a window that resets never runs out, whatever its length.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every fetch stays on the entry's origin, so a `file` playlist never makes
an HTTP request, yet each import built a reqwest client, which loads and
parses the system trust roots. Build it only for an http(s) entry; the
fetch takes the file path when there is none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its SIMD intrinsics compile to out-of-line calls at opt-level 0, as sonora's
did, and they are the whole cost of the resampler's invariant test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The frozen cohort measures 61.0 and 67.5 CPU-seconds in two full runs, under
its 68.92 s target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 18 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5cdec293-fc36-417b-9cf6-01e5187ab851
Commits

Reviewing files that changed from the base of the PR and between 3164bce and cf952d0.

Files selected for processing: 33
  • Cargo.toml
  • quest/m0/ci/README.md
  • quest/m0/ci/five-minutes.md
  • quest/m0/ci/slow-tests.md
  • rs/moq-audio/src/decode/consumer.rs
  • rs/moq-cli/src/publish.rs
  • rs/moq-hls/src/import.rs
  • rs/moq-json/src/snapshot/encoder.rs
  • rs/moq-json/src/stream/encoder.rs
  • rs/moq-json/src/stream/mod.rs
  • rs/moq-json/src/stream/producer.rs
  • rs/moq-json/src/window/encoder.rs
  • rs/moq-json/src/window/properties.rs
  • rs/moq-mux/src/container/ts/export_timing_test.rs
  • rs/moq-mux/src/container/ts/fixture.rs
  • rs/moq-mux/src/container/ts/passthrough_test.rs
  • rs/moq-mux/src/container/ts/recovery_test.rs
  • rs/moq-mux/src/jitter.rs
  • rs/moq-mux/src/rate.rs
  • rs/moq-net/src/ietf/publisher.rs
  • rs/moq-net/src/lite/publisher.rs
  • rs/moq-net/src/model/front.rs
  • rs/moq-net/tests/integration/routing_contracts/handoff.rs
  • rs/moq-net/tests/integration/routing_contracts/model.rs
  • rs/moq-pattern/tests/integration/pattern_exhaustive.rs
  • rs/moq-quic/src/tests/probe.rs
  • rs/moq-quic/src/tests/sim.rs
  • rs/moq-quic/src/tests/sim/workloads.rs
  • rs/moq-quic/src/tests/util.rs
  • rs/moq-relay/tests/integration/cluster_unknown.rs
  • rs/moq-tokio/tests/integration/broadcast.rs
  • rs/moq-video/src/encode/encoder.rs
  • rs/qmux/src/session/tests.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T15:26:51.904467Z 78a9aba New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 12a66db0e24c52adcdda05fc63c6da7a3a011598.

No actionable bugs found. The production changes remain narrow: file-only HLS imports still enforce same-origin checks before fetching, JSON group budgets retain their production defaults across resets, and the single-track jitter shortcut skips only the depth comparison used to choose between tracks. The routing-model and deadline-driven test-harness changes also preserve the relevant ordering and checks on static inspection.

Direction: removing repeated allocation, logging, hashing and setup work is appropriate here. The documented shorter drift runs, smaller busy-stream workloads and different deterministic property-test samples are explicit coverage tradeoffs, not proof of identical coverage; keep the existing bounds and regression checks as these fixtures evolve.

Verification: static review of the changed code and relevant surroundings, including an independent routing-model/export-timing/fixture pass. No tests or benchmarks run; the reported CPU improvements, full-suite results and before/after output comparisons were not independently reproduced.

# Conflicts:
#	rs/moq-json/src/stream/encoder.rs
#	rs/moq-json/src/stream/mod.rs
#	rs/moq-json/src/window/encoder.rs
#	rs/moq-mux/src/container/ts/recovery_test.rs
#	rs/moq-quic/src/tests/util.rs
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merged main into the branch (55fe56c) and resolved the conflicts by hand:

  • moq-json (stream/encoder.rs, stream/mod.rs, window/encoder.rs): main dropped per-record compression (moq-dev/moq#7046). I kept main's version and re-applied this PR's configurable max_group_bytes on top, with the tests' 32 KiB BUDGET in place of MAX_CACHE_BYTES. The window header test lost its per-compression loop, as it did on main.
  • moq-mux recovery_test.rs: I kept main's decoder()/parse() split from moq-dev/moq#7074 and moved this PR's framehash -hash murmur3 swap into it.
  • moq-mux export_timing_test.rs: main's new an_unmarked_step_keeps_the_export_rendering called Live::run with .await, but this PR made run synchronous, so I removed the .await.
  • moq-quic tests/util.rs: I kept main's client::Config signature with this PR's debug_span.

The full just check origin/main run passes, 10514 tests. On the first run, one moq-rtc test failed with a DTLS handshake timeout under load. It passed when run alone and on the rerun.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 30, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated removed this pull request from the merge queue due to a manual request Oct 30, 2026
@moq-bot

moq-bot Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

quest land removed this pull request from the queue: Conflicts with the base: quest/m0/ci/slow-tests.md, rs/moq-mux/src/container/ts/export_timing_test.rs, rs/moq-quic/src/tests/util.rs
.

@moq-bot

moq-bot Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

codex / Codex land fixer: This maintainer pick already used its mechanical fi·x attempt. Run /quest-iterate to diagnose the repeated failure, then pick it again.

(written by codex / Codex)

@kixelated
kixelated enabled auto-merge (squash) November 1, 2026 12:17
@moq-bot
moq-bot Bot disabled auto-merge November 1, 2026 16:37
@moq-bot moq-bot Bot removed the quest:merge label Nov 1, 2026
@moq-bot

moq-bot Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

quest land removed this pull request from the queue: Conflicts with the base: quest/m0/ci/slow-tests.md, rs/moq-mux/src/container/ts/export_timing_test.rs, rs/moq-quic/src/tests/util.rs
.

@moq-bot

moq-bot Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Codex land fixer: fixer crashed (exit 1); run /quest-iterate. Recommendation: run /quest-iterate before picking it again.

quest land check: [Errno 2] No such file or directory: '/var/lib/quest-land-moq/moq-dev-moq/fix-qrkoPz/.git/objects/maintenance.lock'

(written by Codex)

moq-bot Bot pushed a commit to moq-dev/quest that referenced this pull request Nov 1, 2026
A fixer job crashed with ENOENT on a vanished objects/maintenance.lock
(moq-dev/moq#6987): background maintenance removed the lock between
importFixer's readdir and lstat. Fixer checkouts now turn off gc.auto and
maintenance.auto, and the import skips git lock files, which it never
imports.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: moq-bot <moq-bot@users.noreply.github.com>
@kixelated
kixelated enabled auto-merge (squash) November 1, 2026 17:10
@moq-bot
moq-bot Bot disabled auto-merge November 1, 2026 17:19
@moq-bot moq-bot Bot removed the quest:merge label Nov 1, 2026
@moq-bot

moq-bot Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

quest land removed this pull request from the queue: Conflicts with the base: quest/m0/ci/slow-tests.md, rs/moq-mux/src/container/ts/export_timing_test.rs, rs/moq-quic/src/tests/util.rs
.

- quest/m0/ci/slow-tests.md: keep this PR's deletion over main's kio rename.
- export_timing_test: keep the Deadlines-driven harness, renamed to kio::clock.
- moq-quic tests: keep the RUST_LOG-driven subscribe() at info by default and
  drop main's subscribe_at, whose only caller now gets info from subscribe().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@moq-bot

moq-bot Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

codex / Codex land fixer: Base merge conflicts can be re·solved mechanically, but the merged candidate cannot pass tests: &#96;timeout --verbose --kill-after=5s 45s cargo test --locked -p moq-audio -p moq-cli -p moq-hls -p moq-json. Recommendation: Review the export timing fixture drain/finish path with the base kio::clock migration; re·solve the runtime stall with coverage preserved, then rerun all twelve affected packages. Mechanical merge atte

		}
	}

	/// Run the clock to `at`, letting every leg read whenever it would have on its own.
	fn advance(&self, at: Instant, legs: &mut [&mut Leg]) {
		while let Some(wake) = legs
			.iter()
			.filter_map(|leg| leg.wake())
rg: rs/moq-mux/src/container/hang/audio.rs: IO error for operation on rs/moq-mux/src/container/hang/audio.rs: No such file or directory (os error 2)
   Compiling moq-msf v0.5.1 (/work/rs/moq-msf)
   Compiling ciborium-ll v0.2.2
   Compiling webm-iterable v0.7.1
   Compiling ciborium v0.2.2
   Compiling criterion v0.8.2
   Compiling idna_adapter v1.2.2
   Compiling idna v1.1.0
   Compiling url v2.5.8

exec
/bin/bash -lc 'timeout --verbose --kill-after=5s 45s cargo test --locked -p moq-audio -p moq-cli -p moq-hls -p moq-json -p moq-mux -p moq-net -p moq-pattern -p moq-quic -p moq-relay -p moq-tokio -p moq-video -p qmux@0.7.0 container::ts::export_timing_test::the_video_held_at_the_end_goes_out -- --exact --nocapture > /tmp/quest-6987-hang.log 2>&1' in /work
 exited 124 in 30023ms:
exec
/bin/bash -lc 'tail -12 /tmp/quest-6987-isolated.log; tail -12 /tmp/quest-6987-hang.log; ps -eo pid,ppid,comm,etime,pcpu --sort=-pcpu | head -7' in /work
 succeeded in 0ms:
   Compiling idna_adapter v1.2.2
   Compiling idna v1.1.0
   Compiling url v2.5.8
   Compiling bench-harness v0.0.1 (/work/rs/bench-harness)
   Compiling kio v0.6.2 (/work/rs/kio)
   Compiling mp4-atom v0.17.0
   Compiling moq-net v0.3.10 (/work/rs/moq-net)
   Compiling moq-json v0.5.8 (/work/rs/moq-json)
   Compiling moq-loc v0.2.19 (/work/rs/moq-loc)
   Compiling hang v0.21.11 (/work/rs/hang)
   Compiling moq-mux v0.10.11 (/work/rs/moq-mux)
timeout: sending signal TERM to command ‘cargo’

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s

     Running unittests examples/teleop.rs (target/debug/examples/teleop-bb6260ca7764be9c)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/moq_mux-2113f186acb243cc)

running 1 test
    PID    PPID COMMAND             ELAPSED %CPU
  37403   37376 moq_mux-2113f18       00:02  3.1
  37376   37375 cargo                 00:35  1.7
     40      21 python3               08:39  0.5
     83      71 codex                 08:34  0.5
     71      40 quest                 08:34  0.0
     20       1 nsncd                 08:39  0.0

 exited 124 in 44882ms:
exec
/bin/bash -lc "tail -8 /tmp/quest-6987-hang.log; ps -eo pid,ppid,comm | rg 'cargo|moq_mux'" in /work
 exited 1 in 0ms:
running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/moq_mux-2113f186acb243cc)

running 1 test
timeout: sending signal TERM to command ‘cargo’

codex
The filtered rerun reached `the_video_held_at_the_end_goes_out` and timed out with exit 124. I’ll hand PR 6987 back for review of the exporter-drain test fixture, restore its pinned branch, and record the failing command in the result file.

exec
/bin/bash -lc "git merge --abort
python3 - <<'PY'
import json
from pathlib import Path
cmd='timeout --verbose --kill-after=5s 45s cargo test --locked -p moq-audio -p moq-cli -p moq-hls -p moq-json -p moq-mux -p moq-net -p moq-pattern -p moq-quic -p moq-relay -p moq-tokio -p moq-video -p qmux@0.7.0 container::ts::export_timing_test::the_video_held_at_the_end_goes_out -- --exact --nocapture'
result=[{'number':6987,'action':'handback','diagnosis':f'Base merge conflicts can be resolved mechanically, but the merged candidate cannot pass tests: "'`{cmd}` reaches `running 1 test` and exits 124 with "timeout: sending signal TERM to command ‘cargo’"; the full package test run also reports `test container::ts::export_timing_test::the_video_held_at_the_end_goes_out has been running for over 60 seconds` (four other export timing tests also stall).'"','recommendation':'Review the export timing fixture drain/finish path with the base kio::clock migration; resolve the runtime stall with coverage preserved, then rerun all twelve affected packages. Mechanical merge attempt was aborted; pr-6987 remains at its pinned head.'}]
Path('quest-land-result.json').write_text(json.dumps(result,indent=2)+'\\n')
PY
 git status --short
 git rev-parse pr-6987 batch base" in /work
 succeeded in 2595ms:
Updating files:  47% (623/1321)
Updating files:  48% (635/1321)
Updating files:  49% (648/1321)
Updating files:  50% (661/1321)
Updating files:  51% (674/1321)
Updating files:  52% (687/1321)
Updating files:  53% (701/1321)
Updating files:  54% (714/1321)
Updating files:  55% (727/1321)
Updating files:  56% (740/1321)
Updating files:  57% (753/1321)
Updating files:  58% (767/1321)
Updating files:  59% (780/1321)
Updating files:  60% (793/1321)
Updating files:  61% (806/1321)
Updating files:  62% (820/1321)
Updating files:  63% (833/1321)
Updating files:  64% (846/1321)
Updating files:  65% (859/1321)
Updating files:  66% (872/1321)
Updating files:  67% (886/1321)
Updating files:  68% (899/1321)
Updating files:  69% (912/1321)
Updating files:  70% (925/1321)
Updating files:  71% (938/1321)
Updating files:  72% (952/1321)
Updating files:  73% (965/1321)
Updating files:  74% (978/1321)
Updating files:  75% (991/1321)
Updating files:  76% (1004/1321)
Updating files:  77% (1018/1321)
Updating files:  78% (1031/1321)
Updating files:  79% (1044/1321)
Updating files:  80% (1057/1321)
Updating files:  81% (1071/1321)
Updating files:  82% (1084/1321)
Updating files:  83% (1097/1321)
Updating files:  84% (1110/1321)
Updating files:  85% (1123/1321)
Updating files:  86% (1137/1321)
Updating files:  87% (1150/1321)
Updating files:  88% (1163/1321)
Updating files:  89% (1176/1321)
Updating files:  90% (1189/1321)
Updating files:  91% (1203/1321)
Updating files:  92% (1216/1321)
Updating files:  93% (1229/1321)
Updating files:  94% (1242/1321)
Updating files:  95% (1255/1321)
Updating files:  96% (1269/1321)
Updating files:  97% (1282/1321)
Updating files:  98% (1295/1321)
Updating files:  99% (1308/1321)
Updating files: 100% (1321/1321)
Updating files: 100% (1321/1321), done.
?? quest-land-result.json
31df5805b61560b12c84f2649269427ba2d1b4cd
31df5805b61560b12c84f2649269427ba2d1b4cd
4b8e37bd44e407b3416a17992720870ea3f9221c

codex
PR 6987 handed back: the merged candidate builds, but five export timing tests stall; a filtered rerun timed out.

Branch restored unchanged. Diagnosis and command recorded in uncommitted [quest-land-result.json](/work/quest-land-result.json).
tokens used
113,682
PR 6987 handed back: the merged candidate builds, but five export timing tests stall; a filtered rerun timed out.

Branch restored unchanged. Diagnosis and command recorded in uncommitted [quest-land-result.json](/work/quest-land-result.json).
fixer result written

(written by codex / Codex)

kixelated and others added 4 commits November 1, 2026 09:39
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kio::clock::timeout arms its sleep on the entered Deadlines queue, which only
the harness advances, so a drain whose export never reported the end waited
forever. Advance the queue 10 ms instead, as the timeout meant to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: cf952d0c25f0baff1feb51251eb8dabe27bdd629.

No new actionable bugs found in the update since review #5700676453, accounting for the merged base changes. The export-timing fixture now drives its own deadline queue instead of awaiting a timeout on that unadvanced queue. Its final-video count and paired-output assertions remain intact. This addresses the fixture stall described in the land-fixer report; it was not a finding in my previous review.

Direction: keeping the drain synchronous on the harness-owned clock is simpler and appropriate. The merged JSON per-record limit, QUIC logging/buffer adaptations, and HLS file-only path retain the intended narrow optimization scope; no additional public API or wire change is introduced by these updates.

Verification: static inspection of the incremental changes and relevant exporter, clock, and HLS code only. No tests or benchmarks run, and the reported gate results and CPU savings were not independently reproduced.

(Written by OpenAI)

@kixelated
kixelated enabled auto-merge (squash) November 1, 2026 18:12
kixelated added a commit that referenced this pull request Nov 1, 2026
All four cuts from #6987 have landed or land with this PR, so remove the
quest and its link.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant