Skip to content

Experiment: compose scheme registration with output editions - #10040

Closed
connortsui20 wants to merge 3 commits into
developfrom
ct/composable-schemes
Closed

connortsui20 wants to merge 3 commits into
developfrom
ct/composable-schemes

Conversation

@connortsui20

Copy link
Copy Markdown
Member

Summary

Makes scheme registration the writer's opt-in to compression. Encoding packages own their schemes, and compressor construction resolves those registrations against the permitted serialized IDs before starting the cascading engine. Registering an encoding alone still supports reading without enabling its compression scheme.

Changes

  • Move encoding-specific schemes into their owning packages. Keep vortex-btrblocks as a thin initialization and construction facade, with compatibility re-exports.
  • Replace the default scheme list and scheme builder with explicit, atomic session registration. Compact is an additive initialization helper. A compression-session fork isolates per-operation registrations.
  • Declare direct helper outputs as well as top-level encodings. Constants become registered schemes, and the default writer passes its exact serialization permissions to construction.
  • Add a complete draft CUDA edition and explicit edition-set replacement. CUDA writers select it instead of core. Enabling several editions still unions their permissions.

Scoped Clippy passed with warnings denied for the affected core and encoding crates. Compiler checks passed for the affected core crates, including tests, benches, examples, all-feature compression builds, and the facade without file support. Added regression coverage for registration, permissions, concurrency, and edition selection. Rust and TOML formatting passed. Tests were not executed. Snapshot review, CUDA execution, Wasm execution, and benchmarks remain unrun. The whole-workspace pre-push Clippy hook was skipped after scoped checks. Stateful training in #8434 remains separate.

API Changes

Removes BtrBlocksCompressorBuilder, with_compact, and only_cuda_compatible. Call package initialization functions, select editions, then construct BtrBlocksCompressor::from_session. The layout builder remains and accepts an already constructed compressor through with_btrblocks_compressor.

VortexSession::clone continues to share registrations. fork_compression copies the compression registry for local configuration while sharing other services. register_cuda_layout only registers support, while cuda_write_strategy selects the CUDA edition on the supplied session. The cascading engine retains sampling, estimates, exclusions, and recursive compression.

mhk197 and others added 3 commits September 24, 2026 16:34
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20 connortsui20 added the changelog/break A breaking API change label Sep 24, 2026
@connortsui20

Copy link
Copy Markdown
Member Author

this is bad

@connortsui20
connortsui20 deleted the ct/composable-schemes branch September 24, 2026 21:31
@codspeed

codspeed Bot commented Sep 24, 2026

Copy link
Copy Markdown

Merging this PR will regress 2 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 3 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 2 regressed benchmarks
✅ 2218 untouched benchmarks
⏩ 343 skipped benchmarks1
🗄️ 1 archived benchmark run2

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ WallTime decode_avx512[8192, (Inline, OneNullInEight)] 81.7 µs 99.2 µs -17.65%
❌ WallTime decode_avx512[8192, (Inline, AllValid)] 96.6 µs 112.5 µs -14.12%
⚡ Simulation from_vec_drop_arrow[16384] 40.4 µs 31.4 µs +28.7%
⚡ WallTime filtered_sink_i64_avx512[OneNullInEight] 26.3 µs 22.3 µs +18.26%
⚠️ Simulation fixed_16_advancing_ptr_safe[100] < 1 ns < 1 ns N/A
⚠️ Simulation preverify_advancing_ptr_unchecked[1000] < 1 ns < 1 ns N/A
⚠️ Simulation preverify_advancing_ptr_unchecked[10000] < 1 ns < 1 ns N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/composable-schemes (ffe45b2) with develop (035aeff)

Open in CodSpeed

Footnotes

  1. 343 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. 1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants