Skip to content

propogate allowed serialized ids to scheme and have scheme decide - #9991

Closed
mhk197 wants to merge 1 commit into
developfrom
mk/allowed-serialized-ids
Closed

mhk197 wants to merge 1 commit into
developfrom
mk/allowed-serialized-ids

Conversation

@mhk197

@mhk197 mhk197 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Summary

The compressor carries the writer's permitted serialized IDs and exposes them to schemes through CompressorContext. A scheme with an optional newer wire format asks the context before producing it. DecimalScheme uses this to split wide decimals into multi-part DecimalByteParts arrays when vortex.decimal_byte_parts.v2 is permitted, and leaves them canonical otherwise.

How it works

  • Scheme::produced_encodings keeps its meaning: the serialized IDs a scheme always needs. A scheme is dropped when any of them is not permitted.
  • CompressorContext::allows_serialized_id reports whether an optional format may be emitted. Estimate and compress receive the same context, so the decision is per array and consistent.
  • CascadingCompressor holds the allowlist. new permits exactly what the registered schemes declare, so optional formats are off until a writer permits them. with_allowed_serialized_ids replaces the set and drops schemes that need anything outside it.

Changes

  • vortex-compressor: CascadingCompressor::{with_allowed_serialized_ids, allowed_serialized_ids, has_scheme}, CompressorContext::allows_serialized_id, and the produced_encodings doc update.
  • BtrBlocksCompressorBuilder::retain_allowed_encodings keeps its name and callers. Repeated calls intersect, and it applies at build(), so schemes registered later are covered too. New exclude_encodings denies serialized IDs regardless of the allowlist. only_cuda_compatible uses it to deny DBP v2, so it can be called before or after retain_allowed_encodings.
  • DecimalScheme stays a unit struct declaring only v1. It splits via split_decimal and compresses the most significant part and each lower part as children, but only when the context permits v2.
  • An explicit BtrBlocksCompressorBuilder::default() strategy writes wide decimals canonically under the default editions, so it stays writable.

Tests

  • vortex-compressor: default allowlist, scheme filtering, and the context query seen from inside a scheme.
  • vortex-btrblocks: builder intersection, deferred filtering, exclude_encodings, CUDA preset ordering; decimal wire-format selection with round trips.
  • vortex: writer splits wide decimals only when a v2 edition is enabled or editions are disabled; an explicit default strategy keeps them canonical.

Validation

Not run locally. The pinned nightly for fmt is not installed here, so formatting was done by hand.

cargo +nightly-2026-09-10 fmt -p vortex-compressor -p vortex-btrblocks -p vortex
cargo clippy -p vortex-compressor -p vortex-btrblocks -p vortex --all-targets --all-features -- -D warnings
cargo nextest run -p vortex-compressor -p vortex-btrblocks
cargo nextest run -p vortex editions::

@mhk197
mhk197 force-pushed the mk/allowed-serialized-ids branch from 6f69fbf to 36e9657 Compare September 22, 2026 18:39
@codspeed

codspeed Bot commented Sep 22, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 0.08%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 2 regressed benchmarks
✅ 2205 untouched benchmarks
⏩ 293 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation take_fsl_nullable_random[16, 100] 164.3 µs 190.3 µs -13.67%
Simulation take_fsl_f16_random[256, 100] 203.7 µs 231 µs -11.83%
Simulation take_fsl_u32_random[256, 10] 161.6 µs 123.3 µs +31.04%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/allowed-serialized-ids (d0198af) with develop (133aacd)

Open in CodSpeed

Footnotes

  1. 293 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@mhk197
mhk197 force-pushed the mk/allowed-serialized-ids branch from 36e9657 to 1d7f2a5 Compare September 22, 2026 18:51
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197
mhk197 force-pushed the mk/allowed-serialized-ids branch from 1d7f2a5 to d0198af Compare September 22, 2026 18:51
@mhk197 mhk197 changed the title Pass allowed serialized IDs through the compressor context scheme decides mode online Sep 22, 2026
@connortsui20
connortsui20 self-requested a review September 22, 2026 18:59
@mhk197 mhk197 changed the title scheme decides mode online propogate allowed serialized ids to scheme and have scheme decide Sep 22, 2026
@mhk197 mhk197 closed this Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant