Skip to content

Upgrade DecimalScheme to the v2 wire format when the writer permits it - #10035

Closed
mhk197 wants to merge 2 commits into
mk/scheme-registryfrom
mk/decimal-modes
Closed

mhk197 wants to merge 2 commits into
mk/scheme-registryfrom
mk/decimal-modes

Conversation

@mhk197

@mhk197 mhk197 commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #10034.

Summary

Wide i128 and i256 decimals compress into multi-part DecimalByteParts arrays when the writer permits the vortex.decimal_byte_parts.v2 wire format. The registry holds the v1 scheme, which only ever writes the frozen v1 format; permission filtering swaps in the v2 variant wherever the permitted serialized IDs allow it.

Changes

  • vortex-compressor: Scheme::try_upgrade(&self, &HashSet<ArrayId>) -> Option<&'static dyn Scheme> is a new defaulted hook returning a newer variant the permitted IDs support; it must keep the SchemeId and never downgrade. permit_schemes(schemes, &allowed) (the function behind CompressionSessionExt::permit) calls it for each scheme before filtering by declared IDs, and is exported for callers with an explicit permission set.
  • vortex-btrblocks: DEFAULT_SCHEMES registers DecimalScheme::v1(). DecimalScheme gains v1 and v2 modes with v1() and v2() constructors; Default is v1. v1 leaves values wider than i64 canonical after narrowing. v2 splits them with split_decimal and compresses the most significant part and each lower part as children. Single-part arrays serialize as v1 in either mode. try_upgrade returns a static v2 instance when both decimal IDs are permitted; an explicitly listed v2 is dropped, not downgraded, when either ID is missing.
  • vortex-file: with editions disabled the writer now permits every registered serialized ID through permit_schemes, so the upgrade applies there too.
  • vortex-cuda: cuda_compatible_schemes pins the decimal scheme to v1 after permissions are applied, since multi-part arrays have no CUDA decode kernel.

Behavior

  • Default sessions enable the core edition, which permits v1 only, so default writes are unchanged.
  • Sessions that permit v2, or writes with editions disabled, split wide decimals.
  • Strategies built from WriteStrategyBuilder::from_session keep the enabled editions' permissions even when a write disables editions.

Tests

  • vortex-btrblocks/tests/decimal_config.rs: mode selection against the core edition and explicit ID sets, v2 on top of the core edition, explicit modes only upgrade, exclusion is not undone by an upgrade, and signed/nullable i128/i256 round trips with and without child compression.
  • vortex-file: the default writer splits wide decimals only with editions disabled; strategies built from the session keep the enabled editions' permissions.

Validation

Run locally: cargo +nightly-2026-09-10 fmt on the touched crates; cargo clippy --all-targets --all-features -- -D warnings on vortex-compressor, vortex-btrblocks, vortex-file, vortex (clean); cargo nextest run --all-features on vortex-compressor, vortex-btrblocks, vortex-file (296 passed, 1 skipped); doctests on the same three crates (passed). Not run: vortex-cuda, which needs nvcc.

@codspeed

codspeed Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 8.3%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 4 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 4 regressed benchmarks
✅ 2170 untouched benchmarks
⏩ 389 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation take_fsl_f16_force_per_index[256, 10] 38 µs 55.5 µs -31.54%
❌ WallTime filtered_sink_i64_avx2[OneNullInEight] 22.1 µs 26.1 µs -15.21%
❌ WallTime decode_avx512[8192, (Inline, OneNullInEight)] 80.5 µs 93.5 µs -13.87%
❌ Simulation set_indices_vortex_buffer[128] 1.8 µs 2.1 µs -10.73%
⚡ WallTime filtered_sink_i64_avx512[OneNullInEight] 26.3 µs 22.3 µs +17.94%
⚡ WallTime scalar_subtract_neon 13.4 µs 11.9 µs +12.98%
⚠️ Simulation fixed_16_advancing_ptr_safe[100] < 1 ns < 1 ns N/A
⚠️ Simulation preverify_advancing_ptr_unchecked[1000] < 1 ns < 1 ns N/A
⚠️ Simulation preverify_advancing_ptr_unchecked[10000] < 1 ns < 1 ns N/A
⚠️ Simulation bench_compare_sliced_dict_primitive[(3333, 10000)] < 1 ns < 1 ns N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/decimal-modes (9e757a2) with mk/scheme-registry (80af2c5)2

Open in CodSpeed

Footnotes

  1. 389 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on mk/scheme-registry (70b0bfe) during the generation of this report, so 62c3359 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant