Skip to content

Refine DecimalScheme to the v2 wire format when the allowed serialized IDs permit it - #10076

Open
mhk197 wants to merge 1 commit into
developfrom
mk/decimal-refine
Open

mhk197 wants to merge 1 commit into
developfrom
mk/decimal-refine

Conversation

@mhk197

@mhk197 mhk197 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #10069.

Summary

Wide i128 and i256 decimals compress into multi-part DecimalByteParts arrays when the compressor's allowed serialized IDs include vortex.decimal_byte_parts.v2. The session registers the v1 decimal scheme, and build() refines it to v2 wherever both decimal IDs are allowed.

This PR sets the stage for versioned schemes.

Scheme::refine

New defaulted hook in vortex-compressor:

fn refine(&self, _allowed: &dyn Fn(&ArrayId) -> bool) -> Option<&'static dyn Scheme> {
    None
}
  • It returns the variant of the scheme to use for the allowed IDs; None keeps the scheme.
  • A variant must keep the scheme's SchemeId, and every ID it declares in produced_encodings must still be allowed for it to be used.
  • The allowed IDs are a predicate rather than a set, so builders that allow every ID (empty(), unrestricted()) are represented exactly.

Builder

build() now:

  1. drops the mode's excluded schemes,
  2. refines each remaining scheme against the allowed IDs,
  3. keeps the schemes whose produced IDs are all allowed.

The allowed IDs are the session's AllowedIds minus the mode's excluded serialized IDs. The Cuda mode excludes vortex.decimal_byte_parts.v2, since multi-part arrays have no CUDA decode kernel, so the CUDA preset keeps v1.

DecimalScheme

  • DecimalScheme gains v1 and v2 modes with v1() and v2() constructors; Default is v1. The session registers v1.
  • v1 leaves values wider than i64 canonical after narrowing. v2 splits them with split_decimal and compresses the most significant part and each lower part as children. Single-part arrays serialize as v1 in either mode.
  • refine returns v2 when both decimal IDs are allowed and v1 otherwise, from either mode. Unlike Upgrade DecimalScheme to the v2 wire format when the writer permits it #10035's try_upgrade, a v2 scheme is refined down to v1 rather than dropped.

Behaviour

Builder Wide decimals
from_session with the core edition canonical (v1), unchanged
from_session(..).disable_editions(), including the writer with disable_editions() split (v2)
unrestricted() / empty() with the v2 scheme split (v2)
only_cuda_compatible() canonical (v1)

Default writes are unchanged: the core edition allows v1 only. Strategies built from WriteStrategyBuilder::from_session keep the enabled editions' restrictions even when a write disables editions.

Compared with #10035, vortex-file's writer and vortex-cuda need no changes: the writer's disable_editions path already lifts the edition restriction, and the CUDA restriction lives in the Cuda mode.

Tests

  • vortex-btrblocks/src/schemes/decimal.rs: refine for every combination of allowed decimal IDs, from both modes.
  • vortex-btrblocks/tests/decimal_config.rs: decimal output for no editions, the core edition, editions disabled, unrestricted, and the CUDA preset; excluding the decimal scheme is not undone by refinement; signed/nullable i128/i256 round trips with and without child compression.
  • vortex-file: the default writer splits wide decimals only with editions disabled; strategies built from the session keep the enabled editions' restrictions.

Checks

  • cargo clippy -D warnings on vortex-btrblocks, vortex-compressor, vortex-file with --all-targets --all-features, --all-targets, and --no-default-features: passed
  • cargo nextest run --all-features -p vortex-btrblocks -p vortex-compressor -p vortex-file -p vortex -p vortex-layout -p string-bench -p vortex-compat -p vortex-decimal-byte-parts: 988 passed, 1 skipped
  • cargo test --doc -p vortex-btrblocks -p vortex-compressor: passed
  • RUSTDOCFLAGS="-D warnings" cargo doc -p vortex-btrblocks -p vortex-compressor --no-deps --all-features: passed
  • rustfmt +nightly-2026-09-10 --check on touched files: passed
  • Not run: vortex-cuda (needs nvcc), benchmarks, workspace-wide clippy and tests.

@codspeed

codspeed Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 2062 untouched benchmarks
⏩ 503 skipped benchmarks1


Comparing mk/decimal-refine (c8a8c3c) with develop (122ca21)2

Open in CodSpeed

Footnotes

  1. 503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on develop (eb91521) during the generation of this report, so 122ca21 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@mhk197
mhk197 added this pull request to stack #10070 September 25, 2026 20:37
@mhk197 mhk197 added the changelog/chore A trivial change label Sep 25, 2026
Base automatically changed from mk/compressor-scheme-modes to develop September 25, 2026 21:20
Signed-off-by: Matt Katz <mhkatz97@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant