Problem or motivation
|
for &value in values { |
|
if count_distinct_values { |
|
*state.distinct_values.entry(NativeValue(value)).or_insert(0) += 1; |
|
} |
I have a dumb table with 5 u32/u8 fields where 4 are almost always const within a chunk but Vortex still hashes them, and this is a serious hotspot.
Proposed solution
Proposed fixes:
- reorder the control flow for the compressor to check min/max first to identify const before entering this loop
- replace with a run-aware version, something like this
- the best solution to my bespoke problem would be to allow passing in a const array and specializing statistics for that.
#[inline(always)]
fn inner_loop_nonnull<T: IntegerPType>(
values: &[T; 64],
count_distinct_values: bool,
state: &mut LoopState<T>,
) where
NativeValue<T>: Eq + Hash,
{
let mut pending_count = 0u32;
for &value in values {
if value != state.prev {
if count_distinct_values && pending_count != 0 {
*state
.distinct_values
.entry(NativeValue(state.prev))
.or_insert(0) += pending_count;
}
state.prev = value;
state.runs += 1;
pending_count = 0;
}
pending_count += 1;
}
if count_distinct_values {
*state
.distinct_values
.entry(NativeValue(state.prev))
.or_insert(0) += pending_count;
}
}
Additional context
No response
Problem or motivation
vortex/vortex-compressor/src/stats/integer.rs
Lines 490 to 493 in beb7c15
I have a dumb table with 5 u32/u8 fields where 4 are almost always const within a chunk but Vortex still hashes them, and this is a serious hotspot.
Proposed solution
Proposed fixes:
Additional context
No response