Skip to content

Performance improvement for constant integer stats #10029

Description

@tpoterba

Problem or motivation

for &value in values {
if count_distinct_values {
*state.distinct_values.entry(NativeValue(value)).or_insert(0) += 1;
}

I have a dumb table with 5 u32/u8 fields where 4 are almost always const within a chunk but Vortex still hashes them, and this is a serious hotspot.

Proposed solution

Proposed fixes:

  1. reorder the control flow for the compressor to check min/max first to identify const before entering this loop
  2. replace with a run-aware version, something like this
  3. the best solution to my bespoke problem would be to allow passing in a const array and specializing statistics for that.
#[inline(always)]
fn inner_loop_nonnull<T: IntegerPType>(
    values: &[T; 64],
    count_distinct_values: bool,
    state: &mut LoopState<T>,
) where
    NativeValue<T>: Eq + Hash,
{
    let mut pending_count = 0u32;

    for &value in values {
        if value != state.prev {
            if count_distinct_values && pending_count != 0 {
                *state
                    .distinct_values
                    .entry(NativeValue(state.prev))
                    .or_insert(0) += pending_count;
            }

            state.prev = value;
            state.runs += 1;
            pending_count = 0;
        }

        pending_count += 1;
    }

    if count_distinct_values {
        *state
            .distinct_values
            .entry(NativeValue(state.prev))
            .or_insert(0) += pending_count;
    }
}

Additional context

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureA feature request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions