Skip to content

Sketch rollup: a regrouping SummaryMerge so mergeable sketches can roll up to coarser keys #719

Open

Description

@zzylol

Problem, by example

A dashboard has two queries over the same latency metric, one per instance and one per job:

# Q_fine: p99 per (job, instance)
quantile by (job, instance) (0.99, latency_ms)
# Q_coarse: p99 per job
quantile by (job) (0.99, latency_ms)

Say there are 10 jobs with 100 instances each. Stage 1 can answer both with KLL sketches:

Sketches built Rows each sketch reads
Q_fine 1,000 KLLs, one per (job, instance) that instance's samples
Q_coarse 10 KLLs, one per job that job's samples, i.e. the same rows again

Every sample is read and inserted twice. KLL states merge (the family-capability table marks KLL SameAsUnionBuild). So Q_coarse could instead be answered by merging Q_fine's 100 per-instance KLLs of each job into one KLL per job, with no second pass over the raw samples:

today:                                   with sketch rollup:
  Scan ─▶ SummaryAgg(KLL by job,instance)   Scan ─▶ SummaryAgg(KLL by job,instance) ─▶ Q_fine
  Scan ─▶ SummaryAgg(KLL by job)                        └─▶ regroup-merge(by job) ─▶ Q_coarse

Why it isn't done today

Proposal

  1. IR: a regrouping form of SummaryMerge, or a dedicated operator. Its input is states grouped by keys K, its output is states grouped by K' ⊂ K, and it merges every input group whose projection onto K' matches.
  2. Coverage: the output group covers the union of its input groups' rows. Those rows are disjoint, because the finer groups partition the rows.
  3. Stage 2 / Stage 3: timing and materialization like any summary merge. Pricing is one merge per input state, plus the output states' memory.
  4. Executor: reuse each family's existing merge kernel and add a group-by-key step.
  5. Guarantee: taken from the family-capability merged-error fact. KLL, DDSketch, CMS and CountSketch keep SameAsUnionBuild; HLL and KMV allow overlap; heap top-k and UnivMon heap readouts are approximate, so they get no guarantee.
  6. Pass 2: extend the rollup rule from exact accumulators to these sketches.

Why it's deferred

Decided on 2026-10-15. Reuse rarely wins until Stage 2 materialization (PR #717) lets the shared finer sketches be kept and charged once.

Related: #680, #716, #696, #717.

Activity

  1. added
    enhancementNew feature or request
    rollupFine-to-coarse aggregation and summary reuse
    on Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestrollupFine-to-coarse aggregation and summary reuse

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions