Skip to content

feat(planner): whole-expression top-k heap sketch over raw samples - #589

Draft
zzylol wants to merge 2 commits into
stack/509-e1-pass2-identicalfrom
stack/509-e2-keyed-additive-topk
Draft

zzylol wants to merge 2 commits into
stack/509-e1-pass2-identicalfrom
stack/509-e2-keyed-additive-topk

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Rebased on main d4869a7 (DF 54). At this head cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace pass (1517 passed, 0 failed, 10 ignored).

Why

For Q2 topk by (job) (10, sum_over_time(http_requests_total[1m])), Pass 1 offered only a heap sketch over an exact per-series sum_over_time: 1M per-series sums feeding the sketch. The whole expression can instead be one CountSketch+heap (or CMS+heap) per job that reads the raw samples. The item is the series identity and the weight is the sample value. This is the legacy keyed-additive realization. It is also the tree DP's "outer choice drops the inner target" case, which had never been exercised. Stacked on #588. Refs #509, #580, #572.

What

  • Pass 1 (logical_candidates.rs). A top-k over a per-item sum/count whose inner aggregate has no other consumer gets whole-expression Count-Min and CountSketch + heap alternatives. LocalLogicalTarget::absorbs records the inner target each one absorbs. Applicability, weight and weight domain come from the existing realize_keyed_additive_summary_input. The item is the series-identity column when rows carry it, as in the existing top-k path.
  • Enumeration. An absorbed target has no choice: enumerate_choices skips its non-pass-through choices, combination_count counts the valid choices, and choice_index ranks in that order, so ids stay contiguous.
  • DP (plan-selection). The DP did not handle this unchanged. It added best(u) for every alternative of t, so an absorbing alternative would have been credited with the inner target's own saving. Now best(t,c) = local(t,c) + Σ best(u) runs only over read_targets(t,c). The pair guard skips pairs an alternative does not read, and the absorbed target is set to its pass-through.
  • Pricing fix the guard found. Top-k readout rows were min(rows the summary read, k × groups). A sketch over raw samples reads several rows per item, so the readout was sized differently from the exact path. On count(topk(…)) with 3 series this coupled with the outer count and forced exhaustive fallback. Readout rows are now bounded by the series count too.
  • Accuracy/validity unchanged. Count-Min still needs non-negative weights. Raw http_requests_total samples have no proof (UnitCount / ResetAwareCounterDerivative do not apply), so whole-expression CMS+heap is kept in Stage 1 and rejected by Stage 3 and by the runtime for the same reason. CountSketch admits signed weights. Its guarantee is the sketch's own analytical L2 bound on the whole expression.
  • Devtool label: "Q2 whole-expression CountSketch+heap".

How it was checked

  • Pass 1 unit test: 8 valid choices for this Q2, choice_index equals the enumeration position, and the absorbed sum_over_time is not in the composed DAG.
  • Plan-selection unit test: with real costs, and with a bonus that makes the whole-expression alternative cheapest, the DP equals brute force over every valid choice and sets the inner target to pass-through.
  • DP = exhaustive on Example 1 (64 combinations, P58) and on count(topk(…)) (40) at 3 and 1M series.
  • The runtime compiles every valid candidate, including the whole-expression CountSketch ones (stage2_runtime_compiles_exactly_the_candidates_stage3_finds_valid, stage2_count_sketch_heap_topk_compiles_in_the_physical_planner).
  • New q2_roots_keep_one_schema_across_realizations: every Q2 option has the selected-rows schema (job, series identity, value) in Stage 1.

Before this PR (Example 1, after #588)

1 → 48 → 48 → 1, with 16 invalid (Count-Min + heap). Selected P44 "Q1 exact (Sum acc, Rate acc) · Q2 exact (Sum acc) · shared input", 52.201. The only Q2 CountSketch plans sketch the 1M exact per-series sums; the best is 164.201.

After this PR (Example 1)

1 → 64 → 64 → 1 (32 per sharing variant; 24 invalid: all Count-Min + heap, including whole-expression). Selected P58, the same plan as before (renumbered), at 52.201. Stage 3 ranks Q2's options (shared input, Q1 with both accumulators):

Q2 realization Q2 nodes Total
exact: Sum acc (4.0 + 1.0) + sort 1M rows / 100 partitions (14.0) + limit (0.001) 19.001 52.201 (P58, selected)
CountSketch+heap over the exact per-series sums: Sum acc 5.0 + sketch 1M × 126 = 126.0 + estimate 0.001 131.001 164.201 (P62)
whole-expression CountSketch+heap: sketch 4M raw samples × 126 = 504.0 + estimate 0.001; no sum_over_time 504.001 537.201 (P64)

Shared nodes in all three: scan 23.2, range 4.0, Q1 Rate acc 5.0, Q1 Sum acc 1.0. Constants are not tuned. The sketch reads 4× more rows (4M samples vs 1M per-series sums) at depth 125 + 1 heap update per row, so exact wins.

CountSketch sizing (accuracy/estimators/count_sketch.rs): width ceil(3/ε²) = 30,000. Depth is the smallest odd integer ≥ 18·ln(1/δ) (median-of-rows Hoeffding bound exp(−depth/18) ≤ δ): 18·ln(1000) = 124.3 → 125. Stage 3 prices a heap sketch at depth + 1 = 126 ops per row.

Gate

cargo fmt --all --check ✓; cargo clippy --workspace --all-targets --all-features -- -D warnings ✓; cargo test --workspace: 1,517 passed / 10 ignored (#587: 1,508 / 12; #588: 1,514 / 10). The 3 new tests are the Pass 1 whole-expression test, the drop-inner DP test and the Q2 schema test. Viewer: python3 -m unittest discover -s tools/dag-viewer -p test_render.py, 29 OK (6 skipped).

Remaining gaps

  • Stage 2 still exports exact top-k as sort → limit over per-series rows, so the physical Q2 root schema of exact plans carries ts and differs from the sketch readout's (job, identity, value). This predates this PR; the Stage 1 schemas agree.
  • Hydra is still missing (stage1_q2_summary_families_are_heap_sketches_and_hydra stays ignored).

zzylol and others added 2 commits October 11, 2026 16:22
Pass 1 now offers, for a top-k over a per-item sum or count whose inner
aggregate has no other consumer, Count-Min and CountSketch + heap
alternatives that read the inner aggregate's input and absorb it (the
legacy keyed-additive rule decides applicability and the update). For
by series identity and weighted by the sample value.

An absorbed target has no choice of its own: enumeration skips its
choices, choice_index ranks in that order, and the tree DP sums only the
targets an alternative still reads (read_targets) and sets the absorbed
target to its pass-through. Top-k readout rows are bounded by the series
count, so a sketch reading several samples per item is sized like the
other realizations (the coupling guard caught the difference).

Example 1: 1 -> 64 -> 64 -> 1; P58 (all exact, shared input) at 52.201;
the best whole-expression CountSketch + heap plan, P64, costs 537.201.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/509-e2-keyed-additive-topk branch from 861c2ec to f08bd19 Compare October 11, 2026 17:11
zzylol added a commit that referenced this pull request Oct 13, 2026
…e MajorPass (#674)

* feat(planner): run the planning stages as the planner pipeline; retire MajorPass

plan_stages runs logical ASAP-aware optimization (Pass 1 and composition
of every combination, capped), physical ASAP-aware optimization and plan
selection over the frontends' roots, and returns every stage's output.
StagePipeline runs it behind OptimizationPass and is the facade default;
the stage_pipeline devtool uses it (its fixture is unchanged). The
combination count saturates instead of overflowing.

summary_sharing: sharing tests are ignored until Pass 2 (#580); the
UnivMon legacy-search test is deleted; scalar and combined-quantile tests
run on plan_stages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test(planner): logical and physical ASAP-aware optimization import no cost model

From #582: only plan selection prices plans, so the Pass 1 / composition
and physical ASAP-aware optimization modules may not import cost_model,
recurrence, analytical_cost, physical_operator_statistics or
plan_selection (paths or re-exported names) outside tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(planner): compose a per-series avg over rows that carry the series identity

The per-series sum keeps the input's labels, so with
$promql_series_identity after the sample value its last column is the
identity, and composition rejected the only exact option ("non-Float64
per-series avg"). The planner pipeline then had no candidate for
avg_over_time. Read the summed value by its role instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test(executor): take plans and candidates from the planner pipeline

From #629/#633: executor tests take the plan plan_stages selects
(selected_dag) or its composed logical candidates (logical_candidates)
instead of the legacy search and global selection.

- deployment_computation, precompute_candidates, the candidates.rs
  fixture: ported.
- planspace_series_identity_heap: keeps the selection check; the four
  tests of the legacy whole-root proposals are deleted.
- weighted_topk_binding: ported to Pass 1 heap candidates (Count Sketch
  only over rates: no sign proof). The tests of legacy-only candidate
  generation are deleted. Ignored, with reasons: the grouped weighted
  binding (whole-expression heap, #589) and the two rate-heap executions
  (top-k output schema, #672).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test(frontend-promql): take plans and candidates from the planner pipeline

From #629/#633, on support helpers over plan_stages (plan_stages,
selected_dag, logical_candidates, logical_plan):

- count_planning: exact counts select a Count accumulator; count
  updates are read from the composed Count-Min / Count Sketch
  candidates. The Hydra test is ported and ignored: Pass 1 offers no
  Hydra for a grouped count yet (#605).
- univmon_candidates: the synthetic accuracy model is gone; UnivMon
  candidates come from Pass 1 (the shared count at an approximate
  target), the uncalibrated ones carry no guarantee and plan selection
  does not select them.
- promql_corpus, metrics_observability: totality over Pass 1 and
  composition (logical_plan) instead of the legacy strategies; the
  metrics corpus reports composition errors under
  METRICS_OBSERVABILITY_REPORT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test(frontend-sql): take corr's exact plan from the planner pipeline

From #633: the plan plan_stages selects replaces retain_exact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test(planner): the batch SUM example reads one exact accumulator per query

Under the planner pipeline each query's SUM is replaced by its exact
accumulator state; sharing identical producers is Pass 2 (#580).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(planner): stage identifiers carry number and name; enumerate every candidate

User decisions (2026-10-13):
- stage modules stage2_physical_asap_optimization and
  stage3_plan_selection; tests stage1_logical_asap_optimization.rs and
  stage2_physical_asap_optimization_sql.rs; stage_pipeline keys
  stage0_query_language_frontends, stage1_logical_asap_optimization,
  stage2_physical_asap_optimization, stage3_plan_selection (fixture
  regenerated: only the keys change);
- enumeration and selection are never capped: plan_stages enumerates
  every combination; the devtool's --max-displayed (was
  --max-candidates) caps only the listing;
- code docs say "query-language frontends".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test(integration): take plans and candidates from the planner pipeline

From #629/#633, on asap_integration_tests::planning (plan_stages,
selected_dag, logical_candidates):

- sql_to_physical, operator_sharing, materialization_e2e,
  precompute_raw_samples, summary_coverage_e2e,
  promql_numeric_regressions: ported.
- cse.rs: deleted; it pinned the legacy search's memo groups (sharing
  identical sub-DAGs is Pass 2, #580).
- Ignored, with reasons: the UNION ALL summary test (plan selection picks
  the exact plan until materialization, #604), arithmetic over one
  selector (#580), the unit-weight heap over raw samples (#589).
- precompute_raw_samples no longer requires a Count Sketch heap over raw
  samples (#589); materialization_e2e takes the quantile's sketch
  candidate, since plan selection picks the exact plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(devtools): sketch_coverage reports Pass 1's summary candidates

From #633: per lowered query, the summary candidates Pass 1 offers,
instead of the legacy explain_replacements. The CSE-shareable metric is
dropped (sharing is Pass 2). Over every corpus at epsilon 0.01 and a 1 s
ingestion interval: 145/2183 lowered queries get a summary candidate, and
Pass 1 rejects none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test(frontend-promql): the Kubernetes mixin aggregates over topk compose (ignored, #672)

Both corpus queries that do not compose aggregate over a topk output,
which the frontend declares as [keys.., topk_k] with a Utf8 placeholder
and no value column; the outer sum's accumulator has no value to read.
They compose once the top-k output carries the ranked value (#672).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(devtools): run_planning_stages runs the library's plan_stages

One pipeline implementation: the devtool's run_planning_stages (#577)
validates the frontends' roots and every composed candidate, and
otherwise is plan_stages with the built-in models. Its candidate types
are the library's. The Example 1 fixture is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant