feat(planner): Pass 2 identical-expression rule (shared-input variants) by zzylol · Pull Request #588 · ProjectASAP/ASAPPlanner

zzylol · 2026-10-04T15:37:16Z

Rebased on main d4869a7 (DF 54).

Why

#509 Stage 1 is Pass 1 (local alternatives) + Pass 2 (ASAP-aware CSE). Pass 2's identical-expression rule was missing. In Example 1 both queries read http_requests_total[1m], so the doc's "Raw with a shared input" plan was never a candidate. The facade also called share_common_sub_dags unconditionally while the devtool never did, so the two disagreed. In practice the facade's merge was a no-op for PromQL: the scan had no unique key, and CSE's legality rule refuses to share without one. Part of #580; stacked on #587 (#572).

What

Pass 2 identical-expression rule (asap-logical-optimizer/src/pass2/identical_expressions.rs). Stage 1 returns sharing variants: Pass 1 over the queries as written, and Pass 1 over the queries with identical sub-DAGs merged by the existing share_common_sub_dags. The shared variant is added only when the merge removes nodes. Shared candidates are labelled "… · shared input" and numbered after the independent ones.
Selection treats the variant as an outer choice (asap-plan-selection). The per-target DP runs once per variant, and the cheapest wins. In the shared variant, identical summary producers are also merged after composition. They are priced the same way in the DP, in exhaustive enumeration and in the final full check. The coupling guard also checks pairs of targets that read a common input. Fallback and the final full check apply per variant.
One library path. plan_selection::plan_stages runs Stage 1 (Pass 1 + Pass 2) → Stage 2 → Stage 3. The facade calls it and no longer pre-merges. The devtool calls it and only serializes the result (with display set so every candidate is priced).
Scan key. with_promql_series_identity now declares (series identity, ts) as the identified PromQL scan's unique key. A series has at most one sample per timestamp. CSE's legality rule needs this key to share the range selector.

How it was checked

DP = exhaustive on Example 1 (48 combinations, P44) and on a new workload where sharing merges a whole target: sum by (job) (rate(x[1m])) and max by (job) (rate(x[1m])), 16 + 8 combinations, where a shared candidate wins.
Un-ignored stage1_keeps_independent_and_shared_variants and stage3_shared_input_is_not_costlier. Hydra was dropped from both loops; it stays covered by the still-ignored stage1_q2_summary_families_are_heap_sketches_and_hydra, because Pass 1 has no Hydra alternative.
New stage3_selects_a_shared_input_plan: the selected plan shares the input, and it saves exactly one scan plus one range node.
Acceptance spec, its 48-row table and the viewer fixture are regenerated from the planner's output.

Before this PR (Example 1)

1 → 24 → 24 → 1. Selected P20 "Q1 exact (Sum acc, Rate acc) · Q2 exact (Sum acc)", 79.401 cpu ms. The scan (23.2) and range (4.0) are priced twice, once per query.

After this PR (Example 1)

1 → 48 → 48 → 1 (24 independent + 24 shared; 16 Count-Min + heap candidates invalid as before). Selected P44 "Q1 exact (Sum acc, Rate acc) · Q2 exact (Sum acc) · shared input", 52.201 cpu ms. The same choices as P20, with one scan → range:

Node	P20	P44
scan 4,000,000 samples	23.2 × 2	23.2
time range 60 s	4.0 × 2	4.0
Rate acc build + finalize (Q1)	5.0	5.0
Sum acc build + finalize (Q1)	1.0	1.0
Sum acc build + finalize (Q2)	5.0	5.0
sort 1M rows / 100 partitions + limit (Q2)	14.001	14.001
Total	79.401	52.201

The best CountSketch + heap plan is P48 at 164.201 (shared).

Gate

cargo fmt --all --check ✓; cargo clippy --workspace --all-targets --all-features -- -D warnings ✓; cargo test --workspace: 1,514 passed / 10 ignored (#587: 1,508 / 12). The difference is 4 new tests (2 Pass 2 unit tests, shared_target_dp_equals_exhaustive, stage3_selects_a_shared_input_plan) and 2 un-ignored. Viewer: python3 -m unittest discover -s tools/dag-viewer -p test_render.py, 29 OK (6 skipped).

Refs #509, #580, #572.

🤖 Generated with Claude Code

…variant Stage 1 now keeps two forms of the workload (#509 Pass 2): the queries as written, and, when CSE merges something, the queries with identical sub-DAGs shared. Stage 3 treats the variant as an outer choice: the per-target dynamic program runs once per variant (the coupling guard also checks targets reading a common input in the shared variant) and the cheapest result wins. plan_selection::plan_stages runs Stage 1 -> 2 -> 3; the facade and the stage_pipeline devtool both call it, so the facade no longer pre-merges and the devtool shows the shared variants. A PromQL series has one sample per timestamp, so the identified scan now declares (series identity, ts) as a key, which CSE's legality rule needs to share the range selector. Example 1: 1 -> 48 -> 48 -> 1; P44 (P20's choices over a shared input) is selected at 52.201 vs 79.401. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… flat DAGs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

The tests run Pass 1 and composition, optimize_physical_asap and select_plan through a shared tests/planner_layering_common pipeline, and follow the decided behavior: three Pass 1 candidates (Q2 exact, CMS+heap, CountSketch+heap), every sub-DAG NotMaterialized, P1 selected, CMS+heap invalid (weights not proven non-negative) and CountSketch+heap invalid (a Top-K output has no guarantee and no deployment evidence). Hydra (#605), Pass 2 sharing (#588) and the two runtime gaps (#593, #579) stay ignored with the PR that delivers them. Tests are named after the stages; the compile check no longer uses the PromQL fallback. The spec lists where the doc and the planner differ. The expected-only viewer fixture is removed: the generated four-stage fixture replaces it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

The specs and tests from #590 run the current stages through the shared planner_layering_common pipeline (Pass 1 and composition, optimize_physical_asap, select_plan); tests are named after the stages. Materialization is read from the physical DAG (#667); window forms and retention stay pending adapters. Behavior that works passes (29 tests); every expectation that needs a feature not in this stack is ignored with the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596), window composition (#601, #628, #634), materialization enumeration and MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL recognition and COUNT(*) composition (#598, #603) and plan selection row estimates (#632). Each spec's "Today" column states the current output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

The tests run Pass 1 and composition, optimize_physical_asap and select_plan through a shared tests/planner_layering_common pipeline, and follow the decided behavior: three Pass 1 candidates (Q2 exact, CMS+heap, CountSketch+heap), every sub-DAG NotMaterialized, P1 selected, CMS+heap invalid (weights not proven non-negative) and CountSketch+heap invalid (a Top-K output has no guarantee and no deployment evidence). Hydra (#605), Pass 2 sharing (#588) and the two runtime gaps (#593, #579) stay ignored with the PR that delivers them. Tests are named after the stages; the compile check no longer uses the PromQL fallback. The spec lists where the doc and the planner differ. The expected-only viewer fixture is removed: the generated four-stage fixture replaces it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

The specs and tests from #590 run the current stages through the shared planner_layering_common pipeline (Pass 1 and composition, optimize_physical_asap, select_plan); tests are named after the stages. Materialization is read from the physical DAG (#667); window forms and retention stay pending adapters. Behavior that works passes (29 tests); every expectation that needs a feature not in this stack is ignored with the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596), window composition (#601, #628, #634), materialization enumeration and MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL recognition and COUNT(*) composition (#598, #603) and plan selection row estimates (#632). Each spec's "Today" column states the current output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

physical_with and the invalid-candidate check no longer assume one candidate per Q2 option, so #588's shared-input variants need no change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

…per names The 3-logical and 3-physical counts now hold when #588 adds shared-input variants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

…stages (#577) * test: acceptance spec and pending tests for planner-layering Example 1 Define MVP acceptance for #509 Example 1 before the Phase C stage APIs exist: 1 -> 6 -> 6 -> 1 candidates (Pass 1 x identical-expression sharing; physical operator implementation only, no materialization). - Spec with per-stage candidate tables, invariants and doc ambiguities. - Integration tests against todo!() stage stubs, ignored until the stages land; workload and Stage 0 frontend-shape tests run now. - Expected-only asap-stage-pipeline/v1 fixture for the DAG viewer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: run the Example 1 acceptance tests against the real stages Replace the todo!() stubs with adapters over Stage 0-3, un-ignore the 11 tests that pass, and give each still-ignored test the precise difference between the implementation and the spec. Add runtime capability checks: the selected plan compiles in the physical planner; compiling every candidate stays ignored with the two runtime gaps it finds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: read Example 1 logical stages as flat DAGs after the export removal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test: Example 1 acceptance on the composed stages (1 -> 3 -> 3 -> 1) The tests run Pass 1 and composition, optimize_physical_asap and select_plan through a shared tests/planner_layering_common pipeline, and follow the decided behavior: three Pass 1 candidates (Q2 exact, CMS+heap, CountSketch+heap), every sub-DAG NotMaterialized, P1 selected, CMS+heap invalid (weights not proven non-negative) and CountSketch+heap invalid (a Top-K output has no guarantee and no deployment evidence). Hydra (#605), Pass 2 sharing (#588) and the two runtime gaps (#593, #579) stay ignored with the PR that delivers them. Tests are named after the stages; the compile check no longer uses the PromQL fallback. The spec lists where the doc and the planner differ. The expected-only viewer fixture is removed: the generated four-stage fixture replaces it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test: port #590's Example 2-4 acceptance specs and tests The specs and tests from #590 run the current stages through the shared planner_layering_common pipeline (Pass 1 and composition, optimize_physical_asap, select_plan); tests are named after the stages. Materialization is read from the physical DAG (#667); window forms and retention stay pending adapters. Behavior that works passes (29 tests); every expectation that needs a feature not in this stack is ignored with the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596), window composition (#601, #628, #634), materialization enumeration and MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL recognition and COUNT(*) composition (#598, #603) and plan selection row estimates (#632). Each spec's "Today" column states the current output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * feat(devtools): stage_pipeline examples for #509 Example 3 Ported from #590: --example planner-layering-3a (Pattern A, ad hoc batch of five p99 reports) and planner-layering-3b (Pattern B, repeating 5-min p99), the Example 3 acceptance tests' workloads. Example 1's output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test: follow #576's per-consumer charging and exact top-k columns Plan selection charges a NotMaterialized node once per consuming query, so the charging helper checks NodeCost.multiplier and the Example 4 A3 ambiguity is recorded as decided. Example 1 checks that exact Q2 sorts by the ranked value and returns [job, topk_10]. The stage viewer guide's tab 1 row now describes the actual all-exact result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * refactor(devtools): one planning pipeline for stage_pipeline and the acceptance tests run_planning_stages and the #509 example workloads move into the asap-devtools library; stage_pipeline serializes the run and the planner-layering tests adapt it. The shared data workload drops ingestion_rate for data at rest, which validate() requires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * fix(devtools): select over every candidate; --max-displayed limits only the document stage_pipeline capped enumeration at 64, so Example 3a (243 candidates) selected P41 instead of the cheapest P122 (KLL for all five queries). The pipeline now enumerates, optimizes and prices every candidate; the document shows the first --max-displayed (default 64) plus the selected one, with `combinations` and `capped`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * docs: Example 2's SQL candidates compose; the SQL test waits only on #598 Running the ignored SQL test composes every candidate and fails only on Q2's options (per-source counts until #598), so #603 is not a blocker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test: name stages by number and name; say query-language frontends The frontends' Run field, id and locals become stage0_query_language_frontends, and test names carry stage0..stage3 with the stage name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test: charging helper follows plan selection's per-consumer rule With a materialized node the helper accepted any multiplier >= 1. It now computes each node's computations as plan selection does (walks from query roots and materialized nodes' inputs, stopping at materialized nodes), checked on a hand-built candidate with a materialized node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test: Example 1 picks Q2 candidates by (option, no shared input) physical_with and the invalid-candidate check no longer assume one candidate per Q2 option, so #588's shared-input variants need no change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * docs: Example 3 Pattern B charges each node once per consuming query Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * docs(stage-viewer): rows 2, 3a and 3b describe the planner's current output Read from uncapped runs: Example 2's sketch plans are rejected for having no (ε, δ) guarantee, 3a selects P122 (first of 108 tied cheapest plans, nothing shared yet) and 3b selects query-time KLL; panes and the latency check come with the window-summary PRs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * docs(stage-viewer): row 2 from the SQL run; 3a ties 32 plans, not 108 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test: Example 1 counts only unshared candidates; stage3_ charging-helper names The 3-logical and 3-physical counts now hold when #588 adds shared-input variants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * refactor(devtools): PlanningStagesRun fields carry stage number and name Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW * test(devtools): unique temp file per stage_pipeline run cargo test runs the tests as threads of one process, so two runs of the same example shared a file name and one deleted the other's output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

zzylol · 2026-10-13T17:02:48Z

Closing as part of the upper-stack consolidation (G1): its plan_stages entry was rebuilt in #674 on #576's modules; its identical-expression (Pass 2) rule moves to the Example 1 group (G4). Closed with the author's approval.

This was referenced Oct 4, 2026

feat(planner): whole-expression top-k heap sketch over raw samples #589

Draft

refactor: unify logical operators and scalar expressions #528

Closed

zzylol force-pushed the stack/509-e1-pass2-identical branch 2 times, most recently from a52409e to 63eb88c Compare October 8, 2026 16:47

zzylol force-pushed the stack/572-b5-executor branch from 3d570ff to 82aa0c5 Compare October 10, 2026 14:32

zzylol force-pushed the stack/509-e1-pass2-identical branch from 63eb88c to 18f8dae Compare October 10, 2026 14:32

zzylol and others added 2 commits October 11, 2026 16:22

refactor(planner): port the Pass 2 identical rule's stage pipeline to…

901c951

… flat DAGs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

zzylol force-pushed the stack/572-b5-executor branch from 82aa0c5 to ac4d8d7 Compare October 11, 2026 17:11

zzylol force-pushed the stack/509-e1-pass2-identical branch from 18f8dae to 901c951 Compare October 11, 2026 17:11

This was referenced Oct 12, 2026

Pass 1: push down filters and HAVING across levels to create more candidates (and reuse DataFusion's filter push-down) #664

Open

Select complete workload candidates with exhaustive sharing and lifecycles #568

Closed

zzylol mentioned this pull request Oct 13, 2026

test: planner-layering Examples 1–4 acceptance specs on the planning stages #577

Merged

3 tasks

zzylol mentioned this pull request Oct 13, 2026

feat(planner): run the planning stages as the planner pipeline; retire MajorPass #674

Merged

3 tasks

zzylol closed this Oct 13, 2026

zzylol mentioned this pull request Oct 13, 2026

Tracking: end-to-end use case support for ASAPPlanner (inputs, missing components, adding summaries) #685

Open

40 tasks

Navigation Menu

Repository navigation

feat(planner): Pass 2 identical-expression rule (shared-input variants) - #588

feat(planner): Pass 2 identical-expression rule (shared-input variants)#588
zzylol wants to merge 2 commits into
stack/572-b5-executorfrom
stack/509-e1-pass2-identical

zzylol commented Oct 4, 2026 •

edited

Loading

Uh oh!

zzylol commented Oct 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

zzylol commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

How it was checked

Before this PR (Example 1)

After this PR (Example 1)

Gate

Uh oh!

zzylol commented Oct 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

zzylol commented Oct 4, 2026 •

edited

Loading