test: planner-layering Examples 1–4 acceptance specs on the planning stages

[zzylol]

Why

planner-layering.md walks four examples through query-language frontends, logical ASAP-aware optimization, physical ASAP-aware optimization and plan selection. Nothing checked the stages against those examples, so a stage could change its candidates or its selection without a test failing. Acceptance specs written by a test designer, run against the real stages, measure the stages against the doc and record every place where they still differ.

What

• Acceptance specs, one per example, under docs/design_docs/proposals/:
  • planner-layering-example1-acceptance.md (MVP: counts 1 → 3 → 3 → 1).
  • planner-layering-example2-acceptance.md, -example3-acceptance.md, -example4-acceptance.md, ported from test: #509 Examples 2–4 acceptance specs and end-to-end tests #590 (which will be closed).
  • Each lists per-stage candidate tables, invariants with their test and status, and where the doc and the planner differ.
• Tests in crates/integration-tests/tests/: planner_layering_example{1,2,3,4}.rs over adapters in planner_layering_common/. A test whose feature is missing is #[ignore]d with the PR or issue that delivers it. Test names carry the stage number and name (stage0_query_language_frontends_…, stage1_logical_asap_optimization_…, stage2_physical_asap_optimization_…, stage3_plan_selection_…).
• One shared pipeline in the asap-devtools library: planning_stages::run_planning_stages(workload, roots) returns a PlanningStagesRun (stage0_query_language_frontends, stage1_logical_asap_optimization, stage2_physical_asap_optimization, stage2_pruned, stage3_plan_selection) and runs logical ASAP-aware optimization (Pass 1 and composition, recording per-candidate composition errors), physical ASAP-aware optimization and plan selection over every candidate; planner_layering_examples holds the Example 1, 3 and 4 workloads. stage_pipeline serializes the run and the tests adapt it, so both see the same candidates, ids and selection.
• stage_pipeline --example planner-layering-3a | planner-layering-3b: Example 3's Pattern A batch and Pattern B panel in the devtool. Every candidate is optimized and priced; --max-displayed <n> (default 64; replaces --max-candidates) limits only how many the document shows, plus the selected one.
• docs/user_guide_docs/stage-viewer.md: the "What to look for" cells of tabs 1, 2, 3a and 3b describe the planner's current output, and the --max-displayed option.

How

The Example 1 spec and tests were written first against todo!() stubs, then the stubs were replaced by thin adapters over the stages from #575 and #576. Tests identify candidates by structure (which summary each query reads, which nodes queries share), never by id. Rebased on the final #576 (1ad70f9), the tests follow its behavior:

• Exact top-k: sort each job by the ranked value, limit 10, then a Project restoring [job, topk_10].
• Plan selection charges a NotMaterialized node once per consuming query and a materialized node once (NodeCost::multiplier). The shared charging helper now checks that rule instead of "each node once".

The viewer fixture tools/dag-viewer/examples/planner-layering-example1.json was regenerated with stage_pipeline --example planner-layering-1 and is byte-identical to #576's. planner-layering-3a prices all 243 candidates and selects P122 (KLL for all five queries), the first of 32 plans tied at the lowest cost, 78209376.4 (KLL or DDSketch per query, 2⁵; exact on an offset interval costs 2 more, e.g. P121 = 78209378.4); 3b selects P2 (KLL). Runs are uncapped: before, enumeration stopped at 64 candidates and 3a selected P41.

The shared data workload has no ingestion_rate when data is at rest: DataWorkload::validate rejects a positive rate there, and planner-layering.md's at-rest variant has no ingestion. The devtool always set it; the tests dropped it, and the shared definition now follows the tests.

Before

No test runs the planning stages on the doc's examples. Example 4's spec left open whether plan selection charges a non-materialized shared node once per consumer or physical ASAP-aware optimization duplicates it.

After

cargo nextest run -p asap-integration-tests -E 'binary(/planner_layering/)': 85 tests, 52 pass, 33 ignored.

Example 1 (1 → 3 → 3 → 1; the doc's full set is 1 → 54 → 156 → 1):

Id	Q2 top-k	Outcome	Reason
P1	exact: sort by value per job → limit 10 → Project [job, topk_10]	selected	the only valid candidate
P2	Count-Min + heap per job	invalid	sum_over_time weights are not proven non-negative
P3	CountSketch + heap per job	invalid	a top-k output has no guarantee and the deployment has no evidence

Q1 is exact in all three. Every node is NotMaterialized.

Example	Pass	Ignored	Ignored tests wait on
1	23	4	Hydra (#605); identical-expression rule (#588) ×2; every candidate compiling: CMS+heap weights (#593), CountSketch+heap estimate shape (#579)
2	14	10	SQL Entropy/L2 recognition (#598) ×2; specialized entropy/L2 summaries (no PR); summary-capability rule (#595, #596) ×4; UnivMon sizing and accuracy model (#621) ×2; shared UnivMon cheaper than three (#595, #596, #621, #604)
3	12	7	identical-expression rule (#588); window composition across queries (#628, #634) ×3; shared scan pricing (#588, #632); tumbling (#601) and sliding (no PR) windows ×2
4	3	12	Pattern A: shared window segments (#628, #634), materialization enumeration (#604), MatAtQuery (#625) ×6; Pattern B: tumbling windows (#601), #604, #625 ×5; sliding windows (no PR) ×1

Example 4's charging question is now decided: a shared NotMaterialized node is charged once per consuming query, so A3 costs five EH builds and A1 one. No Example 4 test became runnable from it, since the shared EH and its materialization options do not exist yet. Example 2's shared-UnivMon dominance test now also names #604: unmaterialized, the shared UnivMon is charged three times.

Evidence

• New Example 1 test stage2_physical_asap_optimization_exact_topk_returns_job_and_topk_10: Q2's root is Project → Limit 10 → Sort (one descending key on value, partitioned by job), with root columns [job, topk_10].
• assert_each_node_charged_per_consuming_query mirrors plan selection's computation count: walks from each operator root, scalar root and materialized node's inputs, stopping at materialized nodes; a materialized node counts 1. stage3_plan_selection_charging_helper_* check it on a hand-built candidate with a MatAtQuery node read by two queries.
• Example 2's SQL workload composes all 80 candidates; its ignored SQL test fails only on Q2's options, which wait on feat(frontend-sql): recognize the SQL frequency L2 and entropy idioms #598.
• example3a_selects_the_cheapest_of_all_candidates: stage_pipeline --example planner-layering-3a selects what run_planning_stages selects over all candidates (P122).
• Regenerated planner-layering-example1.json: identical to feat(planner): physical ASAP-aware optimization and plan selection for candidate logical ASAP DAGs #576's.

Verification

At e2afcd0:

• cargo fmt --all -- --check: clean.
• cargo clippy --workspace --all-targets -- -D warnings: clean.
• cargo nextest run --workspace: 1654 passed, 0 failed, 35 skipped (33 are the planner-layering ignores).
• python3 tools/dag-viewer/test_render.py: 29 tests OK.
• bash tools/verify_metricsql_external_consumer.sh: passes.

Architectural decisions

• Tests, specs, devtools and the stage-viewer user guide only. No stage or IR changes. New asap-devtools library modules (planning_stages, planner_layering_examples) so the devtool and the tests run one pipeline over the same workloads.
• Adapters in planner_layering_common are the only place that reads stage output, so a stage change touches one file.
• Ignored tests encode the doc's expectation, not today's output; each names what delivers it.
• Charging follows the decision in feat(planner): physical ASAP-aware optimization and plan selection for candidate logical ASAP DAGs #576: per consuming query for NotMaterialized, once for materialized. Physical ASAP-aware optimization does not duplicate shared nodes.
• Plan selection always ranks every candidate; a display cap never changes the selection.
• Example 1's tests count only candidates with no shared input (3 logical, 3 physical), pick a Q2 option's candidate by (option, no shared input) and do not count invalid candidates, so feat(planner): Pass 2 identical-expression rule (shared-input variants) #588's shared-input variants need no test change.

Limitations and follow-up

Where the planner differs from planner-layering.md Example 1 (listed in the spec, not fixed in the doc):

1. Counts: the doc's 54 logical and 156 physical candidates need Pass 2 (feat(planner): Pass 2 identical-expression rule (shared-input variants) #588, Pass 2 window composition: tumbling panes #601) and materialization enumeration (Stage 2 materialization: ingestion vs query time per summary #604); the MVP has 3 and 3.
2. Q2 options: the doc lists exact, Count-Min + heap, Hydra; the planner offers exact, Count-Min + heap, CountSketch + heap. Hydra is feat(planner): Pass 1 offers HydraCms for grouped approximate counts #605, and the doc does not name Hydra's inner sketch.
3. Count-Min + heap is invalid on unproven non-negative weights (feat(planner): declare counter metrics so CMS+heap can prove non-negative weights #593); the doc expects the accuracy model to judge it.
4. No summary is selectable: a composed top-k estimate has no guarantee, so the exact plan wins. The doc expects a summary to win in most deployments.
5. The doc's physical ASAP-aware optimization step for Example 1 is materialization only; the MVP's is operator implementation (exact top-k as sort + limit), one physical candidate per logical one.
6. The doc's Pass 1 diagram draws exact Q2 as sort + limit; the planner rewrites it only in physical ASAP-aware optimization.
7. "Shared summary charged once" does not apply: no summary is shared, and a NotMaterialized shared input is charged per consuming query, so sharing saves nothing until it is materialized.
8. Latency bounds (Q2 ≤ 100 ms) are not checked by plan selection.

Examples 2–4 list their own differences in their specs. No planner-layering-2 or -4 devtool example yet; the specs say what each needs.

Stack: tracker #528 · main → #577 → #578 → …

Human review — do not complete with an agent

• The MVP boundary is correct.
• New conceptual layers or public interfaces are necessary.
• The before/after description matches the intended product behavior.
• Human reviewer:
• Decision and rationale:

🤖 Generated with Claude Code

https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW