Repository navigation
Tracking: end-to-end use case support for ASAPPlanner (inputs, missing components, adding summaries) #685
Description
End-to-end use case support for ASAPPlanner. Design doc: #684 (docs/design_docs/proposals/asapplanner-e2e-use-case.md).
Inputs (doc: Detailed Design, one subsection per input)
- Query workload: queries, language, accuracy targets (on main)
- Latency requirements enforced in selection — Selection per #509: latency bounds and reasons for losing candidates — after #511 #526, Stage 3: memory, disk and latency trade-offs per candidate, and cost models beyond cloud billing #660
- Recurrence and predictability consumed by Stage 2 — Stage 2 materialization: ingestion vs query time per summary #604
- Time selection consumed (window segments) — feat(pass2): shared year-aligned window segments (Q60) #628, Pass 2: the planner chooses the segment grid #634
- Data workload: arrival, rate, cardinality used in Stage 2/3 — Stage 2 materialization: ingestion vs query time per summary #604
- Raw-data retention — Deployment inputs: capabilities and raw-data retention #609
- Declared value domains / sign / counter semantics — proposed
- Distinct-count bounds and HLL confidence — Wire the HLL confidence model into Stage 1 sizing and plan selection #678
- Cost model: deployment prices and calibration — Stage 3 per-second cost model, memory pricing and calibration #594, fix(planner): price state by its size; size KLL for δ #682
- Cost models 1/2/3 — Plan selection cost models: usage over time (cost model 1), peak capacity (cost model 2), own-cluster trade-offs (cost model 3) #683, Stage 3: memory, disk and latency trade-offs per candidate, and cost models beyond cloud billing #660
- Measured costs from sketch-bench — Calibrate plan selection's costs from measured sketch costs #679, cost model: validate analytical estimates against empirical benchmark evidence #347
- Accuracy: guarantee composition and top-k band — Compose accuracy guarantees through arithmetic and comparisons over approximate values #669
- Deployment capabilities — Deployment inputs: capabilities and raw-data retention #609, Deployment inputs: one bundle for cost model, accuracy model and capabilities — after #511 #525
Missing components (doc: Detailed Design, Missing components)
- Early search-space pruning — Prune the candidate search space early without discarding a possibly-best valid plan #673
- Plan search strategies — Planning roadmap: move ASAPQuery planning into ASAPPlanner, MILP plan selection over shared DAGs, benchmark-based accuracy #671
- DataFusion rules in SQL lowering — Run selected DataFusion optimizer rules in SQL lowering; a rule-list API for ASAPPlanner rewrites #670
- Filter/HAVING push-down — Pass 1: push down filters and HAVING across levels to create more candidates (and reuse DataFusion's filter push-down) #664
- Pass 2 reuse rules — Pass 2 reuse rules: rollup, top-k reuse, nested sum merge #680
- Pass 2 ASAP-aware CSE — G4 (feat(planner): Pass 2 identical-expression rule (shared-input variants) #588, feat(planner): whole-expression top-k heap sketch over raw samples #589, feat(planner): declare counter metrics so CMS+heap can prove non-negative weights #593, Stage 3 per-second cost model, memory pricing and calibration #594, fix(logical-optimizer): move whole-expression top-k partitions onto the heap input #640), G5 (Pass 2 summary-capability rule: share one summary sized for the strictest consumer #595–feat(executor): execute the exact SQL entropy fallback; Example 2 status #599, UnivMon: certify L2 from layer 0's F2 #621, Stage 3: satisfies checks the error metric #627), G6 (executor: support tumbling pane merges (per-entity TimeShift panes, temporal merge timestamp) #591, types: evaluation-relative coverage time and mergeable summary families #592, Pass 2 window composition: tumbling panes #601), G10 (feat(pass2): shared year-aligned window segments (Q60) #628, Stage 3: price TimeShift as free and TimeRange by the rows it keeps #632, Pass 2: the planner chooses the segment grid #634)
- Stage 2 materialization — Stage 2 materialization: ingestion vs query time per summary #604, Example 4 acceptance with materialization #606
- Maintained-population top-k — Offer maintained-population top-k candidates in Pass 1 #681
- Top-k output schema — Top-k results drop the item columns (port/job); heavy-hitter candidates resolve columns by name #672
- Executing scalar operator references — Execute scalar operator references in Project/Filter expressions of the physical DAG #668
- Hydra grouped counts in Pass 1 — feat(planner): Pass 1 offers HydraCms for grouped approximate counts #605
- Docs after consolidation — Update the docs after the G1–G11 consolidation groups land #677
- Selected-plan handoff envelope (guarantees, costs, input digest, version compatibility) — proposed
- Plan executability validation (capabilities + deployment dry bind) — Deployment inputs: capabilities and raw-data retention #609, PhysicalASAPDAG::validate does not check per-operator timing #656, dry bind proposed
- Re-planning on workload/data change and state transition — proposed
- Runtime feedback into accuracy/cost models — Feed observed posterior sketch error back into re-planning (execution-time → planning-time loop) #250, cost model: validate analytical estimates against empirical benchmark evidence #347, rest proposed
- Latency estimation per candidate — Selection per #509: latency bounds and reasons for losing candidates — after #511 #526, Stage 3: memory, disk and latency trade-offs per candidate, and cost models beyond cloud billing #660
- User-facing plan explanations — proposed
- End-to-end acceptance on a real deployment (Examples 1–4) — proposed
Adding a summary family (doc: Adding a new summary family end to end, Algorithm 3)
- (a) Capability declaration: intents, statistics, input types, weight domain, grouping, mergeability, windows
- (b) Pass 1 offer in
summary_candidates, unsized - (c) Sizing (
default_size_params) and theoretical bound, or evidence viaoutput_guarantee - (d) Stage 2 physical operator implementation
- (e) Stage 3 state size and pricing (
summary_shape) - (f) Executor kernel via asap_sketchlib; coverage (docs: ASAP primitive schema and summary coverage (definition + selection) #573)
- (g) sketch-bench accuracy and cost measurements
- (h) Tests: catalog, composition, Stage 3, executor vs exact, acceptance example
Activity
docs: link the e2e use-case tracking issue #685
zzylol commentedon Oct 14, 2026
Work items → PRs/issues (moved out of the design doc)
The design doc
docs/design_docs/proposals/asapplanner-e2e-use-case.mdno longer names PRs or issues. This is the mapping it carried (as oforigin/mainat6665fcc3), so status stays tracked here.Design docs cited
planner-layering.md(stages, Assumptions 1–8, Examples, Scenarios)asap-primitive-schema.md(summary coverage, maintained populations)decoupling_op_and_expr.md(operators vs scalar expressions)Inputs not read by the pipeline on main
MatAtIngestion/MatAtQuery, media) (G8)Cost model
PlanningModels.costwith the legacy searchAccuracy model
Tracked missing components
stage_pipeline(G11)validateProposed components (no issue yet when the doc was written)
docs: link the e2e use-case tracking issue #685