refactor(planner): replace the summary lifecycle with #509 materialization options

[zzylol]

Why

#509 Stage 2 (Physical ASAP-aware optimization) decides, per sub-DAG, whether its output is materialized and whether that happens at ingestion or at query time. The Planner's summary maintenance lifecycle made this choice with its own cost model, so Stage 2 and the lifecycle would both own the same decision.

What

Behavior change: planner output no longer materializes anything. Every SummaryAgg and MaintainPopulation is NotMaterialized until Stage 2 materialization (#604) picks an option.

• Remove the summary maintenance lifecycle.
  • Deleted: summary_maintenance_lifecycle, summary_maintenance_cost/, summary_maintenance_dag_export, post_asap::{summary_maintenance, summary_maintenance_lifecycle}, SummaryWindowFramework, the lifecycle CostModel hooks, CandidateCostOverrides, EmpiricalEvidenceProvider::lifecycle_cost_inputs, LifecycleInput and its planning horizon, the lifecycle errors, the lifecycle e2e test, the workload-demand-and-summary-lifecycle.md proposal and the viewer's lifecycle-plan UI.
  • Each plan is a QueryPlan { entry_index, root }. (In this PR the planner of that time, the legacy search, selected candidates by cost and assembled them; since feat(planner): run the planning stages as the planner pipeline; retire MajorPass #674 the planner is run_stage1_to_stage3 / Stage1ToStage3Pass, refactor(planner): name the planner pipeline Stage 1 to Stage 3 and its fields by stage #675 names, which outputs Stage 3 plan selection's choice.)
  • The pane primitives stay; Pass 2 window composition uses them.
• Materialization options per docs: propose workload-wide planning, summary sharing, and materialization #509 (Stage 2).
  • New types: Materialization { NotMaterialized, MatAtIngestion(Medium), MatAtQuery(Medium) } and Medium { Memory, Disk }. Only MatAtIngestion runs at ingestion time.
  • MaterializationAssignment::set accepts any node: per docs: propose workload-wide planning, summary sharing, and materialization #509 every sub-DAG output can be materialized ("Materialization decides, for each sub-DAG, whether its output is stored"; Example 4 stores panes). MatAtIngestion puts the node and everything upstream of it at ingestion time, and a query-time consumer reads its stored output. Default is NotMaterialized everywhere. with_default gives unassigned SummaryAgg and MaintainPopulation nodes a default option where the DAG allows one, and leaves them NotMaterialized where it does not.
  • The assignment holds Rc clones of its keys, so a reused address cannot match a stale entry. split_shared_by_phase returns the assignment remapped onto the rebuilt nodes, so explicit assignments survive the split.
  • MaintainPopulation follows its assignment, as SummaryAgg does; it no longer always runs at ingestion time. EvaluatePopulation reads a timed MaintainPopulation.
• Timing rules (ir::timing). Nothing is silently moved to the other timing; an infeasible assignment is an error:
  • Upstream rule (docs: propose workload-wide planning, summary sharing, and materialization #509 Algorithm 3, line 13): an ingestion-time node over input that exists only at query time fails with UpstreamNotAtIngestion. This covers an explicit MatAtIngestion and nodes a candidate builder fixed at ingestion time (e.g. a fixed-window rate finalize), which previously failed with IllegalChildDataState.
  • Fixed-timing rule: a node whose timing a candidate builder fixed (e.g. the query-time FinalizeExactAccumulator read boundary) keeps that timing. An explicit assignment that agrees is kept; one that needs the other timing fails with MaterializationConflictsWithFixedTiming.
  • Root rule: a root may be MatAtIngestion; the query reads its stored output. A row-producing root that runs at ingestion time without being stored there fails with UnstoredIngestionRoot, which replaces MaintenanceRowsAtRoot.
• Export. OperatorNode, FlatNode and PhysicalASAPDAGNode carry their Materialization. PhysicalASAPDAG validation rejects a materialization that contradicts the node's timing (MaterializationTimingMismatch). Wire version 10. PhysicalASAPDAG::with_execution_phases is removed (docs: propose workload-wide planning, summary sharing, and materialization #509 Assumption 8: the deployment executes the plan as given).
• Workload export. PlanOutput::execution_timed_dag (feat(planner): export a planned workload as one physical ASAP DAG #666) now times the operator roots and the scalar roots' references with apply_materialization_timings under the default assignment and one shared memo, and returns ExecutionDataStateError instead of the lifecycle timing error.
• Query roots are finalized. With the lifecycle gone, an exact-state root is read through FinalizeExactAccumulator without it, as in refactor(planner): plan the workload on the unified operator IR #542. On main, Stage 1 composition (feat(planner): compose candidate logical ASAP DAGs from Pass 1 per-computation candidates #575) builds the finalize into every exact-accumulator candidate.
• Physical planner. frontier_from_timing reads only ingestion timing and documents that MatAtQuery storage is not reflected yet (Stage 2 adds it). The maintained precompute compilers in promql_rows assign MatAtIngestion explicitly.
• Docs and remaining tests use the materialization vocabulary and the Operator<NodeId> payloads.

How

The PR removes the lifecycle and replaces the two-way timing assignment with the three #509 options. Candidate legality in Pass 1 is still checked as if every summary were MatAtIngestion where the DAG allows it (validate_at_ingestion, formerly validate_default), so a candidate stays executable if Stage 2 later materializes it at ingestion; candidate generation is unchanged. planned_data_state uses the same assumption. Tests that exercise ingestion-time execution assign it explicitly (at_ingestion helpers). The four execution tests removed with the lifecycle are ported to crates/integration-tests/tests/materialization_e2e.rs and check that NotMaterialized and MatAtIngestion give the same answers.

Before this PR

The planner chose a lifecycle per summary state (maintain at ingestion, or recompute at query time) using lifecycle costing, and timed plans with that choice. MaintainPopulation always ran at ingestion time, and a candidate-fixed timing that conflicted with its upstream failed with a generic IllegalChildDataState.

After this PR

Nothing is materialized unless a caller assigns an option; choosing options is Stage 2's job (#604). On main, Stage 2 physical ASAP-aware optimization (#576) assigns NotMaterialized to every sub-DAG, and Stage 3 plan selection picks the plan.

API changes:

• UserInput::new / OptimizationInput::new drop the lifecycle argument; LifecycleInput, OptimizationInputError::InvalidHorizon and OptimizeError::{LifecycleSelection, LifecycleAssembly} are removed (assembly errors became OptimizeError::Realization, which feat(planner): run the planning stages as the planner pipeline; retire MajorPass #674 removed with the legacy search).
• QueryLifecyclePlan becomes QueryPlan { entry_index, root }.
• LifecycleAssignment / apply_lifecycle_timings / validate_default become MaterializationAssignment / apply_materialization_timings / validate_at_ingestion.
• MaterializationAssignment::set takes any node; split_shared_by_phase returns the remapped assignment.
• ExecutionDataStateError gains UpstreamNotAtIngestion, MaterializationConflictsWithFixedTiming and UnstoredIngestionRoot (replacing MaintenanceRowsAtRoot); PhysicalASAPDAGValidationError gains MaterializationTimingMismatch.
• PlanOutput::execution_timed_dag returns ExecutionDataStateError.
• PhysicalASAPDAG::with_execution_phases is removed; PHYSICAL_ASAP_DAG_WIRE_VERSION is 10.

Evidence

• A unit test pins the default: nothing is materialized.
• Where lifecycle selection used to force raw recompute, the plan is no longer chosen by lifecycle costing. On main, Stage 3 plan selection (feat(planner): physical ASAP-aware optimization and plan selection for candidate logical ASAP DAGs #576, the planner since feat(planner): run the planning stages as the planner pipeline; retire MajorPass #674) chooses it; see e2e_plan, summary_sharing and operator_design_examples.
• ir::timing unit tests cover each rule: ingestion_time_node_over_query_time_upstream_is_rejected, ingestion_pinned_finalize_over_query_time_state_is_upstream_not_at_ingestion, fixed_timing_keeps_an_agreeing_assignment_and_rejects_a_conflict, a_root_may_be_stored_at_ingestion_time, any_node_may_be_materialized, split_keeps_explicit_assignments. promql_to_post_asap covers UpstreamNotAtIngestion through a frontend.
• materialization_e2e.rs executes the KLL quantile, the current-series population, and sum_over_time arithmetic over one and over different selectors with NotMaterialized and with MatAtIngestion, and gets the same results.
• types/tests/physical_export.rs covers MaterializationTimingMismatch (materialization_inconsistent_with_timing_is_rejected), including on a scalar-only DAG.
• Screenshots and measurements: not applicable.

Verification

• Unit and integration tests: lifecycle-only tests are deleted with the lifecycle; the rule tests above are new. materialization_e2e.rs: materialization_decides_precompute_contents, population_materialization_decides_precompute_contents, arithmetic_over_one_selector_executes, arithmetic_over_different_selectors_matches_prometheus.
• Viewer: dag-viewer test_render (29 passed; the lifecycle-plan UI tests are removed).
• Other checks on the PR head: cargo fmt --check; workspace all-target Clippy with -D warnings; cargo test --workspace (1,540 passed, 0 failed, 2 ignored); tools/verify_metricsql_external_consumer.sh OK. The test count is lower than refactor(ir): remove the legacy QueryExpr and SummaryExpr IRs; move docs and DAG viewer to the unified IR #543's 1,635 because the lifecycle tests are deleted with the lifecycle.

Architectural decisions

• The lifecycle is deleted, not moved. Stage 2 materialization replaces it, and keeping both would give the same decision two owners.
• Names follow docs: propose workload-wide planning, summary sharing, and materialization #509: NotMaterialized / MatAtIngestion / MatAtQuery. The medium is part of the option, so MaintainPopulation gets the same three options.
• No silent re-timing. An infeasible explicit assignment, a conflict with a candidate-fixed timing, or an unstored ingestion-time root is an error.
• Any node can be materialized (docs: propose workload-wide planning, summary sharing, and materialization #509), while with_default only supplies defaults for SummaryAgg / MaintainPopulation; Stage 2 chooses the rest.

Later changes: #674 retired the legacy search (MajorPass) as the planner; the planner is run_stage1_to_stage3 / Stage1ToStage3Pass (#675 names), Stage 2 assigns NotMaterialized everywhere until #604, and identical producers are not merged until Pass 2 (#580).

Limitations and follow-up

• Stage 2 materialization: ingestion vs query time per summary #604 (Stage 2 materialization) chooses the option per sub-DAG, decides retention, and makes frontier_from_timing reflect MatAtQuery storage. Timing: a node shared across batch queries with different timings fails with ConflictingTiming #654 covers batch timing conflicts.

Stack: tracker #528 · main → #542 → #665 → #666 → #543 → #667 → #561.

Human review — do not complete with an agent

• The MVP boundary is correct.
• New conceptual layers or public interfaces are necessary.
• The before/after description matches the intended product behavior.
• Human reviewer:
• Decision and rationale:

🤖 Generated with Claude Code

https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW