Skip to content

refactor(types): arrange asap-types modules by #511 section (#572 B1) - #578

Draft
zzylol wants to merge 45 commits into
stack/509-c4-example1-acceptancefrom
stack/572-b1-types-modules
Draft

zzylol wants to merge 45 commits into
stack/509-c4-example1-acceptancefrom
stack/572-b1-types-modules

Conversation

@zzylol

@zzylol zzylol commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Rebased on main d4869a7 (DF 54).

Step B1 of Phase B of #572 (tracker #528): arrange asap-types by #511 section instead of the pre-ASAP / post-ASAP split.

Why

pre_asap and post_asap no longer describe their contents. There is one operator DAG (#511), pre_asap::schema is the schema of every DAG, and post_asap::sketch is shared vocabulary that the logical stage uses. The later Phase B crates (logical-optimizer, physical-optimizer, plan-selection, executor) should import types by what they are, not by a stage name that no longer exists.

What

Before this PR

crates/types/src/
  ir/          node, non_asap, asap, operator_properties, scalar, aggregate_schema, error,
               timing, summary_coverage, export, physical_export, wire, cse, canonicalize,
               query, schema_support
  pre_asap/    agg_intent, expr_ir, schema, scalar_type_rules, column_resolution
  post_asap/   sketch, guarantee, execution_data_state, maintained_population,
               summary_window, query_time/
  workload.rs, parsed_workload.rs, resources.rs + resources/
  cost.rs, dag_export.rs, serde_f64.rs, types.rs

After this PR

crates/types/src/
  ir/
    operator/    node, non_asap, asap, operator_properties, agg_intent, maintained_population
    scalar/      ScalarExpr (mod.rs), expr_ir items, scalar_type_rules, column_resolution
    schema/      Schema/Field/FieldDataType/DataType (mod.rs), aggregate_schema, error,
                 state_type (was post_asap::sketch)
    properties/  guarantee, timing, summary_coverage,
                 execution (ExecutionTiming / DataPrimitive / ExecutionDataState / ExecutionDataStateError)
    export, physical_export, wire, cse, canonicalize, query, schema_support
  workload/      workload (mod.rs), parsed_workload, resources/
  physical/      execution_data_state (lift_plain, ExactOperationSchemaError), summary_window
  cost.rs, dag_export.rs, serde_f64.rs, types.rs   (unchanged)
  • Public type names do not change. Only module paths change.
  • post_asap::query_time had no callers and is deleted, along with its 20 unit tests.
  • No compatibility re-exports from the old paths. Each section's mod.rs re-exports the vocabulary that pre_asap / post_asap used to re-export, so pre_asap::AggIntent becomes ir::operator::AggIntent, post_asap::SketchAlgorithm becomes ir::schema::SketchAlgorithm, and so on. The ir root keeps its existing re-exports (OperatorNode, ASAPOp, the timing functions, …).
  • lib.rs crate docs and the docs that cite module or file paths are updated.

How

  • git mv for every file, so history follows the moves. Similarity is 88–100 %.
  • A script parses every use tree rooted at asap_types, planner_types / planner (the alias in asap-physical-operators) or crate (inside asap-types), maps each leaf path to its new module, and regroups the result. It does the same for inline qualified paths. Then cargo check iteration and cargo fmt.
  • Measured volume: 217 files changed (204 .rs). About 580 use statements rewritten, plus about 345 inline qualified paths.
  • Behavior-preserving. #[test] count goes from 1793 to 1773, and the 20 removed are exactly the deleted query_time tests. No other test is changed beyond its import paths.

Gate at this tip: cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features -- -D warnings, cargo test --workspace (1518 passed, 0 failed, 12 ignored) and python3 -m unittest discover -s tools/dag-viewer -p test_render.py (29 run, OK, 6 skipped) all pass.

Deviations from #572, for review

  • ExecutionDataStateError and DataPrimitive go to ir::properties together with ExecutionTiming / ExecutionDataState, not to physical. The reason is the one Reorganize crates and modules by #509 stages and the #511 unified IR #572's update gives for the split: ir/timing and ir/physical_export use them. What stays in physical::execution_data_state is lift_plain and ExactOperationSchemaError. Their only callers today are Stage 1 (exact_composition), so physical may not be their final home.
  • expr_ir.rs is now a private file of ir::scalar, and its four items are re-exported there. This avoids an ir::scalar::expr_ir path that keeps the legacy name.
  • Not listed in Reorganize crates and modules by #509 stages and the #511 unified IR #572, so left in place: ir::schema_support (PromQL series-identity realization) stays at the ir root, and cost, dag_export, serde_f64, types stay at the crate root.
  • tools/dag-viewer/post_asap_fixture.json still exists but nothing references it, so it is not renamed. PostAsapDAG / pre_asap_sub_dag are gone from code. PostAsapDAG is still mentioned in prose in docs/design_docs/concepts/post-asap-ir.md, docs/design_docs/physical-planning-and-deployment.md and docs/develop_docs/library-api.md, which this PR leaves alone.
  • Migration tables in docs/develop_docs/*-migration.md keep their historical pre_asap:: paths.
  • Step B0 (moving the CostModel trait and recurrence types into asap-types) is not in this stack, and this PR is based directly on test: planner-layering Examples 1–4 acceptance specs on the planning stages #577. B0 cannot be done as a types-only move:
    • The trait's signature needs Stage 1 types that carry behavior: ReplacementSubDAG, TargetSubDAG, Replacement, Realization, ExactComposition, ExactOperation and OperationPlacement. For example, ExactComposition::compose_with_accuracy uses AccuracyModel, and OperationPlacement::provenance returns ReplacementProvenance.
    • Two default method bodies call behavior: size_params calls accuracy::estimators::size_params, and cse_share_decision_with_recurrence calls recurrence::decide.
    • B0 needs a decision before it can proceed.

Refs #572 · #528

🤖 Generated with Claude Code

@zzylol
zzylol force-pushed the stack/509-c4-example1-acceptance branch from 0afc939 to f620555 Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/572-b1-types-modules branch from dfaa958 to 7eda06e Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/509-c4-example1-acceptance branch from f620555 to b8ad78c Compare October 8, 2026 16:47
@zzylol
zzylol force-pushed the stack/572-b1-types-modules branch 2 times, most recently from 078adb8 to 0babc9d Compare October 10, 2026 14:32
@zzylol
zzylol force-pushed the stack/509-c4-example1-acceptance branch from b8ad78c to 0541f9a Compare October 10, 2026 14:32
zzylol and others added 23 commits October 11, 2026 16:22
Port DF 54's API changes (Expr::Literal metadata, Cast/TryCast field,
ScalarUDFImpl return_field_from_args/invoke_with_args, catalog path,
dialect name) and the planner behaviour changes (SELECT * keeps its
projection, GenericDialect parses aggregate FILTER) from main's
crates/frontend-sql/src/sql onto the unified copy and its tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type coercion turns a literal compared with a column of another type into
a cast: `latency < 100` on a DOUBLE column became
`latency < CAST(100 AS DOUBLE)`, which summary coverage cannot read as a
constant. Fold a cast of a literal to an integer or float type with
DataFusion's `ScalarValue::cast_to`, the evaluation its constant folding
uses; the predicate lowers as `latency < 100.0`. A cast that fails to
evaluate is kept.

DataFusion's whole SimplifyExpressions rule is not run: it also folds
timestamp casts into timestamp literals, which the IR cannot write yet, and
rewrites REGEXP_LIKE into the `~` operator (#650).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
The logical export is phase-free. Physical planning still needs the lifecycle
timing expansion; bring it back unchanged except for carrying coverage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The logical export carries no execution timing. Physical planning output
needs it: reuse the logical payloads and node ids, and add each node's and
edge's data state plus window compatibility, as the earlier post-ASAP export
did. Coverage is carried through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Match the logical export: a batch is one DAG whose roots are its queries,
with shared sub-DAGs exported once. The runtime compiles all roots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s version

DataFusion 54 requires chrono ^0.4.44, so the runtime crate's exact
=0.4.39 pin no longer resolves; keep 0.4.39 as the minimum.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…SAP DAG

Follow #537: the physical DAG now carries each node's operator as
Operator<NodeId> (from ir::flat) instead of the removed wire mirror types.
The runtime matches Operator::NonASAP / Operator::ASAP directly, and
restore() rebuilds the in-memory DAG with map_children instead of a
field-by-field conversion. Node ids are plain usize.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…export

Coverage is derived from the node (#646), so timed copies no longer carry
it and PhysicalASAPDAGNode drops its coverage field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
Cut today's planner over to the unified OperatorNode IR. Frontends return
OperatorNode / QueryRoot, ParsedWorkload keeps scalar roots, Pass 1
(ASAPStrategies), the existing identical-sub-DAG sharing, selection, DAG
assembly and lifecycle costing all run on OperatorNode, and PlanOutput exposes
the whole workload DAG. The native compiler from #541 becomes the canonical
physical_planner and consumes the PhysicalASAPDAG export.

Ported from the earlier #542 (95eef55) without new #509 stage logic. Legacy
QueryExpr/SummaryNode modules stay compiled for their own tests but are no
longer re-exported from post_asap; the cleanup PR removes them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SummaryAgg nodes require SummaryCoverage. Today's planner has no time or
population evidence, so a SummaryAgg it builds declares the whole of the one
source scanned beneath it (no time bound, empty population). The declaration
is trusted, not derived from the scan (#570). Rebuilding the same state over a
re-placed input or with a different grouping strategy keeps its coverage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PlanOutput::execution_timed_dag times every plan root with one shared
TimingMemo and compiles them with compile_physical_asap_workload, so a batch
is one DAG with a root per operator query and shared sub-DAGs exported once.
Lifecycle phases come from the deployments of every plan. The per-plan
SummaryMaintenanceLifecyclePlan::execution_timed_dag is the batch of one.
Standalone scalar roots have no physical form yet and are left out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow #537 and #541: the physical DAG carries Operator<NodeId> instead of
the removed wire mirror types. Port the moved physical_planner, the
planner's lifecycle export and the tests that read or hand-build physical
DAGs; hand-built nodes now name their children by id, matching their edges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Coverage is now checked against the SummaryAgg's subtree, so the planner
can no longer declare an unrestricted population over a filtered scan (for
example a PromQL `{job="worker"}` selector). Build it with
SummaryCoverage::for_summary instead of the removed whole_source_coverage;
time bounds stay unset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Planned summary states no longer declare coverage: it is derived from the
node (#646). Node copies use clone + field updates (the coverage cache is
private), physical DAG fixtures drop the removed coverage field, and the
design example uses with_new_children (#648).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
Lower each example query of the ASAP primitive schema design doc (#573
§4.2.2), plan it with the default cost model and check the coverage of the
selected KLL state: SQL value sets, ranges and renamed columns, a PromQL
offset window as relative time, a matcher below rate kept in the definition,
and a PromQL comparison above rate lifted into the selection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
The planner emits the unified operator IR end to end, so the old pre-ASAP
QueryExpr DAG, its resolver/canonicalizer/CSE, and the post-ASAP
SummaryExpr/PostAsapDAG modules have no live consumers left.

- Move the operator parameter types (GroupKeys, Reduction, Source, ...)
  into ir/operator_properties.rs; pre_asap re-exports them.
- Drop QueryExpr paths from execution_data_state, maintained_population,
  agg_intent, column_resolution, scalar_type_rules and pre_asap/schema.
  with_promql_series_identity keeps only its OperatorNode version in
  ir/schema_support.rs.
- Delete the undeclared unified/ frontend dirs, unified_physical_planner,
  unified_sources, expressions/unified_planner.rs and readout.rs.
- Migrate planner_vocabulary.rs off SchemaResolver; fix the
  scalar_type_rules_fail_closed test name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-applies the documentation half of the earlier legacy cleanup (#543) on
the revised stack, resolving conflicts in favor of the current text where
it is newer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-applies the viewer half of the earlier legacy cleanup (#543): the
viewer categorizes exactly the NonASAPOp/ASAPOp kind names, its fixtures
use the unified export, and devtools/tests/viewer_contract.rs pins the
viewer's category table to every operator variant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 2 materialization (#509) will decide per sub-DAG whether and when to
materialize, so the Planner no longer chooses a maintenance lifecycle.

- MajorPass now runs search_workload_with_targets -> global_selection ->
  assemble_selected_dag per root -> share_common_sub_dags.
- QueryLifecyclePlan becomes QueryPlan { entry_index, root }; PlanOutput's
  execution_timed_dag times the roots directly. LifecycleInput, the
  lifecycle errors and UserInput's `lifecycle` field are gone (public API
  break).
- Delete summary_maintenance_lifecycle, summary_maintenance_cost,
  summary_maintenance_dag_export, post_asap::{summary_maintenance,
  summary_maintenance_lifecycle} and SummaryWindowFramework (the pane
  primitives stay), the lifecycle CostModel hooks, CandidateCostOverrides
  and EmpiricalEvidenceProvider::lifecycle_cost_inputs.
- Delete the lifecycle e2e test and the viewer's lifecycle-plan UI; rewire
  e2e_plan, summary_sharing, operator_design_examples and
  weighted_topk_binding to the new pipeline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol and others added 22 commits October 11, 2026 16:22
Rename LifecycleAssignment to MaterializationAssignment and
apply_lifecycle_timings to apply_materialization_timings. The default
assignment is now all_query_time(): nothing is materialized until Stage 2
materialization (#509) decides per sub-DAG. all_ingestion_time() and set()
assign maintenance explicitly.

- validate_default becomes validate_maintained: candidate legality is still
  checked with every summary maintained, so candidate generation is
  unchanged. planned_data_state and fixed_window_rate_candidates use the
  same maintained assumption, and the maintained precompute compilers in
  promql_rows assign ingestion time explicitly.
- PlanOutput::execution_timed_dag, show_post_asap_ir and the default test
  helpers now emit query-time summaries.
- Tests that exercise maintenance assign it explicitly (new `maintained`
  helpers); a new unit test pins the query-time default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ime default

Remove the lifecycle APIs, recipes and viewer section from the docs, delete
the workload-demand-and-summary-lifecycle proposal, and point materialization
questions to Stage 2 (#509). Rename LifecycleAssignment/apply_lifecycle_timings/
validate_default to their new names and describe the all-query-time default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow #537: the remaining tests read Operator<NodeId> payloads from the
physical DAG, and docs name ir::flat / ir::physical_export instead of the
removed ir::export.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The PromQL frontend's unified module was promoted to the crate root later
in the stack; asap_frontend_promql::unified no longer exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r the export removal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
#509 defines a Stage 1 candidate as one logical ASAP DAG for the whole
workload. #561 lists local alternatives per target but builds no DAG.

compose_logical_candidate(inventory, choice) replaces each chosen target
aggregate with SummaryAgg -> SummaryEstimate (sketch) or SummaryAgg ->
FinalizeExactAccumulator (exact accumulator). Pass-through targets and
untouched sub-DAGs keep their identity. Each SummaryAgg declares trusted
whole-source coverage (#570).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`stage_pipeline --example planner-layering-1 --out <file>` (or
`--promql <q>...`) lowers the workload, runs Pass 1, enumerates the
Cartesian product of local alternatives (capped by --max-candidates and
recorded as `capped`), and writes an asap-stage-pipeline/v1 document with
stage0_logical and stage1_logical_asap only. Generated fixture for #509
Example 1 is committed; a test regenerates it and validates every DAG.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…valuation naming

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…alizes

FinalizeExactAccumulator kept the state column's name (`state`), so a
consumer reading the sample value (e.g. sum by (job) over a finalized
per-series rate) found no `value` column at runtime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…column

Rows that carry the full series identity are closed; Pass 1 rejected them
and the runtime needs explicit item columns for keyed summaries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r naming fix

The finalized exact accumulator now names its column after the aggregate
it realizes (value/sum instead of state). Point the stage_pipeline test at
the renamed fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ges as flat DAGs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
…ction (MVP)

Stage 2 implements one logical candidate physically: exact Aggregate{[TopK]}
becomes Limit(Sort) per group, summaries stay build -> estimate, every node
runs at query time, and the timed roots export as one PhysicalASAPDAG.

Stage 3 is the only stage that computes cost. It rejects candidates whose
summary misses its query's accuracy target (accuracy model, analytical
guarantee) or uses Count-Min over weights not proven non-negative, prices
every valid candidate node by node with analytical_cost (illustrative
statistics), selects the cheapest and reports the rest as costlier.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add stage2_physical_asap (no cost) and stage3_selection (costs, selected,
every other candidate rejected as invalid or costlier) per the v2 contract.
PromQL rows carry the series identity from Stage 0, as runtime state needs.
Regenerate tools/dag-viewer/examples/planner-layering-example1.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the export removal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
Define MVP acceptance for #509 Example 1 before the Phase C stage APIs
exist: 1 -> 6 -> 6 -> 1 candidates (Pass 1 x identical-expression
sharing; physical operator implementation only, no materialization).

- Spec with per-stage candidate tables, invariants and doc ambiguities.
- Integration tests against todo!() stage stubs, ignored until the
  stages land; workload and Stage 0 frontend-shape tests run now.
- Expected-only asap-stage-pipeline/v1 fixture for the DAG viewer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the todo!() stubs with adapters over Stage 0-3, un-ignore the 11
tests that pass, and give each still-ignored test the precise difference
between the implementation and the spec. Add runtime capability checks:
the selected plan compiles in the physical planner; compiling every
candidate stays ignored with the two runtime gaps it finds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the pre_asap/post_asap split with modules named for what they hold:

- ir::operator: node, non_asap, asap, operator_properties, agg_intent,
  maintained_population
- ir::scalar: scalar expressions, expr_ir items, scalar_type_rules,
  column_resolution
- ir::schema: Schema/Field/DataType, aggregate_schema, error, and the
  summary state types (state_type, formerly post_asap::sketch)
- ir::properties: guarantee, timing, summary_coverage, and the execution
  timing/data-state types split out of execution_data_state
- workload: workload, parsed_workload, resources
- physical: the rest of execution_data_state, and summary_window

post_asap::query_time had no callers and is deleted. Public type names are
unchanged; every import across the workspace is rewritten, with no
compatibility re-exports from the old paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/572-b1-types-modules branch from 0babc9d to bd25cf4 Compare October 11, 2026 17:11
@zzylol
zzylol force-pushed the stack/509-c4-example1-acceptance branch from 4f6e4ba to d360107 Compare October 13, 2026 15:32
@zzylol
zzylol force-pushed the stack/509-c4-example1-acceptance branch from d360107 to 074c51a Compare October 13, 2026 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant