Repository navigation
Conversation
zzylol
force-pushed
the
stack/509-c4-example1-acceptance
branch
from
October 5, 2026 06:21
0afc939 to
f620555
Compare
zzylol
force-pushed
the
stack/572-b1-types-modules
branch
from
October 5, 2026 06:21
dfaa958 to
7eda06e
Compare
zzylol
force-pushed
the
stack/509-c4-example1-acceptance
branch
from
October 8, 2026 16:47
f620555 to
b8ad78c
Compare
zzylol
force-pushed
the
stack/572-b1-types-modules
branch
2 times, most recently
from
October 10, 2026 14:32
078adb8 to
0babc9d
Compare
zzylol
force-pushed
the
stack/509-c4-example1-acceptance
branch
from
October 10, 2026 14:32
b8ad78c to
0541f9a
Compare
Port DF 54's API changes (Expr::Literal metadata, Cast/TryCast field, ScalarUDFImpl return_field_from_args/invoke_with_args, catalog path, dialect name) and the planner behaviour changes (SELECT * keeps its projection, GenericDialect parses aggregate FILTER) from main's crates/frontend-sql/src/sql onto the unified copy and its tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type coercion turns a literal compared with a column of another type into a cast: `latency < 100` on a DOUBLE column became `latency < CAST(100 AS DOUBLE)`, which summary coverage cannot read as a constant. Fold a cast of a literal to an integer or float type with DataFusion's `ScalarValue::cast_to`, the evaluation its constant folding uses; the predicate lowers as `latency < 100.0`. A cast that fails to evaluate is kept. DataFusion's whole SimplifyExpressions rule is not run: it also folds timestamp casts into timestamp literals, which the IR cannot write yet, and rewrites REGEXP_LIKE into the `~` operator (#650). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
The logical export is phase-free. Physical planning still needs the lifecycle timing expansion; bring it back unchanged except for carrying coverage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The logical export carries no execution timing. Physical planning output needs it: reuse the logical payloads and node ids, and add each node's and edge's data state plus window compatibility, as the earlier post-ASAP export did. Coverage is carried through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Match the logical export: a batch is one DAG whose roots are its queries, with shared sub-DAGs exported once. The runtime compiles all roots. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s version DataFusion 54 requires chrono ^0.4.44, so the runtime crate's exact =0.4.39 pin no longer resolves; keep 0.4.39 as the minimum. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…SAP DAG Follow #537: the physical DAG now carries each node's operator as Operator<NodeId> (from ir::flat) instead of the removed wire mirror types. The runtime matches Operator::NonASAP / Operator::ASAP directly, and restore() rebuilds the in-memory DAG with map_children instead of a field-by-field conversion. Node ids are plain usize. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…export Coverage is derived from the node (#646), so timed copies no longer carry it and PhysicalASAPDAGNode drops its coverage field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
Cut today's planner over to the unified OperatorNode IR. Frontends return OperatorNode / QueryRoot, ParsedWorkload keeps scalar roots, Pass 1 (ASAPStrategies), the existing identical-sub-DAG sharing, selection, DAG assembly and lifecycle costing all run on OperatorNode, and PlanOutput exposes the whole workload DAG. The native compiler from #541 becomes the canonical physical_planner and consumes the PhysicalASAPDAG export. Ported from the earlier #542 (95eef55) without new #509 stage logic. Legacy QueryExpr/SummaryNode modules stay compiled for their own tests but are no longer re-exported from post_asap; the cleanup PR removes them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SummaryAgg nodes require SummaryCoverage. Today's planner has no time or population evidence, so a SummaryAgg it builds declares the whole of the one source scanned beneath it (no time bound, empty population). The declaration is trusted, not derived from the scan (#570). Rebuilding the same state over a re-placed input or with a different grouping strategy keeps its coverage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PlanOutput::execution_timed_dag times every plan root with one shared TimingMemo and compiles them with compile_physical_asap_workload, so a batch is one DAG with a root per operator query and shared sub-DAGs exported once. Lifecycle phases come from the deployments of every plan. The per-plan SummaryMaintenanceLifecyclePlan::execution_timed_dag is the batch of one. Standalone scalar roots have no physical form yet and are left out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow #537 and #541: the physical DAG carries Operator<NodeId> instead of the removed wire mirror types. Port the moved physical_planner, the planner's lifecycle export and the tests that read or hand-build physical DAGs; hand-built nodes now name their children by id, matching their edges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Coverage is now checked against the SummaryAgg's subtree, so the planner
can no longer declare an unrestricted population over a filtered scan (for
example a PromQL `{job="worker"}` selector). Build it with
SummaryCoverage::for_summary instead of the removed whole_source_coverage;
time bounds stay unset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Planned summary states no longer declare coverage: it is derived from the node (#646). Node copies use clone + field updates (the coverage cache is private), physical DAG fixtures drop the removed coverage field, and the design example uses with_new_children (#648). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
Lower each example query of the ASAP primitive schema design doc (#573 §4.2.2), plan it with the default cost model and check the coverage of the selected KLL state: SQL value sets, ranges and renamed columns, a PromQL offset window as relative time, a matcher below rate kept in the definition, and a PromQL comparison above rate lifted into the selection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
The planner emits the unified operator IR end to end, so the old pre-ASAP QueryExpr DAG, its resolver/canonicalizer/CSE, and the post-ASAP SummaryExpr/PostAsapDAG modules have no live consumers left. - Move the operator parameter types (GroupKeys, Reduction, Source, ...) into ir/operator_properties.rs; pre_asap re-exports them. - Drop QueryExpr paths from execution_data_state, maintained_population, agg_intent, column_resolution, scalar_type_rules and pre_asap/schema. with_promql_series_identity keeps only its OperatorNode version in ir/schema_support.rs. - Delete the undeclared unified/ frontend dirs, unified_physical_planner, unified_sources, expressions/unified_planner.rs and readout.rs. - Migrate planner_vocabulary.rs off SchemaResolver; fix the scalar_type_rules_fail_closed test name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-applies the documentation half of the earlier legacy cleanup (#543) on the revised stack, resolving conflicts in favor of the current text where it is newer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-applies the viewer half of the earlier legacy cleanup (#543): the viewer categorizes exactly the NonASAPOp/ASAPOp kind names, its fixtures use the unified export, and devtools/tests/viewer_contract.rs pins the viewer's category table to every operator variant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 2 materialization (#509) will decide per sub-DAG whether and when to materialize, so the Planner no longer chooses a maintenance lifecycle. - MajorPass now runs search_workload_with_targets -> global_selection -> assemble_selected_dag per root -> share_common_sub_dags. - QueryLifecyclePlan becomes QueryPlan { entry_index, root }; PlanOutput's execution_timed_dag times the roots directly. LifecycleInput, the lifecycle errors and UserInput's `lifecycle` field are gone (public API break). - Delete summary_maintenance_lifecycle, summary_maintenance_cost, summary_maintenance_dag_export, post_asap::{summary_maintenance, summary_maintenance_lifecycle} and SummaryWindowFramework (the pane primitives stay), the lifecycle CostModel hooks, CandidateCostOverrides and EmpiricalEvidenceProvider::lifecycle_cost_inputs. - Delete the lifecycle e2e test and the viewer's lifecycle-plan UI; rewire e2e_plan, summary_sharing, operator_design_examples and weighted_topk_binding to the new pipeline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rename LifecycleAssignment to MaterializationAssignment and apply_lifecycle_timings to apply_materialization_timings. The default assignment is now all_query_time(): nothing is materialized until Stage 2 materialization (#509) decides per sub-DAG. all_ingestion_time() and set() assign maintenance explicitly. - validate_default becomes validate_maintained: candidate legality is still checked with every summary maintained, so candidate generation is unchanged. planned_data_state and fixed_window_rate_candidates use the same maintained assumption, and the maintained precompute compilers in promql_rows assign ingestion time explicitly. - PlanOutput::execution_timed_dag, show_post_asap_ir and the default test helpers now emit query-time summaries. - Tests that exercise maintenance assign it explicitly (new `maintained` helpers); a new unit test pins the query-time default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ime default Remove the lifecycle APIs, recipes and viewer section from the docs, delete the workload-demand-and-summary-lifecycle proposal, and point materialization questions to Stage 2 (#509). Rename LifecycleAssignment/apply_lifecycle_timings/ validate_default to their new names and describe the all-query-time default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow #537: the remaining tests read Operator<NodeId> payloads from the physical DAG, and docs name ir::flat / ir::physical_export instead of the removed ir::export. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The PromQL frontend's unified module was promoted to the crate root later in the stack; asap_frontend_promql::unified no longer exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r the export removal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
#509 defines a Stage 1 candidate as one logical ASAP DAG for the whole workload. #561 lists local alternatives per target but builds no DAG. compose_logical_candidate(inventory, choice) replaces each chosen target aggregate with SummaryAgg -> SummaryEstimate (sketch) or SummaryAgg -> FinalizeExactAccumulator (exact accumulator). Pass-through targets and untouched sub-DAGs keep their identity. Each SummaryAgg declares trusted whole-source coverage (#570). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`stage_pipeline --example planner-layering-1 --out <file>` (or `--promql <q>...`) lowers the workload, runs Pass 1, enumerates the Cartesian product of local alternatives (capped by --max-candidates and recorded as `capped`), and writes an asap-stage-pipeline/v1 document with stage0_logical and stage1_logical_asap only. Generated fixture for #509 Example 1 is committed; a test regenerates it and validates every DAG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…valuation naming Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…alizes FinalizeExactAccumulator kept the state column's name (`state`), so a consumer reading the sample value (e.g. sum by (job) over a finalized per-series rate) found no `value` column at runtime. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…column Rows that carry the full series identity are closed; Pass 1 rejected them and the runtime needs explicit item columns for keyed summaries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r naming fix The finalized exact accumulator now names its column after the aggregate it realizes (value/sum instead of state). Point the stage_pipeline test at the renamed fixture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ges as flat DAGs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
…ction (MVP)
Stage 2 implements one logical candidate physically: exact Aggregate{[TopK]}
becomes Limit(Sort) per group, summaries stay build -> estimate, every node
runs at query time, and the timed roots export as one PhysicalASAPDAG.
Stage 3 is the only stage that computes cost. It rejects candidates whose
summary misses its query's accuracy target (accuracy model, analytical
guarantee) or uses Count-Min over weights not proven non-negative, prices
every valid candidate node by node with analytical_cost (illustrative
statistics), selects the cheapest and reports the rest as costlier.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add stage2_physical_asap (no cost) and stage3_selection (costs, selected, every other candidate rejected as invalid or costlier) per the v2 contract. PromQL rows carry the series identity from Stage 0, as runtime state needs. Regenerate tools/dag-viewer/examples/planner-layering-example1.json. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the export removal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
Define MVP acceptance for #509 Example 1 before the Phase C stage APIs exist: 1 -> 6 -> 6 -> 1 candidates (Pass 1 x identical-expression sharing; physical operator implementation only, no materialization). - Spec with per-stage candidate tables, invariants and doc ambiguities. - Integration tests against todo!() stage stubs, ignored until the stages land; workload and Stage 0 frontend-shape tests run now. - Expected-only asap-stage-pipeline/v1 fixture for the DAG viewer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the todo!() stubs with adapters over Stage 0-3, un-ignore the 11 tests that pass, and give each still-ignored test the precise difference between the implementation and the spec. Add runtime capability checks: the selected plan compiles in the physical planner; compiling every candidate stays ignored with the two runtime gaps it finds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oval Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
Replace the pre_asap/post_asap split with modules named for what they hold: - ir::operator: node, non_asap, asap, operator_properties, agg_intent, maintained_population - ir::scalar: scalar expressions, expr_ir items, scalar_type_rules, column_resolution - ir::schema: Schema/Field/DataType, aggregate_schema, error, and the summary state types (state_type, formerly post_asap::sketch) - ir::properties: guarantee, timing, summary_coverage, and the execution timing/data-state types split out of execution_data_state - workload: workload, parsed_workload, resources - physical: the rest of execution_data_state, and summary_window post_asap::query_time had no callers and is deleted. Public type names are unchanged; every import across the workspace is rewritten, with no compatibility re-exports from the old paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
force-pushed
the
stack/572-b1-types-modules
branch
from
October 11, 2026 17:11
0babc9d to
bd25cf4
Compare
zzylol
force-pushed
the
stack/509-c4-example1-acceptance
branch
from
October 13, 2026 15:32
4f6e4ba to
d360107
Compare
3 tasks
zzylol
force-pushed
the
stack/509-c4-example1-acceptance
branch
from
October 13, 2026 15:34
d360107 to
074c51a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased on main d4869a7 (DF 54).
Step B1 of Phase B of #572 (tracker #528): arrange
asap-typesby #511 section instead of the pre-ASAP / post-ASAP split.Why
pre_asapandpost_asapno longer describe their contents. There is one operator DAG (#511),pre_asap::schemais the schema of every DAG, andpost_asap::sketchis shared vocabulary that the logical stage uses. The later Phase B crates (logical-optimizer, physical-optimizer, plan-selection, executor) should import types by what they are, not by a stage name that no longer exists.What
Before this PR
After this PR
post_asap::query_timehad no callers and is deleted, along with its 20 unit tests.mod.rsre-exports the vocabulary thatpre_asap/post_asapused to re-export, sopre_asap::AggIntentbecomesir::operator::AggIntent,post_asap::SketchAlgorithmbecomesir::schema::SketchAlgorithm, and so on. Theirroot keeps its existing re-exports (OperatorNode,ASAPOp, the timing functions, …).lib.rscrate docs and the docs that cite module or file paths are updated.How
git mvfor every file, so history follows the moves. Similarity is 88–100 %.usetree rooted atasap_types,planner_types/planner(the alias inasap-physical-operators) orcrate(insideasap-types), maps each leaf path to its new module, and regroups the result. It does the same for inline qualified paths. Thencargo checkiteration andcargo fmt..rs). About 580usestatements rewritten, plus about 345 inline qualified paths.#[test]count goes from 1793 to 1773, and the 20 removed are exactly the deletedquery_timetests. No other test is changed beyond its import paths.Gate at this tip:
cargo fmt --all --check,cargo clippy --workspace --all-targets --all-features -- -D warnings,cargo test --workspace(1518 passed, 0 failed, 12 ignored) andpython3 -m unittest discover -s tools/dag-viewer -p test_render.py(29 run, OK, 6 skipped) all pass.Deviations from #572, for review
ExecutionDataStateErrorandDataPrimitivego toir::propertiestogether withExecutionTiming/ExecutionDataState, not tophysical. The reason is the one Reorganize crates and modules by #509 stages and the #511 unified IR #572's update gives for the split:ir/timingandir/physical_exportuse them. What stays inphysical::execution_data_stateislift_plainandExactOperationSchemaError. Their only callers today are Stage 1 (exact_composition), sophysicalmay not be their final home.expr_ir.rsis now a private file ofir::scalar, and its four items are re-exported there. This avoids anir::scalar::expr_irpath that keeps the legacy name.ir::schema_support(PromQL series-identity realization) stays at theirroot, andcost,dag_export,serde_f64,typesstay at the crate root.tools/dag-viewer/post_asap_fixture.jsonstill exists but nothing references it, so it is not renamed.PostAsapDAG/pre_asap_sub_dagare gone from code.PostAsapDAGis still mentioned in prose indocs/design_docs/concepts/post-asap-ir.md,docs/design_docs/physical-planning-and-deployment.mdanddocs/develop_docs/library-api.md, which this PR leaves alone.docs/develop_docs/*-migration.mdkeep their historicalpre_asap::paths.CostModeltrait and recurrence types intoasap-types) is not in this stack, and this PR is based directly on test: planner-layering Examples 1–4 acceptance specs on the planning stages #577. B0 cannot be done as a types-only move:ReplacementSubDAG,TargetSubDAG,Replacement,Realization,ExactComposition,ExactOperationandOperationPlacement. For example,ExactComposition::compose_with_accuracyusesAccuracyModel, andOperationPlacement::provenancereturnsReplacementProvenance.size_paramscallsaccuracy::estimators::size_params, andcse_share_decision_with_recurrencecallsrecurrence::decide.Refs #572 · #528
🤖 Generated with Claude Code