test(promql): enforce range step parity in production E2E by zzylol · Pull Request #509 · ProjectASAP/ASAPQuery-backend

zzylol · 2026-09-04T12:00:58Z

Addresses #487 and is stacked on #505 because that branch supplies the canonical collector-free warm matrix used for the regression.

For every supported PromQL family, the production-process test now:

asserts the exact requested range timestamps and step count;
independently calls /api/v1/query at each requested timestamp;
compares labels and values at each range step with the corresponding instant evaluation;
retains warm provenance and no-fallback assertions.

This closes the detection gap that allowed native pane-close timestamps or ignored step_ms to pass.

Validation: cargo test -p data_plane --test asapquery_compatibility_process_e2e -- --nocapture (1 passed).

Intentionally left open and unmerged.

…ecution stack The backend now executes ASAPPlanner's selected PhysicalASAPDAG as given (#509): the Planner runs Stages 0-3, the backend supplies deployment inputs. - Pin every Planner crate the backend uses to ASAPPlanner main (58999df2): asap-types, asap-frontend-promql, asap-logical-asap-optimization, asap-plan-selection, asap-executor. The sync script updates all five. - control_plane: lower a PromQL PlanningWorkload, run Stages 1-3 (run_stage1_to_stage3) under DeploymentInputs (DeploymentCapabilities and PlanSelectionSettings), and write the selected candidate's DAG, with each PaneChain expanded into its panes, plus the query each root answers (plan.json). - data_plane: load and validate plan.json, accept Remote Write, forward queries to Prometheus until the query engine lands. - Remove the backend's own plan selection, placement pricing, ERP, SDS, precompute and query engines, ClickHouse, OTel and OpAMP paths, which were built on the removed Planner APIs.

…puts Each MatAtIngestion output the query side reads (the executor's materialization frontier) reads one tumbling pane of raw samples. Outputs that differ only in which pane they read share one SummaryDefinition, identified by a SHA-256 of their sub-DAG with node ids blanked and offset TimeShifts skipped. Each output binds to its definition with an offset: at evaluation time T it reads the pane ending at T - offset. SummaryStore keeps built panes by definition and end. A plan with a node kept across evaluations is rejected: the backend declares it cannot keep one. #509 Example 4b on a deployment without raw data: its six 10-minute KLL panes become one definition with offsets 0..50 min.

For Examples 1, 3a, 3b, 4a and 4b: plan the workload, start the data_plane binary against a Prometheus holding generated samples, Remote Write the same samples, and require every planned query to be answered by the plan with the Planner executor's result for the selected logical candidate with nothing materialized (control_plane::plan_selected_at_query_time). 3b also runs, and 4b only runs, on a deployment without raw data, so their panes are built from Remote Write. Exact results must match exactly; Example 3a/4a's five-year KLL windows exceed k and KLL compaction is randomized, so they must agree within twice the 0.005 rank target.

…ld execution stack (#818) refactor!: plan with ASAPPlanner's stage pipeline and drop the old execution stack The backend now executes ASAPPlanner's selected PhysicalASAPDAG as given (#509): the Planner runs Stages 0-3, the backend supplies deployment inputs. - Pin every Planner crate the backend uses to ASAPPlanner main (58999df2): asap-types, asap-frontend-promql, asap-logical-asap-optimization, asap-plan-selection, asap-executor. The sync script updates all five. - control_plane: lower a PromQL PlanningWorkload, run Stages 1-3 (run_stage1_to_stage3) under DeploymentInputs (DeploymentCapabilities and PlanSelectionSettings), and write the selected candidate's DAG, with each PaneChain expanded into its panes, plus the query each root answers (plan.json). - data_plane: load and validate plan.json, accept Remote Write, forward queries to Prometheus until the query engine lands. - Remove the backend's own plan selection, placement pricing, ERP, SDS, precompute and query engines, ClickHouse, OTel and OpAMP paths, which were built on the removed Planner APIs. fix(ci): drop the unused asap-executor pin so the Planner sync succeeds cargo update -p asap-executor exits 101 because no workspace member depends on the crate, so it is absent from Cargo.lock. The sync test's fake cargo now rejects -p specs missing from the real Cargo.lock and runs the script against the repository manifest. fix(data_plane): accept Remote Write up to its limit and forward query headers - Raise the write route's body limit from axum's 2 MB default to Limits::max_compressed_bytes; larger bodies get 413. - Forward end-to-end request headers (Authorization, X-Scope-OrgID, ...), dropping hop-by-hop headers, Host, Content-Length and Accept-Encoding. - Bound forwarded queries with --forward-timeout-seconds (default 120). - State in the README and --prometheus-url help that Remote Write samples are counted and dropped until the precompute engine lands. fix(control_plane): reject a root whose series-identity rewrite fails The rewrite failure was ignored and the root planned without the row identity per-series state needs at runtime. Return it as PlanError::SeriesIdentity, as the Planner's executor lowering does. chore: remove references to deleted files and stale CI text - Delete .dockerignore: no Dockerfile remains. - Drop the hadolint hook and .gitignore entries for removed outputs; ignore __pycache__ from the updater tests. - scripts/ci/README.md: describe what MVP CI runs now and drop the outdated PR reference.

…puts Each MatAtIngestion output the query side reads (the executor's materialization frontier) reads one tumbling pane of raw samples. Outputs that differ only in which pane they read share one SummaryDefinition, identified by a SHA-256 of their sub-DAG with node ids blanked and offset TimeShifts skipped. Each output binds to its definition with an offset: at evaluation time T it reads the pane ending at T - offset. SummaryStore keeps built panes by definition and end. A plan with a node kept across evaluations is rejected: the backend declares it cannot keep one. data_plane depends on asap-executor again, so the workspace pins it at the same ASAPPlanner revision as the other four crates and the Planner sync updates all five. The tests find the frontier with the executor's test-support feature, as the Planner's own tests do. #509 Example 4b on a deployment without raw data: its six 10-minute KLL panes become one definition with offsets 0..50 min.

…puts (#819) feat(sds): define stored summaries from the plan's ingestion-time outputs Each MatAtIngestion output the query side reads (the executor's materialization frontier) reads one tumbling pane of raw samples. Outputs that differ only in which pane they read share one SummaryDefinition, identified by a SHA-256 of their sub-DAG with node ids blanked and offset TimeShifts skipped. Each output binds to its definition with an offset: at evaluation time T it reads the pane ending at T - offset. SummaryStore keeps built panes by definition and end. A plan with a node kept across evaluations is rejected: the backend declares it cannot keep one. data_plane depends on asap-executor again, so the workspace pins it at the same ASAPPlanner revision as the other four crates and the Planner sync updates all five. The tests find the frontier with the executor's test-support feature, as the Planner's own tests do. #509 Example 4b on a deployment without raw data: its six 10-minute KLL panes become one definition with offsets 0..50 min.

…e closed A stored output the plan builds in a query run and keeps (StoredOutputBuild::QueryRun, #509 Example 4's B3) is now executed. The query engine builds its definition's summary from the coverage's raw samples the first time an evaluation needs it, keeps it in memory by (definition, coverage end) once the coverage is closed (its end at or before the precompute engine's watermark), reuses it in later evaluations, and evicts it by the plan's retention_ms measured from the newest evaluation. A summary read while its coverage is still open is built for that evaluation only, so a later one includes late samples. SummaryBuild, shared by the precompute and query engines, compiles a definition's build and runs it over one coverage's samples. Samples of a metric only query-run summaries read move the watermark (counted as samples_query_run_only) without being buffered. The backend now declares query_time_results_retained: true. examples/planner-layering-example4-b3.json is a workload for which the Planner selects B3 under the default cost model.

…e closed A stored output the plan builds in a query run and keeps (StoredOutputBuild::QueryRun, #509 Example 4's B3) is now executed. The query engine builds its definition's summary from the coverage's raw samples the first time an evaluation needs it, and stores it in the SDS store under the same SummaryInstanceKey as ingestion-built summaries once its coverage is closed. Later evaluations reuse it. - Closure is judged by the ingestion progress snapshotted when the evaluation chose its fetch (raw_ranges returns it with the ranges and evaluate takes it back), not when the evaluation runs, so a partial summary built from an earlier fetch is never kept. - Eviction is by the plan's retention_ms plus one pane, measured from the newest closed coverage at or before the watermark, as for ingestion-built summaries. A far-future evaluation no longer stops keeping. - One fetch per query-run definition covers all its missing summaries. - KeptResultStats (built, kept, kept_while_metric_behind, reused) is on /api/v1/status/asap. kept_while_metric_behind counts summaries kept while their own metric's newest sample had not reached their end: the watermark is global. - A query-run definition whose selector does not build no longer fails the install: its queries are forwarded. SummaryBuild, shared by the precompute and query engines, compiles a definition's build and runs it over one coverage's samples. Samples of a metric only query-run summaries read move the watermark (counted as samples_query_run_only) without being buffered. The backend now declares query_time_results_retained: true. examples/planner-layering-example4-b3.json is a workload for which the Planner selects B3 under the default cost model.

A stored output with StoredOutputBuild::QueryRun (materialization MatAtQueryKeptAcrossEvaluations, #509 Example 4's B3) is now executed. The query engine builds its summary state from the pane's raw samples the first time an evaluation needs it, and stores it in the SDS under the same SummaryInstanceKey as stored outputs built in an IngestionRun once the pane is closed. Later evaluations reuse it. - Closure is judged by the ingestion progress snapshotted when the evaluation chose its fetch (raw_ranges returns it with the ranges and evaluate takes it back), not when the evaluation runs, so a summary state built from an earlier, partial fetch is never stored. - Eviction is by the plan's retention_ms plus one pane, measured from the newest closed pane at or before the watermark, as for IngestionRun outputs. A far-future evaluation no longer stops storing. - One fetch per QueryRun definition covers all its missing panes. - QueryRunStats (built, stored, stored_while_metric_behind, reused) is on /api/v1/status/asap under "query_run". stored_while_metric_behind counts summary states stored while their own metric's newest sample had not reached their end: the watermark is global. - A QueryRun definition whose selector does not build no longer fails the install: its queries are forwarded. SummaryBuild, shared by the precompute and query engines, compiles a definition's build and runs it over one pane's samples. Samples of a metric only QueryRun stored outputs read move the watermark (counted as samples_query_run_only) without being buffered. The backend now declares query_time_results_retained: true, and the startup warning about unbuilt QueryRun outputs is gone. examples/planner-layering-example4-b3.json is a workload for which the Planner selects B3 under the default cost model.

A stored output with StoredOutputBuild::QueryRun (materialization MatAtQueryKeptAcrossEvaluations, #509 Example 4's B3) is now executed. The query engine builds its summary state from the pane's raw samples the first time an evaluation needs it, and stores it in the SDS under the same SummaryInstanceKey as stored outputs built in an IngestionRun once the pane is closed. Later evaluations reuse it. - Closure is judged by the ingestion progress snapshotted when the evaluation chose its fetch (bound_raw_sample_windows returns it with the windows and evaluate takes it back), not when the evaluation runs, so a summary state built from an earlier, partial fetch is never stored. - Eviction is by the plan's retention_ms plus one pane, measured from the newest closed pane at or before the watermark, as for IngestionRun outputs. A far-future evaluation no longer stops storing. - One fetch per QueryRun definition covers all its missing panes. - QueryRunStats (built, stored, stored_while_metric_behind, reused) is on /api/v1/status/asap under "query_run". stored_while_metric_behind counts summary states stored while their own metric's newest sample had not reached their end: the watermark is global. - A QueryRun definition whose selector does not build no longer fails the install: its queries are forwarded. SummaryBuild, shared by the precompute and query engines, compiles a definition's build and runs it over one pane's samples. Samples of a metric only QueryRun stored outputs read move the watermark (counted as samples_query_run_only) without being buffered. The backend now declares query_time_results_retained: true, and the startup warning about unbuilt QueryRun outputs is gone. examples/planner-layering-example4-b3.json is a workload for which the Planner selects B3 under the default cost model.

) feat(query): build stored outputs built in a QueryRun on first use A stored output with StoredOutputBuild::QueryRun (materialization MatAtQueryKeptAcrossEvaluations, #509 Example 4's B3) is now executed. The query engine builds its summary state from the pane's raw samples the first time an evaluation needs it, and stores it in the SDS under the same SummaryInstanceKey as stored outputs built in an IngestionRun once the pane is closed. Later evaluations reuse it. - Closure is judged by the ingestion progress snapshotted when the evaluation chose its fetch (bound_raw_sample_windows returns it with the windows and evaluate takes it back), not when the evaluation runs, so a summary state built from an earlier, partial fetch is never stored. - Eviction is by the plan's retention_ms plus one pane, measured from the newest closed pane at or before the watermark, as for IngestionRun outputs. A far-future evaluation no longer stops storing. - One fetch per QueryRun definition covers all its missing panes. - QueryRunStats (built, stored, stored_while_metric_behind, reused) is on /api/v1/status/asap under "query_run". stored_while_metric_behind counts summary states stored while their own metric's newest sample had not reached their end: the watermark is global. - A QueryRun definition whose selector does not build no longer fails the install: its queries are forwarded. SummaryBuild, shared by the precompute and query engines, compiles a definition's build and runs it over one pane's samples. Samples of a metric only QueryRun stored outputs read move the watermark (counted as samples_query_run_only) without being buffered. The backend now declares query_time_results_retained: true, and the startup warning about unbuilt QueryRun outputs is gone. examples/planner-layering-example4-b3.json is a workload for which the Planner selects B3 under the default cost model.

test(promql): compare every range step with instant evaluation

4139e58

zzylol mentioned this pull request Sep 4, 2026

PromQL range warm path ignores step_ms and differential E2E does not detect it #487

Open

zzylol added 3 commits September 8, 2026 07:06

Merge current typed TopK and reviewed workload migration

7b2e8a8

test: compare complete range vectors including changing TopK membership

370cabe

Merge exact-only quote preparation fix

8f85a26

zzylol mentioned this pull request Sep 8, 2026

docs: ASAPPlanner and ASAPQuery-backend integration architecture #512

Merged

zzylol changed the base branch from feat/501-promql-compatibility to main September 8, 2026 20:19

zzylol merged commit d83f36a into main Sep 8, 2026
1 check passed

zzylol deleted the test/487-range-step-parity branch September 21, 2026 18:43

zzylol mentioned this pull request Oct 18, 2026

feat(query): build stored outputs built in a QueryRun on first use #824

Merged

Navigation Menu

Repository navigation

test(promql): enforce range step parity in production E2E - #509

test(promql): enforce range step parity in production E2E#509
zzylol merged 4 commits into
mainfrom
test/487-range-step-parity

zzylol commented Sep 4, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

zzylol commented Sep 4, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant