Skip to content

feat(planner): declare counter metrics so CMS+heap can prove non-negative weights - #593

Draft
zzylol wants to merge 3 commits into
stack/509-w1-stage3-costfrom
stack/509-w6-counter-proof
Draft

zzylol wants to merge 3 commits into
stack/509-w1-stage3-costfrom
stack/509-w6-counter-proof

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Rebased on main d4869a7 (DF 54).

Wave 1 chain: #594 → #593 → #592 → #591 → #595 → #596 → #597 → #598 → #599 (on #589)

Part of #509 (Example 1), decision X1 from #580's planning: a metric-type declaration plus a non-negativity proof, so Count-Min + heap is valid over raw counter samples.

Why

Example 1's Q2 is topk by (job) (10, sum_over_time(http_requests_total[1m])). Every Count-Min + heap candidate for it, including #589's whole-expression heap over raw samples, was rejected by Stage 3 and by the runtime, because nothing proved the update weights (raw sample values, or their sums) non-negative. Neither the workload nor the catalog carried a metric type, and the _total suffix is only a naming convention.

What

  • Declaration: DataWorkload.metric_types: BTreeMap<String, MetricType> with MetricType::{Counter, Gauge} (Prometheus # TYPE). It defaults to empty.
    • It goes in DataWorkload because that struct already holds the facts about the queried data, the PromQL frontend already reads its ingestion interval, and plan_stages already receives it, so Stage 1 can read it without a new input layer.
    • HistogramCatalog was not used: it is histogram-specific and lowering-only, and the IR has no place to carry the declaration on to Stage 1.
  • Proof: a new NonNegativeWeightProof::CounterSamples variant.
  • Stage 3 and runtime: unchanged. Both already accept Count-Min for any WeightDomain::NonNegative proof, so they apply the same rule.

How

  • plan_stages passes &data.metric_types to stage1_logical_candidates, which passes it to enumerate_local_logical_candidates. This is a plain parameter.
  • Pass 1 records counter_input on each target. It is true when the target's input is samples of a metric declared a counter, reached only through TimeRange and plain Sum aggregates (sum_over_time, sum by).
  • A keyed summary update weighted by SampleValue over such an input gets CounterSamples. This covers both the heap over per-series sums and the whole-expression heap over raw samples.
  • Arithmetic, any other operator, a gauge, or an undeclared metric (whatever its name) gets no proof.
  • Example 1's workload declares http_requests_total a counter in the stage_pipeline devtool and in the acceptance test. The fixture and acceptance doc are regenerated, and the workflow doc lists the field. The integration commit updates Stage 3 per-second cost model, memory pricing and calibration #594's per-second ranking test and the Stage 3 cost model's worked example to P60.

Before / After (Example 1, built-in models, no cost tuning)

Costs are per second (#594): per-evaluation CPU-ms × 0.1 evaluations/s.

Before (#594) After
Count-Min + heap candidates 24 invalid ("not proven non-negative") 24 valid, all compile in the runtime
Selected P58 (all exact, shared input), 5.220/s (52.201 per evaluation) P60 (Q1 exact, Rate and Sum acc; Q2 Count-Min + heap over the exact sum_over_time accumulator; shared input), 4.620/s (46.201 per evaluation)
Q2 cost in the winner, per evaluation 19.001 (Sum acc 5.0 + sort and limit 14.001) 13.001 (Sum acc 5.0 + CMS 1M sums × depth 8 = 8.0 + estimate 0.001)
P58 rank 1st 8th. Ranks 1–7 are shared-input Count-Min + heap plans over per-series sums, 4.620–5.220/s; P51 ties P58
Whole-expression Count-Min + heap invalid ranks 17th–20th; P63 is cheapest at 6.520/s (per evaluation, 4M raw samples × depth 8 = 32.0)
Whole-expression CountSketch + heap P64, 53.720/s unchanged

Without the declaration, or with http_requests_total declared a gauge, the 24 Count-Min + heap candidates stay invalid and the runtime still rejects them. This is today's behavior, and a test covers it.

Tests

  • Pass 1: a declared counter gives CounterSamples on all 6 heap updates. An undeclared m_total, a gauge, or a different declared counter gives none. counter_samples stops at arithmetic (- 100, unary minus).
  • Stage 3: Count-Min over declared counter samples is priced, not rejected.
  • Acceptance: with the declaration, all 64 candidates are valid and compile. With no declaration or a gauge, 24 are invalid and the runtime rejects the same ones.

Gate

Refs #509, #580.

@zzylol
zzylol force-pushed the stack/509-w6-counter-proof branch from f14bf7a to ee6f881 Compare October 4, 2026 18:25
zzylol added a commit that referenced this pull request Oct 4, 2026
Pass 1 takes the declared metric types since #593, and #594's year-long scan
test selects its variant by the Sharing enum instead of a shared flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 4, 2026
Pass 1 takes the declared metric types since #593.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 4, 2026
plan_stages takes per-root demand since #594, and DataWorkload declares metric
types since #593; Example 2's pipeline test runs each query once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol changed the base branch from stack/509-e2-keyed-additive-topk to stack/509-w1-stage3-cost October 4, 2026 18:25
@zzylol
zzylol force-pushed the stack/509-w6-counter-proof branch 2 times, most recently from 761b2ce to f6a38fb Compare October 8, 2026 16:47
@zzylol
zzylol force-pushed the stack/509-w1-stage3-cost branch 2 times, most recently from f2b5562 to 8dd0b2f Compare October 10, 2026 14:32
@zzylol
zzylol force-pushed the stack/509-w6-counter-proof branch from f6a38fb to 6fd1d74 Compare October 10, 2026 14:32
zzylol and others added 3 commits October 11, 2026 16:22
…tive weights

DataWorkload gains `metric_types` (metric name -> Counter | Gauge, the
Prometheus TYPE metadata). plan_stages passes it to Stage 1, and Pass 1
attaches the new `CounterSamples` non-negativity proof to a keyed summary
update weighted by the sample value when the target's input is samples of a
declared counter, through a time range and plain sums (sum_over_time). Any
other operator, a gauge or an undeclared metric (whatever its suffix) gets
no proof. Stage 3 and the runtime already accept any NonNegative proof for
Count-Min, so their rule is unchanged.

Example 1 declares http_requests_total a counter: all 64 candidates are
valid and compile (24 Count-Min + heap were invalid). Stage 3 now selects
P60 (Count-Min + heap over the exact sum_over_time accumulator, shared
input) at 46.201 against P58 (all exact) at 52.201, now 8th.

Refs #509, #580.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Example 1 declares http_requests_total a counter, so Stage 3 selects P60 at
46.201 × 0.1 = 4.620 cost per second. Update the per-second ranking test and
the Stage 3 cost model's worked example to P60.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/509-w6-counter-proof branch from 6fd1d74 to da28717 Compare October 11, 2026 17:11
@zzylol
zzylol force-pushed the stack/509-w1-stage3-cost branch from 8dd0b2f to d89f6d2 Compare October 11, 2026 17:11
zzylol added a commit that referenced this pull request Oct 13, 2026
The tests run Pass 1 and composition, optimize_physical_asap and
select_plan through a shared tests/planner_layering_common pipeline, and
follow the decided behavior: three Pass 1 candidates (Q2 exact, CMS+heap,
CountSketch+heap), every sub-DAG NotMaterialized, P1 selected, CMS+heap
invalid (weights not proven non-negative) and CountSketch+heap invalid (a
Top-K output has no guarantee and no deployment evidence). Hydra (#605),
Pass 2 sharing (#588) and the two runtime gaps (#593, #579) stay ignored
with the PR that delivers them. Tests are named after the stages; the
compile check no longer uses the PromQL fallback. The spec lists where the
doc and the planner differ. The expected-only viewer fixture is removed:
the generated four-stage fixture replaces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
The tests run Pass 1 and composition, optimize_physical_asap and
select_plan through a shared tests/planner_layering_common pipeline, and
follow the decided behavior: three Pass 1 candidates (Q2 exact, CMS+heap,
CountSketch+heap), every sub-DAG NotMaterialized, P1 selected, CMS+heap
invalid (weights not proven non-negative) and CountSketch+heap invalid (a
Top-K output has no guarantee and no deployment evidence). Hydra (#605),
Pass 2 sharing (#588) and the two runtime gaps (#593, #579) stay ignored
with the PR that delivers them. Tests are named after the stages; the
compile check no longer uses the PromQL fallback. The spec lists where the
doc and the planner differ. The expected-only viewer fixture is removed:
the generated four-stage fixture replaces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
…stages (#577)

* test: acceptance spec and pending tests for planner-layering Example 1

Define MVP acceptance for #509 Example 1 before the Phase C stage APIs
exist: 1 -> 6 -> 6 -> 1 candidates (Pass 1 x identical-expression
sharing; physical operator implementation only, no materialization).

- Spec with per-stage candidate tables, invariants and doc ambiguities.
- Integration tests against todo!() stage stubs, ignored until the
  stages land; workload and Stage 0 frontend-shape tests run now.
- Expected-only asap-stage-pipeline/v1 fixture for the DAG viewer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run the Example 1 acceptance tests against the real stages

Replace the todo!() stubs with adapters over Stage 0-3, un-ignore the 11
tests that pass, and give each still-ignored test the precise difference
between the implementation and the spec. Add runtime capability checks:
the selected plan compiles in the physical planner; compiling every
candidate stays ignored with the two runtime gaps it finds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read Example 1 logical stages as flat DAGs after the export removal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 acceptance on the composed stages (1 -> 3 -> 3 -> 1)

The tests run Pass 1 and composition, optimize_physical_asap and
select_plan through a shared tests/planner_layering_common pipeline, and
follow the decided behavior: three Pass 1 candidates (Q2 exact, CMS+heap,
CountSketch+heap), every sub-DAG NotMaterialized, P1 selected, CMS+heap
invalid (weights not proven non-negative) and CountSketch+heap invalid (a
Top-K output has no guarantee and no deployment evidence). Hydra (#605),
Pass 2 sharing (#588) and the two runtime gaps (#593, #579) stay ignored
with the PR that delivers them. Tests are named after the stages; the
compile check no longer uses the PromQL fallback. The spec lists where the
doc and the planner differ. The expected-only viewer fixture is removed:
the generated four-stage fixture replaces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: port #590's Example 2-4 acceptance specs and tests

The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* feat(devtools): stage_pipeline examples for #509 Example 3

Ported from #590: --example planner-layering-3a (Pattern A, ad hoc batch of
five p99 reports) and planner-layering-3b (Pattern B, repeating 5-min p99),
the Example 3 acceptance tests' workloads. Example 1's output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: follow #576's per-consumer charging and exact top-k columns

Plan selection charges a NotMaterialized node once per consuming query,
so the charging helper checks NodeCost.multiplier and the Example 4 A3
ambiguity is recorded as decided. Example 1 checks that exact Q2 sorts by
the ranked value and returns [job, topk_10]. The stage viewer guide's
tab 1 row now describes the actual all-exact result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(devtools): one planning pipeline for stage_pipeline and the acceptance tests

run_planning_stages and the #509 example workloads move into the
asap-devtools library; stage_pipeline serializes the run and the
planner-layering tests adapt it. The shared data workload drops
ingestion_rate for data at rest, which validate() requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(devtools): select over every candidate; --max-displayed limits only the document

stage_pipeline capped enumeration at 64, so Example 3a (243 candidates)
selected P41 instead of the cheapest P122 (KLL for all five queries).
The pipeline now enumerates, optimizes and prices every candidate; the
document shows the first --max-displayed (default 64) plus the selected
one, with `combinations` and `capped`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: Example 2's SQL candidates compose; the SQL test waits only on #598

Running the ignored SQL test composes every candidate and fails only on
Q2's options (per-source counts until #598), so #603 is not a blocker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: name stages by number and name; say query-language frontends

The frontends' Run field, id and locals become stage0_query_language_frontends,
and test names carry stage0..stage3 with the stage name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: charging helper follows plan selection's per-consumer rule

With a materialized node the helper accepted any multiplier >= 1. It now
computes each node's computations as plan selection does (walks from
query roots and materialized nodes' inputs, stopping at materialized
nodes), checked on a hand-built candidate with a materialized node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 picks Q2 candidates by (option, no shared input)

physical_with and the invalid-candidate check no longer assume one
candidate per Q2 option, so #588's shared-input variants need no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: Example 3 Pattern B charges each node once per consuming query

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs(stage-viewer): rows 2, 3a and 3b describe the planner's current output

Read from uncapped runs: Example 2's sketch plans are rejected for having
no (ε, δ) guarantee, 3a selects P122 (first of 108 tied cheapest plans,
nothing shared yet) and 3b selects query-time KLL; panes and the latency
check come with the window-summary PRs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs(stage-viewer): row 2 from the SQL run; 3a ties 32 plans, not 108

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 counts only unshared candidates; stage3_ charging-helper names

The 3-logical and 3-physical counts now hold when #588 adds shared-input
variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(devtools): PlanningStagesRun fields carry stage number and name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test(devtools): unique temp file per stage_pipeline run

cargo test runs the tests as threads of one process, so two runs of the same
example shared a file name and one deleted the other's output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant