Skip to content

fix(planner): SQL COUNT(*) candidates count rows, not a sample value - #603

Draft
zzylol wants to merge 2 commits into
stack/509-x5b-example4-acceptancefrom
stack/509-x9-pass1-coverage
Draft

zzylol wants to merge 2 commits into
stack/509-x5b-example4-acceptancefrom
stack/509-x9-pass1-coverage

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Stack: Wave 2 chain: #599 → #601 → #600 → #604 → #606 → #603 → #605

Rebased on main d4869a7 (DF 54).

Why

Part of #509 and #580 (Stage 1 coverage). PR #590's acceptance work found that on SQL flows(ts, src_ip), Stage 1 candidates for COUNT(*) … GROUP BY src_ip fail to compose. Pass 1 builds every count update from ColumnRef::SampleValue, the implicit PromQL value column, and SQL rows don't have one. W7b saw the same failure in Stage 2 for exact Count accumulators.

What

#509 Example 2's design queries (distinct, entropy, and the integer L2 that keeps COUNT(*) GROUP BY src_ip as a target), through plan_stages with every candidate displayed:

Stage 1 build failures selected
Before this PR 240 of 400 (ColumnRef::SampleValue has no value column) P5: UnivMon over a non-existent value item. It passes Stages 2–3 but cannot bind.
After this PR 0 of 400 P2: exact Count accumulator, which executes

How

  • logical_candidates::summary_update: on SQL rows (a closed schema without PromQL series identity), COUNT(*) adds Constant(1.0) per row and carries the UnitCount proof.
    • The exact accumulator reads no column.
    • A sketch hashes one non-null column, a grouping column first. The bare count it answers ignores the item's value.
    • PromQL updates are unchanged.
  • Executor: Operator::unit_count_build is an exact count with no value column, so Constant(1.0) binds. SummaryBuild.value becomes Option<usize>.
  • Legacy origin: the same rule as realize_value_frequency_summary_input in pass1/replacement.rs, which uses a unit weight.

Tests

I wrote tests/pass1_sql_coverage.rs first. Both tests failed on the parent, and both pass now:

  • sql_count_star_group_by_candidates_compose_and_execute: every candidate composes and compiles. Pass-through and the Count accumulator execute to a=3, b=2, c=1.
  • example2_design_candidates_all_build: no Stage 1 or Stage 2 build failure.

Not in this PR

Count-Min, CountSketch and UnivMon count candidates still compile but do not bind. The executor's keyed build needs a weight column, so it rejects Constant(1.0), and PromQL count sketches hit the same limit. Stage 3 never selects them here.

Gates

  • cargo fmt --all --check and cargo clippy --workspace --all-targets --all-features -- -D warnings: pass.
  • cargo test --workspace: 1,638 passed, 23 ignored. Example 4 acceptance with materialization #606 had 1,636 / 23, and this PR adds 2 tests.
  • Viewer tests: 29 ran, 6 skipped (Node).
  • The Example 1, 3a and 3b fixtures are unchanged. They are PromQL.
  • Integration with Stage 2 materialization: ingestion vs query time per summary #604: fix: integrate with #604 adds latency_ms: None to the Example 2 coverage test's RootDemand. The Example 2 design queries still build all 400 candidates and select P2.

@zzylol
zzylol force-pushed the stack/509-x9-pass1-coverage branch from 31ecc84 to 7c0019e Compare October 4, 2026 20:18
@zzylol
zzylol changed the base branch from stack/509-x8a-hydra-kernel to stack/509-x5b-example4-acceptance October 4, 2026 20:18
@zzylol
zzylol force-pushed the stack/509-x5b-example4-acceptance branch from 25d2af0 to 215ed9e Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/509-x9-pass1-coverage branch from 7c0019e to 07adadb Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/509-x5b-example4-acceptance branch from 215ed9e to ff97ad6 Compare October 8, 2026 16:47
@zzylol
zzylol force-pushed the stack/509-x9-pass1-coverage branch 2 times, most recently from 36bc40a to cb917b6 Compare October 10, 2026 14:32
zzylol and others added 2 commits October 11, 2026 16:22
…value

Pass 1 built every COUNT(*) update from ColumnRef::SampleValue, which SQL
rows (e.g. flows(ts, src_ip)) do not have. Exact Count accumulators failed
to compose, and sketches hashed a non-existent `value` item.

SQL COUNT(*) now adds a unit weight per row; a sketch hashes a non-null
column, a grouping column first. The executor builds an exact count with a
unit weight (Operator::unit_count_build), so the accumulator executes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RootDemand gained latency_ms in #604; the Example 2 coverage test's
demand sets it to None (no latency bound).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/509-x5b-example4-acceptance branch from 6161b30 to 96efb42 Compare October 11, 2026 17:11
@zzylol
zzylol force-pushed the stack/509-x9-pass1-coverage branch from cb917b6 to 143195c Compare October 11, 2026 17:11
zzylol added a commit that referenced this pull request Oct 13, 2026
The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
…598

Running the ignored SQL test composes every candidate and fails only on
Q2's options (per-source counts until #598), so #603 is not a blocker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
…stages (#577)

* test: acceptance spec and pending tests for planner-layering Example 1

Define MVP acceptance for #509 Example 1 before the Phase C stage APIs
exist: 1 -> 6 -> 6 -> 1 candidates (Pass 1 x identical-expression
sharing; physical operator implementation only, no materialization).

- Spec with per-stage candidate tables, invariants and doc ambiguities.
- Integration tests against todo!() stage stubs, ignored until the
  stages land; workload and Stage 0 frontend-shape tests run now.
- Expected-only asap-stage-pipeline/v1 fixture for the DAG viewer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run the Example 1 acceptance tests against the real stages

Replace the todo!() stubs with adapters over Stage 0-3, un-ignore the 11
tests that pass, and give each still-ignored test the precise difference
between the implementation and the spec. Add runtime capability checks:
the selected plan compiles in the physical planner; compiling every
candidate stays ignored with the two runtime gaps it finds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read Example 1 logical stages as flat DAGs after the export removal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 acceptance on the composed stages (1 -> 3 -> 3 -> 1)

The tests run Pass 1 and composition, optimize_physical_asap and
select_plan through a shared tests/planner_layering_common pipeline, and
follow the decided behavior: three Pass 1 candidates (Q2 exact, CMS+heap,
CountSketch+heap), every sub-DAG NotMaterialized, P1 selected, CMS+heap
invalid (weights not proven non-negative) and CountSketch+heap invalid (a
Top-K output has no guarantee and no deployment evidence). Hydra (#605),
Pass 2 sharing (#588) and the two runtime gaps (#593, #579) stay ignored
with the PR that delivers them. Tests are named after the stages; the
compile check no longer uses the PromQL fallback. The spec lists where the
doc and the planner differ. The expected-only viewer fixture is removed:
the generated four-stage fixture replaces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: port #590's Example 2-4 acceptance specs and tests

The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* feat(devtools): stage_pipeline examples for #509 Example 3

Ported from #590: --example planner-layering-3a (Pattern A, ad hoc batch of
five p99 reports) and planner-layering-3b (Pattern B, repeating 5-min p99),
the Example 3 acceptance tests' workloads. Example 1's output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: follow #576's per-consumer charging and exact top-k columns

Plan selection charges a NotMaterialized node once per consuming query,
so the charging helper checks NodeCost.multiplier and the Example 4 A3
ambiguity is recorded as decided. Example 1 checks that exact Q2 sorts by
the ranked value and returns [job, topk_10]. The stage viewer guide's
tab 1 row now describes the actual all-exact result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(devtools): one planning pipeline for stage_pipeline and the acceptance tests

run_planning_stages and the #509 example workloads move into the
asap-devtools library; stage_pipeline serializes the run and the
planner-layering tests adapt it. The shared data workload drops
ingestion_rate for data at rest, which validate() requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(devtools): select over every candidate; --max-displayed limits only the document

stage_pipeline capped enumeration at 64, so Example 3a (243 candidates)
selected P41 instead of the cheapest P122 (KLL for all five queries).
The pipeline now enumerates, optimizes and prices every candidate; the
document shows the first --max-displayed (default 64) plus the selected
one, with `combinations` and `capped`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: Example 2's SQL candidates compose; the SQL test waits only on #598

Running the ignored SQL test composes every candidate and fails only on
Q2's options (per-source counts until #598), so #603 is not a blocker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: name stages by number and name; say query-language frontends

The frontends' Run field, id and locals become stage0_query_language_frontends,
and test names carry stage0..stage3 with the stage name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: charging helper follows plan selection's per-consumer rule

With a materialized node the helper accepted any multiplier >= 1. It now
computes each node's computations as plan selection does (walks from
query roots and materialized nodes' inputs, stopping at materialized
nodes), checked on a hand-built candidate with a materialized node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 picks Q2 candidates by (option, no shared input)

physical_with and the invalid-candidate check no longer assume one
candidate per Q2 option, so #588's shared-input variants need no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: Example 3 Pattern B charges each node once per consuming query

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs(stage-viewer): rows 2, 3a and 3b describe the planner's current output

Read from uncapped runs: Example 2's sketch plans are rejected for having
no (ε, δ) guarantee, 3a selects P122 (first of 108 tied cheapest plans,
nothing shared yet) and 3b selects query-time KLL; panes and the latency
check come with the window-summary PRs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs(stage-viewer): row 2 from the SQL run; 3a ties 32 plans, not 108

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 counts only unshared candidates; stage3_ charging-helper names

The 3-logical and 3-physical counts now hold when #588 adds shared-input
variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(devtools): PlanningStagesRun fields carry stage number and name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test(devtools): unique temp file per stage_pipeline run

cargo test runs the tests as threads of one process, so two runs of the same
example shared a file name and one deleted the other's output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant