Skip to content

Pass 2: the planner chooses the segment grid - #634

Draft
zzylol wants to merge 2 commits into
stack/windows-3-pricingfrom
stack/windows-4-segment-grid
Draft

zzylol wants to merge 2 commits into
stack/windows-3-pricingfrom
stack/windows-4-segment-grid

Conversation

@zzylol

@zzylol zzylol commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Stack: #574 → #620 → #618 → #621 → #627 → #625 → #628 → #632 → #634 → #616 → #617 → #622 → #624 → #629 → #630 → #631 → #633 → #635 → #636 → #637

Problem

The shared-segment rule from #628 (share_window_segments, logical-optimizer Pass 2) offered only one grid: even segments whose width is the gcd of every window's lookback and offset. The planner never compared that grid with alternatives. For Example 3a, splitting only at the windows' own boundaries needs 4 segments instead of 5.

Changes (Q67)

  • Two segmentations, each its own Stage 1 variant; Stage 3's cost picks one.
    • (a) Split only at the queries' distinct range boundaries. In 3a the boundaries are 0, 1, 2, 3 and 5 years back, which gives 4 segments, the oldest one 2 years wide.
    • (b) The even gcd grid. In 3a that is 5 one-year segments. It is offered only when it differs from (a); identical segmentations are deduplicated.
    • Labels: · shared segments ×4 and · shared segments ×5. A target's label lists uneven widths, for example Kll · 365d+365d+365d+730d segments. An even grid keeps the old label, 365d segments.
  • WindowForm::Segments { unit_ms, ends }: the grid unit plus a bitmask of where each segment ends, so WindowForm stays Copy. WindowForm::widths gives the pane widths. tumbling_state and the W2 coverage proof (panes_cover_window) now take widths, so panes no longer need equal widths.
  • Sharing::WindowSegments { segments } carries the number of distinct segments, which also keeps the two variants distinct.
  • Accuracy is unchanged: every segment is sized for the group's strictest reader, and merging disjoint KLLs is exact.
  • Stage 2 (materialization.rs): a SummaryMerge over panes of different widths is not a stream of panes, because no pane is an older pane shifted back. Its chain is therefore neither maintained at ingestion time nor kept. In 4a only the even grid has an ingestion-time option.
  • Tests changed for the two grids: pattern_a_shares_four_boundary_or_five_one_year_segments (both forms, their ends masks and labels; 4 or 5 builds and 5 merges after composition), identical_segmentations_are_offered_once (new), stage1_a_shared_segments_serve_all_five (both candidates; each query's segments sum to its range), stage3_a_selects_the_cheaper_segment_grid (new), stage2_a_segments_are_built_once_for_all_consumers (4 and 5 segments, once and monthly), and the example4a_* devtools test (489 plans; only ×5 is maintained at ingestion time).
  • tools/dag-viewer/examples.json: the 3a and 4a stories are updated. planner-layering-example1.json is unchanged (verified by regenerating it).

Costs of each segmentation (after #632's pricing)

Ex ×4 boundary split ×5 even 1-year grid Selected
3a P487 18,104.003889 P488 18,104.004444 P487 ×4
4a P487 25.1444498 (query time; no ingestion-time option) P488 25.1444506; P488-m1 (ingestion time) 1,536.68 P487 ×4

The two grids cost almost the same. Under #632's pricing each row of the scan lands in exactly one segment, so the range and build work is the same for both. The ×4 grid wins only because its merges read fewer states: 4+1+1+1+2 instead of 5+1+1+1+3. Shared input costs 25,112 (3a) and 34.878 (4a).

Test plan

  • cargo fmt --all --check, cargo clippy --workspace --all-targets -D warnings
  • cargo test --workspace: 1692 passed, 0 failed, 13 ignored (after the rebase on main d4869a7; was 1681 passed, 13 ignored)
  • tools/dag-viewer: python3 -m unittest test_viewer: 25 OK
  • Ran stage_pipeline for all six examples. Only the 3a and 4a selections change (to ×4).

Stacked on #632 (stack/windows-3-pricing).

@zzylol
zzylol force-pushed the stack/windows-3-pricing branch from 44291c9 to 9cf2956 Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/windows-4-segment-grid branch from 8d5052f to 1397f2e Compare October 5, 2026 06:21
This was referenced Oct 5, 2026
@zzylol
zzylol force-pushed the stack/windows-3-pricing branch from 9cf2956 to a87cb4d Compare October 8, 2026 16:47
@zzylol
zzylol force-pushed the stack/windows-4-segment-grid branch 2 times, most recently from b2dea0d to 303216a Compare October 10, 2026 14:32
zzylol and others added 2 commits October 11, 2026 16:22
The shared-segment rule (Q60) offered one fixed grid, the gcd of every
window's lookback and offset. It now offers two segmentations, each as
its own Stage 1 variant, and Stage 3's cost decides:

- split only at the windows' own boundaries (fewest segments); for
  Example 3a four segments, the oldest two years wide;
- the even gcd grid (3a: five 1-year segments), when it differs.

WindowForm::Segments carries the grid unit and a bitmask of where each
segment ends, so a target's segments may differ in width; tumbling_state
and the coverage proof take the widths. Sharing::WindowSegments carries
the number of distinct segments, labeled "· shared segments ×4" / "×5".
Each segment is still sized for the group's strictest reader.

Stage 2 does not maintain a merge of panes of different widths at
ingestion time or keep it: it is no stream of panes. So in 4a only the
even grid has an ingestion-time option.

Example 3a and 4a select the four boundary segments: they build the same
rows as the five 1-year ones and merge fewer states (3a: 18,104.0039 vs
18,104.0044 per second).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…places the declared proof

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
@zzylol
zzylol force-pushed the stack/windows-4-segment-grid branch from 303216a to 7adc3ff Compare October 11, 2026 17:11
zzylol added a commit that referenced this pull request Oct 13, 2026
The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
…stages (#577)

* test: acceptance spec and pending tests for planner-layering Example 1

Define MVP acceptance for #509 Example 1 before the Phase C stage APIs
exist: 1 -> 6 -> 6 -> 1 candidates (Pass 1 x identical-expression
sharing; physical operator implementation only, no materialization).

- Spec with per-stage candidate tables, invariants and doc ambiguities.
- Integration tests against todo!() stage stubs, ignored until the
  stages land; workload and Stage 0 frontend-shape tests run now.
- Expected-only asap-stage-pipeline/v1 fixture for the DAG viewer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run the Example 1 acceptance tests against the real stages

Replace the todo!() stubs with adapters over Stage 0-3, un-ignore the 11
tests that pass, and give each still-ignored test the precise difference
between the implementation and the spec. Add runtime capability checks:
the selected plan compiles in the physical planner; compiling every
candidate stays ignored with the two runtime gaps it finds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read Example 1 logical stages as flat DAGs after the export removal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 acceptance on the composed stages (1 -> 3 -> 3 -> 1)

The tests run Pass 1 and composition, optimize_physical_asap and
select_plan through a shared tests/planner_layering_common pipeline, and
follow the decided behavior: three Pass 1 candidates (Q2 exact, CMS+heap,
CountSketch+heap), every sub-DAG NotMaterialized, P1 selected, CMS+heap
invalid (weights not proven non-negative) and CountSketch+heap invalid (a
Top-K output has no guarantee and no deployment evidence). Hydra (#605),
Pass 2 sharing (#588) and the two runtime gaps (#593, #579) stay ignored
with the PR that delivers them. Tests are named after the stages; the
compile check no longer uses the PromQL fallback. The spec lists where the
doc and the planner differ. The expected-only viewer fixture is removed:
the generated four-stage fixture replaces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: port #590's Example 2-4 acceptance specs and tests

The specs and tests from #590 run the current stages through the shared
planner_layering_common pipeline (Pass 1 and composition,
optimize_physical_asap, select_plan); tests are named after the stages.
Materialization is read from the physical DAG (#667); window forms and
retention stay pending adapters. Behavior that works passes (29 tests);
every expectation that needs a feature not in this stack is ignored with
the PR that delivers it (29 tests): Pass 2 sharing (#588, #595, #596),
window composition (#601, #628, #634), materialization enumeration and
MatAtQuery (#604, #625), UnivMon sizing and accuracy (#621), SQL
recognition and COUNT(*) composition (#598, #603) and plan selection row
estimates (#632). Each spec's "Today" column states the current output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* feat(devtools): stage_pipeline examples for #509 Example 3

Ported from #590: --example planner-layering-3a (Pattern A, ad hoc batch of
five p99 reports) and planner-layering-3b (Pattern B, repeating 5-min p99),
the Example 3 acceptance tests' workloads. Example 1's output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: follow #576's per-consumer charging and exact top-k columns

Plan selection charges a NotMaterialized node once per consuming query,
so the charging helper checks NodeCost.multiplier and the Example 4 A3
ambiguity is recorded as decided. Example 1 checks that exact Q2 sorts by
the ranked value and returns [job, topk_10]. The stage viewer guide's
tab 1 row now describes the actual all-exact result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(devtools): one planning pipeline for stage_pipeline and the acceptance tests

run_planning_stages and the #509 example workloads move into the
asap-devtools library; stage_pipeline serializes the run and the
planner-layering tests adapt it. The shared data workload drops
ingestion_rate for data at rest, which validate() requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(devtools): select over every candidate; --max-displayed limits only the document

stage_pipeline capped enumeration at 64, so Example 3a (243 candidates)
selected P41 instead of the cheapest P122 (KLL for all five queries).
The pipeline now enumerates, optimizes and prices every candidate; the
document shows the first --max-displayed (default 64) plus the selected
one, with `combinations` and `capped`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: Example 2's SQL candidates compose; the SQL test waits only on #598

Running the ignored SQL test composes every candidate and fails only on
Q2's options (per-source counts until #598), so #603 is not a blocker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: name stages by number and name; say query-language frontends

The frontends' Run field, id and locals become stage0_query_language_frontends,
and test names carry stage0..stage3 with the stage name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: charging helper follows plan selection's per-consumer rule

With a materialized node the helper accepted any multiplier >= 1. It now
computes each node's computations as plan selection does (walks from
query roots and materialized nodes' inputs, stopping at materialized
nodes), checked on a hand-built candidate with a materialized node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 picks Q2 candidates by (option, no shared input)

physical_with and the invalid-candidate check no longer assume one
candidate per Q2 option, so #588's shared-input variants need no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs: Example 3 Pattern B charges each node once per consuming query

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs(stage-viewer): rows 2, 3a and 3b describe the planner's current output

Read from uncapped runs: Example 2's sketch plans are rejected for having
no (ε, δ) guarantee, 3a selects P122 (first of 108 tied cheapest plans,
nothing shared yet) and 3b selects query-time KLL; panes and the latency
check come with the window-summary PRs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs(stage-viewer): row 2 from the SQL run; 3a ties 32 plans, not 108

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test: Example 1 counts only unshared candidates; stage3_ charging-helper names

The 3-logical and 3-physical counts now hold when #588 adds shared-input
variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* refactor(devtools): PlanningStagesRun fields carry stage number and name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test(devtools): unique temp file per stage_pipeline run

cargo test runs the tests as threads of one process, so two runs of the same
example shared a file name and one deleted the other's output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant