feat(dag-viewer): show the stage pipeline as three stacked lanes

[zzylol]

Supersedes #574 (viewer redesign) and #638 (stacked lanes). Their server.py, generate-sample.sh and dag_export parts conflicted with #686 and are not carried over. #618 was already superseded by #686.

Why

Since #686 the live viewer plans through stage_pipeline, but it still shows one query at a time as a "pre-ASAP" and a "post-ASAP" DAG. That split no longer exists in the planner: planning is Stage 0 → Stage 1 → Stage 2 → Stage 3 over the whole workload, and Pass 2 candidates share nodes across queries. A per-query view hides that sharing, as well as the candidates Stage 3 did not select.

What

The viewer becomes the stage view approved for G11 (G11a):

• Three lanes stacked top to bottom, each showing the whole workload: Stage 0 (query-language frontends), the chosen Stage 1 logical ASAP candidate, and its Stage 2 physical candidate with each node's timing and Stage 3 cost (× n for a node that is not materialized and is computed once per reader). Lanes 1 and 2 have candidate pickers. Choosing a Stage 1 candidate shows its physical plan; Stage 2 is cleared only for a pruned or uncomposed candidate.
• Stage 3 list beside the lanes, ported from feat(dag-viewer): plan through stage_pipeline in the live viewer; delete dag_export #686's candidate table: the selected plan, then valid-but-costlier plans cheapest first, then rejected plans with their reasons ("not judged" when selection failed with error), then Stage 2 pruned and Stage 1 composition-error rows. The header shows the candidate count, the display cap, and the Stage 1 error or selection.error.
• Scalar roots: a standalone scalar query (time(), scalar(v), 1 + 2) is a dashed Scalar node over the nodes it reads. Stage 0/1 read it from {"Scalar": expr} roots and Stage 2 from scalar_roots.
• Examples on demand: examples.json lists Examples 1, 3a and 3b (the ones stage_pipeline --example supports). server.py writes the missing ones into the git-ignored out/ when it starts, and rewrites all of them with --regenerate. Only examples/planner-layering-example1.json stays committed, for the stage_pipeline contract test.
• The SQL + PromQL editor from feat(dag-viewer): plan through stage_pipeline in the live viewer; delete dag_export #686 is kept, moved from planner-ui.js to editor.js, and shows its result in the lanes.
• Removed: the per-query pre/post view (viewer.js, stage-workload.js), render.py (the standalone HTML export) and its test_render.py, dag.example.json, generate-sample.sh, the scalar-queries.json fixture and the old screenshots.
• No deployment-inputs panel. Deployment inputs arrive with stage_pipeline --deployment in G11b, after G9.

How

• stages.js (no DOM) puts both DAG kinds into one shape. A flat logical DAG gets one edge per operator input (every input field of NonASAPOp and ASAPOp: child, children, left/right, lhs/rhs, summary_input, outer/inner) and one per operator a scalar expression reads. A PhysicalASAPDAG uses its edge list, and its query roots are rebuilt from roots plus scalar_roots by query position. From that shape it builds cytoscape elements with readable labels: summary family and parameters, grouping and instances, sort/limit partitions, timing, cost, and "root of q…". It also validates the document, including operand references and scalar_roots positions, and computes the Stage 3 rows and header.
• app.js is feat(dag-viewer): redesign the viewer around the stage pipeline #574's page logic adapted to the current document names (stage0_query_language_frontends, stage1_logical_asap_optimization, stage2_physical_asap_optimization, stage3_plan_selection). The layout is feat(dag-viewer): redesign the viewer around the stage pipeline #574's, with feat(dag-viewer): stack the stage lanes top to bottom #638's stacked lanes and sticky Stage 3 list.
• node-style.js keeps KIND_CATEGORY_JSON, still checked by viewer_contract.rs, and gains nodeGroup (source / relational / summary / scalar).
• crates/devtools/tests/stage_pipeline.rs: the scalar test now checks the scalar_roots positions directly instead of comparing a committed fixture.

Key code interfaces added / modified

No Rust interface changes. The viewer's own interfaces (tools/dag-viewer):

tools/dag-viewer/server.py

# The example list the server writes into out/ and the viewer shows as tabs.
EXAMPLES = HERE / "examples.json"  # new

# Runs stage_pipeline for each examples.json entry whose output file is missing
# (every entry with force); called at startup before serving.
def write_examples(force: bool) -> None: ...  # new: writes each missing examples.json document into out/ (all with force)

# main(): new CLI flag; write_examples(args.regenerate) runs before serving
# --regenerate: rewrite every example document even if it exists; default off.
parser.add_argument("--regenerate", action="store_true", help="rewrite the example documents in out/")

# removed: Handler.do_GET serving GET /api/example (from dag.example.json via render.load_document)
# removed (render.py): check_document(document) -> dict, load_document(path) -> dict, render(document) -> str

tools/dag-viewer/examples.json

[
  {
    // Tab id; also the page's #<id> hash.
    "id": "example1",
    // Tab label.
    "name": "1 · dashboard panels",
    // The `stage_pipeline --example` name that produces the document.
    "example": "planner-layering-1",
    // Where the document is written, under the git-ignored out/.
    "file": "out/example1.json",
    // One-line description shown with the example.
    "story": "Two PromQL panels over the last minute, every 10 s: Q1 the exact per-job request rate, Q2 the top-10 series per job within ε = 0.01, δ = 0.001 and 100 ms."
  }
]

New: example is a stage_pipeline --example name; file is its output under the git-ignored out/. Replaces dag.example.json and examples/scalar-queries.json.

tools/dag-viewer/stages.js (new, no DOM; replaces stage-workload.js)

// The `format` tag of a stage_pipeline document the viewer accepts.
const STAGE_PIPELINE_FORMAT = 'asap-stage-pipeline/v1';
// Whether `doc` is an object tagged with STAGE_PIPELINE_FORMAT.
function isStagePipelineDocument(doc);
// Every shape problem as readable strings; the viewer refuses a document
// with any, rather than guessing.
function validateStagePipeline(doc);        // new: every shape problem as strings; [] = can be shown
// Stage 3's candidate list: selected, other valid ones cheapest first,
// rejected, Stage 2 pruned, then Stage 1 ones that did not compose.
function stageCandidateRows(doc);           // Stage 3 candidate rows, incl. Stage 2 pruned and non-composing ones
// The list header: candidates enumerated and shown, and why Stage 1 or plan
// selection failed, when it did.
function stageCandidateHeader(doc);
// One stage DAG as cytoscape nodes and edges, with a `Scalar` node per
// standalone scalar query; options mark physical timing, costs and query roots.
function stageLaneElements(dag, options);  // new: one stage DAG as cytoscape elements;
                                            // options { physical = false, costPerNode = null, queryIds = [] }
// removed (stage-workload.js): stageScalarRoot(index, expr), stageLabel(kind, body, cost),
//   stageQueryDag(root, nodeOf), stageWorkload(doc)

tools/dag-viewer/app.js (new; replaces viewer.js), editor.js (new; replaces planner-ui.js)

// The viewer's page-global entry point; the editor calls showDocument with a
// planned document.
window.StageViewer = { showDocument: showExternal };  // new; was window.renderPlannerWorkload = showDocument (viewer.js)
// Shows a document from outside the example list (a file or the editor) and
// clears the selected tab.
function showExternal(data, source);  // returns whether the document was shown

// Page URL: ?doc=<path> opens one stage_pipeline document; otherwise #<id> picks an examples.json entry
// The `?doc=<path>` query parameter, if any, read at page load.
const requested = new URLSearchParams(location.search).get('doc');

tools/dag-viewer/node-style.js

// Operator kind -> category map, checked against the IR by viewer_contract.rs (kept; every entry unchanged; the markers below are not in the file).
// nodeGroup draws "data" as source, "summary" as summary and every other category as rel.
const KIND_CATEGORY_JSON = `{
  "Scan": "data",                        // unchanged: a table or metric source
  "Values": "data",                      // unchanged: literal rows
  "Filter": "filter",                    // unchanged
  "PromqlSeriesSample": "sample",        // unchanged: PromQL instant-selector sampling
  "Project": "derive",                   // unchanged
  "PromqlRelabel": "derive",             // unchanged
  "PromqlInfoEnrich": "derive",          // unchanged
  "PromqlVectorFromScalar": "derive",    // unchanged
  "BinaryOp": "derive",                  // unchanged
  "Aggregate": "aggregate",              // unchanged: exact grouped aggregation
  "TimeRange": "window",                 // unchanged
  "PromqlSubquery": "window",            // unchanged
  "TimeShift": "window",                 // unchanged
  "SQLWindowFunc": "window",             // unchanged
  "Join": "join",                        // unchanged
  "Dedup": "set",                        // unchanged
  "SetOp": "set",                        // unchanged
  "Concat": "combine",                   // unchanged
  "Sort": "sort",                        // unchanged
  "Limit": "sort",                       // unchanged
  "SummaryAgg": "summary",               // unchanged: builds a summary
  "SummaryJoin": "summary",              // unchanged
  "SummarySubtract": "summary",          // unchanged
  "SummaryDelete": "summary",            // unchanged
  "SummaryEstimate": "summary",          // unchanged: reads a summary
  "SummaryMerge": "summary",             // unchanged
  "FinalizeExactAccumulator": "summary", // unchanged
  "MaintainPopulation": "summary",       // unchanged
  "EvaluatePopulation": "summary",       // unchanged
  "Extension": "summary"                 // unchanged
}`;
// The color group of an operator kind; an unknown kind is drawn as 'rel'
// and logged.
function nodeGroup(kind);  // new: 'source' | 'rel' | 'summary' | 'scalar'
// removed: const CATEGORIES, const ROOT_BADGE, isDarkMode(), categoryOf(kind),
//   categoryColors(name), applyPaletteToCssVars()

Usage, end to end (the editor path; numbers from running server.py at 90caee78, posting the request below, and calling stages.js on the returned document in V8 as test_viewer.py does):

# 1. Start the viewer. server.py builds stage_pipeline, writes the missing Example 1, 3a and 3b
#    documents into out/, and serves the page and /api/plan on port 8000.
python3 tools/dag-viewer/server.py

// 2. In the editor (editor.js) the user enters one PromQL query and ε = 0.01, then presses Run.
//    `queries` holds the checked editor rows: here p99 latency over 5 minutes. It is posted next.
const queries = [{ name: 'q1', language: 'promql', schemas: [], text: 'quantile_over_time(0.99, latency_ms[5m])' }];
// `response` is server.py's answer. It runs
//   stage_pipeline --promql '<text>' --name q1 --epsilon 0.01 --interval-ms 15000 --out <tmp>
// and returns that asap-stage-pipeline/v1 document (422 when Stage 1 passes --max-combinations).
const response = await fetch('/api/plan', { method: 'POST', headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ queries, schemas: [], epsilon: 0.01 }) });
// `result` is the document: the Stage 0 DAG, 3 Stage 1 candidates (L1 exact, L2 Kll, L3 DDSketch),
// 3 Stage 2 candidates and stage3_plan_selection. Step 3 draws it.
const result = await response.json();

// 3. app.js shows it. showDocument first runs validateStagePipeline(result) (here [], no errors;
//    a malformed document throws with its first 3 errors), then fills the three lanes and the
//    Stage 3 list. `shown` is false when the document is refused; the editor then reports it.
const shown = window.StageViewer.showDocument(result, 'editor · 1 query');

// 4. What the user sees.
//    Status line, from result.stage3_plan_selection:
//      "Stage 3 selected P2 (cost 0.000517 cost_per_second) of 3 candidates."
//    Stage 3 list (stageCandidateRows): P2 selected; P3 costlier (tied with P2, lost on
//    enumeration order); P1 costlier (0.0006).
//    Lane 3, P2's physical DAG, drawn from
//    stageLaneElements(dag, { physical: true, costPerNode, queryIds: ['q1'] }): 5 nodes, 4 edges,
//    bottom to top:
//      Scan            source: latency_ms                             ⏱ query time  cost 0.00035
//      TimeRange       range: 300s                                    ⏱ query time  cost 0.0000833
//      SummaryAgg      summary: Kll · k 269, per series, one instance per series     cost 0.0000833
//      SummaryEstimate query: Quantile(q(0.99))                                      cost 2.78e-7
//      Project         root of q1                                                    cost 2.78e-7

Before this PR

The viewer had query tabs and one canvas with a pre-ASAP and a post-ASAP DAG per selected query. Only the selected plan's physical DAG was visible, as per-query slices. A 27k-line dag.example.json and a scalar fixture were committed and regenerated by hand.

After this PR

python3 tools/dag-viewer/server.py builds stage_pipeline, writes out/example{1,3a,3b}.json, and serves the stage view. Any Stage 1 candidate or Stage 2 plan can be opened from the pickers or the Stage 3 list. A Pass 2 candidate such as Example 1's L2 ("shared Scan #1") shows one Scan read by both queries.

Loading the generated documents through app.js (stub DOM) gives these counts lines:

example1:  1 logical DAG → 9 logical ASAP → 9 physical → 1 selected (6 invalid, 2 valid but costlier); P1, 7.94 cost per second
example3a: 1 logical DAG → 972 logical ASAP (65 shown) → 65 physical → 1 selected (0 invalid, 64 valid but costlier); P485
example3b: 1 logical DAG → 3 logical ASAP → 3 physical → 1 selected (0 invalid, 2 valid but costlier); P2

Screenshot: not applicable. No browser was available to render the page. The checks above run the page script against a stub DOM.

Verification

• python3 -m unittest test_viewer test_server (from tools/dag-viewer, with py_mini_racer and a built stage_pipeline): 45 tests pass. They cover document validation, the Stage 3 rows and header (composition errors, Stage 1 error, selection.error and not-judged rows), the lanes (edges, roots, timing, costs, Pass 2 shared scan), a binary PromQL query's two operand edges in Stage 0, 1 and 2 and scalar roots first/middle/last in Stage 0 and Stage 2 (both generated on demand), lane pickers, node details, the page and editor element ids, the example list, write_examples, and /api/plan.
• cargo nextest run -p asap-devtools: 25/25 pass, including viewer_contract and stage_pipeline (the Example 1 fixture is current).
• Manual: server.py --skip-build wrote the three examples into out/ (git-ignored) and served the page and documents. /api/plan planned a mixed SQL + PromQL request.
• Full workspace on head a3db2ff4 (on main at 3a25450d): cargo fmt --all -- --check and cargo clippy --workspace --all-targets --all-features --locked -- -D warnings pass, and cargo nextest run --workspace passes 1663 tests (46 skipped).

Limitations and follow-ups

• G11b: a read-only deployment-inputs panel fed by stage_pipeline --deployment <json> (after G9).
• docs/user_guide_docs/stage-viewer.md still describes six examples, a deployment panel and a PromQL-only editor. docs/develop_docs/physical-handoff-costs.md still names test_render.py. Both are left for the docs pass (Update the docs after the G1–G11 consolidation groups land #677).
• The Python viewer tests are not run in CI (they were not before either).