Tracker: run #777 eval plans end to end from saved plans #824

Goal

Take ASAP's saved plans from the #777 evaluation (sketch-bench autosketch_vs_asap, results in sketch-bench#203), convert them to StreamingConfig / InferenceConfig without a ControllerConfig or a re-solve, and run them with asap-tools/experiments/experiment_run_e2e.py. This is for the paper experiments, so it sits beside --planner milp, not inside it.

sketch-bench side: ProjectASAP/sketch-bench#204 (plan file, PromQL and label schema in the workload table).

TODO

• B1. Saved plan → deployments (feat(planner): asap-planner --planner saved writes a sketch-bench eval plan without re-solving #828, merged)
  • As built: passes (deployments, Raqe → deployment) to deployments_to_planner_output, which plan_to_planner_output now wraps, instead of a MilpSolution, whose PlanCost a saved plan can't fill.
  • Reuse --planner milp's path (asap-planner-rs/src/main.rs, run_milp): ControllerConfig → build_milp_workload (PromQL → RQEs) → plan_to_planner_output. Only the solve is replaced: load the saved plan instead of calling plan_milp.
  • Build a MilpSolution from plans/<table>.plans.json.gz: one deployment per distinct chosen[].deployment (sketch, params, window, slide; capability, metric and grouping from the table's RQE), and each ASAPQuery RQE pointed at its deployment.
  • The link between ASAPQuery's RQEs and sketch-bench's RQE ids is the table's e2e.queries ({promql, interval_secs, rqes}, sketch-bench#204 P2).
  • The converter is unchanged, with the same refusals (roll-ups, unmapped variants such as Hydra, two deployments for one query).
  • (Was: split the converter to work without a ControllerConfig. Not needed: the table gives us one.)
• B2. --planner saved mode (feat(planner): asap-planner --planner saved writes a sketch-bench eval plan without re-solving #828, merged)
  • As built: --sla-ms N instead of --kind (one SLA plan per SLA; omitted = cheapest). The same-PromQL-at-two-intervals case is left to the existing two-deployments refusal, which fires only when the deployments actually differ.
  • asap-planner --planner saved --plan FILE --workload-table FILE --method asap --weights cpu --kind cheapest --output-dir DIR writes streaming_config.yaml and inference_config.yaml.
  • Builds the ControllerConfig from the table's e2e block: queries grouped by interval_secs → query_groups; metrics label names → metrics.
  • Same PromQL at two intervals reading different RQEs (47 such strings in the r = 8 table) can't be keyed by query string in the inference config: refuse it. The base classic table has none.
• B3. e2e with prebuilt configs (feat(planner): asap-planner --planner saved writes a sketch-bench eval plan without re-solving #828, merged: controller.prebuilt_output_dir; ClickHouse runs in feat(asap-tools): run ClickHouse E2E with frozen plans #860)
  • experiment_run_e2e.py always runs the planner (controller_service.start). Add a flag that skips it and syncs a given controller_output dir to the nodes.
• B4. Synthetic workload experiment config (fix(asap-tools): send paced feeder ticks concurrently, ordered per series, in (t, t+1s] #840, feat(asap-tools): generate the paced feeder's data from a workload table #842, feat(asap-tools): generate records metrics in the paced feeder #843, feat(asap-tools): run sketch-bench workload tables end to end with the paced feeder #841, merged)
  • As built: the paced feeder replaces the fake exporter. fix(asap-tools): send paced feeder ticks concurrently, ordered per series, in (t, t+1s] #840 → feat(asap-tools): generate the paced feeder's data from a workload table #842 → feat(asap-tools): generate records metrics in the paced feeder #843 (feeder data from the table) and feat(asap-tools): run sketch-bench workload tables end to end with the paced feeder #841 (runner and aligned query client); sketch_bench_e2e/sketch_bench_e2e_configs.py writes the experiment config from the table's e2e block.
  • Fake exporter: gauge data and counter data_total (decided in sketch-bench#204), each with labels job and label_0, 1e4 series, 200 samples/s per series (2e6 samples/s per metric), Zipf 1.1 keys, Pareto a = 2 values.
  • label_0 determines job (each of the 1e4 label_0 values belongs to one of 10 jobs). The exporter's --num-values-per-label makes a cross product (1e5 series), so it needs a way to derive job from label_0, using the mapping in the workload table's schema (sketch-bench#204 P2).
  • Check the Rust exporter (--dataset, --metric-type) covers the distributions.
  • Query client: the PromQL from the workload table's queries, each at its RQEs' repeat interval T (1 s for sum by (job) (data) and quantile by (job), 60 s for the rest).
• B6. Engine support for 12 classic queries (feat(planner): serve the #777 eval's sum-of-rate and nested topk queries #827, closed)
  • As resolved: topk(sum by (l) (sum_over_time)) collapses into one CMS heap (feat(planner): collapse topk(sum by (l) (sum_over_time)) into one CMS heap #849, merged). The two rate templates (7 and 9) are dropped from the eval (sketch-bench#210). --planner saved converts the full 46-query classic table at 10 samples/s (sketch-bench#212) for all 12 ASAP plans.
  • sum by (job) (rate(...)) and topk(32, sum by (label_0) (...)) over rate and sum_over_time (12 of 54 classic queries). --planner saved refuses the classic tables until they are served.
• B5. Dependency pin Not needed.
  • Bump the rqe-optimizer pin once sketch-bench P1 lands, if B1 reads its serialized types.
  • sketch-bench#203 writes the plans as hand-written JSON (plans/<table>.plans.json.gz), not serialized rqe-optimizer types, so B1 parses that JSON and needs no new rqe-optimizer types.

Order

sketch-bench P1 plan file (done, #203) ──┐
sketch-bench P2 e2e block ───────────────┼─► B1 saved plan → MilpSolution ─► B2 --planner saved ─┐
                                         │                                                      ├─► e2e run (classic r=1 m=1)
B3 e2e skip-planner ─────────────────────┴──────────────────────────────────────────────────────┤
B4 exporter + query config ◄── P2 ───────────────────────────────────────────────────────────────┤
B6 engine: sum-of-rate, nested topk (#827) ─────────────────────────────────────────────────────┘

Not in scope

• Roll-ups and Hydra in the engine (fix(planner): refuse roll-up plans in milp_output #814 refuses roll-ups). Multi-grouping waits on these.
• Switching --planner milp from milp::minimize to minimize_usage_cost (sketch-bench#188). Not needed here, since the eval's plan is read, not re-solved.
• Arrival-rate mismatch (eval 2e6 samples/s vs. scrape-based e2e): tracked as a decision in the sketch-bench issue.