Skip to content

eval: §6.3 workloads re-planned at 10 samples/s for the end-to-end run - #212

Merged
zzylol merged 11 commits into
mainfrom
eval/e2e-rate10-plans
Oct 14, 2026
Merged

eval: §6.3 workloads re-planned at 10 samples/s for the end-to-end run#212
zzylol merged 11 commits into
mainfrom
eval/e2e-rate10-plans

Conversation

@zzylol

@zzylol zzylol commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #210.

Summary

The §6.3 workloads (classic, multigroup, all at r = 1, m = 1) re-planned at 10 samples/s, 1/20 of the data rate, for the end-to-end run (ASAPQuery#862). The windows, templates, accuracy curves, cost table, p95 targets and runner are unchanged, and §6.3's own results stay at 200 samples/s.

  • --samples-per-sec on export_autosketch_eval_table.py sets the classic rate and scales http/flows. With it, a schema RQE whose smallest covered group falls below the curves' first N (1e3) is left out and logged, together with its e2e query. Without the flag the export is unchanged: it writes the committed 200/s tables byte for byte.
  • scripts/extract_autosketch_plans.py writes plans/*.plans.json.gz from a run with plans kept. On a 200/s rerun it reproduces the committed -hydra plans exactly.
  • rqe-optimizer/results/e2e-rate10/: tables, plans, runner results, and a README covering what's served, what's left out and the predicted costs.

Results

Workload RQEs served at 10/s ASAP, CPU (vCPU), 200/s → 10/s ASAP memory (GiB)
classic 42 / 42 2.38 → 0.309 21.4 → 15.3
multigroup 60 / 62 0.221 → 0.095 1.06 → 1.34
all 102 / 104 2.60 → 0.404 22.4 → 16.7
  • Left out: template 16's 5m p99 by (region, endpoint) and by (region, service, endpoint). Their smallest group holds 403 items, below the accuracy curves' first N (1e3), so no method can serve them. With them kept, the runner drops exactly these two and picks the same plans.
  • Why the plans change: only ingest CPU scales with the rate. With cheaper ingest the MILP picks longer windows; for classic, quantiles go from 300 s to 900 s, rate from 60 s to 180 s, and top-k from 60 s to 300 s. Merge and query CPU barely moves and is now half or more of ASAP's cost.

Testing

The Python suites pass, including a new test for the flag, and so does cargo test --release -p rqe-optimizer. The default export reproduces the committed tables, and the 200/s rerun reproduces the committed plans.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

zzylol and others added 10 commits October 13, 2026 01:11
@zzylol @claude
eval: e2e block for multi-grouping and all tables (#204)
8889e97 
export_autosketch_eval_table.py --synthetic now writes workloads[0].e2e to
every table. Schema templates have no PromQL (SQL is rendered downstream),
so each schema RQE gets one structured query entry: table, value,
capability, grouping, quantile (p99: 0.99), window_secs, interval_secs and
its rqes. Each schema metric is a `records` table (`http`, `flows`;
`data_i_http` under metric `data_i`) with SCHEMAS' labels, rate as
records_per_sec, and values (scaled_by, anomaly, burst as in SCHEMAS).
Classic entries are unchanged.

The multigroup and all tables in autosketch-vs-asap-hydra/inputs/tables are
regenerated; minus e2e they equal the committed ones byte for byte, and the
classic tables are unchanged. Their roll-up plans were already exported by
#203 (plans/*.plans.json.gz); a rerun of multigroup-shared1 and all-shared1
reproduces all 36 plans of each exactly, so the plans stand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
@zzylol @claude
eval: one e2e entry per distinct schema query (#204)
0600fe3 
Schema query entries identical apart from `rqes` (templates 14, 15 and 17
share some groupings) are merged into one entry whose `rqes` lists every
RQE it reads, sorted, so each distinct SQL query appears once. Classic
entries are unchanged. The multigroup and all tables are regenerated;
minus e2e they still equal main's byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
@zzylol @claude
eval: wrap the e2e_block docstring
9c79b96 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
@zzylol @claude
eval: drop classic templates 7 and 9 (rates by job, top-k by rate)
896c152 
`sum by (job) (rate(data_total[w]))` (template 7) and
`topk(32, sum by (label_0) (rate(data_total[w])))` (template 9) leave the
§6.3 evaluation (ProjectASAP/ASAPQuery#777): ASAPQuery's engine would need
per-series counter deltas to serve them. Per-series `rate` (template 6)
stays, so `data_total` is still generated. The dashboard set drops the same
two (D3, D4). Templates keep their numbers.

Classic is now 8 templates, 42 RQEs (46 PromQL queries); all is 104 RQEs.
Multigroup is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
@zzylol @claude
eval: regenerate the classic and all tables without templates 7 and 9
a066fec 
export_autosketch_eval_table.py --synthetic, with each table's committed
--revision (classic 44ac160, all and multigroup beacf68). The multigroup
tables regenerate byte for byte. Classic: 42 RQEs (r = 8: 76; m = 8, 16:
336, 672); all: 104 (200; 832, 1664). The e2e blocks no longer hold
either query; rate(data_total[w]) remains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
@zzylol @claude
eval: render template 2's e2e PromQL as topk(32, sum_over_time(data[w]))
d7d34b5 
label_0 names one series (child_of job, unique values), so
`sum by (label_0)` only drops `job`: the ranking and values are the same.
ASAPQuery's planner supports the two-level shape today, not the
three-level one (ProjectASAP/ASAPQuery#744, #772). The RQE (top-k keyed by
label_0, topk32_sum_by_label0_sum_over_time) is unchanged; only the
classic and all tables' e2e queries change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
@zzylol @claude
eval: rerun §6.3 without classic templates 7 and 9
9b130bc 
The classic and all workloads, rerun exactly as #203 (`--runs 3
--no-chosen`, the same saturation inputs; plans from a `--runs 1` rerun
with chosen plans kept). Multigroup tables are unchanged and keep their
results; a rerun of multigroup-shared1 reproduces its costs and plans
exactly. AutoSketch's benchmark times are reused: every probe was already
measured, and each workload's benchmark time is unchanged.

Classic (42 RQEs): ASAP 2.38 vCPU (was 3.54), PerQuery 7.77 (10.5),
AutoSketch 2,739 (3,716); at AutoSketch's latency ASAP 4.22 (5.67).
All (104 RQEs): ASAP 2.60 (3.76), no roll-ups 2.93 (4.09), roll-ups save
11% (8%). Zero dropped RQEs, sanity violations or Hydra choices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
@milindsrivastava1997
eval: save ASAPQuery SQL in structured e2e queries
cd51be9
@zzylol @claude
Merge #209 (ASAPQuery SQL in structured e2e queries) into #210
4dbc20d 
Table conflicts resolved by regenerating every table with the merged
exporter and each table's original --revision (classic 44ac160, all and
multigroup beacf68). Every structured e2e entry now has `sql`; the
multigroup tables equal #209's byte for byte. Minus `e2e`, every table
equals #210's previous head, so the plans and results stand. Templates 7
and 9 stay dropped and template 2's PromQL stays
topk(32, sum_over_time(data[w])).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
@zzylol @claude
eval: §6.3 workloads re-planned at 10 samples/s for the e2e run
5959c11 
export_autosketch_eval_table.py --synthetic gains --samples-per-sec (default
200): it sets SYNTHETIC_SAMPLES_PER_SEC and scales the schema metrics'
rates, and leaves out (logged) a schema RQE whose smallest covered group
falls below the curves' first N, with its e2e query, where the default
asserts. Without the flag the export is unchanged (the committed 200/s
tables, byte for byte but the final newline).

rqe-optimizer/results/e2e-rate10/: the classic, all and multigroup
tables (shared = 1) at 10 samples/s, the runner's results (p95, --runs 3
--no-chosen) and each method's plans (--runs 1, extracted by the new
scripts/extract_autosketch_plans.py, which reproduces the committed -hydra
plans). Two template-16 RQEs (p99 by (region, endpoint) and (region,
service, endpoint) over 5m, 403 items in the smallest group) are left
out; the runner would drop them as unservable and picks the same plans
either way. Every other RQE is served. Cheapest plan, CPU weights,
200/s -> 10/s: classic ASAP 2.38 -> 0.309 vCPU, PerQuery 7.77 -> 0.725,
AutoSketch 2,739 -> 137; all ASAP 2.60 -> 0.404; multigroup ASAP
0.221 -> 0.095.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
@zzylol
zzylol changed the base branch from eval/drop-group-rate-templates to main October 14, 2026 01:47
@zzylol
Merge origin/main (#210 squashed) into eval/e2e-rate10-plans
7e0bee8 
# Conflicts:
#	rqe-optimizer/README.md
#	scripts/export_autosketch_eval_table.py
@zzylol
zzylol merged commit 56d28a7 into main Oct 14, 2026
2 checks passed
This was referenced Oct 14, 2026
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@zzylol @milindsrivastava1997