eval: drop classic templates 7 and 9 (rates by job, top-k by rate) by zzylol · Pull Request #210 · ProjectASAP/sketch-bench

zzylol · 2026-10-13T02:16:11Z

Stacked on #209 (e2e blocks for multigroup/all tables).

Summary

Drops two classic templates from the §6.3 AutoSketch-vs-ASAP evaluation, because ASAPQuery's engine would need per-series counter deltas to serve them (decided with Milind):

7, sum by (job) (rate(data_total[w])): a rate accumulator per job needs each series' previous counter value to turn samples into increments.
9, topk(32, sum by (label_0) (rate(data_total[w]))): the same, feeding a CMS + heap.

Template 6 (per-series rate) stays, so data_total is still generated. Templates keep their numbers. The dashboard set loses D3 and D4 (the same two queries).

Also, template 2's e2e PromQL is now topk(32, sum_over_time(data[w])) instead of topk(32, sum by (label_0) (sum_over_time(data[w]))). label_0 identifies a series (child_of: job, globally unique values), so the sum by only dropped job; ranking and values are identical. ASAPQuery's planner supports this two-level shape today, but not the three-level one (ASAPQuery#744/#772). The RQE is unchanged. With this, every asap plan of classic-shared1 converts with ASAPQuery#828's --planner saved (checked on CloudLab).

Changes

896c152: templates 7 and 9 removed from synthetic_queries() (classic and dashboard); docstrings, README, tests and one Rust test fixture updated.
a066fec: tables regenerated with each one's original --revision (classic 44ac160…, all/multigroup beacf68). The multigroup tables are byte-identical.
d7d34b5: template 2's e2e PromQL (only e2e.queries changes; every RQE is identical).
9b130bc: the rerun, with the eval: AutoSketch vs. ASAP with Hydra candidates and a roll-up ablation (§6.3) #203 command (--runs 3 --no-chosen). It updates the 8 classic/all *-tp95.json, 8 plan files, both summaries, the 4 figures and the README.
- AutoSketch's benchmark times are reused, since every probed config was already cached.
- The multigroup results are kept: a rerun reproduced its plan file byte for byte, and its costs too.

Results, before → after (no conclusion flips)

	Before	After
Classic RQEs per replica	58 (50 distinct)	50 (42 distinct)
Classic PromQL queries in e2e	54	46
Mixed, AutoSketch	3,716 vCPU, 176 $/h	2,739 vCPU, 136 $/h
Mixed, ASAP at AutoSketch's latency (1.0 s)	5.67 vCPU	4.22 vCPU
Mixed, cheapest: ASAP / PerQuery	3.54 / 10.5 vCPU	2.38 / 7.77 vCPU
Mixed, 100 ms SLA: PerQuery vs. ASAP	64.5 vs. 36.7	60.1 vs. 34.1
Mixed + multi-grouping, roll-up saving	8% (34 → 13 deployments)	11% (32 → 11 deployments)
Planning time, largest workload	22.5 s (1,792 RQEs)	15.6 s (1,664 RQEs)
Multi-grouping (60% roll-up saving)		unchanged

AutoSketch's latency (1.0 s) and the tightest feasible bounds (87.7 ms, 490 ms) are unchanged. Which SLAs AutoSketch misses is unchanged. No plan picks Hydra, and none drops an RQE. The reruns ran on several CloudLab c6320 nodes, so planning times aren't strictly comparable; most of the drop is the smaller RQE count.

Testing

The Python suite and cargo test --release -p rqe-optimizer (with example tests) pass. A rerun of multigroup-shared1 reproduced its committed plan file byte for byte.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

export_autosketch_eval_table.py --synthetic now writes workloads[0].e2e to every table. Schema templates have no PromQL (SQL is rendered downstream), so each schema RQE gets one structured query entry: table, value, capability, grouping, quantile (p99: 0.99), window_secs, interval_secs and its rqes. Each schema metric is a `records` table (`http`, `flows`; `data_i_http` under metric `data_i`) with SCHEMAS' labels, rate as records_per_sec, and values (scaled_by, anomaly, burst as in SCHEMAS). Classic entries are unchanged. The multigroup and all tables in autosketch-vs-asap-hydra/inputs/tables are regenerated; minus e2e they equal the committed ones byte for byte, and the classic tables are unchanged. Their roll-up plans were already exported by #203 (plans/*.plans.json.gz); a rerun of multigroup-shared1 and all-shared1 reproduces all 36 plans of each exactly, so the plans stand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

Schema query entries identical apart from `rqes` (templates 14, 15 and 17 share some groupings) are merged into one entry whose `rqes` lists every RQE it reads, sorted, so each distinct SQL query appears once. Classic entries are unchanged. The multigroup and all tables are regenerated; minus e2e they still equal main's byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

`sum by (job) (rate(data_total[w]))` (template 7) and `topk(32, sum by (label_0) (rate(data_total[w])))` (template 9) leave the §6.3 evaluation (ProjectASAP/ASAPQuery#777): ASAPQuery's engine would need per-series counter deltas to serve them. Per-series `rate` (template 6) stays, so `data_total` is still generated. The dashboard set drops the same two (D3, D4). Templates keep their numbers. Classic is now 8 templates, 42 RQEs (46 PromQL queries); all is 104 RQEs. Multigroup is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

export_autosketch_eval_table.py --synthetic, with each table's committed --revision (classic 44ac160, all and multigroup beacf68). The multigroup tables regenerate byte for byte. Classic: 42 RQEs (r = 8: 76; m = 8, 16: 336, 672); all: 104 (200; 832, 1664). The e2e blocks no longer hold either query; rate(data_total[w]) remains. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

label_0 names one series (child_of job, unique values), so `sum by (label_0)` only drops `job`: the ranking and values are the same. ASAPQuery's planner supports the two-level shape today, not the three-level one (ProjectASAP/ASAPQuery#744, #772). The RQE (top-k keyed by label_0, topk32_sum_by_label0_sum_over_time) is unchanged; only the classic and all tables' e2e queries change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

The classic and all workloads, rerun exactly as #203 (`--runs 3 --no-chosen`, the same saturation inputs; plans from a `--runs 1` rerun with chosen plans kept). Multigroup tables are unchanged and keep their results; a rerun of multigroup-shared1 reproduces its costs and plans exactly. AutoSketch's benchmark times are reused: every probe was already measured, and each workload's benchmark time is unchanged. Classic (42 RQEs): ASAP 2.38 vCPU (was 3.54), PerQuery 7.77 (10.5), AutoSketch 2,739 (3,716); at AutoSketch's latency ASAP 4.22 (5.67). All (104 RQEs): ASAP 2.60 (3.76), no roll-ups 2.93 (4.09), roll-ups save 11% (8%). Zero dropped RQEs, sanity violations or Hydra choices. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

Table conflicts resolved by regenerating every table with the merged exporter and each table's original --revision (classic 44ac160, all and multigroup beacf68). Every structured e2e entry now has `sql`; the multigroup tables equal #209's byte for byte. Minus `e2e`, every table equals #210's previous head, so the plans and results stand. Templates 7 and 9 stay dropped and template 2's PromQL stays topk(32, sum_over_time(data[w])). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

…templates # Conflicts: # rqe-optimizer/results/autosketch-vs-asap-hydra/inputs/tables/synthetic-templatesall-shared1-metrics16.json # rqe-optimizer/results/autosketch-vs-asap-hydra/inputs/tables/synthetic-templatesall-shared1-metrics8.json # rqe-optimizer/results/autosketch-vs-asap-hydra/inputs/tables/synthetic-templatesall-shared1.json # rqe-optimizer/results/autosketch-vs-asap-hydra/inputs/tables/synthetic-templatesall-shared8.json

# Conflicts: # rqe-optimizer/README.md # scripts/export_autosketch_eval_table.py

…, Hydra W 65536) Inputs: the new configs' curves appended to autosketch-vs-asap-inputs (raw records in raw/dense_*), 7 cost rows (63 in all), and 999 Hydra runs at W = 65536 (7,778 in all). Results: the 12 workloads at 95% (with plans) and at 90% and 99%, on #210's tables. CPU cost falls 0-1.4%, Fargate cost up to 13% (smaller KLL/DDSketch nearer the target). 90% still equals 95%. Hydra is never chosen, even at W = 65536. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* study: KLL k 25-800 (2x steps), DDSketch alpha 0.1, Hydra W 65536; --sketches/--configs to add configs to an existing study Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhcWJExZcmVqS6r6rEtjis * study: Hydra cost rows past W 16384 merge 4 shards (16 grids of 3.2 GB do not fit) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: §6.3 rerun on the denser grid (KLL k 25-800, DDSketch alpha 0.1, Hydra W 65536) Inputs: the new configs' curves appended to autosketch-vs-asap-inputs (raw records in raw/dense_*), 7 cost rows (63 in all), and 999 Hydra runs at W = 65536 (7,778 in all). Results: the 12 workloads at 95% (with plans) and at 90% and 99%, on #210's tables. CPU cost falls 0-1.4%, Fargate cost up to 13% (smaller KLL/DDSketch nearer the target). 90% still equals 95%. Hydra is never chosen, even at W = 65536. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * review: one filter helper with name checks and tests; README provenance and claims - filtered_configs()/check_filters(): --sketches or --configs skips the exact rows; a misspelled name exits; tests for both. - The wide-merge comment says per fold, with operands 4x larger. - rqe-optimizer README: the -hydra row's 2026-10-14 rerun, the one solver-tolerance violation, e2e-rate10 planned on the pre-#213 grid. - -hydra README: rerun provenance moved to Reproduce; hydra-kll's figure marked unmerged, with the all-merge-counts value. - raw README: measurement dates, clnode130. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * review round 2: --sketches help names the Hydra flags; planning-time comparability; rewrap Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

zzylol and others added 7 commits October 13, 2026 01:11

eval: wrap the e2e_block docstring

9c79b96

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

zzylol mentioned this pull request Oct 13, 2026

Top-k precision estimate is optimistic: per-seed CMS collisions miss the target in e2e #211

Open

milindsrivastava1997 and others added 2 commits October 13, 2026 12:17

eval: save ASAPQuery SQL in structured e2e queries

cd51be9

zzylol mentioned this pull request Oct 13, 2026

eval: §6.3 workloads re-planned at 10 samples/s for the end-to-end run #212

Merged

milindsrivastava1997 deleted the branch main October 14, 2026 01:34

milindsrivastava1997 closed this Oct 14, 2026

milindsrivastava1997 reopened this Oct 14, 2026

milindsrivastava1997 changed the base branch from feat/e2e-multigroup-export to main October 14, 2026 01:42

zzylol merged commit 0cc2045 into main Oct 14, 2026
2 checks passed

zzylol added a commit that referenced this pull request Oct 14, 2026

Merge origin/main (#210 squashed) into eval/e2e-rate10-plans

7e0bee8

# Conflicts: # rqe-optimizer/README.md # scripts/export_autosketch_eval_table.py

zzylol mentioned this pull request Oct 14, 2026

study: denser KLL/DDSketch grid and Hydra W 65536; §6.3 rerun #213

Merged

Navigation Menu

Repository navigation

eval: drop classic templates 7 and 9 (rates by job, top-k by rate) - #210

eval: drop classic templates 7 and 9 (rates by job, top-k by rate)#210
zzylol merged 10 commits into
mainfrom
eval/drop-group-rate-templates

zzylol commented Oct 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

zzylol commented Oct 13, 2026

Summary

Changes

Results, before → after (no conclusion flips)

Testing

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants