eval: §6.3.1 case studies swept one knob at a time

[milindsrivastava1997]

Summary

Exploration for the §6.3.1 case studies: where does the planner's plan stop being the obvious one? This PR rebuilds both case studies (performance monitoring, anomaly detection) from the paper draft's text as autosketch_vs_asap tables. It sweeps one knob at a time and records every ASAP plan; only ASAP is reported (the result files still hold the runner's baselines). The base workloads reproduce the draft's ASAP plans exactly (perf 0.105 vCPU, 2 MVs; anomaly 0.114, 3 MVs).

• scripts/case_study_tables.py: one table per sweep point (38), plus plan.tsv with each table's SLA grid.
• scripts/summarize_case_studies.py: ASAP's cost, MVs and latency per SLA, and every distinct plan → summary.md.
• rqe-optimizer/results/case-studies/: tables, results (with chosen), summary.md, and a README.md with findings, suggested §6.3.1 text, caveats and Reproduce.

Findings (ASAP vCPU, MVs)

1. Perf, latency SLA builds a hierarchy of windows. The cheapest plan merges 5,760 15 s windows for the 24h query (23.8 s batch latency). At 1 s, the 6h/24h queries get a 30 min tumbling MV while 30m stays on 15 s (0.126, 3). At 100 ms: 15 s / 2 min / 1 h windows (0.229, 4).
2. Perf, refresh interval of the long dashboard. 6h/24h every 30 min: 2 MVs (0.105). Every 1 min: a 2 min / 1 min sliding MV for 10m–24h (0.216, 3).
3. Perf, less data, more MVs. 200 samples/s per series: 2 MVs. 50/s: 3. 20/s: 4. Ingest scales with rate, merging doesn't, so at low rates extra MVs pay for themselves.
4. Anomaly, one more query makes the plan cheaper. Adding a drill-down by (region, service, endpoint, status) lets one finest MV serve all 7 queries: 0.114 (3) → 0.052 (1) at 25 endpoints. At 100 endpoints (40k groups), rolling up every 15 s costs more than a second MV, and the planner splits the coarse queries off to (service, region) (0.078, 2).

No plan change: series count, anomaly ingest rate, one shared anomaly interval. Status cardinality and endpoints without the drill-down only reshuffle by (service) between MVs (< 1% cost).

Design decisions (please approve)

• Workloads rebuilt from the draft's text, not from a colleague's config: same queries, 1e4 series in 10 jobs at 200 samples/s for perf, and the http schema (4 regions, 25 services, 25 endpoints/service, 4 statuses, 2e6 samples/s) for anomaly.
• Runner tables, not a new Rust example: autosketch_vs_asap already plans ASAP with and without an SLA, so no new planner code.
• Only ASAP is reported in the README and summary.md; the case studies are about why the planner's plans are what they are.
• Anomaly metric named requests, not http: no Hydra grid is a candidate, since Hydra's accuracy is measured at http's schema only and the sweeps change it.
• The drill-down covers groups holding ≥ 1% of requests: at every group, the smallest 4-label groups fall below the curves' first N (1e3).
• CPU weights only (--weights cpu), as in the draft. Memory is free, so the base perf plan holds 18 GiB of windows.
• Batch latency SLA (the runner's version 2) stands in for "a latency constraint". Per-case grids sit around each plan's latency: perf 10 ms–10 s, anomaly 0.3–3 ms, drill-down 3–100 ms.
• Sweep ranges stop where a query falls below the curves: ≥ 20 samples/s per series, ≥ 1e4 series, anomaly ingest ≥ 2.5e5/s, ≤ 16 statuses.
• --runs 1: costs are deterministic; planning times are not reported.

Verification

• The base tables reproduce the draft's ASAP numbers in both case studies.
• case_study_tables.py regenerates the committed tables byte for byte.
• Every one of the 38 tables plans all its queries (none dropped as unservable).

🤖 Generated with Claude Code