Export the AutoSketch vs ASAP evaluation table from saturation curves

rqe-optimizer/data/autosketch-eval/README.md

@@ -0,0 +1,103 @@
+# AutoSketch vs ASAPQuery evaluation table
+
+`table.json` holds the accuracy and cost evidence for the AutoSketch vs
+ASAPQuery planner evaluation (plan: ProjectASAP/ASAPQuery#777, section 6). It
+is generated by `scripts/export_autosketch_eval_table.py`.
+
+## Contents
+
+Workloads (W2): one per dataset in ASAPQuery's
+`asap-tools/dataset-analysis/results/skew_summary.csv`.
+
+| Dataset | RQEs | Config rows | Saturated rows |
+|---|---|---|---|
+| alibaba_v2022 | 51 | 1020 | 867 |
+| boom | 20 | 140 | 47 |
+| google_2011 | 27 | 546 | 403 |
+
+Each RQE is one (dataset, query_id, kind, range). Its data parameters are the
+worst case over the **whole dataset** (`worst_theta_cms`, `worst_K`, `max_N`,
+`worst_alpha_rank`/`worst_alpha_memory`), rounded to the measured grid toward
+the harder side by `recommend_config.grid_point`. There is no burst variant:
+these whole-dataset worst cases replace AutoSketch's synthetic burst injection.
+
+Each RQE lists every measured config of every candidate family (keys: CMS,
+CountSketch, CMS-heap top-k; values: KLL, DDSketch), with:
+
+| Field | Meaning |
+|---|---|
+| `autosketch_error` | Single-sketch curve at `N = max_N`. AutoSketch keeps one sketch per query window and never merges, so it has no saturation requirement. |
+| `asap_error_by_m` | Error after merging `m ∈ {1, 4, 16, 64}` shards, at `N = max_N`. CMS, CountSketch, HLL and DDSketch merge exactly: every `m` reads the single-sketch curve. KLL and top-k read the merged curve at `m` (sketch-bench #131); a missing one falls back to the single-sketch curve, flagged "single-sketch curve optimistic". |
+| `saturated` | `max_N ≥ n_sat`. ASAP merges smaller-window sketches, so it may only use a config's value when this is true. |
+| `n_sat`, `not_saturated_in_grid` | From `saturation.csv`; the 1e9 subset wins when it has the point. |
+| `memory_bytes`, `insert_ns_per_item`, `merge_us_per_fold`, `query_phase_us` | `recommend_config.cost_of`. `query_phase_us` is the benchmark's whole query phase, not one query. |
+| `*_flag*` | Grid and interpolation flags: "N below grid", "extrapolated beyond N=…", "cost scaled from rows=3". |
+
+Per family: `error_metric`, `higher_is_better`, `target` (from
+skew_summary), and the grid point used.
+
+## Label sets and windows (assumptions)
+
+`rqe-optimizer` needs `card(ℓ)` (sketch instances) and `λ(ℓ)` (items/s).
+skew_summary.csv describes one population per query, so:
+
+- **`groups = 1`.** A key query (`sum by (k)`) is one sketch whose keys are
+  the group-by values (`keys_per_window = worst_K`). It is not one sketch per
+  group. Value queries have no `by`.
+- **`arrival_rate_per_sec = max_N / lookback_secs`**: the rate in the worst
+  window.
+- **Instant queries:** `lookback_secs = step_s`, because `max_N` counts one
+  step's samples.
+- **BOOM** has no timestamps, window or N in skew_summary, so
+  `boom_query()` derives them from its 20 equal chunks per series:
+  - `N = rows_total / 20`;
+  - `lookback = interval` = one chunk's duration, from the sampling step in
+    the series name (`ds-2187-H` is hourly);
+  - a series without an alpha fit uses the hardest grid alpha for error and
+    the easiest for memory.
+
+  Each RQE's `assumptions` field lists what was assumed.
+
+## Regenerate
+
+The raw curves (2.8 MB) are not committed. Produce them with
+`scripts/study_saturation.py`: the grid from #130 (`--phase all`), the 1e9
+subset, and `--merge-shards-list 1,4,16,64` from #131 for top-k and KLL/DDSketch.
+Then run:
+
+```sh
+python3 scripts/export_autosketch_eval_table.py \
+  --summary <ASAPQuery>/asap-tools/dataset-analysis/results/skew_summary.csv \
+  --curves out_grid_1e7_cost/saturation_curve.csv \
+  --curves-big out_1e9/saturation_curve.csv \
+  --saturation out_grid_1e7_cost/saturation.csv \
+  --saturation-big out_1e9/saturation.csv \
+  --merge-curves out_merge_topk/saturation_merge_curve.csv \
+                 out_merge_quantile/saturation_merge_curve.csv \
+  --revision "curves: sketch-bench #130 study_saturation.py; merge curves: sketch-bench #131 (branch merged-accuracy)" \
+  --out rqe-optimizer/data/autosketch-eval/table.json
+```
+
+The merge curves were produced on #131's branch at a0b8b42. #131 has since
+merged into main as bd644fe with identical content, so the recorded revision
+string above still describes the inputs.
+
+`table.json` records the SHA-256 of every input under `inputs`:
+
+| Input | SHA-256 |
+|---|---|
+| skew_summary.csv (ASAPQuery 46f3942) | `f5984b655e7720449fdf081dd4c0a8d883b5bf2a9fd968f379fa8d44df364382` |
+| out_grid_1e7_cost/saturation_curve.csv | `2d34a925c5bf8e4e72cab5932f63f8ec6e9cdf3e57b07f21aae2e5192144ca1a` |
+| out_grid_1e7_cost/saturation.csv | `569941ea162698963671d2a0ad40faeff53614d3cdb374de2a3b8aa3917ae674` |
+| out_1e9/saturation_curve.csv | `fb8fc62a16e137b831c20c16fe8d9b846e3fe80eb468451f08a727514d1e2bfd` |
+| out_1e9/saturation.csv | `0dd8448349f63317d7f4d385ce3063ab78d8daf3504015f580b7147cd22a3f22` |
+| out_merge_topk/saturation_merge_curve.csv | `ec43995c8b9ac0862fe5c19f038693fbce6e73cf48ebf46fb3bacbfb474d4ef6` |
+| out_merge_quantile/saturation_merge_curve.csv | `d4fb544bf14b06274f2558632faa766b5601ba4f942b48119f45c04aa39898f8` |
+
+## Limits
+
+- Merged curves cover `N ≤ 1e7` and `m ≤ 64`. A deployment with `m > 64`
+  has no measured KLL/top-k value.
+- 530 config rows extrapolate beyond the largest measured N, and 56 rows
+  sit below the grid's smallest N; both are flagged.
+- 720 rows=5 configs take 5/3 of the rows=3 cost (flagged).