eval: instant RQEs read one scrape, not 1 s

[milindsrivastava1997]

Summary

A PromQL instant query (sum by (job) (data), quantile by (job) (0.9, data)) reads one sample per series. The synthetic spatial templates looked back 1 s whatever the data rate, so each per-job sketch held 2e5 items at 200 samples/s and 1e4 at 10/s instead of the 1e3 series in a job. Raising the feed rate made instant RQEs harder, unlike PromQL.

• export_autosketch_eval_table.py: the 6 classic instant RQEs (and the dashboard's quantile-by-job) look back one scrape interval, 1 / samples_per_sec_per_series s: 5 ms at 200/s, 100 ms at 10/s. N = 1e3 per job at any rate. Their ids still end in /1s (the step), so e2e queries and ids are unchanged.
• autosketch_vs_asap: the runner computes N from the scrape interval and series count, not the table's max_N, and scraped every stream once a second. A 100 ms lookback held 0 scrapes and had no candidate windows. A metric with samples_per_sec_per_series (the synthetic data/data_total) is now scraped at that rate, with series = the real 1e4. Record metrics (http, flows) keep 1 s. Temporal RQEs see the same N and ingest rate as before, but their candidate windows are now every divisor of the lookback in scrapes, so sub-second windows are candidates too (see "Candidate windows at 200/s").
• Rate guard: --samples-per-sec (exporter) and samples_per_sec_per_series (runner) must divide 1000, so a scrape is whole ms. 3/s, 0 and 2000/s are rejected.
• Plan output: window_secs/slide_secs are fractional for sub-second deployments (they truncated to 0), and integers otherwise, so existing plans serialize byte for byte.

Results: 10 samples/s replan

Classic, multigroup and all at shared = 1 (--samples-per-sec 10), --runs 1, saturation dir study_merged (a local merged study, not the committed autosketch-vs-asap-inputs). The baseline is main's 10/s tables (the 1 s window) on the same inputs and runner, so only the instant window differs.

Cheapest plan, no latency bound:

Workload	ASAP CPU (vCPU)	ASAP GiB	ASAP deployments	PerQuery, AutoSketch
classic	0.296 → 0.309	15.37 → 15.27	6 → 7	unchanged
all	0.376 → 0.390	16.74 → 16.64	15 → 16	unchanged
multigroup	unchanged (no instant RQEs)

• Only the 6 instant RQEs and sum_by_job_sum_over_time move. That RQE rode the instant 1 s exact-sum deployment and now gets its own 60 s one (+1 deployment, +4% ASAP CPU).
• quantile_by_job keeps DDSketch α = 0.02 at N = 1e3.
• Tightest feasible latency bounds are unchanged (classic 20.7 ms, multigroup/all 407.5 ms). All RQEs are served, none dropped.
• AutoSketch and PerQuery costs don't move: the cost model charges ingest per sample whatever the window holds.

Candidate windows at 200/s

The full scrape-grained candidate set is kept, not restricted to whole seconds: the engine runs sub-second windows, and more candidates can only lower ASAP's cost. Measured on classic, all and multigroup at shared = 1, 200/s, study_merged, --runs 1:

Workload	Runner, table	ASAP candidates	Candidate build (s)	MILP solve (s)	ASAP CPU, cheapest	Temporal sub-second windows chosen
classic	main, committed	420	0.18	0.050	2.224	0
classic	this PR, committed	420	0.77	0.048	2.224	0
classic	this PR, regenerated	420	0.78	0.047	2.549	0
all	main, committed	574	0.30	0.077	2.412	0
all	this PR, committed	574	0.94	0.076	2.412	0
all	this PR, regenerated	574	0.88	0.075	2.738	0
multigroup	all three	154	0.12–0.14	0.03	0.188	0

• Committed tables plan identically. On the committed 200/s tables (1 s instant lookback), main's runner and this PR's give byte-identical plans for all three workloads. The sub-second windows are enumerated, but the reported candidate count is unchanged (420 and 574 both ways) and so is the MILP solve time.
• Only candidate build time grows, about 3–4× (classic 0.18 → 0.77 s), from enumerating the extra divisors. MILP solve time is unchanged. ASAP's planning time stays under 1 s, against AutoSketch's 600 s of benchmarking at the paper's rate, so the comparison doesn't move. The committed planning_secs won't reproduce exactly until the tables are rerun.
• At 200/s the instant RQEs get 5 ms windows. ASAP's CPU rises ~15% (classic 2.22 → 2.55 vCPU) because sum_by_job_sum_over_time no longer shares the instant 1 s deployment (+1 deployment).

No result files are committed: study_merged isn't in the repo (and differs from #213's inputs), so rqe-optimizer/results/e2e-rate10/ stays as #212 left it until we replan on committed inputs.

Design decisions (please approve)

1. An instant RQE's lookback is one scrape interval, 1 / samples_per_sec_per_series s. One sample per series, as PromQL reads it, so N stops depending on the rate. The alternative was keeping 1 s and calling it "everything in the last second" in the paper.
2. The runner scrapes a metric at its table rate when the table gives one, else 1 s. It's needed so a sub-second window holds whole scrapes. It also keeps temporal RQEs' N and ingest identical, and leaves record metrics (no per-series rate) and trace tables as they were.
3. RQE ids keep the /1s suffix. It's the step, so e2e query → RQE mappings and downstream consumers don't change. The lookback lives in lookback_secs.
4. Sub-second windows/slides serialize as fractional seconds, whole seconds as integers. The engine supports sub-second windows (confirmed), so instant RQEs deploy on 100 ms (10/s) or 5 ms (200/s) tumbling windows.
5. Candidate windows are every divisor of the lookback in scrapes, not restricted to whole seconds. That's the engine's true grain. It costs 3–4× candidate build time (still under 1 s) and, measured, changes no plan from a committed table.
6. The samples/s rate must divide 1000 (exporter and runner), so the scrape interval is whole ms.
7. The committed §6.3 tables aren't regenerated here. They plan identically under the new runner; regenerating them (instant RQEs at one scrape) belongs with study: denser KLL/DDSketch grid and Hydra W 65536; §6.3 rerun #213's §6.3 rerun.
8. The exporter tests now expect the new instant values. The byte-for-byte test patches the committed tables' instant RQEs (lookback 5 ms, N = 1e3) before comparing, and checks nothing else changed.

Caveats

• N = 1e3 per job sits exactly on the accuracy curves' first N. At fewer than 1e3 series per job, instant quantiles fall off the curves and become unservable.
• Ingest is still charged for every sample, although an instant query only needs the latest one per series.

Testing

• cargo test --release -p rqe-optimizer and cargo clippy -D warnings pass.
• scripts/test_export_autosketch_eval_table.py: 30 passed (a new test rejects rates that don't divide 1000).
• Trial merge with study: denser KLL/DDSketch grid and Hydra W 65536; §6.3 rerun #213 (eval/dense-grid) is clean.

🤖 Generated with Claude Code

[milindsrivastava1997]

1_000 / per_sec is integer division with no guard on the rate.

• A rate that doesn't divide 1000 truncates. At --samples-per-sec 3, scrape_ms is 333. The temporal lookbacks (60000) and interval (1000) aren't multiples of 333, so lib.rs validation rejects every synthetic RQE. The same happens at 7, 30 and 300/s.
• Above 1000/s, scrape_ms is 0 ("scrape interval is 0").
• samples_per_sec_per_series: 0 panics with divide-by-zero.

This needs a check that the rate divides 1000, here and in the exporter.

[milindsrivastava1997]

Fixed in 813204b: the runner asserts that samples_per_sec_per_series is > 0 and divides 1000, so the scrape interval is whole ms. 3/s, 7/s, 0 and anything above 1000/s are rejected with a message naming the metric.

[milindsrivastava1997]

The committed input tables (results/autosketch-vs-asap-hydra/inputs/tables/*.json, e2e-rate10/inputs/tables/*.json) still have the old 1 s instant lookback, but they already carry samples_per_sec_per_series (200/s and 10/s). This line now reads them with a 5 ms / 100 ms scrape.

Instant RQEs then get candidate windows of 5 ms, 10 ms, … up to 1 s instead of only 1 s, and temporal RQEs get sub-second windows too. So the candidate sets, MILP and plans no longer match the committed results built from these same inputs, and nothing flags the mismatch.

Two options: regenerate those tables, or keep SCRAPE_MS when the table's lookback is the old one.

[milindsrivastava1997]

Measured: on the committed 200/s tables (classic, all, multigroup at shared = 1), main's runner and this PR's give byte-identical plans. The sub-second windows get enumerated, but the reported candidate count is unchanged (420 / 574 / 154) and so is the MILP. What doesn't reproduce is candidate build time (see the other thread). So I'm keeping the finer grain rather than special-casing old tables. Regenerating the committed tables with one-scrape instant RQEs goes with #213's §6.3 rerun. Numbers are in the PR description, "Candidate windows at 200/s".

[milindsrivastava1997]

With a sub-second scrape, temporal RQEs on synthetic streams also change, which contradicts the PR's claim that they don't. At 200/s, a 60 s lookback gives divisors(12000) = 48 candidate windows (5 ms … 60 s) instead of 12, and 24 h gives 220 instead of 96.

• If the MILP picks, say, a 500 ms window/slide, the temporal plan emits a fractional window_secs, which ASAPQuery may not accept (design decision 4 is still unconfirmed).
• The larger candidate set also inflates candidate_build_secs / milp_solve_secs / planning_secs, which are used to compare AutoSketch with ASAP.

[milindsrivastava1997]

Agreed, my claim was wrong; the description now says temporal RQEs get scrape-grained candidates. Keeping them deliberately: the engine supports sub-second windows, so that's the real search space, and more candidates can only lower ASAP's cost. Measured at 200/s: the MILP picks no sub-second window for any temporal RQE, solve time is unchanged, and candidate build time grows ~3–4× (classic 0.18 → 0.77 s, all 0.30 → 0.94 s). ASAP's planning time stays under 1 s against AutoSketch's 600 s of benchmarking, so the planning-time comparison doesn't move. This is design decision 6 in the description.

[milindsrivastava1997]

1 / SYNTHETIC_SAMPLES_PER_SEC is float seconds, and nothing checks that it's a whole number of ms. The runner truncates it with (lookback_secs * 1000.0) as Millis, so at --samples-per-sec 3 or 7 the lookback becomes 333 / 142 ms. Neither lines up with the 1000 ms step or the temporal lookbacks, so the table can't be planned.

The exporter should reject rates that don't divide 1000.

[milindsrivastava1997]

Fixed in 813204b: the exporter rejects a --samples-per-sec that doesn't divide 1000 (parser.error). The new test test_samples_per_sec_must_divide_1000 covers 3, 0 and 2000.

[milindsrivastava1997]

If workload.e2e.metrics[metric] is missing (no e2e block, or a key mismatch: e2e keys swap / for _), scrape_ms silently falls back to 1000. An instant RQE with a 5 ms lookback then gets no candidate windows, so evaluate drops it as unservable before validate_facts runs. Only the dropped count is logged. This should fail loudly when the lookback/interval aren't multiples of the scrape.

[milindsrivastava1997]

Fixed in 460e290. The runner now asserts that each RQE's lookback and interval are multiples of its stream's scrape, before it builds candidates. Checked by deleting e2e.metrics.data from a 10/s table: it now panics with …/sum_by_job/keys/1s: lookback 100 ms and interval 1000 ms must be multiples of …/job/value's 1000 ms scrape instead of dropping the RQE.

[milindsrivastava1997]

(line 696) The docstring says the runner does not read the e2e block, but it now reads samples_per_sec_per_series from it for the scrape interval. test_classic_tables_are_main_s_all_tables_byte_for_byte deletes e2e before comparing (and its comment repeats the claim), so dropping or renaming that field wouldn't fail any test.

[milindsrivastava1997]

Fixed in 460e290. The e2e_block docstring and the test comment now say the runner reads samples_per_sec_per_series. Before the byte-for-byte test drops e2e, it asserts that every RQE metric has samples_per_sec_per_series == 200. Checked: renaming the field fails the test.

[milindsrivastava1997]

(line 519) as Millis truncates. Lookbacks are now fractional seconds (1/rate), so a value like 0.0049999 would become 4 ms, which isn't a multiple of the 5 ms scrape. Use .round() here (and for interval_ms).

[milindsrivastava1997]

Fixed in 460e290: lookback_ms and interval_ms are .round()ed. No rate that divides 1000 truncates wrongly today (I checked all of them), but with the new assert a mis-rounded value would now fail loudly anyway.