Repository navigation
feat(asap-tools): add accelerated random E2E workload artifacts - #879
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 10 samples/s a 1s per-series quantile reads 10 samples, below the N = 1,000 where sketch-bench's accuracy curves start, so no saved plan can serve it. 100s reads exactly 1,000. Re-freezes seed 862 and the workload 01 pilot plan; other queries are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`avg by (G) (f(x[w]))` of a per-series function decomposes to the one leaf `f(x[w])` and is written as that leaf's aggregation plus a query-time avg, as the legacy planner writes it. Averaging per-series quantiles from sum and count leaves would be wrong. A binary arm can't run the stage, so such an arm is still refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces the legacy-planner pilot plan with sketch-bench's saved plan (eval table from --random-manifest at sketch-bench 85699c9, autosketch_vs_asap p95) converted by --planner saved: the coarse sum rolls up from the fine sum, the quantiles share one per-series DDSketch with a query-time avg. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pilot is planned by sketch-bench and --planner saved, which reads its queries from the eval table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ClickHouse renderers take a Query; these shapes are built from RQEs, not parsed from PromQL. A per-series quantile reads a view keyed by every label, and top-k by keeps k rows per group (LIMIT k BY). Raw SQL matches the Prometheus reference and MV SQL matches raw in clickhouse-local. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each RQE maps to the ClickHouse renderers' Query, so the manifest's clickhouse_sql is the same SQL the MV arm is checked against: windows floored to the eval time, Prometheus 2.53 boundaries, top-k per group. The cardinality RQE now names its table, events. Re-freezes seed 862; PromQL is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--pilot-dir writes, under config/experiment_type/random_workload/<id>: ASAPQuery (the archived saved plan) and Prometheus, VictoriaMetrics, raw ClickHouse (the manifest's clickhouse_sql) and ClickHouse+MV (the same RQEs over exact views). Every arm feeds the eval table's metrics with the manifest's feeder seed and evaluates on one schedule: the longest lookback plus the freshness delay, then 10 min aligned to each interval. Replaces pilot_profile and pilot_e2e_table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Composes each of the four configs through Hydra and runs its runner's checks before any service starts: experiment config, paced feeder, and the client configs of each mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-search.latencyOffset defaults to 30 s, so queries sent sooner than that after their window closes miss its last samples. Prometheus has no such delay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 15, 2026
Each exported (sketch-bench f885d7e), solved, and converted by --planner saved like workload 01; none drops an RQE. Tests check every archived plan answers its manifest's pilot queries and every workload renders its run configs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
clickhouse-server 26.6 and later die with an illegal instruction on CloudLab's Xeon E5-2450 nodes (no AVX2), so both ClickHouse arms failed to start. The ClickHouse semantics tests pass on 23.8 too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
compare_arms.py reads each arm's mode directory and prints per-query latency and the share within the repeat interval, agreement with a reference arm (mean relative error, or precision@k for top-k), and each arm's total CPU-seconds and peak memory from remote_monitor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts only where #894 reformatted older versions of the random workload renderer and its tests; kept this branch's versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
milindsrivastava1997
marked this pull request as ready for review
October 15, 2026 04:07
milindsrivastava1997
added a commit
to ProjectASAP/sketch-bench
that referenced
this pull request
Oct 15, 2026
* eval: export ASAPQuery random workload manifests --random-manifest writes the eval table of an ASAPQuery random workload (asap-tools random_workloads.py, ProjectASAP/ASAPQuery#879): its common-promql queries as schema RQEs on metric `data`, so a fine sum can serve a coarse one, quantiles per series (avg of per-series quantiles), and top-k heaps below the curves' 1,000 keys read there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: address review of --random-manifest - closed_labels() takes a label list, so random manifests reuse it: a child-only grouping counts its parents' groups instead of failing. - Other capabilities are left out (logged); a quantile that isn't an avg of per-series quantiles, an accuracy_sla other than 0.95, and a rate that doesn't divide 1000 are refused. - RQE ids carry the exact range; duplicate ids are refused. - --random-manifest refuses the flags it would ignore. Workload 01's table is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: one RQE for random-manifest queries reading the same one Per-series quantiles that differ only in their avg's grouping read the same per-series RQE; ASAPQuery's --planner saved refuses one leaf naming two RQEs. They now share the first query's id. Workloads 03, 04, 06, 08 and 09 of seed 862 hit this; workload 01's table is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: address second review of --random-manifest - RQEs below the curves' first N (1e3 items per group) are left out (logged), as schema_queries() does. - A sub-second interval is a ValueError, not an assert; a bool rate is refused; divides_1000() is shared with --samples-per-sec. - A repeated query is one e2e entry. - The curve flags are refused with --random-manifest. - Each RQE entry is built once; one table writer for both modes. Seed 862's ten tables are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: address third review of --random-manifest - The curves' first-N check skips exact families, which read no curve. - A lookback that isn't whole scrapes, or a query reading other than one RQE, is a ValueError naming the query. - check_unique_ids() is shared with build(); the e2e dedup is a set. Seed 862's ten tables are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * eval: RQE expression trees for random-manifest tables (#228) * eval: RQE expression trees in e2e tables Every e2e query gets `expr`, its RQE expression tree: a leaf per RQE it reads (id, metric or table, capability, lookback_ms, grouping, phi/k/value), and a `/` node for template 10. Rate queries, and schema queries merged across several RQEs (--keep-duplicate-queries), get none. Regenerate the e2e-rate10 tables: only `expr` is added, so the RQE ids and plans are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1 * eval: RQE expression trees for random-manifest tables --random-manifest e2e queries carry `expr` (#227, ProjectASAP/ASAPQuery#886): avg by (G) of per-series quantile_over_time is an avg over a per-series quantile leaf; topk by (G) of sum_over_time a per-series top-k leaf with `topk_by` G (left out for a global top-k); sum by (G) of sum_over_time and spatial quantile by (G) a leaf at G; SQL cardinality a leaf on its table with `value`. Counter-gated top-k and rate have none. Leaf ids are the table's RQE ids; values come from the manifest's RQE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1 * eval: leaf capabilities as a set, renaming only topk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1 * eval: random-manifest trees from RQE fields, not PromQL random_expr() builds each tree from the manifest RQE's capability, grouping_labels, phi and k, as asap-tools' rqe_query() renders them: quantile is avg by G of a per-series quantile leaf, topk_by_value a per-series top-k leaf ranked within G, sum a leaf at G. No PromQL regexes, so no list-order label comparison; the spatial quantile shape (read as a group quantile from a per-series RQE) and the unreachable SQL cardinality branch go. A test builds seed-862 workload-01's table and checks every tree names its query's RQE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--planner savednow plansavg by (G) (f(x[w]))of a per-series function as the per-series leaf plus a query-time avg, as the legacy planner does. Other avg shapes are still refused, and so is this one inside a binary arm.--random-manifest) and converted by--planner saved. The coarse sum rolls up from the fine sum, and the three quantiles share one per-series DDSketch. The eval table, plans, and configs are archived inrandom_workloads/seed-862/workload-01-pilot/.Validation
python3 -m unittest discover -s tests -p "test_*.py"inasap-tools/experiments: 181 pass.cargo testinasap-planner-rs: all pass, with new tests for per-series avg on the MILP-output and saved paths.e2e_avg_of_per_series_quantile_over_time: per-series KLL windows merged over the range, then averaged by group at query time, give the exact expected values.paced_feeder --verify-factspasses on workload 01's feeder table.Remaining
assumptions.🤖 Generated with Claude Code