Skip to content

feat(asap-tools): render sketch-bench e2e queries as ClickHouse SQL - #844

Open
zzylol wants to merge 10 commits into
mainfrom
feat/e2e-sql-from-table
Open

zzylol wants to merge 10 commits into
mainfrom
feat/e2e-sql-from-table

Conversation

@zzylol

@zzylol zzylol commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Builds on #837 (PromQL → ClickHouse SQL for generated2, merged); now based on main.

Summary

Renders a sketch-bench #777 workload table's e2e.queries as ClickHouse SQL, for the ClickHouse baseline of the end-to-end runs (#824). The new datasets/fake_metric/sketch_bench_workloads.py sits next to #837's generate_workloads.py and shares its extrapolatedRate (split out as rate_from_samples, whose generated text is unchanged).

  • Classic templates (PromQL): parsed into a Query record, adding the shapes feat(asap-tools): run generated2 PromQL workloads on ClickHouse #837 lacks: topk(k, sum by (x) (...)), sum by (x) (rate(...)), quantile by (x) (q, instant), the ratio of two quantile_over_time, and the per-series forms.
    • Windows follow Prometheus 2.53, which the bare-metal experiments install (v2.53.2): range selectors are closed, [T−d, T], and instant vectors take each series' latest sample in [T−1s, T]. Measured on CloudLab against a real Prometheus 2.53: its [5s] sums match ClickHouse's ts >= T−5s AND ts <= T exactly, while (T−5s, T] misses the boundary samples. The paced feeder's timestamps and aligned eval times always put samples on that boundary.
    • Note for feat(asap-tools): run generated2 PromQL workloads on ClickHouse #837: generated2 still reads (T−d, T] (left-open is Prometheus 3.0). With 1 s scrapes on whole seconds, a [1m] window reads 60 samples where Prometheus 2.53 reads 61. This PR leaves it unchanged; it's your call there.
  • Multi-grouping templates (structured entries, sketch-bench#209): uniqExact(value) for cardinality and quantileExactInclusive(q)(value) pooled over every sample of a group. Windows are ASAPQuery's left-open (E−W, E], since no Prometheus is involved and ASAPQuery's SQL frontend and precompute windows are (start, end].
    • Also writes <table>/asapquery.sql for the ASAPQuery arm: each entry's sql from the table as written (sketch-bench#209 renders the ASAPQuery dialect once, QUANTILE(q, col), COUNT(DISTINCT col), BETWEEN DATEADD(s, -W, NOW()) AND NOW()), with only NOW() replaced by the eval time floored to T. All 936 distinct queries of the 12 tables parse in ASAPQuery's SQL frontend to (⌊E⌋−W, W).
  • Eval time: {{ end_time }} is floored to a multiple of T in the SQL, matching the aligned query client (feat(asap-tools): run sketch-bench workload tables end to end with the paced feeder #841).
  • Output: one experiment_run_clickhouse.py config per table, one query group per interval_secs (repetition = T), plus init.sql with the tables (gauge/counter: ts, labels, value; records: ts, labels, value fields). Unknown shapes, capabilities or labels raise.

Testing

  • Python reference: a separate implementation of the PromQL 2.53 semantics, and of the pooled multigroup semantics, over small datasets.
    • Classic: 6 series at about 4 samples/s on ms timestamps, with a late-starting counter, a series that stops and a 3 s gap.
    • Multigroup: a skewed http records table with endpoint a child of service.
  • Coverage: 18 classic shapes and template 16's full CUBE plus 7 cardinality groupings, each at several eval times, all run in one clickhouse-local process. Every SQL result matches the reference within 1e-9, including empty windows and every extrapolatedRate branch.
  • Mutation checks: a left-open window on classic, or a closed one on multigroup, makes 84–252 subtests fail.
  • Real tables: every statement for sketch-bench's classic (108), multigroup (104) and all-metrics8 (1696) tables executes in clickhouse-local.
  • ASAPQuery dialect: all 52 multigroup queries, rendered at aligned and unaligned end times, parse with ASAPQuery's SQLPatternParser/SQLPatternMatcher to (start, duration) = (⌊E⌋−W, W).
  • CLICKHOUSE_BINARY=<path> python3 -m pytest -q tests/test_sketch_bench_workloads.py passes (the ClickHouse tests are skipped without a binary). The experiments suite passes. black/flake8 clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

milindsrivastava1997 and others added 8 commits October 12, 2026 13:34
…nerated2 workloads

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e SQL configs

Adds a client_options.simulated_start_time so templated SQL can be evaluated
against bulk-loaded data with fixed timestamps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… configs

sketch_bench_workloads.py reads a sketch-bench table's workloads[0].e2e block
and writes, per table, the metric tables' init SQL, one SQL file per query
interval and a baseline experiment_run_clickhouse.py config whose query
groups repeat every interval.

Queries keep generate_workloads.py's Prometheus 2.53 semantics (range
vectors read (T - d, T], extrapolatedRate for rate/increase,
quantileExactInclusive) and add the classic template shapes: topk over
sum by, sum/count/quantile by over instant vectors (latest sample per series
in (T - 1s, T]) and rates, and the ratio of two quantile_over_time. Eval
time is {{ end_time }} floored to the interval. Unsupported shapes and
structured (non-PromQL) entries raise.

generate_workloads.extrapolated_rate now builds on rate_from_samples, which
the new module reuses; its SQL output is unchanged.

The test compares every generated statement in clickhouse-local against a
Python implementation of the Prometheus semantics at several eval times,
covering empty windows and every extrapolation branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
… SQL

Experiments run Prometheus 2.53.2 (installation/prometheus/install.sh),
whose range selectors and instant lookback are closed, [T - d, T]; left-open
windows arrived in 3.0. Aligned eval times put a sample on the left
boundary, so the SQL now reads ts >= T - d. The Python reference selects
[T - d, T] as well.

generate_workloads.py's generated2 SQL is unchanged; only the shared
rate_from_samples docstring no longer names a window shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
Structured e2e entries {table, value, capability, grouping, quantile,
window_secs, interval_secs} become pooled queries over a records table.
Cardinality is uniqExact of the value field per group, and quantile is
quantileExactInclusive over all of a group's samples. Both read the same
closed [T - W, T] window, with T floored to the interval as the PromQL
queries are. init.sql now creates records tables, with one Float64 column
per value field. Each table's structured queries are also written in
ASAPQuery's SQL frontend dialect (COUNT(DISTINCT col), QUANTILE(q, col),
BETWEEN DATEADD(s, -W, NOW()) AND NOW()) to asapquery.sql.

The clickhouse-local test adds a skewed records dataset in which endpoint is
a child of service. It checks template 16's CUBE over (region, service,
endpoint) and several cardinality groupings against a Python reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
Multigroup (structured) queries involve no Prometheus, so they now read
ASAPQuery's left-open window (E - W, E] instead of the closed [E - W, E]
that classic PromQL keeps to match Prometheus 2.53. ASAPQuery's SQL
frontend and precompute windows are (start, end], so ClickHouse and
ASAPQuery compute over the same samples. Query gets a `left_open` flag that
structured entries set; the raw SQL uses `ts > E - W`.

The ASAPQuery-dialect file ends its window at the floored eval time instead
of NOW(). The frontend cannot floor a literal, so the client's
{{ end_datetime }} is moved back by {{ end_time % T }} inside DATEADD:

  ts BETWEEN DATEADD(s, -{{ end_time % T + W }}, '{{ end_datetime }}')
         AND DATEADD(s, -{{ end_time % T }}, '{{ end_datetime }}')

which the frontend parses to (start, duration) = (E - W, W).

The Python reference for multigroup now pools records in (t - w, t].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
…se configs

The SQL floors {{ end_time }} to T, but the current query client sleeps T
after each repetition's queries finish, so its eval times drift by the
query latency and the arms answer different windows. Each query group now
sets `align_to_interval: true` and `query_time_offset: 10` (PR #841's
default). With #841's client, repetition k is evaluated at
floor((start - O) / T) * T + k * T and sent O seconds later.

The current client ignores `align_to_interval` (unknown client options
become unused attributes) and shifts its eval time back by the offset. The
module docstring records the drift until #841's client lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
…w SQL

ClickHouse 26.10 turns `ts > <integer>` on a DateTime64 key column into
`ts in [<integer> + 1, +Inf)` for index pruning, so the left-open multigroup
SQL dropped the samples in the window's first second from the result (A6:
raw vs MV relative differences of 1e-6 to 2.6e-3 on 49 of 52 queries).
The raw SQL now compares ts with toDateTime64(<bound>, 3) on both ends, for
classic and multigroup queries.

A regression test runs a left-open and a closed query in clickhouse-local
over a table with 8-row granules and samples every 7 ms around the window
start. Without the cast it reproduces the loss (8435 of 8571 rows).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
zzylol and others added 2 commits October 13, 2026 18:33
Brings in #837 (this branch's base, squash-merged) and #828. The one
conflict, generate_workloads.py (add/add), keeps this branch's version:
main's copy is #837's original commit, and this branch later moved
extrapolatedRate into rate_from_samples and closed its windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
sketch-bench's exporter now writes each structured e2e entry's ASAPQuery
SQL (`sql`, e.g. `... WHERE ts BETWEEN DATEADD(s, -300, NOW()) AND NOW()
...`), so it is the single ASAPQuery SQL renderer. asapquery.sql now takes
that SQL as written and only replaces NOW() with the floored eval time; this
module's own dialect renderer is gone. The ClickHouse raw SQL still comes
from the structured fields.

The frontend's DATEADD takes only NOW(), a literal or a CAST as its base, so
NOW() becomes a datetime literal floored in the client's Jinja template:
{{ end_datetime }} (UTC) with its seconds floored for T dividing 60, or its
minutes for 60 | T | 3600. Other intervals raise.

All 936 distinct structured queries in sketch-bench cd51be9's tables
(T = 60 s and 300 s), rendered through the client's QueryTemplate at three
end times, parse in ASAPQuery's SQL frontend to (start, duration) =
(floor(E) - W, W) and pass its matcher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
@zzylol
zzylol changed the base branch from clickhouse-generated2-workloads to main October 13, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants