Repository navigation
Conversation
…nerated2 workloads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e SQL configs Adds a client_options.simulated_start_time so templated SQL can be evaluated against bulk-loaded data with fixed timestamps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… configs
sketch_bench_workloads.py reads a sketch-bench table's workloads[0].e2e block
and writes, per table, the metric tables' init SQL, one SQL file per query
interval and a baseline experiment_run_clickhouse.py config whose query
groups repeat every interval.
Queries keep generate_workloads.py's Prometheus 2.53 semantics (range
vectors read (T - d, T], extrapolatedRate for rate/increase,
quantileExactInclusive) and add the classic template shapes: topk over
sum by, sum/count/quantile by over instant vectors (latest sample per series
in (T - 1s, T]) and rates, and the ratio of two quantile_over_time. Eval
time is {{ end_time }} floored to the interval. Unsupported shapes and
structured (non-PromQL) entries raise.
generate_workloads.extrapolated_rate now builds on rate_from_samples, which
the new module reuses; its SQL output is unchanged.
The test compares every generated statement in clickhouse-local against a
Python implementation of the Prometheus semantics at several eval times,
covering empty windows and every extrapolation branch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
… SQL Experiments run Prometheus 2.53.2 (installation/prometheus/install.sh), whose range selectors and instant lookback are closed, [T - d, T]; left-open windows arrived in 3.0. Aligned eval times put a sample on the left boundary, so the SQL now reads ts >= T - d. The Python reference selects [T - d, T] as well. generate_workloads.py's generated2 SQL is unchanged; only the shared rate_from_samples docstring no longer names a window shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
Structured e2e entries {table, value, capability, grouping, quantile,
window_secs, interval_secs} become pooled queries over a records table.
Cardinality is uniqExact of the value field per group, and quantile is
quantileExactInclusive over all of a group's samples. Both read the same
closed [T - W, T] window, with T floored to the interval as the PromQL
queries are. init.sql now creates records tables, with one Float64 column
per value field. Each table's structured queries are also written in
ASAPQuery's SQL frontend dialect (COUNT(DISTINCT col), QUANTILE(q, col),
BETWEEN DATEADD(s, -W, NOW()) AND NOW()) to asapquery.sql.
The clickhouse-local test adds a skewed records dataset in which endpoint is
a child of service. It checks template 16's CUBE over (region, service,
endpoint) and several cardinality groupings against a Python reference.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
Multigroup (structured) queries involve no Prometheus, so they now read
ASAPQuery's left-open window (E - W, E] instead of the closed [E - W, E]
that classic PromQL keeps to match Prometheus 2.53. ASAPQuery's SQL
frontend and precompute windows are (start, end], so ClickHouse and
ASAPQuery compute over the same samples. Query gets a `left_open` flag that
structured entries set; the raw SQL uses `ts > E - W`.
The ASAPQuery-dialect file ends its window at the floored eval time instead
of NOW(). The frontend cannot floor a literal, so the client's
{{ end_datetime }} is moved back by {{ end_time % T }} inside DATEADD:
ts BETWEEN DATEADD(s, -{{ end_time % T + W }}, '{{ end_datetime }}')
AND DATEADD(s, -{{ end_time % T }}, '{{ end_datetime }}')
which the frontend parses to (start, duration) = (E - W, W).
The Python reference for multigroup now pools records in (t - w, t].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
…se configs
The SQL floors {{ end_time }} to T, but the current query client sleeps T
after each repetition's queries finish, so its eval times drift by the
query latency and the arms answer different windows. Each query group now
sets `align_to_interval: true` and `query_time_offset: 10` (PR #841's
default). With #841's client, repetition k is evaluated at
floor((start - O) / T) * T + k * T and sent O seconds later.
The current client ignores `align_to_interval` (unknown client options
become unused attributes) and shifts its eval time back by the offset. The
module docstring records the drift until #841's client lands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
…w SQL ClickHouse 26.10 turns `ts > <integer>` on a DateTime64 key column into `ts in [<integer> + 1, +Inf)` for index pruning, so the left-open multigroup SQL dropped the samples in the window's first second from the result (A6: raw vs MV relative differences of 1e-6 to 2.6e-3 on 49 of 52 queries). The raw SQL now compares ts with toDateTime64(<bound>, 3) on both ends, for classic and multigroup queries. A regression test runs a left-open and a closed query in clickhouse-local over a table with 8-row granules and samples every 7 ms around the window start. Without the cast it reproduces the loss (8435 of 8571 rows). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
Brings in #837 (this branch's base, squash-merged) and #828. The one conflict, generate_workloads.py (add/add), keeps this branch's version: main's copy is #837's original commit, and this branch later moved extrapolatedRate into rate_from_samples and closed its windows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
sketch-bench's exporter now writes each structured e2e entry's ASAPQuery
SQL (`sql`, e.g. `... WHERE ts BETWEEN DATEADD(s, -300, NOW()) AND NOW()
...`), so it is the single ASAPQuery SQL renderer. asapquery.sql now takes
that SQL as written and only replaces NOW() with the floored eval time; this
module's own dialect renderer is gone. The ClickHouse raw SQL still comes
from the structured fields.
The frontend's DATEADD takes only NOW(), a literal or a CAST as its base, so
NOW() becomes a datetime literal floored in the client's Jinja template:
{{ end_datetime }} (UTC) with its seconds floored for T dividing 60, or its
minutes for 60 | T | 3600. Other intervals raise.
All 936 distinct structured queries in sketch-bench cd51be9's tables
(T = 60 s and 300 s), rendered through the client's QueryTemplate at three
end times, parse in ASAPQuery's SQL frontend to (start, duration) =
(floor(E) - W, W) and pass its matcher.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #837 (PromQL → ClickHouse SQL for generated2, merged); now based on
main.Summary
Renders a sketch-bench #777 workload table's
e2e.queriesas ClickHouse SQL, for the ClickHouse baseline of the end-to-end runs (#824). The newdatasets/fake_metric/sketch_bench_workloads.pysits next to #837'sgenerate_workloads.pyand shares itsextrapolatedRate(split out asrate_from_samples, whose generated text is unchanged).Queryrecord, adding the shapes feat(asap-tools): run generated2 PromQL workloads on ClickHouse #837 lacks:topk(k, sum by (x) (...)),sum by (x) (rate(...)),quantile by (x) (q, instant), the ratio of twoquantile_over_time, and the per-series forms.[T−d, T], and instant vectors take each series' latest sample in[T−1s, T]. Measured on CloudLab against a real Prometheus 2.53: its[5s]sums match ClickHouse'sts >= T−5s AND ts <= Texactly, while(T−5s, T]misses the boundary samples. The paced feeder's timestamps and aligned eval times always put samples on that boundary.(T−d, T](left-open is Prometheus 3.0). With 1 s scrapes on whole seconds, a[1m]window reads 60 samples where Prometheus 2.53 reads 61. This PR leaves it unchanged; it's your call there.uniqExact(value)for cardinality andquantileExactInclusive(q)(value)pooled over every sample of a group. Windows are ASAPQuery's left-open(E−W, E], since no Prometheus is involved and ASAPQuery's SQL frontend and precompute windows are(start, end].<table>/asapquery.sqlfor the ASAPQuery arm: each entry'ssqlfrom the table as written (sketch-bench#209 renders the ASAPQuery dialect once,QUANTILE(q, col),COUNT(DISTINCT col),BETWEEN DATEADD(s, -W, NOW()) AND NOW()), with onlyNOW()replaced by the eval time floored to T. All 936 distinct queries of the 12 tables parse in ASAPQuery's SQL frontend to (⌊E⌋−W, W).{{ end_time }}is floored to a multiple of T in the SQL, matching the aligned query client (feat(asap-tools): run sketch-bench workload tables end to end with the paced feeder #841).experiment_run_clickhouse.pyconfig per table, one query group perinterval_secs(repetition = T), plusinit.sqlwith the tables (gauge/counter:ts, labels,value; records:ts, labels, value fields). Unknown shapes, capabilities or labels raise.Testing
httprecords table withendpointa child ofservice.extrapolatedRatebranch.SQLPatternParser/SQLPatternMatcherto (start, duration) = (⌊E⌋−W, W).CLICKHOUSE_BINARY=<path> python3 -m pytest -q tests/test_sketch_bench_workloads.pypasses (the ClickHouse tests are skipped without a binary). The experiments suite passes. black/flake8 clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae