Repository navigation
fix(asap-tools): let long-window baseline queries finish - #861
Conversation
Prometheus cancels a query after 2 minutes, VictoriaMetrics after 30 s (plus 10 s in its queue), and the ClickHouse query client gave up after 30 s. The e2e baselines run 6h and 24h windows that take minutes, so raise Prometheus' --query.timeout and VictoriaMetrics' search limits to 24h and let the ClickHouse client wait. VictoriaMetrics' retention goes from 1d to 3d so a 24 h window plus the run stays on disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
| timeout: int = 30, | ||
| # None waits for the query to finish; long-window baseline queries | ||
| # can take minutes. | ||
| timeout: Optional[float] = None, |
There was a problem hiding this comment.
timeout=None also disables the connect timeout, so a stalled/unreachable ClickHouse (OOM, silent TCP drop) blocks session.post forever and the prometheus-client container never exits. Suggest a (connect, read) tuple, e.g. (10, 86400): long queries still finish, dead connections still fail. (The Timeout handler's message would then also stop printing None.)
| cmd = f"./prometheus --config.file={PROMETHEUS_CONFIG_FILE} > /dev/null 2>&1 < /dev/null &" | ||
| cmd = ( | ||
| f"./prometheus --config.file={PROMETHEUS_CONFIG_FILE}" | ||
| f" --query.timeout={PROMETHEUS_QUERY_TIMEOUT}" |
There was a problem hiding this comment.
Raising --query.timeout alone may not be enough: Prometheus also caps --query.max-samples (default 50M). A 24h *_over_time over many series can exceed it and fail immediately with "query processing would load too many samples into memory", regardless of the timeout. Same applies to docker_prometheus.py.
| # Keep a 24 h window plus the run, and let long-window queries finish | ||
| # (the defaults are 30 s per query and 10 s waiting in the queue). | ||
| - "-retentionPeriod=3d" | ||
| - "-search.maxQueryDuration=24h" |
There was a problem hiding this comment.
Similarly, VictoriaMetrics has per-query limits besides duration: -search.maxPointsPerTimeseries (default 30000), -search.maxSamplesPerSeries, -search.maxSamplesPerQuery, -search.maxUniqueTimeseries. Long-window/many-series queries can be rejected with a limit-exceeded error, so the 6h/24h baselines could still fail.
- Prometheus: pass --query.max-samples so 24h windows aren't rejected. - VictoriaMetrics: raise maxSamplesPerQuery and maxPointsPerTimeseries. - ClickHouse client: (connect, read) timeout so a dead server fails instead of hanging the querier. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 participants
Summary
The end-to-end evaluation runs 6h and 24h windows on the baselines, and those queries can take minutes. Three timeouts in the current setup would cut them off, and nothing in the experiments framework overrides them:
--query.timeout, default 2 min--query.timeout=24h(PROMETHEUS_QUERY_TIMEOUTinconstants.py)-search.maxQueryDuration30 s,-search.maxQueueDuration10 srequeststimeout 30 sNone: wait for the queryVictoriaMetrics'
-retentionPeriodalso goes from1dto3d, so a 24 h window plus the run stays on disk.The Prometheus query client itself sets no timeout on queries (its 10 s is only the
runtimeinfostatus call), and ASAPQuery's endpoint isn't changed.Testing
query.timeout: 1din/api/v1/status/flags.ClickHouseQueryClient(...).timeoutisNone.🤖 Generated with Claude Code
https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae