Repository navigation
PR CI: stress-test each sanitizer on one architecture, skip builds only skipped jobs use - #123764
Conversation
…anitizer builds only they use Pull requests now skip `Stress test (amd_asan_ubsan)`, `(arm_debug)`, `(arm_tsan)` and `(arm_msan)`, and the `Build (arm_debug)`, `(arm_tsan)` and `(arm_msan)` jobs whose binaries only these stress tests use. Each sanitizer is still stress-tested once per PR (TSan, MSan and debug on amd, ASan on arm next to its `s3` variant and `arm_release`). Master, release and backport workflows are unchanged. The skip uses the exemptions of the small-PR rule (`ci-force-all`, uncounted build inputs, the stress machinery) plus changes under `cmake/` and `base/glibc-compatibility/`. On small PRs the three builds are skipped together with the stress tests. `ci/workflows/pull_request.py` asserts that no other PR job requires the skipped builds' artifacts.
No PR job uses the libFuzzer targets it builds; in a PR it only checks that they compile. Over 2026-08-20 to 2026-10-02 it failed alone in 13 PRs, and the two below 100 changed product lines failed on infrastructure (`Bus error` in the linker). It still runs when a PR touches a `fuzzers/` directory or `tests/fuzz/`, and master builds it on every commit. `Build (wasm64)` keeps running: in the same period it alone found Emscripten-only compile errors in 4 PRs below the threshold.
|
Workflow [PR], commit [cec556f] Summary: ✅ AI ReviewSummaryThis PR reduces PR CI cost by stress-testing each sanitizer on only one architecture and by skipping a few builds that no remaining PR job consumes. The previously reported Findings
Final VerdictRequest changes: the new |
![clickhouse-gh[bot]](https://avatars.githubusercontent.com/in/928874?s=60&v=4){kind=small}
clickhouse-gh
Bot
added
the
comp-ci-infrastructure
…at `cmake/` changes as small With the `ci-build` label the arm sanitizer builds and `Build (amd_fuzzers)` run again. A PR touching `cmake/` or `base/glibc-compatibility/` no longer takes the small-PR skip: one line there (a compiler or sanitizer flag) can change the whole binary, the same reasoning as for the uncounted build inputs.
| """The fuzz targets live in `fuzzers/` directories next to the code they fuzz, and | ||
| `tests/fuzz/` holds the scripts that stage their dictionaries and corpora.""" | ||
| return any( | ||
| "/fuzzers/" in f or f.removeprefix("./").startswith("tests/fuzz/") |
There was a problem hiding this comment.
This still skips Build (amd_fuzzers) for PRs that only touch tests/queries/0_stateless/*. _has_fuzzer_target_changes only exempts */fuzzers/* and tests/fuzz/, but the amd_fuzzers build extends its own digest with tests/queries/0_stateless/ because tests/fuzz/build.sh copies those .sql files into the published seed corpora.
So a corpus-only change now drops the only PR-side check that the fuzzers artifact can still be packaged, and the breakage moves to the nightly libFuzzer pipeline after merge. Can we also keep Build (amd_fuzzers) running for tests/queries/0_stateless/ edits, or derive this exemption from the build's extra digest paths instead of hard-coding only two path families?
robot-ch-test-poll3
added
the
pr-synced-to-cloud
Pull requests stress-test each sanitizer on one architecture and skip the builds whose binaries only the skipped jobs use. Expected saving: about $6.5k per week of PR CI (CIDB, 2026-09-28 to 2026-10-03, on-demand prices).
What pull requests skip (master, release and backport workflows are unchanged):
Stress test (amd_asan_ubsan),(arm_debug),(arm_tsan),(arm_msan)Build (arm_debug),(arm_tsan),(arm_msan)Build (amd_fuzzers), below 100 changed product lines onlyEach sanitizer is still stress-tested once per PR: TSan, MSan and debug on amd, ASan on arm next to its
s3variant andarm_release, so both architectures stay covered.Why one architecture is enough, from PR and master runs over 2026-08-20 to 2026-10-02:
Unknown job error).Exemptions: the stress skip and the build skips do not apply with the
ci-force-alllabel, when a PR touches inputs of the build that the line count does not see (contrib/, the build scripts), the machinery of the stress jobs (as in #120996),cmake/orbase/glibc-compatibility/. A PR touchingcmake/orbase/glibc-compatibility/is also never treated as small. Theci-buildlabel keeps all builds.Build (amd_fuzzers)also runs when a PR touches afuzzers/directory ortests/fuzz/. Over the same period it failed alone (all other builds green) in 13 PRs; the two below the threshold (#105118, #120870) failed on infrastructure.Build (wasm64)also has no consumer, but keeps running: it compiles with the Emscripten clang, and in the same period it alone found compile errors in four PRs below the threshold (#123073, #119805, #117498, #121376).A filtered build is marked like a cache hit, so the jobs that only wait for it (
run_after) still run. A job that requires its binary would find no artifact, soci/workflows/pull_request.pyasserts that no other PR job requires the artifacts of the skipped arm builds.pull_request.ymldoes not change. This PR touches the stress machinery, so its own CI runs everything; the skips show up on the next regular PR.Related: #120996
Changelog category (leave one):
Workflow [PR]
Sync PR [sync-upstream/pr/123764]
Version info
26.10.1.1494-master(included in26.10and later)