Repository navigation
MDBF-1265: Cap MTR --parallel per builder and worker - #1022
Open
RazvanLiviuVarzaru wants to merge 1 commit into
Open
RazvanLiviuVarzaru wants to merge 1 commit into
RazvanLiviuVarzaru wants to merge 1 commit into
Conversation
Buildbot's one-size-fits-all rule has been that a builder asking for N
jobs (vCPUs) runs MTR with `--parallel=2N`, and that this is safe on a
host where several builds are scheduled. It relies on most of our hosts
having about 2 GB of memory per vCPU, and on a test run peaking at about
2 GB per job.
Some builder configurations, such as `--big-test` or the `sanitizers`, are
outliers and no longer fit this rule.
With this patch:
- Builders keep declaring jobs, which sets the compile parallelism and
the job claim on the worker, so they stay fast while compiling.
- A builder can cap MTR `--parallel` on specific workers, with
`get_config(mtr_max_parallel={"<worker>": N})`. The cap is applied
when the build starts, for the worker running it. Workers not listed
keep 2 x jobs, and a worker name that is not one of the builder's
workers fails the config load.
Only the few outliers will need it. The first is
`amd64-ubasan-clang-22-debug` on `sg-bbw1` (32 GB), which hits frequent
OOMs running MTR at `--parallel=24`. It is capped at 10 there.
Memory of its MTR run, reproduced locally with the builder's image,
flags and ASAN options (main, innodb, rpl and binlog, 15 minutes per
run), measured at --parallel=4 and 8 and extrapolated:
| --parallel | Median | Peak (linear fit) | Peak (worst ratio) | On 32 GB |
|-------------|---------|--------------------|--------------------|-----------------------|
| 8 | 12.7 GB | 17.1 GB (measured) | 19 GB | safe |
| 10 | 15.7 GB | 21 GB | 24 GB | safe, 4-7 GB headroom |
| 12 | 18.7 GB | 25 GB | 29 GB | edge, no margin |
| 16 | 24.6 GB | 32 GB | 38 GB | OOM |
| 24 (before) | 36.5 GB | 47 GB | 57 GB | always OOM |
**Plain build** (`RelWithDebInfo`, no sanitizers, no `--big-test`), suites `main,innodb,rpl,binlog`:
| `--parallel` (jobs) | p50 | p99 | Peak | Peak per job | `/dev/shm` at peak |
|---|---|---|---|---|---|
| 8 (4) | 2.8 GB | 5.2 GB | 5.9 GB | 1.5 GB | 2.1 GB |
| 16 (8) | 4.9 GB | 9.5 GB | 10.8 GB | 1.35 GB | 3.5 GB |
| 24 (12) | 6.9 GB | 13.7 GB | 14.1 GB | 1.2 GB | 4.6 GB |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Buildbot's one-size-fits-all rule has been that a builder asking for N jobs (vCPUs) runs MTR with
--parallel=2N, and that this is safe on a host where several builds are scheduled. It relies on most of our hosts having about 2 GB of memory per vCPU, and on a test run peaking at about 2 GB per job.Some builder configurations, such as
--big-testor thesanitizers, are outliers and no longer fit this rule.With this patch:
--parallelon specific workers, withget_config(mtr_max_parallel={"<worker>": N}). The cap is applied when the build starts, for the worker running it. Workers not listed keep 2 x jobs, and a worker name that is not one of the builder's workers fails the config load.Only the few outliers will need it. The first is
amd64-ubasan-clang-22-debugonsg-bbw1(32 GB), which hits frequent OOMs running MTR at--parallel=24. It is capped at 10 there.Memory of its MTR run, reproduced locally with the builder's image, flags and ASAN options (main, innodb, rpl and binlog, 15 minutes per run), measured at --parallel=4 and 8 and extrapolated:
Plain build (
RelWithDebInfo, no sanitizers, no--big-test), suitesmain,innodb,rpl,binlog:--parallel(jobs)/dev/shmat peak