Skip to content

MDBF-1265: Cap MTR --parallel per builder and worker - #1022

Open
RazvanLiviuVarzaru wants to merge 1 commit into
MariaDB:devfrom
RazvanLiviuVarzaru:MDBF-1265
Open

RazvanLiviuVarzaru wants to merge 1 commit into
MariaDB:devfrom
RazvanLiviuVarzaru:MDBF-1265

Conversation

@RazvanLiviuVarzaru

Copy link
Copy Markdown
Collaborator

Buildbot's one-size-fits-all rule has been that a builder asking for N jobs (vCPUs) runs MTR with --parallel=2N, and that this is safe on a host where several builds are scheduled. It relies on most of our hosts having about 2 GB of memory per vCPU, and on a test run peaking at about 2 GB per job.

Some builder configurations, such as --big-test or the sanitizers, are outliers and no longer fit this rule.

With this patch:

  • Builders keep declaring jobs, which sets the compile parallelism and the job claim on the worker, so they stay fast while compiling.
  • A builder can cap MTR --parallel on specific workers, with get_config(mtr_max_parallel={"<worker>": N}). The cap is applied when the build starts, for the worker running it. Workers not listed keep 2 x jobs, and a worker name that is not one of the builder's workers fails the config load.

Only the few outliers will need it. The first is
amd64-ubasan-clang-22-debug on sg-bbw1 (32 GB), which hits frequent OOMs running MTR at --parallel=24. It is capped at 10 there.

Memory of its MTR run, reproduced locally with the builder's image, flags and ASAN options (main, innodb, rpl and binlog, 15 minutes per run), measured at --parallel=4 and 8 and extrapolated:

--parallel Median Peak (linear fit) Peak (worst ratio) On 32 GB
8 12.7 GB 17.1 GB (measured) 19 GB safe
10 15.7 GB 21 GB 24 GB safe, 4-7 GB headroom
12 18.7 GB 25 GB 29 GB edge, no margin
16 24.6 GB 32 GB 38 GB OOM
24 (before) 36.5 GB 47 GB 57 GB always OOM

Plain build (RelWithDebInfo, no sanitizers, no --big-test), suites main,innodb,rpl,binlog:

--parallel (jobs) p50 p99 Peak Peak per job /dev/shm at peak
8 (4) 2.8 GB 5.2 GB 5.9 GB 1.5 GB 2.1 GB
16 (8) 4.9 GB 9.5 GB 10.8 GB 1.35 GB 3.5 GB
24 (12) 6.9 GB 13.7 GB 14.1 GB 1.2 GB 4.6 GB

Buildbot's one-size-fits-all rule has been that a builder asking for N
jobs (vCPUs) runs MTR with `--parallel=2N`, and that this is safe on a
host where several builds are scheduled. It relies on most of our hosts
having about 2 GB of memory per vCPU, and on a test run peaking at about
2 GB per job.

Some builder configurations, such as `--big-test` or the `sanitizers`, are
outliers and no longer fit this rule.

With this patch:
- Builders keep declaring jobs, which sets the compile parallelism and
  the job claim on the worker, so they stay fast while compiling.
- A builder can cap MTR `--parallel` on specific workers, with
  `get_config(mtr_max_parallel={"<worker>": N})`. The cap is applied
  when the build starts, for the worker running it. Workers not listed
  keep 2 x jobs, and a worker name that is not one of the builder's
  workers fails the config load.

Only the few outliers will need it. The first is
`amd64-ubasan-clang-22-debug` on `sg-bbw1` (32 GB), which hits frequent
OOMs running MTR at `--parallel=24`. It is capped at 10 there.

Memory of its MTR run, reproduced locally with the builder's image,
flags and ASAN options (main, innodb, rpl and binlog, 15 minutes per
run), measured at --parallel=4 and 8 and extrapolated:

| --parallel  | Median  | Peak (linear fit)  | Peak (worst ratio) | On 32 GB              |
|-------------|---------|--------------------|--------------------|-----------------------|
| 8           | 12.7 GB | 17.1 GB (measured) | 19 GB              | safe                  |
| 10          | 15.7 GB | 21 GB              | 24 GB              | safe, 4-7 GB headroom |
| 12          | 18.7 GB | 25 GB              | 29 GB              | edge, no margin       |
| 16          | 24.6 GB | 32 GB              | 38 GB              | OOM                   |
| 24 (before) | 36.5 GB | 47 GB              | 57 GB              | always OOM            |

**Plain build** (`RelWithDebInfo`, no sanitizers, no `--big-test`), suites `main,innodb,rpl,binlog`:

| `--parallel` (jobs) | p50 | p99 | Peak | Peak per job | `/dev/shm` at peak |
|---|---|---|---|---|---|
| 8 (4) | 2.8 GB | 5.2 GB | 5.9 GB | 1.5 GB | 2.1 GB |
| 16 (8) | 4.9 GB | 9.5 GB | 10.8 GB | 1.35 GB | 3.5 GB |
| 24 (12) | 6.9 GB | 13.7 GB | 14.1 GB | 1.2 GB | 4.6 GB |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant