diff --git a/.github/workflows/bbm_deploy.yml b/.github/workflows/bbm_deploy.yml index 6d8ab9580..b2127f696 100644 --- a/.github/workflows/bbm_deploy.yml +++ b/.github/workflows/bbm_deploy.yml @@ -22,6 +22,7 @@ on: - "validate_master_cfg.sh" - "worker_locks.yaml" - "dashboards/**" + - "!**/README.md" pull_request: paths: - ".github/workflows/bbm_deploy.yml" @@ -42,6 +43,7 @@ on: - "validate_master_cfg.sh" - "worker_locks.yaml" - "dashboards/**" + - "!**/README.md" jobs: check: diff --git a/docker-compose/README.md b/docker-compose/README.md index be523fca8..1af4e2130 100644 --- a/docker-compose/README.md +++ b/docker-compose/README.md @@ -30,12 +30,28 @@ The [bbm-deploy](../.github/workflows/bbm_deploy.yml) workflow checks the config cd /srv/prod/docker-compose docker-compose stop docker-compose --env-file .env up -d +grep 'BuildMaster is running' logs/.log | tail -n 1 ``` +Restart the next master once `grep` shows this start's time. `` is the master's directory name: `amd64-master-0` for `autogen_amd64-master-0`. + `stop` gives the master up to its `stop_grace_period` (5 minutes, set in `generate-config.py`) to shut down before it is killed. To make it faster, pass a shorter timeout, in seconds: `docker-compose stop -t 30 `. Check the number of autogen masters before deploying a change to `os_info.yaml`; see [Autogen masters](../docs/server-builders.md#autogen-masters). +### Stopping and starting the masters + +Because of [buildbot#9215](https://github.com/buildbot/buildbot/issues/9215), stop and start the production masters one at a time, and start the next only once the previous one has logged `BuildMaster is running`. + +Stop them in this order, and start them in the reverse order, master-libvirt first: + +1. master-protected-branches and master-galera, so that no new server or Galera builds start; +1. the autogen masters; +1. master-docker-nonstandard and master-docker-nonstandard-2; +1. master-migration; +1. master-nonlatent; +1. master-libvirt. + ### Restarting production without disrupting anyone Prefer restarting when no branch protection builds are running, so that contributors already have their results on GitHub. Buildbot is meant to restart the builds a stop interrupts, but doesn't always. If you restart while such builds run, check the Grid View, filtered on the `protected` tag, that they finished or were restarted.