Build once: pin the Helm chart to its images by digest by petr-sandbox · Pull Request #745 · sandbox-quantum/switch

petr-sandbox · 2026-10-09T12:35:46Z

What changes

The published chart pins its images by digest. The release now packages the chart only after the three images are pushed, and first writes each image's registry digest into the chart's values.yaml:

global:
  imageRegistry: "ghcr.io/sandbox-quantum"
switchCore:
  image: switch-core:<version>@sha256:…
  imagePullPolicy: IfNotPresent

(same for gateway and setup). Pinning the chart version therefore pins the exact images. A deployment no longer sets image values, and what runs can be reproduced from the chart pin alone. The tag before the digest is only there for people to read.

Dev builds on every push to main, under <pyproject>-dev.<run>.g<sha>. These never become latest, Helm treats them as pre-releases, and no compose artifact is published for them. This lets a dev environment run main from published artifacts instead of building from source at deploy time.
The digests are recorded. Each publishing run lists all four (the chart's own included) in its summary and uploads them as a release-pins artifact, ready for a pins file to consume.
Checks. scripts/pin_chart_images.py does the pin and checks a rendered chart. The release runs the check on the packaged .tgz before pushing, and confirms the registry serves each pinned reference. PR CI gets a new Helm chart job that runs the pin with placeholder digests, so a new first-party image value that the pinner doesn't know about fails the PR instead of shipping by tag. A dry run (workflow_dispatch) also pins with placeholder digests.
New setting global.preStopSleepSeconds (default 5): a native preStop sleep on switch-core, gateway and Mattermost, so a rollout doesn't answer with 503s while endpoints and load balancers catch up. It needs Kubernetes 1.30+; 0 turns it off.

Worth a look

This changes chart defaults (global.imageRegistry, pull policy, the image references). Anyone who set the images by tag keeps their tag and opts out of the pin. A mirror configured through global.imageRegistry keeps the pin as long as it copies images byte for byte. The changelog says so.
Dev builds run on every push to main, with no path filter. The images copy files from outside core/ and gateway/, so a filter would eventually skip a change that mattered. The cost is a multi-arch build per merge.
Dev builds are not gated on PR CI. They publish whatever reached main. Deploying one is a separate decision, made by changing the pin.
An RC and its final release are still separate builds, with different digests, because the version is stamped into the image. Promoting staging to prod by digest would mean releasing by re-tagging the RC instead of rebuilding. That's a later decision, not part of this PR.

Verified

pin_chart_images unit tests (11) cover the real values.yaml, refusals, and a moved values shape.
Locally, end to end: pin, helm package, helm template of the .tgz, then verify; the unpinned chart fails verify. actionlint passes on both workflows. The run-summary and pins steps were run locally with fake digests.
helm pull oci://…/charts/switch@sha256:… resolves, so the chart can be pinned by digest too.
Not yet run in Actions. The first push to main after merge is the first real publish.

🤖 Generated with Claude Code

petr-sandbox

I left inline comments. The first four matter most for existing installs: double-prefixed full image refs, IfNotPresent on mutable tags, the preStop sleep on Recreate Deployments, and the 1.30+ requirement for the sleep action.

petr-sandbox · 2026-10-09T12:51:02Z

+        raise PinError("version must be non-empty")
+
+    lines = values_text.splitlines(keepends=True)
+    _set_child(lines, "global", "imageRegistry", f'"{registry}"')


Pinning gives global.imageRegistry a non-empty default. Before this, it defaulted to "" and switch-core:latest wasn't usable on its own, so operators set full refs such as switchCore.image: ghcr.io/acme/switch-core:0.29.0. After upgrading, the switch.image helper renders ghcr.io/sandbox-quantum/ghcr.io/acme/switch-core:0.29.0 and the pods fail with ImagePullBackOff. The changelog says setting images by tag opts back out of the pin, but it doesn't mention full references. Could the helper leave refs that already contain a registry alone, or could the changelog call this out as a breaking change?

Both. switch.image now leaves a reference that already names a registry as written: one whose first path component contains . or :, or is localhost, which is the same rule container runtimes use. So ghcr.io/acme/switch-core:0.29.0 renders unchanged. One case still breaks, a Docker Hub namespace with no registry (acme/switch-core:1.0), since a namespace can't be told apart from a registry-less path. The changelog now calls that out as breaking, with the docker.io/ fix. Fixed in cd2f15d.

petr-sandbox · 2026-10-09T12:51:02Z

+    _set_child(lines, "global", "imageRegistry", f'"{registry}"')
+    for name, key in IMAGES.items():
+        _set_child(lines, key, "image", f"{name}:{version}@{digests[name]}")
+        _set_child(lines, key, "imagePullPolicy", "IfNotPresent")


This makes IfNotPresent the default for every first-party image, including when an operator opts out of the pin with a mutable tag (switch-core:latest, main). Those operators were relying on the old Always. Nodes now keep the cached image, so later pushes under that tag are never pulled. The rollout looks fine but runs stale code. Could IfNotPresent apply only when the ref is digest-pinned?

Done. The three imagePullPolicy values now default to "", and a new switch.imagePullPolicy helper renders IfNotPresent only for a reference containing @sha256: and Always for anything else. An explicit policy still wins. The pinner no longer writes a policy at all. cd2f15d

petr-sandbox · 2026-10-09T12:51:02Z

+        {ref for doc in yaml.safe_load_all(rendered) if doc for ref in _images(doc)}
+    )
+    prefix = f"{registry}/"
+    ours = [ref for ref in refs if ref.startswith(prefix)]


verify only checks refs that start with the registry prefix. If a template references .Values.switchCore.image directly or hardcodes switch-core:latest, that ref is ignored as long as some other template renders switch-core pinned. The same goes for templates gated on flags (teamsBridge, hosted) that the render-check values don't turn on. Either way the chart ships an image by tag. Could it also flag any ref whose repository name matches a first-party image, whatever the prefix, and render with the optional templates enabled?

Done. verify now recognises a first-party image by repository name, whatever the prefix. Each one must be exactly <pinning registry>/<name> and digest-pinned, so a bare switch-core:latest or a ref from another registry fails even when a pinned copy renders elsewhere. A ref from the pinning registry that isn't in IMAGES fails too. It also checks a second render with the optional workloads on (render-check-optional-values.yaml), in both PR CI and the release. cd2f15d

petr-sandbox · 2026-10-09T12:51:02Z

              value: /etc/switch-github/github.json
            {{- end }}
            {{- include "switch.coreEnv" (merge (dict "omitOwnerCredentials" true) .) | nindent 12 }}
+          {{- include "switch.preStopDrain" . | nindent 10 }}


switch-core uses strategy: Recreate, so the old pod terminates before the new one exists. No other endpoint is left to take traffic while it sleeps. The drain can't prevent rollout 503s here, and it adds 5s of total unavailability to every rollout. The values.yaml and changelog wording ("does not answer requests with 503s") doesn't hold for this Deployment.

Agreed. The drain is removed from switch-core. cd2f15d

petr-sandbox · 2026-10-09T12:51:02Z

            - name: MM_SERVICESETTINGS_ALLOWEDUNTRUSTEDINTERNALCONNECTIONS
              value: "{{ $core }} {{ $core }}.{{ .Release.Namespace }} {{ $core }}.{{ .Release.Namespace }}.svc.cluster.local"
            {{- end }}
+          {{- include "switch.preStopDrain" . | nindent 10 }}


Same as switch-core: Mattermost uses strategy: Recreate, so this sleep only lengthens the outage on each rollout.

Removed from Mattermost too. Only the gateway keeps it: it rolls with a surge, so its replacement is Ready before the old pod stops. The values and changelog wording now say exactly that. cd2f15d

petr-sandbox · 2026-10-09T12:51:02Z

@@ -25,6 +39,12 @@ on:
  push:


Every push to main now runs three QEMU-emulated multi-arch image builds plus a chart publish. There's no concurrency group, and nothing cleans up the dev versions pushed to GHCR. On a busy merge day that means many parallel arm64 builds competing for runners and the shared gha cache scope. Every dev tag also stays in the public packages forever, cluttering the tag list consumers browse. Could we add a concurrency group with cancel-in-progress for main, plus a retention or cleanup job?

Done. Pushes to main share a concurrency group with cancel-in-progress, so the newest build wins; tags and dispatch runs each get their own group and are never cancelled. Dev builds now publish to separate packages under ghcr.io/<owner>/dev/, which keeps the release packages' tag lists clean. A daily switch-dev-cleanup.yml prunes dev builds older than 30 days that aren't among the 20 newest, keeping each kept build's platform manifests (scripts/prune_dev_packages.py, with a dry-run dispatch). cd2f15d

petr-sandbox · 2026-10-09T12:51:02Z

+            # The `g` keeps the identifier alphanumeric: an all-digit short sha
+            # with a leading zero is not valid semver.
+            tag_version="${pkg_version}-dev.${GITHUB_RUN_NUMBER}.g${GITHUB_SHA::7}"
+            python_version="${pkg_version}.dev${GITHUB_RUN_NUMBER}"


<pyproject>.devN sorts before the release of the same base version, under both PEP 440 and semver (-dev.N). After switch-v0.29.0 ships and before pyproject is bumped, every main build reports itself as older than 0.29.0. A version gate, such as Console's minimum server version, would then treat the dev server as a pre-release of 0.29.0. Could the base version be bumped right after a release, or could dev builds use a post-release or local suffix?

Dev builds now use the next patch above the newest switch-vX.Y.Z tag reachable from main, e.g. 0.29.1-dev.N.g<sha> / 0.29.1.devN after 0.29.0. That sorts after the release the build contains and before the next release or RC, under both semver and PEP 440. It's tag-based, so it doesn't depend on when pyproject gets bumped. cd2f15d

petr-sandbox · 2026-10-09T12:51:02Z

+        # and the check below still run, and nothing is published.
+        run: |
+          args=()
+          for image in switch-core gateway setup; do


The first-party image list and the zero-digest placeholder are duplicated across this pin loop, the pins JSON, the summary loop, pr-ci.yml and IMAGES in pin_chart_images.py. Adding a fourth image means editing all of them together. Missing the summary or pins loop silently leaves the image out of release-pins.json, which the pinner's own checks never see. Could the script emit the image list or the pins JSON itself?

IMAGES in scripts/pin_chart_images.py is now the only list, and it carries each image's Dockerfile. The build matrix comes from pin_chart_images.py images. pin reads the digests directory (or --placeholder-digests), and record writes the pins JSON and the summary, so the workflow has no loops over image names. Every subcommand refuses a digest set that isn't exactly IMAGES. cd2f15d

petr-sandbox · 2026-10-09T12:51:02Z

@@ -212,25 +315,87 @@ jobs:
        # a single tag pins the whole stack.


Re-running a publishing run reuses GITHUB_RUN_NUMBER and the release tag. The images are rebuilt (not bit-for-bit reproducible) and helm push overwrites charts/switch:<version> with a chart pointing at the new digests. An earlier release-pins.json then no longer matches what helm pull --version <version> returns, which breaks the "one pin reproduces what runs" guarantee for anyone pinning by version. Could the job refuse to push when the chart version already exists in the registry?

Done. prep asks the registry whether charts/switch:<version> exists before anything is built, and fails if it does. The chart job asks again right before helm push, because re-running that job alone skips prep. Only a plain not found counts as unpublished; an auth or network error fails the run rather than reading as permission to push. cd2f15d

petr-sandbox · 2026-10-09T12:51:02Z

@@ -110,6 +112,13 @@ jobs:
            # Spans every tree, so it gets its own filter rather than riding


Nit: the new chart filter went in between the artifacts filter's comment and its key. This comment ("Spans every tree… whose versions the registry is checked against") now sits above chart, and artifacts is left without its explanation.

Fixed, thanks. cd2f15d

The release now packages the Helm chart only after the images are pushed, and writes each image's registry digest into the chart's values first, so pinning the chart pins the exact images and what runs is reproducible from that one pin. global.imageRegistry defaults to the publishing registry and the three first-party images pull IfNotPresent. Every push to main publishes a dev build (<version>-dev.<run>.g<sha>, never latest), so the dev environment can run main from published artifacts instead of building from source at deploy time. Each publishing run records all four digests, the chart's included, in its summary and a release-pins artifact. scripts/pin_chart_images.py does the pin and checks a render; the release checks the packaged chart before pushing, and PR CI runs both with placeholder digests. Also adds global.preStopSleepSeconds (default 5): a native preStop sleep on switch-core, gateway and Mattermost so a roll does not answer with 503s. CHOO-3421 phase 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…mespace - switch.image leaves a reference that already names a registry alone, so a full ref no longer gets the new default registry prepended. - imagePullPolicy defaults to "" for the first-party images: IfNotPresent when digest-pinned, Always otherwise, so a mutable tag keeps pulling. The pinner no longer writes a policy. - The preStop drain is gateway-only (switch-core and Mattermost roll with Recreate, where it only lengthens the outage) and renders only on Kubernetes 1.30+. - verify recognises first-party images by repository name under any prefix, flags unknown images from the pinning registry, and runs on a second render with the optional workloads on. - scripts/pin_chart_images.py holds the one image list: the build matrix, the pin and the recorded pins all come from it. - Dev builds publish under <owner>/dev, as <next patch>-dev.<run>.g<sha>, in a cancel-in-progress group; switch-dev-cleanup.yml prunes them. - The release refuses a chart version already in the registry, in prep and again right before pushing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

petr-sandbox requested review from amaudruz and christian-mcdermott as code owners October 9, 2026 12:35

petr-sandbox commented Oct 9, 2026

View reviewed changes

petr-sandbox mentioned this pull request Oct 9, 2026

ci(release): publish the hosted controller image and chart, pinned by digest #748

Merged

amaudruz approved these changes Oct 12, 2026

View reviewed changes

petr-sandbox and others added 2 commits October 12, 2026 12:30

petr-sandbox force-pushed the work/deploy-pipeline branch from cd2f15d to d56ef39 Compare October 12, 2026 10:32

petr-sandbox merged commit dd00626 into main Oct 12, 2026
12 checks passed

petr-sandbox deleted the work/deploy-pipeline branch October 12, 2026 10:43

		@@ -212,25 +315,87 @@ jobs:
		# a single tag pins the whole stack.

		@@ -110,6 +112,13 @@ jobs:
		# Spans every tree, so it gets its own filter rather than riding

Conversation

petr-sandbox commented Oct 9, 2026

What changes

Worth a look

Verified

Uh oh!

petr-sandbox left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants