chore: auto-open a PR when the pinned Go toolchain goes stale - #511
chore: auto-open a PR when the pinned Go toolchain goes stale#511tylerpotts wants to merge 4 commits into
Conversation
Go patch releases carry standard library security fixes, so the day one ships, the govulncheck gates and the Trivy image scan start failing on main and on every open PR at once. That is how go1.26.5 surfaced: as an unrelated frontend PR going red, not as a toolchain signal. check-go-toolchain.sh only warns on a stale patch outside the release gate, so nothing opened the bump. Adds scripts/bump-go-toolchain.sh, which rewrites both pins (the go.mod toolchain directive and the golang builder image tag + digest in the Dockerfile), and a daily workflow that runs it and opens a PR. The script resolves the builder digest from the registry and verifies the image config reports the expected GOLANG_VERSION before pinning it, so a mismatched or mid-push tag cannot be pinned by digest. Patch bumps only. An unsupported release line exits 2 and fails the run rather than opening a PR, since a major-line move also decides the language version and the go directive. The branch name is version-keyed, so a run that finds the previous bump PR still open leaves it alone.
✅ Deploy Preview for nebi-docs canceled.
|
| if git ls-remote --exit-code --heads origin "$BRANCH" >/dev/null 2>&1; then | ||
| echo "$BRANCH already exists upstream; leaving the open PR alone" | ||
| exit 0 |
There was a problem hiding this comment.
This idempotency check treats “branch exists” as the same thing as “there is already an open bump PR,” but those states can diverge. If a previous run successfully pushed chore/go-toolchain-${VERSION} and then failed during gh pr create, or if the PR was closed while the branch remained, every future run will hit this branch check and exit 0, leaving the stale Go pin without an open PR. Maybe we can check for an open PR for this head branch instead, or handle the “branch exists but no open PR” case by creating/reopening the PR or failing loudly?
| # Verify via one concrete platform: the index itself carries no version label. | ||
| amd64 = next( | ||
| m["digest"] for m in index["manifests"] | ||
| if m.get("platform", {}).get("architecture") == "amd64" | ||
| and m.get("platform", {}).get("os") == "linux" | ||
| ) | ||
| config = json.load(fetch(f"{registry}/manifests/{amd64}", token, ACCEPT))["config"]["digest"] | ||
| env = json.load(fetch(f"{registry}/blobs/{config}", token))["config"].get("Env", []) |
There was a problem hiding this comment.
This only verifies the linux/amd64 child manifest before pinning the multi-arch digest, but the Docker workflow also builds linux/arm64. If Docker Hub had a bad or mid-push arm64 manifest while amd64 was correct, this script would still pin the manifest-list digest and the arm64 CI image would use an unverified builder. Should we verify the config for every platform Nebi builds, at least linux/amd64 and linux/arm64, before accepting the digest?
2 participants
Summary
Stops the problem behind #510 from needing a human every time. Every Go patch release closes standard library vulnerabilities, so the day one ships, the govulncheck gates and the Trivy image scan begin failing on
mainand on every open PR at once. That is exactly howgo1.26.5surfaced: as an unrelated frontend PR going red, not as a toolchain signal.check-go-toolchain.shonly warns on a stale patch outside the release gate, so nothing opened the bump.This adds a daily job that turns that into a two-line PR waiting for review.
Changes
scripts/bump-go-toolchain.shrewrites both places the toolchain is pinned: thetoolchaindirective ingo.modand thegolangbuilder image tag + digest in theDockerfile. Follows the existingcheck-go-toolchain.shidiom (bash +curl+python3, no new dependencies) and is runnable locally..github/workflows/go-toolchain-bump.ymlruns it daily plus onworkflow_dispatch, and opens the PR.Design decisions worth reviewing
The digest is verified, not trusted. After resolving the multi-arch index digest, the script pulls the image config and confirms it reports the expected
GOLANG_VERSIONbefore pinning. Trusting the tag alone would let a mismatched or mid-push image get pinned by digest, which is the one failure mode a digest pin exists to prevent. This is the same check I ran by hand for #510.Patch bumps only. A major-line move also decides the language version and the
godirective, so an unsupported line exits 2 and fails the run loudly instead of opening a PR.Idempotent via a version-keyed branch (
chore/go-toolchain-<version>), so a run that finds the previous bump PR still open leaves it alone.One caveat that needs a decision
A PR opened with the default
GITHUB_TOKENdoes not trigger other workflows, so the bump PR would sit with no checks. The workflow usessecrets.BOT_TOKEN || secrets.GITHUB_TOKEN, so:BOT_TOKENsecret (PAT or GitHub App token) and bump PRs get full CI.Creating PRs with
GITHUB_TOKENalso requires Settings -> Actions -> Allow GitHub Actions to create and approve pull requests to be enabled.Test plan
Verified locally before pushing:
go.modandDockerfileto the pre-fixgo1.26.5state and ran the script: the resulting diff against security: bump Go toolchain to 1.26.6 #510's known-good commit is empty.changed=false, no writes.go1.20.0): exits 2 with the manual-upgrade error.toolchaindirective: exits 1.bash -n.git/ghstubbed: commit message and PR body render correctly (variables expanded, backticks literal, no stray indentation) and thegit/ghinvocations are right.Note that CI here exercises the repo's normal gates, not this workflow, which only runs on its schedule. The verification above is where the behavior was checked.