Repository navigation
sec(build): build the shipped image on go1.26.6 to clear the stdlib CVEs - #1835
Merged
Merged
Conversation
#1832 bumped GO_VERSION and the pre-commit go-version pin, which turned govulncheck green in CI, but the artifact we deploy was still compiled by the go1.26.5 builder stage, so all seven stdlib advisories remained in the binary we ship. Bump the golang base image in Dockerfile (the production builder), Dockerfile.dev and Dockerfile.test from 1.26.5-alpine3.24 to 1.26.6-alpine3.24, refreshing the SHA256 digest pin rather than dropping it. The digest was resolved from the registry v2 manifest endpoint and cross-checked against docker buildx imagetools inspect. Bump the go directives in go.work and the five module go.mod files. This is load-bearing rather than tidiness: aws_sanity.yml, azure_sanity.yml and database-migration.yml resolve their toolchain via go-version-file: go.mod, and an azure_sanity run from after #1832 merged logs "Successfully set up Go version 1.26.5". No module carries a toolchain directive, so setup-go reads the go directive directly. tests/e2e/go.mod stays at go 1.25: no workflow resolves a toolchain from it, GOTOOLCHAIN=auto only ever upgrades and never downgrades, and the module is test-only, stdlib-only, and not part of the shipped artifact.
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
Included review availability: 0 reviews are currently available. Based on recent review activity, included reviews refill at 3 per hour. 📝 WalkthroughWalkthroughThe pull request updates the repository to Go 1.26.6. It refreshes pinned Docker images, compiles ChangesGo toolchain update
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: ⚪ Minimal · up to The PR updates the shipped build and related toolchain pins to Go 1.26.6, with the extracted binary verified clean for the targeted stdlib advisories. No actionable merge-blocking risk remains beyond normal checks and review. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
cristim
added
priority/p1
… toolchain The runtime image contains two Go binaries and the base-image bump only moved one of them. /usr/local/bin/migrate was an upstream prebuilt release tarball compiled on go1.25.4, so it still carried all seven stdlib advisories from issue #1833. That binary is not dormant. scripts/entrypoint.sh runs `migrate up` on every container start and DB_AUTO_MIGRATE defaults to true in this image, and the vulnerable symbols govulncheck reports include url.URL.Parse, url.URL.ResolveReference and tls.Conn.Handshake, which are the paths that parse the database URL and negotiate TLS to the database. Build it with `go install` on the stage's pinned toolchain, the same way `make install-tools` does. Bumping the pinned release was not an option: v4.19.1 is the latest upstream tag, and the fix for the 1.25 line is go1.25.13. `go install` refuses GOBIN when cross-compiling and writes to bin/$GOOS_$GOARCH/ instead, so both layouts are resolved and the final `mv` fails the build if neither produced a binary. curl is dropped from the builder stage because the download it served is gone. Verified on full images built from this branch, both target architectures: binary arch toolchain govulncheck -mode=binary /app/cudly arm64 go1.26.6 2 vulnerabilities from 2 modules /app/cudly amd64 go1.26.6 2 vulnerabilities from 2 modules /usr/local/bin/migrate arm64 go1.26.6 No vulnerabilities found. /usr/local/bin/migrate amd64 go1.26.6 No vulnerabilities found. Stdlib advisory count is 0 on all four, counted from `-format json` by module rather than by a text pattern. The previously shipped prebuilt binary measures "69 vulnerabilities from 12 modules and the Go standard library", 35 of them stdlib. The two remaining findings on /app/cudly are pre-existing and non-stdlib (GO-2026-5158, GO-2026-5932); this commit changes no dependency. `migrate -version` still reports v4.19.1 in the final image, and a real `migrate -path /app/migrations -database postgresql://... up` reaches the TCP connect step, confirming both the file source and the postgresql driver are registered under -tags postgres.
…dcoded pin pre-commit.yml was the last workflow carrying a literal `go-version`. aws_sanity, azure_sanity and database-migration already use `go-version-file: go.mod`, so this removes one of the independent places the Go version had to be bumped by hand. That arrangement is what produced issue #1833: #1832 bumped some of them and CI went green while the shipped artifact stayed on the old toolchain. The root go.mod `go` directive is 1.26.6, so the resolved version is unchanged by this commit.
…quire The documented prerequisites had drifted well below the `go` directive in go.mod: "Go 1.23 or later" in CONTRIBUTING.md and README.md, "Go 1.25+" in README.md, docs/DEPLOYMENT.md and docs/DEVELOPMENT.md. Most concretely, CONTRIBUTING.md hands contributors a copy-paste `go.work.local` template whose first line was `go 1.25.0`. A workspace `go` line below the modules' own 1.26.6 is rejected, so following that section verbatim no longer built. It now matches the committed go.work and says why it has to. Also corrects .github/workflows/README.md, which documented the `GO_VERSION` variable default as 1.25 when ci.yml sets 1.26.6.
This was referenced Aug 17, 2026
cristim
added a commit
that referenced
this pull request
Aug 18, 2026
…ce (#1836) Nothing in the pipeline inspected the container image we ship. govulncheck ran in source mode over the six modules, Trivy ran with scan-type fs and config, and docker-build built the image, ran --version against it, and discarded it. A vulnerable binary baked into the image was therefore unreachable by the pipeline by construction. That gap hid two consecutive rounds of CVE work. #1832 bumped the Go toolchain CI reads and govulncheck went green while the image still built on go1.26.5. #1835 bumped the base image and /app/cudly came out clean, but the image shipped a second Go binary: a prebuilt golang-migrate release carrying go1.25.4 with 69 vulnerabilities across 12 modules, executed on every container start by entrypoint.sh with DB_AUTO_MIGRATE=true by default. A new job runs govulncheck -mode=binary over every Go binary in the final stage after docker-build, and is wired into ci-success so it gates. It fails on any advisory with a published fixed version at any severity, and tolerates but always prints advisories with no published fix, so it does not land permanently red on GO-2026-5932 (x/crypto/openpgp, introduced: "0", no fixed event, unreachable here) while still failing on the classes that actually occurred. The scanner was itself fail-open twice, and both were closed before merge. Its jq filter returned an empty array with exit 0 for an empty stream, for {}, and for a well-formed object with no config, all of which the classifier read as zero findings; the existing guards only fired on a parse error and none of those inputs is one, so a govulncheck run that silently produced nothing would have certified the image clean. The parser now requires the stream to open with a config message. That check then accepted any non-empty protocol_version, which would have reproduced the same hole on a tool upgrade emitting a different schema, so it is pinned to v1.0.0 with the expected and found values reported on mismatch. Verified both directions with recorded fixtures: red on a known-bad image carrying the prebuilt migrate binary, green on the current image, empty and malformed and wrong-schema streams all exit 2 rather than clean, an unfixable-only stream is tolerated, and a fixable advisory fails. Suite is 9 passed, 0 failed, asserting message content rather than exit codes alone. Trivy scan-type image was deliberately not added. The pinned runtime base alpine:3.21.3 carries fixable CRITICAL and HIGH OS advisories today, including a libssl3 heap overflow from a large X.509 certificate, so a gating step would land red on day one for something this change does not fix and a non-gating one would print an unactionable wall on every PR. Ordering matters: bump the base first, then add the OS-package gate. Deferred and tracked: #1842 (bump the alpine runtime base, then add Trivy image scanning), #1837 (the unfixable x/crypto advisory this check tolerates by design). Closes #1836
cristim
added a commit
that referenced
this pull request
Aug 18, 2026
…isories The pinned runtime base alpine:3.21.3 shipped 17 fixable CRITICAL and HIGH OS advisories, measured with --ignore-unfixed so every one had a published fix available. The notable one is a libssl3 heap buffer overflow triggered by a large X.509 certificate, in a service that terminates TLS and connects to databases over TLS. Bumped to alpine:3.24.1, re-pinned by digest. The digest was confirmed three independent ways rather than trusting a single server-reported value: docker buildx imagetools inspect, the registry v2 API Docker-Content-Digest header, and a content hash of the fetched manifest. It is the multi-arch OCI image index digest rather than a per-platform manifest, so it stays correct for every TARGETARCH. All three were re-verified immediately before committing, because a genuine digest went stale mid-flight during #1835 when the upstream tag was re-pushed onto a rebuilt image. Every candidate was measured rather than assuming newest-is-cleanest, and that caution was warranted: 3.23.5 and 3.22.5 were rebuilt on 2026-06-22, after 3.24.1 was published on 2026-06-16, so newest-tag and most-recently-rebuilt disagreed. Verified against the built artifact rather than the base tag alone, which is the distinction #1836 exists for: the image was rebuilt and rescanned, and the image scanner added by #1841 still passes on it, so the gate that now blocks CI is not regressed. This unblocks adding Trivy scan-type image to CI. That was deliberately left out of #1841 because gating on OS packages while the base carried fixable advisories would have landed red on day one, and not gating would have printed an unactionable wall on every PR. The ordering was bump first, then gate; the gate is the remaining half. Note on process: #1842 was auto-closed by the #1841 merge without any base bump having landed, because that PR's description contained the phrase resolve #1842 while explaining the ordering. GitHub parses closing keywords in the PR description as well as the commit message and does not read the surrounding prose. It was reopened and the work done here. Closes #1842
This was referenced Aug 25, 2026
Closed
cristim
added a commit
that referenced
this pull request
Sep 27, 2026
…ce (#1836) Nothing in the pipeline inspected the container image we ship. govulncheck ran in source mode over the six modules, Trivy ran with scan-type fs and config, and docker-build built the image, ran --version against it, and discarded it. A vulnerable binary baked into the image was therefore unreachable by the pipeline by construction. That gap hid two consecutive rounds of CVE work. #1832 bumped the Go toolchain CI reads and govulncheck went green while the image still built on go1.26.5. #1835 bumped the base image and /app/cudly came out clean, but the image shipped a second Go binary: a prebuilt golang-migrate release carrying go1.25.4 with 69 vulnerabilities across 12 modules, executed on every container start by entrypoint.sh with DB_AUTO_MIGRATE=true by default. A new job runs govulncheck -mode=binary over every Go binary in the final stage after docker-build, and is wired into ci-success so it gates. It fails on any advisory with a published fixed version at any severity, and tolerates but always prints advisories with no published fix, so it does not land permanently red on GO-2026-5932 (x/crypto/openpgp, introduced: "0", no fixed event, unreachable here) while still failing on the classes that actually occurred. The scanner was itself fail-open twice, and both were closed before merge. Its jq filter returned an empty array with exit 0 for an empty stream, for {}, and for a well-formed object with no config, all of which the classifier read as zero findings; the existing guards only fired on a parse error and none of those inputs is one, so a govulncheck run that silently produced nothing would have certified the image clean. The parser now requires the stream to open with a config message. That check then accepted any non-empty protocol_version, which would have reproduced the same hole on a tool upgrade emitting a different schema, so it is pinned to v1.0.0 with the expected and found values reported on mismatch. Verified both directions with recorded fixtures: red on a known-bad image carrying the prebuilt migrate binary, green on the current image, empty and malformed and wrong-schema streams all exit 2 rather than clean, an unfixable-only stream is tolerated, and a fixable advisory fails. Suite is 9 passed, 0 failed, asserting message content rather than exit codes alone. Trivy scan-type image was deliberately not added. The pinned runtime base alpine:3.21.3 carries fixable CRITICAL and HIGH OS advisories today, including a libssl3 heap overflow from a large X.509 certificate, so a gating step would land red on day one for something this change does not fix and a non-gating one would print an unactionable wall on every PR. Ordering matters: bump the base first, then add the OS-package gate. Deferred and tracked: #1842 (bump the alpine runtime base, then add Trivy image scanning), #1837 (the unfixable x/crypto advisory this check tolerates by design). Closes #1836
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
#1832 bumped
GO_VERSIONinci.ymland thego-versionpin inpre-commit.ymlto 1.26.6, which turned govulncheck green. It did not change what we ship: the production builder stage still compiled on go1.26.5, so all seven stdlib advisories from #1829 stayed in the deployed binary.This makes the whole shipped image stdlib-clean. Three things:
Dockerfile(production builder),Dockerfile.devandDockerfile.test, from1.26.5-alpine3.24to1.26.6-alpine3.24.godirectives ingo.workand the five modulego.modfiles.golang-migrateis now compiled from source instead of downloaded as an upstream prebuilt release tarball. See below: this was the larger half of the problem.No suppression of any kind: no ignore list, no
-scannarrowing, nocontinue-on-error, no allowlist.The image ships two Go binaries, and the base bump only fixed one
/usr/local/bin/migratewas an upstream prebuilt release (golang-migrate v4.19.1) downloaded from GitHub and copied into the final stage. It is not compiled by our toolchain, so nothing about a base-image bump touched it. Measured on an image built from this branch before the fix:35 of those are stdlib, and all seven advisories named in #1833 were present (GO-2026-6218, GO-2026-6091, GO-2026-6090, GO-2026-6089, GO-2026-5026, GO-2026-6088, GO-2026-5972).
This is not dormant code.
scripts/entrypoint.shrunsmigrate -path ... -database ... upon every container start, andDB_AUTO_MIGRATEdefaults totruein this image. The vulnerable symbols govulncheck lists includeurl.URL.Parse,url.URL.ResolveReferenceandtls.Conn.Handshake, which are exactly the paths that parse the database URL and negotiate TLS to the database.Bumping the pinned release was not an option: v4.19.1 is the latest upstream tag, and the fix for the 1.25 line is go1.25.13. So it is now built with
go installon the stage's own pinned toolchain, the same waymake install-toolsdoes.Two notes on that change:
sum.golang.org(noGOPROXY/GOFLAGS/GONOSUMDBoverrides exist in this repo). Themodline in the built binary now carries the module hash, where the prebuilt binary's was blank.-tags=postgres, so only the postgres/postgresql driver is linked in rather than upstream's 14 database drivers and 9 sources. Thefilesource is registered unconditionally by the CLI, andentrypoint.shonly ever constructs apostgresql://URL. The binary drops from 51.9 MB to 5.5 MB.The digest pin was refreshed, not dropped
The
@sha256:pin is kept. The new digest was resolved two independent ways, which agree:Docker-Content-Digestforlibrary/golang:1.26.6-alpine3.24docker buildx imagetools inspect golang:1.26.6-alpine3.24Both return
sha256:3889b425f035be855a72fb4755265311293b6d414521f0a519d819df32222d83, an OCI image index covering linux/amd64 and linux/arm64. Thealpine3.24variant does exist for 1.26.6, so the tag shape is unchanged.Worth recording: the
1.26.6-alpine3.24tag was re-pushed on 2026-08-16 and previously resolved tosha256:af8d6740.... Both digests carryorg.opencontainers.image.version: 1.26.6-alpine3.24, so this was a legitimate rebuild on a patched Alpine base rather than anything suspicious. It is a good illustration of why the digest pin is worth keeping: the tag moved underneath us, and a digest-pinned build does not silently follow it.The
godirective bump is load-bearing, not tidinessaws_sanity.yml,azure_sanity.ymlanddatabase-migration.yml(three call sites) resolve their toolchain withgo-version-file: go.mod, not fromGO_VERSION. No module carries atoolchaindirective, sosetup-goreads thegodirective directly.Measured rather than reasoned about.
azure_sanityrun 31898391027 from 2026-08-15, after #1832 merged, logs:So those jobs were still installing 1.26.5.
aws_sanityshort-circuits beforesetup-gowhen its AWS secrets are absent, anddatabase-migration.ymlisworkflow_dispatch-only so it has no recent runs, but all five call sites read the same rootgo.modand are fixed by the same one-line change.pre-commit.ymlwas the last workflow still carrying a literalgo-version, and now readsgo-version-file: go.modtoo. The resolved version is unchanged; this removes one of the independent places the Go version had to be bumped by hand, which is the arrangement that produced #1833.tests/e2e/go.moddeliberately stays atgo 1.25No workflow resolves a toolchain from it,
GOTOOLCHAIN=autoonly ever upgrades and never downgrades, the module is stdlib-only and test-only, and it is not part of the shipped artifact. It reported clean on both 1.26.5 and 1.26.6. Raising its floor would be an unrelated change.Verification
Full runtime images were built from this branch for both target architectures, and both binaries extracted from each image and scanned with
govulncheck -mode=binaryat the CI-pinned v1.1.4:go version -m/app/cudlygo1.26.6Your code is affected by 2 vulnerabilities from 2 modules./app/cudlygo1.26.6Your code is affected by 2 vulnerabilities from 2 modules./usr/local/bin/migratego1.26.6No vulnerabilities found./usr/local/bin/migratego1.26.6No vulnerabilities found.All seven advisories from #1833 are absent from all four binaries.
On how that was counted. The stdlib column is derived from
govulncheck -format json, counting distinct OSV entries whose finding traces into modulestdlib. It is deliberately not a textgrep: govulncheck v1.1.4 prints a bareStandard libraryline and never the stringModule: stdlib, so agrep -c "Module: stdlib"returns 0 whether or not stdlib CVEs are present. The scan harness was validated against a known-dirty control (the previously shipped prebuilt binary, checksum-matched to the SHA256 this PR removes), where it correctly reports 35 stdlib findings and all seven advisories present.Runtime checks in the final image:
migrateresolves onPATHto/usr/local/bin/migrateownedcudly:cudly,migrate -versionreportsv4.19.1, and a realmigrate -path /app/migrations -database postgresql://... upgets as far as the TCP connect, confirming both the file source and the postgresql driver are registered. The entrypoint is unchanged.The two remaining findings on
/app/cudlyare pre-existing and non-stdlib:golang.org/x/crypto(openpgp, unmaintained)go.opentelemetry.io/otelv1.43.0Neither is introduced here: this PR changes zero dependency versions (
go.sumdiff is empty). GO-2026-5158 has a fix available and is worth a follow-up issue.Also run at the exact CI pins:
go build ./...(0),go vet ./...(0), golangci-lint v2.10.1 (0 issues.), gosec v2.28.0 (0).Known gaps, stated rather than left implicit
scan-types in the workflow set arefsandconfig;docker-buildbuilds the image, runs--version, and discards it. That is the structural reason a vulnerable binary could sit in the image invisibly. Being filed separately rather than bundled here.Dockerfile.devstill downloads a prebuiltmigrate(v4.17.0, older still). It is a local development image, not shipped, so it is out of scope for this PR, but it is the same defect class and worth a follow-up.Closes #1833
Summary by CodeRabbit
Chores
Documentation