sec(ci): nothing scans the container image we ship, only source; a baked-in vulnerable binary is invisible by construction

[cristim]

Surfaced by the adversarial review of #1835 (which closes #1833). This is the structural reason #1833 happened and why its first fix was incomplete.

What

Nothing in the pipeline inspects the container image that actually ships.

• The only two scan-type keys across the whole workflow set are fs (repo source) and config (Terraform): .github/workflows/ci.yml:546 and :585.
• govulncheck runs in source mode over the six Go modules.
• docker-build (ci.yml:326) builds the image, runs --version / --help against it, and discards it.

So a vulnerable third-party binary baked into the image is unreachable by the pipeline by construction.

Why it matters, concretely

This is not hypothetical. The review of #1835 found that the image ships two Go binaries:

• /app/cudly, compiled by our toolchain, which the sec(build): build the shipped image on go1.26.6 to clear the stdlib CVEs #1835 base-image bump fixes
• /usr/local/bin/migrate, an upstream prebuilt golang-migrate v4.19.1 release downloaded in Dockerfile:43-54 and copied in at Dockerfile:140

The prebuilt binary reports go1.25.4 and carries all seven advisories from #1833 plus 62 more. scripts/entrypoint.sh:69 runs it on every container start (DB_AUTO_MIGRATE=true by default, Dockerfile:155-156), and the vulnerable symbols include url.URL.Parse and tls.Conn.Handshake, the paths that parse the database URL and negotiate TLS to the database.

A source scan cannot see any of that. Two rounds of "fix the Go version" both went green while the shipped image still executed the CVEs at startup.

Fix direction

After docker-build, scan the artifact rather than the source. Either:

1. A Trivy step with scan-type: image against the built tag, or
2. A govulncheck -mode=binary loop over the binaries in the final stage (/app/cudly, /usr/local/bin/migrate)

Either would have caught this automatically.

Verification bar

• Both directions. Prove the new scan fails on an image known to contain a vulnerable binary, and passes on a clean one. A scan step that always passes is worse than none, because it looks like coverage.
• Assert with a sound method. The review also found that the manual check used to date, grep -c "Module: stdlib", matches nothing in govulncheck v1.1.4, which prints a bare Standard library line instead. That grep returns 0 whether or not stdlib CVEs are present. Assert on the summary line, or use -format json and count OSV entries whose module is stdlib. Never a bare grep -c whose zero result is indistinguishable from a pattern typo.
• Allowlist on success. A check written as != "failure" reports success for cancelled and skipped.
• Watch the runtime cost: an image scan on every PR is slower than a source scan. If that is a problem, gate it on the docker-build job rather than dropping it.

Related

• sec(build): shipped image still builds on go1.26.5, so the stdlib CVEs remain in the artifact after #1832 #1833 — shipped image still built on go1.26.5 (the instance)
• sec(build): build the shipped image on go1.26.6 to clear the stdlib CVEs #1835 — the fix, where this was found
• fix(ci): govulncheck fails repo-wide on 7 stdlib CVEs fixed in go1.26.6, blocking every merge #1829 / fix(ci): bump pinned Go toolchain to 1.26.6 for stdlib CVE fixes #1832 — the original CI-only toolchain bump that went green while the artifact stayed vulnerable