perf(build): cache Go build + module dirs in Dockerfile builder#194
Merged
Conversation
The builder stage had no BuildKit cache mount, so every `docker build` started with an empty Go build cache and recompiled the entire module — client-go, controller-runtime, all deps — from scratch (~100s), even for a one-line source change. This is the dominant cost of the e2e image-refresh leg, which deliberately changes a source file and rebuilds the controller ~5 times per run to prove that a change triggers a rebuild + rollout. On the sharded CI matrix that leg is the critical path at ~15 min (run 28681013051), almost all of it cold recompiles. Mount the Go build cache (/root/.cache/go-build) and module cache (/go/pkg/mod) as BuildKit cache mounts so they persist across builds. A source change still recompiles (only the changed packages + dependents), so the image-refresh assertions still hold, but a rebuild no longer recompiles the world. Verified locally: after appending a comment to cmd/main.go, the go build step dropped from a cold full compile to ~2s incremental. Also speeds up local `task test-e2e` rebuilds and the CI build job on warm cache. The cache is content-addressed and keyed by GOOS/GOARCH/flags, so cross-arch and coverage (GOCOVER) builds stay isolated. Requires BuildKit, the default on modern Docker. See docs/design/e2e-ci-runner-sharding-plan.md for the sharding analysis that surfaced this. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe Dockerfile was updated to use BuildKit cache mounts for Go module downloads and build steps, persisting the module cache and build cache across builds while preserving existing build flags and cross-platform variables. ChangesDockerfile BuildKit Caching
Estimated code review effort: 1 (Trivial) | ~5 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@Dockerfile`:
- Around line 25-27: The current Dockerfile cache mount in the build step only
helps within the same BuildKit builder and will not persist across separate
GitHub-hosted CI runs. Update the Dockerfile/build workflow around the go mod
download step and the related cache setup to use a workflow-level cache
restore/save approach (for example, a cache-dance style step) so the module
cache is actually reused across runs, while keeping the existing RUN
--mount=type=cache usage for in-job reuse.
- Around line 45-47: Add a Dockerfile syntax directive at the very top of the
Dockerfile so the BuildKit cache mounts in the RUN step are supported; the issue
is that the current Dockerfile uses --mount=type=cache without declaring the
required frontend. Update the Dockerfile header to use the Dockerfile 1.x syntax
directive, then keep the existing RUN instruction with the cache mounts
unchanged. Locate the fix near the RUN build step that invokes go build.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add BuildKit cache mounts for the Go build cache (
/root/.cache/go-build) and module cache (/go/pkg/mod) to theDockerfilebuilder stage.Why
The builder stage had no cache mount, so every
docker buildstarted with an empty Go build cache and recompiled the entire module (client-go, controller-runtime, all deps) from scratch (~100 s) — even for a one-line source change.That is the dominant cost of the e2e
image-refreshleg, which deliberately changes a source file and rebuilds the controller ~5 times per run to prove a change triggers a rebuild + rollout. On the newly-sharded CI matrix that leg is the critical path at ~15 min (run28681013051), almost all of it cold recompiles. This surfaced while analysing the e2e sharding (seedocs/design/e2e-ci-runner-sharding-plan.md).Effect
A source change still recompiles (only the changed packages + dependents), so the image-refresh assertions still hold — but a rebuild no longer recompiles the world.
Verified locally: after appending a comment to
cmd/main.go(exactly what image-refreshS2does), thego buildstep dropped from a cold full compile to ~2 s incremental (5 s total build). Expected to cut theimage-refreshleg roughly in half and also speed up localtask test-e2erebuilds and the CIbuildjob on warm cache.Safety
GOOS/GOARCH/flags, so cross-arch and coverage (GOCOVER) builds stay isolated.Validation
task lint(incl.hadolint) — greendocker buildtwice with a source change between — builds succeed, rebuild is ~2 stask test-e2edeferred to CI (this PR's e2e run exercises the image-refresh chain directly)🤖 Generated with Claude Code
Summary by CodeRabbit