Skip to content

perf(build): cache Go build + module dirs in Dockerfile builder#194

Merged
sunib merged 1 commit into
mainfrom
perf/dockerfile-build-cache
Jul 5, 2026
Merged

perf(build): cache Go build + module dirs in Dockerfile builder#194
sunib merged 1 commit into
mainfrom
perf/dockerfile-build-cache

Conversation

@sunib

@sunib sunib commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

What

Add BuildKit cache mounts for the Go build cache (/root/.cache/go-build) and module cache (/go/pkg/mod) to the Dockerfile builder stage.

Why

The builder stage had no cache mount, so every docker build started with an empty Go build cache and recompiled the entire module (client-go, controller-runtime, all deps) from scratch (~100 s) — even for a one-line source change.

That is the dominant cost of the e2e image-refresh leg, which deliberately changes a source file and rebuilds the controller ~5 times per run to prove a change triggers a rebuild + rollout. On the newly-sharded CI matrix that leg is the critical path at ~15 min (run 28681013051), almost all of it cold recompiles. This surfaced while analysing the e2e sharding (see docs/design/e2e-ci-runner-sharding-plan.md).

Effect

A source change still recompiles (only the changed packages + dependents), so the image-refresh assertions still hold — but a rebuild no longer recompiles the world.

Verified locally: after appending a comment to cmd/main.go (exactly what image-refresh S2 does), the go build step dropped from a cold full compile to ~2 s incremental (5 s total build). Expected to cut the image-refresh leg roughly in half and also speed up local task test-e2e rebuilds and the CI build job on warm cache.

Safety

  • The cache is content-addressed and keyed by GOOS/GOARCH/flags, so cross-arch and coverage (GOCOVER) builds stay isolated.
  • Requires BuildKit — the default on modern Docker.
  • Final binary is byte-for-byte the same; only build caching changes.

Validation

  • task lint (incl. hadolint) — green
  • Local docker build twice with a source change between — builds succeed, rebuild is ~2 s
  • Full task test-e2e deferred to CI (this PR's e2e run exercises the image-refresh chain directly)

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
    • Improved build performance by reusing cached dependencies during container builds.
    • Reduced rebuild times for development and CI workflows while preserving existing build behavior.

The builder stage had no BuildKit cache mount, so every `docker build`
started with an empty Go build cache and recompiled the entire module —
client-go, controller-runtime, all deps — from scratch (~100s), even for a
one-line source change.

This is the dominant cost of the e2e image-refresh leg, which deliberately
changes a source file and rebuilds the controller ~5 times per run to prove
that a change triggers a rebuild + rollout. On the sharded CI matrix that
leg is the critical path at ~15 min (run 28681013051), almost all of it cold
recompiles.

Mount the Go build cache (/root/.cache/go-build) and module cache
(/go/pkg/mod) as BuildKit cache mounts so they persist across builds. A
source change still recompiles (only the changed packages + dependents), so
the image-refresh assertions still hold, but a rebuild no longer recompiles
the world. Verified locally: after appending a comment to cmd/main.go, the
go build step dropped from a cold full compile to ~2s incremental. Also
speeds up local `task test-e2e` rebuilds and the CI build job on warm cache.

The cache is content-addressed and keyed by GOOS/GOARCH/flags, so cross-arch
and coverage (GOCOVER) builds stay isolated. Requires BuildKit, the default
on modern Docker. See docs/design/e2e-ci-runner-sharding-plan.md for the
sharding analysis that surfaced this.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 5, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The Dockerfile was updated to use BuildKit cache mounts for Go module downloads and build steps, persisting the module cache and build cache across builds while preserving existing build flags and cross-platform variables.

Changes

Dockerfile BuildKit Caching

Layer / File(s) Summary
Add cache mounts for module download and build steps
Dockerfile
Adds BuildKit cache mounts for the Go module cache (/go/pkg/mod) during go mod download, and both the module cache and build cache (/root/.cache/go-build) during go build, keeping existing GOOS/GOARCH/coverage flag behavior.

Estimated code review effort: 1 (Trivial) | ~5 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the Dockerfile BuildKit cache-mount optimization in the builder stage.
Description check ✅ Passed The description is detailed and covers the change, motivation, validation, and safety, but it does not follow the full template structure.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch perf/dockerfile-build-cache

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Dockerfile`:
- Around line 25-27: The current Dockerfile cache mount in the build step only
helps within the same BuildKit builder and will not persist across separate
GitHub-hosted CI runs. Update the Dockerfile/build workflow around the go mod
download step and the related cache setup to use a workflow-level cache
restore/save approach (for example, a cache-dance style step) so the module
cache is actually reused across runs, while keeping the existing RUN
--mount=type=cache usage for in-job reuse.
- Around line 45-47: Add a Dockerfile syntax directive at the very top of the
Dockerfile so the BuildKit cache mounts in the RUN step are supported; the issue
is that the current Dockerfile uses --mount=type=cache without declaring the
required frontend. Update the Dockerfile header to use the Dockerfile 1.x syntax
directive, then keep the existing RUN instruction with the cache mounts
unchanged. Locate the fix near the RUN build step that invokes go build.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1156e239-8955-4455-9e79-644bb0b76f2f

📥 Commits

Reviewing files that changed from the base of the PR and between bda7e63 and ee40615.

📒 Files selected for processing (1)
  • Dockerfile

Comment thread Dockerfile
Comment thread Dockerfile
@codecov

codecov Bot commented Jul 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@sunib sunib merged commit 55ed662 into main Jul 5, 2026
17 checks passed
@sunib sunib deleted the perf/dockerfile-build-cache branch July 5, 2026 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant