Skip to content

ci: darwin release builds are uncached — port rivet sccache/cache-mount Dockerfile pattern #1639

Description

@NathanFlurry

The "Build darwin binaries" jobs are the critical path for Publish npm in publish.yaml (~10.5 min per release) and rebuild ~388 crates from scratch every run.

Current state

  • docker/build/darwin.Dockerfile uses FROM ghcr.io/rivet-dev/rivet/builder-base-osxcross:0e33ceb98 — the same prebaked base rivet uses (osxcross + MacOSX 11.3 SDK + both apple Rust targets + sccache preinstalled) — but uses none of its caching affordances:
    • re-installs the rust toolchain/targets the base already has
    • no RUSTC_WRAPPER=sccache, no cache backend, no BuildKit cache mounts
    • plain docker build with no --cache-from/--cache-to
  • Measured on the 0.2.5 stable run (28801821648): base image pull ~1 min, rustup ~1 min, cargo build --release of 388 crates = 7m52s, every run, forever.
  • By contrast the linux sidecar/plugin jobs use Swatinem/rust-cache and finish in 1–5 min warm.

Fix: port the rivet consumer pattern

Rivet already solved this for the identical build in rivet-dev/rivet:

  • docker/build/darwin-{arm64,x64}.Dockerfile: RUSTC_WRAPPER=sccache + SCCACHE_WEBDAV_ENDPOINT=https://cache.depot.dev, --mount=type=secret,id=DEPOT_TOKEN,env=SCCACHE_WEBDAV_TOKEN with a graceful health-check fallback that disables sccache when the token/backend is unavailable, plus --mount=type=cache for cargo registry/git/target.
  • Base image build/push lives in docker/builder-base/osxcross.Dockerfile + scripts/docker-builder-base/build-push.sh.

Port notes for agentos:

  • agentos builds on GitHub-hosted runners (not Depot builders), so BuildKit cache mounts alone will not persist across runs — the cross-run win is the sccache remote backend. Either:
    • (a) add a DEPOT_TOKEN secret and reuse rivet's Depot cache config verbatim (preferred: byte-for-byte port of a production pattern), or
    • (b) use sccache's native GHA backend (SCCACHE_GHA_ENABLED=on).
  • Drop the redundant rustup toolchain/target install from darwin.Dockerfile (already in the base).
  • Pass the secret through in the build-sidecar-darwin job in .github/workflows/publish.yaml (docker buildx build --secret id=DEPOT_TOKEN).

Expected: darwin job drops from ~10 min to ~2–3 min warm, cutting release npm-publish latency roughly in half.

Related smaller wins (same investigation, can be separate PRs)

  • Publish crates.io job: give it its own concurrency group (or workflow) so a slow crates tail (crates.io new-crate rate-limit backoff can sleep for many minutes) doesn't queue the next release's npm path behind it (concurrency: publish-${{ github.ref }}, cancel-in-progress: false caused the 0.2.6 dispatch to sit pending behind 0.2.5's crates job). Also surface the "rate limited, retrying at " sleep as a ::notice:: so the job doesn't look hung.
  • Release rust-cache eviction: linux jobs key caches by <target>-<trigger>, so -release caches are only warmed by prior release runs and get LRU-evicted during quiet stretches while -preview stays warm; consider a shared key or a scheduled warm job.
  • WASM Commands (~8.6 min): cache packages/runtime-core/commands keyed on hashFiles('registry/native/**') and skip make on hit.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions