build(cache): share sccache across worktrees with mode-aware wrappers

[hyperb1iss]

🗄️ One compile cache for every worktree

Executes the Sibyl sidequest "Share Hypercolor build caches across worktrees", filed after a disk audit found ~700 GB of build artifacts across the root checkout, one worktree, and an orphaned cache tree. Complements the incremental-state cleanup that already reclaimed ~132 GB; this PR is the part that stops the regrowth.

💡 What this is

Sharing one Cargo target dir between worktrees is the wrong primitive for this repo: Cargo's target lock is coarse, so two agents building in parallel would serialize, and alternating feature shapes would thrash the shared tree. Instead, every worktree keeps its own target dir and the sharing happens one layer down — a single bounded sccache under ~/.cache/hypercolor/sccache that every worktree's compiles hit. A second worktree's cold build becomes mostly cache hits without any cross-worktree locking.

The wrappers previously enabled sccache only for release builds (and it was not installed on the primary dev machine at all), so every fingerprint bust — toolchain bump, lockfile change, profile switch, new worktree — recompiled Servo at full price, per profile tree, per worktree.

🎯 The invariant

sccache 0.17 hard-errors the moment it sees incremental compilation, both via the CARGO_INCREMENTAL env var and via arg-level -Cincremental (verified by probe; there is no passthrough). The property to anchor on: no code path in either wrapper can select sccache and leave incremental enabled in the same invocation. Everything else in the wrapper diff is plumbing around that exclusion.

🛠️ How it works

The wrappers pick a mode per command, from the cargo subcommand and profile flags:

Command shape	Mode	Reason
cargo build / test / bench / nextest, any --release/--profile release|bench	sccache, CARGO_INCREMENTAL=0	whole-tree codegen; identical units shared across worktrees
cargo run	incremental	edit-run loop; measured hypercolor-core edit-rebuild: 44.8s non-incremental vs 11.4s incremental
cargo check / clippy	incremental	sccache cannot cache --emit=metadata units at all
anything else (deny, doc, …)	incremental	nothing to gain

Incremental mode also drops an ambient sccache RUSTC_WRAPPER if one leaks in from the environment, so the sccache+incremental hard-error is unreachable from any wrapper path.

1. scripts/cargo-cache-build.ps1 — subcommand detection, the mode split, SCCACHE_CACHE_SIZE (default 75G, HYPERCOLOR_SCCACHE_SIZE to override), and Windows C/C++ caching: CC/CXX = sccache cl for cc-rs consumers (mozangle's ANGLE build, the largest repeat C++ cost at 12 GB of stale build output on the audited machine) and CMAKE_*_COMPILER_LAUNCHER for CMake consumers (turbojpeg). Also sets rust-lld as the MSVC linker via CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER for non-release builds only — env-only so the committed .cargo/config.toml stays portable, and gated off release-like profiles so shipped artifacts (CI sidecars, the installer, app bundles built outside this wrapper) all keep link.exe.
2. scripts/cargo-cache-build.sh — the same mode logic for Linux/macOS. The C/C++ ccache routing it already had is unchanged.
3. Escape hatches, in precedence order — a pre-set nonzero CARGO_INCREMENTAL always wins; HYPERCOLOR_NO_SCCACHE=1 (session) and HYPERCOLOR_ITERATE=1 (per invocation) force incremental mode; HYPERCOLOR_FORCE_SCCACHE=1 forces sccache mode. CI does not run sccache at all today (verification grep: zero sccache references in .github/, none preinstalled on hosted runners) — it sets CARGO_INCREMENTAL=0 globally, which both old and new wrappers preserve identically, so CI behavior is bit-for-bit unchanged. The doc's CI section previously claimed a CI sccache setup that never existed; it now describes reality, and the future CI sccache lane is a filed sidequest.
4. Iteration recipes pin incremental — just test-crate, test-one, app, and the daemon build inside scripts/dev-windows.ps1 export HYPERCOLOR_ITERATE=1, because those are edit loops even though their subcommands are test/build. Alternating one profile tree between modes rebuilds only workspace crates (~50s measured), never dependencies.
5. justfile disk hygiene — just disk (usage report incl. sccache stats), just gc (cargo-sweep: orphaned toolchains, then >14-day artifacts), just gc-worktrees (the same sweep across every worktree lane), just gc-deep (also drops incremental state and the cpu-smoke lane). just e2e-build-cpu now builds into target/cpu-smoke so the --no-default-features feature unification stops churning the daily tree; an ambient CARGO_TARGET_DIR (CI) still wins.
6. docs/development/SERVO_BUILD_CACHING.md — documents the mode split, the cross-worktree topology, and the disk bounds.

flowchart LR
    A[worktree A\ntarget/] -->|compile units| S[(sccache\n75G bounded)]
    B[worktree B\ntarget/] -->|compile units| S
    C[worktree C\ntarget/] -->|compile units| S
    A -.-> M[(mozbuild state)]
    B -.-> M
    A --> X[target/cpu-smoke\nisolated feature shape]

Loading

🧪 Validation

• Probe: RUSTC_WRAPPER=sccache + CARGO_INCREMENTAL=1 → sccache: incremental compilation is prohibited (exit 101); same with the env unset and dev-profile -Cincremental args. This is the measured basis for the mode split.
• Measurement: touch-rebuild of hypercolor-core --features servo, warm deps, same machine: 44.8s with CARGO_INCREMENTAL=0 vs 11.4s incremental.
• Wrapper smoke on Windows, build-turbo worktree: cargo build -p hypercolor-types in sccache mode → exit 0, sccache stats 15 misses (cold fill); after cargo clean, rebuild → 11 cache hits (the 5–8 non-cacheable calls are proc-macro crates, which sccache can never cache). cargo check → incremental mode banner, exit 0. cargo run --help → incremental mode banner. cargo test -p hypercolor-types --no-run → sccache mode banner, Finished in 17.6s.
• Adversarial verification round (separate agent, 14-scenario mode/precedence harness + live ps1 probes): verdict PASS, no reachable path to the sccache+incremental combo. Its findings are fixed in the follow-up commit: rust-lld gated off release-like builds, the doc's fictional CI sccache section rewritten to reality, nextest added to the sccache list, the sh no-args default moved ahead of mode detection, ambient RUSTC_WRAPPER dropped in incremental mode, the ps1 --profile scan made monotonic, and just disk hardened against empty targets.
• Post-fix probes: --release build → sccache mode with no rust-lld line; dev build → sccache mode + rust-lld; ambient RUSTC_WRAPPER=sccache + cargo check → "unset ambient sccache RUSTC_WRAPPER" + incremental mode. PowerShell parser 0 errors, bash -n clean, just --list exit 0, just disk and just gc run live.
• ⚠️ Not exercised in this session: the unix wrapper at runtime (syntax-checked and logic-harnessed only; CI is the backstop), the mozangle sccache cl path end to end, and a full daemon build — the first just daemon on this branch proves the C++ cache path, and the first non-release build after the linker change re-runs build scripts and relinks once (one-time warm-up, cold C++ cache included).

🔍 What reviewers should focus on

• The subcommand-detection loops in both wrappers (flag skipping, +toolchain, the no-args default) — a wrong subcommand read routes a command into the wrong mode. Known nit, deliberately unfixed: a value-taking flag before the subcommand (cargo --config x.y build) mis-detects and fails safe into incremental mode; no caller does this.
• The precedence chain among HYPERCOLOR_FORCE_SCCACHE, HYPERCOLOR_NO_SCCACHE, HYPERCOLOR_ITERATE, and a pre-set CARGO_INCREMENTAL, especially against CI's global CARGO_INCREMENTAL=0.
• CC='sccache cl' on MSVC relies on cc-rs's known-wrapper splitting; if any build script mishandles it, the C/C++ launcher lines are the first thing to pull.

📌 Follow-ups (deliberate non-fixes)

• 251 integration-test binaries (one per file) dominate just verify link time and deps disk; consolidation to one harness per crate + cargo-nextest is filed as a Sibyl sidequest.
• A Windows CI servo lane sharing this cache topology (and possibly an S3-compatible remote backend) is filed as a second sidequest.
• The dev/preview profile pair still means two Servo trees per worktree; whether preview's debug-assertions trade still earns two trees is an open question, not changed here.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Performance
  • Improved build caching and incremental compilation behavior across development, testing, and release workflows.
  • Added clearer cache controls and platform-specific linker and compiler caching support.
• Developer Experience
  • Added housekeeping commands for inspecting disk usage, collecting stale artifacts, and cleaning worktree and incremental-build data.
  • Improved deterministic behavior for iterative tests, application builds, and CPU smoke-stack builds.
• Documentation
  • Expanded guidance on caching, incremental builds, cache sharing, platform behavior, and CI cache workflows.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9ef869fc-6227-4db5-99dc-2d405741842e

📥 Commits

Reviewing files that changed from the base of the PR and between ff66e1c and 7459147.

📒 Files selected for processing (5)

• docs/development/SERVO_BUILD_CACHING.md
• justfile
• scripts/cargo-cache-build.ps1
• scripts/cargo-cache-build.sh
• scripts/dev-windows.ps1

---

📝 Walkthrough

Walkthrough

Build wrappers now select between sccache and Cargo incremental compilation based on command context and environment. Development recipes use explicit iteration and CPU-smoke lanes, while documentation and housekeeping recipes describe cache topology, CI reuse, disk limits, and cleanup.

Changes

Build caching and iteration behavior

Layer / File(s)	Summary
Wrapper cache-mode selection scripts/cargo-cache-build.sh, scripts/cargo-cache-build.ps1	Cargo commands and profiles now determine whether sccache or incremental compilation is enabled, with environment overrides, compiler cache configuration, and conditional Windows rust-lld selection.
Iteration and lane-specific recipes justfile, scripts/dev-windows.ps1	Development and test recipes opt into iteration mode, and CPU smoke-stack builds use a dedicated target directory.
Cache topology and maintenance guidance docs/development/SERVO_BUILD_CACHING.md, justfile	Documentation and recipes describe shared cache layout, CI cache reuse, disk reporting, garbage collection, and deep cleanup.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Developer
  participant BuildWrapper
  participant Cargo
  participant Sccache
  Developer->>BuildWrapper: run a build, test, or development recipe
  BuildWrapper->>BuildWrapper: inspect command, profile, and environment
  BuildWrapper->>Sccache: configure compiler cache when selected
  BuildWrapper->>Cargo: set incremental mode and execute command
  Cargo-->>Developer: produce build or test result

Loading

Poem

A rabbit hops through caches bright,
With incremental steps just right.
Sccache guards the compiled trail,
Smoke lanes keep their targets pale.
Cleanup sweeps the disk with cheer—
Fresh builds bloom throughout the year!

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}