build(cache): share sccache across worktrees with mode-aware wrappers

docs/development/SERVO_BUILD_CACHING.md

@@ -58,11 +58,68 @@ The shared wrapper configures:
 
 - `CARGO_TARGET_DIR=<repo>/target` (unless already set)
 - `MOZBUILD_STATE_PATH=$HOME/.cache/hypercolor/mozbuild` (unless already set)
-- Cargo incremental compilation for local dev and preview-style builds
-- `sccache` as `RUSTC_WRAPPER` for release/bench builds, or whenever
-  `HYPERCOLOR_FORCE_SCCACHE=1`
+- `sccache` as `RUSTC_WRAPPER` for whole-tree codegen commands
+  (`cargo build`, `test`, `bench`, and anything release/bench-profiled)
+  when installed, with a bounded on-disk cache (default `75G`, override
+  with `HYPERCOLOR_SCCACHE_SIZE`). sccache and incremental compilation are
+  mutually exclusive, so these commands run with `CARGO_INCREMENTAL=0`.
+- Cargo incremental compilation for iteration and metadata commands:
+  `cargo run` (the edit-run loop; a measured hypercolor-core edit-rebuild
+  is ~45s non-incremental vs ~11s incremental), `cargo check`, and
+  `clippy` (sccache cannot cache `--emit=metadata` units). The
+  iteration-shaped recipes (`just test-crate`, `test-one`, the Unix `app`
+  build, and the Windows `just dev` daemon build) pin incremental via
+  `HYPERCOLOR_ITERATE=1`; the Windows `app` recipe uses `cargo run` and
+  lands there by subcommand.
+- Opt-outs: `HYPERCOLOR_NO_SCCACHE=1` disables sccache for the session;
+  `HYPERCOLOR_ITERATE=1` does the same per invocation when you want
+  incremental rebuilds in a tight edit loop; a pre-set non-zero
+  `CARGO_INCREMENTAL` always wins. Alternating the same profile tree
+  between the two modes rebuilds only workspace crates (~50s measured),
+  never dependencies.
+- `rust-lld` as the linker on `x86_64-pc-windows-msvc` for non-release
+  builds (`HYPERCOLOR_NO_FAST_LINK=1` to opt out); release-like builds
+  keep `link.exe` so shipped artifacts all come off the same linker
 - `clang` + `ld.lld` for faster link steps on `x86_64-unknown-linux-gnu` when available
-- `ccache` for `CC`/`CXX` when installed, otherwise `sccache` if available
+- C/C++ caching for `cc`- and CMake-driven native deps (mozangle/ANGLE,
+  turbojpeg): `ccache` or `sccache` on Unix (both modes), `sccache` around
+  `cl.exe` on Windows (sccache mode only)
+
+## Cross-Worktree Topology
+
+Multiple worktrees (and multiple agents) build this repo concurrently. The
+sharing layer is the compile cache, not the target dir:
+
+- **Per-worktree `target/`** stays the default. Cargo's target lock is
+  coarse; a shared target dir would serialize parallel builds across
+  worktrees and thrash on feature-shape differences.
+- **Shared, bounded caches** live under `$HOME/.cache/hypercolor`
+  (`HYPERCOLOR_CACHE_DIR` to relocate): `sccache/` for compiled units,
+  `mozbuild/` for SpiderMonkey build state. A second worktree's cold build
+  becomes mostly cache hits without any cross-worktree locking.
+- **Incompatible feature shapes get isolated lanes**: `just e2e-build-cpu`
+  builds into `target/cpu-smoke` so the `--no-default-features` unification
+  never churns the daily tree.
+- **`mozjs_sys` uses prebuilt SpiderMonkey archives by default.** It falls
+  back to a source build silently, e.g. when a package profile override
+  drops `mozjs_sys` below `-O3`; keep the `opt-level = 3` overrides in
+  `Cargo.toml` intact.
+
+## Disk Bounds
+
+Target dirs grow without bound as toolchains, lockfiles, and feature shapes
+churn — Cargo never garbage-collects them. Bound them with:
+
+```bash
+just disk          # per-profile + shared-cache usage report
+just gc            # sweep orphaned-toolchain and >14-day artifacts here
+just gc-worktrees  # the same sweep across every worktree lane
+just gc-deep       # additionally drop incremental state + the cpu-smoke lane
+```
+
+`sccache` trims itself to `SCCACHE_CACHE_SIZE`; the cache size only applies
+when the server starts, so after changing it run `sccache --stop-server`
+first.
 
 ## Verify Cache Hits
 
@@ -78,26 +135,22 @@ Look for increasing cache hit counts after the first Servo build.
 The reusable action `.github/actions/rust-build-cache` configures GitHub
 Actions builds with:
 
-- `mozilla-actions/sccache-action` using GitHub's sccache backend
-- `HYPERCOLOR_FORCE_SCCACHE=1`
-- `CARGO_INCREMENTAL=0`
 - `Swatinem/rust-cache` for Cargo and extra cache directories
+- `CARGO_INCREMENTAL=0` (set workflow-wide)
 - `.cache/hypercolor/target` for CI-selected Cargo target shards
 - `.cache/hypercolor/mozbuild`
 - `.cache/hypercolor/toolchain`
 - `.cache/hypercolor/ccache`
 
-The manual `.github/workflows/servo-cache-warm.yml` workflow warms three
-compatible shapes when a maintainer deliberately refreshes Servo caches:
-
-| Suite        | Shared Key     | Extra Key    | Purpose                                          |
-| ------------ | -------------- | ------------ | ------------------------------------------------ |
-| Core Servo   | `servo-core`   | empty        | core Servo check, test, and clippy artifacts     |
-| Daemon Servo | `servo-daemon` | empty        | daemon Servo check, test, and clippy artifacts   |
-| E2E Servo    | `servo-daemon` | `e2e-dev-v1` | daemon and CLI binaries for the normal E2E stack |
+CI does not run sccache today: hosted runners do not preinstall it and the
+workflows do not set it up. The wrappers honor `HYPERCOLOR_FORCE_SCCACHE=1`
+and a pre-set `CARGO_INCREMENTAL=0`, so a future CI sccache lane only needs
+to install the binary and set the flag.
 
-The main CI workflow reuses those same shared keys in the explicit Servo check,
-test, and E2E build lanes. Pull requests keep the separate Servo check/test
+The manual `.github/workflows/servo-cache-warm.yml` workflow warms the
+`servo` shared cache key when a maintainer deliberately refreshes Servo
+caches; the main CI workflow reuses that key in its Servo check, test, and
+E2E build lanes. Pull requests keep the separate Servo check/test
 lanes out of the default path and rely on the normal Servo E2E stack for HTML
 renderer coverage. Pushes to `main`, tags, and manual CI dispatches still run
 the full Servo check/test gates.

justfile

@@ -142,23 +142,23 @@ test *args='':
 test *args='':
     powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -File scripts/cargo-cache-build.ps1 cargo test {{ workspace_args }} {{ args }}
 
-# Run tests for a specific crate
+# Run tests for a specific crate (iteration-shaped: keeps incremental rebuilds)
 [unix]
 test-crate crate *args='':
-    ./scripts/cargo-cache-build.sh cargo test -p {{ crate }} {{ args }}
+    HYPERCOLOR_ITERATE=1 ./scripts/cargo-cache-build.sh cargo test -p {{ crate }} {{ args }}
 
 [windows]
 test-crate crate *args='':
-    powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -File scripts/cargo-cache-build.ps1 cargo test -p {{ crate }} {{ args }}
+    HYPERCOLOR_ITERATE=1 powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -File scripts/cargo-cache-build.ps1 cargo test -p {{ crate }} {{ args }}
 
-# Run a specific test by name
+# Run a specific test by name (iteration-shaped: keeps incremental rebuilds)
 [unix]
 test-one name *args='':
-    ./scripts/cargo-cache-build.sh cargo test {{ workspace_args }} {{ name }} {{ args }}
+    HYPERCOLOR_ITERATE=1 ./scripts/cargo-cache-build.sh cargo test {{ workspace_args }} {{ name }} {{ args }}
 
 [windows]
 test-one name *args='':
-    powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -File scripts/cargo-cache-build.ps1 cargo test {{ workspace_args }} {{ name }} {{ args }}
+    HYPERCOLOR_ITERATE=1 powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -File scripts/cargo-cache-build.ps1 cargo test {{ workspace_args }} {{ name }} {{ args }}
 
 # Manually run the Cinder/Leptos extension design audit snapshot generator
 cinder-audit:
@@ -364,10 +364,10 @@ app-assets:
     just ui-build
     just effects-build
 
-# Run the unified desktop app
+# Run the unified desktop app (iteration-shaped: keeps incremental rebuilds)
 [unix]
 app *args='': app-assets
-    ./scripts/cargo-cache-build.sh cargo build -p hypercolor-daemon --bin hypercolor-daemon -p hypercolor-app --bin hypercolor-app --profile preview
+    HYPERCOLOR_ITERATE=1 ./scripts/cargo-cache-build.sh cargo build -p hypercolor-daemon --bin hypercolor-daemon -p hypercolor-app --bin hypercolor-app --profile preview
     "${CARGO_TARGET_DIR:-target}/preview/hypercolor-app" {{ args }}
 
 [windows]
@@ -674,10 +674,14 @@ e2e-build:
     just effects-build
     just ui-build
 
-# Build the fallback CPU smoke stack without the Servo renderer
+# Build the fallback CPU smoke stack without the Servo renderer.
+# Isolated target lane: the --no-default-features shape unifies crate features
+# differently from the daily builds, and letting it share target/ churns and
+# strands artifacts for the whole dependency graph on every alternation.
+# CI pins CARGO_TARGET_DIR per lane, so an ambient value wins.
 e2e-build-cpu:
-    ./scripts/cargo-cache-build.sh cargo build -p hypercolor-daemon --no-default-features --features builtin-drivers
-    ./scripts/cargo-cache-build.sh cargo build -p hypercolor-cli
+    CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target/cpu-smoke}" ./scripts/cargo-cache-build.sh cargo build -p hypercolor-daemon --no-default-features --features builtin-drivers
+    CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target/cpu-smoke}" ./scripts/cargo-cache-build.sh cargo build -p hypercolor-cli
     just effects-build
     just ui-build
 
@@ -814,6 +818,38 @@ udev-install:
 clean:
     ./scripts/cargo-cache-build.sh cargo clean
 
+# Plain sh lines, no shebang blocks: shebang recipes resolve `bash` from
+# PATH, which on Windows can be WSL bash that cannot read the temp script
+# path. `sh -cu` lines are the pattern every [windows] recipe already
+# proves.
+
+# Report build artifact and shared cache disk usage for this checkout
+disk:
+    @echo '── target profiles ──'
+    @if [ -d "${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target}" ]; then du -sh "${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target}"/* 2>/dev/null | sort -rh | head -15 || true; else echo '(no target dir)'; fi
+    @echo "── shared caches (${HYPERCOLOR_CACHE_DIR:-$HOME/.cache/hypercolor}) ──"
+    @if [ -d "${HYPERCOLOR_CACHE_DIR:-$HOME/.cache/hypercolor}" ]; then du -sh "${HYPERCOLOR_CACHE_DIR:-$HOME/.cache/hypercolor}"/* 2>/dev/null | sort -rh || true; else echo '(no cache dir)'; fi
+    @if command -v sccache >/dev/null 2>&1; then echo '── sccache ──'; sccache --show-stats | grep -E 'Cache hits|Cache misses|Cache size|Max cache' || true; fi
+
+# Sweep stale build artifacts (orphaned toolchains, then >14 days old) from this checkout
+gc:
+    @command -v cargo-sweep >/dev/null 2>&1 || { echo 'cargo-sweep not found; install with: cargo install --locked cargo-sweep'; exit 1; }
+    cargo sweep --installed
+    cargo sweep --time 14
+    @echo '🧹 stale artifacts swept'
+
+# Sweep every worktree of this repo (run after merges or when disk runs hot)
+gc-worktrees:
+    @command -v cargo-sweep >/dev/null 2>&1 || { echo 'cargo-sweep not found; install with: cargo install --locked cargo-sweep'; exit 1; }
+    git worktree prune
+    git worktree list --porcelain | sed -n 's/^worktree //p' | while read -r wt; do [ -d "$wt/target" ] || continue; echo "── sweeping $wt"; cargo sweep --installed "$wt" || true; cargo sweep --time 14 "$wt" || true; done
+    @echo '🧹 all worktree lanes swept'
+
+# Deep clean: sweep, then drop incremental state and the CPU-smoke lane
+gc-deep: gc
+    rm -rf "${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target}"/*/incremental "${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target}/cpu-smoke"
+    @echo '🧹 incremental state and cpu-smoke lane dropped'
+
 # Show workspace dependency tree
 deps:
     cargo tree --workspace

scripts/cargo-cache-build.ps1

@@ -89,33 +89,102 @@ function Initialize-HypercolorCargoCache {
         if ($arg -eq '--release') {
             $usesReleaseLikeProfile = $true
         } elseif ($arg -eq '--profile' -and ($i + 1) -lt $CommandArgs.Count) {
-            $usesReleaseLikeProfile = $CommandArgs[$i + 1] -in @('release', 'bench')
+            if ($CommandArgs[$i + 1] -in @('release', 'bench')) {
+                $usesReleaseLikeProfile = $true
+            }
         } elseif ($arg -match '^--profile=(release|bench)$') {
             $usesReleaseLikeProfile = $true
         }
     }
 
-    $sccache = Get-Command sccache.exe -ErrorAction SilentlyContinue
+    # sccache is the cross-worktree sharing layer: every worktree keeps its own
+    # target dir (so parallel agent builds never contend on Cargo's target
+    # lock), while identical compiles hit one bounded cache under the shared
+    # cache root. sccache and incremental compilation are mutually exclusive
+    # (sccache 0.17 hard-errors on either the env var or -Cincremental), so
+    # the wrapper picks per command: codegen-heavy tree ops go through
+    # sccache; metadata-only ops (check/clippy) keep incremental because
+    # sccache cannot cache --emit=metadata units at all.
+    $cargoSubcommand = ''
+    if ($CommandArgs.Count -gt 1 -and $CommandArgs[0] -match 'cargo(\.exe)?$') {
+        for ($i = 1; $i -lt $CommandArgs.Count; $i += 1) {
+            if ($CommandArgs[$i] -notmatch '^[-+]') {
+                $cargoSubcommand = $CommandArgs[$i]
+                break
+            }
+        }
+    }
+    # `run` stays incremental: it is the edit-run iteration loop, and a
+    # measured edit-rebuild of hypercolor-core is ~45s non-incremental vs
+    # ~11s incremental. Whole-tree ops win with sccache instead.
+    $sccacheSubcommands = @('build', 'test', 'bench', 'nextest')
     $forceSccache = $env:HYPERCOLOR_FORCE_SCCACHE -in @('1', 'true', 'TRUE')
-    if ($null -ne $sccache -and ($usesReleaseLikeProfile -or $forceSccache)) {
+    $wantsSccache = $forceSccache -or $usesReleaseLikeProfile -or ($cargoSubcommand -in $sccacheSubcommands)
+
+    $sccache = Get-Command sccache.exe -ErrorAction SilentlyContinue
+    $disableSccache = ($env:HYPERCOLOR_NO_SCCACHE -in @('1', 'true', 'TRUE')) -or
+        ($env:HYPERCOLOR_ITERATE -in @('1', 'true', 'TRUE')) -or
+        ($env:CARGO_INCREMENTAL -and $env:CARGO_INCREMENTAL -ne '0')
+    if ($null -ne $sccache -and $wantsSccache -and -not $disableSccache) {
         if (-not $env:SCCACHE_DIR) {
             $env:SCCACHE_DIR = Join-Path $cacheRoot 'sccache'
         }
         New-Item -ItemType Directory -Force -Path $env:SCCACHE_DIR | Out-Null
+        if (-not $env:SCCACHE_CACHE_SIZE) {
+            $env:SCCACHE_CACHE_SIZE = if ($env:HYPERCOLOR_SCCACHE_SIZE) {
+                $env:HYPERCOLOR_SCCACHE_SIZE
+            } else {
+                '75G'
+            }
+        }
         if (-not $env:RUSTC_WRAPPER) {
             $env:RUSTC_WRAPPER = $sccache.Source
         }
-        $env:CARGO_BUILD_INCREMENTAL = 'false'
-        $env:CARGO_PROFILE_RELEASE_INCREMENTAL = 'false'
-        $env:CARGO_PROFILE_BENCH_INCREMENTAL = 'false'
-        Write-Host "[cargo-cache] sccache enabled for Rust compilation"
-        Write-Host "[cargo-cache] SCCACHE_DIR=$env:SCCACHE_DIR"
+        # C/C++ caching: mozangle (ANGLE) builds through the cc crate and
+        # turbojpeg through CMake; both honor these and route cl.exe through
+        # sccache, which is where repeat ANGLE builds go to die.
+        if (-not $env:CMAKE_C_COMPILER_LAUNCHER) {
+            $env:CMAKE_C_COMPILER_LAUNCHER = $sccache.Source
+        }
+        if (-not $env:CMAKE_CXX_COMPILER_LAUNCHER) {
+            $env:CMAKE_CXX_COMPILER_LAUNCHER = $sccache.Source
+        }
+        if (-not $env:CC) {
+            $env:CC = 'sccache cl'
+        }
+        if (-not $env:CXX) {
+            $env:CXX = 'sccache cl'
+        }
+        $env:CARGO_INCREMENTAL = '0'
+        Write-Host "[cargo-cache] sccache mode: Rust + C/C++ cached, incremental off (cap $env:SCCACHE_CACHE_SIZE)"
+        Write-Host "[cargo-cache] SCCACHE_DIR=$env:SCCACHE_DIR (HYPERCOLOR_NO_SCCACHE=1 or HYPERCOLOR_ITERATE=1 to disable)"
     } else {
+        # An ambient sccache RUSTC_WRAPPER combined with incremental
+        # hard-fails every compile; drop it rather than let the build die.
+        if ($env:RUSTC_WRAPPER -and (Split-Path -Leaf $env:RUSTC_WRAPPER) -like 'sccache*') {
+            Remove-Item Env:RUSTC_WRAPPER -ErrorAction SilentlyContinue
+            Write-Host "[cargo-cache] unset ambient sccache RUSTC_WRAPPER for incremental mode"
+        }
         if (-not $env:CARGO_INCREMENTAL) {
             $env:CARGO_INCREMENTAL = '1'
         }
-        Write-Host "[cargo-cache] using Cargo incremental compilation on Windows"
-        Write-Host "[cargo-cache] CARGO_INCREMENTAL=$env:CARGO_INCREMENTAL"
+        if ($null -eq $sccache) {
+            Write-Host "[cargo-cache] sccache not found; run 'cargo install --locked sccache' for shared build caching"
+        }
+        Write-Host "[cargo-cache] incremental mode: CARGO_INCREMENTAL=$env:CARGO_INCREMENTAL"
+    }
+
+    # rust-lld ships with the toolchain and links large statically-linked
+    # binaries (daemon + Servo + mozjs) far faster than link.exe. Dev-loop
+    # only: release-like builds keep link.exe so shipped artifacts (CI
+    # sidecars, the installer, app bundles built outside this wrapper) all
+    # come off the same linker.
+    $disableFastLink = $env:HYPERCOLOR_NO_FAST_LINK -in @('1', 'true', 'TRUE')
+    if (-not $env:CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER -and
+        -not $disableFastLink -and
+        -not $usesReleaseLikeProfile) {
+        $env:CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER = 'rust-lld'
+        Write-Host "[cargo-cache] linker: rust-lld (HYPERCOLOR_NO_FAST_LINK=1 to disable)"
     }
 
     Write-Host "[cargo-cache] CARGO_TARGET_DIR=$env:CARGO_TARGET_DIR"