From 8a000606f72a11be6b335cc8d51e18a20170dfbf Mon Sep 17 00:00:00 2001 From: Stefanie Jane Date: Wed, 29 Jul 2026 00:09:28 -0700 Subject: [PATCH 1/2] build(cache): share sccache across worktrees with mode-aware wrappers The cargo-cache wrappers only enabled sccache for release builds, and it was not even installed locally, so every fingerprint bust recompiled Servo at full price in every worktree and profile tree (measured: 343 GB target in the root checkout, 268 GB in one worktree, 41k stale files in debug/deps). sccache and incremental compilation are mutually exclusive (sccache 0.17 hard-errors on the env var and on arg-level -Cincremental), so the wrappers now pick a mode per command: - build/test/bench and release-like profiles: RUSTC_WRAPPER=sccache with CARGO_INCREMENTAL=0 and a bounded cache (75G default) shared by every worktree under the hypercolor cache root. On Windows this also routes cl.exe through sccache (cc + CMake launchers), which caches mozangle's ANGLE builds. - run/check/clippy: Cargo incremental. rustc metadata units are not cacheable by sccache, and an edit-rebuild of hypercolor-core measures ~45s non-incremental vs ~11s incremental, so the edit loops keep incremental. Iteration-shaped recipes (test-crate, test-one, app, the Windows dev daemon build) pin this via HYPERCOLOR_ITERATE=1. rust-lld becomes the local linker on x86_64-pc-windows-msvc; the shared .cargo/config.toml stays portable for CI and release. Disk growth gets bounds: the cpu-smoke feature shape builds into its own target/cpu-smoke lane instead of churning the daily tree, and new just recipes (disk, gc, gc-worktrees, gc-deep) report and sweep stale artifacts with cargo-sweep. Verified on Windows: mode selection per subcommand, sccache fill and cache hits across a cargo clean, CI env compat (HYPERCOLOR_FORCE_SCCACHE and explicit CARGO_INCREMENTAL are respected). Co-Authored-By: Nova (Claude Fable 5) --- docs/development/SERVO_BUILD_CACHING.md | 63 ++++++++++++- justfile | 72 +++++++++++--- scripts/cargo-cache-build.ps1 | 75 +++++++++++++-- scripts/cargo-cache-build.sh | 119 ++++++++++++++---------- scripts/dev-windows.ps1 | 3 + 5 files changed, 261 insertions(+), 71 deletions(-) diff --git a/docs/development/SERVO_BUILD_CACHING.md b/docs/development/SERVO_BUILD_CACHING.md index 8013d02d6..d3f48c729 100644 --- a/docs/development/SERVO_BUILD_CACHING.md +++ b/docs/development/SERVO_BUILD_CACHING.md @@ -58,11 +58,66 @@ The shared wrapper configures: - `CARGO_TARGET_DIR=/target` (unless already set) - `MOZBUILD_STATE_PATH=$HOME/.cache/hypercolor/mozbuild` (unless already set) -- Cargo incremental compilation for local dev and preview-style builds -- `sccache` as `RUSTC_WRAPPER` for release/bench builds, or whenever - `HYPERCOLOR_FORCE_SCCACHE=1` +- `sccache` as `RUSTC_WRAPPER` for whole-tree codegen commands + (`cargo build`, `test`, `bench`, and anything release/bench-profiled) + when installed, with a bounded on-disk cache (default `75G`, override + with `HYPERCOLOR_SCCACHE_SIZE`). sccache and incremental compilation are + mutually exclusive, so these commands run with `CARGO_INCREMENTAL=0`. +- Cargo incremental compilation for iteration and metadata commands: + `cargo run` (the edit-run loop; a measured hypercolor-core edit-rebuild + is ~45s non-incremental vs ~11s incremental), `cargo check`, and + `clippy` (sccache cannot cache `--emit=metadata` units). The + iteration-shaped recipes (`just test-crate`, `test-one`, `app`, and the + Windows `just dev` daemon build) pin incremental via + `HYPERCOLOR_ITERATE=1`. +- Opt-outs: `HYPERCOLOR_NO_SCCACHE=1` disables sccache for the session; + `HYPERCOLOR_ITERATE=1` does the same per invocation when you want + incremental rebuilds in a tight edit loop; a pre-set non-zero + `CARGO_INCREMENTAL` always wins. Alternating the same profile tree + between the two modes rebuilds only workspace crates (~50s measured), + never dependencies. +- `rust-lld` as the linker on `x86_64-pc-windows-msvc` + (`HYPERCOLOR_NO_FAST_LINK=1` to opt out) - `clang` + `ld.lld` for faster link steps on `x86_64-unknown-linux-gnu` when available -- `ccache` for `CC`/`CXX` when installed, otherwise `sccache` if available +- C/C++ caching for `cc`- and CMake-driven native deps (mozangle/ANGLE, + turbojpeg): `ccache` or `sccache` on Unix, `sccache` around `cl.exe` on + Windows + +## Cross-Worktree Topology + +Multiple worktrees (and multiple agents) build this repo concurrently. The +sharing layer is the compile cache, not the target dir: + +- **Per-worktree `target/`** stays the default. Cargo's target lock is + coarse; a shared target dir would serialize parallel builds across + worktrees and thrash on feature-shape differences. +- **Shared, bounded caches** live under `$HOME/.cache/hypercolor` + (`HYPERCOLOR_CACHE_DIR` to relocate): `sccache/` for compiled units, + `mozbuild/` for SpiderMonkey build state. A second worktree's cold build + becomes mostly cache hits without any cross-worktree locking. +- **Incompatible feature shapes get isolated lanes**: `just e2e-build-cpu` + builds into `target/cpu-smoke` so the `--no-default-features` unification + never churns the daily tree. +- **`mozjs_sys` uses prebuilt SpiderMonkey archives by default.** It falls + back to a source build silently, e.g. when a package profile override + drops `mozjs_sys` below `-O3`; keep the `opt-level = 3` overrides in + `Cargo.toml` intact. + +## Disk Bounds + +Target dirs grow without bound as toolchains, lockfiles, and feature shapes +churn — Cargo never garbage-collects them. Bound them with: + +```bash +just disk # per-profile + shared-cache usage report +just gc # sweep orphaned-toolchain and >14-day artifacts here +just gc-worktrees # the same sweep across every worktree lane +just gc-deep # additionally drop incremental state + the cpu-smoke lane +``` + +`sccache` trims itself to `SCCACHE_CACHE_SIZE`; the cache size only applies +when the server starts, so after changing it run `sccache --stop-server` +first. ## Verify Cache Hits diff --git a/justfile b/justfile index 5836d26a4..69bb7c448 100644 --- a/justfile +++ b/justfile @@ -142,23 +142,23 @@ test *args='': test *args='': powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -File scripts/cargo-cache-build.ps1 cargo test {{ workspace_args }} {{ args }} -# Run tests for a specific crate +# Run tests for a specific crate (iteration-shaped: keeps incremental rebuilds) [unix] test-crate crate *args='': - ./scripts/cargo-cache-build.sh cargo test -p {{ crate }} {{ args }} + HYPERCOLOR_ITERATE=1 ./scripts/cargo-cache-build.sh cargo test -p {{ crate }} {{ args }} [windows] test-crate crate *args='': - powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -File scripts/cargo-cache-build.ps1 cargo test -p {{ crate }} {{ args }} + HYPERCOLOR_ITERATE=1 powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -File scripts/cargo-cache-build.ps1 cargo test -p {{ crate }} {{ args }} -# Run a specific test by name +# Run a specific test by name (iteration-shaped: keeps incremental rebuilds) [unix] test-one name *args='': - ./scripts/cargo-cache-build.sh cargo test {{ workspace_args }} {{ name }} {{ args }} + HYPERCOLOR_ITERATE=1 ./scripts/cargo-cache-build.sh cargo test {{ workspace_args }} {{ name }} {{ args }} [windows] test-one name *args='': - powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -File scripts/cargo-cache-build.ps1 cargo test {{ workspace_args }} {{ name }} {{ args }} + HYPERCOLOR_ITERATE=1 powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -File scripts/cargo-cache-build.ps1 cargo test {{ workspace_args }} {{ name }} {{ args }} # Manually run the Cinder/Leptos extension design audit snapshot generator cinder-audit: @@ -364,10 +364,10 @@ app-assets: just ui-build just effects-build -# Run the unified desktop app +# Run the unified desktop app (iteration-shaped: keeps incremental rebuilds) [unix] app *args='': app-assets - ./scripts/cargo-cache-build.sh cargo build -p hypercolor-daemon --bin hypercolor-daemon -p hypercolor-app --bin hypercolor-app --profile preview + HYPERCOLOR_ITERATE=1 ./scripts/cargo-cache-build.sh cargo build -p hypercolor-daemon --bin hypercolor-daemon -p hypercolor-app --bin hypercolor-app --profile preview "${CARGO_TARGET_DIR:-target}/preview/hypercolor-app" {{ args }} [windows] @@ -674,10 +674,14 @@ e2e-build: just effects-build just ui-build -# Build the fallback CPU smoke stack without the Servo renderer +# Build the fallback CPU smoke stack without the Servo renderer. +# Isolated target lane: the --no-default-features shape unifies crate features +# differently from the daily builds, and letting it share target/ churns and +# strands artifacts for the whole dependency graph on every alternation. +# CI pins CARGO_TARGET_DIR per lane, so an ambient value wins. e2e-build-cpu: - ./scripts/cargo-cache-build.sh cargo build -p hypercolor-daemon --no-default-features --features builtin-drivers - ./scripts/cargo-cache-build.sh cargo build -p hypercolor-cli + CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target/cpu-smoke}" ./scripts/cargo-cache-build.sh cargo build -p hypercolor-daemon --no-default-features --features builtin-drivers + CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target/cpu-smoke}" ./scripts/cargo-cache-build.sh cargo build -p hypercolor-cli just effects-build just ui-build @@ -814,6 +818,52 @@ udev-install: clean: ./scripts/cargo-cache-build.sh cargo clean +# Report build artifact and shared cache disk usage for this checkout +disk: + #!/usr/bin/env bash + set -euo pipefail + cache_root="${HYPERCOLOR_CACHE_DIR:-$HOME/.cache/hypercolor}" + target="${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target}" + echo "── target profiles ──" + [ -d "$target" ] && du -sh "$target"/* 2>/dev/null | sort -rh | head -15 + echo "── shared caches ($cache_root) ──" + [ -d "$cache_root" ] && du -sh "$cache_root"/* 2>/dev/null | sort -rh + if command -v sccache >/dev/null 2>&1; then + echo "── sccache ──" + sccache --show-stats | grep -E 'Cache hits|Cache misses|Cache size|Max cache' || true + fi + +# Sweep stale build artifacts (orphaned toolchains, then >14 days old) from this checkout +gc: + #!/usr/bin/env bash + set -euo pipefail + command -v cargo-sweep >/dev/null 2>&1 || { echo 'cargo-sweep not found; install with: cargo install --locked cargo-sweep'; exit 1; } + cargo sweep --installed + cargo sweep --time 14 + echo '🧹 stale artifacts swept' + +# Sweep every worktree of this repo (run after merges or when disk runs hot) +gc-worktrees: + #!/usr/bin/env bash + set -euo pipefail + command -v cargo-sweep >/dev/null 2>&1 || { echo 'cargo-sweep not found; install with: cargo install --locked cargo-sweep'; exit 1; } + git worktree prune + git worktree list --porcelain | sed -n 's/^worktree //p' | while read -r wt; do + [ -d "$wt/target" ] || continue + echo "── sweeping $wt" + cargo sweep --installed "$wt" || true + cargo sweep --time 14 "$wt" || true + done + echo '🧹 all worktree lanes swept' + +# Deep clean: sweep, then drop incremental state and the CPU-smoke lane +gc-deep: gc + #!/usr/bin/env bash + set -euo pipefail + target="${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target}" + rm -rf "$target"/*/incremental "$target/cpu-smoke" + echo '🧹 incremental state and cpu-smoke lane dropped' + # Show workspace dependency tree deps: cargo tree --workspace diff --git a/scripts/cargo-cache-build.ps1 b/scripts/cargo-cache-build.ps1 index 6113813e1..b9fee9405 100644 --- a/scripts/cargo-cache-build.ps1 +++ b/scripts/cargo-cache-build.ps1 @@ -95,27 +95,84 @@ function Initialize-HypercolorCargoCache { } } - $sccache = Get-Command sccache.exe -ErrorAction SilentlyContinue + # sccache is the cross-worktree sharing layer: every worktree keeps its own + # target dir (so parallel agent builds never contend on Cargo's target + # lock), while identical compiles hit one bounded cache under the shared + # cache root. sccache and incremental compilation are mutually exclusive + # (sccache 0.17 hard-errors on either the env var or -Cincremental), so + # the wrapper picks per command: codegen-heavy tree ops go through + # sccache; metadata-only ops (check/clippy) keep incremental because + # sccache cannot cache --emit=metadata units at all. + $cargoSubcommand = '' + if ($CommandArgs.Count -gt 1 -and $CommandArgs[0] -match 'cargo(\.exe)?$') { + for ($i = 1; $i -lt $CommandArgs.Count; $i += 1) { + if ($CommandArgs[$i] -notmatch '^[-+]') { + $cargoSubcommand = $CommandArgs[$i] + break + } + } + } + # `run` stays incremental: it is the edit-run iteration loop, and a + # measured edit-rebuild of hypercolor-core is ~45s non-incremental vs + # ~11s incremental. Whole-tree ops win with sccache instead. + $sccacheSubcommands = @('build', 'test', 'bench') $forceSccache = $env:HYPERCOLOR_FORCE_SCCACHE -in @('1', 'true', 'TRUE') - if ($null -ne $sccache -and ($usesReleaseLikeProfile -or $forceSccache)) { + $wantsSccache = $forceSccache -or $usesReleaseLikeProfile -or ($cargoSubcommand -in $sccacheSubcommands) + + $sccache = Get-Command sccache.exe -ErrorAction SilentlyContinue + $disableSccache = ($env:HYPERCOLOR_NO_SCCACHE -in @('1', 'true', 'TRUE')) -or + ($env:HYPERCOLOR_ITERATE -in @('1', 'true', 'TRUE')) -or + ($env:CARGO_INCREMENTAL -and $env:CARGO_INCREMENTAL -ne '0') + if ($null -ne $sccache -and $wantsSccache -and -not $disableSccache) { if (-not $env:SCCACHE_DIR) { $env:SCCACHE_DIR = Join-Path $cacheRoot 'sccache' } New-Item -ItemType Directory -Force -Path $env:SCCACHE_DIR | Out-Null + if (-not $env:SCCACHE_CACHE_SIZE) { + $env:SCCACHE_CACHE_SIZE = if ($env:HYPERCOLOR_SCCACHE_SIZE) { + $env:HYPERCOLOR_SCCACHE_SIZE + } else { + '75G' + } + } if (-not $env:RUSTC_WRAPPER) { $env:RUSTC_WRAPPER = $sccache.Source } - $env:CARGO_BUILD_INCREMENTAL = 'false' - $env:CARGO_PROFILE_RELEASE_INCREMENTAL = 'false' - $env:CARGO_PROFILE_BENCH_INCREMENTAL = 'false' - Write-Host "[cargo-cache] sccache enabled for Rust compilation" - Write-Host "[cargo-cache] SCCACHE_DIR=$env:SCCACHE_DIR" + # C/C++ caching: mozangle (ANGLE) builds through the cc crate and + # turbojpeg through CMake; both honor these and route cl.exe through + # sccache, which is where repeat ANGLE builds go to die. + if (-not $env:CMAKE_C_COMPILER_LAUNCHER) { + $env:CMAKE_C_COMPILER_LAUNCHER = $sccache.Source + } + if (-not $env:CMAKE_CXX_COMPILER_LAUNCHER) { + $env:CMAKE_CXX_COMPILER_LAUNCHER = $sccache.Source + } + if (-not $env:CC) { + $env:CC = 'sccache cl' + } + if (-not $env:CXX) { + $env:CXX = 'sccache cl' + } + $env:CARGO_INCREMENTAL = '0' + Write-Host "[cargo-cache] sccache mode: Rust + C/C++ cached, incremental off (cap $env:SCCACHE_CACHE_SIZE)" + Write-Host "[cargo-cache] SCCACHE_DIR=$env:SCCACHE_DIR (HYPERCOLOR_NO_SCCACHE=1 or HYPERCOLOR_ITERATE=1 to disable)" } else { if (-not $env:CARGO_INCREMENTAL) { $env:CARGO_INCREMENTAL = '1' } - Write-Host "[cargo-cache] using Cargo incremental compilation on Windows" - Write-Host "[cargo-cache] CARGO_INCREMENTAL=$env:CARGO_INCREMENTAL" + if ($null -eq $sccache) { + Write-Host "[cargo-cache] sccache not found; run 'cargo install --locked sccache' for shared build caching" + } + Write-Host "[cargo-cache] incremental mode: CARGO_INCREMENTAL=$env:CARGO_INCREMENTAL" + } + + # rust-lld ships with the toolchain and links large statically-linked + # binaries (daemon + Servo + mozjs) far faster than link.exe. Local-only: + # the shared .cargo/config.toml stays portable for CI and release. + $disableFastLink = $env:HYPERCOLOR_NO_FAST_LINK -in @('1', 'true', 'TRUE') + if (-not $env:CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER -and -not $disableFastLink) { + $env:CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER = 'rust-lld' + Write-Host "[cargo-cache] linker: rust-lld (HYPERCOLOR_NO_FAST_LINK=1 to disable)" } Write-Host "[cargo-cache] CARGO_TARGET_DIR=$env:CARGO_TARGET_DIR" diff --git a/scripts/cargo-cache-build.sh b/scripts/cargo-cache-build.sh index 97d323e00..50cdabc92 100755 --- a/scripts/cargo-cache-build.sh +++ b/scripts/cargo-cache-build.sh @@ -50,63 +50,88 @@ fi SCCACHE_BIN="$(command -v sccache || true)" CCACHE_BIN="$(command -v ccache || true)" -ENABLE_RUST_SCCACHE=1 -FORCE_RUST_SCCACHE="${HYPERCOLOR_FORCE_SCCACHE:-0}" - -if [ "$FORCE_RUST_SCCACHE" != "1" ] && [ "$FORCE_RUST_SCCACHE" != "true" ]; then - for ((i = 1; i <= $#; i++)); do - arg="${!i}" - case "$arg" in - --release) - ENABLE_RUST_SCCACHE=1 - ;; - --profile) - next_index=$((i + 1)) - if [ "$next_index" -le "$#" ]; then - next_arg="${!next_index}" - if [ "$next_arg" != "release" ] && [ "$next_arg" != "bench" ]; then - ENABLE_RUST_SCCACHE=0 - fi +DISABLE_SCCACHE="${HYPERCOLOR_NO_SCCACHE:-0}" + +RELEASE_LIKE_PROFILE=0 +for ((i = 1; i <= $#; i++)); do + arg="${!i}" + case "$arg" in + --release) + RELEASE_LIKE_PROFILE=1 + ;; + --profile) + next_index=$((i + 1)) + if [ "$next_index" -le "$#" ]; then + next_arg="${!next_index}" + if [ "$next_arg" = "release" ] || [ "$next_arg" = "bench" ]; then + RELEASE_LIKE_PROFILE=1 fi - ;; - --profile=*) - profile_name="${arg#--profile=}" - if [ "$profile_name" != "release" ] && [ "$profile_name" != "bench" ]; then - ENABLE_RUST_SCCACHE=0 - fi - ;; - esac - done + fi + ;; + --profile=release | --profile=bench) + RELEASE_LIKE_PROFILE=1 + ;; + esac +done - if [ "$#" -gt 0 ] && ! printf '%s\n' "$*" | grep -Eq -- '(^| )--release($| )|(^| )--profile(=| )(release|bench)($| )'; then - ENABLE_RUST_SCCACHE=0 - fi +# sccache is the cross-worktree sharing layer: every worktree keeps its own +# target dir (so parallel agent builds never contend on Cargo's target lock), +# while identical compiles hit one bounded cache under the shared cache root. +# sccache and incremental compilation are mutually exclusive (sccache 0.17 +# hard-errors on either the env var or -Cincremental), so the wrapper picks +# per command: codegen-heavy tree ops go through sccache; metadata-only ops +# (check/clippy) keep incremental because sccache cannot cache +# --emit=metadata units at all. +CARGO_SUBCOMMAND="" +if [ "$#" -gt 1 ]; then + case "$(basename "$1")" in + cargo | cargo.exe) + for ((i = 2; i <= $#; i++)); do + arg="${!i}" + case "$arg" in + -* | +*) continue ;; + *) + CARGO_SUBCOMMAND="$arg" + break + ;; + esac + done + ;; + esac +fi + +FORCE_SCCACHE="${HYPERCOLOR_FORCE_SCCACHE:-0}" +ITERATE="${HYPERCOLOR_ITERATE:-0}" +# `run` stays incremental: it is the edit-run iteration loop, and a measured +# edit-rebuild of hypercolor-core is ~45s non-incremental vs ~11s +# incremental. Whole-tree ops win with sccache instead. +WANTS_SCCACHE=0 +case "$CARGO_SUBCOMMAND" in + build | test | bench) WANTS_SCCACHE=1 ;; +esac +if [ "$RELEASE_LIKE_PROFILE" -eq 1 ] || [ "$FORCE_SCCACHE" = "1" ] || [ "$FORCE_SCCACHE" = "true" ]; then + WANTS_SCCACHE=1 +fi +if [ "$DISABLE_SCCACHE" = "1" ] || [ "$DISABLE_SCCACHE" = "true" ] \ + || [ "$ITERATE" = "1" ] || [ "$ITERATE" = "true" ] \ + || { [ -n "${CARGO_INCREMENTAL:-}" ] && [ "$CARGO_INCREMENTAL" != "0" ]; }; then + WANTS_SCCACHE=0 fi -if [ -n "$SCCACHE_BIN" ] && [ "$ENABLE_RUST_SCCACHE" -eq 1 ]; then +if [ -n "$SCCACHE_BIN" ] && [ "$WANTS_SCCACHE" -eq 1 ]; then export SCCACHE_DIR="${SCCACHE_DIR:-$CACHE_ROOT/sccache}" mkdir -p "$SCCACHE_DIR" + export SCCACHE_CACHE_SIZE="${SCCACHE_CACHE_SIZE:-${HYPERCOLOR_SCCACHE_SIZE:-75G}}" export RUSTC_WRAPPER="${RUSTC_WRAPPER:-$SCCACHE_BIN}" - # sccache rejects incremental compilation entirely, so prefer the - # compiler cache and force a compatible Cargo setting. - unset CARGO_INCREMENTAL || true - export CARGO_BUILD_INCREMENTAL="false" - export CARGO_PROFILE_DEV_INCREMENTAL="false" - export CARGO_PROFILE_RELEASE_INCREMENTAL="false" - export CARGO_PROFILE_TEST_INCREMENTAL="false" - export CARGO_PROFILE_BENCH_INCREMENTAL="false" - export CARGO_PROFILE_PREVIEW_INCREMENTAL="false" - echo "[cargo-cache] sccache enabled for Rust compilation" - echo "[cargo-cache] SCCACHE_DIR=$SCCACHE_DIR" - echo "[cargo-cache] incremental compilation disabled for sccache compatibility" + export CARGO_INCREMENTAL="0" + echo "[cargo-cache] sccache mode: Rust cached, incremental off (cap $SCCACHE_CACHE_SIZE)" + echo "[cargo-cache] SCCACHE_DIR=$SCCACHE_DIR (HYPERCOLOR_NO_SCCACHE=1 or HYPERCOLOR_ITERATE=1 to disable)" else export CARGO_INCREMENTAL="${CARGO_INCREMENTAL:-1}" - if [ -n "$SCCACHE_BIN" ]; then - echo "[cargo-cache] skipping rust sccache for dev/preview-style build; using Cargo incremental instead" - else - echo "[cargo-cache] sccache not found; Rust compilation will use Cargo incremental only" + if [ -z "$SCCACHE_BIN" ]; then + echo "[cargo-cache] sccache not found; run 'cargo install --locked sccache' for shared build caching" fi - echo "[cargo-cache] CARGO_INCREMENTAL=$CARGO_INCREMENTAL" + echo "[cargo-cache] incremental mode: CARGO_INCREMENTAL=$CARGO_INCREMENTAL" fi if [ "$HOST_TRIPLE" = "x86_64-unknown-linux-gnu" ] \ diff --git a/scripts/dev-windows.ps1 b/scripts/dev-windows.ps1 index 801b825a3..7e6e49b8e 100644 --- a/scripts/dev-windows.ps1 +++ b/scripts/dev-windows.ps1 @@ -339,6 +339,9 @@ $ui = $null try { Write-Host '[dev] building daemon' Write-Host "[dev] CARGO_TARGET_DIR=$env:CARGO_TARGET_DIR" + # This is the edit-run loop: keep incremental rebuilds instead of the + # wrapper's sccache mode for `cargo build`. + $env:HYPERCOLOR_ITERATE = '1' & powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -File $cargoCacheBuild cargo @cargoBuildArguments if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE From 7459147be0e84005a096bfd5b7863a296fe910c0 Mon Sep 17 00:00:00 2001 From: Stefanie Jane Date: Wed, 29 Jul 2026 02:02:53 -0700 Subject: [PATCH 2/2] build(cache): address adversarial verification findings The verification pass on the sccache topology change returned PASS with two majors and five minors; this lands the fixes. rust-lld is now gated off release-like builds so shipped Windows artifacts (CI sidecars, the installer, app bundles built outside the wrapper) all keep link.exe; the fast linker remains for dev iteration. The caching doc's CI section claimed a mozilla-actions sccache setup that has never existed in .github; it now documents reality: CI runs rust-cache with CARGO_INCREMENTAL=0 and no sccache, and the wrappers preserve that environment unchanged. Robustness fixes from the same report: nextest joins the sccache subcommand list (CI's test runner, and local nextest runs deserve the cache); the sh wrapper's bare-invocation default is injected before mode detection so it routes like an explicit workspace build; an ambient sccache RUSTC_WRAPPER is dropped in incremental mode instead of hard-failing every compile; the ps1 --profile scan can no longer unset an earlier --release match; and the housekeeping recipes use plain sh lines instead of shebang blocks, which resolve bash from PATH and can land on WSL bash on Windows. just disk survives empty target dirs, and just disk/gc are verified live (gc: cargo sweep --installed plus --time 14, clean no-op on a fresh tree). Co-Authored-By: Nova (Claude Fable 5) --- docs/development/SERVO_BUILD_CACHING.md | 38 ++++++++++---------- justfile | 48 +++++++++---------------- scripts/cargo-cache-build.ps1 | 22 +++++++++--- scripts/cargo-cache-build.sh | 21 ++++++++--- 4 files changed, 68 insertions(+), 61 deletions(-) diff --git a/docs/development/SERVO_BUILD_CACHING.md b/docs/development/SERVO_BUILD_CACHING.md index d3f48c729..9b0ed0168 100644 --- a/docs/development/SERVO_BUILD_CACHING.md +++ b/docs/development/SERVO_BUILD_CACHING.md @@ -67,21 +67,23 @@ The shared wrapper configures: `cargo run` (the edit-run loop; a measured hypercolor-core edit-rebuild is ~45s non-incremental vs ~11s incremental), `cargo check`, and `clippy` (sccache cannot cache `--emit=metadata` units). The - iteration-shaped recipes (`just test-crate`, `test-one`, `app`, and the - Windows `just dev` daemon build) pin incremental via - `HYPERCOLOR_ITERATE=1`. + iteration-shaped recipes (`just test-crate`, `test-one`, the Unix `app` + build, and the Windows `just dev` daemon build) pin incremental via + `HYPERCOLOR_ITERATE=1`; the Windows `app` recipe uses `cargo run` and + lands there by subcommand. - Opt-outs: `HYPERCOLOR_NO_SCCACHE=1` disables sccache for the session; `HYPERCOLOR_ITERATE=1` does the same per invocation when you want incremental rebuilds in a tight edit loop; a pre-set non-zero `CARGO_INCREMENTAL` always wins. Alternating the same profile tree between the two modes rebuilds only workspace crates (~50s measured), never dependencies. -- `rust-lld` as the linker on `x86_64-pc-windows-msvc` - (`HYPERCOLOR_NO_FAST_LINK=1` to opt out) +- `rust-lld` as the linker on `x86_64-pc-windows-msvc` for non-release + builds (`HYPERCOLOR_NO_FAST_LINK=1` to opt out); release-like builds + keep `link.exe` so shipped artifacts all come off the same linker - `clang` + `ld.lld` for faster link steps on `x86_64-unknown-linux-gnu` when available - C/C++ caching for `cc`- and CMake-driven native deps (mozangle/ANGLE, - turbojpeg): `ccache` or `sccache` on Unix, `sccache` around `cl.exe` on - Windows + turbojpeg): `ccache` or `sccache` on Unix (both modes), `sccache` around + `cl.exe` on Windows (sccache mode only) ## Cross-Worktree Topology @@ -133,26 +135,22 @@ Look for increasing cache hit counts after the first Servo build. The reusable action `.github/actions/rust-build-cache` configures GitHub Actions builds with: -- `mozilla-actions/sccache-action` using GitHub's sccache backend -- `HYPERCOLOR_FORCE_SCCACHE=1` -- `CARGO_INCREMENTAL=0` - `Swatinem/rust-cache` for Cargo and extra cache directories +- `CARGO_INCREMENTAL=0` (set workflow-wide) - `.cache/hypercolor/target` for CI-selected Cargo target shards - `.cache/hypercolor/mozbuild` - `.cache/hypercolor/toolchain` - `.cache/hypercolor/ccache` -The manual `.github/workflows/servo-cache-warm.yml` workflow warms three -compatible shapes when a maintainer deliberately refreshes Servo caches: +CI does not run sccache today: hosted runners do not preinstall it and the +workflows do not set it up. The wrappers honor `HYPERCOLOR_FORCE_SCCACHE=1` +and a pre-set `CARGO_INCREMENTAL=0`, so a future CI sccache lane only needs +to install the binary and set the flag. -| Suite | Shared Key | Extra Key | Purpose | -| ------------ | -------------- | ------------ | ------------------------------------------------ | -| Core Servo | `servo-core` | empty | core Servo check, test, and clippy artifacts | -| Daemon Servo | `servo-daemon` | empty | daemon Servo check, test, and clippy artifacts | -| E2E Servo | `servo-daemon` | `e2e-dev-v1` | daemon and CLI binaries for the normal E2E stack | - -The main CI workflow reuses those same shared keys in the explicit Servo check, -test, and E2E build lanes. Pull requests keep the separate Servo check/test +The manual `.github/workflows/servo-cache-warm.yml` workflow warms the +`servo` shared cache key when a maintainer deliberately refreshes Servo +caches; the main CI workflow reuses that key in its Servo check, test, and +E2E build lanes. Pull requests keep the separate Servo check/test lanes out of the default path and rely on the normal Servo E2E stack for HTML renderer coverage. Pushes to `main`, tags, and manual CI dispatches still run the full Servo check/test gates. diff --git a/justfile b/justfile index 69bb7c448..b4ff98cc7 100644 --- a/justfile +++ b/justfile @@ -818,51 +818,37 @@ udev-install: clean: ./scripts/cargo-cache-build.sh cargo clean +# Plain sh lines, no shebang blocks: shebang recipes resolve `bash` from +# PATH, which on Windows can be WSL bash that cannot read the temp script +# path. `sh -cu` lines are the pattern every [windows] recipe already +# proves. + # Report build artifact and shared cache disk usage for this checkout disk: - #!/usr/bin/env bash - set -euo pipefail - cache_root="${HYPERCOLOR_CACHE_DIR:-$HOME/.cache/hypercolor}" - target="${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target}" - echo "── target profiles ──" - [ -d "$target" ] && du -sh "$target"/* 2>/dev/null | sort -rh | head -15 - echo "── shared caches ($cache_root) ──" - [ -d "$cache_root" ] && du -sh "$cache_root"/* 2>/dev/null | sort -rh - if command -v sccache >/dev/null 2>&1; then - echo "── sccache ──" - sccache --show-stats | grep -E 'Cache hits|Cache misses|Cache size|Max cache' || true - fi + @echo '── target profiles ──' + @if [ -d "${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target}" ]; then du -sh "${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target}"/* 2>/dev/null | sort -rh | head -15 || true; else echo '(no target dir)'; fi + @echo "── shared caches (${HYPERCOLOR_CACHE_DIR:-$HOME/.cache/hypercolor}) ──" + @if [ -d "${HYPERCOLOR_CACHE_DIR:-$HOME/.cache/hypercolor}" ]; then du -sh "${HYPERCOLOR_CACHE_DIR:-$HOME/.cache/hypercolor}"/* 2>/dev/null | sort -rh || true; else echo '(no cache dir)'; fi + @if command -v sccache >/dev/null 2>&1; then echo '── sccache ──'; sccache --show-stats | grep -E 'Cache hits|Cache misses|Cache size|Max cache' || true; fi # Sweep stale build artifacts (orphaned toolchains, then >14 days old) from this checkout gc: - #!/usr/bin/env bash - set -euo pipefail - command -v cargo-sweep >/dev/null 2>&1 || { echo 'cargo-sweep not found; install with: cargo install --locked cargo-sweep'; exit 1; } + @command -v cargo-sweep >/dev/null 2>&1 || { echo 'cargo-sweep not found; install with: cargo install --locked cargo-sweep'; exit 1; } cargo sweep --installed cargo sweep --time 14 - echo '🧹 stale artifacts swept' + @echo '🧹 stale artifacts swept' # Sweep every worktree of this repo (run after merges or when disk runs hot) gc-worktrees: - #!/usr/bin/env bash - set -euo pipefail - command -v cargo-sweep >/dev/null 2>&1 || { echo 'cargo-sweep not found; install with: cargo install --locked cargo-sweep'; exit 1; } + @command -v cargo-sweep >/dev/null 2>&1 || { echo 'cargo-sweep not found; install with: cargo install --locked cargo-sweep'; exit 1; } git worktree prune - git worktree list --porcelain | sed -n 's/^worktree //p' | while read -r wt; do - [ -d "$wt/target" ] || continue - echo "── sweeping $wt" - cargo sweep --installed "$wt" || true - cargo sweep --time 14 "$wt" || true - done - echo '🧹 all worktree lanes swept' + git worktree list --porcelain | sed -n 's/^worktree //p' | while read -r wt; do [ -d "$wt/target" ] || continue; echo "── sweeping $wt"; cargo sweep --installed "$wt" || true; cargo sweep --time 14 "$wt" || true; done + @echo '🧹 all worktree lanes swept' # Deep clean: sweep, then drop incremental state and the CPU-smoke lane gc-deep: gc - #!/usr/bin/env bash - set -euo pipefail - target="${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target}" - rm -rf "$target"/*/incremental "$target/cpu-smoke" - echo '🧹 incremental state and cpu-smoke lane dropped' + rm -rf "${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target}"/*/incremental "${CARGO_TARGET_DIR:-{{ justfile_directory() }}/target}/cpu-smoke" + @echo '🧹 incremental state and cpu-smoke lane dropped' # Show workspace dependency tree deps: diff --git a/scripts/cargo-cache-build.ps1 b/scripts/cargo-cache-build.ps1 index b9fee9405..4f243f0df 100644 --- a/scripts/cargo-cache-build.ps1 +++ b/scripts/cargo-cache-build.ps1 @@ -89,7 +89,9 @@ function Initialize-HypercolorCargoCache { if ($arg -eq '--release') { $usesReleaseLikeProfile = $true } elseif ($arg -eq '--profile' -and ($i + 1) -lt $CommandArgs.Count) { - $usesReleaseLikeProfile = $CommandArgs[$i + 1] -in @('release', 'bench') + if ($CommandArgs[$i + 1] -in @('release', 'bench')) { + $usesReleaseLikeProfile = $true + } } elseif ($arg -match '^--profile=(release|bench)$') { $usesReleaseLikeProfile = $true } @@ -115,7 +117,7 @@ function Initialize-HypercolorCargoCache { # `run` stays incremental: it is the edit-run iteration loop, and a # measured edit-rebuild of hypercolor-core is ~45s non-incremental vs # ~11s incremental. Whole-tree ops win with sccache instead. - $sccacheSubcommands = @('build', 'test', 'bench') + $sccacheSubcommands = @('build', 'test', 'bench', 'nextest') $forceSccache = $env:HYPERCOLOR_FORCE_SCCACHE -in @('1', 'true', 'TRUE') $wantsSccache = $forceSccache -or $usesReleaseLikeProfile -or ($cargoSubcommand -in $sccacheSubcommands) @@ -157,6 +159,12 @@ function Initialize-HypercolorCargoCache { Write-Host "[cargo-cache] sccache mode: Rust + C/C++ cached, incremental off (cap $env:SCCACHE_CACHE_SIZE)" Write-Host "[cargo-cache] SCCACHE_DIR=$env:SCCACHE_DIR (HYPERCOLOR_NO_SCCACHE=1 or HYPERCOLOR_ITERATE=1 to disable)" } else { + # An ambient sccache RUSTC_WRAPPER combined with incremental + # hard-fails every compile; drop it rather than let the build die. + if ($env:RUSTC_WRAPPER -and (Split-Path -Leaf $env:RUSTC_WRAPPER) -like 'sccache*') { + Remove-Item Env:RUSTC_WRAPPER -ErrorAction SilentlyContinue + Write-Host "[cargo-cache] unset ambient sccache RUSTC_WRAPPER for incremental mode" + } if (-not $env:CARGO_INCREMENTAL) { $env:CARGO_INCREMENTAL = '1' } @@ -167,10 +175,14 @@ function Initialize-HypercolorCargoCache { } # rust-lld ships with the toolchain and links large statically-linked - # binaries (daemon + Servo + mozjs) far faster than link.exe. Local-only: - # the shared .cargo/config.toml stays portable for CI and release. + # binaries (daemon + Servo + mozjs) far faster than link.exe. Dev-loop + # only: release-like builds keep link.exe so shipped artifacts (CI + # sidecars, the installer, app bundles built outside this wrapper) all + # come off the same linker. $disableFastLink = $env:HYPERCOLOR_NO_FAST_LINK -in @('1', 'true', 'TRUE') - if (-not $env:CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER -and -not $disableFastLink) { + if (-not $env:CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER -and + -not $disableFastLink -and + -not $usesReleaseLikeProfile) { $env:CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER = 'rust-lld' Write-Host "[cargo-cache] linker: rust-lld (HYPERCOLOR_NO_FAST_LINK=1 to disable)" } diff --git a/scripts/cargo-cache-build.sh b/scripts/cargo-cache-build.sh index 50cdabc92..f1af24b3d 100755 --- a/scripts/cargo-cache-build.sh +++ b/scripts/cargo-cache-build.sh @@ -48,6 +48,13 @@ if [ "$HOST_TRIPLE" = "x86_64-pc-windows-msvc" ]; then exec powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -File "$PS_WRAPPER" "$@" fi +# Default the bare invocation before mode detection so it routes like an +# explicit `cargo build --workspace` instead of falling through to +# incremental mode. +if [ "$#" -eq 0 ]; then + set -- cargo build --workspace +fi + SCCACHE_BIN="$(command -v sccache || true)" CCACHE_BIN="$(command -v ccache || true)" DISABLE_SCCACHE="${HYPERCOLOR_NO_SCCACHE:-0}" @@ -107,7 +114,7 @@ ITERATE="${HYPERCOLOR_ITERATE:-0}" # incremental. Whole-tree ops win with sccache instead. WANTS_SCCACHE=0 case "$CARGO_SUBCOMMAND" in - build | test | bench) WANTS_SCCACHE=1 ;; + build | test | bench | nextest) WANTS_SCCACHE=1 ;; esac if [ "$RELEASE_LIKE_PROFILE" -eq 1 ] || [ "$FORCE_SCCACHE" = "1" ] || [ "$FORCE_SCCACHE" = "true" ]; then WANTS_SCCACHE=1 @@ -127,6 +134,14 @@ if [ -n "$SCCACHE_BIN" ] && [ "$WANTS_SCCACHE" -eq 1 ]; then echo "[cargo-cache] sccache mode: Rust cached, incremental off (cap $SCCACHE_CACHE_SIZE)" echo "[cargo-cache] SCCACHE_DIR=$SCCACHE_DIR (HYPERCOLOR_NO_SCCACHE=1 or HYPERCOLOR_ITERATE=1 to disable)" else + # An ambient sccache RUSTC_WRAPPER combined with incremental hard-fails + # every compile; drop it rather than let the build die. + case "$(basename "${RUSTC_WRAPPER:-}")" in + sccache*) + unset RUSTC_WRAPPER + echo "[cargo-cache] unset ambient sccache RUSTC_WRAPPER for incremental mode" + ;; + esac export CARGO_INCREMENTAL="${CARGO_INCREMENTAL:-1}" if [ -z "$SCCACHE_BIN" ]; then echo "[cargo-cache] sccache not found; run 'cargo install --locked sccache' for shared build caching" @@ -229,9 +244,5 @@ done echo "[cargo-cache] CARGO_TARGET_DIR=$CARGO_TARGET_DIR" echo "[cargo-cache] MOZBUILD_STATE_PATH=$MOZBUILD_STATE_PATH" -if [ "$#" -eq 0 ]; then - set -- cargo build --workspace -fi - echo "[cargo-cache] running: $*" exec "$@"