From 8efd38e0bf77f2d538a906ab8a91bf240f4011a5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 22:40:48 +0000 Subject: [PATCH] Fix CUDA-Linux build: retry without sccache on nvcc device-compile crash CUDA 13.3's nvcc broke sccache's nvcc wrapping for -virtual architecture targets ("fatbinary fatal: Could not open input file 'acc.compute_75.ptx'" followed by "sccache: Compiler killed by signal 1"), caught by the CUDA-Linux job on the b10333/CUDA-13.3 publish dispatch (run 31339594933). The existing sccache-error retry-without-cache fallback in build.sh didn't trigger because its regex didn't cover this wording. Extend the regex so this failure mode falls back to an uncached build like every other known sccache/nvcc incompatibility, instead of redding the job. --- .github/build.sh | 8 ++++++-- CLAUDE.md | 10 +++++++++- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/.github/build.sh b/.github/build.sh index 6257904b8..95245f9ed 100755 --- a/.github/build.sh +++ b/.github/build.sh @@ -137,12 +137,16 @@ cmake -Bbuild $LAUNCH $@ || exit 1 # still fails fast (and is reported) instead of triggering a wasteful uncached rebuild. The # "Compiler not supported" signature additionally covers the CUDA case: if wrapping nvcc breaks # (sccache declining/erroring on the nvcc driver), the retry rebuilds the full-arch CUDA job -# without any launcher rather than redding it. +# without any launcher rather than redding it. "Compiler killed by signal" covers a second CUDA +# failure mode seen with CUDA 13.3's nvcc: sccache's nvcc wrapping loses an intermediate .ptx file +# for a "-virtual" architecture target (e.g. 75-virtual), so fatbinary aborts with "Could not open +# input file '*.ptx'" and sccache reports the underlying nvcc invocation as killed — an sccache/nvcc +# incompatibility, not a real compile error, so it gets the same uncached-retry treatment. build_log="$(mktemp 2>/dev/null || echo "/tmp/jllama-build.$$.log")" cmake --build build --config Release -j"${JOBS}" 2>&1 | tee "$build_log" build_rc=${PIPESTATUS[0]} if [ "$build_rc" -ne 0 ]; then - if [ -n "$LAUNCH" ] && grep -qiE 'sccache: error|Server startup failed|cache storage failed|Compiler not supported' "$build_log"; then + if [ -n "$LAUNCH" ] && grep -qiE 'sccache: error|Server startup failed|cache storage failed|Compiler not supported|Compiler killed by signal' "$build_log"; then echo "build.sh: build failed via an sccache cache error — retrying WITHOUT cache (clean reconfigure)." rm -f "$build_log" rm -rf build && mkdir -p build diff --git a/CLAUDE.md b/CLAUDE.md index 5fef23280..5c903605a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -526,7 +526,15 @@ v0.16.0 + the probe this is no longer a risk.) Job-by-job status: **100%** on CUDA / CUBIN / device-code (139 CUDA hits, 99.86% overall, 3 misses), cutting the job from **~51 min cold to ~15 min warm**. The first-run debug diagnostics (`SCCACHE_LOG` / `SCCACHE_ERROR_LOG` / `RUST_BACKTRACE`) were dropped once confirmed; `sccache --show-stats` still - prints the hit table every run. + prints the hit table every run. **CUDA 13.3 regression (caught on the b10333/CUDA-13.3 publish + dispatch, run 31339594933):** nvcc 13.3's device-compile pipeline broke sccache's nvcc wrapping — + `fatbinary fatal: Could not open input file 'acc.compute_75.ptx'` immediately followed by + `sccache: Compiler killed by signal 1`, on the very first `.cu` TU (a cold-cache miss, not a hit + issue) — an sccache/nvcc incompatibility for `-virtual` architecture targets (e.g. `75-virtual`), + not a real compile error. The existing mid-build retry-without-cache mechanism (see below) didn't + catch it because its trigger regex didn't include this failure's wording; `build.sh`'s regex now + also matches `Compiler killed by signal`, so this failure mode falls back to an uncached, green + `-O3` build like every other sccache/nvcc incompatibility instead of redding the job. 3. `crosscompile-linux-aarch64` — ✅ **enabled**, now a **native `ubuntu-24.04-arm` build** (not dockcross): `build.sh` self-fetches the aarch64 static-musl sccache (the fetch block in `build.sh` maps `uname -m` → `x86_64`/`aarch64`) and the probe guards it. See "Linux aarch64: