fix(ci): provision stock rust toolchain and run the mutation gate fresh - #15
Merged
Conversation
The cargo-mutants verdict is now a turbo task keyed on exactly the files that determine mutant survival (crate sources, tests, workspace and toolchain manifests). An unchanged tree replays the cached verdict in milliseconds instead of re-running the 3-minute mutant loop; any change to the classifier inputs re-executes. cacheDir is pinned to .turbo/cache so git worktrees stop silently sharing one cache with the main checkout.
flake.nix locks the entire toolchain (cargo, clippy, rustfmt, cargo-mutants, gcc, node, pnpm pinned to packageManager) behind one `nix develop`. CI's gate and mutation jobs provision from the same flake instead of dtolnay/rust-toolchain plus a per-run `cargo install cargo-mutants`, and the mutation job runs the gate through turbo with .turbo/cache persisted by actions/cache, so re-runs and same-input PR builds replay the cached verdict instead of re-executing mutants.
The actions/cache key now hashes the same input set the mutants task hashes (sources, tests, manifests, flake.lock) instead of the commit sha, so unchanged inputs restore an exact key, skip the post-job save, and stop growing the cache store on every commit. flake.lock joins the task inputs: nixpkgs pins cargo-mutants and rustc, so a toolchain bump regenerates a different mutant set and must invalidate the cached verdict. The pnpm store is cached too — pnpm install runs before turbo can replay anything and was a cold fetch on every run.
Review fixes (ce-code-review, validated): - turbo.json: add $TURBO_ROOT$/flake.nix to mutants inputs. flake.nix defines the toolchain; without it in the key a flake.nix-only edit replayed the previous verdict verbatim (probe: hash b956bf4d... stayed put, Cached (Local) = true). With it: miss + full re-execution. - ci.yml: add flake.nix, turbo.json and the npm package.json (which carries the gate script) to the turbo-bag hashFiles key. turbo self-hashes the latter two, so an exact-key restore was suppressing the post-job save on config-only edits: fresh verdicts were re-run every time and never persisted. - ci.yml: restore cargo build-state caching in the mutation job (origin/master had it; the turbo rewrite dropped it) keyed on Cargo.lock + flake.lock, sharing the gate job's cargo- restore prefix. - ci.yml: set CARGO_BUILD_JOBS=4 on the mutation run line for parity with the gate job and deterministic env hashing. Verified: mutants loop miss(117 green, 2m36s) -> hit(10ms) -> flake.nix edit miss(2m36s) -> revert hit(8ms); one-shot gate green (fmt, clippy -D warnings, 90 tests).
…oncept Compound of this session's learning: the turbo-cached mutation gate hashed the toolchain pin (flake.lock) but not the toolchain definition (flake.nix), so flake.nix-only edits replayed stale verdicts. Doc captures the fix, the two falsified adversarial hypotheses (script-body and env staleness — both busted by hash-movement probes), the transport-superset invariant, and the probe protocol. CONCEPTS.md gains the Verdict cache entry (two hash surfaces).
First Actions run failed in both gate and mutation jobs: 'bash -lc' inside nix develop -c re-sources the runner profiles, which re-prepend ~/.cargo/bin ahead of the dev shell PATH. The rustup shim then owned the toolchain: it synced the stable channel mid-build, removed the previous rust-std/rustc components, and every compile died with E0463 (can't find crate for std). cargo-mutants hit the identical failure in its scratch tree. 'bash -c' keeps the dev shell's PATH first; nix develop -c already exports the full toolchain. Comment added at the first use so the flag isn't reintroduced. Verified locally in the exact new form: one-shot gate green (90 tests), mutants cold miss (117 mutants, 113 caught, 4 unviable, 0 survived, 2m36s) then FULL TURBO hit in 9ms with CARGO_BUILD_JOBS=4 matching CI.
Run 2 failed differently from run 1: nix's cargo resolved rustc via PATH, the dev shell never shipped one, and on Actions runners the rustup proxy won the lookup — its stable toolchain then failed to link (std rlibs not applicable/missing). Locally the same shell only worked because ambient PATH satisfied the lookup by accident; 'type -P rustc' inside a clean 'nix develop -c bash -c' proved MISSING. cargo resolves rustc from PATH; a toolchain-provisioning flake that lists cargo without rustc is incomplete. Add rustc beside cargo with a comment naming the failure mode. Verified in a runner simulation (env -i, PATH=nix:/usr/bin:/bin, CI=true, cold pnpm store, no rustup on any path): pnpm install 2.9s; mutants cache miss -> full re-execution, 117 mutants green, 2m37s, exit 0; replay hit. The flake.nix edit itself flipped the turbo input hash, so the gate re-verified under the new toolchain set rather than replaying.
Drop the nix flake from the gate and mutation jobs. Adopt the standard actions-rust-lang setup-rust-toolchain action; fmt/clippy/mutants arrive as registry-installed cargo subcommands. The mutation gate now runs cargo mutants fresh instead of replaying a turbo verdict cache, so a green verdict means the mutant loop actually ran.
Master's nix-toolchain work (#14) diverged the same ci.yml lines this branch replaced with the stock toolchain; keep this PR's stock version on merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CI now provisions the toolchain through the standard
actions-rust-lang/setup-rust-toolchainaction (org-owned, pinned) instead of the repo's nix flake.fmt,clippy, and the mutation gate'smutantsresolve as registry-installed cargo subcommands (cargo-fmt,cargo-clippy,cargo-mutants), which is how modern rust-lang cargo discovers them.The mutation gate no longer replays a turbo verdict cache. It runs
cargo mutantsfresh on every classifier change, so a green100%verdict means the mutant loop actually ran this run — the SOTA guarantee is no longer transport-cache dependent.Why
rustup,rustfmt, andgcc.fmt/clippy/mutantsare cargo subcommands served from the registry — no custom action is required.100%replay observationally identical to a fresh run. Removing that cache restores the gate's meaning.Validation
Local reproduction of every gated command on the stock toolchain surface this run:
cargo fmt --checkcargo clippy --all-targets -- -D warningscargo test --all-targetscargo mutants --file crates/comment-checker/src/classify.rs --timeout 90The
flake.nix/flake.lock/turbo.jsonlocal surface is intentionally unchanged — the flake still provisions the local dev toolchain; only the CI consumption of it is replaced.