Roadmap: Mutation-Gated Testing to 1.0 (Phase 1–3)

[Hero-Gamer]

Suggested labels: testing, meta, roadmap
This is a tracking/meta issue — each item will be a separate PR. Checkboxes track progress.

Related

#676 Mutation-Gated Testing, #668 (313 mutants canceled at 30m), #678 (short gates share one runner), #679 (4-way sharded mutants after test), #685 (policy docs), #687 (ast-grep exact-variant)
Docs: docs/road-to-1.0.md (1.0 promises: datadir freeze, chain, IBD ~1s, 2 GiB RSS, P2P eclipse resistance), docs/quality.md Q-41 Rank 1 = 71 run / 196 skip, SCHEMA.md v24, docs/crash-recovery.md, TESTING.md, AGENTS.md
CI: .github/workflows/ci.yml required jobs fmt, deny, clippy, ast-grep, test, windows, macos, coverage, nixos-module-eval (9 jobs), fuzz.yml nightly 600s daily 07:17 UTC (17 7 * * *) + Sunday 3600s weekly long (17 7 * * 0), mutants.yml weekly Sunday 8-shard full sweep - both advisory not required PR checks, miri.yml nightly primitives-only Q-53

Motivation

Testing strength is the biggest gap to rbitcoin v1.0 release and goal is automated suite so strong only little real-world testing needed for each release. This aligns with road-to-1.0.md freeze-last gate.

Current gap:

• store: two-scan SH extract (unique keys, then fuse-hit postings) #668: 313 mutants = ~50 min work canceled at 30m cap, partial result
• Weak assertions is_err()/is_ok() and matches!(_) pass coverage but not kill mutants like pack_key_for_slot > vs >= and pwrite_all + vs -
• Time: 426 SystemTime::now / Instant::now hits, NodeClock AtomicI64 mock (0=wall) in crates/rbitcoin-consensus/src/clock.rs + thread-local duplicate, INV age gate wall-clock fallback
• Crash: docs/crash-recovery.md checkpoints but no fault injection
• Equivalent mutants: manual waiver, no expiry enforcement

Research: "Investigating the application of differential fuzzing to support the identification and suppression of equivalent mutants: An experimental study by Bruno Ely Reis Garcia" - 1090 mutants, 6 functions, 98-100% kill in 5/6 with 5 min per mutant, most killed in seconds, log-normal variance (budget by executions not wall-clock), seed corpora kill mutants unit tests miss. Limitation §7.3: return values + exception types only, not side effects (output buffers, pass-by-ref, global state) - relevant for rbitcoin-store torn-write.

Phase 1 — Contract, Lints, Feedback Loop (no production Rust changes — docs + YAML only)

• 1.1 Policy docs (docs: Mutation-Gated Testing — demonstrable killer per #676 #685) File: TESTING.md, AGENTS.md, .agents/skills/ship-pr/SKILL.md. Exact error-variant standard, waiver // MUTANT-WAIVER expires YYYY-MM-DD reason. No prod Rust. Solves: no standard.
• 1.2 ast-grep exact-variant rules (lint: weak asserts outside test files, and a waiver date token #687) Files: weak-assert.yml, waiver-expiry.yml via scripts/ast-grep.sh. Flags is_err()/is_ok() without exact variant, loose matches!(_), requires expiry date, scope src/ incl #[cfg(test)], ignores benches/fixtures. Defer time-source ban (426 hits per code-shape.md structural lint only). Verify: zero hits on master. Solves: weak assertions. Risk 🟢.
• 1.3 Mutants baseline-diff feedback (next 1-2d) Builds on ci: shard PR cargo-mutants next to coverage #679 + ci: run short gates on one runner #678 merged. ci: shard PR cargo-mutants next to coverage #679: 4-way --in-diff sharding needs: [fmt,clippy,test] overlapping coverage, continue-on-error, cargo mutants --workspace --in-diff git.diff --shard k/4 --sharding slice -j 2, baseline stays (few seconds auto-sets 20s per-mutant timeout, no --timeout 240), mutants.out per shard, skip when diff doesn't touch crates/, Cargo.toml, Cargo.lock, ci.yml, mutants.yml. Remaining: PR comment bot new vs docs/mutants/ baseline, fail on unexplained new survivors (diff only, weekly Sunday full stays advisory). Solves: actionable diff. Risk 🟢🟡.
• 1.4 Seed corpus mining (after 1.3) Files: fuzz/ corpora. Extract inputs that kill mutants unit tests miss → assert_eq! tests. Verify: count killed by corpora vs unit tests. Solves: free wins. Risk 🟢.

Phase 2 — Determinism Infrastructure

• 2.1 Unified clock API (2-3d, 🟡🔴) Consolidate to wall_time() + mockable override, remove Instant::now workarounds. Depends 1.2. Solves: time flake. Why 1.0: IBD ~1s measurement.
• 2.2 Wall vs monotonic split (1-2d, 🟡🔴) monotonic() for deadlines, NTP-immune, both mockable. Depends 2.1. Solves: deadline drift.
• 2.3 Exposed decision state (2-3d, 🟡) Return Decision { peer, window, reason }, assert intermediate values not just final height. Depends 1.1. Solves: intermediate-state assertions writable. Why 1.0: P2P eclipse gate.
• 2.4 Deterministic RNG injection (1-2d, 🟡) Single seed point, ban rand::thread_rng outside it, stable seed for peer selection/addrman. Depends 2.3. Solves: RNG flake.
• 2.5 Crash/recovery harness (1-2w, 🔴) Per docs/crash-recovery.md: 3 outcomes recover-to-prefix / rebuild-from-lower-layer / typed Corrupt. Harness fault → kill → reopen → verify invariant, covers 0/N,1/N,N-1/N torn writes where contract requires. Depends 1.1. Solves: Q-57/Q-58 fail-closed, freeze store last. Enables 2 remaining big store PRs.

Phase 3 — Evidence-Driven Advanced (triggered by measurements)

• 3.1 Differential fuzzing harness Trigger: 1.3 shows unexplained survivors. Duplicate target → same input to original + mutated → compare outputs. Budget 500K-1M execs (~5 min per mutant per paper, most killed in seconds, log-normal, parallelize high variance). Kill → promote input to unit test, Survive → candidate equivalent → brief review. 98-100% on Bitcoin Core funcs. ⚠️ Limitation §7.3: return values only, not side effects - for rbitcoin-store torn-write supplement with 2.5. Depends 2.3,2.4. Solves: equivalent mutant bottleneck for <5% FP needed for promotion.
• 3.2 Deterministic in-mem P2P transport Trigger: journey flake >0. Abstract tokio::net → control delay/reorder. Depends 2.2,2.4.
• 3.3 Loom concurrency Trigger: audit shows reducible lock ordering. cfg(loom) skip if atomics-only, gated on 2.5 fault-seam patterns. Depends 2.5.
• 3.4 Shuttle async Trigger: hard-to-reproduce bug. Depends 3.2.
• 3.5 Electrum golden transcripts Trigger: survivor rate in index paths. Record/replay vs Fulcrum/Electrs. Depends 2.3.
• 3.6 Selective mutant promotion Trigger: 20-30 PRs <5% FP stable runtime. Changed-path subset → optional required, full weekly stays weekly. Depends 1.3.

Verification for AI

1. Phase 1 zero prod Rust? Yes - docs + YAML only.
2. Does ci: shard PR cargo-mutants next to coverage #679 solve concurrency? Check ci.yml needs: [fmt,clippy,test], 4 shards overlapping coverage, skip logic.
3. Time-source ban deferred per code-shape.md? Yes - 1.2 defers.
4. Does 2.5 map to docs/crash-recovery.md 3 outcomes? Yes.
5. Does 3.1 budget match paper? Paper 5 min per mutant, execution budget not wall-clock, log-normal, seed corpora kills missed mutants. 500K-1M execs translation correct.
6. Does roadmap block LDK REST / CLN plugin? No - after store PRs, Phase 3 gated, After 1.0 per road-to-1.0.
7. Schedule claim verified: fuzz.yml nightly 600s daily (17 7 * * *) + Sunday 3600s weekly long (17 7 * * 0)【4674037048076786992†L5-L8】, mutants.yml weekly Sunday full 8-shard sweep, both advisory not required PR checks per road-to-1.0.md Nightly fuzz.yml (not a required PR check)【2230000870980475997†L37-L40】.

Non-Goals

Not solving 2 GiB RSS measurement, IBD ~1s re-baseline, Tor, BIP331, Apple notarization - separate 1.0 gates in quality.md.

Next

1. Merge docs: Mutation-Gated Testing — demonstrable killer per #676 #685, lint: weak asserts outside test files, and a waiver date token #687
2. 1.3 bot (1-2d) - ci: shard PR cargo-mutants next to coverage #679 merged
3. 1.4 seed mining
4. 2.1 unified clock

---

Draft 2.1 - 2026-09-23
Will continue to update as conversation goes on and more things I can integrate from the research paper