[Mikola Lysenko (mikolalysenko)]

Problem

vlt patch compatibility is the most re-run workflow in the repo: of its last 500 runs (~36 h), 31 needed a re-run or ended red. Clustering the first-attempt failures, the biggest group is the native backtest (native (…) › Backtest against production) failing on a transport blip that the backtest's own retry does not recognize, and that failure then also turns lock-diff red:

run	job	first-attempt result
37784462525 (PR, re-run)	native (ubuntu, 1.0.0-rc.14)	BAD 1.0.0-rc.14-hosted-direct: safe-refusal (expected patched) failing=[]
same run	lock-diff	vlt-lock.json differs across OS for 1.0.0-rc.14 hosted direct (a cascade: the refused Linux lock compared against a patched Windows lock)
37643810504 (PR, red)	native (ubuntu, 1.0.10)	BAD 1.0.10-hosted-two-versions: safe-refusal (expected patched) failing=[]
37685198846 (PR, re-run)	native (macos, 1.2.0)	retry attempt hit vlt install exited 1: … code: 'ETIMEDOUT', syscall: 'connect' and was not retried again
37603380236, 37646658494, 37702066914, 37652886935 …	native / lock-diff	same family (7 lock-diff first-attempt failures in the window)

Each one costs a manual re-run of the failed vlt legs and blocks the PR's compat status until then.

Root cause

run_with_retries re-runs a cell only when transient(row) says the failure was transport. It matched just two CLI strings (error sending request for url (, API request failed with status 5xx). Two transport failures slip through:

1. The hosted vlt artifact preflight's own fetch drops. Before pinning a vlt lock, the CLI fetches the artifact the way vlt does (vlt_preflight.rs). On fetch error <e> or http 5xx it withholds the dep with redirect_vlt_artifact_unverifiable ("vlt would fail to verify : ; nothing was written …"), writes nothing and exits 0. The backtest then sees no changed files and scores a safe refusal (failing=[]), even though its own probe of the same URL had just verified. Since the run succeeded, the envelope never reached the row (cliFailures holds only non-zero exits), so no retry could see the reason.
2. vlt's own registry fetch times out or resets (ETIMEDOUT, ECONNRESET, EAI_AGAIN, socket hang up), surfacing as RuntimeError: vlt install exited 1: … on the row.

Fix

scripts/backtest-vlt.py only:

• Cell.patch_run keeps every redirect_vlt_artifact_unverifiable detail from the scan/get envelope (exit 0 or not) on the row as preflightFailures.
• transient() also matches:
  • PREFLIGHT_TRANSPORT_FAILURE: vlt would fail to verify <url>: http 5xx or : fetch error …, except the deterministic fetch errors (too large cap, refusing a non-http(s) artifact URL);
  • VLT_TRANSPORT_FAILURE: ETIMEDOUT / ECONNRESET / EAI_AGAIN / socket hang up.
• These stay final: content-encoding <v> (the real BLOCKED case, still handled by blocked_refusal), sha512 mismatch, 4xx, offline, ENOTFOUND and ECONNREFUSED (a wrongly rewritten URL fails that way too).

This is the existing narrowly scoped retry (3 attempts, fresh tree each, earlier attempts recorded under transportRetries on the row), extended to two more documented transport signatures. A cell that keeps failing still fails, and matchesExpectation rows never retry. No assertion is weakened and no cell or leg moves.

Proof

• New unit tests in scripts/tests/test_backtest_harnesses.py (VltRetryTests): each transient preflight reason and network errno retries, each deterministic one does not, and a successful (exit-0) run's preflight reason reaches result.json and the classifier. Without the fix: 10 failures + 1 error; with it: all pass.
• python3 -B -m unittest discover -s scripts/tests (what ci.yml runs): 257 tests OK.
• The PR's paths include scripts/backtest-vlt.py, so the vlt compatibility workflow runs the full native matrix on this PR.

Limitation: the artifact blobs from the failing runs can't be downloaded from this environment, so the exact preflight detail of the two hosted-*: safe-refusal failures wasn't read directly. Exit 0 with nothing written and failing=[] on a must-patch hosted cell, right after the cell's own probe verified the same URL, matches the preflight-withheld path, and both cells passed on re-run against the same binary. I haven't ruled out other exit-0 no-write paths from the logs alone. If a future failure shows a different reason, preflightFailures now puts it in result.json.

Where each test still runs

Unchanged: every cell, leg and check runs where it did before (PR, main push, nightly).

🤖 Generated with Claude Code

https://claude.ai/code/session_01MhQVTZiJczw9jt27ZysyRy

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 12061bb. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Burn-down agent: labeled Ready for review at head 12061bba6a.

• CI: 261/261 non-skipped check runs green on this head (11 skipped), mergeable clean.
• Bugbot: reviewed 12061bb, no findings.
• Already approved once; ready for merge-queue.
• Reviewer focus: scripts/backtest-vlt.py now treats the CLI's exit-0 hosted artifact preflight fetch drops and vlt ETIMEDOUT/ECONNRESET as transport retries instead of scoring them as safe-refusal/error.

---

Generated by Claude Code