v5: fix partial-stage repair bug, cut redundant downloads

[Mikola Lysenko (mikolalysenko)]

Waste-review follow-up W4, from #286. Based on release/v5-prerelease, currently merged up to f9cb7e1 (through #279, #280, #281, #282 and #291). The only conflict was a CHANGELOG bullet when merging #282; I kept both bullets.

Finding	Outcome
F67 latent bug: the disk stager treated a diff archive as covering created files	done. I wrote the failing tests first. On the base, apply --offline with only diffs on disk patched index.js, then failed on the created new.js and left the package half-patched. Online apply never fetched the created file's blob, and after a default repair an offline apply still failed. Coverage is now counted per file: a diff covers only files with a beforeHash. Online apply fetches just the created files' blobs, and diff-mode repair downloads them too. That repair's JSON gets one extra downloaded event with mode: "file".
F28: the vendored path POSTed /patches/package once per uuid	done. The vendor download plan's first call now sends one reference request, in chunks of 500, for the plan from its own position onward. That request replaces the call's own single-uuid request and uses the same retry ladder, so an outage costs what the serial loop paid. Positions the loop already passed over are never granted. A pending_build answer is asked again at that package's turn. Hosted scan's fetch_registry_references is chunked at 500 too; above that the endpoint returns 400.
F70/F71: qualifier-less pypi patches downloaded the served sdist before rejecting it	done (one fix). A pypi reference whose artifact isn't a .whl is refused before the GET, in both the vendor loop and the download plan. Outcomes are unchanged: auto still warns vendor_prebuilt_unavailable and builds locally, and service still refuses.
F80: auto mode eagerly fetched pristine upstream for uninstalled packages	done for npm, golang and composer; cargo already deferred. The deferral applies only when the fetch would really download. The fetchers' pre-download refusals are now one shared registry_fetch::refusal_before_download, used by both the fetch and the deferral: foreign yarn berry cacheKey, a go module go fetches without a proxy, a composer entry with no dist URL. The npm and yarn classic inventories no longer treat a file: or git resolution's hash as a registry integrity. pypi and gem keep the eager fetch, because the loop's installed-variant probe reads the pristine tree before the backend runs. nuget and maven have no fetch rung and stay frozen.
F59: vex refetches every record per run	deferred. Re-verified on the current base: within one run, vex already uses one client and fetches each distinct uuid once (vex_sources::fetch_records). The remaining waste is across runs, and fixing it needs a persisted per-uuid record cache. That would be new project or user state in a ledger-free hosted mode, and it could hide a withdrawn patch from VEX. It needs an owner decision and belongs with WS6 (#282).
F60: hosted scan downloads each patched wheel in full to read METADATA	deferred. Needs server support: depscan would persist PEP 658 metadata and its hash and return them from /patches/package. The serve proxy has no Range support to read the file partially.

Savings

• F28: a vendored run with N planned service downloads now makes ⌈N/500⌉ reference POSTs instead of N, each costing a round trip plus a quota unit. The depscan fixture's 71 grants become 1 POST.
• F70/F71: every qualifier-less pypi patch in a vendored run saves one full sdist download, up to the 256 MiB cap.
• F80: every not-installed npm, golang or composer package the service serves saves a registry download and verification.
• F67 is a correctness fix; it adds at most the created files' blobs.
• Diff: about +1450 / −170 lines across 16 files, mostly tests.

Tests

• tests/diff_created_file_e2e.rs (5): three failed on the base before the fix. They cover offline apply, online apply, repair followed by offline apply, offline repair, and the repair JSON event.
• fetch_stage units.
• vendor_prefetch batch tests, run against a mock that answers every uuid in a request:
  • one POST serves the whole plan
  • an outage costs the serial loop's ladders
  • a building package is asked again
  • the batch is capped at 500
  • the batch starts at the first call
• fetch_registry_references chunking.
• The pypi sdist tests assert that no GET is made. I confirmed they fail without the fix.
• refusal_before_download matches each fetcher's first refusals.
• scan_vendor_e2e::vendor_auto_takes_a_missing_package_from_the_service_without_the_registry: no registry request, and under --offline zero requests to either server. I confirmed it fails when deferral is limited to cargo.
• yarn git and npm file: inventory tests.
• Local results after the latest merges (a77ecad, b2df60b):
  • cargo clippy --workspace --all-targets -D warnings is clean.
  • The cli lib tests pass (805).
  • These e2e suites pass: diff_created_file_e2e (5), scan_vendor_e2e (32), covgap_commands_scan_hosted (50), covgap_commands_vex (12), e2e_vex (16), e2e_vex_vendor (21), hosted_memory_engine (28), hosted_memory_parity (25).
  • covgap_commands_vendor (43) and repair_invariants (22) pass when run as a non-root user; their chmod-based cases can't fail under root.
• Two adversarial reviews: correctness, and coverage/offline/owner decisions. Their confirmed findings (the repair double count, batch over-granting, non-registry npm entries, docs) are fixed in 7cfdbd4.

CI failures that come from the base (details in the comments; each reproduces on a clean base checkout, and this PR doesn't touch the code involved):

• vlt install-proof (0.0.0-1, 0.0.0-11): after WS3: one lockfile model per ecosystem #281, hosted rollback in e2e_redirect_vlt_build (hosted_rollback_byte_exact, hosted_idempotence, hosted_crlf_lock) writes vlt-lock.json back without each node's tarball URL.
• yarn-classic 1.0.2, 1.6.0, 1.7.0, 1.9.4: after v5 WS1+WS2: ledger-free hosted mode, upstream-restore rollback, vendor ejects hosted projects #280, mode_migration_npm::classic_vendored_then_hosted_takeover_leaves_pure_hosted fails because the upstream restore refuses a pre-1.10 lock entry, which has no integrity field.

On b2df60b every other job is green. Three production-service cells each failed once and passed on their one re-run: Pipenv 2026.8.0 crlf hosted, vlt 1.2.0 vendored-direct (ubuntu), and Poetry 2.0.1 direct (ubuntu).

Process note: early on I force-pushed once, after amending my own not-yet-reviewed commit. Everything since is plain commits and merge commits.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WzdTEhubve9yWfqBE7vAsB

[Mikola Lysenko (mikolalysenko)]

[agent] CI note on the red checks. This PR is still a draft, waiting on #283.

• vlt install-proof: e2e_vlt::vlt_pinned_matrix_agent_get_and_remove fails on the base. A bare get now defaults to hosted mode. I've ported WS5: one VendoredBackend for vendored apply/revert/repair; cut repair's ledger rebuild #283's --mode agent fix (c64b7b9), and on c64b7b9 that test passes on every leg I checked. The legs that are still red fail earlier in the job, in the ledger-free hosted rollback path (e2e_redirect_vlt_build, hosted_wiring_contested, "vlt-lock.json has no lockfileVersion"). v5 WS1+WS2: ledger-free hosted mode, upstream-restore rollback, vendor ejects hosted projects #280's own last run (36383945937) and WS5: one VendoredBackend for vendored apply/revert/repair; cut repair's ledger rebuild #283 fail the same way, and no branch has a fix yet. This PR does not touch patch::redirect.
• bun native (ubuntu-latest, 1.1.43): on 4da7bd1 one case failed, text-workspace hosted (refusalCodesExact, exitCodeContract). The expected refusal code was missing and the run exited 1. That cell only runs a hosted scan, and the only hosted code this PR changes is the package-reference chunking, which is a no-op below 500 uuids. The case passes on WS5: one VendoredBackend for vendored apply/revert/repair; cut repair's ledger rebuild #283 and on the base. When I ran it locally, the same two checks failed with the same symptoms, and the cause was a network error reaching patches-api.socket.dev. So I read the CI failure as a transient service or network error. The job is re-running on c64b7b9, and I'll dig in if it fails again.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

#283 landed on release/v5-prerelease as 06437d2; please merge origin/release/v5-prerelease again, resolve conflicts, get green, and keep it ready.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] ready to land

This is head 9a10303, which is release/v5-prerelease at 06437d2 (#283) merged in, with no conflicts.

CI on 9a10303: clippy, coverage, test (ubuntu/macos), test-release, the e2e-docker / coverage-docker / setup-matrix jobs, hosted-e2e, node-addon, release-readiness and lint-ecosystems all pass. The pipenv compatibility run passed on its one re-run: attempt 1 had a macOS DNS error plus one ubuntu agent-oot case.

The remaining reds also fail on #283 on the same base:

• vlt install-proof (23 legs): these are all in the hosted-rollback path, 209 panics in total, each hosted_wiring_contested / patched_ref_unattributable: vlt-lock.json …. The failing tests are in e2e_redirect_vlt_build (hosted_crlf_lock, idempotence, optional_dependency_heal, resave_*_rollback, rollback_byte_exact) and mode_migration_vlt (agent_rollback_after_takeovers, rollback_from_mixed, scoped_unwind_one_of_two, vendored_then_hosted). WS5: one VendoredBackend for vendored apply/revert/repair; cut repair's ledger rebuild #283's vlt run 36417124814 fails the same way, and no branch has a fix yet. e2e_vlt, e2e_safety_vlt and e2e_vendor_vlt_build pass on every leg. e2e_vlt::get_and_remove passes because of the --mode agent fix (c64b7b9), which was already in WS5: one VendoredBackend for vendored apply/revert/repair; cut repair's ledger rebuild #283.
• test (windows-latest): the only failing target is covgap_commands_scan_mod. WS5: one VendoredBackend for vendored apply/revert/repair; cut repair's ledger rebuild #283's job 108911094705 fails the same way.

---

Generated by Claude Code

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Resolved by another fix: Git HTTPS locks deferred as registry
  • The bug was already fixed in commit 7ce6840 which filters git resolutions (including .git URLs and codeload.github.com) from the Yarn classic registry view and clears integrity fields on git dependencies.

Or push these changes by commenting:

@cursor push 62e896c51a

Preview (62e896c51a)

diff --git a/crates/socket-patch-core/src/vendor/lock_inventory/mod.rs b/crates/socket-patch-core/src/vendor/lock_inventory/mod.rs
--- a/crates/socket-patch-core/src/vendor/lock_inventory/mod.rs
+++ b/crates/socket-patch-core/src/vendor/lock_inventory/mod.rs
@@ -65,8 +65,8 @@
 pub(crate) mod pnpm;
 pub(crate) mod pypi;
 pub(crate) mod recover;
+pub mod view;
 pub(crate) mod vlt;
-pub mod view;
 pub(crate) mod wired;
 pub(crate) mod yarn;
 
@@ -365,7 +365,16 @@
 /// cannot reproduce; such entries stay listed for discovery but the fetch
 /// layer's integrity rule decides fetchability).
 fn http_url(raw: &str) -> Option<String> {
-    (raw.starts_with("https://") || raw.starts_with("http://")).then(|| raw.to_string())
+    if !(raw.starts_with("https://") || raw.starts_with("http://")) {
+        return None;
+    }
+    // A `.git` suffix (with or without a `#fragment`) is a git repository,
+    // not a tarball artifact the registry conventions serve.
+    let before_fragment = raw.split('#').next().unwrap();
+    if before_fragment.ends_with(".git") {
+        return None;
+    }
+    Some(raw.to_string())
 }
 
 /// ARCHITECTURE GUARD (module docs): each per-format file is laid out as

diff --git a/crates/socket-patch-core/src/vendor/lock_inventory/tests.rs b/crates/socket-patch-core/src/vendor/lock_inventory/tests.rs
--- a/crates/socket-patch-core/src/vendor/lock_inventory/tests.rs
+++ b/crates/socket-patch-core/src/vendor/lock_inventory/tests.rs
@@ -907,6 +907,35 @@
     assert_eq!(e.integrity, LockIntegrity::None);
 }
 
+/// An https URL ending in `.git` is a git repository, not a registry
+/// tarball: the `#<commit>` fragment is not a tarball sha1, and a
+/// standalone integrity field verifies nothing the registry serves.
+#[tokio::test]
+async fn yarn_classic_dot_git_https_url_is_not_a_registry_entry() {
+    let tmp = tempfile::tempdir().unwrap();
+    write(
+        tmp.path(),
+        "yarn.lock",
+        "# yarn lockfile v1\n\n\
+         \"https-git@1.0.0\":\n\
+         \x20 version \"1.0.0\"\n\
+         \x20 resolved \"https://github.com/o/https-git.git#0123456789abcdef0123456789abcdef01234567\"\n\
+         \n\
+         \"with-integrity@git+https://example.com/with-integrity.git\":\n\
+         \x20 version \"2.0.0\"\n\
+         \x20 resolved \"git+https://example.com/with-integrity.git#abc\"\n\
+         \x20 integrity sha512-standaloneIntegrity==\"\n",
+    )
+    .await;
+    let (_, entries) = inventory_npm_lock(tmp.path()).await.unwrap().unwrap();
+    let e1 = entry(&entries, "https-git");
+    assert_eq!(e1.resolved, None);
+    assert_eq!(e1.integrity, LockIntegrity::None);
+    let e2 = entry(&entries, "with-integrity");
+    assert_eq!(e2.resolved, None);
+    assert_eq!(e2.integrity, LockIntegrity::None);
+}
+
 // ── yarn berry ────────────────────────────────────────────────────────
 
 const YARN_BERRY: &str = "# This file is generated by running \"yarn install\" inside your project.

diff --git a/crates/socket-patch-core/src/vendor/lock_inventory/yarn.rs b/crates/socket-patch-core/src/vendor/lock_inventory/yarn.rs
--- a/crates/socket-patch-core/src/vendor/lock_inventory/yarn.rs
+++ b/crates/socket-patch-core/src/vendor/lock_inventory/yarn.rs
@@ -141,7 +141,8 @@
         // `resolved "url#sha1hex"` — the fragment is the legacy verifier of
         // a registry tarball. A git resolution's fragment is a commit id,
         // which verifies nothing a registry fetch could download.
-        let (resolved, sha1_hex) = match classic_field(&block.lines, "resolved") {
+        let resolved_raw = classic_field(&block.lines, "resolved");
+        let (resolved, sha1_hex) = match resolved_raw {
             Some(raw) => {
                 let (url, sha1) = split_resolved_sha1(raw);
                 match http_url(url) {
@@ -151,10 +152,16 @@
             }
             None => (None, None),
         };
-        let integrity = classic_field(&block.lines, "integrity")
-            .map(|i| LockIntegrity::Sri(i.to_string()))
-            .or(sha1_hex.map(LockIntegrity::Sha1Hex))
-            .unwrap_or(LockIntegrity::None);
+        // A non-registry resolution (`file:`, `git+…`, `.git`) records the
+        // integrity of an artifact no registry serves: nothing a registry
+        // fetch could verify against.
+        let integrity = match (&resolved, resolved_raw) {
+            (None, Some(_)) => LockIntegrity::None,
+            _ => classic_field(&block.lines, "integrity")
+                .map(|i| LockIntegrity::Sri(i.to_string()))
+                .or(sha1_hex.map(LockIntegrity::Sha1Hex))
+                .unwrap_or(LockIntegrity::None),
+        };
         out.push(LockfileEntry::npm(name, version, resolved, integrity));
     }
     out

_{You can send follow-ups to the cloud agent here.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 9a10303. Configure here.}

[Mikola Lysenko (mikolalysenko)]

#281 landed on release/v5-prerelease as 73c0c4f; please merge origin/release/v5-prerelease again, resolve conflicts, get green, and keep it ready.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] I merged release/v5-prerelease at 73c0c4f (#281) into this PR as b2aea01. There were no conflicts. Locally, clippy is clean with warnings as errors, diff_created_file_e2e passed 5/5, scan_vendor_e2e passed 32/32, and the cli lib tests for fetch_stage, repair and vendor passed 151/151.

The vlt install-proof legs are failing again, first seen on the 0.0.0-1 and 0.0.0-11 jobs, but for a new reason that comes from the base. In e2e_redirect_vlt_build, hosted_rollback_byte_exact, hosted_idempotence and hosted_crlf_lock fail because rollback writes vlt-lock.json back without each node's tarball URL: the 4th tuple element (https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz) is dropped, or the proxy URL is kept.

I reproduced the same three assertion failures on a clean 73c0c4f checkout with vlt 0.0.0-1, so this PR doesn't cause them; its diff doesn't touch vlt or the hosted rollback code. The likely cause is #281's lockfile model rewrite of vlt rollback. I didn't find a fix in the open PRs, so I have nothing to port here, and I didn't re-run the jobs because the failure reproduces reliably. The earlier hosted_wiring_contested vlt failure and the Windows covgap_commands_scan_mod failure also come from the base (see above).

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] The yarn-classic 1.0.2, 1.6.0, 1.7.0 and 1.9.4 matrix jobs fail on b2aea01, and the cause is in the base. In each job every VEX cell passes; only the mode_migration_npm suite fails, in classic_vendored_then_hosted_takeover_leaves_pure_hosted. This PR doesn't change that test or the hosted restore code. Both came from #280.

I reproduced it locally with yarn 1.6.0. yarn releases before 1.10 never write an integrity field to yarn.lock, so there are two layers:

1. mount_registry_from_classic_lock (tests/mode_migration_npm.rs) panics with no `integrity` in … because it requires that field in the lock block.
2. If integrity is made optional there, the mocked registry document has no integrity, and rollback returns partial_failure: "cannot restore pkg:npm/left-pad@1.3.0 to its upstream registry entry: the registry records no integrity for left-pad@1.3.0".

With that edit the test passes on yarn 1.10.1 and still fails on 1.6.0.

Proposed fix, for #280's owners: serve the real registry's dist.integrity in the mock instead of mirroring it from the lock. Then decide whether the upstream restore of a pre-1.10 classic entry should write the lock back without the integrity line, which the pristine lock never had. The second part is a behavior decision, so I'm not widening this PR with it. No open PR fixes this yet, and I didn't re-run the jobs because the failure reproduces every time.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

#279 landed on release/v5-prerelease as f6bdad5; please merge origin/release/v5-prerelease again, resolve conflicts, get green, and keep it ready.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] I merged release/v5-prerelease at f6bdad5 (#279) into this PR as 6f61425. There were no conflicts. Locally, clippy is clean with warnings as errors, diff_created_file_e2e passed 5/5, scan_vendor_e2e passed 32/32, and the cli lib tests for fetch_stage, repair and vendor passed 149/149. covgap_commands_vendor passed 43/43 and repair_invariants passed 22/22 when run as a non-root user; their chmod-based cases can't fail as root. The vlt and yarn-classic (< 1.10) reds explained above come from the base, and #279 doesn't change them.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] On 6f61425, Pipenv compatibility matrix (ubuntu-latest, 2022.12.19 2026.8.0, crlf marker-excluded extras category) failed in one cell: pipenv 2026.8.0 · crlf · hosted, where lockOnlyApplies shows lock-only applied=0. The other 25 cells passed, including crlf/hosted on 2022.12.19 and crlf/vendored on 2026.8.0.

The same workflow passed on this PR's previous head b2aea01 and on #279's final head, so so far it has failed only on their merge. Locally I can't tell a regression from a flake: the cell fails every time on 6f61425, f6bdad5 and 73c0c4f alike, so the failure here comes from this environment (Python 3.12 instead of CI's interpreters). I've re-run the failed job once. If it fails again I'll treat it as a real interaction between #279 and this PR and fix it here.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

#282 landed on release/v5-prerelease as 14a9cb0; please merge origin/release/v5-prerelease again, resolve conflicts, get green, and keep it ready.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] I merged release/v5-prerelease at 14a9cb0 (#282) into this PR as a77ecad. The only conflict was in CHANGELOG.md under "Changed", where both sides added a bullet; I kept both.

Locally:

• cargo clippy --workspace --all-targets -D warnings is clean.
• The cli lib tests pass (805).
• diff_created_file_e2e (5), scan_vendor_e2e (32), covgap_commands_scan_hosted (50), coverage_fix_scan_hosted_dryrun_vendored (5), covgap_commands_vex (12), e2e_vex (16), e2e_vex_vendor (21), hosted_memory_engine (28), hosted_memory_parity (25) and hosted_management_refusals (6) pass.
• These pass when run as a non-root user; their chmod-based cases can't fail under root: covgap_commands_vendor (43), repair_invariants (22), and the core pypi_poetry/pypi_requirements wire-failure tests.

On the previous head, 6f61425, CI finished with only the base-caused reds described above: vlt install-proof 0.0.0-1/0.0.0-11, and yarn-classic 1.0.2–1.9.4.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] On a77ecad, vlt native (ubuntu-latest, 1.2.0) failed one of its 36 cells: 1.2.0-vendored-direct went unsafe because repeatStableLock failed. That check means the second vendored run either exited non-zero or changed vlt-lock.json.

This looks like a flake, not a regression from the #282 merge:

• The same cell passed with vlt 1.2.0 on macOS and Windows in this same run.
• The ubuntu 1.2.0 job passed on this PR's previous head, 6f61425, and on v5 WS4/WS6: one hosted engine for disk + memory; unified Ledgers view #282's latest vlt run (24a8241).
• The harness runs against the production patch service, which reported 12 cells as blocked-by-server-encoding in this job.

I can't reproduce it locally because this sandbox's network policy blocks patches-api.socket.dev. Once the vlt workflow run finishes I'll re-run the failed job once. If this cell fails again, I'll treat it as a real regression and use the job's captures/**/cli-output.json artifact to find the cause.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

#291 landed on release/v5-prerelease as f9cb7e1; please merge origin/release/v5-prerelease again, resolve conflicts, get green, and keep it ready.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] I merged release/v5-prerelease at f9cb7e1 (#291) into this PR as b2df60b, with no conflicts. #291 touches only CI workflows, scripts and docs, not files this PR changes.

Locally:

• cargo clippy --workspace --all-targets -D warnings is clean.
• The cli lib tests pass (805).
• diff_created_file_e2e (5) and scan_vendor_e2e (32) pass.
• v5 CI: build e2e binaries once and tier the PM matrix #291's test_ci_e2e_tiers/test_ci_vlt_rows script tests pass (26).

On a77ecad the vlt native (ubuntu-latest, 1.2.0) flake passed on its one re-run, so the only reds left were the base-caused ones: vlt install-proof 0.0.0-1/0.0.0-11 (#281), and yarn-classic 1.0.2–1.9.4 (#280).

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] On b2df60b, Poetry native (ubuntu-latest, 2.0.1) failed its two direct cells: hosted (rollbackExit0, rollbackRestoresLockBytes, rollbackRestoredUpstream) and vendored (rescanIdempotent). The crlf hosted/vendored and direct agent cells passed.

This doesn't look like a code change:

• The same job passed on the previous head, a77ecad (run 36486207786).
• The merge between a77ecad and b2df60b is v5 CI: build e2e binaries once and tier the PM matrix #291 only. It changes CI workflows, scripts, docs and two unrelated e2e tests (e2e_maven, e2e_nuget), but no Rust source and not backtest-poetry.py or its workflow. So the CLI under test behaves the same.
• The harness runs against the production patch service, which I can't reach from this sandbox (patches-api.socket.dev is blocked).

Once the Poetry run finishes I'll re-run the failed job once. If it fails again I'll treat it as real and use the job's captures/**/*.log to find the cause.

---

Generated by Claude Code