WS5: one VendoredBackend for vendored apply/revert/repair; cut repair's ledger rebuild

[Mikola Lysenko (mikolalysenko)]

WS5 from docs/design/v5-plan.md. Based on release/v5-prerelease at 686e5fb (includes #280, ledger-free hosted).

Depscan check (blocking step)

Checked from a fresh clone of SocketDev/depscan master (784013d6). Depscan does not use the CLI's local pack/rebuild path.

• Its server-side prebuilt artifacts come from its own TypeScript repackers (workspaces/patches/src/repack/). It never spawns socket-patch, and it has no Cargo or napi dependency on the socket-patch crates.
• Depscan uses the CLI binary only through apply/get/scan/vex, in dev tools and live e2e tests. It also reads fixtures from the submodule.
• The packing code (npm_pack, pypi_wheel, berry_zip, registry_fetch, prestage) is kept. It is the CLI's own --vendor-source build/auto fallback. Only helpers that just the ledger rebuild used were removed: fetch_npm_unverified and gem::reconstruct_gem_wiring.

What changed

• New commands/vendored_backend/ with VendoredBackend { apply, revert, repair }.
  • apply: stages patch content in memory, then calls vendor_records_reusing.
    • Used by vendor, scan --mode vendored, get --mode vendored, and v5 WS1+WS2: ledger-free hosted mode, upstream-restore rollback, vendor ejects hosted projects #280's hosted-project eject in vendor.
    • The three boxed_scan_vendor_step* shims, stage_and_vendor and boxed_vendor_records collapse into one boxed_vendor_step.
    • The engine future stays boxed inside apply, for the 1 MiB main-thread stack on Windows.
  • revert: used by vendor --revert, the manifest reconcile, rollback's vendored leg and both remove paths.
  • repair: replaces repair_vendor.rs (2.6k lines).
    • It health-checks ledger entries and re-vendors missing or corrupt artifacts through apply. Under --vendor-source auto it tries the service's prebuilt artifact first and falls back to a local build.
    • Before moving a corrupt artifact aside, it harvests the artifact's afterHash-verified members as patch content, so --offline can still repair it.
    • The result is checked against the original ledger entry. A source that produces different bytes, such as a service archive re-gzipped since vendoring, is never committed. That candidate's wiring files and ledger entry are put back, and repair retries with a build-only local rebuild. Lockfiles and state.json stay byte-identical.
    • A verified rebuild that the backend migrated (the cargo retag) is kept. So is a carried-inventory refresh (vendor_inventory_refreshed).
• Ledger re-synthesis is cut. A lockfile reference that has no ledger entry now fails with vendor_ledger_missing: an artifact-level event with uuid and details.{ecosystem,path}, and exit 1. The remedy is to restore state.json from version control. The rollback and vendor messages for this case now say that too.
• Docs and CI: CLI_CONTRACT, README, CHANGELOG, the v5 plan, the CI path filters and docs/testing/vlt-coverage.json.
• The gated real-toolchain get fixtures now pass --mode agent. This covers a gap 5e5f5ed left on the base branch.

Behavior notes

• A drifted installed copy of a gem or pypi release variant now fails vendor's installed-variant probe during repair. Before, repair force-overwrote it. Repair's failure details are now the engine's own messages.
• Removed tests: reconstruction, soft-restore, gem wiring backfill, and their persist-failure variants. They are replaced by vendor_ledger_missing tests for npm, pnpm, yarn berry, bun, bun.lockb and vlt.
• Review regressions (offline harvest, repair identity) are pinned by repair_offline_harvests_a_corrupt_artifacts_valid_members and repair_never_rewires_to_different_service_bytes.

Validation (head 5f8573b)

• cargo clippy --workspace --all-targets --all-features -- -D warnings: clean.
• Full workspace cargo test, run locally as root: 10,162 passed, 18 failed. The same 18 fail on the base branch here. They are chmod-based tests that do nothing as root, including v5 WS1+WS2: ledger-free hosted mode, upstream-restore rollback, vendor ejects hosted projects #280's new partial_lockfile_write_failure_exits_1_and_writes_no_ledger, plus hosted cargo fetches. Re-running the permission-dependent tests as non-root passes them.
• CI failures that come from the base branch, each explained in a PR comment:
  • e2e_redirect_cargo_build: hosted cargo VEX, F77. A separate routine is fixing it.
  • Windows covgap_commands_scan_mod.
  • vlt install-proof / e2e_redirect_vlt_build: hosted rollback returns hosted_wiring_contested. The same failure is in v5 WS1+WS2: ledger-free hosted mode, upstream-restore rollback, vendor ejects hosted projects #280's run 36383945937.
• Two macOS hosted-mode jobs, Poetry 1.8.5 and Bun 1.0.0, hit a runner network error (ConnectionError / DNS nodename nor servname). The vendored cases in the same jobs pass. I've re-run them once.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DMhWChtaNX5FJYNDq3NBJa

---

Note

Medium Risk
Touches every vendored workflow and changes recovery when the vendor ledger is missing; incorrect repair/revert behavior could leave lockfiles pointing at bad artifacts, but verification and fail-closed ledger rules limit blast radius.

Overview
Introduces a shared VendoredBackend (apply / revert / repair) so vendor, vendored scan/get, rollback, remove, and repair all use the same vendoring engine and revert policy instead of the deleted repair_vendor.rs monolith.

repair now re-vendors broken artifacts like a fresh vendor run (patch-service prebuilt under --vendor-source auto, local build as fallback) and verifies against the existing ledger fingerprint; it no longer rebuilds .socket/vendor/state.json from lockfile references—those cases fail with vendor_ledger_missing and docs/rollback messages tell users to restore state.json from VCS. CI path filters point at vendored_backend/**; contract/README/CHANGELOG reflect the new semantics.

^{Reviewed by Cursor Bugbot for commit 5f8573b. Configure here.}

[Mikola Lysenko (mikolalysenko)]

install-proof (ubuntu-latest, 0.0.0-1) failed in vlt_pinned_matrix_agent_get_and_remove: the copy was Absent after get <uuid>. This PR didn't cause it. 5e5f5ed on release/v5-prerelease made get default to hosted mode and moved the agent-mode fixtures to --mode agent, but it missed the #[ignore]d real-toolchain suites. The vlt-compatibility workflow ran here only because this PR touches a file on its path filter.

I added the same fix to this PR in f206eba: --mode agent on the agent-mode get calls in e2e_vlt, e2e_npm, e2e_pypi, e2e_gem and e2e_safety_pnpm. It does nothing if the base branch fixes this separately. I couldn't run these suites locally because the sandbox can't reach registry.npmjs.org, so CI on f206eba is the check.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

coverage on e3717a0 fails only in e2e_redirect_cargo_build, in two tests: cargo_get_uuid_hosted_fresh_checkout_fetch and cargo_hosted_fresh_checkout_fetch_pulls_patched_crate_and_vex_verifies. This PR didn't cause it:

• The same two tests fail on the base branch. See CI run 36352437716 on release/v5-prerelease @ 8ae7dc3: coverage, plus test on ubuntu, macos and windows. They reproduce locally on a clean base checkout too.
• The failing assertion is step (3) of the hosted cargo VEX check (e2e_redirect_cargo_build.rs:829). An offline vex with the redirect ledger removed still attests the redirected crate: it exits 0, where the test expects 1 with record_unavailable. That is the hosted/VEX record lookup, which this PR doesn't touch (WS1 territory).
• No fix exists yet on any branch I can see, so there is nothing to port. I'm leaving it for WS1 or the base branch rather than widening this PR.

Many checks on e3717a0 are still running. The vlt install-proof legs that failed earlier now pass.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

v5 review at e3717a0: the shared vendored backend and removal of ledger re-synthesis are useful reductions. However, the new repair adapter changes behavior in two reproducible ways (inline).

I built this head and compared identical fixtures with the previous repair implementation on #279: offline repair regresses from success to failure; service-based repair reports success while changing both the lock and stored fingerprint. These should be fixed before landing.

The simplification that addresses both is to share artifact acquisition while making the operation policy explicit. Repair should retain an immutable expected artifact identity and preserve wiring; ordinary vendor may deliberately select new bytes and rewire. Sharing a general apply method without that distinction imports the wrong side effects.

Also return typed per-package outcomes from the backend. repair currently runs it into a scratch JSON-facing Envelope and interprets event codes back into execution state. Let human/JSON output consume the same result instead. Reuse the shared project context for ledger and format reads.

Validation: clean CLI build; two isolated npm repair reproductions, both compared against the prior implementation. Full workspace/toolchain matrix not rerun.

[Mikola Lysenko (mikolalysenko)]

Thanks for the review. Both P2s are fixed in 6b04d2f and pinned with your reproductions. Replies are on the inline threads.

On the design points in the summary:

• Explicit repair vs vendor policy. 6b04d2f is the minimal version: repair keeps the original entry as its immutable expected identity and undoes any wiring or ledger change for a candidate whose result isn't that artifact. Ordinary vendor still rewires. It isn't yet a policy flag on the backend.
• Typed per-package outcomes. Agreed that reading scratch-envelope event codes back into execution state is the weak spot. The proper fix is for vendor_records_reusing to return a Vec<(purl, PackageOutcome)> (applied / in-sync / rebuilt-artifact / refused{code, detail} / no-source, plus advisories), with Envelope events rendered from it for every command. That touches the whole engine loop and every vendored caller. I'd rather land it as its own commit on top of this PR than mix it into the regression fixes. Say if you want it in this PR.
• Shared project context for ledger and format reads. In the v5 plan that's WS6 (ProjectContext: crawl snapshot, lock set, ledgers). Repair would switch to it there, next to scan/vendor/vex/list/get.

Validation for 6b04d2f:

• clippy is clean;
• the repair and vendor suites pass, both as root and as a non-root user (the permission-gated persist-failure tests only run as non-root);
• the cargo, bun and composer build e2e suites pass.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

test (windows-latest) on 6b04d2f fails in two test targets: covgap_commands_scan_mod and e2e_redirect_cargo_build. Both also fail on release/v5-prerelease itself: its Windows job in run 36352437716 on 8ae7dc3 lists exactly these two targets.

This PR doesn't touch commands/scan/mod.rs beyond one re-export rename. I couldn't pull the per-test panic for covgap_commands_scan_mod: only the last 5000 lines of the job log can be fetched, and the panic is earlier than that. No fix exists on any branch I can see, so there is nothing to port.

For 6b04d2f's other jobs: in test on ubuntu and macos and in coverage, the only failing target is the hosted-cargo e2e_redirect_cargo_build pair covered above. I haven't gone through the rest of the matrix yet.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Restack request from the v5 coordinator (please act now).

#280 (ledger-free hosted) is now merged into release/v5-prerelease as 686e5fb4. Please merge origin/release/v5-prerelease into this branch, resolve the conflicts, get CI to "base-inherited reds only", and mark the PR ready for review (not draft) when done.

• Resolve toward v5 WS1+WS2: ledger-free hosted mode, upstream-restore rollback, vendor ejects hosted projects #280's model. There is no hosted ledger any more: rollback/remove/vendor restore upstream via patch::redirect::upstream. Don't reintroduce redirect-state.json writes or fragment-replay paths. Keep this PR's own intent on top.
• Landing order: WS5: one VendoredBackend for vendored apply/revert/repair; cut repair's ledger rebuild #283 → WS3: one lockfile model per ecosystem #281 → v5: remove setup (WS7) + patch UI streamlining (WS8) #279 → v5 WS4/WS6: one hosted engine for disk + memory; unified Ledgers view #282, then the design PRs (docs: v5 waste review + repacking-to-depscan design #286 docs, Add the NuGet vendoring design (docs only) #285/Vendored Maven and Gradle: design and prototype #287). After each one lands, the next ones merge the base again. Expect a small second round of conflicts.
• Base-inherited reds that aren't yours to fix: vlt install-proof (vlt_pinned_matrix_agent_get_and_remove; fixed on v5: remove setup (WS7) + patch UI streamlining (WS8) #279) and Windows covgap_commands_scan_mod.
• The repo only allows squash merges. Post a comment with your final head SHA and CI summary when you're ready.

[Mikola Lysenko (mikolalysenko)]

Owner decision (via coordinator): typed per-package outcomes can be a follow-up PR. The 6b04d2f fix (repair keeps the recorded identity) is enough to land. Priority now: merge the base (#280), resolve conflicts, get green, and mark ready.

[Mikola Lysenko (mikolalysenko)]

v5 coordinator: #283 is first in the landing order but isn't ready to land yet. On head 5f8573b:

• It's still a draft. Mark it ready for review once CI settles.
• CI on 5f8573b, from the 11:15 UTC push, is still running. Every failing check has to be base-inherited: vlt install-proof vlt_pinned_matrix_agent_get_and_remove, Windows covgap_commands_scan_mod, or a check that fails at the same job name on the base's latest run.

The base-merge requirement is met, since the branch contains the current release/v5-prerelease. The next hourly run will squash-merge the PR once it's out of draft and CI is done.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

vlt install-proof on 5f8573b fails in e2e_redirect_vlt_build, in 5 hosted tests: crlf_lock, idempotence, resave_crlf_rollback, resave_install_rollback and rollback_byte_exact. rollback restores the pin but then reports hosted_wiring_contested / patched_ref_unattributable on vlt-lock.json, and exits 1.

This comes from the base, not this PR:

• v5 WS1+WS2: ledger-free hosted mode, upstream-restore rollback, vendor ejects hosted projects #280's own last vlt-compatibility run (36383945937, on fc9a52a, the commit that was merged) fails the same 5 tests with the same hosted_wiring_contested error. It fails in the same install-proof jobs: 61 of them there.
• The failure is in the ledger-free hosted rollback path from v5 WS1+WS2: ledger-free hosted mode, upstream-restore rollback, vendor ejects hosted projects #280 (patch::redirect::upstream). This PR doesn't touch it.
• In the same jobs, e2e_vlt passes here, including vlt_pinned_matrix_agent_get_and_remove, so this PR also carries the fix for that other base-inherited red. e2e_vendor_vlt_build passes too.

No fix exists on any branch I can see, so there is nothing to port. I'll mark the PR ready once the rest of CI on 5f8573b settles, if it shows only base-inherited reds.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Ready to land. Head: 5f8573be9e21be26a248467277e71b8d05873db1, which contains release/v5-prerelease @ 686e5fb (#280). The PR is marked ready for review.

CI summary on 5f8573b. Every failure I've found is one of these three, each inherited from the base branch and explained in earlier comments:

• e2e_redirect_cargo_build: hosted cargo VEX (F77, being fixed separately). It fails the test jobs and coverage.
• Windows covgap_commands_scan_mod.
• vlt install-proof: 5 hosted-rollback tests in e2e_redirect_vlt_build, failing with hosted_wiring_contested, the same as v5 WS1+WS2: ledger-free hosted mode, upstream-restore rollback, vendor ejects hosted projects #280's run 36383945937. On this PR, e2e_vlt, e2e_vendor_vlt_build and get_and_remove pass.

Two macOS hosted-only jobs hit a runner network error: Poetry native (macos-latest, 1.8.5) and Bun native (macos-latest, 1.0.0). The vendored cases in those same jobs passed. I've re-run the Poetry job once. The Bun run is still queued, so I'll re-run its job once when the run finishes.

Review: both P2 threads are fixed in 6b04d2f and resolved. The owner decided typed outcomes can go in a follow-up.

Follow-ups (non-blocking):

1. Have vendor_records_reusing return typed per-package outcomes, so repair stops reading event codes out of a scratch Envelope.
2. Make the repair and vendor policy an explicit backend option: repair keeps the recorded identity, vendor rewires. 6b04d2f does this with a snapshot and undo instead.
3. Move repair onto the shared ProjectContext for ledger and lock reads (WS6).
4. The rustfmt debt on the base branch is untouched here, and CI doesn't check formatting.

---

Generated by Claude Code

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Repair snapshot uses unguarded reads
  • Replaced bare tokio::fs::read calls with read_regular_to_bytes in snapshot_wiring and undo_candidate to prevent FIFO blocking and properly handle symlink lockfiles.

Or push these changes by commenting:

@cursor push 89e28f4bcb

Preview (89e28f4bcb)

diff --git a/crates/socket-patch-cli/src/commands/vendored_backend/repair.rs b/crates/socket-patch-cli/src/commands/vendored_backend/repair.rs
--- a/crates/socket-patch-cli/src/commands/vendored_backend/repair.rs
+++ b/crates/socket-patch-cli/src/commands/vendored_backend/repair.rs
@@ -30,7 +30,7 @@
 use socket_patch_core::constants::SOCKET_DIR;
 use socket_patch_core::manifest::schema::{PatchManifest, PatchRecord};
 use socket_patch_core::patch::copy_tree::remove_tree;
-use socket_patch_core::utils::fs::read_regular_to_string;
+use socket_patch_core::utils::fs::{read_regular_to_bytes, read_regular_to_string};
 use socket_patch_core::utils::purl::normalize_purl;
 use socket_patch_core::vendor::{
     self, artifact_is_file_shaped, check_vendored_artifact, load_state, parse_vendor_path,
@@ -1058,14 +1058,10 @@
     let mut out = Vec::with_capacity(rels.len());
     for rel in rels {
         let path = cwd.join(&rel);
-        match tokio::fs::symlink_metadata(&path).await {
-            Ok(meta) if meta.is_file() => {
-                if let Ok(bytes) = tokio::fs::read(&path).await {
-                    out.push((path, Some(bytes)));
-                }
-            }
-            Ok(_) => {}
-            Err(_) => out.push((path, None)),
+        match read_regular_to_bytes(&path).await {
+            Ok(bytes) => out.push((path, Some(bytes))),
+            Err(e) if e.kind() == std::io::ErrorKind::NotFound => out.push((path, None)),
+            Err(_) => {}
         }
     }
     out
@@ -1088,7 +1084,7 @@
 async fn undo_candidate(cwd: &Path, c: &Candidate, snapshot: &[(PathBuf, Option<Vec<u8>>)]) {
     let wired: HashSet<PathBuf> = c.entry.wiring.iter().map(|w| cwd.join(&w.file)).collect();
     for (path, bytes) in snapshot.iter().filter(|(p, _)| wired.contains(p)) {
-        if tokio::fs::read(path).await.ok().as_ref() == bytes.as_ref() {
+        if read_regular_to_bytes(path).await.ok().as_ref() == bytes.as_ref() {
             continue;
         }
         match bytes {

_{You can send follow-ups to the cloud agent here.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 5f8573b. Configure here.}

[Mikola Lysenko (mikolalysenko)]

CI settled on 9d46742e69dda3459c95f371e0d37d523dbcee9a. The only failures are ones the base branch also has:

• CI: 38 of 39 jobs pass, 5 are skipped. The one failure is test (windows-latest), and only in covgap_commands_scan_mod.
• vlt compatibility: install-proof fails in e2e_redirect_vlt_build hosted rollback (hosted_wiring_contested), the same as v5 WS1+WS2: ledger-free hosted mode, upstream-restore rollback, vendor ejects hosted projects #280's own run.
• Other compatibility workflows: all pass (Poetry, Bun, npm, pnpm, PDM, Pipenv, Go, pin-check). The earlier macOS network flakes didn't come back.

Every review thread is resolved. Ready for the coordinator to squash-merge into release/v5-prerelease.

---

Generated by Claude Code