Defer pristine fetches the service makes moot

With the patch service on, `vendor` deferred the registry download
of a not-installed package only for cargo; npm, golang and composer
packages were downloaded and verified up front even when the
service's prebuilt artifact made the pristine copy unnecessary.

Those backends also ask the service first and read the pristine
tree only on a local-build fallback, so their download is now
deferred the same way. A package is deferred only when its fetch
would really download: the fetchers' pre-download refusals (a
foreign yarn berry cacheKey, a go module go fetches without a
proxy, a composer entry with no dist URL) are now one shared check
that both the fetch and the deferral use. pypi and gem keep the
up-front fetch, which their installed-variant probe reads.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WzdTEhubve9yWfqBE7vAsB

Author: claude

crates/socket-patch-cli/CLI_CONTRACT.md

@@ -127,7 +127,7 @@ For a **9.0 root lock**, the CLI ensures `pnpm-workspace.yaml` carries `trustLoc
 
 **Lockfile supplement (v3.4)**: `scan` discovery is no longer limited to installed trees. The project's lockfiles (`package-lock.json`/`npm-shrinkwrap.json`, `pnpm-lock.yaml` v9, `yarn.lock` classic + berry, `bun.lock`, `vlt-lock.json` (registry nodes, Socket-hosted pins included; vendored `file` nodes are left to the vendor ledger), `Cargo.lock`, `go.sum`, `composer.lock`, `Gemfile.lock`, `uv.lock`/`poetry.lock`/pinned `requirements.txt`) are inventoried and dependencies with NO installed copy join discovery — counts, the API lookup, the table (flagged ` [NOT INSTALLED]`, plus a stderr note), and the prune "scanned" set (a wiped node_modules no longer prunes lockfile-listed entries). JSON gains a top-level `lockfileOnlyPackages` count and an additive `notInstalled: true` on matching `packages[]` entries. `--apply` partitions lockfile-only patches out BEFORE download (calm `skipped`/`package_not_installed` records — never an error exit, never a manifest write); `--vendor` passes them through to the vendor engine's auto-fetch. Vendored-ledger entries likewise stay discoverable on a fresh clone (the committed artifact is the dependency). Global scans (`--global`) get no supplement. **Rush monorepos** (no root lockfile, `rush.json` present): the npm-lock inventory falls back to the Rush source-of-truth locks — `common/config/rush/pnpm-lock.yaml` plus every `common/config/subspaces/*/pnpm-lock.yaml` (`read_dir`-sorted, repo-relative paths preserved) — so a Rush repo's dependencies still join discovery. **Plug'n'Play layouts are an explicit refusal, not an empty inventory**: a `.pnp.*` loader means the npm packages are structurally unreachable in EVERY mode (under yarn PnP the installed-tree crawl is empty too — no `node_modules/`), so `scan` surfaces an additive top-level `warnings[]` array (`{code, detail}` objects, omitted when empty) carrying `yarn_pnp_unsupported` (same code as apply's refusal; remedy `yarn patch <pkg>`) or `pnpm_pnp_unsupported` (pnpm's `node-linker=pnp` twin; pnpm remedies), plus a stderr `Warning (<code>): …` line on the human path. Exit code and `status` are deliberately unchanged (exit 0 / `success` — the same posture as hosted refusals, which exit 0 with `redirected: 0`); the warning is the machine-readable signal that nothing was checked. Pinned by `tests/e2e_safety_yarn_pnp.rs`.
 
-**Vendor auto-fetch (v3.4)**: `vendor`/`scan --vendor` no longer fail on lockfile-resolved packages with no installed copy. Already-vendored purls stage from their committed artifact (sha256-verified against the vendor ledger — a vlt directory artifact against its file inventory, which leaves out the links vlt creates inside it; offline-safe) when the ledger entry is at the manifest record's patch uuid; a superseding uuid fetches the pristine package instead, since the older artifact holds the older patch's bytes. Otherwise the pristine artifact is fetched per the lockfile resolution and verified against the lock's recorded integrity FAIL-CLOSED before any write: npm SRI (or yarn classic's sha1 fragment; for vlt the registry node's slot [2]), yarn berry's cache-zip checksum (rebuilt from the fetched tarball; cacheKey 10c0 only), Cargo.lock sha256 over the .crate, go.sum `h1:` dirhash over the module zip, composer `dist.shasum` (sha1), Gemfile.lock `CHECKSUMS` sha256, uv.lock wheel sha256 (pure `py3-none-any` wheels only). Entries the lock cannot verify are NEVER fetched (`vendor_fetch_unverifiable` warning + the calm `package_not_installed` skip). Registry bases honor `SOCKET_NPM_REGISTRY`, `SOCKET_CRATES_REGISTRY`, `SOCKET_GOPROXY` (else `GOPROXY`, `GONOPROXY` and `GOPRIVATE` the way go reads them — see the env table); npm/yarn/composer/gem/uv lock-recorded URLs are used verbatim. `--offline` refuses the fetch with the calm skip (the detail names the lockfile resolution). The fetch stages into a private tempdir — the project tree is never touched. **Deferred fetch (v5.0):** a purl the vendor ledger already covers (its entry records the record's patch uuid and the committed artifact is on disk — a file artifact only while it hashes to the ledger's `sha256`; not under `--force`), and a lockfile-only cargo crate the registry could fetch and verify (a crates.io `Cargo.lock` entry with a checksum, or the pre-vendor resolution the ledger recovers) while the patch service is enabled, are NOT downloaded up front: the fetch runs only if the backend reaches a branch that reads the pristine tree (a drifted committed copy rebuilt locally, a service miss). An in-sync re-run therefore makes no registry request, reports no `vendor_fetched_missing`, and succeeds with no network or under `--offline`. A deferred fetch that does run records its `vendor_fetched_missing` just ahead of the package's own event; one that fails, is unverifiable, or is refused by `--offline` reports the same events the up-front fetch would have. A git, path or custom-registry cargo crate is never deferred, so it keeps `vendor_fetch_unverifiable` + `package_not_installed` and is never vendored from the service's crates.io build. **Gem, local build only** (`--vendor-source build`, or no service config): a not-installed gem the lock can verify (bundler >= 2.6 `CHECKSUMS`) and no ledger entry covers is refused `gem_spec_missing` (`failed`, the backend's own detail) BEFORE any download — a downloaded `.gem` carries no eval-able stub gemspec, so a local build can never vendor it; no `vendor_fetched_missing` precedes it, and a refusal the backend would have reached first on the fetched copy reports as `gem_spec_missing` too. Not under `--dry-run`, which still fetches and previews the gem (`vendor_fetched_missing` + `verified`).
+**Vendor auto-fetch (v3.4)**: `vendor`/`scan --vendor` no longer fail on lockfile-resolved packages with no installed copy. Already-vendored purls stage from their committed artifact (sha256-verified against the vendor ledger — a vlt directory artifact against its file inventory, which leaves out the links vlt creates inside it; offline-safe) when the ledger entry is at the manifest record's patch uuid; a superseding uuid fetches the pristine package instead, since the older artifact holds the older patch's bytes. Otherwise the pristine artifact is fetched per the lockfile resolution and verified against the lock's recorded integrity FAIL-CLOSED before any write: npm SRI (or yarn classic's sha1 fragment; for vlt the registry node's slot [2]), yarn berry's cache-zip checksum (rebuilt from the fetched tarball; cacheKey 10c0 only), Cargo.lock sha256 over the .crate, go.sum `h1:` dirhash over the module zip, composer `dist.shasum` (sha1), Gemfile.lock `CHECKSUMS` sha256, uv.lock wheel sha256 (pure `py3-none-any` wheels only). Entries the lock cannot verify are NEVER fetched (`vendor_fetch_unverifiable` warning + the calm `package_not_installed` skip). Registry bases honor `SOCKET_NPM_REGISTRY`, `SOCKET_CRATES_REGISTRY`, `SOCKET_GOPROXY` (else `GOPROXY`, `GONOPROXY` and `GOPRIVATE` the way go reads them — see the env table); npm/yarn/composer/gem/uv lock-recorded URLs are used verbatim. `--offline` refuses the fetch with the calm skip (the detail names the lockfile resolution). The fetch stages into a private tempdir — the project tree is never touched. **Deferred fetch (v5.0):** a purl the vendor ledger already covers (its entry records the record's patch uuid and the committed artifact is on disk — a file artifact only while it hashes to the ledger's `sha256`; not under `--force`), and a lockfile-only npm, cargo, golang or composer package the registry would fetch and verify (a lock entry with an integrity, or the pre-vendor resolution the ledger recovers, that none of its fetcher's pre-download refusals applies to: a yarn berry cacheKey other than 10c0, a go module go fetches without a proxy, a composer entry with no dist URL) while the patch service is enabled, are NOT downloaded up front (those backends ask the service first and read the pristine tree only on a local-build fallback; pypi and gem keep the up-front fetch, which their installed-variant probe reads): the fetch runs only if the backend reaches a branch that reads the pristine tree (a drifted committed copy rebuilt locally, a service miss). An in-sync re-run therefore makes no registry request, reports no `vendor_fetched_missing`, and succeeds with no network or under `--offline`. A deferred fetch that does run records its `vendor_fetched_missing` just ahead of the package's own event; one that fails, is unverifiable, or is refused by `--offline` reports the same events the up-front fetch would have. A package whose fetch would be refused is never deferred (a git, path or custom-registry cargo crate, say), so it keeps `vendor_fetch_unverifiable` + `package_not_installed` and is never vendored from the service's registry build. **Gem, local build only** (`--vendor-source build`, or no service config): a not-installed gem the lock can verify (bundler >= 2.6 `CHECKSUMS`) and no ledger entry covers is refused `gem_spec_missing` (`failed`, the backend's own detail) BEFORE any download — a downloaded `.gem` carries no eval-able stub gemspec, so a local build can never vendor it; no `vendor_fetched_missing` precedes it, and a refusal the backend would have reached first on the fetched copy reports as `gem_spec_missing` too. Not under `--dry-run`, which still fetches and previews the gem (`vendor_fetched_missing` + `verified`).
 
 **Vendored write durability (v5.0)**: every write is atomic (stage + rename), but only the durable commit points — lockfiles, `go.mod`/`go.sum`, `pom.xml`, `nuget.config`, `package.json`, `pnpm-workspace.yaml`, `.cargo/config.toml`, the Python/Ruby manifests, `.socket/vendor/state.json` and `redirect-state.json` — are fsynced on write. The content-verified artifacts under `.socket/vendor/<eco>/<uuid>/` (patched copies, packed/rebuilt archives and sidecars, markers) are written without an fsync and made durable by one barrier (file + directory fsync, one `F_FULLFSYNC` per device on macOS) ahead of the next commit point — and, for an artifact rebuilt in place that no commit point follows, at the end of the vendored run's commit and when the command releases the apply lock — so a crash can only lose an artifact that no durable commit point names yet, which the next run rebuilds.
 
@@ -1315,7 +1315,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified
 | `vendor_override_conflict` | `failed`        | vendor (pnpm/yarn-berry): a user-authored override/resolution for the package already exists. |
 | `vendor_integrity_unverified` | `skipped` (warning) | vendor (pipenv): the lockfile format does not hash-check file entries; the committed wheel bytes are the protection. |
 | `vendor_content_mismatch_overwritten` | `skipped` (warning) | vendor: a staged file matched NEITHER beforeHash nor afterHash (patch built against different bytes, or local edits); the stage was overwritten with the verified patched content and the vendor succeeded. |
-| `vendor_fetched_missing` | `skipped` (warning) | vendor: the package was not installed; its pristine artifact was fetched per the lockfile resolution (or staged from the committed vendor artifact), integrity-verified, and vendored — the project tree was not touched. Not emitted when no fetch happened: an in-sync re-run of a ledger-covered purl, or a cargo crate the patch service served (see Vendor auto-fetch § Deferred fetch). For `poetry.lock` (which records hashes but no URLs) the pure-Python wheel's sha256 selects the file through PyPI's JSON API (`SOCKET_PYPI_JSON_API` overrides the endpoint); Poetry 0.12's bare `[metadata.hashes]` names no wheel, so those locks still need an installed copy (`vendor_fetch_unverifiable`). |
+| `vendor_fetched_missing` | `skipped` (warning) | vendor: the package was not installed; its pristine artifact was fetched per the lockfile resolution (or staged from the committed vendor artifact), integrity-verified, and vendored — the project tree was not touched. Not emitted when no fetch happened: an in-sync re-run of a ledger-covered purl, or an npm, cargo, golang or composer package the patch service served (see Vendor auto-fetch § Deferred fetch). For `poetry.lock` (which records hashes but no URLs) the pure-Python wheel's sha256 selects the file through PyPI's JSON API (`SOCKET_PYPI_JSON_API` overrides the endpoint); Poetry 0.12's bare `[metadata.hashes]` names no wheel, so those locks still need an installed copy (`vendor_fetch_unverifiable`). |
 | `vendor_fetch_failed` | `failed` | vendor: the lockfile-resolved fetch was attempted and failed (HTTP error, size cap, integrity mismatch, or a PRESENT-but-corrupt committed artifact — pointed at `socket-patch repair`). A MISSING committed artifact no longer lands here: it falls through to the ledger-recovered registry fetch. Suppresses the duplicate `package_not_installed` skip. |
 | `vendor_fetch_unverifiable` | `skipped` (warning) | vendor: the lockfile records no usable integrity for the missing package; nothing was fetched (fail-closed) and the `package_not_installed` skip follows. Unchanged for gems by the build-mode `gem_spec_missing` refusal below, which fires only where a fetch WOULD have run. |
 | `vendor_vlt_transitive_unsupported` | `failed` | vendor (vlt): the target has an inbound edge from another package in `vlt-lock.json` (the detail names it); vendored mode rewires only direct dependencies of the root or a workspace member, because vlt silently reverts transitive lock surgery. Remedy: `--mode hosted`. Refused before any download or write, dry runs included (`would_refuse`). |

crates/socket-patch-cli/src/commands/vendor.rs

@@ -1697,6 +1697,45 @@ pub(crate) async fn pristine_fetch_is_verifiable(
     }
 }
 
+/// Whether [`fetch_pristine_package`] would reach the download for this
+/// purl: the same entry choice (see [`pristine_fetch_is_verifiable`]), and
+/// none of the refusals its fetcher raises before the first request (a
+/// foreign yarn berry cacheKey, a go module go would not fetch through a
+/// proxy, a composer entry with no dist URL). Deferring a fetch that would
+/// refuse `vendor_fetch_unverifiable` behind the patch service would
+/// instead vendor the patch over a package it does not describe.
+async fn pristine_fetch_reaches_download(
+    project_root: &Path,
+    inventory: &[lock_inventory::LockfileEntry],
+    purl: &str,
+    ledger_entry: Option<&VendorEntry>,
+) -> bool {
+    let entry = match lock_inventory::lookup(inventory, purl)
+        .filter(|e| e.integrity != lock_inventory::LockIntegrity::None)
+    {
+        Some(e) => e.clone(),
+        None => match ledger_entry {
+            Some(le) => match lock_inventory::recover_lock_entry(project_root, le).await {
+                Ok(e) => e,
+                Err(_) => return false,
+            },
+            None => return false,
+        },
+    };
+    registry_fetch::refusal_before_download(&entry).is_none()
+}
+
+/// The ecosystems whose backend asks the patch service before it reads the
+/// pristine tree, and reads it only on a local-build fallback. pypi and gem
+/// read it earlier, in the loop's installed-variant probe; nuget and maven
+/// have no registry fetch.
+fn backend_reads_pristine_only_on_fallback(purl: &str) -> bool {
+    matches!(
+        Ecosystem::from_purl(purl),
+        Some(Ecosystem::Npm | Ecosystem::Cargo | Ecosystem::Golang | Ecosystem::Composer)
+    )
+}
+
 /// The purls among `purls` with an installed copy, found exactly as the
 /// vendor loop finds them: the qualified-aware resolver
 /// ([`find_packages_for_rollback_reusing`]), then the npm `package.json`
@@ -2376,11 +2415,12 @@ pub(crate) async fn vendor_records_reusing(
             //    backend's in-sync hot path answers it from the committed
             //    bytes alone, so a re-run needs no network. `--force` may
             //    rebuild anyway, so it keeps the eager fetch.
-            //  * a cargo crate the patch service can serve: the backend reads
-            //    the pristine tree only if it falls back to the local build.
-            //    Only a crate the registry ladder COULD fetch (see
-            //    `pristine_fetch_is_verifiable`) — a git, path or
-            //    custom-registry crate keeps the eager rung, whose
+            //  * a package the patch service can serve, in an ecosystem whose
+            //    backend reads the pristine tree only if it falls back to the
+            //    local build (`backend_reads_pristine_only_on_fallback`).
+            //    Only one the registry ladder would really download (see
+            //    `pristine_fetch_reaches_download`) — a git, path or
+            //    custom-registry crate, say, keeps the eager rung, whose
             //    `vendor_fetch_unverifiable` refusal keeps a crates.io patch
             //    off it.
             //
@@ -2400,10 +2440,10 @@ pub(crate) async fn vendor_records_reusing(
                         }
                         None => false,
                     };
-                let cargo_via_service = service_enabled
+                let via_service = service_enabled
                     && matches!(rung, MissingRung::Fetch)
-                    && Ecosystem::from_purl(purl) == Some(Ecosystem::Cargo)
-                    && pristine_fetch_is_verifiable(
+                    && backend_reads_pristine_only_on_fallback(purl)
+                    && pristine_fetch_reaches_download(
                         &common.cwd,
                         inventory
                             .get_or_init(|| lock_inventory::inventory_project(&common.cwd))
@@ -2412,7 +2452,7 @@ pub(crate) async fn vendor_records_reusing(
                         lookup_entry(&state.entries, purl),
                     )
                     .await;
-                if covered || cargo_via_service {
+                if covered || via_service {
                     *rung = MissingRung::Deferred;
                 }
             }