diff --git a/CHANGELOG.md b/CHANGELOG.md index 9163f2bf..6de7b16c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -345,6 +345,26 @@ into the new version's section — see docs/releasing.md. ### Fixed +- **A vendoring-service outage no longer re-vendors packages.** An npm + re-run (every lock flavor, `bun.lockb` included) re-acquired its tarball + from whichever source answered — the service's prebuilt, or a local pack + with different bytes — so an outage or its recovery rewrote the lock's + integrity and the committed tarball and reported `applied`. A re-run now + keeps the committed artifact whenever the vendor ledger vouches for it + (uuid-bound path, no symlink, sha256 + size equal to the ledger, a + canonical archive an installer extracts exactly as decoded, every + patched file verified from the same bytes) and is `already_vendored` + with no service request, in every `--vendor-source` mode (including + `service` + `--offline`, as cargo and composer already did; golang now + matches). A pypi re-scan after a relock re-wires the committed wheel + instead of pinning a new sha, the PDM partial-relock guard holds + whichever source built the wheel, a wiring failure no longer deletes a + committed wheel, and a missing prebuilt wheel during an outage now says + to wait for the service. `--dry-run` previews the same reuse, so it no + longer predicts a `service` + `--offline` refusal the real run does not + have. Transient service failures (network, timeouts, 429, 5xx) are + retried with backoff, each attempt is time-bounded, and after two + consecutive exhausted fetches the run stops calling the service. - **Terminal output is clean on every command.** Progress lines no longer leave stale text behind (`scan` printed e.g. `Found 7 patches for 1 packagesatch 7/7)`) or run into warnings printed while they are active. diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 39e39453..736e04ae 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -481,6 +481,21 @@ per service outcome: | `--offline` | local build | refuse (`vendor_service_offline_conflict`) | | no API client configured (library callers of the vendor engine; the CLI always configures one) | local build | refuse (`vendor_prebuilt_required`) | +`--vendor-source` governs ACQUISITION, not reuse: a re-run whose committed artifact the ledger +vouches for (npm tarball / pypi wheel: path under this patch uuid, no symlink, whole-file sha256 and +size equal to the ledger, every afterHash verified from the same bytes; the dir-shaped ecosystems: +the wired copy's afterHashes) keeps it in every mode — no service request, no local build, no +rewrite — whichever source built it. So a service outage (or its recovery) never re-vendors an +already-vendored package: the re-run is `already_vendored`, including under `service` + +`--offline`, and `build` does not rebuild a service-built artifact (delete the uuid dir to force +a rebuild). The ledger records no provenance, so `service` cannot tell a locally built committed +artifact from a prebuilt one; it keeps what verifies. A lock that drifted off a verified committed +artifact (a relock, a hand revert) is re-wired to those exact bytes (pypi re-scans report the +Verbose `vendor_artifact_reused`). Service round trips are retried on transport errors and +429/500/502/503/504 (3 attempts, exponential backoff with jitter, `Retry-After` honored, 4s cap); +after 2 consecutive exhausted fetches the rest of the run skips the service (`auto` builds +locally, `service` refuses). + **golang service leg staging (v5.0)**: the module zip is downloaded, extracted and `h1:`-verified in a `.socket-stage` sibling and swapped into place only afterwards; a failed re-download of a WIRED, present copy keeps the copy and its `replace` directive (previously both were torn down), while a missing copy still drops the dangling directive. Coverage today: **npm** (all lock flavors), **pypi** (wheel — sdist falls back / refuses), **cargo** @@ -1125,6 +1140,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `vendor_fetch_unverifiable` | `skipped` (warning) | vendor: the lockfile records no usable integrity for the missing package; nothing was fetched (fail-closed) and the `package_not_installed` skip follows. | | `vendor_artifact_missing` | `skipped` (warning) / `failed` | vendor: the committed artifact is gone — the registry resolution is recovered from the ledger and the artifact rebuilt (warning); repair `--offline` with no local source surfaces it as the per-entry failure instead. | | `vendor_artifact_corrupt` | `failed` | repair `--offline`: the committed artifact fails verification (member afterHashes or the ledger's whole-file sha256) and no local source can rebuild it. Online repairs rebuild instead. | +| `vendor_artifact_reused` | `skipped` (verbose note) | vendor / scan `--vendor` (pypi): the wiring was dropped by a relock but the committed wheel the ledger vouches for verified, so it was re-wired as-is — no service download, no rebuild; the lock pins the first run's sha again. | | `vendor_artifact_rebuilt` | `skipped` (warning) | vendor / scan `--vendor`: a wired-but-missing/stale artifact was rebuilt in place. The lockfiles are untouched, except that nuget re-pins `packages.lock.json` to the rebuilt bytes. gem/maven/nuget: the package's event is `applied` (also for a rebuild from the patch service), and the ledger entry's artifact fingerprint (gem `fileInventory`, maven/nuget `sha256` + `size`, and the nuget lock pin) is refreshed to the rebuilt bytes, and its wiring records are kept unchanged, so `--revert` still restores the pre-vendor files. A rebuild whose ledger has no entry for the package, or only one from another patch uuid, records none. cargo/composer/gem rebuilds honour `--vendor-source` like a fresh vendor (`service` downloads the prebuilt artifact and refuses when it cannot). Other ecosystems leave the ledger entry untouched. (Under `repair` the `rebuilt` event carries this signal.) | | `vendor_artifact_rebuild_failed` | `failed` | repair: the rebuild ran but the result failed verification against the recorded fingerprint (e.g. an edited state.json sha); the unverifiable artifact was removed. | | `vendor_artifact_unrepairable` | `failed` | repair: no verifiable pristine source exists (not installed + lockfile rewired + no recoverable ledger fragment), the wheel is platform-locked with no installed copy, or the ledger entry itself cannot be trusted. | diff --git a/crates/socket-patch-cli/src/commands/vendor.rs b/crates/socket-patch-cli/src/commands/vendor.rs index c02e1e4f..5de59bde 100644 --- a/crates/socket-patch-cli/src/commands/vendor.rs +++ b/crates/socket-patch-cli/src/commands/vendor.rs @@ -368,8 +368,9 @@ enum AdvisoryTier { fn advisory_tier(code: &str) -> AdvisoryTier { match code { - // Every successful service vendor emits this, one per package. - "vendor_prebuilt_downloaded" => AdvisoryTier::Verbose, + // Every successful service vendor emits this, one per package; a + // relock re-scan that re-wires the committed wheel emits the other. + "vendor_prebuilt_downloaded" | "vendor_artifact_reused" => AdvisoryTier::Verbose, // The run did what was asked; these explain how. "vendor_fetched_missing" | "vendor_would_revert_redirect" @@ -4208,6 +4209,11 @@ mod ui_format_tests { ), Some("Note: vendored x from the service".to_string()) ); + assert_eq!(format_advisory("vendor_artifact_reused", "r", false), None); + assert_eq!( + format_advisory("vendor_artifact_reused", "re-wired x", true), + Some("Note: re-wired x".to_string()) + ); assert_eq!( format_advisory("vendor_fetched_missing", "fetched", false), Some("Note: fetched".to_string()) diff --git a/crates/socket-patch-cli/tests/covgap_commands_vendor.rs b/crates/socket-patch-cli/tests/covgap_commands_vendor.rs index a948b005..b837f058 100644 --- a/crates/socket-patch-cli/tests/covgap_commands_vendor.rs +++ b/crates/socket-patch-cli/tests/covgap_commands_vendor.rs @@ -1358,3 +1358,433 @@ fn human_classic_migration_risk_prints_stderr_warning() { "the run-level advisory prints for humans: {stderr}" ); } + +// ─────────────── service outage / source-flip idempotence ─────────────── +// +// A re-run whose committed artifact the ledger vouches for is +// `already_vendored` whichever source built it and whatever the service +// answers now: exit 0, the lock byte-identical, no service request. + +const PACKAGE_PATH: &str = "/v0/orgs/acme/patches/package"; + +/// `vendor --json` against the mock service at `uri` (authenticated, org +/// `acme`, `--vendor-url` pointed at the mock too). +fn vendor_via_service(root: &Path, uri: &str) -> (i32, Value, String) { + let args = [ + "vendor", + "--json", + "--cwd", + root.to_str().unwrap(), + "--api-url", + uri, + "--vendor-url", + uri, + "--api-token", + "sktsec_placeholder_value_for_tests_api", + "--org", + "acme", + "--lock-timeout", + "5", + ]; + let (code, stdout, stderr) = run_cli(root, &args, &[]); + let env: Value = serde_json::from_str(&stdout).unwrap_or_else(|e| { + panic!("vendor --json must emit an envelope: {e}\nstdout:\n{stdout}\nstderr:\n{stderr}") + }); + (code, env, stderr) +} + +fn regzip(tgz: &[u8]) -> Vec { + use std::io::{Read as _, Write as _}; + let mut raw = Vec::new(); + flate2::read::GzDecoder::new(tgz) + .read_to_end(&mut raw) + .unwrap(); + let mut enc = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::fast()); + enc.write_all(&raw).unwrap(); + let out = enc.finish().unwrap(); + assert_ne!(out, tgz); + out +} + +async fn mount_granted_artifact(server: &MockServer, leaf: &str, bytes: &[u8]) { + let serve = format!("/serve/{UUID}/{leaf}"); + let url = format!("{}{serve}", server.uri()); + Mock::given(method("POST")) + .and(path(PACKAGE_PATH)) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "results": { UUID: { "status": "granted", "url": url, + "artifacts": [{ "kind": "tarball", "url": url, + "integrity": { "sha512": sri_of(bytes) } }] } } + }))) + .mount(server) + .await; + Mock::given(method("GET")) + .and(path(serve)) + .respond_with(ResponseTemplate::new(200).set_body_bytes(bytes.to_vec())) + .mount(server) + .await; +} + +async fn mount_outage(server: &MockServer) { + server.reset().await; + Mock::given(method("POST")) + .and(path(PACKAGE_PATH)) + .respond_with(ResponseTemplate::new(503).set_body_string("upstream unavailable")) + .mount(server) + .await; +} + +async fn package_posts(server: &MockServer) -> usize { + server + .received_requests() + .await + .unwrap() + .iter() + .filter(|r| r.url.path() == PACKAGE_PATH) + .count() +} + +/// The run-2 contract: exit 0, applied 0, skipped 1, exactly one +/// `already_vendored` event, no outage advisory. +fn assert_already_vendored(code: i32, env: &Value, stderr: &str) { + assert_eq!(code, 0, "{env:#}\n{stderr}"); + assert_eq!(env["summary"]["applied"], 0, "{env:#}"); + assert_eq!(env["summary"]["skipped"], 1, "{env:#}"); + let in_sync = events(env) + .iter() + .filter(|e| e["errorCode"] == "already_vendored") + .count(); + assert_eq!(in_sync, 1, "{env:#}"); + assert!( + events(env) + .iter() + .all(|e| e["errorCode"] != "vendor_prebuilt_unavailable"), + "{env:#}" + ); +} + +/// Swap an npm fixture's package-lock for a bun.lock project. +fn to_bun(fx: &NpmFixture) { + std::fs::remove_file(fx.lock_path()).unwrap(); + std::fs::write( + fx.root().join("package.json"), + "{\n \"name\": \"bn3-lockonly\",\n \"version\": \"1.0.0\",\n \"dependencies\": {\n \"left-pad\": \"1.3.0\"\n }\n}\n", + ) + .unwrap(); + std::fs::write( + fx.root().join("bun.lock"), + r#"{ + "lockfileVersion": 1, + "configVersion": 1, + "workspaces": { + "": { + "name": "bn3-lockonly", + "dependencies": { + "left-pad": "1.3.0", + }, + }, + }, + "packages": { + "left-pad": ["left-pad@1.3.0", "", {}, "sha512-XI5MPzVNApjAyhQzphX8BkmKsKUxD4LdyK24iZeQGinBN9yTQT3bFlCBy/aVx2HrNcqQGsdot8ghrjyrvMCoEA=="], + } +} +"#, + ) + .unwrap(); +} + +/// `(fixture, lock file name)` for a flavor. +fn flavor_fixture(bun: bool) -> (NpmFixture, &'static str) { + let fx = npm_fixture(); + if bun { + to_bun(&fx); + (fx, "bun.lock") + } else { + (fx, "package-lock.json") + } +} + +/// The service's prebuilt artifact: the local build's members, re-encoded. +fn prebuilt_for(bun: bool) -> Vec { + let (probe, _) = flavor_fixture(bun); + let (code, stdout, stderr) = run_cli( + probe.root(), + &[ + "vendor", + "--json", + "--offline", + "--cwd", + probe.root().to_str().unwrap(), + ], + &[], + ); + assert_eq!(code, 0, "{stdout}\n{stderr}"); + regzip(&std::fs::read(probe.tgz_path()).unwrap()) +} + +async fn service_then_outage(bun: bool) { + let alt = prebuilt_for(bun); + let (fx, lock) = flavor_fixture(bun); + let server = MockServer::start().await; + mount_granted_artifact(&server, "left-pad-1.3.0.tgz", &alt).await; + let (code, env, stderr) = vendor_via_service(fx.root(), &server.uri()); + assert_eq!(code, 0, "{env:#}\n{stderr}"); + assert_eq!(env["summary"]["applied"], 1, "{env:#}"); + assert_eq!( + std::fs::read(fx.tgz_path()).unwrap(), + alt, + "run 1 used the service" + ); + let lock1 = std::fs::read(fx.root().join(lock)).unwrap(); + + mount_outage(&server).await; + let (code, env, stderr) = vendor_via_service(fx.root(), &server.uri()); + assert_already_vendored(code, &env, &stderr); + assert_eq!( + std::fs::read(fx.root().join(lock)).unwrap(), + lock1, + "{lock} unchanged" + ); + assert_eq!(std::fs::read(fx.tgz_path()).unwrap(), alt); + assert_eq!( + package_posts(&server).await, + 0, + "no service request on the re-run" + ); +} + +async fn outage_then_service(bun: bool) { + let alt = prebuilt_for(bun); + let (fx, lock) = flavor_fixture(bun); + let server = MockServer::start().await; + mount_outage(&server).await; + let (code, env, stderr) = vendor_via_service(fx.root(), &server.uri()); + assert_eq!(code, 0, "{env:#}\n{stderr}"); + assert_eq!(env["summary"]["applied"], 1, "{env:#}"); + assert!( + events(&env) + .iter() + .any(|e| e["errorCode"] == "vendor_prebuilt_unavailable"), + "run 1 fell back to a local build: {env:#}" + ); + let lock1 = std::fs::read(fx.root().join(lock)).unwrap(); + let tgz1 = std::fs::read(fx.tgz_path()).unwrap(); + + server.reset().await; + mount_granted_artifact(&server, "left-pad-1.3.0.tgz", &alt).await; + let (code, env, stderr) = vendor_via_service(fx.root(), &server.uri()); + assert_already_vendored(code, &env, &stderr); + assert_eq!( + std::fs::read(fx.root().join(lock)).unwrap(), + lock1, + "{lock} unchanged" + ); + assert_eq!(std::fs::read(fx.tgz_path()).unwrap(), tgz1); + assert_eq!(package_posts(&server).await, 0); +} + +#[tokio::test(flavor = "multi_thread")] +async fn npm_service_then_outage_rerun_is_already_vendored() { + service_then_outage(false).await; +} + +#[tokio::test(flavor = "multi_thread")] +async fn npm_outage_then_service_rerun_is_already_vendored() { + outage_then_service(false).await; +} + +#[tokio::test(flavor = "multi_thread")] +async fn bun_lock_service_then_outage_rerun_is_already_vendored() { + service_then_outage(true).await; +} + +#[tokio::test(flavor = "multi_thread")] +async fn bun_lock_outage_then_service_rerun_is_already_vendored() { + outage_then_service(true).await; +} + +const PDM_REGISTRY_LOCK: &str = r#"# This file is @generated by PDM. +# It is not intended for manual editing. + +[metadata] +groups = ["default"] +strategy = ["inherit_metadata"] +lock_version = "4.5.0" +content_hash = "sha256:d49d286986c5de41ec9879b6d710389b0be11cd096d883c069123b489ac6e6ea" + +[[metadata.targets]] +requires_python = "==3.14.*" + +[[package]] +name = "six" +version = "1.16.0" +requires_python = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*" +summary = "Python 2 and 3 compatibility utilities" +groups = ["default"] +files = [ + {file = "six-1.16.0-py2.py3-none-any.whl", hash = "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"}, + {file = "six-1.16.0.tar.gz", hash = "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926"}, +] +"#; +const SIX_WHEEL: &str = "six-1.16.0-py2.py3-none-any.whl"; + +/// A PDM project: pdm.lock pinning registry six, six installed in a +/// project `.venv`, and the manifest + blob for a patch to `six.py`. +fn pdm_fixture() -> tempfile::TempDir { + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + const ORIG: &[u8] = b"# six, original\n"; + const PATCHED: &[u8] = b"# six, patched\n"; + std::fs::write(root.join("pdm.lock"), PDM_REGISTRY_LOCK).unwrap(); + let sp = if cfg!(windows) { + root.join(".venv/Lib/site-packages") + } else { + root.join(".venv/lib/python3.12/site-packages") + }; + let di = sp.join("six-1.16.0.dist-info"); + std::fs::create_dir_all(&di).unwrap(); + std::fs::write(sp.join("six.py"), ORIG).unwrap(); + std::fs::write( + di.join("METADATA"), + "Metadata-Version: 2.1\nName: six\nVersion: 1.16.0\n\nbody\n", + ) + .unwrap(); + std::fs::write( + di.join("WHEEL"), + "Wheel-Version: 1.0\nRoot-Is-Purelib: true\nTag: py2-none-any\nTag: py3-none-any\n", + ) + .unwrap(); + std::fs::write( + di.join("RECORD"), + "six.py,sha256=AAAA,20\nsix-1.16.0.dist-info/METADATA,,\nsix-1.16.0.dist-info/WHEEL,,\nsix-1.16.0.dist-info/RECORD,,\n", + ) + .unwrap(); + let before = compute_git_sha256_from_bytes(ORIG); + let after = compute_git_sha256_from_bytes(PATCHED); + let socket = root.join(".socket"); + std::fs::create_dir_all(socket.join("blobs")).unwrap(); + std::fs::write(socket.join("blobs").join(&after), PATCHED).unwrap(); + let manifest = json!({ "patches": { "pkg:pypi/six@1.16.0": { + "uuid": UUID, + "exportedAt": "2026-01-01T00:00:00Z", + "files": { "six.py": { "beforeHash": before, "afterHash": after } }, + "vulnerabilities": {}, + "description": "synthetic pdm outage test patch", + "license": "MIT", + "tier": "free" + } } }); + std::fs::write( + socket.join("manifest.json"), + serde_json::to_vec_pretty(&manifest).unwrap(), + ) + .unwrap(); + tmp +} + +fn wheel_path(root: &Path) -> PathBuf { + root.join(format!(".socket/vendor/pypi/{UUID}/{SIX_WHEEL}")) +} + +/// The same wheel members, re-encoded (stored): the service's prebuilt. +fn rezip(whl: &[u8]) -> Vec { + use std::io::{Read as _, Write as _}; + let mut src = zip::ZipArchive::new(std::io::Cursor::new(whl)).unwrap(); + let mut out = zip::ZipWriter::new(std::io::Cursor::new(Vec::new())); + let opts = + zip::write::SimpleFileOptions::default().compression_method(zip::CompressionMethod::Stored); + for i in 0..src.len() { + let mut entry = src.by_index(i).unwrap(); + let mut bytes = Vec::new(); + entry.read_to_end(&mut bytes).unwrap(); + out.start_file(entry.name().to_string(), opts).unwrap(); + out.write_all(&bytes).unwrap(); + } + let alt = out.finish().unwrap().into_inner(); + assert_ne!(alt, whl); + alt +} + +/// PDM relock twin (P1 end to end): vendor from the service, `pdm lock` +/// restores the registry unit, re-vendor during an outage — the committed +/// wheel is re-wired (the service sha, no request), and `rollback` then +/// restores the relocked bytes exactly. +#[tokio::test(flavor = "multi_thread")] +async fn pdm_relock_rescan_under_outage_rewires_the_committed_wheel() { + use sha2::{Digest as _, Sha256}; + let probe = pdm_fixture(); + let (code, stdout, stderr) = run_cli( + probe.path(), + &[ + "vendor", + "--json", + "--offline", + "--cwd", + probe.path().to_str().unwrap(), + ], + &[], + ); + assert_eq!(code, 0, "{stdout}\n{stderr}"); + let alt = rezip(&std::fs::read(wheel_path(probe.path())).unwrap()); + let alt_sha = hex::encode(Sha256::digest(&alt)); + + let tmp = pdm_fixture(); + let root = tmp.path(); + let server = MockServer::start().await; + mount_granted_artifact(&server, SIX_WHEEL, &alt).await; + let (code, env, stderr) = vendor_via_service(root, &server.uri()); + assert_eq!(code, 0, "{env:#}\n{stderr}"); + assert_eq!(env["summary"]["applied"], 1, "{env:#}"); + let wired = std::fs::read_to_string(root.join("pdm.lock")).unwrap(); + assert!( + wired.contains(&alt_sha), + "run 1 pins the service wheel: {wired}" + ); + + // `pdm lock` re-resolves the registry unit. + std::fs::write(root.join("pdm.lock"), PDM_REGISTRY_LOCK).unwrap(); + mount_outage(&server).await; + let (code, env, stderr) = vendor_via_service(root, &server.uri()); + assert_eq!(code, 0, "{env:#}\n{stderr}"); + assert_eq!( + env["summary"]["applied"], 1, + "the relock is re-wired: {env:#}" + ); + assert!( + events(&env) + .iter() + .any(|e| e["errorCode"] == "vendor_artifact_reused"), + "{env:#}" + ); + assert!( + events(&env) + .iter() + .all(|e| e["errorCode"] != "vendor_prebuilt_unavailable"), + "{env:#}" + ); + assert_eq!( + std::fs::read_to_string(root.join("pdm.lock")).unwrap(), + wired, + "the same service sha is pinned again" + ); + assert_eq!(std::fs::read(wheel_path(root)).unwrap(), alt); + assert_eq!(package_posts(&server).await, 0); + + let (code, stdout, stderr) = run_cli( + root, + &[ + "rollback", + "--json", + "--offline", + "--yes", + "--cwd", + root.to_str().unwrap(), + ], + &[], + ); + assert_eq!(code, 0, "{stdout}\n{stderr}"); + assert_eq!( + std::fs::read_to_string(root.join("pdm.lock")).unwrap(), + PDM_REGISTRY_LOCK, + "rollback restores the relocked bytes" + ); +} diff --git a/crates/socket-patch-cli/tests/e2e_bun_lockb.rs b/crates/socket-patch-cli/tests/e2e_bun_lockb.rs index eed6e888..496052e1 100644 --- a/crates/socket-patch-cli/tests/e2e_bun_lockb.rs +++ b/crates/socket-patch-cli/tests/e2e_bun_lockb.rs @@ -623,6 +623,46 @@ async fn native_binary_hosted_vendored_takeover_roundtrip() { assert_eq!(repeat["summary"]["applied"], 0, "vendor rerun: {repeat}"); assert_eq!(repeat["summary"]["skipped"], 1, "vendor rerun: {repeat}"); assert_eq!(fixture.lock(), vendor_lock); + // The same rerun during a vendoring-service outage (closed port): the + // committed archive is reused, so bun.lockb and the tarball stay + // byte-identical, and no outage advisory is raised (the pre-reuse + // fallback to a local pack also stayed in sync here, but only after + // warning `vendor_prebuilt_unavailable` — that event is the tell). + let tgz_path = project.join(format!(".socket/vendor/npm/{UUID}/minimist-1.2.2.tgz")); + let vendor_tgz = std::fs::read(&tgz_path).unwrap(); + let outage = cli( + project, + &[ + "vendor", + "--api-url", + &server.uri(), + "--vendor-url", + "http://127.0.0.1:9", + "--api-token", + "fake", + "--org", + ORG, + ], + ); + assert_eq!(outage["summary"]["applied"], 0, "outage rerun: {outage}"); + assert_eq!(outage["summary"]["skipped"], 1, "outage rerun: {outage}"); + assert_eq!(fixture.lock(), vendor_lock); + assert_eq!(std::fs::read(&tgz_path).unwrap(), vendor_tgz); + let outage_events = outage["events"].as_array().cloned().unwrap_or_default(); + assert_eq!( + outage_events + .iter() + .filter(|e| e["errorCode"] == "already_vendored") + .count(), + 1, + "outage rerun: {outage}" + ); + assert!( + outage_events + .iter() + .all(|e| e["errorCode"] != "vendor_prebuilt_unavailable"), + "outage rerun must not touch the service: {outage}" + ); // Rebuild a deleted artifact from the manifest, preserve binary wiring. std::fs::remove_dir_all(project.join(".socket/vendor/npm")).unwrap(); diff --git a/crates/socket-patch-core/src/api/client.rs b/crates/socket-patch-core/src/api/client.rs index 9a58ec2d..ba3ff596 100644 --- a/crates/socket-patch-core/src/api/client.rs +++ b/crates/socket-patch-core/src/api/client.rs @@ -1,5 +1,7 @@ use std::collections::HashSet; -use std::sync::atomic::AtomicBool; +use std::sync::atomic::{AtomicBool, AtomicU32, Ordering}; +use std::sync::Arc; +use std::time::Duration; use reqwest::header::{self, HeaderMap, HeaderValue}; use reqwest::StatusCode; @@ -110,8 +112,108 @@ pub struct ApiClient { api_token: Option, use_public_proxy: bool, org_slug: Option, + /// Retry policy for the vendoring service's two round trips. + vendor_retry: VendorRetryPolicy, + /// Consecutive [`Self::fetch_vendor_package`] calls that ended in a + /// retryable failure (transport / 429 / 5xx after every retry) — the + /// run-level circuit breaker. Shared by clones: one CLI run, one count. + vendor_outage: Arc, } +/// Retry policy for the vendoring service's package-reference POST and +/// archive GET: `attempts` tries in total, exponential delays from `base` +/// with ±25% jitter, each capped at `max_delay` (a `Retry-After` in seconds +/// is honored under the same cap). Retried: transport errors (per-attempt +/// timeouts and bodies cut off mid-transfer included) and HTTP 429, +/// 500, 502, 503, 504. Never retried: auth (401/403), terminal misses +/// (404/410), still-building (408), other 4xx, parse errors. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct VendorRetryPolicy { + pub attempts: u32, + pub base: Duration, + pub max_delay: Duration, + /// Bound on one attempt's POST round trip, and on the archive GET's + /// connect + response headers (a black-holed or stalled host fails the + /// attempt instead of hanging the run). + pub attempt_timeout: Duration, + /// Bound on one archive GET's body read. + pub body_timeout: Duration, +} + +impl Default for VendorRetryPolicy { + fn default() -> Self { + Self { + attempts: 3, + base: Duration::from_millis(400), + max_delay: Duration::from_secs(4), + attempt_timeout: Duration::from_secs(30), + body_timeout: Duration::from_secs(300), + } + } +} + +impl VendorRetryPolicy { + /// A single attempt, no retry (the default per-attempt timeouts). + pub fn none() -> Self { + Self { + attempts: 1, + base: Duration::ZERO, + max_delay: Duration::ZERO, + ..Self::default() + } + } + + /// The pause before retry number `retry` (1-based), given the server's + /// `Retry-After` (seconds), and a jitter sample in `[0, 1)`. + fn delay(&self, retry: u32, retry_after: Option, jitter: f64) -> Duration { + if let Some(after) = retry_after { + return after.min(self.max_delay); + } + let exp = self + .base + .saturating_mul(1u32 << retry.saturating_sub(1).min(16)); + // ±25%: scale by [0.75, 1.25). + exp.mul_f64(0.75 + 0.5 * jitter.clamp(0.0, 1.0)) + .min(self.max_delay) + } +} + +/// Consecutive retryable vendor-service failures after which the rest of the +/// run skips the service without any I/O (`auto` then builds locally, +/// `service` fails closed — the existing miss policy). +const VENDOR_BREAKER_THRESHOLD: u32 = 2; + +/// A jitter sample in `[0, 1)` from std's randomly keyed hasher (no RNG +/// dependency; the quality needed here is "not synchronized"). +fn jitter_sample() -> f64 { + use std::hash::{BuildHasher as _, Hasher as _}; + let mut hasher = std::collections::hash_map::RandomState::new().build_hasher(); + hasher.write_u64(0); + (hasher.finish() >> 11) as f64 / (1u64 << 53) as f64 +} + +/// Is this vendor-service HTTP status worth another attempt? +fn vendor_status_retryable(status: StatusCode) -> bool { + matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504) +} + +/// A `Retry-After: ` header (the HTTP-date form is ignored). +fn retry_after_secs(headers: &HeaderMap) -> Option { + headers + .get(header::RETRY_AFTER)? + .to_str() + .ok()? + .trim() + .parse::() + .ok() + .map(Duration::from_secs) +} + +/// One vendor-service attempt's failure: the error, and — when the failure +/// is retryable — the server's `Retry-After` hint (`Some(None)` = retryable +/// without a hint). +type VendorAttemptError = (ApiError, Option>); + /// Body payload for the batch search POST endpoint. #[derive(Serialize)] struct BatchSearchBody { @@ -167,9 +269,18 @@ impl ApiClient { api_token: options.api_token, use_public_proxy: options.use_public_proxy, org_slug: options.org_slug, + vendor_retry: VendorRetryPolicy::default(), + vendor_outage: Arc::new(AtomicU32::new(0)), } } + /// Override the vendoring-service retry policy (tests; a policy of + /// [`VendorRetryPolicy::none`] makes a single attempt). + pub fn with_vendor_retry(mut self, policy: VendorRetryPolicy) -> Self { + self.vendor_retry = policy; + self + } + /// Returns the API token, if set. pub fn api_token(&self) -> Option<&String> { self.api_token.as_ref() @@ -718,6 +829,44 @@ impl ApiClient { "Invalid patch UUID: {uuid}" ))); } + // Circuit breaker: after consecutive retryable failures the service + // is down for this run — skip it without any I/O instead of paying + // every package's retries (and mixing sources package by package). + let failures = self.vendor_outage.load(Ordering::Relaxed); + if failures >= VENDOR_BREAKER_THRESHOLD { + // Worded to read inside the callers' "patch service request + // failed (...)" wrapper: no request was made, and the skip is + // scoped to this run. + return VendorServiceOutcome::Failed(ApiError::Other(format!( + "not attempted: the service failed for the previous {failures} packages in \ + this run" + ))); + } + let (outcome, retryable_failure) = self + .fetch_vendor_package_once(uuid, free_only, vendor_url, patch_server_url) + .await; + match &outcome { + VendorServiceOutcome::Failed(_) if retryable_failure => { + self.vendor_outage.fetch_add(1, Ordering::Relaxed); + } + // A non-retryable failure (auth, parse) says nothing about + // availability either way. + VendorServiceOutcome::Failed(_) => {} + _ => self.vendor_outage.store(0, Ordering::Relaxed), + } + outcome + } + + /// [`Self::fetch_vendor_package`] without the breaker: the outcome, and + /// whether a `Failed` one was a retryable (availability) failure. + async fn fetch_vendor_package_once( + &self, + uuid: &str, + free_only: bool, + vendor_url: Option<&str>, + patch_server_url: Option<&str>, + ) -> (VendorServiceOutcome, bool) { + let done = |o: VendorServiceOutcome| (o, false); // ── Step 1: resolve the grant URL + integrity ────────────────────── let result = match self @@ -725,21 +874,25 @@ impl ApiClient { .await { Ok(r) => r, - Err(e) => return VendorServiceOutcome::Failed(e), + Err((e, retryable)) => return (VendorServiceOutcome::Failed(e), retryable), }; // Classify the build/grant status before attempting any download. match result.status.as_str() { "granted" | "reused" => {} - "pending_build" => return VendorServiceOutcome::Pending, + "pending_build" => return done(VendorServiceOutcome::Pending), "build_failed" | "withdrawn" | "not_found" => { - return VendorServiceOutcome::Unavailable(result.status.clone()) + return done(VendorServiceOutcome::Unavailable(result.status.clone())) } "forbidden" => { - return VendorServiceOutcome::Failed(ApiError::Forbidden( + return done(VendorServiceOutcome::Failed(ApiError::Forbidden( "Forbidden: not entitled to this patch (paid tier or no org access).".into(), - )) + ))) + } + other => { + return done(VendorServiceOutcome::Unavailable(format!( + "unknown status `{other}`" + ))) } - other => return VendorServiceOutcome::Unavailable(format!("unknown status `{other}`")), } // Select the native tarball artifact and its sha512 (the universal @@ -750,22 +903,26 @@ impl ApiClient { .as_ref() .and_then(|arts| arts.iter().find(|a| a.kind == "tarball")) else { - return VendorServiceOutcome::Unavailable("no tarball artifact in response".into()); + return done(VendorServiceOutcome::Unavailable( + "no tarball artifact in response".into(), + )); }; let Some(sha512_raw) = artifact.integrity.sha512.as_deref() else { - return VendorServiceOutcome::Unavailable( + return done(VendorServiceOutcome::Unavailable( "tarball artifact has no sha512 integrity".into(), - ); + )); }; let integrity_sri = normalize_sha512_sri(sha512_raw); // The artifact's own URL wins; fall back to the top-level `url`. let Some(download_url) = artifact.url.as_deref().or(result.url.as_deref()) else { - return VendorServiceOutcome::Unavailable("granted result has no download url".into()); + return done(VendorServiceOutcome::Unavailable( + "granted result has no download url".into(), + )); }; let download_url = match patch_server_url { Some(base) => match rewrite_url_host(download_url, base) { Ok(u) => u, - Err(e) => return VendorServiceOutcome::Failed(e), + Err(e) => return done(VendorServiceOutcome::Failed(e)), }, None => download_url.to_string(), }; @@ -800,19 +957,21 @@ impl ApiClient { } // ── Step 2: download the prebuilt archive ────────────────────────── - match self.download_vendor_archive(&download_url).await { - ServeDownload::Ok(bytes) => VendorServiceOutcome::Ready(FetchedVendorPackage { - tarball: bytes, - integrity_sri, - dirhash_h1: artifact.integrity.dirhash_h1.clone(), - source_url: download_url, - secondary_artifacts, - }), - ServeDownload::NotFound => { - VendorServiceOutcome::Unavailable("serve returned 404/410".into()) + match self.download_vendor_archive_retrying(&download_url).await { + (ServeDownload::Ok(bytes), _) => { + done(VendorServiceOutcome::Ready(FetchedVendorPackage { + tarball: bytes, + integrity_sri, + dirhash_h1: artifact.integrity.dirhash_h1.clone(), + source_url: download_url, + secondary_artifacts, + })) } - ServeDownload::Pending => VendorServiceOutcome::Pending, - ServeDownload::Failed(e) => VendorServiceOutcome::Failed(e), + (ServeDownload::NotFound, _) => done(VendorServiceOutcome::Unavailable( + "serve returned 404/410".into(), + )), + (ServeDownload::Pending, _) => done(VendorServiceOutcome::Pending), + (ServeDownload::Failed(e), retryable) => (VendorServiceOutcome::Failed(e), retryable), } } @@ -848,14 +1007,49 @@ impl ApiClient { } } - /// Step 1 of [`Self::fetch_vendor_package`]: POST the package-reference - /// endpoint and return the single requested UUID's result. + /// Pause before retry number `retry` (see [`VendorRetryPolicy`]). + async fn vendor_backoff(&self, retry: u32, retry_after: Option) { + let delay = self.vendor_retry.delay(retry, retry_after, jitter_sample()); + debug_log(&format!("vendor service retry {retry} in {delay:?}")); + tokio::time::sleep(delay).await; + } + + /// Step 1 of [`Self::fetch_vendor_package`], retried per the client's + /// [`VendorRetryPolicy`]. `Err` carries whether the final failure was a + /// retryable (availability) one. async fn request_vendor_package( &self, uuid: &str, free_only: bool, vendor_url: Option<&str>, - ) -> Result { + ) -> Result { + let attempts = self.vendor_retry.attempts.max(1); + let mut attempt = 1; + loop { + match self + .request_vendor_package_once(uuid, free_only, vendor_url) + .await + { + Ok(result) => return Ok(result), + Err((e, Some(retry_after))) if attempt < attempts => { + debug_log(&format!( + "vendor package request attempt {attempt} failed: {e}" + )); + self.vendor_backoff(attempt, retry_after).await; + attempt += 1; + } + Err((e, hint)) => return Err((e, hint.is_some())), + } + } + } + + /// One package-reference POST: the single requested UUID's result. + async fn request_vendor_package_once( + &self, + uuid: &str, + free_only: bool, + vendor_url: Option<&str>, + ) -> Result { let body = PackageVendorRequest { uuids: vec![uuid.to_string()], // Only send freeOnly when forcing it (the public-proxy contract); @@ -865,11 +1059,15 @@ impl ApiClient { let (url, use_auth) = self.vendor_package_url(vendor_url); debug_log(&format!("POST {url}")); + // The whole round trip (connect, response, JSON body) is bounded per + // attempt, so attempts × timeout + backoff bounds the step. + let timeout = self.vendor_retry.attempt_timeout; let resp = if use_auth { self.client .post(&url) .header(header::CONTENT_TYPE, "application/json") .json(&body) + .timeout(timeout) .send() .await } else { @@ -879,57 +1077,122 @@ impl ApiClient { .header(header::CONTENT_TYPE, "application/json") .header(header::ACCEPT, "application/json") .json(&body) + .timeout(timeout) .send() .await }; let resp = resp.map_err(|e| { - ApiError::Network(format!("Network error: {}", network_error_detail(&e))) + ( + ApiError::Network(format!("Network error: {}", network_error_detail(&e))), + Some(None), + ) })?; let status = resp.status(); if status == StatusCode::OK { - let parsed = resp - .json::() - .await - .map_err(|e| ApiError::Parse(format!("Failed to parse package response: {e}")))?; + let parsed = resp.json::().await.map_err(|e| { + // A body cut off (or timed out) mid-transfer is transport, + // not a malformed answer. + let hint = (e.is_timeout() || e.is_body()).then_some(None); + ( + ApiError::Parse(format!("Failed to parse package response: {e}")), + hint, + ) + })?; return parsed.results.get(uuid).cloned().ok_or_else(|| { - ApiError::Other(format!("package response missing a result for {uuid}")) + ( + ApiError::Other(format!("package response missing a result for {uuid}")), + None, + ) }); } + // 429 classifies as RateLimited but is still retried (the hint); + // 401/403 carry no hint. + let hint = vendor_status_retryable(status).then(|| retry_after_secs(resp.headers())); if let Some(err) = classify_auth_error(status, !use_auth) { - return Err(err); + return Err((err, hint)); } let text = resp.text().await.unwrap_or_default(); - Err(ApiError::Other(status_error( - "package request failed with status", - status, - &text, - ))) + Err(( + ApiError::Other(status_error( + "package request failed with status", + status, + &text, + )), + hint, + )) + } + + /// Step 2 of [`Self::fetch_vendor_package`], retried per the client's + /// [`VendorRetryPolicy`]; the flag says whether a final `Failed` was a + /// retryable (availability) failure. + async fn download_vendor_archive_retrying(&self, url: &str) -> (ServeDownload, bool) { + let attempts = self.vendor_retry.attempts.max(1); + let mut attempt = 1; + loop { + match self.download_vendor_archive_once(url).await { + (ServeDownload::Failed(e), Some(retry_after)) if attempt < attempts => { + debug_log(&format!( + "vendor package download attempt {attempt} failed: {e}" + )); + self.vendor_backoff(attempt, retry_after).await; + attempt += 1; + } + (outcome, hint) => return (outcome, hint.is_some()), + } + } } - /// Step 2 of [`Self::fetch_vendor_package`]: GET the grant-tokenized serve - /// URL. The grant token in the path is the authorization, so this uses a - /// plain (no-auth) client. + /// [`Self::download_vendor_archive_retrying`] without the flag. async fn download_vendor_archive(&self, url: &str) -> ServeDownload { + self.download_vendor_archive_retrying(url).await.0 + } + + /// One GET of the grant-tokenized serve URL. The grant token in the path + /// is the authorization, so this uses a plain (no-auth) client. The hint + /// is `Some` iff a `Failed` outcome is retryable. + async fn download_vendor_archive_once( + &self, + url: &str, + ) -> (ServeDownload, Option>) { if !(url.starts_with("https://") || url.starts_with("http://")) { - return ServeDownload::Failed(ApiError::Other(format!( - "refusing non-http(s) artifact URL `{url}`" - ))); + return ( + ServeDownload::Failed(ApiError::Other(format!( + "refusing non-http(s) artifact URL `{url}`" + ))), + None, + ); } debug_log(&format!("GET vendor package {url}")); - let resp = match self - .plain - .get(url) - .header(header::ACCEPT, "application/octet-stream") - .send() - .await - { - Ok(r) => r, - Err(e) => { - return ServeDownload::Failed(ApiError::Network(format!( - "Network error fetching vendor package: {}", - network_error_detail(&e) - ))) + // Connect + response headers are bounded per attempt; the body read + // below gets its own (larger) bound — archives can be big. + let sent = tokio::time::timeout( + self.vendor_retry.attempt_timeout, + self.plain + .get(url) + .header(header::ACCEPT, "application/octet-stream") + .send(), + ) + .await; + let resp = match sent { + Ok(Ok(r)) => r, + Ok(Err(e)) => { + return ( + ServeDownload::Failed(ApiError::Network(format!( + "Network error fetching vendor package: {}", + network_error_detail(&e) + ))), + Some(None), + ) + } + Err(_) => { + return ( + ServeDownload::Failed(ApiError::Network(format!( + "Network error fetching vendor package: no response within {:?}", + self.vendor_retry.attempt_timeout + ))), + Some(None), + ) } }; let status = resp.status(); @@ -937,25 +1200,48 @@ impl ApiClient { StatusCode::OK => {} // 404 (build_failed / not stored) and 410 (withdrawn) are terminal // misses; the caller decides build-fallback vs hard-fail. - StatusCode::NOT_FOUND | StatusCode::GONE => return ServeDownload::NotFound, - // 408 = the archive is still building (Retry-After) — retryable. - StatusCode::REQUEST_TIMEOUT => return ServeDownload::Pending, + StatusCode::NOT_FOUND | StatusCode::GONE => return (ServeDownload::NotFound, None), + // 408 = the archive is still building (Retry-After) — the + // caller's pending policy, never retried here. + StatusCode::REQUEST_TIMEOUT => return (ServeDownload::Pending, None), _ => { + let hint = + vendor_status_retryable(status).then(|| retry_after_secs(resp.headers())); if let Some(err) = classify_auth_error(status, true) { - return ServeDownload::Failed(err); + return (ServeDownload::Failed(err), hint); } let text = resp.text().await.unwrap_or_default(); - return ServeDownload::Failed(ApiError::Other(format!( - "vendor package download failed with status {}: {text}", - status.as_u16(), - ))); + return ( + ServeDownload::Failed(ApiError::Other(format!( + "vendor package download failed with status {}: {text}", + status.as_u16(), + ))), + hint, + ); } } - match crate::utils::http::read_capped(resp, MAX_VENDOR_PACKAGE_BYTES, "vendor package") - .await - { - Ok(bytes) => ServeDownload::Ok(bytes), - Err(e) => ServeDownload::Failed(ApiError::Network(e)), + use crate::utils::http::{read_capped_typed, ReadCappedError}; + let body = tokio::time::timeout( + self.vendor_retry.body_timeout, + read_capped_typed(resp, MAX_VENDOR_PACKAGE_BYTES, "vendor package"), + ) + .await + .unwrap_or_else(|_| { + Err(ReadCappedError::Truncated(format!( + "vendor package body not received within {:?}", + self.vendor_retry.body_timeout + ))) + }); + match body { + Ok(bytes) => (ServeDownload::Ok(bytes), None), + // A body cut off mid-transfer is a transport failure (retryable); + // a size-cap breach is not (the same bytes would breach it again). + Err(ReadCappedError::Truncated(e)) => { + (ServeDownload::Failed(ApiError::Network(e)), Some(None)) + } + Err(ReadCappedError::CapExceeded(e)) => { + (ServeDownload::Failed(ApiError::Network(e)), None) + } } } @@ -3754,6 +4040,585 @@ mod vendor_package_tests { } } +#[cfg(test)] +mod vendor_retry_tests { + //! Retry + circuit breaker for the vendoring service's two round trips. + use super::*; + use base64::Engine as _; + use sha2::{Digest as _, Sha512}; + use wiremock::matchers::{method, path}; + use wiremock::{Mock, MockServer, ResponseTemplate}; + + const UUID_A: &str = "11111111-1111-4111-8111-111111111111"; + const UUID_B: &str = "22222222-2222-4222-8222-222222222222"; + const UUID_C: &str = "33333333-3333-4333-8333-333333333333"; + const POST_PATH: &str = "/v0/orgs/acme/patches/package"; + const SERVE: &str = "/serve/pkg.tgz"; + const BYTES: &[u8] = b"prebuilt bytes"; + + /// A fast policy: three attempts, millisecond delays. + fn fast() -> VendorRetryPolicy { + VendorRetryPolicy { + attempts: 3, + base: Duration::from_millis(1), + max_delay: Duration::from_millis(5), + ..VendorRetryPolicy::default() + } + } + + fn client(uri: &str, policy: VendorRetryPolicy) -> ApiClient { + ApiClient::new(ApiClientOptions { + api_url: uri.to_string(), + api_token: Some("sktsec_placeholder_value_for_tests_api".into()), + use_public_proxy: false, + org_slug: Some("acme".into()), + }) + .with_vendor_retry(policy) + } + + fn granted(server: &MockServer, uuid: &str) -> ResponseTemplate { + let url = format!("{}{SERVE}", server.uri()); + let sri = format!( + "sha512-{}", + base64::engine::general_purpose::STANDARD.encode(Sha512::digest(BYTES)) + ); + ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "results": { uuid: { "status": "granted", "url": url, + "artifacts": [{ "kind": "tarball", "url": url, + "integrity": { "sha512": sri } }] } } + })) + } + + async fn mount_serve(server: &MockServer) { + Mock::given(method("GET")) + .and(path(SERVE)) + .respond_with(ResponseTemplate::new(200).set_body_bytes(BYTES.to_vec())) + .mount(server) + .await; + } + + /// `n` responses of `status` on the POST, then (optionally) a grant. + async fn mount_post_failures_then(server: &MockServer, status: u16, n: u64, then_grant: bool) { + Mock::given(method("POST")) + .and(path(POST_PATH)) + .respond_with(ResponseTemplate::new(status)) + .up_to_n_times(n) + .with_priority(1) + .mount(server) + .await; + if then_grant { + Mock::given(method("POST")) + .and(path(POST_PATH)) + .respond_with(granted(server, UUID_A)) + .with_priority(2) + .mount(server) + .await; + } + } + + async fn posts(server: &MockServer) -> usize { + server + .received_requests() + .await + .unwrap() + .iter() + .filter(|r| r.method == wiremock::http::Method::POST) + .count() + } + + #[tokio::test] + async fn post_503_twice_then_200_is_ready_after_three_requests() { + let server = MockServer::start().await; + mount_post_failures_then(&server, 503, 2, true).await; + mount_serve(&server).await; + let outcome = client(&server.uri(), fast()) + .fetch_vendor_package(UUID_A, false, None, None) + .await; + assert!( + matches!(outcome, VendorServiceOutcome::Ready(ref p) if p.tarball == BYTES), + "{outcome:?}" + ); + assert_eq!(posts(&server).await, 3); + } + + #[tokio::test] + async fn post_503_three_times_is_failed_after_three_requests() { + let server = MockServer::start().await; + mount_post_failures_then(&server, 503, 3, false).await; + let outcome = client(&server.uri(), fast()) + .fetch_vendor_package(UUID_A, false, None, None) + .await; + assert!( + matches!(outcome, VendorServiceOutcome::Failed(_)), + "{outcome:?}" + ); + assert_eq!(posts(&server).await, 3); + } + + #[tokio::test] + async fn every_retryable_status_is_retried() { + for status in [429u16, 500, 502, 503, 504] { + let server = MockServer::start().await; + mount_post_failures_then(&server, status, 1, true).await; + mount_serve(&server).await; + let outcome = client(&server.uri(), fast()) + .fetch_vendor_package(UUID_A, false, None, None) + .await; + assert!( + matches!(outcome, VendorServiceOutcome::Ready(_)), + "{status}: {outcome:?}" + ); + assert_eq!(posts(&server).await, 2, "{status}"); + } + } + + #[tokio::test] + async fn get_502_then_200_is_ready() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path(POST_PATH)) + .respond_with(granted(&server, UUID_A)) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path(SERVE)) + .respond_with(ResponseTemplate::new(502)) + .up_to_n_times(1) + .with_priority(1) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path(SERVE)) + .respond_with(ResponseTemplate::new(200).set_body_bytes(BYTES.to_vec())) + .with_priority(2) + .mount(&server) + .await; + let outcome = client(&server.uri(), fast()) + .fetch_vendor_package(UUID_A, false, None, None) + .await; + assert!( + matches!(outcome, VendorServiceOutcome::Ready(ref p) if p.tarball == BYTES), + "{outcome:?}" + ); + assert_eq!( + posts(&server).await, + 1, + "the POST is not repeated for a GET retry" + ); + } + + #[tokio::test] + async fn transport_errors_are_retried() { + // A closed port: every attempt is a connect error. + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let uri = format!("http://{}", listener.local_addr().unwrap()); + drop(listener); + let c = client(&uri, fast()); + let outcome = c.fetch_vendor_package(UUID_A, false, None, None).await; + assert!( + matches!(outcome, VendorServiceOutcome::Failed(ApiError::Network(_))), + "{outcome:?}" + ); + // A transport failure counts toward the breaker. + assert_eq!(c.vendor_outage.load(Ordering::Relaxed), 1); + } + + #[tokio::test] + async fn terminal_statuses_are_never_retried() { + // POST side: auth and other 4xx. + for status in [401u16, 403, 400, 404] { + let server = MockServer::start().await; + mount_post_failures_then(&server, status, 10, false).await; + let _ = client(&server.uri(), fast()) + .fetch_vendor_package(UUID_A, false, None, None) + .await; + assert_eq!(posts(&server).await, 1, "POST {status}"); + } + // GET side: 404/410 (terminal miss), 408 (pending), 403 (auth). + for status in [404u16, 410, 408, 403] { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path(POST_PATH)) + .respond_with(granted(&server, UUID_A)) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path(SERVE)) + .respond_with(ResponseTemplate::new(status)) + .mount(&server) + .await; + let outcome = client(&server.uri(), fast()) + .fetch_vendor_package(UUID_A, false, None, None) + .await; + let gets = server + .received_requests() + .await + .unwrap() + .iter() + .filter(|r| r.method == wiremock::http::Method::GET) + .count(); + assert_eq!(gets, 1, "GET {status}: {outcome:?}"); + } + } + + #[tokio::test] + async fn malformed_200_is_not_retried() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path(POST_PATH)) + .respond_with(ResponseTemplate::new(200).set_body_string("not json")) + .mount(&server) + .await; + let c = client(&server.uri(), fast()); + let outcome = c.fetch_vendor_package(UUID_A, false, None, None).await; + assert!( + matches!(outcome, VendorServiceOutcome::Failed(ApiError::Parse(_))), + "{outcome:?}" + ); + assert_eq!(posts(&server).await, 1); + assert_eq!( + c.vendor_outage.load(Ordering::Relaxed), + 0, + "not an availability failure" + ); + } + + /// `Retry-After` is honored (the pause is at least the header's second, + /// far above the 1ms base) and capped at `max_delay`. + #[tokio::test] + async fn retry_after_is_honored_and_capped() { + let honoring = VendorRetryPolicy { + attempts: 2, + base: Duration::from_millis(1), + max_delay: Duration::from_secs(5), + ..VendorRetryPolicy::default() + }; + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path(POST_PATH)) + .respond_with(ResponseTemplate::new(503).insert_header("Retry-After", "1")) + .up_to_n_times(1) + .with_priority(1) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path(POST_PATH)) + .respond_with(granted(&server, UUID_A)) + .with_priority(2) + .mount(&server) + .await; + mount_serve(&server).await; + let started = std::time::Instant::now(); + let outcome = client(&server.uri(), honoring) + .fetch_vendor_package(UUID_A, false, None, None) + .await; + assert!( + matches!(outcome, VendorServiceOutcome::Ready(_)), + "{outcome:?}" + ); + assert!( + started.elapsed() >= Duration::from_millis(950), + "{:?}", + started.elapsed() + ); + + // Retry-After: 30 with a 20ms cap retries almost immediately. + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path(POST_PATH)) + .respond_with(ResponseTemplate::new(503).insert_header("Retry-After", "30")) + .mount(&server) + .await; + let capped = VendorRetryPolicy { + attempts: 2, + base: Duration::from_millis(1), + max_delay: Duration::from_millis(20), + ..VendorRetryPolicy::default() + }; + let started = std::time::Instant::now(); + let _ = client(&server.uri(), capped) + .fetch_vendor_package(UUID_A, false, None, None) + .await; + assert!(started.elapsed() < Duration::from_secs(10)); + assert_eq!(posts(&server).await, 2); + } + + /// A policy whose attempts time out after `timeout`. + fn timing_out(timeout: Duration) -> VendorRetryPolicy { + VendorRetryPolicy { + attempts: 2, + attempt_timeout: timeout, + body_timeout: timeout, + ..fast() + } + } + + /// A stalled POST fails the attempt at the per-attempt timeout (and is + /// retried as a transport failure), so the step's latency is bounded by + /// attempts × timeout + backoff instead of hanging. + #[tokio::test] + async fn stalled_post_times_out_per_attempt_and_is_retried() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path(POST_PATH)) + .respond_with(granted(&server, UUID_A).set_delay(Duration::from_secs(30))) + .mount(&server) + .await; + let started = std::time::Instant::now(); + let outcome = client(&server.uri(), timing_out(Duration::from_millis(200))) + .fetch_vendor_package(UUID_A, false, None, None) + .await; + assert!( + matches!(outcome, VendorServiceOutcome::Failed(ApiError::Network(_))), + "{outcome:?}" + ); + assert!( + started.elapsed() < Duration::from_secs(10), + "{:?}", + started.elapsed() + ); + assert_eq!(posts(&server).await, 2, "the timed-out attempt is retried"); + } + + /// The archive GET's response headers are bounded the same way. + #[tokio::test] + async fn stalled_get_times_out_per_attempt_and_is_retried() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path(POST_PATH)) + .respond_with(granted(&server, UUID_A)) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path(SERVE)) + .respond_with( + ResponseTemplate::new(200) + .set_body_bytes(BYTES.to_vec()) + .set_delay(Duration::from_secs(30)), + ) + .mount(&server) + .await; + let started = std::time::Instant::now(); + let outcome = client(&server.uri(), timing_out(Duration::from_millis(200))) + .fetch_vendor_package(UUID_A, false, None, None) + .await; + assert!( + matches!(outcome, VendorServiceOutcome::Failed(ApiError::Network(_))), + "{outcome:?}" + ); + assert!( + started.elapsed() < Duration::from_secs(10), + "{:?}", + started.elapsed() + ); + let gets = server + .received_requests() + .await + .unwrap() + .iter() + .filter(|r| r.method == wiremock::http::Method::GET) + .count(); + assert_eq!(gets, 2); + } + + /// A raw HTTP server for the serve GET: connection `i` gets + /// `responses[min(i, last)]` verbatim and is then closed. Returns the + /// base URL and the connection counter. + fn raw_serve(responses: Vec>) -> (String, Arc) { + use std::io::{Read as _, Write as _}; + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let addr = listener.local_addr().unwrap(); + let count = Arc::new(AtomicU32::new(0)); + let seen = count.clone(); + std::thread::spawn(move || { + for stream in listener.incoming() { + let Ok(mut stream) = stream else { return }; + let i = seen.fetch_add(1, Ordering::SeqCst) as usize; + // Drain the request head. + let mut buf = [0u8; 4096]; + let mut head = Vec::new(); + while !head.windows(4).any(|w| w == b"\r\n\r\n") { + match stream.read(&mut buf) { + Ok(0) | Err(_) => break, + Ok(n) => head.extend_from_slice(&buf[..n]), + } + } + let _ = stream.write_all(&responses[i.min(responses.len() - 1)]); + let _ = stream.flush(); + drop(stream); + } + }); + (format!("http://{addr}"), count) + } + + async fn mount_grant_to(server: &MockServer, serve_base: &str) { + let url = format!("{serve_base}{SERVE}"); + let sri = format!( + "sha512-{}", + base64::engine::general_purpose::STANDARD.encode(Sha512::digest(BYTES)) + ); + Mock::given(method("POST")) + .and(path(POST_PATH)) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "results": { UUID_A: { "status": "granted", "url": url, + "artifacts": [{ "kind": "tarball", "url": url, + "integrity": { "sha512": sri } }] } } + }))) + .mount(server) + .await; + } + + /// A body cut off mid-transfer (fewer bytes than `Content-Length`) is a + /// transport failure: retried, and the second, whole body is Ready. + #[tokio::test] + async fn truncated_body_is_retried() { + let mut cut = format!( + "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", + BYTES.len() + 90 + ) + .into_bytes(); + cut.extend_from_slice(&BYTES[..4]); + let mut whole = format!( + "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", + BYTES.len() + ) + .into_bytes(); + whole.extend_from_slice(BYTES); + let (base, conns) = raw_serve(vec![cut, whole]); + let server = MockServer::start().await; + mount_grant_to(&server, &base).await; + let outcome = client(&server.uri(), fast()) + .fetch_vendor_package(UUID_A, false, None, None) + .await; + assert!( + matches!(outcome, VendorServiceOutcome::Ready(ref p) if p.tarball == BYTES), + "{outcome:?}" + ); + assert_eq!(conns.load(Ordering::SeqCst), 2); + } + + /// A size-cap breach is not retried: the same bytes would breach it + /// again. + #[tokio::test] + async fn cap_breach_is_not_retried() { + let over = format!( + "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", + MAX_VENDOR_PACKAGE_BYTES + 1 + ) + .into_bytes(); + let (base, conns) = raw_serve(vec![over]); + let server = MockServer::start().await; + mount_grant_to(&server, &base).await; + let outcome = client(&server.uri(), fast()) + .fetch_vendor_package(UUID_A, false, None, None) + .await; + assert!( + matches!(outcome, VendorServiceOutcome::Failed(ApiError::Network(ref m)) if m.contains("too large")), + "{outcome:?}" + ); + assert_eq!(conns.load(Ordering::SeqCst), 1); + } + + #[test] + fn delay_is_exponential_jittered_and_capped() { + let p = VendorRetryPolicy::default(); + assert_eq!(p.attempts, 3); + // Midpoint jitter (0.5) is the nominal exponential value. + assert_eq!(p.delay(1, None, 0.5), Duration::from_millis(400)); + assert_eq!(p.delay(2, None, 0.5), Duration::from_millis(800)); + // ±25%. + assert_eq!(p.delay(1, None, 0.0), Duration::from_millis(300)); + assert!(p.delay(1, None, 0.999_999) < Duration::from_millis(500)); + // Capped at max_delay. + assert_eq!(p.delay(10, None, 0.5), Duration::from_secs(4)); + // Retry-After wins, under the cap. + assert_eq!( + p.delay(1, Some(Duration::from_secs(2)), 0.5), + Duration::from_secs(2) + ); + assert_eq!( + p.delay(1, Some(Duration::from_secs(60)), 0.5), + Duration::from_secs(4) + ); + let j = jitter_sample(); + assert!((0.0..1.0).contains(&j)); + assert_eq!(VendorRetryPolicy::none().attempts, 1); + } + + /// Two uuids exhaust their retries; the third makes NO request and + /// fails fast. A later success resets the breaker. + #[tokio::test] + async fn breaker_opens_after_two_exhausted_fetches_and_resets_on_success() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path(POST_PATH)) + .respond_with(ResponseTemplate::new(503)) + .mount(&server) + .await; + let c = client(&server.uri(), fast()); + for uuid in [UUID_A, UUID_B] { + let o = c.fetch_vendor_package(uuid, false, None, None).await; + assert!(matches!(o, VendorServiceOutcome::Failed(_)), "{o:?}"); + } + assert_eq!(posts(&server).await, 6); + let o = c.fetch_vendor_package(UUID_C, false, None, None).await; + match o { + VendorServiceOutcome::Failed(ApiError::Other(msg)) => { + assert!( + msg.contains( + "not attempted: the service failed for the previous 2 packages in this run" + ), + "{msg}" + ) + } + other => panic!("expected the breaker's Failed, got {other:?}"), + } + assert_eq!(posts(&server).await, 6, "the open breaker makes no request"); + // Clones share the breaker (one run, one count). + let clone = c.clone(); + assert!(matches!( + clone.fetch_vendor_package(UUID_C, false, None, None).await, + VendorServiceOutcome::Failed(_) + )); + assert_eq!(posts(&server).await, 6); + + // A success resets it. + c.vendor_outage.store(1, Ordering::Relaxed); + server.reset().await; + Mock::given(method("POST")) + .and(path(POST_PATH)) + .respond_with(granted(&server, UUID_A)) + .mount(&server) + .await; + mount_serve(&server).await; + assert!(matches!( + c.fetch_vendor_package(UUID_A, false, None, None).await, + VendorServiceOutcome::Ready(_) + )); + assert_eq!(c.vendor_outage.load(Ordering::Relaxed), 0); + } + + /// Pending / Unavailable answers prove the service is up: they reset + /// the count, so an isolated failure never opens the breaker. + #[tokio::test] + async fn non_failure_answers_reset_the_breaker() { + for status in ["pending_build", "not_found"] { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path(POST_PATH)) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "results": { UUID_A: { "status": status, "url": null, "artifacts": [] } } + }))) + .mount(&server) + .await; + let c = client(&server.uri(), fast()); + c.vendor_outage.store(1, Ordering::Relaxed); + let _ = c.fetch_vendor_package(UUID_A, false, None, None).await; + assert_eq!(c.vendor_outage.load(Ordering::Relaxed), 0, "{status}"); + } + } +} + #[cfg(test)] mod authenticated_batch_tests { use super::*; diff --git a/crates/socket-patch-core/src/patch/package.rs b/crates/socket-patch-core/src/patch/package.rs index ca305937..54ed7a7f 100644 --- a/crates/socket-patch-core/src/patch/package.rs +++ b/crates/socket-patch-core/src/patch/package.rs @@ -46,6 +46,35 @@ pub enum ArchiveError { EntryTooLarge { path: String, size: u64, max: u64 }, #[error("archive contains more than {0} entries")] TooManyEntries(usize), + /// Strict decoding only: the archive is not in the canonical shape an + /// installer extracts exactly as decoded (see + /// [`read_archive_bytes_to_map_strict`]). + #[error("entry {0:?} is not canonical")] + NonCanonical(String), +} + +/// The key two archive member names collide under on a case-insensitive +/// filesystem, or `None` when the name is outside the strict shape: ASCII +/// only (no Unicode case-folding / normalization aliases), no backslash, +/// no empty/`.`/`..` segment smuggled past a string compare. Used by the +/// strict tarball and wheel decoders so every member an installer would +/// write has exactly one decoded twin. +pub(crate) fn canonical_member_key(name: &str) -> Option { + if !name.is_ascii() || name.contains('\\') || name.bytes().any(|b| b.is_ascii_control()) { + return None; + } + let mut segs = Vec::new(); + for seg in name.split('/') { + match seg { + "" | "." => continue, + ".." => return None, + s => segs.push(s.to_ascii_lowercase()), + } + } + if segs.is_empty() { + return None; + } + Some(segs.join("/")) } /// Read a `.tar.gz` archive into a map of `normalized_path -> bytes`. @@ -86,17 +115,48 @@ pub fn read_archive_to_map(archive_path: &Path) -> Result Result>, ArchiveError> { - read_tgz_to_map(bytes) + read_archive_from_reader(bytes, false) } -fn read_tgz_to_map(reader: R) -> Result>, ArchiveError> { +/// [`read_archive_bytes_to_map`] that additionally refuses any archive an +/// npm-family installer (npm, pnpm, yarn, bun) would extract DIFFERENTLY +/// from how it decodes, so a member check over the decoded map is a check +/// over what gets installed. Those installers strip the first path segment +/// whatever it is, extract more entry types than `Regular` as files, and +/// let case-variant names overwrite each other on a case-insensitive +/// filesystem; the lenient decoder strips only a literal `package/`, keeps +/// `Regular` entries, and lets the last duplicate win. Strict mode fails +/// ([`ArchiveError::NonCanonical`]) when: +/// +/// - an entry's path does not start with `package/`; +/// - an entry is anything but `Regular` or `Directory` (pax / GNU long-name +/// metadata headers are consumed or skipped); +/// - an entry name is not in [`canonical_member_key`]'s shape, or two +/// entries share a key (exact or ASCII-case-folded duplicates). +/// +/// For re-verifying a COMMITTED artifact before it is reused unchanged. +pub fn read_archive_bytes_to_map_strict( + bytes: &[u8], +) -> Result>, ArchiveError> { + read_archive_from_reader(bytes, true) +} + +/// The shared decoder behind [`read_archive_to_map`] and +/// [`read_archive_bytes_to_map`]: gunzip → tar walk with every cap and the +/// post-normalization path-safety gate (plus the canonical-shape gate when +/// `strict`). +fn read_archive_from_reader( + reader: R, + strict: bool, +) -> Result>, ArchiveError> { // Hard-cap decompressed bytes to defuse gzip / tar bombs. Reads // beyond the limit yield EOF, which the tar parser surfaces as a // truncated-archive error. @@ -104,6 +164,9 @@ fn read_tgz_to_map(reader: R) -> Result>, Archi let mut tar = Archive::new(bounded); let mut out: HashMap> = HashMap::new(); + // Strict mode: canonical keys of every file / directory seen. + let mut file_keys: std::collections::HashSet = std::collections::HashSet::new(); + let mut dir_keys: std::collections::HashSet = std::collections::HashSet::new(); let mut entry_count: usize = 0; for entry in tar.entries()? { let mut entry = entry?; @@ -113,8 +176,52 @@ fn read_tgz_to_map(reader: R) -> Result>, Archi return Err(ArchiveError::TooManyEntries(MAX_ENTRIES)); } + let entry_type = entry.header().entry_type(); + if strict { + use tar::EntryType; + let is_meta = matches!( + entry_type, + EntryType::XGlobalHeader + | EntryType::XHeader + | EntryType::GNULongName + | EntryType::GNULongLink + ); + if !is_meta { + let raw = entry.path()?.to_string_lossy().to_string(); + let is_dir = entry_type == EntryType::Directory; + if !(entry_type == EntryType::Regular || is_dir) { + return Err(ArchiveError::NonCanonical(raw)); + } + // The installers strip the FIRST segment whatever it is; + // only `package/` maps onto the decoded key space. + let stripped = if is_dir && raw.trim_end_matches('/') == "package" { + None + } else if let Some(rest) = raw.strip_prefix("package/") { + Some(rest) + } else { + return Err(ArchiveError::NonCanonical(raw)); + }; + if let Some(rest) = stripped { + let Some(key) = canonical_member_key(rest) else { + return Err(ArchiveError::NonCanonical(raw)); + }; + let collides = if is_dir { + file_keys.contains(&key) + } else { + dir_keys.contains(&key) || !file_keys.insert(key.clone()) + }; + if collides { + return Err(ArchiveError::NonCanonical(raw)); + } + if is_dir { + dir_keys.insert(key); + } + } + } + } + // Only regular files. Skip directories, symlinks, hardlinks, etc. - if entry.header().entry_type() != tar::EntryType::Regular { + if entry_type != tar::EntryType::Regular { continue; } @@ -289,6 +396,32 @@ mod tests { write_raw_tar_gz(path, &[raw_entry(name, data.len() as u64, data)]); } + /// The in-memory reader decodes exactly what the path reader decodes and + /// keeps the same path-safety gate (it is the same core). + #[test] + fn test_read_archive_bytes_matches_path_reader_and_gate() { + let dir = tempfile::tempdir().unwrap(); + let archive = dir.path().join("arc.tar.gz"); + write_archive( + &archive, + &[("package/index.js", b"hello"), ("package/lib/a.js", b"a")], + ); + let bytes = std::fs::read(&archive).unwrap(); + assert_eq!( + read_archive_bytes_to_map(&bytes).unwrap(), + read_archive_to_map(&archive).unwrap() + ); + + let bad = dir.path().join("bad.tar.gz"); + write_raw_archive(&bad, b"/etc/passwd", b"evil"); + let bytes = std::fs::read(&bad).unwrap(); + assert!(matches!( + read_archive_bytes_to_map(&bytes), + Err(ArchiveError::UnsafePath(_)) + )); + assert!(read_archive_bytes_to_map(b"not a gzip").is_err()); + } + #[test] fn test_read_archive_rejects_absolute_paths() { let dir = tempfile::tempdir().unwrap(); diff --git a/crates/socket-patch-core/src/utils/http.rs b/crates/socket-patch-core/src/utils/http.rs index 269f1c82..c96c6580 100644 --- a/crates/socket-patch-core/src/utils/http.rs +++ b/crates/socket-patch-core/src/utils/http.rs @@ -1,5 +1,25 @@ //! Small shared HTTP primitives. +/// Why [`read_capped_typed`] gave up — typed so a caller can tell a body cut +/// off mid-transfer (a transport failure, worth a retry) from a cap breach +/// (the same bytes would breach it again) without matching on wording. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum ReadCappedError { + /// The body stream failed before it ended (connection reset, timeout, + /// truncated `Content-Length`, …). + Truncated(String), + /// The declared or streamed size exceeded the cap. + CapExceeded(String), +} + +impl std::fmt::Display for ReadCappedError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Truncated(m) | Self::CapExceeded(m) => f.write_str(m), + } + } +} + /// Stream a response body into memory with a hard byte cap, rejecting both /// an over-large declared `Content-Length` and an actual stream that /// exceeds the cap mid-flight. `what` names the payload in error messages @@ -8,25 +28,38 @@ /// Hoisted from `api/client.rs` so the self-update downloader shares the /// exact cap semantics the vendor/artifact fetches already have. pub(crate) async fn read_capped( - mut resp: reqwest::Response, + resp: reqwest::Response, max: u64, what: &str, ) -> Result, String> { + read_capped_typed(resp, max, what) + .await + .map_err(|e| e.to_string()) +} + +/// [`read_capped`] with a typed error. +pub(crate) async fn read_capped_typed( + mut resp: reqwest::Response, + max: u64, + what: &str, +) -> Result, ReadCappedError> { if let Some(len) = resp.content_length() { if len > max { - return Err(format!( + return Err(ReadCappedError::CapExceeded(format!( "{what} too large: declared {len} bytes > {max} cap" - )); + ))); } } let mut bytes: Vec = Vec::new(); while let Some(chunk) = resp .chunk() .await - .map_err(|e| format!("error reading {what} body: {e}"))? + .map_err(|e| ReadCappedError::Truncated(format!("error reading {what} body: {e}")))? { if bytes.len() as u64 + chunk.len() as u64 > max { - return Err(format!("{what} exceeded {max}-byte cap mid-stream")); + return Err(ReadCappedError::CapExceeded(format!( + "{what} exceeded {max}-byte cap mid-stream" + ))); } bytes.extend_from_slice(&chunk); } diff --git a/crates/socket-patch-core/src/vendor/bun_binary.rs b/crates/socket-patch-core/src/vendor/bun_binary.rs index 3133e33f..4b426372 100644 --- a/crates/socket-patch-core/src/vendor/bun_binary.rs +++ b/crates/socket-patch-core/src/vendor/bun_binary.rs @@ -595,33 +595,75 @@ mod symlink_tests { #[cfg(test)] mod rebuild_tests { use super::*; - use crate::api::client::{ApiClient, ApiClientOptions}; use crate::hash::git_sha256::compute_git_sha256_from_bytes; use crate::vendor::state::carry_forward_wiring; + use crate::vendor::test_support as ts; use crate::vendor::{VendorServiceConfig, VendorSource}; - use base64::{engine::general_purpose::STANDARD, Engine}; - use sha2::Sha512; use wiremock::matchers::{method, path}; use wiremock::{Mock, MockServer, ResponseTemplate}; - /// A service outage can switch an existing UUID from a prebuilt archive - /// to a locally packed one. Different archive bytes must advance the - /// integrity snapshot without losing the pristine registry predecessor. - #[tokio::test] - async fn same_uuid_prebuilt_then_local_fallback_reverts_exact_binary_and_mirrors() { - const UUID: &str = "11111111-1111-4111-8111-111111111111"; - const PURL: &str = "pkg:npm/minimist@1.2.2"; - const BEFORE: &[u8] = b"module.exports = 'original';\n"; - const AFTER: &[u8] = b"module.exports = 'patched';\n"; - const PACKAGE: &[u8] = br#"{"name":"minimist","version":"1.2.2"}"#; - let root = tempfile::tempdir().unwrap(); - let original = include_bytes!("../../tests/fixtures/bun-lockb/1.1.45-extensions/bun.lockb"); - std::fs::write(root.path().join(LOCK), original).unwrap(); - let installed = root.path().join("node_modules/minimist"); + const UUID: &str = "11111111-1111-4111-8111-111111111111"; + const PURL: &str = "pkg:npm/minimist@1.2.2"; + const BEFORE: &[u8] = b"module.exports = 'original';\n"; + const AFTER: &[u8] = b"module.exports = 'patched';\n"; + const PACKAGE: &[u8] = br#"{"name":"minimist","version":"1.2.2"}"#; + const ORIGINAL: &[u8] = + include_bytes!("../../tests/fixtures/bun-lockb/1.1.45-extensions/bun.lockb"); + + pub(super) struct Fixture { + tmp: tempfile::TempDir, + record: PatchRecord, + } + + impl Fixture { + fn root(&self) -> &Path { + self.tmp.path() + } + fn installed(&self) -> PathBuf { + self.root().join("node_modules/minimist") + } + } + + impl ts::FlipFixture for Fixture { + fn flip_root(&self) -> &Path { + self.root() + } + fn flip_key(&self) -> String { + PURL.to_string() + } + fn flip_uuid(&self) -> String { + UUID.to_string() + } + fn flip_artifact_rel(&self) -> String { + format!(".socket/vendor/npm/{UUID}/minimist-1.2.2.tgz") + } + /// The lock plus every workspace mirror the ledger records. + fn flip_files(&self) -> Vec { + let mut files = vec![LOCK.to_string()]; + if let Ok(bytes) = std::fs::read(self.root().join(".socket/vendor/state.json")) { + let state: crate::vendor::state::VendorState = + serde_json::from_slice(&bytes).unwrap(); + for entry in state.entries.values() { + for rec in entry.wiring.iter().filter(|r| r.kind == MIRROR_KIND) { + files.push(rec.file.clone()); + } + } + } + files.sort(); + files.dedup(); + files + } + } + + pub(super) async fn flip_fixture() -> Fixture { + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + std::fs::write(root.join(LOCK), ORIGINAL).unwrap(); + let installed = root.join("node_modules/minimist"); std::fs::create_dir_all(&installed).unwrap(); std::fs::write(installed.join("package.json"), PACKAGE).unwrap(); std::fs::write(installed.join("index.js"), BEFORE).unwrap(); - let blobs = root.path().join(".socket/blobs"); + let blobs = root.join(".socket/blobs"); std::fs::create_dir_all(&blobs).unwrap(); let after_hash = compute_git_sha256_from_bytes(AFTER); std::fs::write(blobs.join(&after_hash), AFTER).unwrap(); @@ -631,6 +673,30 @@ mod rebuild_tests { }}, "vulnerabilities": {}, "description": "", "license": "MIT", "tier": "free", })) .unwrap(); + Fixture { tmp, record } + } + + pub(super) async fn flip_run(fx: &Fixture, cfg: Option<&VendorServiceConfig>) -> VendorOutcome { + let blobs = fx.root().join(".socket/blobs"); + vendor( + PURL, + &fx.installed(), + fx.root(), + &fx.record, + &PatchSources::blobs_only(&blobs), + "", + false, + false, + cfg, + ) + .await + } + + ts::npm_flip_suite!(flip_suite, Fixture, flip_fixture, flip_run); + + /// A prebuilt archive whose tar headers deliberately differ from the + /// local packer's (so its bytes never equal a local build's). + fn prebuilt_archive() -> Vec { let mut tar = tar::Builder::new(flate2::write::GzEncoder::new( Vec::new(), flate2::Compression::default(), @@ -645,63 +711,77 @@ mod rebuild_tests { tar.append_data(&mut header, format!("package/{name}"), bytes) .unwrap(); } - let archive = tar.into_inner().unwrap().finish().unwrap(); - let server = MockServer::start().await; - let url = format!("{}/minimist.tgz", server.uri()); + tar.into_inner().unwrap().finish().unwrap() + } + + async fn mount_403(server: &MockServer) { + server.reset().await; Mock::given(method("POST")) - .and(path("/v0/orgs/acme/patches/package")) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "results": {UUID: {"status": "granted", "url": url, "artifacts": [{ - "kind": "tarball", "url": url, - "integrity": {"sha512": format!("sha512-{}", STANDARD.encode(Sha512::digest(&archive)))}, - }]}}, - }))) - .mount(&server) - .await; - Mock::given(method("GET")) - .and(path("/minimist.tgz")) - .respond_with(ResponseTemplate::new(200).set_body_bytes(archive.clone())) - .mount(&server) + .and(path(ts::PACKAGE_PATH)) + .respond_with(ResponseTemplate::new(403)) + .mount(server) .await; - let config = VendorServiceConfig { - source: VendorSource::Auto, - client: Some(ApiClient::new(ApiClientOptions { - api_url: server.uri(), - api_token: Some("sktsec_placeholder_value_for_tests_api".into()), - use_public_proxy: false, - org_slug: Some("acme".into()), - })), - use_public_proxy: false, - vendor_url: None, - patch_server_url: None, - offline: false, - }; + } + + /// A service outage (here a 403) after a prebuilt vendor: the committed + /// prebuilt archive is anchored by the ledger, so the re-run reuses it — + /// entry `None`, bun.lockb and every workspace mirror byte-unchanged, no + /// request (the flip no longer re-pins). + #[tokio::test] + async fn same_uuid_prebuilt_then_outage_reuses_the_committed_archive() { + let archive = prebuilt_archive(); + let server = MockServer::start().await; + ts::mount_granted(&server, UUID, "minimist-1.2.2.tgz", &archive).await; + let fx = flip_fixture().await; + let config = ts::service_cfg(&server.uri(), VendorSource::Auto, false); + let (result, entry, warnings) = ts::expect_done(flip_run(&fx, Some(&config)).await); + assert!(result.success, "{result:?}"); + assert!(ts::has_warning(&warnings, "vendor_prebuilt_downloaded")); + ts::persist(fx.root(), PURL, entry.unwrap()).await; + let before = ts::snapshot(&fx).await; + assert!( + before.len() > 2, + "fixture wires workspace mirrors: {:?}", + before.iter().map(|b| &b.0).collect::>() + ); + + mount_403(&server).await; + let (result, entry, warnings) = ts::expect_done(flip_run(&fx, Some(&config)).await); + assert!(result.success, "{result:?}"); + assert!(entry.is_none(), "in sync: nothing re-pinned"); + assert!(warnings.is_empty(), "{warnings:?}"); + assert_eq!(ts::snapshot(&fx).await, before); + assert_eq!(ts::request_count(&server).await, 0); + } + + /// With the canonical tarball GONE, an outage switches the same UUID + /// from a prebuilt archive to a locally packed one. Different archive + /// bytes must advance the integrity snapshot without losing the pristine + /// registry predecessor, and revert must restore everything exactly. + #[tokio::test] + async fn same_uuid_prebuilt_then_local_fallback_reverts_exact_binary_and_mirrors() { + let archive = prebuilt_archive(); + let server = MockServer::start().await; + ts::mount_granted(&server, UUID, "minimist-1.2.2.tgz", &archive).await; + let fx = flip_fixture().await; + let root = fx.tmp.path(); + let config = ts::service_cfg(&server.uri(), VendorSource::Auto, false); let mut prior: Option = None; for prebuilt in [true, false] { if !prebuilt { - server.reset().await; - Mock::given(method("POST")) - .and(path("/v0/orgs/acme/patches/package")) - .respond_with(ResponseTemplate::new(403)) - .mount(&server) - .await; + mount_403(&server).await; + // The committed artifact is missing (deleted, never + // committed): reuse cannot apply, so acquisition runs. + std::fs::remove_file( + root.join(format!(".socket/vendor/npm/{UUID}/minimist-1.2.2.tgz")), + ) + .unwrap(); } let VendorOutcome::Done { result, entry: Some(mut entry), warnings, - } = vendor( - PURL, - &installed, - root.path(), - &record, - &PatchSources::blobs_only(&blobs), - "", - false, - false, - Some(&config), - ) - .await + } = flip_run(&fx, Some(&config)).await else { panic!("vendoring must write a new binary snapshot"); }; @@ -719,30 +799,28 @@ mod rebuild_tests { assert_eq!(binary.len(), 1, "discard the superseded integrity snapshot"); assert_eq!(binary[0].original, previous.wiring[0].original); } - let lock = BunLockb::parse(&std::fs::read(root.path().join(LOCK)).unwrap()).unwrap(); + let lock = BunLockb::parse(&std::fs::read(root.join(LOCK)).unwrap()).unwrap(); let binary = entry.wiring.iter().find(|r| r.kind == KIND).unwrap(); let id = binary.key.as_ref().unwrap().parse().unwrap(); assert!(lock .matches_snapshot(id, binary.new.as_ref().unwrap()) .unwrap()); - let bytes = std::fs::read(root.path().join(&entry.artifact.path)).unwrap(); + let bytes = std::fs::read(root.join(&entry.artifact.path)).unwrap(); assert_eq!(bytes == archive, prebuilt); for mirror in entry.wiring.iter().filter(|r| r.kind == MIRROR_KIND) { - assert_eq!( - std::fs::read(root.path().join(&mirror.file)).unwrap(), - bytes - ); + assert_eq!(std::fs::read(root.join(&mirror.file)).unwrap(), bytes); } + ts::persist(root, PURL, entry.clone()).await; prior = Some(entry); } let entry = prior.unwrap(); - let outcome = revert(&entry, root.path(), RevertOpts::new(false)).await; + let outcome = revert(&entry, root, RevertOpts::new(false)).await; assert!(outcome.success, "{outcome:?}"); assert!(outcome.warnings.is_empty(), "{outcome:?}"); - assert_eq!(std::fs::read(root.path().join(LOCK)).unwrap(), original); - assert!(!root.path().join(&entry.artifact.path).exists()); + assert_eq!(std::fs::read(root.join(LOCK)).unwrap(), ORIGINAL); + assert!(!root.join(&entry.artifact.path).exists()); for mirror in entry.wiring.iter().filter(|r| r.kind == MIRROR_KIND) { - assert!(!root.path().join(&mirror.file).exists()); + assert!(!root.join(&mirror.file).exists()); } } } diff --git a/crates/socket-patch-core/src/vendor/bun_lock.rs b/crates/socket-patch-core/src/vendor/bun_lock.rs index 955398e0..4fa45bdb 100644 --- a/crates/socket-patch-core/src/vendor/bun_lock.rs +++ b/crates/socket-patch-core/src/vendor/bun_lock.rs @@ -434,10 +434,12 @@ pub(crate) async fn vendor_bun( // The sha512 of the artifact already sitting at the target path, if // any — the one witness a digest-less in-sync tuple (see `classify`) // still has of the digest Bun dropped: the lock line was written from - // these bytes. Read BEFORE staging, which overwrites the file; a - // missing or non-regular path (a `repair` rebuild after deletion, a - // FIFO) yields `None`, which the in-sync check below treats as "not - // provably the same bytes". Only a digest-less 2-tuple of OURS at this + // these bytes. With a ledger anchor the shared pipeline reuses exactly + // these bytes (so this equals the staged integrity); without one it + // keeps today's behavior. Read BEFORE staging, which may overwrite the + // file; a missing or non-regular path (a `repair` rebuild after + // deletion, a FIFO) yields `None`, which the in-sync check below treats + // as "not provably the same bytes". Only a digest-less 2-tuple of OURS at this // path can consume it, so every other re-run skips the read + hash. let has_digestless_own_tuple = entries.iter().any(|e| { e.elems.len() == 2 @@ -557,10 +559,13 @@ pub(crate) async fn vendor_bun( continue; } // Same path, different digest — or a digest-less line - // whose artifact was missing or differed before staging - // (a `repair` rebuild, a service-prebuilt ↔ local-pack - // flip): re-pinned below like any stale tuple of ours, - // so the returned entry carries the rebuilt artifact's + // whose artifact was missing or differed before staging. + // A source flip no longer reaches here when the ledger + // verifies the committed tarball (it is reused, so the + // digests agree); only a missing, corrupt or unanchored + // artifact (a `repair` rebuild, a lost state.json) is + // re-pinned below like any stale tuple of ours, so the + // returned entry carries the rebuilt artifact's // fingerprint (`carry_forward_wiring` refills the // pristine original from the entry it replaces). _ => {} @@ -592,8 +597,9 @@ pub(crate) async fn vendor_bun( if !changed { // Every instance already points at this uuid with the packed // integrity (or with the digest Bun dropped, now re-pinned): in - // sync. The tarball re-pack above was byte-identical by - // determinism; synthesize AlreadyPatched and record nothing. The + // sync. The integrity is that of the reused committed tarball (or of + // a fresh acquisition that reproduced it when reuse missed); + // synthesize AlreadyPatched and record nothing. The // heal is the one write of an in-sync run, and a failed write // leaves the still-installable digest-less lock — reported as the // failure it is, like every other lock write below. @@ -1144,6 +1150,329 @@ mod tests { } } + // ── source-flip / outage idempotence (vendor::test_support::npm_flip_suite) ── + + impl crate::vendor::test_support::FlipFixture for Fixture { + fn flip_root(&self) -> &Path { + self.root() + } + fn flip_key(&self) -> String { + "pkg:npm/left-pad@1.3.0".to_string() + } + fn flip_uuid(&self) -> String { + self.record.uuid.clone() + } + fn flip_artifact_rel(&self) -> String { + self.rel_tgz() + } + fn flip_files(&self) -> Vec { + vec![BUN_LOCK.to_string(), "package.json".to_string()] + } + } + + async fn flip_run( + fx: &Fixture, + cfg: Option<&crate::vendor::VendorServiceConfig>, + ) -> VendorOutcome { + let blobs = fx.root().join(".socket/blobs"); + vendor_bun( + "pkg:npm/left-pad@1.3.0", + &fx.installed, + fx.root(), + &fx.record, + &PatchSources::blobs_only(&blobs), + "2026-06-09T00:00:00Z", + false, + false, + cfg, + ) + .await + } + + async fn flip_fixture() -> Fixture { + fixture_with(BN3_BEFORE_LOCK, "node_modules/left-pad").await + } + + async fn flip_fixture_nested() -> Fixture { + fixture_with( + BN4C_BEFORE_LOCK, + "node_modules/haspad/node_modules/left-pad", + ) + .await + } + + crate::vendor::test_support::npm_flip_suite!(flip_suite, Fixture, flip_fixture, flip_run); + crate::vendor::test_support::npm_flip_suite!( + flip_suite_nested, + Fixture, + flip_fixture_nested, + flip_run + ); + + /// Run 1 from the service (`alt` = a re-encoding of the local build), + /// persisted like the CLI does; the server is left answering 503. + async fn bun_service_vendored() -> (Fixture, wiremock::MockServer, Vec, String) { + use crate::vendor::test_support as ts; + let probe = flip_fixture().await; + let _ = expect_done(flip_run(&probe, None).await); + let local = tokio::fs::read(probe.root().join(probe.rel_tgz())) + .await + .unwrap(); + let alt = ts::regzip(&local); + let server = wiremock::MockServer::start().await; + ts::mount_granted(&server, UUID, "left-pad-1.3.0.tgz", &alt).await; + let fx = flip_fixture().await; + let cfg = ts::service_cfg(&server.uri(), crate::vendor::VendorSource::Auto, false); + let (r, e, _) = expect_done(flip_run(&fx, Some(&cfg)).await); + assert!(r.success, "{:?}", r.error); + let e = e.unwrap(); + ts::persist(fx.root(), "pkg:npm/left-pad@1.3.0", e).await; + let wired = fx.read_lock().await; + assert!(wired.contains(&ts::sri(&alt))); + server.reset().await; + ts::mount_503(&server).await; + (fx, server, alt, wired) + } + + async fn bun_outage_rerun( + fx: &Fixture, + server: &wiremock::MockServer, + ) -> (ApplyResult, Option, Vec) { + let cfg = crate::vendor::test_support::service_cfg( + &server.uri(), + crate::vendor::VendorSource::Auto, + false, + ); + expect_done(flip_run(fx, Some(&cfg)).await) + } + + /// Digest-less 2-tuple (a Bun < 1.3.10 re-save) over a SERVICE-built + /// artifact, re-run during an outage: the reused committed bytes are the + /// witness, so the digest heals back to the service SRI with no record, + /// no request and no rebuild. + #[tokio::test] + async fn digestless_tuple_over_service_artifact_heals_under_outage() { + let (fx, server, alt, wired) = bun_service_vendored().await; + let state = crate::vendor::state::load_state(fx.root()).await.unwrap(); + let entry = state.entries.values().next().unwrap().clone(); + let (_, digestless) = digestless_lock(&wired, &entry); + tokio::fs::write(fx.root().join(BUN_LOCK), &digestless) + .await + .unwrap(); + let (r, e, w) = bun_outage_rerun(&fx, &server).await; + assert!(r.success, "{:?}", r.error); + assert!(e.is_none(), "healed in place, nothing recorded"); + assert!(w.is_empty(), "{w:?}"); + assert_eq!( + fx.read_lock().await, + wired, + "digest healed to the service SRI" + ); + assert_eq!( + tokio::fs::read(fx.root().join(fx.rel_tgz())).await.unwrap(), + alt + ); + assert_eq!(crate::vendor::test_support::request_count(&server).await, 0); + } + + /// F6: a re-gzipped committed tarball with the ledger untouched fails + /// the anchor and is rebuilt (the lock is re-pinned to the local build). + #[tokio::test] + async fn bun_regzipped_tarball_with_untouched_ledger_is_not_reused() { + let (fx, server, alt, _) = bun_service_vendored().await; + let reencoded = crate::vendor::test_support::regzip_at(&alt, flate2::Compression::best()); + tokio::fs::write(fx.root().join(fx.rel_tgz()), &reencoded) + .await + .unwrap(); + let (r, e, _) = bun_outage_rerun(&fx, &server).await; + assert!(r.success, "{:?}", r.error); + assert!(e.is_some(), "not reused: re-acquired and re-pinned"); + let lock = fx.read_lock().await; + assert!(!lock.contains(&crate::vendor::test_support::sri(&reencoded))); + assert!(lock.contains(&fx.actual_integrity().await)); + } + + /// F7: a patched member edited AND the ledger sha forged to match fails + /// the afterHash check; the tampered bytes are never pinned. + #[tokio::test] + async fn bun_forged_ledger_over_edited_patched_member_is_not_reused() { + use sha2::Sha256; + let (fx, server, _, _) = bun_service_vendored().await; + let evil = { + let mut b = tar::Builder::new(flate2::write::GzEncoder::new( + Vec::new(), + flate2::Compression::default(), + )); + for (n, data) in [ + ( + "package/package.json", + br#"{"name":"left-pad","version":"1.3.0"}"#.to_vec(), + ), + ("package/index.js", b"module.exports = 'evil';\n".to_vec()), + ] { + let mut h = tar::Header::new_gnu(); + h.set_size(data.len() as u64); + h.set_mode(0o644); + h.set_cksum(); + b.append_data(&mut h, n, data.as_slice()).unwrap(); + } + b.into_inner().unwrap().finish().unwrap() + }; + tokio::fs::write(fx.root().join(fx.rel_tgz()), &evil) + .await + .unwrap(); + let mut state = crate::vendor::state::load_state(fx.root()).await.unwrap(); + let e = state.entries.get_mut("pkg:npm/left-pad@1.3.0").unwrap(); + e.artifact.sha256 = hex::encode(Sha256::digest(&evil)); + e.artifact.size = Some(evil.len() as u64); + crate::vendor::state::save_state(fx.root(), &state) + .await + .unwrap(); + let (r, e, _) = bun_outage_rerun(&fx, &server).await; + assert!(r.success, "{:?}", r.error); + assert!(e.is_some(), "not reused: rebuilt and re-pinned"); + let lock = fx.read_lock().await; + assert!(!lock.contains(&crate::vendor::test_support::sri(&evil))); + assert!(lock.contains(&fx.actual_integrity().await)); + } + + /// F9: a new record uuid acquires under the new uuid dir; the old + /// uuid's committed artifact is left alone. + #[tokio::test] + async fn bun_new_record_uuid_acquires_under_the_new_uuid_dir() { + const NEXT: &str = "1a2b3c4d-5e6f-4a1b-8c2d-3e4f5a6b7c8d"; + let (mut fx, server, alt, _) = bun_service_vendored().await; + fx.record.uuid = NEXT.to_string(); + let (r, e, _) = bun_outage_rerun(&fx, &server).await; + assert!(r.success, "{:?}", r.error); + assert_eq!( + e.expect("a new uuid re-wires").artifact.path, + format!(".socket/vendor/npm/{NEXT}/left-pad-1.3.0.tgz") + ); + assert_eq!( + tokio::fs::read( + fx.root() + .join(format!(".socket/vendor/npm/{UUID}/left-pad-1.3.0.tgz")) + ) + .await + .unwrap(), + alt + ); + } + + /// F8: no ledger, no anchor — today's re-pin (the documented residual). + #[tokio::test] + async fn bun_missing_ledger_keeps_todays_repin() { + let (fx, server, alt, _) = bun_service_vendored().await; + tokio::fs::remove_file(fx.root().join(".socket/vendor/state.json")) + .await + .unwrap(); + let (r, e, w) = bun_outage_rerun(&fx, &server).await; + assert!(r.success, "{:?}", r.error); + assert!(e.is_some()); + assert!(w.iter().any(|w| w.code == "vendor_prebuilt_unavailable")); + assert!(!fx + .read_lock() + .await + .contains(&crate::vendor::test_support::sri(&alt))); + } + + /// F10: the tuple reset to the registry line (a `bun install` relock) + /// with tarball + ledger intact is re-pinned to the COMMITTED bytes. + #[tokio::test] + async fn bun_relocked_tuple_is_repinned_to_the_committed_bytes_without_network() { + let (fx, server, alt, wired) = bun_service_vendored().await; + tokio::fs::write(fx.root().join(BUN_LOCK), BN3_BEFORE_LOCK) + .await + .unwrap(); + let (r, e, w) = bun_outage_rerun(&fx, &server).await; + assert!(r.success, "{:?}", r.error); + assert!(e.is_some(), "re-wired (Applied)"); + assert!(w.is_empty(), "{w:?}"); + assert_eq!(fx.read_lock().await, wired); + assert_eq!( + tokio::fs::read(fx.root().join(fx.rel_tgz())).await.unwrap(), + alt + ); + assert_eq!(crate::vendor::test_support::request_count(&server).await, 0); + } + + /// F11: a FIFO at the artifact path never wedges and is never reused. + #[cfg(unix)] + #[tokio::test] + async fn bun_fifo_artifact_is_not_reused() { + let (fx, server, alt, _) = bun_service_vendored().await; + let tgz = fx.root().join(fx.rel_tgz()); + tokio::fs::remove_file(&tgz).await.unwrap(); + let c = std::ffi::CString::new(tgz.to_str().unwrap()).unwrap(); + assert_eq!(unsafe { libc::mkfifo(c.as_ptr(), 0o644) }, 0); + let (r, e, _) = tokio::time::timeout( + std::time::Duration::from_secs(30), + bun_outage_rerun(&fx, &server), + ) + .await + .expect("a FIFO must never wedge a vendor re-run"); + assert!(r.success, "{:?}", r.error); + assert!(e.is_some(), "not reused"); + assert!(!fx + .read_lock() + .await + .contains(&crate::vendor::test_support::sri(&alt))); + } + + /// F12: a package.json-rewriting patch reuses the committed bytes and the + /// lock stays byte-identical across a flip. + #[tokio::test] + async fn bun_package_json_patch_reuse_is_byte_stable() { + use crate::vendor::test_support as ts; + async fn pkg_fixture() -> Fixture { + let mut fx = flip_fixture().await; + let before = tokio::fs::read(fx.installed.join("package.json")) + .await + .unwrap(); + let after: &[u8] = br#"{"name":"left-pad","version":"1.3.0","description":"patched"}"#; + let after_hash = compute_git_sha256_from_bytes(after); + tokio::fs::write(fx.root().join(".socket/blobs").join(&after_hash), after) + .await + .unwrap(); + fx.record.files.insert( + "package/package.json".to_string(), + PatchFileInfo { + before_hash: compute_git_sha256_from_bytes(&before), + after_hash, + }, + ); + fx + } + let probe = pkg_fixture().await; + let _ = expect_done(flip_run(&probe, None).await); + let alt = ts::regzip( + &tokio::fs::read(probe.root().join(probe.rel_tgz())) + .await + .unwrap(), + ); + let server = wiremock::MockServer::start().await; + ts::mount_granted(&server, UUID, "left-pad-1.3.0.tgz", &alt).await; + let fx = pkg_fixture().await; + let cfg = ts::service_cfg(&server.uri(), crate::vendor::VendorSource::Auto, false); + let (r, e, _) = expect_done(flip_run(&fx, Some(&cfg)).await); + assert!(r.success, "{:?}", r.error); + ts::persist(fx.root(), "pkg:npm/left-pad@1.3.0", e.unwrap()).await; + let lock1 = fx.read_lock().await; + server.reset().await; + ts::mount_503(&server).await; + let (r, e, w) = expect_done(flip_run(&fx, Some(&cfg)).await); + assert!(r.success, "{:?}", r.error); + assert!(e.is_none()); + // The (pre-existing, every-run) dependency-object advisory only. + assert!( + w.iter().all(|w| w.code == "vendor_dep_manifest_stale"), + "{w:?}" + ); + assert_eq!(fx.read_lock().await, lock1); + assert_eq!(ts::request_count(&server).await, 0); + } + async fn fixture_with(lock: &str, installed_rel: &str) -> Fixture { let tmp = tempfile::tempdir().unwrap(); let root = tmp.path(); diff --git a/crates/socket-patch-core/src/vendor/cargo.rs b/crates/socket-patch-core/src/vendor/cargo.rs index 288eab0f..57640c3d 100644 --- a/crates/socket-patch-core/src/vendor/cargo.rs +++ b/crates/socket-patch-core/src/vendor/cargo.rs @@ -2107,12 +2107,15 @@ mod tests { fn cargo_service_cfg(uri: &str, source: VendorSource, offline: bool) -> VendorServiceConfig { VendorServiceConfig { source, - client: Some(ApiClient::new(ApiClientOptions { - api_url: uri.to_string(), - api_token: Some("sktsec_placeholder_value_for_tests_api".into()), - use_public_proxy: false, - org_slug: Some("acme".into()), - })), + client: Some( + ApiClient::new(ApiClientOptions { + api_url: uri.to_string(), + api_token: Some("sktsec_placeholder_value_for_tests_api".into()), + use_public_proxy: false, + org_slug: Some("acme".into()), + }) + .with_vendor_retry(crate::api::client::VendorRetryPolicy::none()), + ), use_public_proxy: false, vendor_url: None, patch_server_url: None, @@ -3162,6 +3165,88 @@ mod tests { assert!(stage.exists(), "the stage is left for the caller's cleanup"); } + // ── source-flip regression: the hot path decides "in sync" from the + // COMMITTED copy before any service call, so a service ↔ local flip + // between runs is a byte-identical no-op with no request. ── + + async fn flip_run( + root: &Path, + blobs: &Path, + pristine: &Path, + record: &PatchRecord, + uri: &str, + ) -> (ApplyResult, Option, Vec) { + let sources = PatchSources::blobs_only(blobs); + expect_done( + vendor_cargo_crate( + PURL, + pristine, + root, + record, + &sources, + "2026-06-09T00:00:00Z", + false, + false, + Some(&cargo_service_cfg(uri, VendorSource::Auto, false)), + ) + .await, + ) + } + + /// A service crate that differs from the local build in NON-patched bytes. + fn flip_service_crate() -> Vec { + make_crate_tgz( + "cfg-if-1.0.4", + &[ + ("src/lib.rs", PATCHED), + ( + "Cargo.toml", + b"[package]\nname = \"cfg-if\"\nversion = \"1.0.4\"\n# service\n", + ), + ("README.service.md", b"built by the service\n"), + ], + ) + } + + #[tokio::test] + async fn flip_local_then_service_is_noop() { + use crate::vendor::test_support as ts; + let (dir, blobs, pristine, record) = fixture().await; + let root = dir.path(); + let down = wiremock::MockServer::start().await; + ts::mount_503(&down).await; + let (r1, e1, _) = flip_run(root, &blobs, &pristine, &record, &down.uri()).await; + assert!(r1.success && e1.is_some()); + let before = ts::tree_snapshot(root); + let up = wiremock::MockServer::start().await; + let tgz = flip_service_crate(); + mount_cargo_granted(&up, &sri_sha512(&tgz), &tgz).await; + let (r2, e2, w2) = flip_run(root, &blobs, &pristine, &record, &up.uri()).await; + assert!(r2.success && e2.is_none() && r2.files_patched.is_empty() && w2.is_empty()); + assert_eq!(ts::tree_snapshot(root), before, "tree byte-identical"); + assert_eq!(ts::request_count(&up).await, 0); + } + + #[tokio::test] + async fn flip_service_then_local_is_noop() { + use crate::vendor::test_support as ts; + let (dir, blobs, pristine, record) = fixture().await; + let root = dir.path(); + let up = wiremock::MockServer::start().await; + let tgz = flip_service_crate(); + mount_cargo_granted(&up, &sri_sha512(&tgz), &tgz).await; + let (r1, e1, w1) = flip_run(root, &blobs, &pristine, &record, &up.uri()).await; + assert!(r1.success && e1.is_some()); + assert!(ts::has_warning(&w1, "vendor_prebuilt_downloaded")); + let before = ts::tree_snapshot(root); + let down = wiremock::MockServer::start().await; + ts::mount_503(&down).await; + let (r2, e2, w2) = flip_run(root, &blobs, &pristine, &record, &down.uri()).await; + assert!(r2.success && e2.is_none() && w2.is_empty()); + assert_eq!(ts::tree_snapshot(root), before); + assert_eq!(ts::request_count(&down).await, 0); + } + /// `--vendor-source=service` with no configured client must fail /// closed, never quietly build locally. #[tokio::test] diff --git a/crates/socket-patch-core/src/vendor/composer_lock.rs b/crates/socket-patch-core/src/vendor/composer_lock.rs index e69740f8..22b7e2a9 100644 --- a/crates/socket-patch-core/src/vendor/composer_lock.rs +++ b/crates/socket-patch-core/src/vendor/composer_lock.rs @@ -1744,12 +1744,15 @@ mod tests { fn composer_service_cfg(uri: &str, source: VendorSource, offline: bool) -> VendorServiceConfig { VendorServiceConfig { source, - client: Some(ApiClient::new(ApiClientOptions { - api_url: uri.to_string(), - api_token: Some("sktsec_placeholder_value_for_tests_api".into()), - use_public_proxy: false, - org_slug: Some("acme".into()), - })), + client: Some( + ApiClient::new(ApiClientOptions { + api_url: uri.to_string(), + api_token: Some("sktsec_placeholder_value_for_tests_api".into()), + use_public_proxy: false, + org_slug: Some("acme".into()), + }) + .with_vendor_retry(crate::api::client::VendorRetryPolicy::none()), + ), use_public_proxy: false, vendor_url: None, patch_server_url: None, @@ -3486,6 +3489,116 @@ mod tests { ); } + // ── source-flip regression: the hot path decides "in sync" from the + // COMMITTED copy before any service call, so a service ↔ local flip + // between runs is a byte-identical no-op with no request. ── + + fn flip_service_zip() -> Vec { + make_dist_zip( + "php-fig-log-f16e1d5", + &[ + ( + "composer.json", + b"{\"name\": \"psr/log\", \"_service\": true}\n", + ), + ("src/LoggerInterface.php", PATCHED), + ("SERVICE_ONLY.md", b"x\n"), + ], + ) + } + + async fn flip( + root: &Path, + blobs: &Path, + installed: &Path, + record: &PatchRecord, + uri: &str, + source: VendorSource, + ) -> VendorOutcome { + vendor_with_service( + root, + blobs, + installed, + record, + &composer_service_cfg(uri, source, false), + ) + .await + } + + #[tokio::test] + async fn flip_local_then_service_is_noop() { + use crate::vendor::test_support as ts; + let lock = lock_value("psr/log", "3.0.2", false); + let (dir, blobs, installed, record) = fixture(&lock).await; + let root = dir.path(); + let down = wiremock::MockServer::start().await; + ts::mount_503(&down).await; + let (r1, e1, _) = unwrap_done( + flip( + root, + &blobs, + &installed, + &record, + &down.uri(), + VendorSource::Auto, + ) + .await, + ); + assert!(r1.success && e1.is_some()); + let before = ts::tree_snapshot(root); + let up = wiremock::MockServer::start().await; + let z = flip_service_zip(); + mount_composer_granted(&up, &sri_sha512(&z), &z).await; + let (r2, e2, w2) = unwrap_done( + flip( + root, + &blobs, + &installed, + &record, + &up.uri(), + VendorSource::Auto, + ) + .await, + ); + assert!(r2.success && e2.is_none() && w2.is_empty()); + assert_eq!(ts::tree_snapshot(root), before); + assert_eq!(ts::request_count(&up).await, 0); + } + + #[tokio::test] + async fn flip_service_then_local_is_noop() { + use crate::vendor::test_support as ts; + let lock = lock_value("psr/log", "3.0.2", false); + let (dir, blobs, installed, record) = fixture(&lock).await; + let root = dir.path(); + let up = wiremock::MockServer::start().await; + let z = flip_service_zip(); + mount_composer_granted(&up, &sri_sha512(&z), &z).await; + let (r1, e1, w1) = unwrap_done( + flip( + root, + &blobs, + &installed, + &record, + &up.uri(), + VendorSource::Auto, + ) + .await, + ); + assert!(r1.success && e1.is_some()); + assert!(ts::has_warning(&w1, "vendor_prebuilt_downloaded")); + let before = ts::tree_snapshot(root); + let down = wiremock::MockServer::start().await; + ts::mount_503(&down).await; + for source in [VendorSource::Auto, VendorSource::Service] { + let (r2, e2, w2) = + unwrap_done(flip(root, &blobs, &installed, &record, &down.uri(), source).await); + assert!(r2.success && e2.is_none() && w2.is_empty(), "{source:?}"); + assert_eq!(ts::tree_snapshot(root), before, "{source:?}"); + } + assert_eq!(ts::request_count(&down).await, 0); + } + /// An integrity mismatch is a hard failure under `auto` too — /// never a quiet local-build fallback (service_fetch's contract). #[tokio::test] diff --git a/crates/socket-patch-core/src/vendor/gem.rs b/crates/socket-patch-core/src/vendor/gem.rs index cb8e6985..3debb926 100644 --- a/crates/socket-patch-core/src/vendor/gem.rs +++ b/crates/socket-patch-core/src/vendor/gem.rs @@ -4653,12 +4653,15 @@ mod tests { fn gem_service_cfg(uri: &str, source: VendorSource, offline: bool) -> VendorServiceConfig { VendorServiceConfig { source, - client: Some(ApiClient::new(ApiClientOptions { - api_url: uri.to_string(), - api_token: Some("sktsec_placeholder_value_for_tests_api".into()), - use_public_proxy: false, - org_slug: Some("acme".into()), - })), + client: Some( + ApiClient::new(ApiClientOptions { + api_url: uri.to_string(), + api_token: Some("sktsec_placeholder_value_for_tests_api".into()), + use_public_proxy: false, + org_slug: Some("acme".into()), + }) + .with_vendor_retry(crate::api::client::VendorRetryPolicy::none()), + ), use_public_proxy: false, vendor_url: None, patch_server_url: None, @@ -8119,6 +8122,91 @@ mod tests { ); } + // ── source-flip regression: the hot path decides "in sync" from the + // COMMITTED copy before any service call, so a service ↔ local flip + // between runs is a byte-identical no-op with no request. ── + + async fn flip_run( + root: &Path, + installed: &Path, + blobs: &Path, + record: &PatchRecord, + server: &wiremock::MockServer, + ) -> (ApplyResult, Option, Vec) { + let sources = PatchSources::blobs_only(blobs); + unwrap_done( + vendor_gem( + PURL, + installed, + root, + record, + &sources, + "2026-06-09T00:00:00Z", + false, + false, + Some(&gem_service_cfg(&server.uri(), VendorSource::Auto, false)), + ) + .await, + ) + } + + async fn flip_granted() -> wiremock::MockServer { + let gem = make_gem(&[("lib/rack.rb", PATCHED)]); + let sri = sri_sha512(&gem); + let stub_sri = sri_sha512(SERVICE_STUB); + let server = wiremock::MockServer::start().await; + mount_gem_granted(&server, &gem, &sri, Some((SERVICE_STUB, &stub_sri))).await; + server + } + + #[tokio::test] + async fn flip_local_then_service_is_noop() { + use crate::vendor::test_support as ts; + let (_tmp, root, installed, blobs, record) = fixture(GEMFILE_DIRECT, LOCK_DIRECT).await; + let down = wiremock::MockServer::start().await; + ts::mount_503(&down).await; + let (r1, e1, w1) = flip_run(&root, &installed, &blobs, &record, &down).await; + assert!(r1.success && e1.is_some()); + assert!(ts::has_warning(&w1, "vendor_prebuilt_unavailable")); + let before = ts::tree_snapshot(&root); + let up = flip_granted().await; + let (r2, e2, _) = flip_run(&root, &installed, &blobs, &record, &up).await; + assert!(r2.success, "{:?}", r2.error); + assert!( + e2.is_none(), + "re-run must be the in-sync no-op (entry None)" + ); + assert!(r2 + .files_verified + .iter() + .all(|f| f.status == VerifyStatus::AlreadyPatched)); + assert_eq!(before, ts::tree_snapshot(&root), "tree byte-identical"); + assert_eq!(ts::request_count(&up).await, 0); + } + + #[tokio::test] + async fn flip_service_then_local_is_noop() { + use crate::vendor::test_support as ts; + let (_tmp, root, installed, blobs, record) = fixture(GEMFILE_DIRECT, LOCK_DIRECT).await; + let up = flip_granted().await; + let (r1, e1, w1) = flip_run(&root, &installed, &blobs, &record, &up).await; + assert!(r1.success && e1.is_some()); + assert!(ts::has_warning(&w1, "vendor_prebuilt_downloaded")); + assert_eq!( + tokio::fs::read(copy_gemspec(&root)).await.unwrap(), + SERVICE_STUB + ); + let before = ts::tree_snapshot(&root); + let down = wiremock::MockServer::start().await; + ts::mount_503(&down).await; + let (r2, e2, w2) = flip_run(&root, &installed, &blobs, &record, &down).await; + assert!(r2.success); + assert!(e2.is_none()); + assert!(w2.is_empty(), "{w2:?}"); + assert_eq!(before, ts::tree_snapshot(&root)); + assert_eq!(ts::request_count(&down).await, 0); + } + /// Vendored from the service (served stub gemspec), the /// committed copy is lost and rebuilt LOCALLY (local gemspec → different /// tree). The ledger the CLI persists must carry the rebuilt tree's diff --git a/crates/socket-patch-core/src/vendor/golang.rs b/crates/socket-patch-core/src/vendor/golang.rs index e3fffd53..9a4aec1e 100644 --- a/crates/socket-patch-core/src/vendor/golang.rs +++ b/crates/socket-patch-core/src/vendor/golang.rs @@ -147,9 +147,6 @@ pub async fn vendor_go_module( wired && wired_version_ok && copy_matches_after_hashes(©_dir, &record.files).await; let mut warnings: Vec = Vec::new(); - if let Some(refusal) = service_offline_conflict(service) { - return refusal; - } // Hot path (mirrors cargo.rs / composer_lock.rs): already wired to this // uuid with the committed copy intact → touch nothing and never consult @@ -166,6 +163,15 @@ pub async fn vendor_go_module( warnings, ); } + // After the hot path (as in cargo.rs): an in-sync re-run acquires + // nothing, so `--vendor-source service --offline` must not refuse it. + // Gated on `copy_was_ok` rather than the return above so a dry run of an + // in-sync module previews the same success the real run reports. + if !copy_was_ok { + if let Some(refusal) = service_offline_conflict(service) { + return refusal; + } + } // Acquire the patched module: prefer the prebuilt module zip from the patch // service (download → verify → extract → wire the `replace`, no pristine @@ -1549,12 +1555,15 @@ mod tests { fn go_service_cfg(uri: &str, source: VendorSource, offline: bool) -> VendorServiceConfig { VendorServiceConfig { source, - client: Some(ApiClient::new(ApiClientOptions { - api_url: uri.to_string(), - api_token: Some("sktsec_placeholder_value_for_tests_api".into()), - use_public_proxy: false, - org_slug: Some("acme".into()), - })), + client: Some( + ApiClient::new(ApiClientOptions { + api_url: uri.to_string(), + api_token: Some("sktsec_placeholder_value_for_tests_api".into()), + use_public_proxy: false, + org_slug: Some("acme".into()), + }) + .with_vendor_retry(crate::api::client::VendorRetryPolicy::none()), + ), use_public_proxy: false, vendor_url: None, patch_server_url: None, @@ -2002,6 +2011,76 @@ mod tests { expect_refused(outcome, "vendor_service_offline_conflict"); } + /// An ALREADY-vendored module under `--offline` + `--vendor-source + /// service` is `already_vendored` (the hot path acquires nothing), as in + /// cargo and composer — the conflict only refuses real acquisition. + #[tokio::test] + async fn offline_service_mode_in_sync_rerun_is_already_vendored() { + let (dir, blobs, pristine, record) = fixture().await; + let root = dir.path(); + let (result, entry, _) = + expect_done(run_vendor(PURL, root, &blobs, &pristine, &record, false).await); + assert!(result.success, "{:?}", result.error); + assert!(entry.is_some()); + let gomod = tokio::fs::read(root.join("go.mod")).await.unwrap(); + let sources = PatchSources::blobs_only(&blobs); + let outcome = vendor_go_module( + PURL, + &pristine, + root, + &record, + &sources, + "2026-06-09T00:00:00Z", + false, + false, + Some(&go_service_cfg( + "http://127.0.0.1:1", + VendorSource::Service, + true, + )), + ) + .await; + let (result, entry, warnings) = expect_done(outcome); + assert!(result.success, "{:?}", result.error); + assert!(entry.is_none(), "in sync: nothing recorded"); + assert!(warnings.is_empty(), "{warnings:?}"); + assert_eq!(tokio::fs::read(root.join("go.mod")).await.unwrap(), gomod); + } + + /// Dry-run parity for the case above: previewing an in-sync re-run under + /// `--offline` + `--vendor-source service` predicts success, not the + /// refusal the real run never raises. + #[tokio::test] + async fn offline_service_mode_in_sync_dry_run_is_not_refused() { + let (dir, blobs, pristine, record) = fixture().await; + let root = dir.path(); + let (result, _, _) = + expect_done(run_vendor(PURL, root, &blobs, &pristine, &record, false).await); + assert!(result.success, "{:?}", result.error); + let gomod = tokio::fs::read(root.join("go.mod")).await.unwrap(); + let sources = PatchSources::blobs_only(&blobs); + let outcome = vendor_go_module( + PURL, + &pristine, + root, + &record, + &sources, + "2026-06-09T00:00:00Z", + true, + false, + Some(&go_service_cfg( + "http://127.0.0.1:1", + VendorSource::Service, + true, + )), + ) + .await; + let (result, entry, _) = expect_done(outcome); + assert!(result.success, "{:?}", result.error); + assert!(entry.is_none(), "a dry run records nothing"); + assert_eq!(tokio::fs::read(root.join("go.mod")).await.unwrap(), gomod); + } + // ── missing-patch-target pre-check (fail-closed vs `--force`) ───────── /// A patch-target file absent from the pristine module cache fails closed @@ -2601,6 +2680,95 @@ mod tests { ); } + // ── source-flip regression: the hot path decides "in sync" from the + // COMMITTED copy before any service call, so a service ↔ local flip + // between runs is a byte-identical no-op with no request. ── + + async fn flip_run( + root: &Path, + blobs: &Path, + pristine: &Path, + record: &PatchRecord, + uri: &str, + ) -> (ApplyResult, Option, Vec) { + let sources = PatchSources::blobs_only(blobs); + expect_done( + vendor_go_module( + PURL, + pristine, + root, + record, + &sources, + "2026-06-09T00:00:00Z", + false, + false, + Some(&go_service_cfg(uri, VendorSource::Auto, false)), + ) + .await, + ) + } + + fn flip_service_zip() -> Vec { + make_module_zip(&[ + ( + "go.mod", + b"module github.com/foo/bar\n\ngo 1.22 // service\n", + ), + ("bar.go", PATCHED), + ("SERVICE_ONLY.txt", b"x\n"), + ]) + } + + async fn flip_go_sum(root: &Path) { + tokio::fs::write( + root.join("go.sum"), + "github.com/foo/bar v1.4.2 h1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\ngithub.com/foo/bar v1.4.2/go.mod h1:BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB=\n", + ) + .await + .unwrap(); + } + + #[tokio::test] + async fn flip_local_then_service_is_noop() { + use crate::vendor::test_support as ts; + let (dir, blobs, pristine, record) = fixture().await; + let root = dir.path(); + flip_go_sum(root).await; + let down = wiremock::MockServer::start().await; + ts::mount_503(&down).await; + let (r1, e1, _) = flip_run(root, &blobs, &pristine, &record, &down.uri()).await; + assert!(r1.success && e1.is_some()); + let before = ts::tree_snapshot(root); + let up = wiremock::MockServer::start().await; + let z = flip_service_zip(); + mount_go_granted(&up, &sri_sha512(&z), None, &z).await; + let (r2, e2, w2) = flip_run(root, &blobs, &pristine, &record, &up.uri()).await; + assert!(r2.success && e2.is_none() && w2.is_empty()); + assert_eq!(ts::tree_snapshot(root), before); + assert_eq!(ts::request_count(&up).await, 0); + } + + #[tokio::test] + async fn flip_service_then_local_is_noop() { + use crate::vendor::test_support as ts; + let (dir, blobs, pristine, record) = fixture().await; + let root = dir.path(); + flip_go_sum(root).await; + let up = wiremock::MockServer::start().await; + let z = flip_service_zip(); + mount_go_granted(&up, &sri_sha512(&z), None, &z).await; + let (r1, e1, w1) = flip_run(root, &blobs, &pristine, &record, &up.uri()).await; + assert!(r1.success && e1.is_some()); + assert!(ts::has_warning(&w1, "vendor_prebuilt_downloaded")); + let before = ts::tree_snapshot(root); + let down = wiremock::MockServer::start().await; + ts::mount_503(&down).await; + let (r2, e2, w2) = flip_run(root, &blobs, &pristine, &record, &down.uri()).await; + assert!(r2.success && e2.is_none() && w2.is_empty()); + assert_eq!(ts::tree_snapshot(root), before); + assert_eq!(ts::request_count(&down).await, 0); + } + /// An integrity mismatch is a hard failure under `auto` too — /// never a quiet local-build fallback (service_fetch's contract). #[tokio::test] diff --git a/crates/socket-patch-core/src/vendor/maven_repo.rs b/crates/socket-patch-core/src/vendor/maven_repo.rs index 5587caef..b33e8cb3 100644 --- a/crates/socket-patch-core/src/vendor/maven_repo.rs +++ b/crates/socket-patch-core/src/vendor/maven_repo.rs @@ -2501,6 +2501,7 @@ mod tests { use_public_proxy: false, org_slug: Some("acme".into()), }) + .with_vendor_retry(crate::api::client::VendorRetryPolicy::none()) }), use_public_proxy: false, vendor_url: None, @@ -3633,6 +3634,107 @@ mod tests { assert_eq!(strip_empty_repositories(no_close), no_close); } + // ── source-flip regression: the hot path decides "in sync" from the + // COMMITTED copy before any service call, so a service ↔ local flip + // between runs is a byte-identical no-op with no request. ── + + /// A service jar: same members, STORED — bytes distinct from the local + /// deflate re-zip, as a real service build would be. + async fn flip_granted_jar() -> (wiremock::MockServer, Vec) { + use std::io::Write as _; + use wiremock::matchers::{method, path}; + use wiremock::{Mock, MockServer, ResponseTemplate}; + let body = { + let mut zw = zip::ZipWriter::new(std::io::Cursor::new(Vec::new())); + let opts = zip::write::SimpleFileOptions::default() + .compression_method(zip::CompressionMethod::Stored); + for (name, bytes) in [ + ("META-INF/MANIFEST.MF", &b"Manifest-Version: 1.0\n"[..]), + (JAR_FILE, PATCHED), + ( + "org/apache/commons/text/StringSubstitutor.class", + &b"\xca\xfe\xba\xbe-fake-class"[..], + ), + ] { + zw.start_file(name, opts).unwrap(); + zw.write_all(bytes).unwrap(); + } + zw.finish().unwrap().into_inner() + }; + let sri = crate::vendor::test_support::sri(&body); + let serve_path = "/patch/maven/commons-text/1.10.0/tok/uuid/commons-text-1.10.0.jar"; + let server = MockServer::start().await; + let serve_url = format!("{}{serve_path}", server.uri()); + Mock::given(method("POST")) + .and(path("/v0/orgs/acme/patches/package")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "results": { UUID: { + "status": "granted", "url": serve_url, + "artifacts": [{ "kind": "tarball", "url": serve_url, + "integrity": { "sha512": sri } }] + }} + }))) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path(serve_path)) + .respond_with(ResponseTemplate::new(200).set_body_bytes(body.clone())) + .mount(&server) + .await; + (server, body) + } + + fn flip_cfg(s: &wiremock::MockServer) -> VendorServiceConfig { + service_cfg(Some(&s.uri()), crate::vendor::VendorSource::Auto, false) + } + + #[tokio::test] + async fn flip_local_then_service_is_noop() { + use crate::vendor::test_support as ts; + let (dir, blobs, installed, record) = fixture(Some(project_pom()), true, true).await; + let root = dir.path(); + let down = wiremock::MockServer::start().await; + ts::mount_503(&down).await; + let (r1, e1, _) = unwrap_done( + run_vendor_with_service(root, &blobs, &installed, &record, &flip_cfg(&down)).await, + ); + assert!(r1.success && e1.is_some()); + let local_jar = tokio::fs::read(root.join(jar_rel())).await.unwrap(); + let before = ts::tree_snapshot(root); + let (up, served) = flip_granted_jar().await; + assert_ne!(local_jar, served, "sources produce different bytes"); + let (r2, e2, _) = unwrap_done( + run_vendor_with_service(root, &blobs, &installed, &record, &flip_cfg(&up)).await, + ); + assert!(r2.success); + assert!(e2.is_none()); + assert_eq!(before, ts::tree_snapshot(root)); + assert_eq!(ts::request_count(&up).await, 0); + } + + #[tokio::test] + async fn flip_service_then_local_is_noop() { + use crate::vendor::test_support as ts; + let (dir, blobs, installed, record) = fixture(Some(project_pom()), true, true).await; + let root = dir.path(); + let (up, served) = flip_granted_jar().await; + let (r1, e1, _) = unwrap_done( + run_vendor_with_service(root, &blobs, &installed, &record, &flip_cfg(&up)).await, + ); + assert!(r1.success && e1.is_some()); + assert_eq!(tokio::fs::read(root.join(jar_rel())).await.unwrap(), served); + let before = ts::tree_snapshot(root); + let down = wiremock::MockServer::start().await; + ts::mount_503(&down).await; + let (r2, e2, _) = unwrap_done( + run_vendor_with_service(root, &blobs, &installed, &record, &flip_cfg(&down)).await, + ); + assert!(r2.success); + assert!(e2.is_none()); + assert_eq!(before, ts::tree_snapshot(root)); + assert_eq!(ts::request_count(&down).await, 0); + } + /// Mount a granted service response serving `body` as the prebuilt jar. async fn mount_granted_jar(body: &[u8]) -> wiremock::MockServer { use wiremock::matchers::{method, path}; diff --git a/crates/socket-patch-core/src/vendor/mod.rs b/crates/socket-patch-core/src/vendor/mod.rs index 1aca167e..d5a812bd 100644 --- a/crates/socket-patch-core/src/vendor/mod.rs +++ b/crates/socket-patch-core/src/vendor/mod.rs @@ -81,7 +81,10 @@ mod pypi_requirements; mod pypi_uv; mod pypi_wheel; pub mod registry_fetch; +pub(crate) mod reuse; pub(crate) mod service_fetch; +#[cfg(test)] +pub(crate) mod test_support; mod toml_surgery; pub(crate) mod verify; pub(crate) mod yarn_berry_lock; diff --git a/crates/socket-patch-core/src/vendor/npm_common.rs b/crates/socket-patch-core/src/vendor/npm_common.rs index 04eb4fb4..08efab75 100644 --- a/crates/socket-patch-core/src/vendor/npm_common.rs +++ b/crates/socket-patch-core/src/vendor/npm_common.rs @@ -29,6 +29,7 @@ use super::common::{ }; use super::npm_pack::{pack_deterministic, PackedTarball}; use super::path::vendor_uuid_dir_rel; +use super::reuse; use super::service_fetch::{fetch_verified_archive, ServiceArtifact}; use super::{RevertOutcome, VendorOutcome, VendorServiceConfig, VendorWarning}; @@ -127,6 +128,12 @@ pub(super) struct NpmStagedPack { /// (a same-uuid re-vendor's dir may still be referenced by live wiring); /// backends feed this to [`done_failure_unstage`]. pub uuid_dir_preexisted: bool, + /// The exact bytes a committed-artifact reuse hashed and verified + /// (`None` for a fresh pack / download, which this run just wrote). + /// A consumer that needs the tarball's bytes (yarn berry's checksum) + /// uses these instead of re-reading the file, so nothing swapped in + /// after verification can reach the lock. + pub verified_bytes: Option>, } /// Stage → patch → pack one installed npm package. @@ -162,13 +169,34 @@ pub(super) async fn stage_patch_pack( ) -> Result<(Option, ApplyResult), Box> { let coords = guard_coordinates(purl, record)?; + // ── Reuse the committed artifact (before any acquisition) ─────────── + // A re-run whose tarball the ledger vouches for — sha256 anchored, + // every afterHash verified from the same bytes — keeps those bytes: + // no service call, no local pack, no write. Acquiring anew would make + // the lock's digests depend on which source answered THIS run (the + // service's prebuilt encoding and the local deterministic pack carry the + // same members but different bytes), so a service outage or its + // recovery would re-vendor every package. `--vendor-source` governs + // acquisition, not reuse; checked before `service_offline_conflict` so + // an in-sync `service` + `--offline` re-run succeeds (as cargo and + // composer already do). The probe is read-only and offline, so a dry + // run runs it too: on a hit it skips the offline refusal (the real run + // would not raise it) and keeps previewing the local build. + let mut reusable = false; + if let Some(pair) = reuse_committed_pack(purl, project_root, &coords, record).await { + if !dry_run { + return Ok(pair); + } + reusable = true; + } + // ── Service-download fast path (Tier A: write the prebuilt tarball) ── // When the vendoring service is configured, try to download the already- // built, integrity-verified tarball instead of staging+patching+packing // locally. A dry run previews the local build (no network). Per the // `auto`/`service` policy a non-fatal miss falls back to the local build // below; under `service` it fails closed. - if let Some(refusal) = service_offline_conflict(service) { + if let Some(refusal) = service_offline_conflict(service).filter(|_| !reusable) { return Err(Box::new(refusal)); } if let Some(cfg) = service { @@ -315,6 +343,68 @@ pub(super) async fn stage_patch_pack( packed, staged_pkg_json, uuid_dir_preexisted, + verified_bytes: None, + }), + result, + )) +} + +/// The staged pack for a verified committed tarball (see +/// [`super::reuse`]), or `None` to acquire as usual. Nothing is written, so +/// the pack reports `uuid_dir_preexisted: true` — a later wiring failure's +/// [`done_failure_unstage`] must never delete the committed artifact the +/// live ledger entry still names. +async fn reuse_committed_pack( + purl: &str, + project_root: &Path, + coords: &NpmCoords, + record: &PatchRecord, +) -> Option<(Option, ApplyResult)> { + let rel_tgz = format!( + "{}/{}", + coords.uuid_dir_rel, + tgz_rel_leaf(&coords.name, &coords.version) + ); + let art = match reuse::reusable_committed_artifact(project_root, "npm", record, Some(&rel_tgz)) + .await + { + Ok(art) => art, + Err(miss) => { + reuse::log_miss(purl, &miss); + return None; + } + }; + // A patched package.json feeds the flavor's dependency-mirror + // recompute; read it from the SAME verified bytes. + let staged_pkg_json = if record + .files + .keys() + .any(|k| normalize_file_path(k) == "package.json") + { + match art + .members + .get("package.json") + .and_then(|b| serde_json::from_slice::(b).ok()) + { + Some(pkg) => Some(pkg), + None => { + reuse::log_miss(purl, &reuse::ReuseMiss::Unreadable); + return None; + } + } + } else { + None + }; + let result = already_patched_result(purl, &project_root.join(&rel_tgz), &record.files); + Some(( + Some(NpmStagedPack { + name: coords.name.clone(), + version: coords.version.clone(), + rel_tgz, + packed: PackedTarball::from_bytes(&art.bytes), + staged_pkg_json, + uuid_dir_preexisted: true, + verified_bytes: Some(art.bytes), }), result, )) @@ -519,6 +609,7 @@ async fn staged_pack_from_service_bytes( packed, staged_pkg_json, uuid_dir_preexisted, + verified_bytes: None, }) } @@ -1141,12 +1232,15 @@ mod tests { fn service_cfg(server_uri: &str, source: VendorSource) -> VendorServiceConfig { VendorServiceConfig { source, - client: Some(ApiClient::new(ApiClientOptions { - api_url: server_uri.to_string(), - api_token: Some("sktsec_placeholder_value_for_tests_api".into()), - use_public_proxy: false, - org_slug: Some("acme".into()), - })), + client: Some( + ApiClient::new(ApiClientOptions { + api_url: server_uri.to_string(), + api_token: Some("sktsec_placeholder_value_for_tests_api".into()), + use_public_proxy: false, + org_slug: Some("acme".into()), + }) + .with_vendor_retry(crate::api::client::VendorRetryPolicy::none()), + ), use_public_proxy: false, vendor_url: None, patch_server_url: None, @@ -1439,6 +1533,58 @@ mod tests { assert!(!tmp.path().join(".socket/vendor").exists()); } + /// A reuse hands the flavor the EXACT bytes it verified (yarn berry + /// derives its checksum from them rather than re-reading a file that + /// may have been swapped since); a fresh download carries none. + #[tokio::test] + async fn reused_pack_carries_the_verified_bytes() { + let tmp = tempfile::tempdir().unwrap(); + let mut record = record_with_uuid(UUID); + record.files.get_mut("package/index.js").unwrap().after_hash = + crate::hash::git_sha256::compute_git_sha256_from_bytes(PATCHED_INDEX); + let tgz = build_tgz(&[ + ("index.js", PATCHED_INDEX), + ("package.json", br#"{"name":"left-pad","version":"1.3.0"}"#), + ]) + .await; + let sri = PackedTarball::from_bytes(&tgz).integrity; + let fresh = service_bytes(tmp.path(), &record, &tgz, &sri) + .await + .unwrap_or_else(|e| panic!("{e:?}")); + assert!(fresh.verified_bytes.is_none()); + let entry = crate::vendor::state::VendorEntry { + ecosystem: "npm".into(), + base_purl: LP_PURL.into(), + uuid: record.uuid.clone(), + artifact: crate::vendor::state::VendorArtifact { + path: fresh.rel_tgz.clone(), + sha256: fresh.packed.sha256_hex.clone(), + size: Some(fresh.packed.size), + platform_locked: None, + file_inventory: None, + }, + wiring: Vec::new(), + lock: None, + took_over_go_patches: false, + flavor: Some("yarn-berry".into()), + uv: None, + pnpm: None, + poetry: None, + pdm: None, + pipenv: None, + detached: false, + record: None, + }; + crate::vendor::test_support::persist(tmp.path(), LP_PURL, entry).await; + let coords = guard_coordinates(LP_PURL, &record).unwrap(); + let (staged, _) = reuse_committed_pack(LP_PURL, tmp.path(), &coords, &record) + .await + .expect("the committed tarball is reused"); + let staged = staged.unwrap(); + assert_eq!(staged.verified_bytes.as_deref(), Some(tgz.as_slice())); + assert!(staged.uuid_dir_preexisted); + } + /// Full service-bytes success: the tarball lands verbatim at the same /// rel path a local build uses, the `PackedTarball` facts describe the /// served bytes, and the patched package.json is extracted through the diff --git a/crates/socket-patch-core/src/vendor/npm_lock.rs b/crates/socket-patch-core/src/vendor/npm_lock.rs index 424a2f72..7a6a145d 100644 --- a/crates/socket-patch-core/src/vendor/npm_lock.rs +++ b/crates/socket-patch-core/src/vendor/npm_lock.rs @@ -319,9 +319,12 @@ pub async fn vendor_npm( if !changed { // Every instance already points at this uuid with the packed - // integrity: the project is in sync. Touch nothing (the tarball - // rewrite above was byte-identical by determinism) and synthesize an - // AlreadyPatched-style success, mirroring the go_redirect hot path. + // integrity: the project is in sync. The facts are those of the + // REUSED committed artifact (the shared pipeline wrote nothing) or, + // when reuse missed (no ledger anchor, a tampered/missing tarball), + // of a freshly acquired one that reproduced the pinned bytes. Touch + // nothing and synthesize an AlreadyPatched-style success, mirroring + // the go_redirect hot path. return done( already_patched_result(purl, &project_root.join(&rel_tgz), &record.files), None, @@ -3198,12 +3201,15 @@ mod tests { fn service_cfg(server_uri: &str, source: VendorSource, offline: bool) -> VendorServiceConfig { VendorServiceConfig { source, - client: Some(ApiClient::new(ApiClientOptions { - api_url: server_uri.to_string(), - api_token: Some("sktsec_placeholder_value_for_tests_api".into()), - use_public_proxy: false, - org_slug: Some("acme".into()), - })), + client: Some( + ApiClient::new(ApiClientOptions { + api_url: server_uri.to_string(), + api_token: Some("sktsec_placeholder_value_for_tests_api".into()), + use_public_proxy: false, + org_slug: Some("acme".into()), + }) + .with_vendor_retry(crate::api::client::VendorRetryPolicy::none()), + ), use_public_proxy: false, vendor_url: None, patch_server_url: None, @@ -3533,6 +3539,510 @@ mod tests { } } + // ─────────────── source-flip / outage idempotence ─────────────── + // + // A re-run whose committed tarball the ledger vouches for reuses it — + // whichever source (service prebuilt, local pack) built it — so a + // service outage or its recovery never re-vendors. The shared suite + // covers the four flips per flavor; the cases below pin the reuse gate's + // fail-closed edges at the flavor level. + + use crate::vendor::test_support as ts; + + impl ts::FlipFixture for Fixture { + fn flip_root(&self) -> &Path { + self.root() + } + fn flip_key(&self) -> String { + self.purl() + } + fn flip_uuid(&self) -> String { + self.record.uuid.clone() + } + fn flip_artifact_rel(&self) -> String { + format!( + ".socket/vendor/npm/{}/{}", + self.record.uuid, + tgz_rel_leaf(&self.name, &self.version) + ) + } + fn flip_files(&self) -> Vec { + vec![PACKAGE_LOCK.to_string()] + } + } + + async fn flip_run(fx: &Fixture, cfg: Option<&VendorServiceConfig>) -> VendorOutcome { + let blobs = fx.root().join(".socket/blobs"); + let sources = PatchSources::blobs_only(&blobs); + vendor_npm( + &fx.purl(), + &fx.installed(), + fx.root(), + &fx.record, + &sources, + "2026-06-09T00:00:00Z", + false, + false, + cfg, + ) + .await + } + + async fn flip_fixture_v3() -> Fixture { + fixture().await + } + + /// v2: the legacy `dependencies` mirror must stay byte-stable too. + async fn flip_fixture_v2() -> Fixture { + let mut lock = default_lock(); + lock["lockfileVersion"] = json!(2); + lock["dependencies"] = json!({ + "foo": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/foo/-/foo-2.0.0.tgz", + "integrity": "sha512-foo==", + "requires": { "left-pad": "^1.3.0" }, + "dependencies": { + "left-pad": { "version": "1.3.0", "resolved": REG_RESOLVED, "integrity": "sha512-orig==" } + } + }, + "left-pad": { "version": "1.3.0", "resolved": REG_RESOLVED, "integrity": "sha512-orig==" } + }); + fixture_with("left-pad", "1.3.0", lock).await + } + + ts::npm_flip_suite!(flip_suite_v3, Fixture, flip_fixture_v3, flip_run); + ts::npm_flip_suite!(flip_suite_v2, Fixture, flip_fixture_v2, flip_run); + + /// Run 1 from the service (`alt` = a re-encoding of the local build), + /// persisted like the CLI does. Returns (fixture, server, alt bytes). + async fn service_vendored() -> (Fixture, wiremock::MockServer, Vec) { + let (local, _) = locally_built_artifact().await; + let alt = ts::regzip(&local); + let server = wiremock::MockServer::start().await; + ts::mount_granted(&server, UUID, "left-pad-1.3.0.tgz", &alt).await; + let fx = fixture().await; + let cfg = ts::service_cfg(&server.uri(), VendorSource::Auto, false); + let (r, e, _) = expect_done(flip_run(&fx, Some(&cfg)).await); + assert!(r.success, "{:?}", r.error); + ts::persist(fx.root(), &fx.purl(), e.unwrap()).await; + server.reset().await; + ts::mount_503(&server).await; + (fx, server, alt) + } + + /// Run 2 under the outage; asserts it was NOT a reuse (the gate missed, + /// so acquisition fell back to a local build and re-pinned the lock). + async fn assert_not_reused(fx: &Fixture, server: &wiremock::MockServer, alt: &[u8]) { + let cfg = ts::service_cfg(&server.uri(), VendorSource::Auto, false); + let (r, e, w) = expect_done(flip_run(fx, Some(&cfg)).await); + assert!(r.success, "{:?}", r.error); + assert!(e.is_some(), "a failed reuse gate re-acquires and re-pins"); + assert!(ts::has_warning(&w, "vendor_prebuilt_unavailable"), "{w:?}"); + let lock = fx.read_lock().await; + assert_ne!( + lock_integrity(&lock, "node_modules/left-pad"), + ts::sri(alt), + "the unverified bytes were not pinned" + ); + let tgz = tokio::fs::read(fx.root().join(fx.expected_rel_tgz())) + .await + .unwrap(); + assert_eq!( + lock_integrity(&lock, "node_modules/left-pad"), + ts::sri(&tgz) + ); + assert!( + !std::fs::symlink_metadata(fx.root().join(fx.expected_rel_tgz())) + .unwrap() + .file_type() + .is_symlink() + ); + } + + /// F6: a re-gzipped tarball (members intact, bytes changed) with the + /// ledger untouched fails the sha anchor; today's rebuild heals it. + #[tokio::test] + async fn regzipped_tarball_with_untouched_ledger_is_not_reused() { + let (fx, server, alt) = service_vendored().await; + let tgz = fx.root().join(fx.expected_rel_tgz()); + let reencoded = ts::regzip_at(&alt, flate2::Compression::best()); + tokio::fs::write(&tgz, &reencoded).await.unwrap(); + assert_not_reused(&fx, &server, &reencoded).await; + } + + /// Tamper: an UNPATCHED member edited and the tarball re-gzipped, the + /// ledger sha stale — never reused, never pinned. + #[tokio::test] + async fn edited_unpatched_member_with_stale_ledger_sha_is_not_reused() { + let (fx, server, alt) = service_vendored().await; + let tgz = fx.root().join(fx.expected_rel_tgz()); + let tampered = { + let mut members = crate::patch::package::read_archive_bytes_to_map(&alt).unwrap(); + members.insert( + "package.json".into(), + b"{\"name\":\"left-pad\",\"evil\":1}".to_vec(), + ); + let mut b = tar::Builder::new(flate2::write::GzEncoder::new( + Vec::new(), + flate2::Compression::default(), + )); + let mut names: Vec<_> = members.keys().cloned().collect(); + names.sort(); + for n in names { + let data = &members[&n]; + let mut h = tar::Header::new_gnu(); + h.set_size(data.len() as u64); + h.set_mode(0o644); + h.set_cksum(); + b.append_data(&mut h, format!("package/{n}"), data.as_slice()) + .unwrap(); + } + b.into_inner().unwrap().finish().unwrap() + }; + tokio::fs::write(&tgz, &tampered).await.unwrap(); + assert_not_reused(&fx, &server, &alt).await; + } + + /// F7: a patched member edited AND the ledger sha forged to match fails + /// the afterHash check. + #[tokio::test] + async fn forged_ledger_over_edited_patched_member_is_not_reused() { + use sha2::Sha256; + let (fx, server, alt) = service_vendored().await; + let evil = { + let mut b = tar::Builder::new(flate2::write::GzEncoder::new( + Vec::new(), + flate2::Compression::default(), + )); + for (n, data) in [ + ( + "package/package.json", + installed_pkg_json("left-pad", "1.3.0"), + ), + ("package/index.js", b"module.exports = 'evil';\n".to_vec()), + ] { + let mut h = tar::Header::new_gnu(); + h.set_size(data.len() as u64); + h.set_mode(0o644); + h.set_cksum(); + b.append_data(&mut h, n, data.as_slice()).unwrap(); + } + b.into_inner().unwrap().finish().unwrap() + }; + tokio::fs::write(fx.root().join(fx.expected_rel_tgz()), &evil) + .await + .unwrap(); + let mut state = crate::vendor::state::load_state(fx.root()).await.unwrap(); + let e = state.entries.get_mut(&fx.purl()).unwrap(); + e.artifact.sha256 = hex::encode(Sha256::digest(&evil)); + e.artifact.size = Some(evil.len() as u64); + crate::vendor::state::save_state(fx.root(), &state) + .await + .unwrap(); + assert_not_reused(&fx, &server, &alt).await; + } + + /// Tamper: a symlinked artifact (pointing at verified bytes outside the + /// uuid dir) is never reused. + #[cfg(unix)] + #[tokio::test] + async fn symlinked_artifact_is_not_reused() { + let (fx, server, alt) = service_vendored().await; + let tgz = fx.root().join(fx.expected_rel_tgz()); + let outside = fx.root().join("outside.tgz"); + tokio::fs::write(&outside, &alt).await.unwrap(); + tokio::fs::remove_file(&tgz).await.unwrap(); + std::os::unix::fs::symlink(&outside, &tgz).unwrap(); + assert_not_reused(&fx, &server, &alt).await; + } + + /// F11 / tamper: a FIFO at the artifact path returns promptly, no reuse. + #[cfg(unix)] + #[tokio::test] + async fn fifo_artifact_is_not_reused_and_never_wedges() { + let (fx, server, alt) = service_vendored().await; + let tgz = fx.root().join(fx.expected_rel_tgz()); + tokio::fs::remove_file(&tgz).await.unwrap(); + let c = std::ffi::CString::new(tgz.to_str().unwrap()).unwrap(); + assert_eq!(unsafe { libc::mkfifo(c.as_ptr(), 0o644) }, 0); + tokio::time::timeout( + std::time::Duration::from_secs(30), + assert_not_reused(&fx, &server, &alt), + ) + .await + .expect("a FIFO must never wedge a vendor re-run"); + } + + /// Tamper: verified bytes under ANOTHER uuid's directory (the ledger + /// path rewritten to point there) are never reused for this record. + #[tokio::test] + async fn artifact_under_a_wrong_uuid_dir_is_not_reused() { + const OTHER: &str = "1a2b3c4d-5e6f-4a1b-8c2d-3e4f5a6b7c8d"; + let (fx, server, alt) = service_vendored().await; + let other_rel = format!(".socket/vendor/npm/{OTHER}/left-pad-1.3.0.tgz"); + tokio::fs::create_dir_all(fx.root().join(&other_rel).parent().unwrap()) + .await + .unwrap(); + tokio::fs::rename( + fx.root().join(fx.expected_rel_tgz()), + fx.root().join(&other_rel), + ) + .await + .unwrap(); + let mut state = crate::vendor::state::load_state(fx.root()).await.unwrap(); + state.entries.get_mut(&fx.purl()).unwrap().artifact.path = other_rel.clone(); + crate::vendor::state::save_state(fx.root(), &state) + .await + .unwrap(); + assert_not_reused(&fx, &server, &alt).await; + assert_eq!( + tokio::fs::read(fx.root().join(&other_rel)).await.unwrap(), + alt, + "the other uuid's artifact is left alone" + ); + } + + /// F8: with the ledger gone there is no anchor — today's behavior (the + /// lock is re-pinned to the fresh local build). Documents the residual. + #[tokio::test] + async fn missing_ledger_keeps_todays_repin() { + let (fx, server, alt) = service_vendored().await; + tokio::fs::remove_file(fx.root().join(".socket/vendor/state.json")) + .await + .unwrap(); + assert_not_reused(&fx, &server, &alt).await; + } + + /// F9: a new record uuid acquires under the new uuid dir. + #[tokio::test] + async fn new_record_uuid_acquires_under_the_new_uuid_dir() { + const NEXT: &str = "1a2b3c4d-5e6f-4a1b-8c2d-3e4f5a6b7c8d"; + let (mut fx, server, alt) = service_vendored().await; + fx.record.uuid = NEXT.to_string(); + let cfg = ts::service_cfg(&server.uri(), VendorSource::Auto, false); + let (r, e, _) = expect_done(flip_run(&fx, Some(&cfg)).await); + assert!(r.success, "{:?}", r.error); + let e = e.expect("a new uuid re-wires"); + assert_eq!( + e.artifact.path, + format!(".socket/vendor/npm/{NEXT}/left-pad-1.3.0.tgz") + ); + assert_eq!( + tokio::fs::read( + fx.root() + .join(format!(".socket/vendor/npm/{UUID}/left-pad-1.3.0.tgz")) + ) + .await + .unwrap(), + alt, + "the old uuid's artifact is untouched" + ); + } + + /// F10: a lock reset to the registry resolution (a relock / hand + /// revert) with the tarball + ledger intact is re-pinned to the + /// COMMITTED bytes — no request, the service SRI kept. + #[tokio::test] + async fn relocked_lock_is_repinned_to_the_committed_bytes_without_network() { + let (fx, server, alt) = service_vendored().await; + tokio::fs::write(fx.lock_path(), &fx.lock_bytes) + .await + .unwrap(); + let cfg = ts::service_cfg(&server.uri(), VendorSource::Auto, false); + let (r, e, w) = expect_done(flip_run(&fx, Some(&cfg)).await); + assert!(r.success, "{:?}", r.error); + assert!(e.is_some(), "the lock was re-wired (Applied)"); + assert!(w.is_empty(), "{w:?}"); + let lock = fx.read_lock().await; + assert_eq!( + lock_integrity(&lock, "node_modules/left-pad"), + ts::sri(&alt) + ); + assert_eq!( + lock_integrity(&lock, "node_modules/foo/node_modules/left-pad"), + ts::sri(&alt) + ); + assert_eq!( + tokio::fs::read(fx.root().join(fx.expected_rel_tgz())) + .await + .unwrap(), + alt + ); + assert_eq!(ts::request_count(&server).await, 0); + } + + /// A fixture whose patch also rewrites `package/package.json` (adds a + /// `wow` dependency). + async fn pkg_json_patch_fixture() -> Fixture { + let mut fx = fixture().await; + let before = installed_pkg_json("left-pad", "1.3.0"); + let after: &[u8] = + br#"{"name":"left-pad","version":"1.3.0","dependencies":{"wow":"^1.0.0"}}"#; + let after_hash = compute_git_sha256_from_bytes(after); + tokio::fs::write(fx.root().join(".socket/blobs").join(&after_hash), after) + .await + .unwrap(); + fx.record.files.insert( + "package/package.json".to_string(), + PatchFileInfo { + before_hash: compute_git_sha256_from_bytes(&before), + after_hash, + }, + ); + fx + } + + /// Run 1 of [`pkg_json_patch_fixture`] from the service (a re-encoding + /// of the local build), persisted; the server then answers 503. + /// Returns (fixture, server, run-1 lock bytes). + async fn pkg_json_patch_service_vendored() -> (Fixture, wiremock::MockServer, Vec) { + let probe = pkg_json_patch_fixture().await; + let _ = expect_done(flip_run(&probe, None).await); + let local = tokio::fs::read(probe.root().join(probe.expected_rel_tgz())) + .await + .unwrap(); + let alt = ts::regzip(&local); + let server = wiremock::MockServer::start().await; + ts::mount_granted(&server, UUID, "left-pad-1.3.0.tgz", &alt).await; + let fx = pkg_json_patch_fixture().await; + let cfg = ts::service_cfg(&server.uri(), VendorSource::Auto, false); + let (r, e, _) = expect_done(flip_run(&fx, Some(&cfg)).await); + assert!(r.success, "{:?}", r.error); + ts::persist(fx.root(), &fx.purl(), e.unwrap()).await; + let lock1 = tokio::fs::read(fx.lock_path()).await.unwrap(); + assert_eq!( + fx.read_lock().await["packages"]["node_modules/left-pad"]["dependencies"], + json!({ "wow": "^1.0.0" }) + ); + server.reset().await; + ts::mount_503(&server).await; + (fx, server, lock1) + } + + /// F10 + F12: a relock (registry dependency map) re-pinned from the + /// reused bytes recomputes the dependency mirror from THEIR patched + /// package.json — not the registry's map — with no request. + #[tokio::test] + async fn relock_with_pkg_json_patch_recomputes_deps_from_reused_bytes() { + let (fx, server, lock1) = pkg_json_patch_service_vendored().await; + tokio::fs::write(fx.lock_path(), &fx.lock_bytes) + .await + .unwrap(); + let cfg = ts::service_cfg(&server.uri(), VendorSource::Auto, false); + let (r, e, w) = expect_done(flip_run(&fx, Some(&cfg)).await); + assert!(r.success, "{:?}", r.error); + assert!(e.is_some(), "the relocked lock is re-wired"); + assert!(!ts::has_warning(&w, "vendor_prebuilt_unavailable"), "{w:?}"); + assert_eq!( + fx.read_lock().await["packages"]["node_modules/left-pad"]["dependencies"], + json!({ "wow": "^1.0.0" }) + ); + assert_eq!(tokio::fs::read(fx.lock_path()).await.unwrap(), lock1); + assert_eq!(ts::request_count(&server).await, 0); + } + + /// A wiring failure after a reuse (the relocked lock's staged write + /// fails) must never unstage the uuid dir: it holds the committed + /// tarball the live ledger entry still names. + #[cfg(unix)] + #[tokio::test] + async fn wiring_failure_after_reuse_keeps_the_committed_tarball() { + use std::os::unix::fs::PermissionsExt as _; + if unsafe { libc::geteuid() } == 0 { + return; // root ignores the read-only dir + } + let (fx, server, alt) = service_vendored().await; + tokio::fs::write(fx.lock_path(), &fx.lock_bytes) + .await + .unwrap(); + std::fs::set_permissions(fx.root(), std::fs::Permissions::from_mode(0o555)).unwrap(); + let cfg = ts::service_cfg(&server.uri(), VendorSource::Auto, false); + let outcome = flip_run(&fx, Some(&cfg)).await; + std::fs::set_permissions(fx.root(), std::fs::Permissions::from_mode(0o755)).unwrap(); + let (r, e, _) = expect_done(outcome); + assert!(!r.success, "the lock write must fail"); + assert!(e.is_none()); + assert_eq!( + tokio::fs::read(fx.root().join(fx.expected_rel_tgz())) + .await + .unwrap(), + alt, + "the committed tarball the ledger names survives" + ); + assert_eq!( + tokio::fs::read(fx.lock_path()).await.unwrap(), + fx.lock_bytes, + "lock untouched" + ); + } + + /// Dry run and real run agree for an in-sync package under + /// `service` + `--offline`: the real run reuses (already_vendored), so + /// the dry run must not predict the offline refusal. + #[tokio::test] + async fn dry_run_agrees_with_real_run_under_service_offline_in_sync() { + let (fx, server, alt) = service_vendored().await; + let before = ts::snapshot(&fx).await; + let cfg = ts::service_cfg(&server.uri(), VendorSource::Service, true); + let blobs = fx.root().join(".socket/blobs"); + let sources = PatchSources::blobs_only(&blobs); + for dry_run in [true, false] { + let outcome = vendor_npm( + &fx.purl(), + &fx.installed(), + fx.root(), + &fx.record, + &sources, + "2026-06-09T00:00:00Z", + dry_run, + false, + Some(&cfg), + ) + .await; + let (r, e, _) = expect_done(outcome); + assert!(r.success, "dry_run={dry_run}: {:?}", r.error); + assert!(e.is_none(), "dry_run={dry_run}"); + assert_eq!(ts::snapshot(&fx).await, before, "dry_run={dry_run}"); + } + assert_eq!(before[0].1.as_deref(), Some(alt.as_slice())); + assert_eq!(ts::request_count(&server).await, 0); + // A reuse miss (tarball gone) keeps the refusal in the dry run. + tokio::fs::remove_file(fx.root().join(fx.expected_rel_tgz())) + .await + .unwrap(); + let outcome = vendor_npm( + &fx.purl(), + &fx.installed(), + fx.root(), + &fx.record, + &sources, + "2026-06-09T00:00:00Z", + true, + false, + Some(&cfg), + ) + .await; + assert!( + matches!(outcome, VendorOutcome::Refused { code, .. } if code == "vendor_service_offline_conflict"), + "{outcome:?}" + ); + } + + /// F12: a patch that rewrites package.json — the reused pack's parsed + /// manifest equals the fresh one, so the dependency mirror (and the + /// whole lock) stays byte-identical across a flip. + #[tokio::test] + async fn package_json_patch_reuse_keeps_the_dependency_mirror() { + let (fx, server, lock1) = pkg_json_patch_service_vendored().await; + let cfg = ts::service_cfg(&server.uri(), VendorSource::Auto, false); + let (r, e, _) = expect_done(flip_run(&fx, Some(&cfg)).await); + assert!(r.success, "{:?}", r.error); + assert!(e.is_none()); + assert_eq!(tokio::fs::read(fx.lock_path()).await.unwrap(), lock1); + assert_eq!(ts::request_count(&server).await, 0); + } + /// An integrity mismatch is a hard failure under `auto` too — /// never a quiet local-build fallback (service_fetch's contract); /// the project is byte-untouched. diff --git a/crates/socket-patch-core/src/vendor/nuget_feed.rs b/crates/socket-patch-core/src/vendor/nuget_feed.rs index 4b18f8a0..3b703f0d 100644 --- a/crates/socket-patch-core/src/vendor/nuget_feed.rs +++ b/crates/socket-patch-core/src/vendor/nuget_feed.rs @@ -4050,12 +4050,15 @@ mod tests { .await; let cfg = VendorServiceConfig { source: VendorSource::Service, - client: Some(ApiClient::new(ApiClientOptions { - api_url: server.uri(), - api_token: Some("sktsec_placeholder_value_for_tests_api".into()), - use_public_proxy: false, - org_slug: Some("acme".into()), - })), + client: Some( + ApiClient::new(ApiClientOptions { + api_url: server.uri(), + api_token: Some("sktsec_placeholder_value_for_tests_api".into()), + use_public_proxy: false, + org_slug: Some("acme".into()), + }) + .with_vendor_retry(crate::api::client::VendorRetryPolicy::none()), + ), use_public_proxy: false, vendor_url: None, patch_server_url: None, @@ -4128,12 +4131,15 @@ mod tests { .await; let cfg = VendorServiceConfig { source: VendorSource::Service, - client: Some(ApiClient::new(ApiClientOptions { - api_url: server.uri(), - api_token: Some("sktsec_placeholder_value_for_tests_api".into()), - use_public_proxy: false, - org_slug: Some("acme".into()), - })), + client: Some( + ApiClient::new(ApiClientOptions { + api_url: server.uri(), + api_token: Some("sktsec_placeholder_value_for_tests_api".into()), + use_public_proxy: false, + org_slug: Some("acme".into()), + }) + .with_vendor_retry(crate::api::client::VendorRetryPolicy::none()), + ), use_public_proxy: false, vendor_url: None, patch_server_url: None, @@ -4615,12 +4621,15 @@ mod tests { .await; let cfg = VendorServiceConfig { source: VendorSource::Service, - client: Some(ApiClient::new(ApiClientOptions { - api_url: server.uri(), - api_token: Some("sktsec_placeholder_value_for_tests_api".into()), - use_public_proxy: false, - org_slug: Some("acme".into()), - })), + client: Some( + ApiClient::new(ApiClientOptions { + api_url: server.uri(), + api_token: Some("sktsec_placeholder_value_for_tests_api".into()), + use_public_proxy: false, + org_slug: Some("acme".into()), + }) + .with_vendor_retry(crate::api::client::VendorRetryPolicy::none()), + ), use_public_proxy: false, vendor_url: None, patch_server_url: None, @@ -4718,6 +4727,109 @@ mod tests { ); } + // ── source-flip regression: the hot path decides "in sync" from the + // COMMITTED copy before any service call, so a service ↔ local flip + // between runs is a byte-identical no-op with no request. ── + + async fn flip_granted_nupkg() -> (wiremock::MockServer, Vec) { + use wiremock::matchers::{method, path}; + use wiremock::{Mock, MockServer, ResponseTemplate}; + let server = MockServer::start().await; + let served = make_nupkg(PATCHED); + let sri = crate::vendor::test_support::sri(&served); + let serve_path = + "/patch/nuget/newtonsoft.json/13.0.3/tok/uuid/newtonsoft.json.13.0.3.nupkg"; + let serve_url = format!("{}{serve_path}", server.uri()); + Mock::given(method("POST")) + .and(path("/v0/orgs/acme/patches/package")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "results": { UUID: { + "status": "granted", "url": serve_url, + "artifacts": [{ "kind": "tarball", "url": serve_url, + "integrity": { "sha512": sri } }] + }} + }))) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path(serve_path)) + .respond_with(ResponseTemplate::new(200).set_body_bytes(served.clone())) + .mount(&server) + .await; + (server, served) + } + + async fn flip_run( + root: &Path, + blobs: &Path, + installed: &Path, + record: &PatchRecord, + s: &wiremock::MockServer, + ) -> (ApplyResult, Option, Vec) { + let cfg = crate::vendor::test_support::service_cfg( + &s.uri(), + crate::vendor::VendorSource::Auto, + false, + ); + let sources = PatchSources::blobs_only(blobs); + unwrap_done( + vendor_nuget( + PURL, + installed, + root, + record, + &sources, + "2026-06-09T00:00:00Z", + false, + false, + Some(&cfg), + ) + .await, + ) + } + + #[tokio::test] + async fn flip_local_then_service_is_noop() { + use crate::vendor::test_support as ts; + let (dir, blobs, installed, record) = fixture(true, None).await; + let root = dir.path(); + let down = wiremock::MockServer::start().await; + ts::mount_503(&down).await; + let (r1, e1, _) = flip_run(root, &blobs, &installed, &record, &down).await; + assert!(r1.success && e1.is_some()); + let local = tokio::fs::read(root.join(copy_rel())).await.unwrap(); + let before = ts::tree_snapshot(root); + let (up, served) = flip_granted_nupkg().await; + assert_ne!(local, served); + let (r2, e2, _) = flip_run(root, &blobs, &installed, &record, &up).await; + assert!(r2.success); + assert!(e2.is_none()); + assert_eq!(before, ts::tree_snapshot(root)); + assert_eq!(ts::request_count(&up).await, 0); + } + + #[tokio::test] + async fn flip_service_then_local_is_noop() { + use crate::vendor::test_support as ts; + let (dir, blobs, installed, record) = fixture(true, None).await; + let root = dir.path(); + let (up, served) = flip_granted_nupkg().await; + let (r1, e1, _) = flip_run(root, &blobs, &installed, &record, &up).await; + assert!(r1.success && e1.is_some()); + assert_eq!( + tokio::fs::read(root.join(copy_rel())).await.unwrap(), + served + ); + let before = ts::tree_snapshot(root); + let down = wiremock::MockServer::start().await; + ts::mount_503(&down).await; + let (r2, e2, _) = flip_run(root, &blobs, &installed, &record, &down).await; + assert!(r2.success); + assert!(e2.is_none()); + assert_eq!(before, ts::tree_snapshot(root)); + assert_eq!(ts::request_count(&down).await, 0); + } + fn integrity_cfg( server: Option<&wiremock::MockServer>, source: crate::vendor::VendorSource, diff --git a/crates/socket-patch-core/src/vendor/pnpm_lock.rs b/crates/socket-patch-core/src/vendor/pnpm_lock.rs index 0b0b49ff..1740ead0 100644 --- a/crates/socket-patch-core/src/vendor/pnpm_lock.rs +++ b/crates/socket-patch-core/src/vendor/pnpm_lock.rs @@ -350,9 +350,11 @@ pub async fn vendor_pnpm( if !pkg_changed && !lock_changed && ws_edit.new_text.is_none() { // Everything already carries this uuid + the packed integrity: the - // project is in sync. The tarball re-pack above was byte-identical - // by determinism; synthesize AlreadyPatched and record nothing (the - // existing ledger entry stays authoritative). + // project is in sync. The integrity is that of the reused committed + // tarball (the shared pipeline wrote nothing) or, when reuse missed, + // of a fresh acquisition that reproduced the pinned bytes; + // synthesize AlreadyPatched and record nothing (the existing ledger + // entry stays authoritative). return done( already_patched_result(purl, &project_root.join(&rel_tgz), &record.files), None, @@ -2951,6 +2953,55 @@ snapshots: } } + // ── source-flip / outage idempotence (vendor::test_support::npm_flip_suite) ── + + impl crate::vendor::test_support::FlipFixture for Fixture { + fn flip_root(&self) -> &Path { + self.root() + } + fn flip_key(&self) -> String { + "pkg:npm/left-pad@1.3.0".to_string() + } + fn flip_uuid(&self) -> String { + self.record.uuid.clone() + } + fn flip_artifact_rel(&self) -> String { + self.rel_tgz() + } + fn flip_files(&self) -> Vec { + vec![ + PACKAGE_JSON.to_string(), + PNPM_LOCK.to_string(), + PNPM_WORKSPACE.to_string(), + ] + } + } + + async fn flip_run( + fx: &Fixture, + cfg: Option<&crate::vendor::VendorServiceConfig>, + ) -> VendorOutcome { + let blobs = fx.root().join(".socket/blobs"); + vendor_pnpm( + "pkg:npm/left-pad@1.3.0", + &fx.installed(), + fx.root(), + &fx.record, + &PatchSources::blobs_only(&blobs), + "2026-06-09T00:00:00Z", + false, + false, + cfg, + ) + .await + } + + async fn flip_fixture() -> Fixture { + fixture_with(P1_BEFORE_PKG, P1_BEFORE_LOCK).await + } + + crate::vendor::test_support::npm_flip_suite!(flip_suite, Fixture, flip_fixture, flip_run); + async fn fixture_with(pkg_json: &str, lock: &str) -> Fixture { let tmp = tempfile::tempdir().unwrap(); let root = tmp.path(); diff --git a/crates/socket-patch-core/src/vendor/pnpm_lock_legacy.rs b/crates/socket-patch-core/src/vendor/pnpm_lock_legacy.rs index 24fbce61..f9631e74 100644 --- a/crates/socket-patch-core/src/vendor/pnpm_lock_legacy.rs +++ b/crates/socket-patch-core/src/vendor/pnpm_lock_legacy.rs @@ -2198,6 +2198,56 @@ packages: } } + // ── source-flip / outage idempotence (vendor::test_support::npm_flip_suite) ── + + impl crate::vendor::test_support::FlipFixture for Fixture { + fn flip_root(&self) -> &Path { + self.root() + } + fn flip_key(&self) -> String { + "pkg:npm/left-pad@1.3.0".to_string() + } + fn flip_uuid(&self) -> String { + self.record.uuid.clone() + } + fn flip_artifact_rel(&self) -> String { + self.rel_tgz() + } + fn flip_files(&self) -> Vec { + vec![PACKAGE_JSON.to_string(), PNPM_LOCK.to_string()] + } + } + + async fn flip_run( + fx: &Fixture, + cfg: Option<&crate::vendor::VendorServiceConfig>, + ) -> VendorOutcome { + let blobs = fx.root().join(".socket/blobs"); + vendor_pnpm_legacy( + "pkg:npm/left-pad@1.3.0", + &fx.installed(), + fx.root(), + &fx.record, + &PatchSources::blobs_only(&blobs), + "2026-08-18T00:00:00Z", + false, + false, + cfg, + ) + .await + } + + async fn flip_fixture_v5() -> Fixture { + fixture_with(T_BEFORE_PKG, T7_BEFORE_LOCK).await + } + + async fn flip_fixture_v6() -> Fixture { + fixture_with(T_BEFORE_PKG, T8_BEFORE_LOCK).await + } + + crate::vendor::test_support::npm_flip_suite!(flip_suite_v5, Fixture, flip_fixture_v5, flip_run); + crate::vendor::test_support::npm_flip_suite!(flip_suite_v6, Fixture, flip_fixture_v6, flip_run); + async fn fixture_with(pkg_json: &str, lock: &str) -> Fixture { let tmp = tempfile::tempdir().unwrap(); let root = tmp.path(); diff --git a/crates/socket-patch-core/src/vendor/pypi.rs b/crates/socket-patch-core/src/vendor/pypi.rs index 19bfe0b3..4212d36f 100644 --- a/crates/socket-patch-core/src/vendor/pypi.rs +++ b/crates/socket-patch-core/src/vendor/pypi.rs @@ -35,8 +35,10 @@ use super::pypi_uv::{ check_target_guards, load_uv_project, revert_uv, wire_uv, UvProject, UvTarget, }; use super::pypi_wheel::{ - build_patched_wheel, locate_installed_dist, wheel_file_name, WheelArtifact, + build_patched_wheel, escape_wheel_version, locate_installed_dist, wheel_file_name, + WheelArtifact, }; +use super::reuse; use super::service_fetch::{fetch_verified_archive, ServiceArtifact}; use super::state::{ write_marker_or_warn, PdmMeta, PipenvMeta, PoetryMeta, UvMeta, VendorArtifact, VendorEntry, @@ -770,19 +772,72 @@ pub async fn vendor_pypi_with_pipenv_version( // flavor pre-flight read out of it. Only when the wired file yields no // pin either does the unguarded rebuild remain (the local build is // deterministic for locally-vendored projects). + // + // The ledger entry anchoring this uuid (read once): the rebuild pin, the + // Fresh-path reuse anchor, and the PDM partial-relock guard's prior sha. + let prior: Option = reuse::prior_entry(project_root, "pypi", record, None) + .await + .ok(); let expected_pin: Option<(String, String)> = if in_sync { - match super::state::load_state(project_root).await { - Ok(state) => state - .entries - .into_values() - .find(|e| e.ecosystem == "pypi" && e.uuid == record.uuid) - .map(|e| (e.artifact.path, e.artifact.sha256)), - Err(_) => None, - } - .or(wired_pin) + prior + .as_ref() + .map(|e| (e.artifact.path.clone(), e.artifact.sha256.clone())) + .or(wired_pin) + } else { + None + }; + + // Fresh-path reuse: the wiring dropped the vendored reference (a relock + // restored the registry unit) but the committed wheel the ledger + // vouches for is intact — re-wire those exact bytes instead of acquiring + // anew, so the re-scan pins the first run's sha whichever source is + // reachable now (no service call, no local build). + // + // The probe is read-only and offline, so a dry run runs it too: its + // preview must agree with the real run, which re-wires without the + // service, the installed dist or the blobs (and so is never refused by + // `service` + `--offline`). + let reused_wheel = if !in_sync { + fresh_reuse_wheel( + base, + project_root, + &uuid_dir_rel, + record, + prior.as_ref(), + &canon_name, + version, + ) + .await } else { None }; + if dry_run { + if let Some(acquired) = &reused_wheel { + warnings.push(VendorWarning::new( + "vendor_artifact_reused", + format!( + "would re-wire the committed wheel {} for {base} (no rebuild, no service \ + download)", + acquired.rel_wheel + ), + )); + return done( + reuse_preview_result(base, &project_root.join(&acquired.rel_wheel), record), + None, + warnings, + ); + } + } + let reused = reused_wheel.is_some(); + if let Some(acquired) = &reused_wheel { + warnings.push(VendorWarning::new( + "vendor_artifact_reused", + format!( + "re-wired the committed wheel {} for {base} (no rebuild, no service download)", + acquired.rel_wheel + ), + )); + } // Acquire the patched wheel: prefer the prebuilt service artifact (which // skips needing the package installed), else build it locally. A refusal / @@ -794,34 +849,37 @@ pub async fn vendor_pypi_with_pipenv_version( artifact, platform_locked, platform_tags_display, - } = match acquire_patched_wheel( - base, - raw_name, - version, - site_packages, - &uuid_dir_rel, - project_root, - record, - sources, - dry_run, - force, - service, - expected_pin.as_ref(), - &mut warnings, - ) - .await - { - Ok(a) => a, - Err(outcome) => { - // A refused/hard-failed acquisition may have scaffolded the - // empty uuid dir (and the ecosystem / vendor levels on a fresh - // project) before failing: prune them so the failure leaves no - // committable husk. Dry runs create nothing. - if !dry_run { - prune_empty_vendor_levels(&project_root.join(&uuid_dir_rel)).await; + } = match reused_wheel { + Some(acquired) => acquired, + None => match acquire_patched_wheel( + base, + raw_name, + version, + site_packages, + &uuid_dir_rel, + project_root, + record, + sources, + dry_run, + force, + service, + expected_pin.as_ref(), + &mut warnings, + ) + .await + { + Ok(a) => a, + Err(outcome) => { + // A refused/hard-failed acquisition may have scaffolded the + // empty uuid dir (and the ecosystem / vendor levels on a fresh + // project) before failing: prune them so the failure leaves no + // committable husk. Dry runs create nothing. + if !dry_run { + prune_empty_vendor_levels(&project_root.join(&uuid_dir_rel)).await; + } + return outcome; } - return outcome; - } + }, }; if !result.success { prune_empty_vendor_levels(&project_root.join(&uuid_dir_rel)).await; @@ -879,12 +937,29 @@ pub async fn vendor_pypi_with_pipenv_version( prune_empty_vendor_levels(&project_root.join(&uuid_dir_rel)).await; let mut result = result; result.success = false; - result.error = Some(format!( - "the rebuilt wheel ({rel_wheel}, sha256 {}) does not match the wheel the \ - lockfile still pins ({pin_path}, sha256 {pin_sha}); run `socket-patch \ - vendor --revert` for {base} and re-vendor to re-wire the lockfile", - artifact.sha256_hex - )); + // A service outage is the likely cause when the pin came from + // a prebuilt wheel: waiting for the service fixes it, while a + // revert + re-vendor would needlessly re-wire the lockfile. + let service_down = warnings.iter().any(|w| { + w.code == "vendor_prebuilt_unavailable" || w.code == "vendor_prebuilt_pending" + }); + result.error = Some(if service_down { + format!( + "the patch service was unavailable, and the local rebuild ({rel_wheel}, \ + sha256 {}) cannot reproduce the prebuilt wheel the lockfile pins \ + ({pin_path}, sha256 {pin_sha}); re-run vendor once the service is \ + reachable, or run `socket-patch vendor --revert` for {base} and \ + re-vendor to pin a local build", + artifact.sha256_hex + ) + } else { + format!( + "the rebuilt wheel ({rel_wheel}, sha256 {}) does not match the wheel the \ + lockfile still pins ({pin_path}, sha256 {pin_sha}); run `socket-patch \ + vendor --revert` for {base} and re-vendor to re-wire the lockfile", + artifact.sha256_hex + ) + }); return done(result, None, warnings); } } @@ -969,18 +1044,28 @@ pub async fn vendor_pypi_with_pipenv_version( ) .await .map(|(wiring, meta)| (wiring, MetaSlot::Poetry(meta))), - WiringPlan::Pdm(project) => super::pypi_pdm::wire_pdm( - &project, - project_root, - &canon_name, - version, - &rel_wheel, - &wheel_name, - &artifact.sha256_hex, - &record.uuid, - ) - .await - .map(|(wiring, meta)| (wiring, MetaSlot::Pdm(meta))), + WiringPlan::Pdm(project) => { + // Every sha this patch's wheel is known by: this run's, and the + // ledger's (a source flip since the first vendor changes it) — + // the partial-relock guard must not depend on the source. + let mut known_patched: Vec<&str> = vec![artifact.sha256_hex.as_str()]; + if let Some(prior) = &prior { + known_patched.push(prior.artifact.sha256.as_str()); + } + super::pypi_pdm::wire_pdm( + &project, + project_root, + &canon_name, + version, + &rel_wheel, + &wheel_name, + &artifact.sha256_hex, + &record.uuid, + &known_patched, + ) + .await + .map(|(wiring, meta)| (wiring, MetaSlot::Pdm(meta))) + } WiringPlan::Pipenv(project) => super::pypi_pipenv::wire_pipenv( &project, project_root, @@ -998,8 +1083,12 @@ pub async fn vendor_pypi_with_pipenv_version( let (wiring, meta) = match wired { Ok(pair) => pair, Err((code, detail)) => { - let _ = tokio::fs::remove_dir_all(project_root.join(&uuid_dir_rel)).await; - prune_empty_vendor_levels(&project_root.join(&uuid_dir_rel)).await; + // A REUSED wheel is the committed artifact the live ledger entry + // still names: never sweep it (nothing was acquired to undo). + if !reused { + let _ = tokio::fs::remove_dir_all(project_root.join(&uuid_dir_rel)).await; + prune_empty_vendor_levels(&project_root.join(&uuid_dir_rel)).await; + } let mut result = result; result.success = false; result.error = Some(format!("{code}: {detail}")); @@ -1337,6 +1426,106 @@ pub async fn revert_pypi_opts( /// The patched wheel plus the facts the wiring + ledger need, however it was /// acquired (service download or local build). +/// The committed wheel for a Fresh-plan re-run, when the ledger anchors it +/// and it verifies (see [`reuse`]): directly under `uuid_dir_rel`, a +/// well-formed wheel filename for THIS distribution and version (the leaf +/// comes from the committed ledger, and the wirings splice it verbatim into +/// requirements.txt / uv.lock / poetry.lock — see [`reusable_wheel_leaf`]), +/// and not platform-locked by either the ledger flag or the filename's own +/// tags (a platform-specific wheel committed on another OS keeps today's +/// acquire-and-pin behavior). `None` acquires as usual. +async fn fresh_reuse_wheel( + base: &str, + project_root: &Path, + uuid_dir_rel: &str, + record: &PatchRecord, + prior: Option<&VendorEntry>, + canon_name: &str, + version: &str, +) -> Option { + let prior = prior?; + let rel = prior.artifact.path.replace('\\', "/"); + let leaf = match rel + .strip_prefix(uuid_dir_rel) + .and_then(|rest| rest.strip_prefix('/')) + .filter(|leaf| reusable_wheel_leaf(leaf, canon_name, version)) + { + Some(leaf) => leaf.to_string(), + None => { + reuse::log_miss(base, &reuse::ReuseMiss::PathUnsafe); + return None; + } + }; + let (locked, platform_tags_display) = wheel_platform_from_filename(&leaf); + if locked || prior.artifact.platform_locked == Some(true) { + reuse::log_miss(base, &reuse::ReuseMiss::PlatformLocked); + return None; + } + let art = match reuse::verify_committed_artifact(project_root, prior, record).await { + Ok(art) => art, + Err(miss) => { + reuse::log_miss(base, &miss); + return None; + } + }; + let abs = project_root.join(&art.rel_path); + Some(AcquiredWheel { + rel_wheel: art.rel_path.clone(), + result: already_patched_result(base, &abs, &record.files), + artifact: Some(WheelArtifact { + file_name: leaf.clone(), + sha256_hex: art.entry.artifact.sha256.to_ascii_lowercase(), + size: art.bytes.len() as u64, + }), + platform_tags_display, + wheel_name: leaf, + platform_locked: false, + }) +} + +/// A ledger-supplied wheel leaf is reusable only as a well-formed PEP 427 +/// filename (`name-version(-build)?-py-abi-plat.whl`) in the wheel-filename +/// charset — no whitespace, control, `#`, `;` or `/`, which a wiring would +/// splice verbatim into a requirements line or lock path — whose name is +/// THIS distribution (PEP 503-normalized) and whose version is THIS version +/// (as [`escape_wheel_version`] spells it, ASCII case-insensitively). +fn reusable_wheel_leaf(leaf: &str, canon_name: &str, version: &str) -> bool { + let Some(stem) = leaf.strip_suffix(".whl") else { + return false; + }; + if !stem + .bytes() + .all(|b| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'_' | b'+' | b'!' | b'-')) + { + return false; + } + let parts: Vec<&str> = stem.split('-').collect(); + if !(parts.len() == 5 || parts.len() == 6) || parts.iter().any(|p| p.is_empty()) { + return false; + } + canonicalize_pypi_name(parts[0]) == canonicalize_pypi_name(canon_name) + && parts[1].eq_ignore_ascii_case(&escape_wheel_version(version)) +} + +/// The dry-run preview of a Fresh-path reuse: the shape a dry-run local +/// build reports (every patched file verified, ready to wire) — the CLI +/// renders it as a verified preview, as it would the build. +fn reuse_preview_result(base: &str, abs: &Path, record: &PatchRecord) -> ApplyResult { + let files_verified = record + .files + .keys() + .map(|f| crate::patch::apply::VerifyResult { + file: f.clone(), + status: crate::patch::apply::VerifyStatus::Ready, + message: None, + current_hash: None, + expected_hash: None, + target_hash: None, + }) + .collect(); + super::common::synthesized_result(base, abs, files_verified, true, None) +} + struct AcquiredWheel { wheel_name: String, rel_wheel: String, @@ -2517,12 +2706,15 @@ wheels = [ ) -> VendorServiceConfig { VendorServiceConfig { source, - client: Some(ApiClient::new(ApiClientOptions { - api_url: server_uri.to_string(), - api_token: Some("sktsec_placeholder_value_for_tests_api".into()), - use_public_proxy: false, - org_slug: Some("acme".into()), - })), + client: Some( + ApiClient::new(ApiClientOptions { + api_url: server_uri.to_string(), + api_token: Some("sktsec_placeholder_value_for_tests_api".into()), + use_public_proxy: false, + org_slug: Some("acme".into()), + }) + .with_vendor_retry(crate::api::client::VendorRetryPolicy::none()), + ), use_public_proxy: false, vendor_url: None, patch_server_url: None, @@ -6076,6 +6268,518 @@ wheels = [{url = "https://files.pythonhosted.org/six.whl", hash = "sha256:upstre .any(|warning| warning.code == "pypi_unmatched_lockfiles")); } + // ─────────────── source-flip / outage idempotence ─────────────── + // + // In-sync re-runs never consult the service (the wiring is the anchor). + // A relock that dropped the wiring (a Fresh plan) re-wires the COMMITTED + // wheel the ledger vouches for, whichever source is reachable now; the + // PDM partial-relock guard knows every sha the patch's wheel went by. + mod outage_idempotence { + use super::*; + use crate::vendor::test_support as ts; + use std::io::{Read as _, Write as _}; + + const KEY: &str = "pkg:pypi/six@1.16.0"; + const WIRED_FILES: [&str; 7] = [ + "poetry.lock", + "pdm.lock", + "Pipfile.lock", + "Pipfile", + "uv.lock", + "requirements.txt", + "pyproject.toml", + ]; + const HATCH_PYPROJECT: &str = "[build-system]\nrequires = [\"hatchling\"]\nbuild-backend = \"hatchling.build\"\n\n[project]\nname = \"proj\"\nversion = \"0.1.0\"\ndependencies = [\"six==1.16.0\"]\n"; + + type Snap = Vec>>; + + async fn snap(fx: &E2eFixture) -> Snap { + let mut out = Vec::new(); + for f in WIRED_FILES { + out.push(tokio::fs::read(fx.root.join(f)).await.ok()); + } + out + } + + async fn restore(fx: &E2eFixture, snap: &Snap) { + for (f, bytes) in WIRED_FILES.iter().zip(snap) { + match bytes { + Some(b) => tokio::fs::write(fx.root.join(f), b).await.unwrap(), + None => { + let _ = tokio::fs::remove_file(fx.root.join(f)).await; + } + } + } + } + + fn wheel(fx: &E2eFixture) -> PathBuf { + uuid_dir_of(fx).join(WHEEL_NAME) + } + + /// The deterministic local build's wheel (from a throwaway copy). + async fn local_wheel() -> Vec { + let probe = e2e_fixture().await; + let sources = PatchSources::blobs_only(&probe.blobs); + let (r, e, _) = ts::expect_done(vendor_six(&probe, &sources, None).await); + assert!(r.success && e.is_some(), "{:?}", r.error); + tokio::fs::read(wheel(&probe)).await.unwrap() + } + + /// The same members re-encoded (stored, not deflated): the stand-in + /// for the service's prebuilt wheel — verifies, different sha. + fn rezip(whl: &[u8]) -> Vec { + let mut src = zip::ZipArchive::new(std::io::Cursor::new(whl)).unwrap(); + let mut out = zip::ZipWriter::new(std::io::Cursor::new(Vec::new())); + let opts: zip::write::SimpleFileOptions = zip::write::SimpleFileOptions::default() + .compression_method(zip::CompressionMethod::Stored); + for i in 0..src.len() { + let mut entry = src.by_index(i).unwrap(); + let mut bytes = Vec::new(); + entry.read_to_end(&mut bytes).unwrap(); + out.start_file(entry.name().to_string(), opts).unwrap(); + out.write_all(&bytes).unwrap(); + } + let alt = out.finish().unwrap().into_inner(); + assert_ne!(alt, whl); + alt + } + + fn sha(bytes: &[u8]) -> String { + hex::encode(Sha256::digest(bytes)) + } + + /// One run against a fresh mock: `serve` = the prebuilt wheel, or + /// `None` for a 503 outage. Returns the outcome + the request count. + async fn run( + fx: &E2eFixture, + serve: Option<&[u8]>, + source: VendorSource, + offline: bool, + ) -> (VendorOutcome, usize) { + let server = wiremock::MockServer::start().await; + match serve { + Some(bytes) => { + mount_pypi_granted(&server, WHEEL_NAME, &sri_sha512(bytes), bytes).await + } + None => ts::mount_503(&server).await, + } + let sources = PatchSources::blobs_only(&fx.blobs); + let cfg = ts::service_cfg(&server.uri(), source, offline); + let outcome = vendor_six(fx, &sources, Some(&cfg)).await; + (outcome, ts::request_count(&server).await) + } + + /// Run 1, persisted like the CLI does. + async fn first_run(fx: &E2eFixture, serve: Option<&[u8]>) -> VendorEntry { + let (outcome, _) = run(fx, serve, VendorSource::Auto, false).await; + let (r, e, _) = ts::expect_done(outcome); + assert!(r.success, "run 1: {:?}", r.error); + let e = e.expect("run 1 wires"); + ts::persist(&fx.root, KEY, e.clone()).await; + e + } + + async fn flavor_fixture(files: &[(&str, &str)]) -> E2eFixture { + let fx = e2e_fixture().await; + if !files.is_empty() { + swap_to_lock_flavor(&fx, files).await; + } + fx + } + + fn flavors() -> Vec<(&'static str, Vec<(&'static str, &'static str)>)> { + vec![ + ("poetry", vec![("poetry.lock", POETRY_LOCK_REGISTRY)]), + ("pdm", vec![("pdm.lock", PDM_LOCK_REGISTRY)]), + ("pipenv", vec![("Pipfile.lock", PIPENV_LOCK_REGISTRY)]), + ( + "uv", + vec![ + ("pyproject.toml", UV_PYPROJECT), + ("uv.lock", UV_LOCK_REGISTRY), + ], + ), + ("requirements", vec![]), + ("hatch", vec![("pyproject.toml", HATCH_PYPROJECT)]), + ] + } + + /// Regression (the analysts' repro): an in-sync re-run after a flip, + /// in both directions, is a no-op with no request, for every flavor. + #[tokio::test] + async fn all_flavors_rerun_flip_is_in_sync() { + let alt = rezip(&local_wheel().await); + for (name, files) in flavors() { + for first_svc in [true, false] { + let fx = flavor_fixture(&files).await; + let first = first_run(&fx, first_svc.then_some(alt.as_slice())).await; + assert_eq!(first.flavor.as_deref(), Some(name), "{name}"); + let s1 = snap(&fx).await; + let w1 = tokio::fs::read(wheel(&fx)).await.unwrap(); + let (outcome, requests) = run( + &fx, + (!first_svc).then_some(alt.as_slice()), + VendorSource::Auto, + false, + ) + .await; + let (r, e, w) = ts::expect_done(outcome); + assert!(r.success, "{name}: {:?}", r.error); + assert!(e.is_none(), "{name}: in sync"); + assert!( + !ts::has_warning(&w, "vendor_prebuilt_unavailable"), + "{name}: {w:?}" + ); + assert_eq!(snap(&fx).await, s1, "{name}: locks byte-identical"); + assert_eq!(tokio::fs::read(wheel(&fx)).await.unwrap(), w1, "{name}"); + assert_eq!(requests, 0, "{name}: no request"); + } + } + } + + /// P1 + P2: a relock restored the registry unit (the wiring dropped), + /// re-run under the OTHER source: the committed wheel is re-wired — + /// entry Some, the first run's sha pinned, wheel bytes unchanged, no + /// request, and the `vendor_artifact_reused` advisory. + #[tokio::test] + async fn relock_rescan_rewires_the_committed_wheel_without_network() { + let local = local_wheel().await; + let alt = rezip(&local); + for (name, files) in flavors() + .into_iter() + .filter(|(n, _)| ["pdm", "uv", "poetry"].contains(n)) + { + for first_svc in [true, false] { + let fx = flavor_fixture(&files).await; + let registry = snap(&fx).await; + let first = first_run(&fx, first_svc.then_some(alt.as_slice())).await; + let committed = tokio::fs::read(wheel(&fx)).await.unwrap(); + assert_eq!(&committed, if first_svc { &alt } else { &local }, "{name}"); + assert_eq!(first.artifact.sha256, sha(&committed)); + let wired = snap(&fx).await; + restore(&fx, ®istry).await; // the relock + let (outcome, requests) = run( + &fx, + (!first_svc).then_some(alt.as_slice()), + VendorSource::Auto, + false, + ) + .await; + let (r, e, w) = ts::expect_done(outcome); + assert!(r.success, "{name}: {:?}", r.error); + let e = e.expect("the relocked wiring is re-applied"); + assert_eq!(e.artifact.sha256, first.artifact.sha256, "{name}"); + assert_eq!(e.artifact.path, first.artifact.path, "{name}"); + assert!( + ts::has_warning(&w, "vendor_artifact_reused"), + "{name}: {w:?}" + ); + assert!( + !ts::has_warning(&w, "vendor_prebuilt_unavailable"), + "{name}: {w:?}" + ); + assert_eq!(requests, 0, "{name}: no request"); + assert_eq!(tokio::fs::read(wheel(&fx)).await.unwrap(), committed); + assert_eq!(snap(&fx).await, wired, "{name}: re-wired byte-identically"); + } + } + } + + /// The Fresh-path reuse runs before the offline conflict: a relock + /// re-scan under `service` + `--offline` still re-wires. + #[tokio::test] + async fn relock_rescan_reuse_survives_service_mode_offline() { + let alt = rezip(&local_wheel().await); + let fx = flavor_fixture(&[("pdm.lock", PDM_LOCK_REGISTRY)]).await; + let registry = snap(&fx).await; + let first = first_run(&fx, Some(&alt)).await; + restore(&fx, ®istry).await; + let (outcome, requests) = run(&fx, None, VendorSource::Service, true).await; + let (r, e, _) = ts::expect_done(outcome); + assert!(r.success, "{:?}", r.error); + assert_eq!(e.unwrap().artifact.sha256, first.artifact.sha256); + assert_eq!(requests, 0); + } + + fn partial_pdm_lock(patched_sha: &str) -> String { + let partial = PDM_LOCK_REGISTRY.replace( + "files = [\n {file = \"six-1.16.0-py2.py3-none-any.whl\", hash = \"sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254\"},\n {file = \"six-1.16.0.tar.gz\", hash = \"sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926\"},\n]", + &format!("files = [\n {{file = \"six-1.16.0-py2.py3-none-any.whl\", hash = \"sha256:{patched_sha}\"}},\n]"), + ); + assert_ne!(partial, PDM_LOCK_REGISTRY); + partial + } + + /// P3 + P4: a `pdm add` partial relock (path dropped, the SERVICE + /// wheel's sha kept) re-run during an outage refuses whichever + /// source is reachable — with the wheel present (reused, sha A) and + /// with it deleted (local rebuild, sha B, still guarded by the + /// ledger's A). The ledger is untouched, and a reused wheel is never + /// swept by the wiring failure. + #[tokio::test] + async fn pdm_partial_relock_after_a_flip_refuses_whichever_source() { + let alt = rezip(&local_wheel().await); + for wheel_present in [true, false] { + let fx = flavor_fixture(&[("pdm.lock", PDM_LOCK_REGISTRY)]).await; + let first = first_run(&fx, Some(&alt)).await; + assert_eq!(first.artifact.sha256, sha(&alt)); + let partial = partial_pdm_lock(&first.artifact.sha256); + tokio::fs::write(fx.root.join("pdm.lock"), &partial) + .await + .unwrap(); + if !wheel_present { + tokio::fs::remove_file(wheel(&fx)).await.unwrap(); + } + let ledger = tokio::fs::read(fx.root.join(".socket/vendor/state.json")) + .await + .unwrap(); + let (outcome, _) = run(&fx, None, VendorSource::Auto, false).await; + let (r, e, _) = ts::expect_done(outcome); + assert!(!r.success, "present={wheel_present}: must refuse"); + assert!(e.is_none()); + let err = r.error.unwrap(); + assert!(err.contains("pypi_pdm_source_already_exists"), "{err}"); + assert_eq!( + tokio::fs::read_to_string(fx.root.join("pdm.lock")) + .await + .unwrap(), + partial, + "lock untouched" + ); + assert_eq!( + tokio::fs::read(fx.root.join(".socket/vendor/state.json")) + .await + .unwrap(), + ledger, + "the ledger original is never replaced by the patched unit" + ); + if wheel_present { + assert_eq!( + tokio::fs::read(wheel(&fx)).await.unwrap(), + alt, + "P4: the reused committed wheel survives the wiring failure" + ); + } + } + } + + /// P5: in-sync wiring, the SERVICE-built wheel missing, service + /// down: fails closed with a message that names the outage; the lock + /// is unchanged and nothing is left behind. + #[tokio::test] + async fn missing_service_wheel_under_outage_names_the_outage() { + let alt = rezip(&local_wheel().await); + let fx = flavor_fixture(&[("pdm.lock", PDM_LOCK_REGISTRY)]).await; + let _ = first_run(&fx, Some(&alt)).await; + let wired = snap(&fx).await; + tokio::fs::remove_file(wheel(&fx)).await.unwrap(); + let (outcome, _) = run(&fx, None, VendorSource::Auto, false).await; + let (r, e, _) = ts::expect_done(outcome); + assert!(!r.success); + assert!(e.is_none()); + let err = r.error.unwrap(); + assert!(err.contains("patch service was unavailable"), "{err}"); + assert!(err.contains("once the service is reachable"), "{err}"); + assert_eq!(snap(&fx).await, wired, "lock unchanged"); + assert!(!wheel(&fx).exists()); + } + + /// P6: a platform-locked ledger entry is never reused on the Fresh + /// path (a platform wheel committed on another OS keeps today's + /// acquire-and-pin behavior). + #[tokio::test] + async fn platform_locked_entry_is_not_reused() { + let alt = rezip(&local_wheel().await); + let fx = flavor_fixture(&[("pdm.lock", PDM_LOCK_REGISTRY)]).await; + let registry = snap(&fx).await; + let mut first = first_run(&fx, Some(&alt)).await; + first.artifact.platform_locked = Some(true); + ts::persist(&fx.root, KEY, first.clone()).await; + restore(&fx, ®istry).await; + let (outcome, requests) = run(&fx, None, VendorSource::Auto, false).await; + let (r, e, w) = ts::expect_done(outcome); + assert!(r.success, "{:?}", r.error); + assert!(!ts::has_warning(&w, "vendor_artifact_reused"), "{w:?}"); + assert!(ts::has_warning(&w, "vendor_prebuilt_unavailable"), "{w:?}"); + assert_ne!(e.unwrap().artifact.sha256, first.artifact.sha256); + assert_eq!(requests, 1, "acquisition ran (the 503 POST)"); + } + + /// Dry run of the relock re-scan: the preview agrees with the real + /// run (which re-wires offline, see above) — success, a verified + /// preview, the reuse note, nothing written, no request — instead + /// of the `service` + `--offline` refusal the acquisition preview + /// raised. + #[tokio::test] + async fn relock_rescan_dry_run_previews_the_reuse_under_service_offline() { + let alt = rezip(&local_wheel().await); + let fx = flavor_fixture(&[("pdm.lock", PDM_LOCK_REGISTRY)]).await; + let registry = snap(&fx).await; + let _ = first_run(&fx, Some(&alt)).await; + restore(&fx, ®istry).await; + let server = wiremock::MockServer::start().await; + ts::mount_503(&server).await; + let cfg = ts::service_cfg(&server.uri(), VendorSource::Service, true); + let outcome = vendor_pypi( + KEY, + &fx.site_packages, + &fx.root, + &fx.record, + &PatchSources::blobs_only(&fx.blobs), + "2026-06-09T00:00:00Z", + true, + false, + Some(&cfg), + ) + .await; + let (r, e, w) = ts::expect_done(outcome); + assert!(r.success, "{:?}", r.error); + assert!(e.is_none(), "a dry run records nothing"); + assert!(ts::has_warning(&w, "vendor_artifact_reused"), "{w:?}"); + assert!( + r.files_verified + .iter() + .all(|f| f.status == crate::patch::apply::VerifyStatus::Ready), + "a verified preview, as the dry-run build reports" + ); + assert_eq!(snap(&fx).await, registry, "nothing wired"); + assert_eq!(tokio::fs::read(wheel(&fx)).await.unwrap(), alt); + assert_eq!(ts::request_count(&server).await, 0); + } + + /// Rename the committed wheel to `leaf` and point every ledger + /// entry at it (a forged, committed state.json), then relock. + async fn forge_leaf(fx: &E2eFixture, registry: &Snap, leaf: &str) { + tokio::fs::rename(wheel(fx), uuid_dir_of(fx).join(leaf)) + .await + .unwrap(); + let state_p = fx.root.join(".socket/vendor/state.json"); + let mut state: crate::vendor::state::VendorState = + serde_json::from_slice(&tokio::fs::read(&state_p).await.unwrap()).unwrap(); + for e in state.entries.values_mut() { + let dir = e.artifact.path.rsplit_once('/').unwrap().0.to_string(); + e.artifact.path = format!("{dir}/{leaf}"); + } + tokio::fs::write(&state_p, serde_json::to_vec_pretty(&state).unwrap()) + .await + .unwrap(); + restore(fx, registry).await; + } + + /// The reused leaf comes from the committed ledger and is spliced + /// verbatim into the wiring: a leaf carrying a newline must never + /// inject a requirements.txt option line, and a leaf naming another + /// distribution must never be wired for this one. + #[tokio::test] + async fn forged_ledger_leaf_is_never_reused() { + for leaf in [ + "six-1.16.0-py3-none-any.whl\n--trusted-host evil.example\n#.whl", + "evil-9.9-py3-none-any.whl", + "six-6.6.6-py3-none-any.whl", + ] { + // Windows rejects a newline in a filename, so the forged + // file cannot exist there to tempt the reuse path. + if cfg!(windows) && leaf.contains('\n') { + continue; + } + let fx = flavor_fixture(&[]).await; + let registry = snap(&fx).await; + let _ = first_run(&fx, None).await; + forge_leaf(&fx, ®istry, leaf).await; + let (outcome, _) = run(&fx, None, VendorSource::Auto, false).await; + let (r, _, w) = ts::expect_done(outcome); + assert!( + !ts::has_warning(&w, "vendor_artifact_reused"), + "{leaf:?}: {w:?}" + ); + let req = tokio::fs::read_to_string(fx.root.join("requirements.txt")) + .await + .unwrap(); + assert!( + !req.lines() + .any(|l| l.trim_start().starts_with("--trusted-host")), + "{leaf:?}: injected\n{req}" + ); + assert!(!req.contains("evil"), "{leaf:?}\n{req}"); + assert!(r.success, "{leaf:?}: acquisition re-vendors: {:?}", r.error); + } + } + + /// A platform-specific wheel (by its own filename tags) is not + /// reused even when the ledger lacks the `platform_locked` flag. + #[tokio::test] + async fn platform_tagged_leaf_without_ledger_flag_is_not_reused() { + let fx = flavor_fixture(&[("pdm.lock", PDM_LOCK_REGISTRY)]).await; + let registry = snap(&fx).await; + let mut first = first_run(&fx, None).await; + first.artifact.platform_locked = None; + ts::persist(&fx.root, KEY, first).await; + forge_leaf( + &fx, + ®istry, + "six-1.16.0-cp311-cp311-manylinux_2_17_x86_64.whl", + ) + .await; + let (outcome, requests) = run(&fx, None, VendorSource::Auto, false).await; + let (_, _, w) = ts::expect_done(outcome); + assert!(!ts::has_warning(&w, "vendor_artifact_reused"), "{w:?}"); + assert_eq!(requests, 1, "acquisition ran (the 503 POST)"); + } + + #[test] + fn reusable_wheel_leaf_accepts_only_this_dist_and_version() { + assert!(reusable_wheel_leaf( + "six-1.16.0-py2.py3-none-any.whl", + "six", + "1.16.0" + )); + assert!(reusable_wheel_leaf( + "Six-1.16.0-1-py3-none-any.whl", + "six", + "1.16.0" + )); + assert!(reusable_wheel_leaf( + "zope_interface-5.0-py3-none-any.whl", + "zope-interface", + "5.0" + )); + assert!(reusable_wheel_leaf( + "torch-2.0.0+cu118-cp311-cp311-linux_x86_64.whl", + "torch", + "2.0.0+cu118" + )); + for bad in [ + "six-1.16.0-py3-none-any.whl\n--x\n#.whl", + "six-1.16.0-py3-none-any .whl", + "six-1.16.0-py3-none-any.whl#x.whl", + "evil-1.16.0-py3-none-any.whl", + "six-1.17.0-py3-none-any.whl", + "six-1.16.0-any.whl", + "six-1.16.0-a-b-py3-none-any.whl", + "six-1.16.0-py3-none-any.tar.gz", + "six--1.16.0-py3-none-any.whl", + ] { + assert!(!reusable_wheel_leaf(bad, "six", "1.16.0"), "{bad:?}"); + } + } + + /// P7: the in-sync path is unchanged — `service` mode + 503 on an + /// already-vendored package is `already_vendored`. + #[tokio::test] + async fn service_mode_in_sync_rerun_under_outage_is_already_vendored() { + let alt = rezip(&local_wheel().await); + let fx = flavor_fixture(&[("pdm.lock", PDM_LOCK_REGISTRY)]).await; + let _ = first_run(&fx, Some(&alt)).await; + let wired = snap(&fx).await; + let (outcome, requests) = run(&fx, None, VendorSource::Service, false).await; + let (r, e, _) = ts::expect_done(outcome); + assert!(r.success, "{:?}", r.error); + assert!(e.is_none()); + assert_eq!(snap(&fx).await, wired); + assert_eq!(requests, 0); + } + } + /// An integrity mismatch is a hard failure under `auto` too — /// never a quiet local-build fallback (service_fetch's contract). #[tokio::test] diff --git a/crates/socket-patch-core/src/vendor/pypi_pdm.rs b/crates/socket-patch-core/src/vendor/pypi_pdm.rs index dcb8a825..3966d968 100644 --- a/crates/socket-patch-core/src/vendor/pypi_pdm.rs +++ b/crates/socket-patch-core/src/vendor/pypi_pdm.rs @@ -322,8 +322,10 @@ fn check_target_unit( Ok(PdmTarget::Fresh) } -/// `true` when the target `[[package]]` unit's `files` already list THIS run's -/// patched wheel sha256. +/// `true` when the target `[[package]]` unit's `files` list ANY sha256 this +/// patch's wheel is known by (this run's, and the ledger's — after a +/// service ↔ local source flip the two differ, and the guard must hold +/// whichever source built the wheel the stale unit still carries). /// /// A `pdm add ` / partial relock on an already-vendored lock reuses the /// stale `files` entry — which still carries our PATCHED wheel hash — while @@ -339,9 +341,13 @@ fn check_target_unit( fn target_carries_patched_wheel_hash( lock: &DocumentMut, canon_name: &str, - wheel_sha256_hex: &str, + known_patched_sha256: &[&str], ) -> bool { - let needle = format!("sha256:{}", wheel_sha256_hex.to_ascii_lowercase()); + let needles: Vec = known_patched_sha256 + .iter() + .filter(|sha| !sha.is_empty()) + .map(|sha| format!("sha256:{}", sha.to_ascii_lowercase())) + .collect(); lock_units_named(lock, canon_name).into_iter().any(|unit| { crate::utils::pdm_lock::files_for(lock, unit) .map(|files| { @@ -349,7 +355,7 @@ fn target_carries_patched_wheel_hash( file.as_inline_table() .and_then(|table| table.get("hash")) .and_then(Value::as_str) - == Some(needle.as_str()) + .is_some_and(|hash| needles.iter().any(|n| n == hash)) }) }) .unwrap_or(false) @@ -361,6 +367,8 @@ fn target_carries_patched_wheel_hash( /// committed atomically). `rel_wheel` is the project-relative wheel path /// (`.socket/vendor/pypi//`, no `./` prefix — the `./` idiom of /// pdm's own `path` serialization is applied here, fixture-pinned). +/// `known_patched_sha256`: every sha256 this patch's wheel is known by +/// (this run's and the ledger's), for the partial-relock guard. #[allow(clippy::too_many_arguments)] pub async fn wire_pdm( p: &PdmProject, @@ -371,6 +379,7 @@ pub async fn wire_pdm( wheel_file_name: &str, wheel_sha256_hex: &str, record_uuid: &str, + known_patched_sha256: &[&str], ) -> Result<(Vec, PdmMeta), (&'static str, String)> { // Before ANY write: a symlinked lock would be replaced by the rename-over. refuse_symlinked(root, &[LOCK_FILE], "pypi_pdm_symlink_unsupported").await?; @@ -399,7 +408,7 @@ pub async fn wire_pdm( // true registry original would be lost. Refuse with the repair path (a full // `pdm lock` re-resolves the registry hashes) so the ledger's real original // survives untouched. - if target_carries_patched_wheel_hash(&p.lock, canon_name, wheel_sha256_hex) { + if target_carries_patched_wheel_hash(&p.lock, canon_name, known_patched_sha256) { return Err(( "pypi_pdm_source_already_exists", format!( @@ -727,7 +736,15 @@ distribution = false async fn wire_default(p: &PdmProject, root: &Path) -> (Vec, PdmMeta) { wire_pdm( - p, root, "six", "1.16.0", REL_WHEEL, WHEEL_NAME, WHEEL_SHA, UUID, + p, + root, + "six", + "1.16.0", + REL_WHEEL, + WHEEL_NAME, + WHEEL_SHA, + UUID, + &[WHEEL_SHA], ) .await .unwrap() @@ -951,6 +968,7 @@ distribution = false WHEEL_NAME, WHEEL_SHA, UUID, + &[WHEEL_SHA], ) .await .unwrap_err(); @@ -962,6 +980,53 @@ distribution = false ); } + /// Partial relock after a service ↔ local source flip: the stale unit + /// carries the PRIOR (ledger) sha, not this run's. The guard matches any + /// known patched sha, so it still refuses — and without the prior sha + /// it would have wired over it (the pre-fix, source-dependent hole). + #[tokio::test] + async fn partial_relock_guard_matches_any_known_patched_sha() { + let prior_sha = "c".repeat(64); + let stale = LOCK_DIRECT_REGISTRY.replace( + "8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254", + &prior_sha, + ); + assert_ne!(stale, LOCK_DIRECT_REGISTRY); + let tmp = write_project(&stale, PYPROJECT_DIRECT).await; + let p = load_pdm_project(tmp.path()).await.unwrap(); + let err = wire_pdm( + &p, + tmp.path(), + "six", + "1.16.0", + REL_WHEEL, + WHEEL_NAME, + WHEEL_SHA, + UUID, + &[WHEEL_SHA, prior_sha.as_str()], + ) + .await + .unwrap_err(); + assert_eq!(err.0, "pypi_pdm_source_already_exists"); + assert!(err.1.contains("pdm lock"), "{}", err.1); + assert_eq!(read_lock(tmp.path()).await, stale, "refusal writes nothing"); + + // Only this run's sha known: the stale unit reads as a registry one. + assert!(wire_pdm( + &p, + tmp.path(), + "six", + "1.16.0", + REL_WHEEL, + WHEEL_NAME, + WHEEL_SHA, + UUID, + &[WHEEL_SHA], + ) + .await + .is_ok()); + } + #[tokio::test] async fn unknown_lock_version_refuses_before_writing() { let lock = LOCK_DIRECT_REGISTRY.replace("4.5.0", "4.6.0"); @@ -1016,6 +1081,7 @@ distribution = false WHEEL_NAME, WHEEL_SHA, UUID, + &[WHEEL_SHA], ) .await .unwrap_err(); @@ -1201,6 +1267,7 @@ distribution = false WHEEL_NAME, WHEEL_SHA, UUID, + &[WHEEL_SHA], ) .await; @@ -1438,7 +1505,15 @@ distribution = false let p = load_pdm_project(&root).await.unwrap(); let err = wire_pdm( - &p, &root, "six", "1.16.0", REL_WHEEL, WHEEL_NAME, WHEEL_SHA, UUID, + &p, + &root, + "six", + "1.16.0", + REL_WHEEL, + WHEEL_NAME, + WHEEL_SHA, + UUID, + &[WHEEL_SHA], ) .await .unwrap_err(); @@ -1510,6 +1585,7 @@ distribution = false WHEEL_NAME, WHEEL_SHA, UUID, + &[WHEEL_SHA], ) .await .unwrap_err(); diff --git a/crates/socket-patch-core/src/vendor/pypi_wheel.rs b/crates/socket-patch-core/src/vendor/pypi_wheel.rs index 445285b6..0f85a932 100644 --- a/crates/socket-patch-core/src/vendor/pypi_wheel.rs +++ b/crates/socket-patch-core/src/vendor/pypi_wheel.rs @@ -202,7 +202,7 @@ fn escape_wheel_component(s: &str) -> String { /// version (`packaging.utils.parse_wheel_filename` raises on /// `torch-2.0.0_cu118-…` but parses `torch-2.0.0+cu118-…`), so escaping /// those two would make the rebuilt wheel uninstallable. -fn escape_wheel_version(s: &str) -> String { +pub(crate) fn escape_wheel_version(s: &str) -> String { escape_wheel_chars(s, true) } diff --git a/crates/socket-patch-core/src/vendor/reuse.rs b/crates/socket-patch-core/src/vendor/reuse.rs new file mode 100644 index 00000000..79a9eeb5 --- /dev/null +++ b/crates/socket-patch-core/src/vendor/reuse.rs @@ -0,0 +1,905 @@ +//! Reuse of an already-committed, file-shaped vendored artifact (npm +//! tarball, pypi wheel) instead of acquiring a new one. +//! +//! The directory-shaped backends (cargo, golang, composer, gem, maven, +//! nuget) decide "in sync" from the COMMITTED artifact before they ever +//! consult the patch service. The archive-shaped backends used to acquire +//! first (service download, else a local deterministic pack) and compare the +//! lock's digests with those NEW bytes — so a prebuilt ↔ local source flip +//! between two runs (a service outage, or its recovery) rewrote the lock and +//! the tarball even though nothing needed vendoring. This module gives them +//! the same rule: when the ledger vouches for the committed artifact and the +//! bytes verify, reuse them. +//! +//! Anchor: the vendor ledger entry (`.socket/vendor/state.json`) recorded +//! the artifact's path + sha256 when it was wired. Reuse requires, fail +//! closed at every step (any miss falls through to the caller's normal +//! acquisition, exactly today's behavior): +//! +//! 1. a non-empty patch record (nothing to verify ⇒ never reused); +//! 2. a canonical, uuid-bound artifact path (`checked_artifact_path`) — an +//! artifact under another uuid's directory is never reused; +//! 3. no symlink anywhere on the path below the project root; +//! 4. a regular file (FIFO-safe open), at most `MAX_HEALTH_HASH_BYTES`, read +//! ONCE into memory — every later check runs on that one buffer; +//! 5. `sha256(bytes)` == the ledger sha256 (and the ledger size, when +//! recorded) — the tamper anchor for unpatched members and re-encodings; +//! 6. the archive is CANONICAL (strict decode: every tarball entry under +//! `package/`, only regular/directory entries, no exact or case-folded +//! duplicate names; the same name rules for wheels) — so the decoded +//! members are exactly what an installer extracts, and the afterHash +//! check below cannot be satisfied by one entry while a sibling the +//! installer prefers (another top-level dir, a type-`7` twin, a +//! case-variant name) carries different bytes; +//! 7. every `record.files` afterHash verifies inside the decoded members. +//! +//! The lockfile is deliberately NOT an input: the flavor's own in-sync code +//! runs afterwards against the reused bytes' facts, so a lock that already +//! pins them is a true no-op and a lock that drifted is re-pinned to the +//! verified committed bytes. +//! +//! Residual trust (the same level `repair` and `vex`'s +//! `check_vendored_artifact` already grant the ledger): an attacker who edits +//! an unpatched member AND rewrites the ledger sha256 keeps the edit, because +//! the afterHash check only covers patched members. +//! +//! Read-only: nothing here writes or touches the network. + +use std::collections::HashMap; +use std::path::Path; + +use sha2::{Digest, Sha256}; + +use crate::manifest::schema::PatchRecord; +use crate::utils::env_compat::is_debug_enabled; + +use super::state::{load_state, VendorEntry}; +use super::verify::{ + checked_artifact_path, read_zip_bytes_to_map_strict, verify_member_map, MAX_HEALTH_HASH_BYTES, +}; + +/// A committed artifact that passed every reuse check. +#[derive(Debug)] +pub(crate) struct CommittedArtifact { + /// Normalized (forward-slashed) ledger `artifact.path`. + pub rel_path: String, + /// The EXACT bytes that were hashed and verified (read once). + pub bytes: Vec, + /// Members decoded from `bytes` (tarball keys `package/`-stripped, the + /// `normalize_file_path` key space; wheel keys as stored). + pub members: HashMap>, + /// The anchoring ledger entry. + pub entry: VendorEntry, +} + +/// Why the committed artifact was not reused. Never surfaced to users (a +/// miss simply means "acquire as usual"); debug-logged only. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum ReuseMiss { + NoFiles, + NoLedger, + NoEntry, + Ambiguous, + PathUnsafe, + UuidMismatch, + Missing, + NotRegular, + TooLarge, + Sha256Mismatch, + SizeMismatch, + Unreadable, + NonCanonical, + MemberMismatch(String), + PlatformLocked, +} + +/// Debug-log a reuse miss (`SOCKET_PATCH_DEBUG`); the caller then acquires. +pub(crate) fn log_miss(purl: &str, miss: &ReuseMiss) { + if is_debug_enabled() { + eprintln!("[socket-patch debug] vendor reuse skipped for {purl}: {miss:?}"); + } +} + +fn norm(path: &str) -> String { + path.replace('\\', "/") +} + +/// The ledger entry anchoring `record.uuid` for `ecosystem`: same ecosystem +/// and uuid, a non-empty artifact sha256, and (when given) a normalized +/// artifact path equal to `expected_rel`. Qualified-purl twins may each +/// carry an entry for the same uuid; they must all agree on (path, sha256) +/// or the answer is [`ReuseMiss::Ambiguous`]. An unreadable ledger is +/// [`ReuseMiss::NoLedger`] (reuse is skipped; acquisition runs as before). +pub(crate) async fn prior_entry( + project_root: &Path, + ecosystem: &str, + record: &PatchRecord, + expected_rel: Option<&str>, +) -> Result { + let state = load_state(project_root) + .await + .map_err(|_| ReuseMiss::NoLedger)?; + let expected = expected_rel.map(norm); + let mut hits: Vec = state + .entries + .into_values() + .filter(|e| { + e.ecosystem == ecosystem + && e.uuid == record.uuid + && !e.artifact.sha256.is_empty() + && expected + .as_deref() + .is_none_or(|want| norm(&e.artifact.path) == want) + }) + .collect(); + let Some(first) = hits.pop() else { + return Err(ReuseMiss::NoEntry); + }; + let agree = hits.iter().all(|e| { + norm(&e.artifact.path) == norm(&first.artifact.path) + && e.artifact + .sha256 + .eq_ignore_ascii_case(&first.artifact.sha256) + }); + if !agree { + return Err(ReuseMiss::Ambiguous); + } + Ok(first) +} + +/// Verify the committed artifact `entry` names against the ledger anchor and +/// `record`'s afterHashes (see the module docs for the ordered checks). +/// Read-only; never touches the network. +pub(crate) async fn verify_committed_artifact( + project_root: &Path, + entry: &VendorEntry, + record: &PatchRecord, +) -> Result { + use tokio::io::AsyncReadExt as _; + + if record.files.is_empty() { + return Err(ReuseMiss::NoFiles); + } + let abs = checked_artifact_path(project_root, entry, record).map_err(|tag| { + if tag == "vendor_uuid_mismatch" { + ReuseMiss::UuidMismatch + } else { + ReuseMiss::PathUnsafe + } + })?; + let rel_path = norm(&entry.artifact.path); + + // No link anywhere below the project root: a symlinked uuid dir (or + // artifact) would let bytes from outside the vendored tree be "reused" + // and then pinned into the lock. `checked_artifact_path` already + // rejected `..`/empty segments, so each prefix is a real path level. + let mut prefix = project_root.to_path_buf(); + for seg in rel_path.split('/') { + prefix.push(seg); + match tokio::fs::symlink_metadata(&prefix).await { + Ok(meta) if meta.file_type().is_symlink() => return Err(ReuseMiss::NotRegular), + Ok(_) => {} + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Err(ReuseMiss::Missing), + Err(_) => return Err(ReuseMiss::Unreadable), + } + } + + // One FIFO-safe open (O_NONBLOCK + fstat on the handle): the size gate + // and every byte below come from the same inode. + let (file, meta) = match crate::utils::fs::open_regular_file(&abs).await { + Ok(pair) => pair, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Err(ReuseMiss::Missing), + Err(e) if e.kind() == std::io::ErrorKind::InvalidInput => { + return Err(ReuseMiss::NotRegular) + } + Err(_) => return Err(ReuseMiss::Unreadable), + }; + if meta.len() > MAX_HEALTH_HASH_BYTES { + return Err(ReuseMiss::TooLarge); + } + let mut bytes = Vec::with_capacity(meta.len() as usize); + // +1: a file that grew past the cap after the fstat reads one byte over + // and is rejected rather than truncated. + file.take(MAX_HEALTH_HASH_BYTES + 1) + .read_to_end(&mut bytes) + .await + .map_err(|_| ReuseMiss::Unreadable)?; + if bytes.len() as u64 > MAX_HEALTH_HASH_BYTES { + return Err(ReuseMiss::TooLarge); + } + + // The ledger anchor. + let sha = hex::encode(Sha256::digest(&bytes)); + if !sha.eq_ignore_ascii_case(&entry.artifact.sha256) { + return Err(ReuseMiss::Sha256Mismatch); + } + if entry + .artifact + .size + .is_some_and(|size| size != bytes.len() as u64) + { + return Err(ReuseMiss::SizeMismatch); + } + + // Members from the SAME buffer. + let is_tarball = rel_path.ends_with(".tgz") || rel_path.ends_with(".tar.gz"); + let is_wheel = rel_path.ends_with(".whl"); + if !is_tarball && !is_wheel { + return Err(ReuseMiss::Unreadable); + } + let (bytes, members) = tokio::task::spawn_blocking(move || { + let members = if is_tarball { + crate::patch::package::read_archive_bytes_to_map_strict(&bytes).map_err(|e| match e { + crate::patch::package::ArchiveError::NonCanonical(_) => ReuseMiss::NonCanonical, + _ => ReuseMiss::Unreadable, + }) + } else { + read_zip_bytes_to_map_strict(&bytes).map_err(|e| { + if e == "vendor_artifact_non_canonical" { + ReuseMiss::NonCanonical + } else { + ReuseMiss::Unreadable + } + }) + }; + (bytes, members) + }) + .await + .map_err(|_| ReuseMiss::Unreadable)?; + let members = members?; + verify_member_map(&members, record).map_err(ReuseMiss::MemberMismatch)?; + + Ok(CommittedArtifact { + rel_path, + bytes, + members, + entry: entry.clone(), + }) +} + +/// [`prior_entry`] followed by [`verify_committed_artifact`]. +pub(crate) async fn reusable_committed_artifact( + project_root: &Path, + ecosystem: &str, + record: &PatchRecord, + expected_rel: Option<&str>, +) -> Result { + let entry = prior_entry(project_root, ecosystem, record, expected_rel).await?; + verify_committed_artifact(project_root, &entry, record).await +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::hash::git_sha256::compute_git_sha256_from_bytes; + use crate::manifest::schema::PatchFileInfo; + use crate::vendor::state::{save_state, VendorArtifact, VendorState}; + use std::io::Write as _; + + const UUID: &str = "9f6b2c4e-1d3a-4f6b-8c2d-7e5a9b1c3d5f"; + const OTHER_UUID: &str = "1a2b3c4d-5e6f-4a1b-8c2d-3e4f5a6b7c8d"; + const PATCHED: &[u8] = b"module.exports = 'patched';\n"; + + fn record(uuid: &str) -> PatchRecord { + let mut files = HashMap::new(); + files.insert( + "package/index.js".to_string(), + PatchFileInfo { + before_hash: "a".repeat(64), + after_hash: compute_git_sha256_from_bytes(PATCHED), + }, + ); + PatchRecord { + uuid: uuid.to_string(), + exported_at: String::new(), + files, + vulnerabilities: HashMap::new(), + description: String::new(), + license: String::new(), + tier: String::new(), + } + } + + fn tgz(members: &[(&str, &[u8])]) -> Vec { + let mut builder = tar::Builder::new(flate2::write::GzEncoder::new( + Vec::new(), + flate2::Compression::default(), + )); + for (name, data) in members { + let mut header = tar::Header::new_gnu(); + header.set_size(data.len() as u64); + header.set_mode(0o644); + header.set_cksum(); + builder.append_data(&mut header, name, *data).unwrap(); + } + builder.into_inner().unwrap().finish().unwrap() + } + + fn good_tgz() -> Vec { + tgz(&[ + ("package/index.js", PATCHED), + ("package/package.json", b"{\"name\":\"left-pad\"}"), + ]) + } + + fn rel_for(uuid: &str) -> String { + format!(".socket/vendor/npm/{uuid}/left-pad-1.3.0.tgz") + } + + fn entry_for(uuid: &str, rel: &str, bytes: &[u8]) -> VendorEntry { + VendorEntry { + ecosystem: "npm".into(), + base_purl: "pkg:npm/left-pad@1.3.0".into(), + uuid: uuid.into(), + artifact: VendorArtifact { + path: rel.into(), + sha256: hex::encode(Sha256::digest(bytes)), + size: Some(bytes.len() as u64), + platform_locked: None, + file_inventory: None, + }, + wiring: Vec::new(), + lock: None, + took_over_go_patches: false, + flavor: Some("package-lock".into()), + uv: None, + pnpm: None, + poetry: None, + pdm: None, + pipenv: None, + detached: false, + record: None, + } + } + + /// A project with `bytes` committed at the UUID's tarball path and a + /// ledger entry anchoring them. + async fn project(bytes: &[u8]) -> (tempfile::TempDir, VendorEntry) { + let tmp = tempfile::tempdir().unwrap(); + let rel = rel_for(UUID); + let abs = tmp.path().join(&rel); + tokio::fs::create_dir_all(abs.parent().unwrap()) + .await + .unwrap(); + tokio::fs::write(&abs, bytes).await.unwrap(); + let entry = entry_for(UUID, &rel, bytes); + write_ledger(tmp.path(), &[("pkg:npm/left-pad@1.3.0", entry.clone())]).await; + (tmp, entry) + } + + async fn write_ledger(root: &Path, entries: &[(&str, VendorEntry)]) { + let mut state = VendorState::new(); + for (k, e) in entries { + state.entries.insert(k.to_string(), e.clone()); + } + save_state(root, &state).await.unwrap(); + } + + async fn reuse(root: &Path, rec: &PatchRecord) -> Result { + reusable_committed_artifact(root, "npm", rec, Some(&rel_for(&rec.uuid))).await + } + + #[tokio::test] + async fn verified_artifact_is_reused_with_its_exact_bytes() { + let bytes = good_tgz(); + let (tmp, entry) = project(&bytes).await; + let art = reuse(tmp.path(), &record(UUID)).await.unwrap(); + assert_eq!(art.bytes, bytes); + assert_eq!(art.rel_path, rel_for(UUID)); + assert_eq!(art.entry, entry); + assert_eq!( + art.members.get("index.js").map(Vec::as_slice), + Some(PATCHED) + ); + assert!(art.members.contains_key("package.json")); + } + + #[tokio::test] + async fn empty_record_is_never_reused() { + let (tmp, _) = project(&good_tgz()).await; + let mut rec = record(UUID); + rec.files.clear(); + assert_eq!( + reuse(tmp.path(), &rec).await.unwrap_err(), + ReuseMiss::NoFiles + ); + } + + #[tokio::test] + async fn unreadable_or_missing_ledger_misses() { + let (tmp, _) = project(&good_tgz()).await; + tokio::fs::write(tmp.path().join(".socket/vendor/state.json"), b"{not json") + .await + .unwrap(); + assert_eq!( + reuse(tmp.path(), &record(UUID)).await.unwrap_err(), + ReuseMiss::NoLedger + ); + tokio::fs::remove_file(tmp.path().join(".socket/vendor/state.json")) + .await + .unwrap(); + assert_eq!( + reuse(tmp.path(), &record(UUID)).await.unwrap_err(), + ReuseMiss::NoEntry + ); + } + + #[tokio::test] + async fn entry_must_match_ecosystem_uuid_path_and_carry_a_sha() { + let bytes = good_tgz(); + let (tmp, entry) = project(&bytes).await; + // Wrong ecosystem. + assert_eq!( + reusable_committed_artifact(tmp.path(), "pypi", &record(UUID), None) + .await + .unwrap_err(), + ReuseMiss::NoEntry + ); + // Different expected path. + assert_eq!( + reusable_committed_artifact( + tmp.path(), + "npm", + &record(UUID), + Some(".socket/vendor/npm/x/other.tgz") + ) + .await + .unwrap_err(), + ReuseMiss::NoEntry + ); + // A new record uuid finds no entry (acquisition runs; the old dir is + // never read). + assert_eq!( + reuse(tmp.path(), &record(OTHER_UUID)).await.unwrap_err(), + ReuseMiss::NoEntry + ); + // An entry without a sha anchors nothing. + let mut no_sha = entry.clone(); + no_sha.artifact.sha256.clear(); + write_ledger(tmp.path(), &[("pkg:npm/left-pad@1.3.0", no_sha)]).await; + assert_eq!( + reuse(tmp.path(), &record(UUID)).await.unwrap_err(), + ReuseMiss::NoEntry + ); + } + + #[tokio::test] + async fn qualified_twins_must_agree() { + let bytes = good_tgz(); + let (tmp, entry) = project(&bytes).await; + // Agreeing twins (same path, sha in a different case): reused. + let mut twin = entry.clone(); + twin.artifact.sha256 = twin.artifact.sha256.to_ascii_uppercase(); + write_ledger( + tmp.path(), + &[ + ("pkg:npm/left-pad@1.3.0", entry.clone()), + ("pkg:npm/left-pad@1.3.0?artifact_id=x", twin), + ], + ) + .await; + assert!(reuse(tmp.path(), &record(UUID)).await.is_ok()); + // Disagreeing twins: ambiguous. + let mut twin = entry.clone(); + twin.artifact.sha256 = "0".repeat(64); + write_ledger( + tmp.path(), + &[ + ("pkg:npm/left-pad@1.3.0", entry), + ("pkg:npm/left-pad@1.3.0?artifact_id=x", twin), + ], + ) + .await; + assert_eq!( + reuse(tmp.path(), &record(UUID)).await.unwrap_err(), + ReuseMiss::Ambiguous + ); + } + + #[tokio::test] + async fn unsafe_and_uuid_mismatched_paths_miss() { + let bytes = good_tgz(); + let (tmp, entry) = project(&bytes).await; + let mut escaping = entry.clone(); + escaping.artifact.path = format!(".socket/vendor/npm/{UUID}/../../../../etc/x.tgz"); + assert_eq!( + verify_committed_artifact(tmp.path(), &escaping, &record(UUID)) + .await + .unwrap_err(), + ReuseMiss::PathUnsafe + ); + // The same verified bytes committed under ANOTHER uuid's directory + // are never reused for this record. + let other_rel = rel_for(OTHER_UUID); + let abs = tmp.path().join(&other_rel); + tokio::fs::create_dir_all(abs.parent().unwrap()) + .await + .unwrap(); + tokio::fs::write(&abs, &bytes).await.unwrap(); + let wrong_dir = entry_for(UUID, &other_rel, &bytes); + assert_eq!( + verify_committed_artifact(tmp.path(), &wrong_dir, &record(UUID)) + .await + .unwrap_err(), + ReuseMiss::UuidMismatch + ); + } + + #[tokio::test] + async fn missing_artifact_misses() { + let (tmp, _) = project(&good_tgz()).await; + tokio::fs::remove_file(tmp.path().join(rel_for(UUID))) + .await + .unwrap(); + assert_eq!( + reuse(tmp.path(), &record(UUID)).await.unwrap_err(), + ReuseMiss::Missing + ); + } + + #[tokio::test] + async fn regzipped_artifact_fails_the_ledger_anchor() { + let bytes = good_tgz(); + let (tmp, _) = project(&bytes).await; + let alt = super::super::test_support::regzip(&bytes); + tokio::fs::write(tmp.path().join(rel_for(UUID)), &alt) + .await + .unwrap(); + assert_eq!( + reuse(tmp.path(), &record(UUID)).await.unwrap_err(), + ReuseMiss::Sha256Mismatch + ); + } + + /// An UNPATCHED member edited and re-gzipped, with the ledger still + /// recording the original sha: the anchor rejects it (today's rebuild + /// then heals it). + #[tokio::test] + async fn edited_unpatched_member_with_stale_ledger_sha_misses() { + let (tmp, _) = project(&good_tgz()).await; + let tampered = tgz(&[ + ("package/index.js", PATCHED), + ( + "package/package.json", + b"{\"name\":\"left-pad\",\"evil\":1}", + ), + ]); + tokio::fs::write(tmp.path().join(rel_for(UUID)), &tampered) + .await + .unwrap(); + assert_eq!( + reuse(tmp.path(), &record(UUID)).await.unwrap_err(), + ReuseMiss::Sha256Mismatch + ); + } + + #[tokio::test] + async fn size_mismatch_misses() { + let bytes = good_tgz(); + let (tmp, mut entry) = project(&bytes).await; + entry.artifact.size = Some(bytes.len() as u64 + 1); + assert_eq!( + verify_committed_artifact(tmp.path(), &entry, &record(UUID)) + .await + .unwrap_err(), + ReuseMiss::SizeMismatch + ); + // An absent size is not checked. + entry.artifact.size = None; + assert!(verify_committed_artifact(tmp.path(), &entry, &record(UUID)) + .await + .is_ok()); + } + + /// A patched member edited AND the ledger sha forged to match: the + /// afterHash check still rejects it. + #[tokio::test] + async fn forged_ledger_over_edited_patched_member_misses() { + let forged = tgz(&[("package/index.js", b"module.exports = 'evil';\n")]); + let (tmp, _) = project(&forged).await; + assert_eq!( + reuse(tmp.path(), &record(UUID)).await.unwrap_err(), + ReuseMiss::MemberMismatch("vendor_hash_mismatch".into()) + ); + let no_member = tgz(&[("package/other.js", PATCHED)]); + let (tmp, _) = project(&no_member).await; + assert_eq!( + reuse(tmp.path(), &record(UUID)).await.unwrap_err(), + ReuseMiss::MemberMismatch("file_not_found".into()) + ); + } + + #[tokio::test] + async fn undecodable_or_unknown_shape_misses() { + let (tmp, _) = project(b"not a tarball").await; + assert_eq!( + reuse(tmp.path(), &record(UUID)).await.unwrap_err(), + ReuseMiss::Unreadable + ); + // A dir-shaped / unknown extension is not this helper's business. + let tmp = tempfile::tempdir().unwrap(); + let rel = format!(".socket/vendor/npm/{UUID}/left-pad-1.3.0.zip"); + let abs = tmp.path().join(&rel); + tokio::fs::create_dir_all(abs.parent().unwrap()) + .await + .unwrap(); + tokio::fs::write(&abs, good_tgz()).await.unwrap(); + let entry = entry_for(UUID, &rel, &good_tgz()); + assert_eq!( + verify_committed_artifact(tmp.path(), &entry, &record(UUID)) + .await + .unwrap_err(), + ReuseMiss::Unreadable + ); + } + + #[tokio::test] + async fn wheel_members_verify_from_the_same_buffer() { + let mut zip = zip::ZipWriter::new(std::io::Cursor::new(Vec::new())); + zip.start_file::<_, ()>("index.js", Default::default()) + .unwrap(); + zip.write_all(PATCHED).unwrap(); + let whl = zip.finish().unwrap().into_inner(); + let tmp = tempfile::tempdir().unwrap(); + let rel = format!(".socket/vendor/pypi/{UUID}/six-1.0-py3-none-any.whl"); + let abs = tmp.path().join(&rel); + tokio::fs::create_dir_all(abs.parent().unwrap()) + .await + .unwrap(); + tokio::fs::write(&abs, &whl).await.unwrap(); + let mut entry = entry_for(UUID, &rel, &whl); + entry.ecosystem = "pypi".into(); + let art = verify_committed_artifact(tmp.path(), &entry, &record(UUID)) + .await + .unwrap(); + assert_eq!(art.bytes, whl); + } + + #[cfg(unix)] + #[tokio::test] + async fn symlinked_artifact_or_uuid_dir_misses() { + let bytes = good_tgz(); + // Symlinked artifact file. + let (tmp, _) = project(&bytes).await; + let abs = tmp.path().join(rel_for(UUID)); + let outside = tmp.path().join("outside.tgz"); + tokio::fs::write(&outside, &bytes).await.unwrap(); + tokio::fs::remove_file(&abs).await.unwrap(); + std::os::unix::fs::symlink(&outside, &abs).unwrap(); + assert_eq!( + reuse(tmp.path(), &record(UUID)).await.unwrap_err(), + ReuseMiss::NotRegular + ); + + // Symlinked uuid directory. + let (tmp, _) = project(&bytes).await; + let uuid_dir = tmp.path().join(format!(".socket/vendor/npm/{UUID}")); + let real = tmp.path().join("real-dir"); + tokio::fs::rename(&uuid_dir, &real).await.unwrap(); + std::os::unix::fs::symlink(&real, &uuid_dir).unwrap(); + assert_eq!( + reuse(tmp.path(), &record(UUID)).await.unwrap_err(), + ReuseMiss::NotRegular + ); + } + + #[cfg(unix)] + #[tokio::test] + async fn fifo_artifact_misses_promptly() { + let (tmp, _) = project(&good_tgz()).await; + let abs = tmp.path().join(rel_for(UUID)); + tokio::fs::remove_file(&abs).await.unwrap(); + let c = std::ffi::CString::new(abs.to_str().unwrap()).unwrap(); + assert_eq!(unsafe { libc::mkfifo(c.as_ptr(), 0o644) }, 0); + let res = tokio::time::timeout( + std::time::Duration::from_secs(10), + reuse(tmp.path(), &record(UUID)), + ) + .await + .expect("a FIFO must never wedge the reuse probe"); + assert_eq!(res.unwrap_err(), ReuseMiss::NotRegular); + } + + #[tokio::test] + async fn oversized_artifact_misses_before_reading() { + let (tmp, _) = project(&good_tgz()).await; + let abs = tmp.path().join(rel_for(UUID)); + let f = std::fs::OpenOptions::new().write(true).open(&abs).unwrap(); + // Sparse: no real disk use. + f.set_len(MAX_HEALTH_HASH_BYTES + 1).unwrap(); + drop(f); + assert_eq!( + reuse(tmp.path(), &record(UUID)).await.unwrap_err(), + ReuseMiss::TooLarge + ); + } + + // ── Canonical-archive gate: an archive whose decoded members differ + // from what an installer extracts is never reused, even with the + // ledger sha recomputed (a plain sha256 anyone committing + // state.json can forge). + + const EVIL: &[u8] = b"module.exports = 'UNPATCHED';\n"; + const PKG_JSON: &[u8] = b"{\"name\":\"left-pad\",\"version\":\"1.3.0\"}"; + + fn tgz_typed(members: &[(&str, &[u8], tar::EntryType)]) -> Vec { + let mut builder = tar::Builder::new(flate2::write::GzEncoder::new( + Vec::new(), + flate2::Compression::default(), + )); + for (name, data, ty) in members { + let mut header = tar::Header::new_gnu(); + header.set_size(data.len() as u64); + header.set_mode(0o644); + header.set_entry_type(*ty); + header.set_cksum(); + builder.append_data(&mut header, name, *data).unwrap(); + } + builder.into_inner().unwrap().finish().unwrap() + } + + async fn forged_reuse(bytes: &[u8]) -> Result { + let (tmp, _) = project(bytes).await; + reuse(tmp.path(), &record(UUID)).await + } + + /// npm/pnpm/yarn/bun strip the FIRST segment whatever it is: a second + /// top-level dir shadows the verified `package/index.js` at install. + #[tokio::test] + async fn second_top_level_dir_is_not_canonical() { + use tar::EntryType::Regular; + let bytes = tgz_typed(&[ + ("package/index.js", PATCHED, Regular), + ("package/package.json", PKG_JSON, Regular), + ("zzz/index.js", EVIL, Regular), + ]); + assert_eq!( + forged_reuse(&bytes).await.unwrap_err(), + ReuseMiss::NonCanonical + ); + } + + /// node-tar extracts a type-'7' (contiguous) entry as a file; the + /// lenient decoder skips it. + #[tokio::test] + async fn contiguous_twin_entry_is_not_canonical() { + use tar::EntryType::{Continuous, Regular}; + let bytes = tgz_typed(&[ + ("package/index.js", PATCHED, Regular), + ("package/package.json", PKG_JSON, Regular), + ("package/index.js", EVIL, Continuous), + ]); + assert_eq!( + forged_reuse(&bytes).await.unwrap_err(), + ReuseMiss::NonCanonical + ); + // A symlink / hardlink entry is refused the same way. + let bytes = tgz_typed(&[ + ("package/index.js", PATCHED, Regular), + ("package/evil.js", b"", tar::EntryType::Symlink), + ]); + assert_eq!( + forged_reuse(&bytes).await.unwrap_err(), + ReuseMiss::NonCanonical + ); + } + + /// A case-insensitive filesystem keeps one of `index.js` / `INDEX.js` + /// (the later write) — and an exact duplicate lets the LAST one win in + /// both the decoder and the installer, but whichever wins is not ours + /// to guess. + #[tokio::test] + async fn case_folded_or_exact_duplicate_is_not_canonical() { + use tar::EntryType::Regular; + let bytes = tgz_typed(&[ + ("package/index.js", PATCHED, Regular), + ("package/INDEX.js", EVIL, Regular), + ]); + assert_eq!( + forged_reuse(&bytes).await.unwrap_err(), + ReuseMiss::NonCanonical + ); + let bytes = tgz_typed(&[ + ("package/index.js", EVIL, Regular), + ("package/index.js", PATCHED, Regular), + ]); + assert_eq!( + forged_reuse(&bytes).await.unwrap_err(), + ReuseMiss::NonCanonical + ); + // `./` aliases collapse onto the same key. + let bytes = tgz_typed(&[ + ("package/index.js", PATCHED, Regular), + ("package/./Index.js", EVIL, Regular), + ]); + assert_eq!( + forged_reuse(&bytes).await.unwrap_err(), + ReuseMiss::NonCanonical + ); + } + + /// Honest shapes stay reusable: directory entries, a `package/` root + /// dir entry, and a pax/GNU long-name member. + #[tokio::test] + async fn canonical_archive_with_dirs_and_long_names_is_reused() { + let long = format!("package/{}/deep.js", "d".repeat(120)); + let mut builder = tar::Builder::new(flate2::write::GzEncoder::new( + Vec::new(), + flate2::Compression::default(), + )); + for dir in ["package/", "package/lib/"] { + let mut h = tar::Header::new_gnu(); + h.set_entry_type(tar::EntryType::Directory); + h.set_size(0); + h.set_mode(0o755); + h.set_cksum(); + builder.append_data(&mut h, dir, std::io::empty()).unwrap(); + } + for (name, data) in [ + ("package/index.js", PATCHED), + ("package/lib/a.js", b"a" as &[u8]), + (long.as_str(), b"deep"), + ] { + let mut h = tar::Header::new_gnu(); + h.set_size(data.len() as u64); + h.set_mode(0o644); + h.set_cksum(); + builder.append_data(&mut h, name, data).unwrap(); + } + let bytes = builder.into_inner().unwrap().finish().unwrap(); + let art = forged_reuse(&bytes).await.unwrap(); + assert!(art.members.contains_key("lib/a.js")); + assert!(art.members.keys().any(|k| k.ends_with("/deep.js"))); + } + + async fn wheel_reuse(whl: &[u8]) -> Result { + let tmp = tempfile::tempdir().unwrap(); + let rel = format!(".socket/vendor/pypi/{UUID}/six-1.0-py3-none-any.whl"); + let abs = tmp.path().join(&rel); + tokio::fs::create_dir_all(abs.parent().unwrap()) + .await + .unwrap(); + tokio::fs::write(&abs, whl).await.unwrap(); + let mut entry = entry_for(UUID, &rel, whl); + entry.ecosystem = "pypi".into(); + verify_committed_artifact(tmp.path(), &entry, &record(UUID)).await + } + + fn zip_of(members: &[(&str, &[u8])]) -> Vec { + let mut zip = zip::ZipWriter::new(std::io::Cursor::new(Vec::new())); + for (name, data) in members { + zip.start_file::<_, ()>(*name, Default::default()).unwrap(); + zip.write_all(data).unwrap(); + } + zip.finish().unwrap().into_inner() + } + + /// Wheel twin of the case-fold shape, plus an exact duplicate name + /// (made by renaming a same-length sibling in place: the name is not + /// covered by the CRC). + #[tokio::test] + async fn case_folded_or_exact_duplicate_wheel_member_is_not_reused() { + let whl = zip_of(&[("index.js", PATCHED), ("INDEX.js", EVIL)]); + assert_eq!( + wheel_reuse(&whl).await.unwrap_err(), + ReuseMiss::NonCanonical + ); + + let whl = zip_of(&[("index.js", PATCHED), ("indeX.js", EVIL)]); + let mut dup = whl.clone(); + let (from, to) = (b"indeX.js", b"index.js"); + let mut i = 0; + while i + from.len() <= dup.len() { + if &dup[i..i + from.len()] == from { + dup[i..i + from.len()].copy_from_slice(to); + } + i += 1; + } + assert!( + wheel_reuse(&dup).await.is_err(), + "an exact duplicate must never be reused" + ); + // The canonical wheel is still reused. + assert!(wheel_reuse(&zip_of(&[("index.js", PATCHED)])).await.is_ok()); + } +} diff --git a/crates/socket-patch-core/src/vendor/service_fetch.rs b/crates/socket-patch-core/src/vendor/service_fetch.rs index 81e53d6e..9aa9c585 100644 --- a/crates/socket-patch-core/src/vendor/service_fetch.rs +++ b/crates/socket-patch-core/src/vendor/service_fetch.rs @@ -310,12 +310,15 @@ mod tests { fn cfg_for(server: &MockServer) -> VendorServiceConfig { VendorServiceConfig { source: VendorSource::Service, - client: Some(ApiClient::new(ApiClientOptions { - api_url: server.uri(), - api_token: Some("sktsec_placeholder_value_for_tests_api".into()), - use_public_proxy: false, - org_slug: Some("acme".into()), - })), + client: Some( + ApiClient::new(ApiClientOptions { + api_url: server.uri(), + api_token: Some("sktsec_placeholder_value_for_tests_api".into()), + use_public_proxy: false, + org_slug: Some("acme".into()), + }) + .with_vendor_retry(crate::api::client::VendorRetryPolicy::none()), + ), use_public_proxy: false, vendor_url: None, patch_server_url: None, diff --git a/crates/socket-patch-core/src/vendor/test_support.rs b/crates/socket-patch-core/src/vendor/test_support.rs new file mode 100644 index 00000000..8130ff28 --- /dev/null +++ b/crates/socket-patch-core/src/vendor/test_support.rs @@ -0,0 +1,366 @@ +//! Shared tooling for the source-flip / outage tests of the +//! archive-shaped backends. + +use std::io::{Read as _, Write as _}; +use std::path::Path; + +use base64::Engine as _; +use sha2::{Digest, Sha512}; + +use crate::api::client::{ApiClient, ApiClientOptions, VendorRetryPolicy}; +use crate::patch::apply::ApplyResult; +use crate::vendor::state::{load_state, save_state, VendorEntry}; +use crate::vendor::{VendorOutcome, VendorServiceConfig, VendorSource, VendorWarning}; + +pub(crate) const PACKAGE_PATH: &str = "/v0/orgs/acme/patches/package"; + +/// Re-gzip at `Compression::fast`: identical members, different bytes +/// (and so a different sha512) — the stand-in for the service's +/// prebuilt encoding of the same patched package. +pub(crate) fn regzip(tgz: &[u8]) -> Vec { + regzip_at(tgz, flate2::Compression::fast()) +} + +/// [`regzip`] at an explicit level (`best` re-encodes a `fast` archive). +pub(crate) fn regzip_at(tgz: &[u8], level: flate2::Compression) -> Vec { + let mut raw = Vec::new(); + flate2::read::GzDecoder::new(tgz) + .read_to_end(&mut raw) + .unwrap(); + let mut enc = flate2::write::GzEncoder::new(Vec::new(), level); + enc.write_all(&raw).unwrap(); + let out = enc.finish().unwrap(); + assert_ne!(out, tgz, "regzip must change the bytes"); + out +} + +pub(crate) fn sri(bytes: &[u8]) -> String { + format!( + "sha512-{}", + base64::engine::general_purpose::STANDARD.encode(Sha512::digest(bytes)) + ) +} + +/// A service config against `server_uri` (org `acme`, authenticated), with +/// retries disabled (tests about retry build their own client). +pub(crate) fn service_cfg( + server_uri: &str, + source: VendorSource, + offline: bool, +) -> VendorServiceConfig { + VendorServiceConfig { + source, + client: Some( + ApiClient::new(ApiClientOptions { + api_url: server_uri.to_string(), + api_token: Some("sktsec_placeholder_value_for_tests_api".into()), + use_public_proxy: false, + org_slug: Some("acme".into()), + }) + .with_vendor_retry(VendorRetryPolicy::none()), + ), + use_public_proxy: false, + vendor_url: None, + patch_server_url: None, + offline, + } +} + +/// Mount a granted package reference for `uuid` serving `bytes` (file +/// name `leaf`) with a matching SRI. +pub(crate) async fn mount_granted( + server: &wiremock::MockServer, + uuid: &str, + leaf: &str, + bytes: &[u8], +) { + use wiremock::matchers::{method, path}; + use wiremock::{Mock, ResponseTemplate}; + let serve_path = format!("/serve/{uuid}/{leaf}"); + let url = format!("{}{serve_path}", server.uri()); + Mock::given(method("POST")) + .and(path(PACKAGE_PATH)) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "results": { uuid: { + "status": "granted", + "url": url, + "artifacts": [{ "kind": "tarball", "url": url, + "integrity": { "sha512": sri(bytes) } }] + }} + }))) + .mount(server) + .await; + Mock::given(method("GET")) + .and(path(serve_path)) + .respond_with(ResponseTemplate::new(200).set_body_bytes(bytes.to_vec())) + .mount(server) + .await; +} + +/// Mount a 503 on the package-reference POST (the outage). +pub(crate) async fn mount_503(server: &wiremock::MockServer) { + use wiremock::matchers::{method, path}; + use wiremock::{Mock, ResponseTemplate}; + Mock::given(method("POST")) + .and(path(PACKAGE_PATH)) + .respond_with(ResponseTemplate::new(503).set_body_string("upstream unavailable")) + .mount(server) + .await; +} + +pub(crate) async fn request_count(server: &wiremock::MockServer) -> usize { + server.received_requests().await.unwrap_or_default().len() +} + +pub(crate) fn expect_done( + outcome: VendorOutcome, +) -> (ApplyResult, Option, Vec) { + match outcome { + VendorOutcome::Done { + result, + entry, + warnings, + } => (result, entry, warnings), + VendorOutcome::Refused { code, detail } => { + panic!("expected Done, got Refused {code}: {detail}") + } + } +} + +/// Persist `entry` under `key` the way the CLI does after a successful +/// vendor (carry-forward included), so the next run sees the ledger. +pub(crate) async fn persist(root: &Path, key: &str, mut entry: VendorEntry) { + let mut state = load_state(root).await.unwrap(); + if let Some(prev) = state.entries.get(key) { + crate::vendor::carry_forward_wiring(prev, &mut entry); + } + state.entries.insert(key.to_string(), entry); + save_state(root, &state).await.unwrap(); +} + +pub(crate) fn has_warning(warnings: &[VendorWarning], code: &str) -> bool { + warnings.iter().any(|w| w.code == code) +} + +/// A vendoring fixture the shared npm-family flip suite +/// ([`npm_flip_suite`]) can drive. +pub(crate) trait FlipFixture { + fn flip_root(&self) -> &Path; + /// The ledger key the CLI would persist the entry under (the purl). + fn flip_key(&self) -> String; + fn flip_uuid(&self) -> String; + /// Project-relative path of the canonical committed artifact. + fn flip_artifact_rel(&self) -> String; + /// Every other committed file the wiring may touch (the lock plus any + /// mirrors), project-relative. + fn flip_files(&self) -> Vec; +} + +/// `(path, bytes)` for the artifact and every wired file (`None` = absent). +pub(crate) type Snapshot = Vec<(String, Option>)>; + +pub(crate) async fn snapshot(fx: &impl FlipFixture) -> Snapshot { + let mut out = Vec::new(); + for rel in std::iter::once(fx.flip_artifact_rel()).chain(fx.flip_files()) { + let bytes = tokio::fs::read(fx.flip_root().join(&rel)).await.ok(); + out.push((rel, bytes)); + } + out +} + +pub(crate) fn artifact_leaf(fx: &impl FlipFixture) -> String { + fx.flip_artifact_rel() + .rsplit('/') + .next() + .unwrap() + .to_string() +} + +/// The npm-family source-flip suite: one module of tests per flavor, each +/// asserting that a re-run after a service ↔ local source flip (or under +/// an outage) is a true no-op — lock and artifact byte-identical, entry +/// `None` (the CLI's `already_vendored`), zero service requests, and no +/// outage warning. +/// +/// `$suite`: the generated module's name; `$fixture`: `async fn() -> $fx`; `$run`: +/// `async fn(&$fx, Option<&VendorServiceConfig>) -> VendorOutcome`; +/// `$fx: FlipFixture`. All three resolve in the invoking module. +macro_rules! npm_flip_suite { + ($suite:ident, $fx:ident, $fixture:ident, $run:ident) => { + mod $suite { + use super::{$fixture, $run}; + use crate::vendor::test_support::{self as ts, FlipFixture as _}; + use crate::vendor::VendorSource; + + use super::$fx as Fx; + + /// The deterministic local build's bytes (from a throwaway copy). + async fn local_bytes() -> Vec { + let probe = $fixture().await; + let (r, e, _) = ts::expect_done($run(&probe, None).await); + assert!(r.success && e.is_some(), "{:?}", r.error); + tokio::fs::read(probe.flip_root().join(probe.flip_artifact_rel())) + .await + .unwrap() + } + + async fn mount(fx: &Fx, server: &wiremock::MockServer, serve: Option<&[u8]>) { + server.reset().await; + match serve { + Some(bytes) => { + ts::mount_granted(server, &fx.flip_uuid(), &ts::artifact_leaf(fx), bytes) + .await + } + None => ts::mount_503(server).await, + } + } + + /// Run 1: wires, and persists the entry the way the CLI would. + async fn first_run( + fx: &Fx, + server: &wiremock::MockServer, + source: VendorSource, + serve: Option<&[u8]>, + ) -> ts::Snapshot { + mount(fx, server, serve).await; + let cfg = ts::service_cfg(&server.uri(), source, false); + let (r, e, w) = ts::expect_done($run(fx, Some(&cfg)).await); + assert!(r.success, "run 1: {:?}", r.error); + assert_eq!( + ts::has_warning(&w, "vendor_prebuilt_downloaded"), + serve.is_some(), + "run 1 source: {w:?}" + ); + ts::persist(fx.flip_root(), &fx.flip_key(), e.expect("run 1 wires")).await; + ts::snapshot(fx).await + } + + /// Run 2 must be a true no-op with no network. + async fn assert_noop_rerun( + fx: &Fx, + server: &wiremock::MockServer, + source: VendorSource, + offline: bool, + serve: Option<&[u8]>, + before: &ts::Snapshot, + ) { + mount(fx, server, serve).await; + let cfg = ts::service_cfg(&server.uri(), source, offline); + let (r, e, w) = ts::expect_done($run(fx, Some(&cfg)).await); + assert!(r.success, "run 2: {:?}", r.error); + assert!(e.is_none(), "run 2 must be already_vendored (entry None)"); + assert!(w.is_empty(), "run 2 carries no advisory: {w:?}"); + assert_eq!( + &ts::snapshot(fx).await, + before, + "lock + artifact byte-identical" + ); + assert_eq!(ts::request_count(server).await, 0, "run 2 makes no request"); + } + + #[tokio::test] + async fn service_then_outage_rerun_is_in_sync() { + let alt = ts::regzip(&local_bytes().await); + let server = wiremock::MockServer::start().await; + let fx = $fixture().await; + let before = first_run(&fx, &server, VendorSource::Auto, Some(&alt)).await; + assert_eq!( + before[0].1.as_deref(), + Some(alt.as_slice()), + "run 1 used the service bytes" + ); + assert_noop_rerun(&fx, &server, VendorSource::Auto, false, None, &before).await; + } + + #[tokio::test] + async fn outage_then_service_rerun_is_in_sync() { + let local = local_bytes().await; + let alt = ts::regzip(&local); + let server = wiremock::MockServer::start().await; + let fx = $fixture().await; + let before = first_run(&fx, &server, VendorSource::Auto, None).await; + assert_eq!( + before[0].1.as_deref(), + Some(local.as_slice()), + "run 1 built locally" + ); + assert_noop_rerun(&fx, &server, VendorSource::Auto, false, Some(&alt), &before) + .await; + } + + #[tokio::test] + async fn outage_then_outage_rerun_is_in_sync_and_quiet() { + let server = wiremock::MockServer::start().await; + let fx = $fixture().await; + let before = first_run(&fx, &server, VendorSource::Auto, None).await; + assert_noop_rerun(&fx, &server, VendorSource::Auto, false, None, &before).await; + } + + #[tokio::test] + async fn service_mode_in_sync_rerun_survives_outage_and_offline() { + let alt = ts::regzip(&local_bytes().await); + let server = wiremock::MockServer::start().await; + let fx = $fixture().await; + let before = first_run(&fx, &server, VendorSource::Service, Some(&alt)).await; + assert_noop_rerun(&fx, &server, VendorSource::Service, false, None, &before).await; + assert_noop_rerun(&fx, &server, VendorSource::Service, true, None, &before).await; + } + + /// F3: a healthy service is not re-contacted, and the committed + /// artifact is not even rewritten (mtime unchanged). + #[tokio::test] + async fn service_then_service_rerun_skips_the_service() { + let alt = ts::regzip(&local_bytes().await); + let server = wiremock::MockServer::start().await; + let fx = $fixture().await; + let before = first_run(&fx, &server, VendorSource::Auto, Some(&alt)).await; + let art = fx.flip_root().join(fx.flip_artifact_rel()); + let mtime = std::fs::metadata(&art).unwrap().modified().unwrap(); + assert_noop_rerun(&fx, &server, VendorSource::Auto, false, Some(&alt), &before) + .await; + assert_eq!(std::fs::metadata(&art).unwrap().modified().unwrap(), mtime); + } + + /// F5: `--vendor-source build` reuses (it never contacts the + /// service, and reuse contacts nothing). + #[tokio::test] + async fn build_rerun_after_service_keeps_the_service_bytes() { + let alt = ts::regzip(&local_bytes().await); + let server = wiremock::MockServer::start().await; + let fx = $fixture().await; + let before = first_run(&fx, &server, VendorSource::Auto, Some(&alt)).await; + assert_noop_rerun( + &fx, + &server, + VendorSource::Build, + false, + Some(&alt), + &before, + ) + .await; + } + } + }; +} +pub(crate) use npm_flip_suite; + +/// Every regular file under `root` (relative path → bytes), for the +/// whole-tree byte-identity oracle of the directory-shaped backends. +pub(crate) fn tree_snapshot(root: &Path) -> std::collections::BTreeMap> { + fn walk(base: &Path, dir: &Path, out: &mut std::collections::BTreeMap>) { + for e in std::fs::read_dir(dir).unwrap() { + let p = e.unwrap().path(); + if p.is_dir() { + walk(base, &p, out); + } else { + out.insert( + p.strip_prefix(base).unwrap().to_string_lossy().into_owned(), + std::fs::read(&p).unwrap(), + ); + } + } + } + let mut out = std::collections::BTreeMap::new(); + walk(root, root, &mut out); + out +} diff --git a/crates/socket-patch-core/src/vendor/verify.rs b/crates/socket-patch-core/src/vendor/verify.rs index 4909433c..87aba918 100644 --- a/crates/socket-patch-core/src/vendor/verify.rs +++ b/crates/socket-patch-core/src/vendor/verify.rs @@ -44,7 +44,7 @@ const MAX_WHEEL_ENTRIES: usize = 10_000; /// `..`/absolute/NUL components, and (c) carry the uuid of the patch record /// being attested — a poisoned path must neither read outside the project /// tree nor launder one patch's artifact into another's attestation. -fn checked_artifact_path( +pub(crate) fn checked_artifact_path( project_root: &Path, entry: &VendorEntry, record: &PatchRecord, @@ -147,18 +147,65 @@ fn read_wheel_to_map(whl: &Path) -> Result>, String> { // reader streams from it. let (file, _metadata) = crate::utils::fs::open_regular_file_sync(whl) .map_err(|_| "vendor_artifact_unreadable".to_string())?; + read_zip_to_map(file, false) +} + +/// [`read_wheel_to_map`] over in-memory zip bytes — the same entry and +/// decompressed-size caps — for callers that hash and decode the SAME +/// buffer (a committed wheel read exactly once). +#[cfg(test)] +pub(crate) fn read_zip_bytes_to_map(bytes: &[u8]) -> Result>, String> { + read_zip_to_map(std::io::Cursor::new(bytes), false) +} + +/// [`read_zip_bytes_to_map`] that also refuses a wheel an installer could +/// extract differently from how it decodes: a symlink entry, a name outside +/// [`canonical_member_key`]'s shape (non-ASCII, backslash, `..`, absolute), +/// or two entries that collide exactly or after ASCII case-folding (a +/// case-insensitive filesystem keeps only one of `six.py` / `SIX.py`, and +/// which one is the installer's choice, not ours). Fails with +/// `vendor_artifact_non_canonical`. For re-verifying a COMMITTED wheel +/// before it is reused unchanged. +/// +/// [`canonical_member_key`]: crate::patch::package::canonical_member_key +pub(crate) fn read_zip_bytes_to_map_strict( + bytes: &[u8], +) -> Result>, String> { + read_zip_to_map(std::io::Cursor::new(bytes), true) +} + +/// The shared bounded zip decoder behind [`read_wheel_to_map`] and +/// [`read_zip_bytes_to_map`] (plus the canonical-shape gate when `strict`). +fn read_zip_to_map( + reader: R, + strict: bool, +) -> Result>, String> { let mut zip = - zip::ZipArchive::new(file).map_err(|_| "vendor_artifact_unreadable".to_string())?; + zip::ZipArchive::new(reader).map_err(|_| "vendor_artifact_unreadable".to_string())?; if zip.len() > MAX_WHEEL_ENTRIES { return Err("vendor_artifact_unreadable".to_string()); } let mut out = HashMap::new(); + let mut seen: std::collections::HashSet = std::collections::HashSet::new(); let mut declared: u64 = 0; let mut actual: u64 = 0; for i in 0..zip.len() { let mut entry = zip .by_index(i) .map_err(|_| "vendor_artifact_unreadable".to_string())?; + if strict { + let non_canonical = || "vendor_artifact_non_canonical".to_string(); + if entry.is_symlink() || entry.name().starts_with('/') { + return Err(non_canonical()); + } + let key = crate::patch::package::canonical_member_key(entry.name()) + .ok_or_else(non_canonical)?; + // Directories share the key space: a dir `six.py/` next to a + // file `SIX.py` collides on disk too. + if !seen.insert(key) { + return Err(non_canonical()); + } + } if !entry.is_file() { continue; } @@ -193,7 +240,7 @@ fn read_wheel_to_map(whl: &Path) -> Result>, String> { /// Hard cap on whole-artifact bytes hashed by the health check — committed /// artifacts are small (a package tarball/wheel); a tampered multi-GiB file /// must not stall `repair`. -const MAX_HEALTH_HASH_BYTES: u64 = 512 * 1024 * 1024; +pub(crate) const MAX_HEALTH_HASH_BYTES: u64 = 512 * 1024 * 1024; /// Hard cap on inventoried files, mirroring the zip reader's entry cap — a /// committed artifact dir is one package; a tampered dir must not stall an @@ -416,7 +463,7 @@ pub async fn file_sha256_hex(path: &Path) -> Option { Some(hex::encode(hasher.finalize())) } -fn verify_member_map( +pub(crate) fn verify_member_map( members: &HashMap>, record: &PatchRecord, ) -> Result<(), String> { @@ -512,6 +559,25 @@ mod tests { zip.finish().unwrap(); } + /// The in-memory zip reader decodes exactly what the path reader does. + #[test] + fn zip_bytes_reader_matches_path_reader() { + let dir = tempfile::tempdir().unwrap(); + let whl = dir.path().join("x-1.0-py3-none-any.whl"); + write_whl(&whl, "x/__init__.py", PATCHED); + let bytes = std::fs::read(&whl).unwrap(); + let from_bytes = read_zip_bytes_to_map(&bytes).unwrap(); + assert_eq!(from_bytes, read_wheel_to_map(&whl).unwrap()); + assert_eq!( + from_bytes.get("x/__init__.py").map(Vec::as_slice), + Some(PATCHED) + ); + assert_eq!( + read_zip_bytes_to_map(b"not a zip").unwrap_err(), + "vendor_artifact_unreadable" + ); + } + #[tokio::test] async fn binary_workspace_health_attests_every_installable_tarball() { use base64::Engine as _; diff --git a/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs b/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs index 4becca1a..6995e06a 100644 --- a/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs +++ b/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs @@ -31,7 +31,7 @@ use std::path::Path; use serde_json::Value; -use sha2::{Digest, Sha512}; +use sha2::{Digest, Sha256, Sha512}; use crate::constants::SOCKET_DIR; use crate::manifest::schema::PatchRecord; @@ -310,19 +310,36 @@ pub async fn vendor_yarn_berry( let dest = project_root.join(&rel_tgz); // ── 8. Berry identity facts of the packed tarball ───────────────────── - let tgz_bytes = match tokio::fs::read(&dest).await { - Ok(b) => b, - Err(e) => { - return done_failure_unstage( - purl, - format!("cannot re-read the packed tarball: {e}"), - project_root, - &uuid_dir_rel, - uuid_dir_preexisted, - ) - .await - } + // A reuse hands over the exact bytes it verified; a fresh pack is + // re-read and must still be the bytes the pack hashed (the lock's + // checksum and `hash=` are derived from these, so a file swapped after + // verification must fail, never be pinned). + let tgz_bytes = match staged.verified_bytes { + Some(bytes) => bytes, + None => match tokio::fs::read(&dest).await { + Ok(b) => b, + Err(e) => { + return done_failure_unstage( + purl, + format!("cannot re-read the packed tarball: {e}"), + project_root, + &uuid_dir_rel, + uuid_dir_preexisted, + ) + .await + } + }, }; + if hex::encode(Sha256::digest(&tgz_bytes)) != packed.sha256_hex { + return done_failure_unstage( + purl, + format!("the packed tarball {rel_tgz} changed on disk after it was verified"), + project_root, + &uuid_dir_rel, + uuid_dir_preexisted, + ) + .await; + } let tgz_sha512 = hex::encode(Sha512::digest(&tgz_bytes)); // `hash=` — the first 6 hex chars of sha512(tgz): the lock-committed // tamper guard on the tarball itself (spike B3, flips on any byte edit). @@ -1195,6 +1212,55 @@ __metadata: } } + // ── source-flip / outage idempotence (vendor::test_support::npm_flip_suite) ── + + impl crate::vendor::test_support::FlipFixture for Fixture { + fn flip_root(&self) -> &Path { + self.root() + } + fn flip_key(&self) -> String { + "pkg:npm/left-pad@1.3.0".to_string() + } + fn flip_uuid(&self) -> String { + self.record.uuid.clone() + } + fn flip_artifact_rel(&self) -> String { + format!(".socket/vendor/npm/{UUID}/left-pad-1.3.0.tgz") + } + fn flip_files(&self) -> Vec { + vec![ + PACKAGE_JSON.to_string(), + YARN_LOCK.to_string(), + YARNRC.to_string(), + ] + } + } + + async fn flip_run( + fx: &Fixture, + cfg: Option<&crate::vendor::VendorServiceConfig>, + ) -> VendorOutcome { + let blobs = fx.root().join(".socket/blobs"); + vendor_yarn_berry( + "pkg:npm/left-pad@1.3.0", + &fx.installed(), + fx.root(), + &fx.record, + &PatchSources::blobs_only(&blobs), + "2026-06-09T00:00:00Z", + false, + false, + cfg, + ) + .await + } + + async fn flip_fixture() -> Fixture { + fixture().await + } + + crate::vendor::test_support::npm_flip_suite!(flip_suite, Fixture, flip_fixture, flip_run); + async fn fixture_with(pkg: &str, lock: &str) -> Fixture { let tmp = tempfile::tempdir().unwrap(); let root = tmp.path(); diff --git a/crates/socket-patch-core/src/vendor/yarn_classic_lock.rs b/crates/socket-patch-core/src/vendor/yarn_classic_lock.rs index 9cecff08..84c9e31d 100644 --- a/crates/socket-patch-core/src/vendor/yarn_classic_lock.rs +++ b/crates/socket-patch-core/src/vendor/yarn_classic_lock.rs @@ -233,8 +233,10 @@ pub async fn vendor_yarn_classic( if wiring.is_empty() { // Every block already points at this uuid with the packed hashes: - // in sync. Touch nothing (the tarball re-pack above was - // byte-identical by determinism) and synthesize AlreadyPatched. + // in sync. `#sha1` and `integrity` were derived from the reused + // committed tarball (or, when reuse missed, from a fresh acquisition + // that reproduced them); touch nothing and synthesize + // AlreadyPatched. return VendorOutcome::Done { result: already_patched_result(purl, &dest, &record.files), entry: None, @@ -1053,6 +1055,51 @@ left-pad@^1.3.0, left-pad@~1.3.0: } } + // ── source-flip / outage idempotence (vendor::test_support::npm_flip_suite) ── + + impl crate::vendor::test_support::FlipFixture for Fixture { + fn flip_root(&self) -> &Path { + self.root() + } + fn flip_key(&self) -> String { + "pkg:npm/left-pad@1.3.0".to_string() + } + fn flip_uuid(&self) -> String { + self.record.uuid.clone() + } + fn flip_artifact_rel(&self) -> String { + format!(".socket/vendor/npm/{UUID}/left-pad-1.3.0.tgz") + } + fn flip_files(&self) -> Vec { + vec![YARN_LOCK.to_string(), "package.json".to_string()] + } + } + + async fn flip_run( + fx: &Fixture, + cfg: Option<&crate::vendor::VendorServiceConfig>, + ) -> VendorOutcome { + let blobs = fx.root().join(".socket/blobs"); + vendor_yarn_classic( + "pkg:npm/left-pad@1.3.0", + &fx.installed(), + fx.root(), + &fx.record, + &PatchSources::blobs_only(&blobs), + "2026-06-09T00:00:00Z", + false, + false, + cfg, + ) + .await + } + + async fn flip_fixture() -> Fixture { + fixture_with_lock(Y2_BEFORE).await + } + + crate::vendor::test_support::npm_flip_suite!(flip_suite, Fixture, flip_fixture, flip_run); + /// Build a project tempdir: installed left-pad, patched blob, the given /// yarn.lock bytes, and the PatchRecord. async fn fixture_with_lock(lock_text: &str) -> Fixture { @@ -1302,6 +1349,66 @@ left-pad@^1.3.0: ); } + /// F10 + F12 twin: a relock back to the registry block, re-pinned + /// from the REUSED tarball (no request under the outage), recomputes + /// the dependency sub-maps from the reused bytes' patched package.json. + #[tokio::test] + async fn relock_with_pkg_json_patch_recomputes_submaps_from_reused_bytes() { + use crate::vendor::test_support as ts; + let lock = r#"# yarn lockfile v1 + +left-pad@^1.3.0: + version "1.3.0" + resolved "https://registry.yarnpkg.com/left-pad/-/left-pad-1.3.0.tgz#5b8a3a7765dfe001261dde915589e782f8c94d1e" + integrity sha512-XI5MPzVNApjAyhQzphX8BkmKsKUxD4LdyK24iZeQGinBN9yTQT3bFlCBy/aVx2HrNcqQGsdot8ghrjyrvMCoEA== + dependencies: + old-dep "^1.0.0" +"#; + let mut fx = fixture_with_lock(lock).await; + let before: &[u8] = br#"{"name":"left-pad","version":"1.3.0"}"#; + let after: &[u8] = + br#"{"name":"left-pad","version":"1.3.0","dependencies":{"wow":"^1.0.0"}}"#; + let after_hash = compute_git_sha256_from_bytes(after); + tokio::fs::write(fx.root().join(".socket/blobs").join(&after_hash), after) + .await + .unwrap(); + fx.record.files.insert( + "package/package.json".to_string(), + PatchFileInfo { + before_hash: compute_git_sha256_from_bytes(before), + after_hash, + }, + ); + let (r, e, _) = expect_done(flip_run(&fx, None).await); + assert!(r.success, "{:?}", r.error); + ts::persist(fx.root(), "pkg:npm/left-pad@1.3.0", e.unwrap()).await; + let lock1 = fx.lock_text().await; + let tgz1 = tokio::fs::read(fx.tgz_path()).await.unwrap(); + // The relock. + tokio::fs::write(fx.lock_path(), &fx.lock_bytes) + .await + .unwrap(); + let server = wiremock::MockServer::start().await; + ts::mount_503(&server).await; + let cfg = ts::service_cfg(&server.uri(), crate::vendor::VendorSource::Auto, false); + let (r, e, w) = expect_done(flip_run(&fx, Some(&cfg)).await); + assert!(r.success, "{:?}", r.error); + assert!(e.is_some(), "the relocked block is re-wired"); + assert!( + !ts::has_warning(&w, "vendor_prebuilt_unavailable"), + "reused: {w:?}" + ); + assert_eq!(ts::request_count(&server).await, 0); + let text = fx.lock_text().await; + assert!(!text.contains("old-dep"), "{text}"); + assert!( + text.contains(" dependencies:\n wow \"^1.0.0\"\n"), + "{text}" + ); + assert_eq!(text, lock1); + assert_eq!(tokio::fs::read(fx.tgz_path()).await.unwrap(), tgz1); + } + #[tokio::test] async fn rerun_is_in_sync_and_byte_stable() { let fx = fixture_with_lock(Y2_BEFORE).await; diff --git a/scripts/backtest-bun.py b/scripts/backtest-bun.py index 27254699..cd49456c 100644 --- a/scripts/backtest-bun.py +++ b/scripts/backtest-bun.py @@ -956,6 +956,18 @@ def install(binary, label, flags=(), cache=None): row['repeat'] = parse_envelope(repeat) checks['repeatStableLock'] = lock.read_bytes() == patched_lock checks['repeatClean'] = rerun_clean(code, row['repeat'], main_mode) + if main_mode != 'hosted': + # The same re-run during a vendoring-service outage (a + # closed port: every service call is a transport + # failure). The committed artifact is reused, so the + # lock stays byte-identical and the run is the same + # already_vendored no-op — whichever source built it. + outage_env = dict(env, SOCKET_VENDOR_URL='http://127.0.0.1:9') + code, outage = run(command, project, outage_env, case / 'repeat-outage.log', False) + exit_codes['repeatOutage'] = code + row['repeatOutage'] = parse_envelope(outage) + checks['repeatOutageStableLock'] = lock.read_bytes() == patched_lock + checks['repeatOutageClean'] = rerun_clean(code, row['repeatOutage'], main_mode) if shape == 'already-vendored-workspace' and main_mode != 'hosted': # A deleted committed artifact is rebuilt by `repair` # (locally, so its tarball digest may differ from the diff --git a/scripts/backtest-pdm.py b/scripts/backtest-pdm.py index 75494653..3fce34ce 100644 --- a/scripts/backtest-pdm.py +++ b/scripts/backtest-pdm.py @@ -1139,6 +1139,23 @@ def uninstall(log): rollback_note = {"exit": rb1.rc, "status": erb1.get("status"), "failed": failures[:3], "lockEqualsRelocked": (project / lockname).read_bytes() == relocked} if target_kept: check("rescanAfterRelockApplies", rs.ok() and marker in rescanned, info["rescanAfterRelock"]) + if mode == "vendored": + # The re-scan re-wires the COMMITTED wheel (no service + # call, no rebuild): the patched sha the first scan + # wired is the one wired again. The sha alone also + # holds whenever the source did not flip between the + # scans, so when the relock dropped the vendored + # reference (a re-wire, not an in-sync skip) the + # re-scan must also report `vendor_artifact_reused` — + # the proof the committed wheel, not a rebuild, was used. + sha_re = rb"sha256:([a-f0-9]{64})" + patched_shas = set(re.findall(sha_re, lock_after)) - set(re.findall(sha_re, pristine_lock)) + sha_kept = bool(patched_shas) and patched_shas <= set(re.findall(sha_re, rescanned)) + rewired = marker not in relocked + reuse_event = "vendor_artifact_reused" in info["rescanAfterRelock"]["codes"] + reused = sha_kept and (reuse_event or not rewired) + info["rescanAfterRelock"].update({"shaKept": sha_kept, "rewired": rewired, "reuseEvent": reuse_event, "reusesWheel": reused}) + check("rescanReusesWheel", reused, info["rescanAfterRelock"]) check("rollbackAfterRelockPristine", rb1.ok() and rollback_note["lockEqualsRelocked"], rollback_note) else: # The relock resolved urllib3 away from 1.26.18 (PDM < 2.0