Skip to content

Commit 03d9237

Browse files
Fix vendored Pipenv sibling requirements.txt (#612) (#1309)
* WIP: Fix vendored Pipenv sibling requirements.txt (#612) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * WIP: Fix vendored Pipenv sibling requirements.txt (#612) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Wire Pipenv's exported requirements in vendor A Pipenv project often installs through a `pipenv requirements > requirements.txt` export (Docker, plain pip). Vendored mode wired only Pipfile.lock, so that path kept installing the unpatched release, and the hosted -> vendored takeover turned the export's hosted pin back into a plain PyPI pin. Re-running vendor answered `already_vendored`, so `vendor --check` and `vex` stayed red. Vendoring a Pipenv entry now also rewrites an exact registry pin of the package in requirements.txt (or an in-root `-r` include) to the same committed wheel, the way hosted mode rewrites both files. The ledger entry records both, every revert restores both, a superseding patch re-wires both, and a re-run over an already wired Pipfile.lock wires a newly made export. Pins the requirements wiring cannot rewrite stay named by `pypi_multiple_lockfiles`. Fixes #612 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Vendor Pipenv's exported requirements.txt too A Pipenv project often keeps a requirements.txt exported by `pipenv requirements` for Docker or plain pip installs. Vendored mode wired only Pipfile.lock, so `pip install -r requirements.txt` kept installing the unpatched release, and a hosted to vendored switch turned that file's patched pin back into a registry pin. A fresh Pipenv vendor now wires the root requirements.txt (and its -r includes) along with Pipfile.lock when it pins the package, and revert restores both. If the lock is wired but the requirements file can't be, the vendor fails as a whole and the lock is put back. A requirements file that can't be co-wired, or a project vendored before this change, is still named in the loud pypi_multiple_lockfiles warning. Refs #612 Assisted-by: Claude Code:claude-opus-5-5 * Test that co-wired Pipenv files are not contested `vendor --check` and `vex` read lockfile discovery. With Pipfile.lock and requirements.txt both pointing at the same vendored wheel, the regression test now checks that discovery finds the patch in both files and reports no contest between them. Refs #612 Assisted-by: Claude Code:claude-opus-5-5 * Pin the both-files takeover contract (#612) The hosted -> vendored takeover test asserted the old warning that requirements.txt stays UNPATCHED; it now asserts that the export's pin is wired to the vendored wheel too. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Harden the in-sync Pipenv export wiring and its revert (#612) - An in-sync re-run whose committed wheel is missing now rebuilds it and then wires the pending exported requirements pin, instead of returning after the rebuild with the export still on the registry. - The in-sync sibling wiring pins only a committed wheel that verify_committed_artifact accepts (canonical path for this uuid, ledger sha256, afterHashes), and wire/rewire_requirements refuse a wheel path or digest that is not a single plain token (no CR/LF, whitespace or #). - Re-wiring after a re-export replaces the entry's requirements_line records for that file instead of appending duplicates, so a later superseding patch can still re-wire the export. - revert_pipenv_with_sibling reverts the export first and restores it if the Pipfile.lock revert then fails, so neither failure leaves the two files half reverted. - Moves wire_requirements' doc comment back onto it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent c60845d commit 03d9237

6 files changed

Lines changed: 727 additions & 34 deletions

File tree

‎crates/socket-patch-cli/CLI_CONTRACT.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1389,7 +1389,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified
13891389
| `redirect_unconfirmed` | `redirect.patches[]` `unpinned` row | hosted `scan` / `get` (v5.0, additive): the patch was granted but no lockfile entry pinning it could be rewritten. The status and exit code are unchanged for now, pending the open hosted exit-policy decision (#704); `--silent` hides the human line. |
13901390
| `lockfile_unreadable` / `lockfile_unparseable` / `patched_ref_invalid` / `patched_ref_unattributable` | run-level `warnings[]` | vex (every form): lockfile-discovery diagnostics — see "Manifest-less VEX (lockfile discovery)". Never flip the exit on their own. |
13911391
| `vex_npm_shrinkwrap_only` | run warning and `failed[].reason` | vex (every form, #899): the patch is wired only in a root `npm-shrinkwrap.json` with no `package-lock.json` twin, which npm >= 12 never reads; no statement until the twin exists. `list` / `rollback` / `remove` still manage the wiring. |
1392-
| `vendor_multiple_lockfiles` / `pypi_multiple_lockfiles` | `skipped` (warning) | vendor: a sibling lockfile of another package manager (for PyPI, also a root `requirements.txt` that pins the package beside the wired tool lock) will still install UNPATCHED bytes; names the wired winner + the ignored locks. |
1392+
| `vendor_multiple_lockfiles` / `pypi_multiple_lockfiles` | `skipped` (warning) | vendor: a sibling lockfile of another package manager (for PyPI, also a root `requirements.txt` that pins the package beside the wired tool lock) will still install UNPATCHED bytes; names the wired winner + the ignored locks. Beside `Pipfile.lock`, an exact registry pin of the package in `requirements.txt` or an in-root `-r` include (`pipenv requirements` output) is wired with the lock instead and never named here (#612): both files then refer to the committed wheel, the ledger entry records both, every revert restores both, and a re-run over a project whose `Pipfile.lock` is already wired (a vendor from before, or an export made since) wires the export too (`pypi_requirements_sibling_wired`). A pin the requirements wiring cannot rewrite (a range, extras, an include outside the root) stays a loser. |
13931393
| `vendor_npm_shrinkwrap_only` | `skipped` (warning) | vendor / scan / get `--mode vendored` (npm, #899): the package was wired into `npm-shrinkwrap.json`, the only npm lock (also on an in-sync re-run). npm >= 12 never reads the shrinkwrap and installs the unpatched registry bytes; rename the lock to `package-lock.json` (or commit a copy under that name) and re-run. |
13941394
| `vendor_workspace_member_skipped` | `skipped` (warning) | vendor / scan / get `--mode vendored` (npm, #688): a lock entry with the package's `name@version` is the project's own source (a workspace member or a `file:` directory), so it is left alone while the lock's registry copies are vendored; patch that source directly if it needs the fix. When it is the only instance, vendoring refuses instead. |
13951395
| `vendor_npm_allow_file` | `skipped` (warning, printed as `Warning (vendor_npm_allow_file)`) | vendor / scan / get `--mode vendored` (package-lock, #969): the effective npm `allow-file` setting (env > project `.npmrc` > user > global > builtin) is not `all`, so npm >= 11.14 refuses the vendored `file:` tarball (EALLOWFILE) on install. The setting is respected, never rewritten; the detail names where it comes from and the remedy (`allow-file=all`). `vendor --check` fails the entry for the same reason. |

‎crates/socket-patch-cli/tests/mode_migration_pypi.rs‎

Lines changed: 123 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2022,12 +2022,11 @@ async fn platform_wheel_takeover_is_refused_before_revert() {
20222022

20232023
/// #612: a Pipenv project with a `requirements.txt` exported beside its
20242024
/// lock (`pipenv requirements`). Hosted mode pins both; the hosted →
2025-
/// vendored takeover (`vendor` over the hosted project) wires only the
2026-
/// governing `Pipfile.lock` and restores the requirements pin to upstream,
2027-
/// so the run must name `requirements.txt` among the install sources left
2028-
/// UNPATCHED instead of passing in silence.
2025+
/// vendored takeover (`vendor` over the hosted project) must leave both
2026+
/// patched too: `Pipfile.lock` and the export's pin both refer to the
2027+
/// vendored wheel, and no install source is left UNPATCHED.
20292028
#[tokio::test]
2030-
async fn pipenv_hosted_to_vendored_names_the_unpatched_requirements() {
2029+
async fn pipenv_hosted_to_vendored_keeps_the_requirements_patched() {
20312030
let (_tmp, root) = project();
20322031
stage_pipenv(&root);
20332032
std::fs::write(
@@ -2062,13 +2061,127 @@ async fn pipenv_hosted_to_vendored_names_the_unpatched_requirements() {
20622061
lock.contains(&format!(".socket/vendor/pypi/{UUID}/")),
20632062
"Pipfile.lock is wired to the vendored wheel:\n{lock}\n{env:#}"
20642063
);
2065-
let rendered = env.to_string();
2064+
let reqs = std::fs::read_to_string(root.join("requirements.txt")).unwrap();
20662065
assert!(
2067-
rendered.contains("\"pypi_multiple_lockfiles\"")
2068-
&& rendered.contains("wiring `Pipfile.lock`")
2069-
&& rendered.contains("requirements.txt will still install the UNPATCHED"),
2070-
"the takeover names requirements.txt as an unpatched install source: {env:#}"
2066+
reqs.contains(&format!(".socket/vendor/pypi/{UUID}/")) && !reqs.contains(&hosted_url),
2067+
"requirements.txt is wired to the vendored wheel too:\n{reqs}\n{env:#}"
20712068
);
2069+
assert!(
2070+
!env.to_string().contains("UNPATCHED"),
2071+
"no install source is left unpatched: {env:#}"
2072+
);
2073+
}
2074+
2075+
/// The `pipenv requirements > requirements.txt` export of [`stage_pipenv`]'s
2076+
/// lock: the file Docker / plain-pip installs read.
2077+
const PIPENV_EXPORT: &str = "-i https://pypi.org/simple\nsix==1.16.0 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2'\n";
2078+
2079+
/// #612: vendored mode in a Pipenv project with an exported
2080+
/// requirements.txt must keep that install path patched too, the way
2081+
/// hosted mode rewrites both files: `vendor` wires Pipfile.lock AND the
2082+
/// export's pin, `vendor --check` is green, and the revert restores both.
2083+
/// The same holds when the export is added after a first vendor (a re-run
2084+
/// wires it instead of answering `already_vendored`), and for the hosted
2085+
/// → vendored takeover, which used to turn the export's hosted pin back
2086+
/// into a plain PyPI pin.
2087+
#[tokio::test]
2088+
async fn pipenv_vendor_wires_the_exported_requirements_too() {
2089+
let wheel_dir = format!(".socket/vendor/pypi/{UUID}/");
2090+
let assert_both_vendored = |root: &Path, label: &str| {
2091+
for f in ["Pipfile.lock", "requirements.txt"] {
2092+
let text = std::fs::read_to_string(root.join(f)).unwrap();
2093+
assert!(
2094+
text.contains(&wheel_dir),
2095+
"{label}: {f} is vendored:\n{text}"
2096+
);
2097+
}
2098+
let (code, env) = run_cli(root, &["vendor", "--check"], &[]);
2099+
assert_eq!(code, 0, "{label}: vendor --check is green: {env:#}");
2100+
};
2101+
let assert_reverts = |root: &Path, pipfile_lock: &str, label: &str| {
2102+
let (code, env) = run_cli(root, &["vendor", "--revert"], &[]);
2103+
assert_eq!(code, 0, "{label}: revert: {env:#}");
2104+
assert_eq!(
2105+
std::fs::read_to_string(root.join("requirements.txt")).unwrap(),
2106+
PIPENV_EXPORT,
2107+
"{label}: the export is restored byte for byte"
2108+
);
2109+
let lock = |t: &str| serde_json::from_str::<Value>(t).unwrap();
2110+
assert_eq!(
2111+
lock(&std::fs::read_to_string(root.join("Pipfile.lock")).unwrap()),
2112+
lock(pipfile_lock),
2113+
"{label}: Pipfile.lock is restored"
2114+
);
2115+
assert!(
2116+
!root.join(&wheel_dir).exists(),
2117+
"{label}: the wheel is reclaimed"
2118+
);
2119+
};
2120+
2121+
// Fresh vendor.
2122+
let (_tmp, root) = project();
2123+
stage_pipenv(&root);
2124+
let pristine = std::fs::read_to_string(root.join("Pipfile.lock")).unwrap();
2125+
std::fs::write(root.join("requirements.txt"), PIPENV_EXPORT).unwrap();
2126+
stage_manifest(&root);
2127+
let (code, env) = run_cli(&root, &["vendor"], &[]);
2128+
assert_eq!(code, 0, "vendor: {env:#}");
2129+
assert!(
2130+
!env.to_string().contains("UNPATCHED"),
2131+
"no install path is left unpatched: {env:#}"
2132+
);
2133+
assert_both_vendored(&root, "fresh");
2134+
assert_reverts(&root, &pristine, "fresh");
2135+
2136+
// The export reached through an in-root `-r` include is wired too.
2137+
let (_tmp, root) = project();
2138+
let files = stage_pipenv(&root);
2139+
std::fs::write(root.join("requirements.txt"), "-r req/base.txt\n").unwrap();
2140+
std::fs::create_dir_all(root.join("req")).unwrap();
2141+
std::fs::write(root.join("req/base.txt"), PIPENV_EXPORT).unwrap();
2142+
vendor_project(&root, files);
2143+
let base = std::fs::read_to_string(root.join("req/base.txt")).unwrap();
2144+
assert!(
2145+
base.contains(&wheel_dir),
2146+
"the included export is vendored:\n{base}"
2147+
);
2148+
let (code, env) = run_cli(&root, &["vendor", "--check"], &[]);
2149+
assert_eq!(code, 0, "include: vendor --check is green: {env:#}");
2150+
let (code, env) = run_cli(&root, &["vendor", "--revert"], &[]);
2151+
assert_eq!(code, 0, "include: revert: {env:#}");
2152+
assert_eq!(
2153+
std::fs::read_to_string(root.join("req/base.txt")).unwrap(),
2154+
PIPENV_EXPORT
2155+
);
2156+
2157+
// The export appears after a first vendor: the re-run wires it.
2158+
let (_tmp, root) = project();
2159+
let files = stage_pipenv(&root);
2160+
vendor_project(&root, files);
2161+
std::fs::write(root.join("requirements.txt"), PIPENV_EXPORT).unwrap();
2162+
let (code, env) = run_cli(&root, &["vendor"], &[]);
2163+
assert_eq!(code, 0, "re-run: {env:#}");
2164+
assert_both_vendored(&root, "re-run");
2165+
assert_reverts(&root, &pristine, "re-run");
2166+
2167+
// Hosted → vendored takeover keeps both files patched.
2168+
let server = MockServer::start().await;
2169+
let hosted_url = mount_hosted_api(&server, true).await;
2170+
let uri = server.uri();
2171+
let (_tmp, root) = project();
2172+
stage_pipenv(&root);
2173+
std::fs::write(root.join("requirements.txt"), PIPENV_EXPORT).unwrap();
2174+
let (code, env) = hosted_scan(&root, &server);
2175+
assert_eq!(code, 0, "hosted scan: {env:#}");
2176+
for f in ["Pipfile.lock", "requirements.txt"] {
2177+
let text = std::fs::read_to_string(root.join(f)).unwrap();
2178+
assert!(text.contains(&hosted_url), "hosted first: {f}:\n{text}");
2179+
}
2180+
stage_manifest(&root);
2181+
prebuilt_common::mount_project(&server, &root).await;
2182+
let (code, env) = run_cli(&root, &["vendor", "--patch-server-url", uri.as_str()], &[]);
2183+
assert_eq!(code, 0, "takeover: {env:#}");
2184+
assert_both_vendored(&root, "takeover");
20722185
}
20732186

20742187
const PDM_LOCK: &str = r#"# This file is @generated by PDM.

0 commit comments

Comments
 (0)