diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 69fb09df9..7e642078d 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -189,7 +189,7 @@ The hidden alias `--no-apply` on `get --save-only` is **part of the contract** `repair` keeps its `gc` visible alias. -**Python stale-install guard**: after a hosted redirect, `scan` / `get` use the Python crawler to inspect every matching installed package, including Poetry's out-of-tree virtualenvs and `--global-prefix`. A readable file that differs from the patch's `afterHash` emits `redirect_pypi_stale_install` in JSON `redirect.warnings[]` and human stderr. The probe changes no installed files, re-runs on idempotent scans, and falls back to persisted patch records when fresh record fetching fails. Missing/unreadable files alone do not prove staleness; lock-only checkouts stay quiet. Dry runs skip the probe. Same-run VEX excludes positively stale Python packages (qualifier-insensitive), even with `--vex-no-verify` or a healthy copy in another interpreter; if nothing remains to attest, the command exits 1 with `no_applicable_patches`. Reinstall from the rewritten lock in the affected interpreter and verify with `socket-patch vex`. +**Python stale-install guard**: after a hosted redirect, `scan` / `get` use the Python crawler to inspect every matching installed package, including Poetry's out-of-tree virtualenvs, the project's Hatch environments (Hatch's data dir / `HATCH_DATA_DIR`, `[dirs.env] virtual`, explicit env `path`s) and `--global-prefix`. A readable file that differs from the patch's `afterHash` emits `redirect_pypi_stale_install` in JSON `redirect.warnings[]` and human stderr. The probe changes no installed files, re-runs on idempotent scans, and falls back to persisted patch records when fresh record fetching fails. Missing/unreadable files alone do not prove staleness; lock-only checkouts stay quiet. Dry runs skip the probe. Same-run VEX excludes positively stale Python packages (qualifier-insensitive), even with `--vex-no-verify` or a healthy copy in another interpreter; if nothing remains to attest, the command exits 1 with `no_applicable_patches`. Reinstall from the rewritten lock in the affected interpreter and verify with `socket-patch vex`. A stale Hatch environment instead names `hatch env remove ` / `hatch env prune`: Hatch's pip installer (and uv before Hatch 1.16) keeps a same-version release, so only a recreated env picks up the patch. The same Hatch envs are what agent mode patches and `vex` judges for a Hatch project. ### socket.yml patch policy (v5.0) @@ -327,7 +327,7 @@ Contract details: * **JSON success surface**: `apply` adds a top-level `vex` object to its envelope; `scan` adds a top-level `vex` key to its result. Both carry `{ path, statements, format: "openvex-0.2.0" }`. * `apply`'s no-manifest early exit (the `noManifest` success no-op; v5.0: its human line is `No patch manifest found; nothing to apply.` — it names the missing `.socket/manifest.json`, not the folder, since `.socket/` may legitimately hold vendored state) and `vendor`'s (`No manifest found, nothing to vendor.` — a project with hosted pins ejects instead, v5.0) still generate the document from the lockfiles and the vendor ledger (manifest-less VEX: hosted / vendored checkouts carry no manifest). Nothing referenced anywhere keeps the calm exit 0 (a stale document at the path is removed; `--json` carries any discovery diagnostics in `warnings[]`); any other VEX failure fails the command with exit 1 — including a run whose only candidates are omitted `record_unavailable` (an `--offline` run over a lockfile-wired checkout with no local records), so an ambient `SOCKET_VEX` there fails the install. `--dry-run` skips generation on both, and so does `apply --check` — it stays read-only and offline-safe, leaving the output path untouched. `scan` has no such early exit: with no manifest and nothing wired anywhere its `--vex` fails with `manifest_not_found`. * **Stale-doc removal (v3.5)**: a run that ends in a VEX error removes a recognizably-OpenVEX file (JSON whose `@context` names openvex.dev) already sitting at the output path — a pipeline reusing one path can never ship yesterday's attestation for a now-unpatched tree. Unrelated files at the path are never touched; a mid-write partial that no longer parses as JSON is left for downstream parsers to reject loudly. -* **Additive warnings (v3.5)**: `product_not_iri` (the `--product`/`--vex-product` override is neither a `pkg:` purl nor an absolute IRI; honored verbatim, warned) and `vendored_tree_out_of_sync` (a healthy vendored attestation stands on the committed artifact + lock wiring while the PRESENT installed tree hash-mismatches the patched bytes — run the package manager's install; the attestation itself is unchanged). Both ride stderr in human mode and `warnings[]` in the standalone `vex --json` envelope. Same channel for `product_multiple_manifests` (auto-detect found several project manifests and names the one it used), `vex_stale_doc_removed` (the stale-doc removal above happened), the manifest-less plan's advisories — `vex_wiring_conflict` (the lockfiles wire a package to different patches: which files, which uuids, how to fix it), `vex_record_superseded` (a recorded patch replaced by the lockfile-wired one), `vex_claim_unwired` (a ledger claim whose patch the lockfiles still mention, but not as wiring), `vex_record_offline` / `vex_record_not_found` / `vex_record_fetch_failed` (why a lockfile-wired patch has no record — the detail behind a `record_unavailable` skip) and `api_auth_fallback` (the authenticated API refused the credentials and the public proxy served free patches only; `get` / `scan`'s warning text) — and, standalone only, `org_looks_like_path` (`-o`/`--org` given a file-shaped value — `-O` is `--output`). The standalone error envelope carries `warnings[]` too. An embedded `--vex` that fails also folds each omitted patch into the host command's `warnings[]` as `vex_omitted` (`: ()` — standalone `vex` lists them as `skipped` events), and `--silent` lists them as `omitted: ()` lines under the error. A corrupt `.socket/vendor/state.json` is no longer degraded with a warning: every form of vex fails with `vendor_ledger_corrupt` (see the error-code table). A malformed pre-v5 `redirect-state.json` is, as of v5.0, only the `redirect_ledger_corrupt` warning (hosted mode keeps no ledger; the file is an optional migration record source). +* **Additive warnings (v3.5)**: `product_not_iri` (the `--product`/`--vex-product` override is neither a `pkg:` purl nor an absolute IRI; honored verbatim, warned) and `vendored_tree_out_of_sync` (a healthy vendored attestation stands on the committed artifact + lock wiring while the PRESENT installed tree hash-mismatches the patched bytes — run the package manager's install (for a project with Hatch environments the detail also names `hatch env remove `, since Hatch keeps an installed release); the attestation itself is unchanged). Both ride stderr in human mode and `warnings[]` in the standalone `vex --json` envelope. Same channel for `product_multiple_manifests` (auto-detect found several project manifests and names the one it used), `vex_stale_doc_removed` (the stale-doc removal above happened), the manifest-less plan's advisories — `vex_wiring_conflict` (the lockfiles wire a package to different patches: which files, which uuids, how to fix it), `vex_record_superseded` (a recorded patch replaced by the lockfile-wired one), `vex_claim_unwired` (a ledger claim whose patch the lockfiles still mention, but not as wiring), `vex_record_offline` / `vex_record_not_found` / `vex_record_fetch_failed` (why a lockfile-wired patch has no record — the detail behind a `record_unavailable` skip) and `api_auth_fallback` (the authenticated API refused the credentials and the public proxy served free patches only; `get` / `scan`'s warning text) — and, standalone only, `org_looks_like_path` (`-o`/`--org` given a file-shaped value — `-O` is `--output`). The standalone error envelope carries `warnings[]` too. An embedded `--vex` that fails also folds each omitted patch into the host command's `warnings[]` as `vex_omitted` (`: ()` — standalone `vex` lists them as `skipped` events), and `--silent` lists them as `omitted: ()` lines under the error. A corrupt `.socket/vendor/state.json` is no longer degraded with a warning: every form of vex fails with `vendor_ledger_corrupt` (see the error-code table). A malformed pre-v5 `redirect-state.json` is, as of v5.0, only the `redirect_ledger_corrupt` warning (hosted mode keeps no ledger; the file is an optional migration record source). ### VEX provenance markers (contract) @@ -1234,6 +1234,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `pypi_poetry_symlink_unsupported` / `pypi_pipenv_symlink_unsupported` / `pypi_requirements_symlink_unsupported` | `failed` | vendor (pypi, v5.0): a target file (`pyproject.toml` / `poetry.lock`, `Pipfile` / `Pipfile.lock`, or any planned `requirements*.txt`) is a symlink — refused before any write on wire AND on revert (the revert keeps the artifact, `kept_artifact`); the twins of the existing pdm/uv symlink refusals. | | `pypi_poetry_changed` / `pypi_pdm_changed` / `pypi_pipenv_changed` / `pypi_uv_changed` | `failed` | vendor (pypi, v5.0): the lock / project file changed between the read that planned the edit and the first write — refused before any write (worded like `pypi_lock_changed`: " changed during vendoring; re-run"). | | `pypi_pipenv_stale_install` | `skipped` (warning) | vendor (pipenv): the vendored twin of `redirect_pypi_stale_install` — the project's venv still holds the upstream release Pipenv will not reinstall over; the detail names the `pipenv run pip uninstall -y && pipenv sync` remedy. | +| `pypi_hatch_stale_install` | `skipped` (warning) | vendor (hatch): an existing Hatch environment of the project (found under Hatch's data dir, `dirs.env.virtual` or an explicit env `path`) still holds the upstream release; Hatch keeps it on the next `hatch run`, so the detail names the env and the `hatch env remove ` / `hatch env prune` remedy. | | `pypi_pipenv_installer_unknown` | `skipped` (warning) | vendor (pipenv): no `pipenv` answered on PATH; the vendored references assume Pipenv 2018 or later (7–11 cannot consume them — use hosted mode there); `SOCKET_PIPENV_MAJOR` pins the release. | | `vendor_lock_entry_relocked` | revert `warnings[]` | vendor `--revert` / rollback (pipenv): a relock regenerated the wired entry to a registry reference, or removed it; the record is retired (artifact removed, ledger entry dropped) instead of drift-kept. | | `pypi_{poetry,pdm,pipenv}_no_lockfile` | `failed` | vendor (pypi): a lock-less tool marker with no `requirements.txt` fallback — run ` lock`. | diff --git a/crates/socket-patch-cli/src/commands/scan/hosted/python.rs b/crates/socket-patch-cli/src/commands/scan/hosted/python.rs index 8a5445673..77e27a9ae 100644 --- a/crates/socket-patch-cli/src/commands/scan/hosted/python.rs +++ b/crates/socket-patch-cli/src/commands/scan/hosted/python.rs @@ -44,6 +44,13 @@ pub(super) async fn stale_install_warnings( // fallback to the global interpreters would judge an unrelated Python's // copy of the release (a tool venv on PATH) and warn falsely. // --global / --global-prefix keep their meaning. + // Hatch envs need their own remedy: a reinstall from the rewritten + // pyproject does nothing there (#335). + let hatch_envs = if common.is_global() { + Vec::new() + } else { + socket_patch_core::crawlers::hatch_env::hatch_environments(&common.cwd).await + }; let paths = if common.is_global() { crawler .get_site_packages_paths(&common.crawler_options()) @@ -105,7 +112,11 @@ pub(super) async fn stale_install_warnings( // (`install`, `install --deploy`, `sync` all keep the installed // bytes on every major), and `pipenv uninstall` rewrites the // Pipfile and re-locks the patch away — name the verified remedy. - let remedy = if pipenv_purls.contains(&purl) { + let hatch_env = + socket_patch_core::crawlers::hatch_env::environment_of(&hatch_envs, &site); + let remedy = if let Some(env) = hatch_env { + socket_patch_core::crawlers::hatch_env::stale_install_remedy(&env.name) + } else if pipenv_purls.contains(&purl) { let name = strip_purl_qualifiers(&purl) .strip_prefix("pkg:pypi/") .and_then(|rest| rest.split('@').next()) @@ -214,6 +225,55 @@ mod tests { assert!(out.warnings.is_empty()); } + /// #335: a Hatch env keeps the upstream release after the rewrite, and + /// the probe must find it (Hatch keeps envs out of `./.venv`) and name + /// Hatch's remedy, not "reinstall from the rewritten lock". + #[tokio::test] + async fn hatch_env_gets_the_stale_install_warning_with_hatch_remedy() { + let tmp = tempfile::tempdir().unwrap(); + let project = tmp.path().join("app"); + std::fs::create_dir_all(&project).unwrap(); + std::fs::write( + project.join("pyproject.toml"), + "[project]\nname = \"app\"\nversion = \"0.1.0\"\ndependencies = [\"six==1.16.0\"]\n\n[tool.hatch.envs.default]\npath = \"../hatch-envs/app\"\n", + ) + .unwrap(); + let env = tmp.path().join("hatch-envs").join("app"); + std::fs::create_dir_all(&env).unwrap(); + std::fs::write(env.join("pyvenv.cfg"), "home = /usr/bin\n").unwrap(); + let site = if cfg!(windows) { + env.join("Lib").join("site-packages") + } else { + env.join("lib").join("python3.12").join("site-packages") + }; + let dist = site.join("six-1.16.0.dist-info"); + std::fs::create_dir_all(&dist).unwrap(); + std::fs::write(dist.join("METADATA"), "Name: six\nVersion: 1.16.0\n").unwrap(); + std::fs::write(site.join("six.py"), b"upstream").unwrap(); + + let common = crate::args::GlobalArgs { + cwd: project.clone(), + ..Default::default() + }; + let purl = "pkg:pypi/six@1.16.0"; + let confirmed = vec![(purl.to_string(), "six-uuid".to_string())]; + let ledger = BTreeMap::from([("k".into(), record("six-uuid", "six.py", b"patched"))]); + let out = stale_install_warnings(&common, &confirmed, &BTreeSet::new(), &ledger).await; + assert_eq!(out.stale_purls, BTreeSet::from([purl.to_string()])); + assert_eq!(out.warnings.len(), 1); + let detail = out.warnings[0]["detail"].as_str().unwrap(); + assert!(detail.contains("hatch env remove default"), "{detail}"); + assert!( + !detail.contains("Reinstall from the rewritten lock"), + "{detail}" + ); + + // Patched in the env: nothing to warn about. + std::fs::write(site.join("six.py"), b"patched").unwrap(); + let out = stale_install_warnings(&common, &confirmed, &BTreeSet::new(), &ledger).await; + assert!(out.warnings.is_empty()); + } + /// A legacy `.egg-info` install (pip < 23.1 building an sdist without /// `wheel`) is a real copy pip keeps on `install -r`, so the hosted /// stale-install guard must judge it like a `.dist-info` one (#447). diff --git a/crates/socket-patch-cli/src/commands/vex.rs b/crates/socket-patch-cli/src/commands/vex.rs index 43eff8991..6d5615806 100644 --- a/crates/socket-patch-cli/src/commands/vex.rs +++ b/crates/socket-patch-cli/src/commands/vex.rs @@ -674,6 +674,26 @@ async fn generate_vex( // installed tree is present and running different bytes. Say so — a // build that bypasses the vendor wiring is unpatched until the next // package-manager install. + // A Hatch env is never resynced by an install: Hatch keeps a present + // release (#335), so name the remedy that recreates it. + let hatch_note = if outcome.vendored_out_of_sync.is_empty() || common.is_global() { + String::new() + } else { + match socket_patch_core::crawlers::hatch_env::hatch_environments(&common.cwd) + .await + .as_slice() + { + [] => String::new(), + envs => format!( + " A Hatch environment keeps an installed release on the next `hatch run`; \ + recreate it instead ({}).", + envs.iter() + .map(|env| format!("`hatch env remove {}`", env.name)) + .collect::>() + .join(", ") + ), + } + }; for purl in &outcome.vendored_out_of_sync { note_warning( warnings, @@ -683,7 +703,7 @@ async fn generate_vex( "{purl}: the installed tree does not match its vendored artifact; the \ attestation is based on the committed .socket/vendor artifact (the lockfile \ consumes it), but the live tree carries different bytes — re-run your \ - package manager's install to resync it." + package manager's install to resync it.{hatch_note}" ), ); } diff --git a/crates/socket-patch-cli/tests/e2e_vex_build/hatch.rs b/crates/socket-patch-cli/tests/e2e_vex_build/hatch.rs index 042001150..b00e56480 100644 --- a/crates/socket-patch-cli/tests/e2e_vex_build/hatch.rs +++ b/crates/socket-patch-cli/tests/e2e_vex_build/hatch.rs @@ -579,3 +579,163 @@ fn hatch_toml_environment_dependency_hosted() { fn hatch_toml_environment_dependency_vendored() { flow(Flavor::HatchTomlEnv, Mode::Vendored); } + +/// #335: on a project whose Hatch env ALREADY exists (any developer +/// checkout, a warm CI cache), Hatch keeps the upstream `six` on the next +/// `hatch run`: pip, and uv before Hatch 1.16, never reinstall a present +/// release, and Hatch then records the env as synced. Hatch keeps that env +/// out of tree, so socket-patch must find it on its own (no `VIRTUAL_ENV`): +/// the scan warns with the Hatch remedy and names the env, `vex` from the +/// project root refuses to attest the unpatched env, and the named remedy +/// (`hatch env remove default`) really yields the patched bytes, which vex +/// then attests. +fn existing_env_flow(mode: Mode) { + let Some(hatch) = hatch() else { return }; + let version = hatch.version.clone(); + let what = format!("hatch {version} existing-env {}", mode.label()); + let tmp = tempfile::tempdir().unwrap(); + let project = tmp.path().join("proj"); + std::fs::create_dir_all(project.join("app")).unwrap(); + std::fs::write(project.join("app/__init__.py"), "").unwrap(); + write_native(&project, Flavor::Project); + let case_envs = hatch.case_env(&tmp.path().join("case")); + + // The env as a developer has it: created from PyPI, upstream six. + let out = hatch.run(&project, &case_envs, &["env", "create"]); + if !out.status.success() { + skip_or_fail( + REQUIRED, + &format!("{what}: hatch env create: {}", out_text(&out)), + ); + return; + } + let found = hatch.run(&project, &case_envs, &["env", "find"]); + assert_ok(&found, &format!("{what}: hatch env find")); + let env_dir = PathBuf::from( + String::from_utf8_lossy(&found.stdout) + .trim() + .lines() + .last() + .unwrap() + .trim(), + ); + let (_, pristine, marked) = + six_oracle(&venv_bin(&env_dir, "python"), &project).expect("pristine six"); + assert!(!marked, "{what}: the env starts unpatched"); + let patched = [pristine.as_slice(), PATCH_SUFFIX].concat(); + let api = RealApi::start(mode.uuid(), &pristine, &patched); + + // Hatch's own state (data dir, config, HOME) but no VIRTUAL_ENV: the + // crawler has to locate Hatch's out-of-tree env itself. + let (code, env, stderr) = socket_scan(&project, &api, &scan_mode_args(mode), &case_envs); + assert_eq!(code, Some(0), "{what}: scan failed: {env}\n{stderr}"); + let (warnings, code_name) = match mode { + Mode::Hosted => ( + env["redirect"]["warnings"].clone(), + "redirect_pypi_stale_install", + ), + Mode::Vendored => (env["vendor"]["events"].clone(), "pypi_hatch_stale_install"), + }; + let details: Vec = warnings + .as_array() + .into_iter() + .flatten() + .filter(|w| w["code"] == code_name || w["errorCode"] == code_name) + .map(|w| w.to_string()) + .collect(); + assert_eq!( + details.len(), + 1, + "{what}: {code_name} expected: {env}\n{stderr}" + ); + assert!( + details[0].contains("hatch env remove default"), + "{what}: Hatch remedy: {}", + details[0] + ); + + // The next `hatch run` keeps the upstream bytes, as the warning says. + let out = hatch.run(&project, &case_envs, &["run", "python", "-c", "import six"]); + assert_ok(&out, &format!("{what}: hatch run")); + let (_, _, marked) = six_oracle(&venv_bin(&env_dir, "python"), &project).unwrap(); + assert!( + !marked, + "{what}: Hatch reinstalled; the premise no longer holds" + ); + + // vex from the project root sees the Hatch env: hosted attests + // nothing over it; vendored attests the committed artifact (its + // contract) but discloses the out-of-sync env with Hatch's remedy. + let patch_api = vex_e2e_common::PatchApi::start(vec![( + mode.uuid().to_string(), + view(mode.uuid(), &pristine, &patched), + )]); + let run = vex_e2e_common::VexRun { + patch_server_url: Some(api.uri()), + product: Some(PRODUCT.into()), + envs: case_envs + .iter() + .map(|(k, v)| (k.clone(), v.into())) + .collect(), + ..vex_e2e_common::VexRun::online(&patch_api) + }; + let out = vex_e2e_common::run_vex(&vex_e2e_common::binary(), &project, &run); + match mode { + Mode::Hosted => vex_e2e_common::assert_absent(out.doc.as_ref(), PURL), + Mode::Vendored => { + assert_attested(out.doc(), PURL, mode.uuid(), mode.marker(), VULNS); + let disclosed = out.envelope["warnings"] + .as_array() + .into_iter() + .flatten() + .any(|w| { + w["code"] == "vendored_tree_out_of_sync" + && w["detail"] + .as_str() + .is_some_and(|d| d.contains("hatch env remove default")) + }); + assert!( + disclosed, + "{what}: vendored_tree_out_of_sync with Hatch remedy: {out}" + ); + } + } + record( + "hatch", + &version, + &format!("existing-env/{}", mode.label()), + "stale-warned", + "pass", + ); + + // The remedy works: the recreated env holds the patch, and vex attests. + let out = hatch.run(&project, &case_envs, &["env", "remove", "default"]); + assert_ok(&out, &format!("{what}: hatch env remove")); + let out = hatch.run(&project, &case_envs, &["run", "python", "-c", "import six"]); + assert_ok(&out, &format!("{what}: hatch run after remove")); + let (_, bytes, marked) = six_oracle(&venv_bin(&env_dir, "python"), &project).unwrap(); + assert!(marked, "{what}: the recreated env must hold the patch"); + assert_eq!(git_sha256(&bytes), git_sha256(&patched), "{what}"); + let out = vex_e2e_common::run_vex(&vex_e2e_common::binary(), &project, &run); + assert_eq!(out.code, Some(0), "{what}: vex after the remedy: {out}"); + assert_attested(out.doc(), PURL, mode.uuid(), mode.marker(), VULNS); + record( + "hatch", + &version, + &format!("existing-env/{}", mode.label()), + "remedy-attested", + "pass", + ); +} + +#[test] +#[ignore = "real Hatch + PyPI; run with --ignored (CI: SOCKET_PATCH_HATCH_E2E_REQUIRED=1)"] +fn hatch_existing_env_hosted() { + existing_env_flow(Mode::Hosted); +} + +#[test] +#[ignore = "real Hatch + PyPI; run with --ignored (CI: SOCKET_PATCH_HATCH_E2E_REQUIRED=1)"] +fn hatch_existing_env_vendored() { + existing_env_flow(Mode::Vendored); +} diff --git a/crates/socket-patch-core/src/crawlers/hatch_env.rs b/crates/socket-patch-core/src/crawlers/hatch_env.rs new file mode 100644 index 000000000..d2bb431f6 --- /dev/null +++ b/crates/socket-patch-core/src/crawlers/hatch_env.rs @@ -0,0 +1,738 @@ +//! Where Hatch keeps a project's virtual environments. +//! +//! Hatch never uses `./.venv`: `hatch run` / `hatch shell` / `hatch test` +//! install into envs under its data directory, keyed by the project name and +//! a hash of the project root. Modelled on `hatch/env/virtual.py`, +//! `hatch/cli/application.py::get_env_directory` and +//! `hatch/utils/fs.py::Path.id`, unchanged in layout from Hatch 1.0 through +//! 1.18 except where noted: +//! +//! - env type directory: `[dirs.env] virtual` from Hatch's config file +//! (absolute, else relative to the project), else +//! `/env/virtual`; the data dir is `HATCH_DATA_DIR`, else +//! `dirs.data` from the config file, else the platform data dir; +//! - an env with an explicit `path` (`[tool.hatch.envs.] path`, +//! `hatch.toml`'s `[envs.] path`, or `HATCH_ENV_TYPE_VIRTUAL_PATH`) +//! lives exactly there; +//! - when the env type directory is `~/.virtualenvs` or inside the project, +//! envs sit flat in it (`/`); +//! - Hatch 1.0 - 1.2 keep every env at `/-/`; +//! - otherwise `///`, where the +//! project name is the PEP 503-normalized `[project] name` (or +//! `-unmanaged` without a `[project]` table), the id is the first 8 +//! chars of the URL-safe base64 sha256 of the project root (casefolded on +//! Windows, and on macOS from Hatch 1.10), and the `default` env is named +//! after the project. + +use std::path::{Path, PathBuf}; + +use toml_edit::{DocumentMut, Item}; + +/// The verified remedy for a Hatch env still holding the upstream release +/// after its dependency was rewired. Hatch only syncs a changed dependency +/// with `pip install` (keeps a same-version release that is already +/// installed) or, before Hatch 1.16, accepts the installed release as +/// satisfying the new reference outright; either way it then records the +/// env as synced and never retries. Only recreating the env helps. +pub fn stale_install_remedy(env_name: &str) -> String { + format!( + "Hatch does not reinstall a release that is already present in an existing \ + environment (its pip installer keeps the installed bytes, uv before Hatch 1.16 \ + skips the sync, and Hatch then records the environment as synced), so the \ + rewired dependency only reaches fresh environments. Recreate this one with \ + `hatch env remove {env_name}` (the next `hatch run` rebuilds it), or run `hatch \ + env prune` for every environment of the project; then `socket-patch vex` \ + re-verifies the installed files." + ) +} + +/// The Hatch env among `envs` whose tree holds `site`, however either is +/// spelled (an activated `VIRTUAL_ENV` may name the env through a symlink, +/// e.g. macOS's `/var` -> `/private/var`). +pub fn environment_of<'e>( + envs: &'e [HatchEnvironment], + site: &Path, +) -> Option<&'e HatchEnvironment> { + if let Some(env) = envs.iter().find(|env| site.starts_with(&env.prefix)) { + return Some(env); + } + let site = std::fs::canonicalize(site).ok()?; + envs.iter() + .find(|env| std::fs::canonicalize(&env.prefix).is_ok_and(|prefix| site.starts_with(prefix))) +} + +/// One Hatch virtual environment of a project, as Hatch names it. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct HatchEnvironment { + /// The env name `hatch env remove ` takes (`default` for the + /// env named after the project). + pub name: String, + /// The venv root (the directory holding `pyvenv.cfg`). + pub prefix: PathBuf, +} + +/// The existing Hatch virtual environments of the project at `cwd`. Empty +/// when `cwd` has neither `pyproject.toml` nor `hatch.toml`, or Hatch has +/// created none. +pub async fn hatch_environments(cwd: &Path) -> Vec { + let var = |name: &str| std::env::var(name).ok(); + hatch_environments_with(cwd, &var).await +} + +/// [`hatch_environments`] over an explicit environment. +pub(crate) async fn hatch_environments_with( + cwd: &Path, + var: &impl Fn(&str) -> Option, +) -> Vec { + let pyproject = read_toml(&cwd.join("pyproject.toml")).await; + let hatch_toml = read_toml(&cwd.join("hatch.toml")).await; + if pyproject.is_none() && hatch_toml.is_none() { + return Vec::new(); + } + let root = std::fs::canonicalize(cwd).unwrap_or_else(|_| cwd.to_path_buf()); + let config = read_config(var).await; + let configured = configured_envs(pyproject.as_ref(), hatch_toml.as_ref()); + let project_name = pyproject.as_ref().and_then(project_name); + + let mut found: Vec = Vec::new(); + let mut push = |name: String, prefix: PathBuf| { + if prefix.join("pyvenv.cfg").is_file() && !found.iter().any(|e| e.prefix == prefix) { + found.push(HatchEnvironment { name, prefix }); + } + }; + + // Explicit paths win for the env they name. + let override_path = var("HATCH_ENV_TYPE_VIRTUAL_PATH").filter(|v| !v.trim().is_empty()); + for env in &configured { + if let Some(path) = override_path.as_deref().or(env.path.as_deref()) { + push(env.name.clone(), resolve_env_path(&root, path)); + } + } + if override_path.is_some() && configured.iter().all(|e| e.name != "default") { + push( + "default".to_string(), + resolve_env_path(&root, override_path.as_deref().unwrap()), + ); + } + + let Some(env_dir) = virtual_env_dir(&root, config.as_ref(), var) else { + return found; + }; + let in_project = env_dir.starts_with(&root) + || std::fs::canonicalize(&env_dir).is_ok_and(|d| d.starts_with(&root)); + let shared_flat = home_dir(var).is_some_and(|h| same_path(&env_dir, &h.join(".virtualenvs"))); + + for id in project_ids(&root) { + // Hatch 1.0 - 1.2: `/-/`, whatever + // the directory (no flat or unmanaged layouts yet). + if let Some(name) = &project_name { + for (dir_name, prefix) in subdirs(&env_dir.join(format!("{name}-{id}"))) { + push(env_name_for(&dir_name, name), prefix); + } + } + let name = project_name + .clone() + .unwrap_or_else(|| format!("{id}-unmanaged")); + if in_project { + // A directory inside the project holds only this project's envs. + for (dir_name, prefix) in subdirs(&env_dir) { + push(env_name_for(&dir_name, &name), prefix); + } + break; + } + if shared_flat { + // `~/.virtualenvs` is shared, so only the names this project + // configures are taken from it. + push("default".to_string(), env_dir.join(&name)); + for env in configured.iter().filter(|e| e.name != "default") { + push(env.name.clone(), env_dir.join(&env.name)); + } + break; + } + let storage = env_dir.join(&name).join(&id); + for (dir_name, prefix) in subdirs(&storage) { + push(env_name_for(&dir_name, &name), prefix); + } + } + found +} + +/// `default` is stored under the project's name; every other env under its +/// own. +fn env_name_for(dir_name: &str, project_name: &str) -> String { + if dir_name == project_name { + "default".to_string() + } else { + dir_name.to_string() + } +} + +/// An env Hatch's project config declares, with its explicit `path`. +struct ConfiguredEnv { + name: String, + path: Option, +} + +/// `[tool.hatch.envs.*]` from pyproject, unless hatch.toml carries `envs` +/// (Hatch then reads hatch.toml's `[envs.*]` only). Envs whose `type` is +/// not `virtual` are left out. +fn configured_envs( + pyproject: Option<&DocumentMut>, + hatch_toml: Option<&DocumentMut>, +) -> Vec { + let table = hatch_toml + .and_then(|doc| doc.get("envs")) + .or_else(|| { + pyproject + .and_then(|doc| doc.get("tool")) + .and_then(|tool| tool.get("hatch")) + .and_then(|hatch| hatch.get("envs")) + }) + .and_then(Item::as_table_like); + let Some(table) = table else { + return Vec::new(); + }; + table + .iter() + .filter_map(|(name, item)| { + let env = item.as_table_like()?; + if env + .get("type") + .and_then(Item::as_str) + .is_some_and(|kind| kind != "virtual") + { + return None; + } + Some(ConfiguredEnv { + name: name.to_string(), + path: env + .get("path") + .and_then(Item::as_str) + .filter(|p| !p.is_empty()) + .map(str::to_string), + }) + }) + .collect() +} + +/// The PEP 503-normalized `[project] name` (hatchling's +/// `normalize_project_name`). `None` without a `[project]` table; a table +/// without a name is not a project Hatch can load, so `None` too. +fn project_name(pyproject: &DocumentMut) -> Option { + let name = pyproject.get("project")?.get("name")?.as_str()?; + let mut out = String::with_capacity(name.len()); + let mut in_run = false; + for c in name.chars() { + if matches!(c, '-' | '_' | '.') { + if !in_run { + out.push('-'); + } + in_run = true; + } else { + out.extend(c.to_lowercase()); + in_run = false; + } + } + Some(out) +} + +/// The project ids Hatch may have used for `root`: the hash of the path as +/// written, and of its casefolded form where some Hatch release casefolds +/// (Windows always; macOS from Hatch 1.10). +fn project_ids(root: &Path) -> Vec { + let text = strip_windows_verbatim_prefix(&root.to_string_lossy()); + let mut ids = Vec::new(); + if cfg!(windows) || cfg!(target_os = "macos") { + ids.push(path_id(&text.to_lowercase())); + } + let raw = path_id(&text); + if !ids.contains(&raw) { + ids.push(raw); + } + ids +} + +/// `hatch.utils.fs.Path.id` over an already normalized path string. +fn path_id(text: &str) -> String { + use base64::Engine as _; + use sha2::{Digest, Sha256}; + let digest = Sha256::digest(text.as_bytes()); + let encoded = base64::engine::general_purpose::URL_SAFE.encode(digest); + encoded[..8].to_string() +} + +/// `\\?\C:\x` -> `C:\x`, `\\?\UNC\srv\share` -> `\\srv\share` (Python's +/// `Path.cwd()` never carries the verbatim prefix `canonicalize` adds). +fn strip_windows_verbatim_prefix(text: &str) -> String { + if let Some(rest) = text.strip_prefix(r"\\?\UNC\") { + format!(r"\\{rest}") + } else if let Some(rest) = text.strip_prefix(r"\\?\") { + rest.to_string() + } else { + text.to_string() + } +} + +/// The env type directory for `virtual` (see the module docs). +fn virtual_env_dir( + root: &Path, + config: Option<&DocumentMut>, + var: &impl Fn(&str) -> Option, +) -> Option { + let dirs = config.and_then(|c| c.get("dirs")); + if let Some(configured) = dirs + .and_then(|d| d.get("env")) + .and_then(|e| e.get("virtual")) + .and_then(Item::as_str) + { + return Some(absolutize(root, &expand(configured, var))); + } + let data = var("HATCH_DATA_DIR") + .filter(|v| !v.trim().is_empty()) + .map(|v| expand(&v, var)) + .or_else(|| { + dirs.and_then(|d| d.get("data")) + .and_then(Item::as_str) + .map(|v| expand(v, var)) + }) + .map(PathBuf::from) + .or_else(|| default_data_dir(var))?; + Some(data.join("env").join("virtual")) +} + +/// Hatch's config file: `HATCH_CONFIG`, else `config.toml` in +/// `platformdirs.user_config_dir("hatch")`. +async fn read_config(var: &impl Fn(&str) -> Option) -> Option { + let path = var("HATCH_CONFIG") + .filter(|v| !v.trim().is_empty()) + .map(PathBuf::from) + .or_else(|| default_config_dir(var).map(|d| d.join("config.toml")))?; + read_toml(&path).await +} + +/// `platformdirs.user_data_dir("hatch", appauthor=False)`. +fn default_data_dir(var: &impl Fn(&str) -> Option) -> Option { + if cfg!(windows) { + local_app_data(var).map(|d| d.join("hatch")) + } else if cfg!(target_os = "macos") { + Some( + home_dir(var)? + .join("Library") + .join("Application Support") + .join("hatch"), + ) + } else { + Some( + xdg(var, "XDG_DATA_HOME") + .or_else(|| home_dir(var).map(|h| h.join(".local").join("share")))? + .join("hatch"), + ) + } +} + +/// `platformdirs.user_config_dir("hatch", appauthor=False)`. +fn default_config_dir(var: &impl Fn(&str) -> Option) -> Option { + if cfg!(windows) { + local_app_data(var).map(|d| d.join("hatch")) + } else if cfg!(target_os = "macos") { + Some( + home_dir(var)? + .join("Library") + .join("Application Support") + .join("hatch"), + ) + } else { + Some( + xdg(var, "XDG_CONFIG_HOME") + .or_else(|| home_dir(var).map(|h| h.join(".config")))? + .join("hatch"), + ) + } +} + +fn local_app_data(var: &impl Fn(&str) -> Option) -> Option { + var("LOCALAPPDATA") + .filter(|v| !v.trim().is_empty()) + .map(PathBuf::from) + .or_else(|| home_dir(var).map(|h| h.join("AppData").join("Local"))) +} + +/// An XDG base directory, honoured only when absolute (as platformdirs). +fn xdg(var: &impl Fn(&str) -> Option, name: &str) -> Option { + var(name) + .filter(|v| !v.trim().is_empty()) + .map(PathBuf::from) + .filter(|p| p.is_absolute()) +} + +fn home_dir(var: &impl Fn(&str) -> Option) -> Option { + var("HOME") + .or_else(|| var("USERPROFILE")) + .filter(|v| !v.trim().is_empty()) + .map(PathBuf::from) +} + +/// Hatch's `Path.expand`: `~` and `$VAR` / `${VAR}` (`%VAR%` on Windows +/// is left to the shell that set it and not modelled). +fn expand(text: &str, var: &impl Fn(&str) -> Option) -> String { + let text = match text.strip_prefix('~') { + Some(rest) if rest.is_empty() || rest.starts_with('/') || rest.starts_with('\\') => { + match home_dir(var) { + Some(home) => format!("{}{rest}", home.display()), + None => text.to_string(), + } + } + _ => text.to_string(), + }; + let mut out = String::with_capacity(text.len()); + let mut rest = text.as_str(); + while let Some(at) = rest.find('$') { + out.push_str(&rest[..at]); + let after = &rest[at + 1..]; + let (name, tail) = if let Some(braced) = after.strip_prefix('{') { + match braced.find('}') { + Some(end) => (&braced[..end], &braced[end + 1..]), + None => ("", after), + } + } else { + let end = after + .find(|c: char| !(c.is_ascii_alphanumeric() || c == '_')) + .unwrap_or(after.len()); + (&after[..end], &after[end..]) + }; + match (!name.is_empty()).then(|| var(name)).flatten() { + Some(value) => out.push_str(&value), + None => { + // Python's expandvars leaves an unknown variable as written. + out.push('$'); + out.push_str(&after[..after.len() - tail.len()]); + } + } + rest = tail; + } + out.push_str(rest); + out +} + +/// `path` against the project root, as Hatch joins it. +fn absolutize(root: &Path, path: &str) -> PathBuf { + let path = PathBuf::from(path); + if path.is_absolute() { + path + } else { + root.join(path) + } +} + +/// An env's explicit `path`, resolved like Hatch's `(root / path).resolve()`. +fn resolve_env_path(root: &Path, path: &str) -> PathBuf { + let joined = absolutize(root, path); + std::fs::canonicalize(&joined).unwrap_or(joined) +} + +fn same_path(a: &Path, b: &Path) -> bool { + match (std::fs::canonicalize(a), std::fs::canonicalize(b)) { + (Ok(a), Ok(b)) => a == b, + _ => a == b, + } +} + +/// The subdirectories of `dir` by name (none when it is missing). +fn subdirs(dir: &Path) -> Vec<(String, PathBuf)> { + let Ok(entries) = std::fs::read_dir(dir) else { + return Vec::new(); + }; + let mut out: Vec<(String, PathBuf)> = entries + .flatten() + .filter(|e| e.file_type().is_ok_and(|t| t.is_dir())) + .filter_map(|e| Some((e.file_name().into_string().ok()?, e.path()))) + .collect(); + out.sort(); + out +} + +/// A regular file's TOML (a FIFO or device planted at the path is never +/// opened, so discovery cannot block on it). +async fn read_toml(path: &Path) -> Option { + crate::utils::fs::read_regular_to_string(path) + .await + .ok()? + .parse() + .ok() +} + +#[cfg(test)] +mod tests { + use super::*; + use std::collections::HashMap; + + fn env_of(pairs: &[(&str, String)]) -> impl Fn(&str) -> Option { + let map: HashMap = pairs + .iter() + .map(|(k, v)| (k.to_string(), v.clone())) + .collect(); + move |name: &str| map.get(name).cloned() + } + + fn make_venv(prefix: &Path) { + std::fs::create_dir_all(prefix).unwrap(); + std::fs::write(prefix.join("pyvenv.cfg"), "home = /usr/bin\n").unwrap(); + } + + const PYPROJECT: &str = "[project]\nname = \"My_App.Core\"\nversion = \"0.1.0\"\n"; + + /// Hatch 1.18.1 computes `Path("/home/user/app").id` as `zrSR0Z2A` + /// (`urlsafe_b64encode(sha256(b"/home/user/app").digest())[:8]`). + #[test] + fn path_id_matches_hatch() { + assert_eq!(path_id("/home/user/app"), "zrSR0Z2A"); + } + + #[test] + fn project_name_is_pep503_normalized() { + let doc: DocumentMut = PYPROJECT.parse().unwrap(); + assert_eq!(project_name(&doc).as_deref(), Some("my-app-core")); + } + + #[tokio::test] + async fn finds_default_and_named_envs_under_the_data_dir() { + let tmp = tempfile::tempdir().unwrap(); + let project = tmp.path().join("app"); + std::fs::create_dir_all(&project).unwrap(); + std::fs::write(project.join("pyproject.toml"), PYPROJECT).unwrap(); + let data = tmp.path().join("data"); + let root = std::fs::canonicalize(&project).unwrap(); + let id = project_ids(&root).pop().unwrap(); + let storage = data.join("env/virtual/my-app-core").join(&id); + make_venv(&storage.join("my-app-core")); + make_venv(&storage.join("test")); + // Another project's env with the same name is not ours. + make_venv(&data.join("env/virtual/my-app-core/XXXXXXXX/my-app-core")); + + let var = env_of(&[ + ("HATCH_DATA_DIR", data.display().to_string()), + ("HOME", tmp.path().join("home").display().to_string()), + ]); + let found = hatch_environments_with(&project, &var).await; + assert_eq!( + found, + vec![ + HatchEnvironment { + name: "default".into(), + prefix: storage.join("my-app-core") + }, + HatchEnvironment { + name: "test".into(), + prefix: storage.join("test") + }, + ] + ); + } + + /// Hatch 1.0 - 1.2 (`hatch/env/virtual.py` there): `-`. + #[tokio::test] + async fn legacy_hatch_layout() { + let tmp = tempfile::tempdir().unwrap(); + let project = tmp.path().join("app"); + std::fs::create_dir_all(&project).unwrap(); + std::fs::write(project.join("pyproject.toml"), PYPROJECT).unwrap(); + let data = tmp.path().join("data"); + let root = std::fs::canonicalize(&project).unwrap(); + let id = project_ids(&root).pop().unwrap(); + let prefix = data + .join("env/virtual") + .join(format!("my-app-core-{id}")) + .join("my-app-core"); + make_venv(&prefix); + let var = env_of(&[("HATCH_DATA_DIR", data.display().to_string())]); + assert_eq!( + hatch_environments_with(&project, &var).await, + vec![HatchEnvironment { + name: "default".into(), + prefix + }] + ); + } + + #[tokio::test] + async fn default_data_dir_and_config_dirs_env_virtual() { + let tmp = tempfile::tempdir().unwrap(); + let home = tmp.path().join("home"); + let project = tmp.path().join("app"); + std::fs::create_dir_all(&project).unwrap(); + std::fs::write(project.join("pyproject.toml"), PYPROJECT).unwrap(); + let root = std::fs::canonicalize(&project).unwrap(); + let id = project_ids(&root).pop().unwrap(); + let var = env_of(&[ + ("HOME", home.display().to_string()), + ("LOCALAPPDATA", home.join("lad").display().to_string()), + ]); + let data = default_data_dir(&var).unwrap(); + let prefix = data + .join("env/virtual/my-app-core") + .join(&id) + .join("my-app-core"); + make_venv(&prefix); + let found = hatch_environments_with(&project, &var).await; + assert_eq!(found.len(), 1); + assert_eq!(found[0].prefix, prefix); + + // `[dirs.env] virtual` moves the env type directory. + let config_dir = default_config_dir(&var).unwrap(); + std::fs::create_dir_all(&config_dir).unwrap(); + let elsewhere = tmp.path().join("envs"); + std::fs::write( + config_dir.join("config.toml"), + format!("[dirs.env]\nvirtual = '{}'\n", elsewhere.display()), + ) + .unwrap(); + assert!(hatch_environments_with(&project, &var).await.is_empty()); + let moved = elsewhere.join("my-app-core").join(&id).join("lint"); + make_venv(&moved); + let found = hatch_environments_with(&project, &var).await; + assert_eq!( + found, + vec![HatchEnvironment { + name: "lint".into(), + prefix: moved + }] + ); + } + + #[tokio::test] + async fn in_project_env_dir_is_flat_and_explicit_paths_win() { + let tmp = tempfile::tempdir().unwrap(); + let project = tmp.path().join("app"); + std::fs::create_dir_all(&project).unwrap(); + std::fs::write( + project.join("pyproject.toml"), + format!("{PYPROJECT}\n[tool.hatch.envs.docs]\npath = \"envs/docs\"\n"), + ) + .unwrap(); + let config = tmp.path().join("hatch-config.toml"); + std::fs::write(&config, "[dirs.env]\nvirtual = \".hatch\"\n").unwrap(); + make_venv(&project.join(".hatch/my-app-core")); + make_venv(&project.join(".hatch/test")); + make_venv(&project.join("envs/docs")); + let var = env_of(&[("HATCH_CONFIG", config.display().to_string())]); + let found = hatch_environments_with(&project, &var).await; + let root = std::fs::canonicalize(&project).unwrap(); + let names: Vec<_> = found + .iter() + .map(|e| (e.name.as_str(), e.prefix.clone())) + .collect(); + assert_eq!( + names, + vec![ + ("docs", root.join("envs/docs")), + ("default", root.join(".hatch/my-app-core")), + ("test", root.join(".hatch/test")), + ] + ); + } + + #[tokio::test] + async fn unmanaged_project_and_no_project_files() { + let tmp = tempfile::tempdir().unwrap(); + let project = tmp.path().join("app"); + std::fs::create_dir_all(&project).unwrap(); + let data = tmp.path().join("data"); + let var = env_of(&[("HATCH_DATA_DIR", data.display().to_string())]); + assert!(hatch_environments_with(&project, &var).await.is_empty()); + + std::fs::write(project.join("hatch.toml"), "[envs.default]\n").unwrap(); + let root = std::fs::canonicalize(&project).unwrap(); + let id = project_ids(&root).pop().unwrap(); + let name = format!("{id}-unmanaged"); + let prefix = data.join("env/virtual").join(&name).join(&id).join(&name); + make_venv(&prefix); + let found = hatch_environments_with(&project, &var).await; + assert_eq!( + found, + vec![HatchEnvironment { + name: "default".into(), + prefix + }] + ); + } + + #[cfg(unix)] + #[tokio::test] + async fn fifo_configuration_never_blocks() { + for filename in ["pyproject.toml", "hatch.toml", "config.toml"] { + let tmp = tempfile::tempdir().unwrap(); + let project = tmp.path().join("app"); + std::fs::create_dir_all(&project).unwrap(); + if filename != "pyproject.toml" { + std::fs::write(project.join("pyproject.toml"), PYPROJECT).unwrap(); + } + let fifo = if filename == "config.toml" { + tmp.path().join(filename) + } else { + project.join(filename) + }; + assert!(std::process::Command::new("mkfifo") + .arg(&fifo) + .status() + .unwrap() + .success()); + let var = env_of(&[ + ("HATCH_CONFIG", fifo.display().to_string()), + ( + "HATCH_DATA_DIR", + tmp.path().join("data").display().to_string(), + ), + ]); + let result = tokio::time::timeout( + std::time::Duration::from_secs(2), + hatch_environments_with(&project, &var), + ) + .await; + if result.is_err() { + drop( + std::fs::OpenOptions::new() + .read(true) + .write(true) + .open(&fifo) + .unwrap(), + ); + } + assert!(result.unwrap().is_empty(), "{filename}"); + } + } + + /// macOS spells temp dirs `/var/...` and `/private/var/...`: an + /// activated env named through a symlink is still that Hatch env. + #[cfg(unix)] + #[test] + fn environment_of_sees_through_symlinked_spellings() { + let tmp = tempfile::tempdir().unwrap(); + let real = tmp.path().join("real"); + let site = real.join("env/lib/python3.12/site-packages"); + std::fs::create_dir_all(&site).unwrap(); + let link = tmp.path().join("link"); + std::os::unix::fs::symlink(&real, &link).unwrap(); + let envs = vec![HatchEnvironment { + name: "default".into(), + prefix: real.join("env"), + }]; + let via_link = link.join("env/lib/python3.12/site-packages"); + assert_eq!(environment_of(&envs, &via_link).unwrap().name, "default"); + assert_eq!(environment_of(&envs, &site).unwrap().name, "default"); + assert!(environment_of(&envs, tmp.path()).is_none()); + } + + #[test] + fn expand_matches_python() { + let var = env_of(&[("HOME", "/h".to_string()), ("X", "ex".to_string())]); + assert_eq!(expand("~/a/$X/${X}b/$NOPE/c", &var), "/h/a/ex/exb/$NOPE/c"); + assert_eq!(expand("~user/a", &var), "~user/a"); + } +} + +/// [`project_ids`] for tests elsewhere in the crate. +#[cfg(test)] +pub(crate) fn project_ids_for_tests(root: &Path) -> Vec { + project_ids(root) +} diff --git a/crates/socket-patch-core/src/crawlers/mod.rs b/crates/socket-patch-core/src/crawlers/mod.rs index b0c257f50..a983d2d2e 100644 --- a/crates/socket-patch-core/src/crawlers/mod.rs +++ b/crates/socket-patch-core/src/crawlers/mod.rs @@ -3,6 +3,7 @@ pub mod composer_crawler; pub mod deno_crawler; pub mod fuzzy_match; pub mod go_crawler; +pub mod hatch_env; mod listing; pub mod maven_crawler; #[cfg(test)] diff --git a/crates/socket-patch-core/src/crawlers/python_crawler.rs b/crates/socket-patch-core/src/crawlers/python_crawler.rs index dfdee4f52..8d00bf5ce 100644 --- a/crates/socket-patch-core/src/crawlers/python_crawler.rs +++ b/crates/socket-patch-core/src/crawlers/python_crawler.rs @@ -331,7 +331,9 @@ async fn find_site_packages_under( /// 3. Poetry's out-of-tree virtualenv(s), when Poetry itself would not use /// `./.venv` for the project (see [`find_poetry_virtualenv_site_packages`]) /// 4. `.venv` directory in `cwd` -/// 5. `venv` directory in `cwd` +/// 5. `venv` directory in `cwd` (a PDM project with neither: PEP 582) +/// 6. Hatch's out-of-tree envs for the project (see +/// [`super::hatch_env::hatch_environments`]), added to 4/5 pub async fn find_local_venv_site_packages(cwd: &Path) -> Vec { let var = |name: &str| std::env::var(name).ok(); find_local_venv_site_packages_with(cwd, &var).await @@ -378,6 +380,9 @@ async fn find_local_venv_site_packages_with( let matches = find_site_packages_under(&venv_path, "site-packages").await; results.extend(matches); if !results.is_empty() { + // `hatch run` / `hatch shell` never use an activated venv that + // is not one of Hatch's own (#335): its envs stay the project's. + add_hatch_site_packages(cwd, var, &mut results).await; return results; } } @@ -419,9 +424,36 @@ async fn find_local_venv_site_packages_with( results = pdm_pep582_dirs(cwd).await; } + // 6. Hatch never installs into `./.venv` / `./venv`: `hatch run` uses + // its own out-of-tree envs (#335). Every existing one belongs to the + // project, next to whatever a generic probe found. + add_hatch_site_packages(cwd, var, &mut results).await; + results } +/// Appends the `site-packages` of every existing Hatch env of the project +/// at `cwd` (see [`super::hatch_env::hatch_environments`]) not already in +/// `results`. +async fn add_hatch_site_packages( + cwd: &Path, + var: &impl Fn(&str) -> Option, + results: &mut Vec, +) { + for env in super::hatch_env::hatch_environments_with(cwd, var).await { + for site in hatch_env_site_packages(&env).await { + if !results.contains(&site) { + results.push(site); + } + } + } +} + +/// The `site-packages` directories of one Hatch env. +pub async fn hatch_env_site_packages(env: &super::hatch_env::HatchEnvironment) -> Vec { + find_site_packages_under(&env.prefix, "site-packages").await +} + /// The `site-packages` of the env the project's package manager records for /// `cwd`, when that env exists. `None` means the manager records nothing /// (or nothing installed yet), and the generic probes decide. @@ -2545,6 +2577,74 @@ mod tests { } } + /// #335: Hatch keeps a project's envs out of tree, under + /// `/env/virtual///`, and never uses `./.venv` + /// for them. Every existing env is the project's (stale-install probes, + /// VEX's installed basis and agent mode see them all), alongside a + /// `./.venv` another tool made. + #[tokio::test] + async fn hatch_out_of_tree_envs_are_project_envs() { + let tmp = tempfile::tempdir().unwrap(); + let project = tmp.path().join("app"); + std::fs::create_dir_all(&project).unwrap(); + std::fs::write( + project.join("pyproject.toml"), + "[build-system]\nrequires = [\"hatchling\"]\nbuild-backend = \"hatchling.build\"\n\n[project]\nname = \"app\"\nversion = \"0.1.0\"\ndependencies = [\"six==1.16.0\"]\n", + ) + .unwrap(); + let data = tmp.path().join("hatch-data"); + let var = env_of(&[ + ("HATCH_DATA_DIR", data.to_string_lossy().into_owned()), + ( + "HOME", + tmp.path().join("home").to_string_lossy().into_owned(), + ), + ]); + assert!(find_local_venv_site_packages_with(&project, &var) + .await + .is_empty()); + + let envs = super::super::hatch_env::hatch_environments_with(&project, &var).await; + assert!(envs.is_empty()); + let storage_root = data.join("env").join("virtual").join("app"); + // Hatch's own id for the project root, whichever casefolding applies. + let real = std::fs::canonicalize(&project).unwrap(); + let mut sites = Vec::new(); + for id in super::super::hatch_env::project_ids_for_tests(&real) { + let (_, site) = fake_venv_root(&storage_root.join(&id).join("app")); + sites.push(site); + } + let found = find_local_venv_site_packages_with(&project, &var).await; + assert!(!found.is_empty()); + assert!(found.iter().all(|s| sites.contains(s)), "{found:?}"); + + // An activated venv that is not Hatch's does not hide them. + let other = tempfile::tempdir().unwrap(); + let activated_site = fake_venv(other.path(), "tool-venv"); + let mut activated = vec![( + "VIRTUAL_ENV", + other + .path() + .join("tool-venv") + .to_string_lossy() + .into_owned(), + )]; + activated.push(("HATCH_DATA_DIR", data.to_string_lossy().into_owned())); + activated.push(( + "HOME", + tmp.path().join("home").to_string_lossy().into_owned(), + )); + let found = find_local_venv_site_packages_with(&project, &env_of(&activated)).await; + assert_eq!(found.first(), Some(&activated_site), "{found:?}"); + assert!(found.iter().any(|s| sites.contains(s)), "{found:?}"); + + // A `./.venv` beside them is kept too. + let dot = fake_venv(&project, ".venv"); + let found = find_local_venv_site_packages_with(&project, &var).await; + assert!(found.contains(&dot)); + assert!(found.iter().any(|s| sites.contains(s))); + } + /// #502: PDM installs into the interpreter saved in `.pdm-python` (an /// out-of-tree `venv.in_project = false` venv, or one bound with /// `pdm use`), ahead of a stray `./.venv` and an activated venv. diff --git a/crates/socket-patch-core/src/vendor/pypi.rs b/crates/socket-patch-core/src/vendor/pypi.rs index 74883c23e..8f7fe84f0 100644 --- a/crates/socket-patch-core/src/vendor/pypi.rs +++ b/crates/socket-patch-core/src/vendor/pypi.rs @@ -609,10 +609,81 @@ async fn pipenv_stale_install_warning( record: &PatchRecord, listings: &InstalledSiteListings, ) -> Option { - use crate::crawlers::python_crawler::{find_local_venv_site_packages, PythonCrawler}; + use crate::crawlers::python_crawler::find_local_venv_site_packages; + let sites = find_local_venv_site_packages(project_root).await; + let stale_dirs = stale_install_sites(&sites, purl, record, listings).await; + if stale_dirs.is_empty() { + return None; + } + let name = parse_pypi_purl(strip_purl_qualifiers(purl)) + .map(|(name, _)| name.to_string()) + .unwrap_or_else(|| purl.to_string()); + let listed = stale_dirs + .iter() + .map(|d| d.display().to_string()) + .collect::>() + .join(", "); + Some(VendorWarning::new( + "pypi_pipenv_stale_install", + format!( + "{purl}: the UNPATCHED upstream release is still installed in {listed}. Pipenv does not reinstall a release that is already present (`pipenv install`, `pipenv install --deploy` and `pipenv sync` all keep those bytes), so the wired Pipfile.lock only protects fresh installs. Reinstall it from the lock without touching the Pipfile: `pipenv run pip uninstall -y {name} && pipenv sync` (`pipenv install --deploy` before Pipenv 2018), or `pipenv --rm && pipenv sync` for a clean virtualenv — NOT `pipenv uninstall`, which rewrites the Pipfile and re-locks the patch away; then `socket-patch vex` re-verifies the installed files." + ), + )) +} + +/// The Hatch twin of [`pipenv_stale_install_warning`]: Hatch keeps the +/// upstream release installed in an existing env (#335), so each env that +/// still holds it gets `pypi_hatch_stale_install` with the env-recreating +/// remedy. Only Hatch's own envs are judged (an activated one included): a +/// `./.venv` another tool made is not where `hatch run` installs. +async fn hatch_stale_install_warning( + project_root: &Path, + purl: &str, + record: &PatchRecord, + listings: &InstalledSiteListings, +) -> Vec { + use crate::crawlers::hatch_env::{environment_of, hatch_environments, stale_install_remedy}; + let envs = hatch_environments(project_root).await; + if envs.is_empty() { + return Vec::new(); + } + // Judged over Hatch's own envs only (an activated one is among them): a + // `./.venv` another tool made is not where `hatch run` installs. + let mut sites: Vec = Vec::new(); + for env in &envs { + sites.extend(crate::crawlers::python_crawler::hatch_env_site_packages(env).await); + } + stale_install_sites(&sites, purl, record, listings) + .await + .into_iter() + .filter_map(|site| { + let env = environment_of(&envs, &site)?; + Some(VendorWarning::new( + "pypi_hatch_stale_install", + format!( + "{purl}: the UNPATCHED upstream release is still installed in the Hatch environment `{}` ({}). {}", + env.name, + site.display(), + stale_install_remedy(&env.name) + ), + )) + }) + .collect() +} + +/// The `sites` that hold the package of `purl` with positive evidence of +/// unpatched bytes (a readable file at the upstream or another hash), and no +/// copy that verifies as patched. +async fn stale_install_sites( + sites: &[std::path::PathBuf], + purl: &str, + record: &PatchRecord, + listings: &InstalledSiteListings, +) -> Vec { + use crate::crawlers::python_crawler::PythonCrawler; use crate::patch::apply::{verify_file_patch, VerifyStatus}; if record.files.is_empty() { - return None; + return Vec::new(); } // Judged over the PROJECT'S venvs (VIRTUAL_ENV, ./.venv, ./venv, Pipenv's // WORKON_HOME venv) — never the staging dir a lock-only vendor fetched @@ -622,20 +693,20 @@ async fn pipenv_stale_install_warning( let base = strip_purl_qualifiers(purl).to_string(); let crawler = PythonCrawler::new(); let mut stale_dirs: Vec = Vec::new(); - for site in find_local_venv_site_packages(project_root).await { - let listed = listings.of(&site).await; - let found = crawler.find_by_purls_listed(&site, &listed, std::slice::from_ref(&base)); + for site in sites { + let listed = listings.of(site).await; + let found = crawler.find_by_purls_listed(site, &listed, std::slice::from_ref(&base)); if !found.contains_key(&base) { continue; } - if crate::vex::verify::verify_patch_record(&site, record) + if crate::vex::verify::verify_patch_record(site, record) .await .is_ok() { continue; } for (file, info) in &record.files { - let result = verify_file_patch(&site, file, info).await; + let result = verify_file_patch(site, file, info).await; if matches!( result.status, VerifyStatus::Ready | VerifyStatus::HashMismatch @@ -646,23 +717,7 @@ async fn pipenv_stale_install_warning( } } } - if stale_dirs.is_empty() { - return None; - } - let name = parse_pypi_purl(strip_purl_qualifiers(purl)) - .map(|(name, _)| name.to_string()) - .unwrap_or_else(|| purl.to_string()); - let listed = stale_dirs - .iter() - .map(|d| d.display().to_string()) - .collect::>() - .join(", "); - Some(VendorWarning::new( - "pypi_pipenv_stale_install", - format!( - "{purl}: the UNPATCHED upstream release is still installed in {listed}. Pipenv does not reinstall a release that is already present (`pipenv install`, `pipenv install --deploy` and `pipenv sync` all keep those bytes), so the wired Pipfile.lock only protects fresh installs. Reinstall it from the lock without touching the Pipfile: `pipenv run pip uninstall -y {name} && pipenv sync` (`pipenv install --deploy` before Pipenv 2018), or `pipenv --rm && pipenv sync` for a clean virtualenv — NOT `pipenv uninstall`, which rewrites the Pipfile and re-locks the patch away; then `socket-patch vex` re-verifies the installed files." - ), - )) + stale_dirs } /// Everything [`vendor_pypi_with_pipenv_version`] decides before it can @@ -780,7 +835,17 @@ async fn pypi_prelude<'p>( } } PypiFlavor::Hatch => { - match super::pypi_hatch::load(project_root, &canon_name, version, &record.uuid).await { + let loaded = + super::pypi_hatch::load(project_root, &canon_name, version, &record.uuid).await; + // Both a fresh vendor and a re-run over already-wired + // dependencies keep warning while a Hatch env still holds the + // upstream release (#335). A refusal probes nothing. + if loaded.is_ok() { + warnings.extend( + hatch_stale_install_warning(project_root, purl, record, installed_sites).await, + ); + } + match loaded { Ok(project) if project.in_sync => { wired_pin = project.pin; WiringPlan::InSync @@ -4990,6 +5055,85 @@ wheels = [ } } + /// #335: Hatch keeps the upstream release in an existing env (pip, and + /// uv before Hatch 1.16, never reinstall it), so vendoring a Hatch + /// project — fresh and re-run in sync — must name each such env with + /// the `hatch env remove` remedy. A `./.venv` Hatch does not use stays + /// out of it; a patched env gets nothing. + #[tokio::test] + async fn hatch_vendor_warns_about_a_stale_hatch_env() { + let fx = e2e_fixture().await; + swap_to_lock_flavor( + &fx, + &[( + "pyproject.toml", + "[build-system]\nrequires = [\"hatchling\"]\nbuild-backend = \"hatchling.build\"\n\n[project]\nname = \"proj\"\nversion = \"0.1.0\"\ndependencies = [\"six==1.16.0\"]\n\n[tool.hatch.envs.default]\npath = \".hatch-env\"\n", + )], + ) + .await; + let env = fx.root.join(".hatch-env"); + let site = if cfg!(windows) { + env.join("Lib").join("site-packages") + } else { + env.join("lib").join("python3.12").join("site-packages") + }; + tokio::fs::create_dir_all(site.join("six-1.16.0.dist-info")) + .await + .unwrap(); + touch(&env, "pyvenv.cfg", "home = /usr/bin\n").await; + touch(&site, "six.py", std::str::from_utf8(ORIG).unwrap()).await; + touch( + &site.join("six-1.16.0.dist-info"), + "METADATA", + "Metadata-Version: 2.1\nName: six\nVersion: 1.16.0\n", + ) + .await; + + let stale = |warnings: &[VendorWarning]| -> Vec { + warnings + .iter() + .filter(|w| w.code == "pypi_hatch_stale_install") + .map(|w| w.detail.clone()) + .collect() + }; + let VendorOutcome::Done { + result, + entry, + warnings, + } = vendor_six(&fx, &PatchSources::blobs_only(&fx.blobs), None).await + else { + panic!("vendor must be Done"); + }; + assert!(result.success, "{:?}", result.error); + crate::vendor::test_support::persist(&fx.root, "pkg:pypi/six@1.16.0", entry.unwrap()).await; + let details = stale(&warnings); + assert_eq!(details.len(), 1, "{warnings:?}"); + assert!( + details[0].contains("hatch env remove default"), + "{}", + details[0] + ); + assert!(details[0].contains("six.py") || details[0].contains("site-packages")); + assert!(!details[0].contains(".venv"), "{}", details[0]); + + // The in-sync re-run keeps warning while the env is stale. + let VendorOutcome::Done { warnings, .. } = + vendor_six(&fx, &PatchSources::blobs_only(&fx.blobs), None).await + else { + panic!("re-run must be Done"); + }; + assert_eq!(stale(&warnings).len(), 1, "{warnings:?}"); + + // Once the env holds the patched bytes there is nothing to say. + touch(&site, "six.py", std::str::from_utf8(PATCHED).unwrap()).await; + let VendorOutcome::Done { warnings, .. } = + vendor_six(&fx, &PatchSources::blobs_only(&fx.blobs), None).await + else { + panic!("re-run must be Done"); + }; + assert!(stale(&warnings).is_empty(), "{warnings:?}"); + } + /// One full vendor → revert cycle through `vendor_pypi` for a lock-splice /// flavor: plan arm, wire arm, `entry.flavor` tag (PypiFlavor::as_str), /// the matching MetaSlot, and the byte-identical lock restore. diff --git a/docs/testing/hatch.md b/docs/testing/hatch.md index fa1742976..543db2867 100644 --- a/docs/testing/hatch.md +++ b/docs/testing/hatch.md @@ -33,10 +33,23 @@ the direct-reference permission. Selective and preserved rollback retain the set project direct reference remains, and restore its original value after the last reference is unwired. +Existing environments: Hatch keeps a project's environments out of tree +(`/env/virtual///`, or `HATCH_DATA_DIR`, +`[dirs.env] virtual`, an env's `path`; Hatch 1.0-1.2 use +`-/`), and on the next `hatch run` its pip installer (and +uv before Hatch 1.16) keeps the release already installed there. Socket +Patch finds those environments itself: a hosted scan warns +`redirect_pypi_stale_install`, a vendored one `pypi_hatch_stale_install`, +both naming `hatch env remove ` / `hatch env prune`; hosted `vex` +does not attest over a stale env, and vendored `vex` discloses it with +`vendored_tree_out_of_sync`. Agent mode crawls the same environments. + Focused Rust checks: ```sh cargo test --locked -p socket-patch-core --lib hatch +# real Hatch, existing envs (needs uv + PyPI): +SOCKET_PATCH_HATCH_E2E_REQUIRED=1 cargo test -p socket-patch-cli --test e2e_vex_build -- hatch::hatch_existing_env --ignored ``` The depscan companion PR runs real released Hatch binaries, actual CLI