Fix Python crawler missing .egg-info installs (#447)

[Mikola Lysenko (mikolalysenko)]

LLM Description written by Claude Code:claude-opus-5-5

Fixes #447

Root cause

list_dist_info_packages_sync in crates/socket-patch-core/src/crawlers/python_crawler.rs keeps only *.dist-info entries. Legacy installs record their metadata as <name>-<ver>[-pyX.Y].egg-info instead. That covers pip < 23.1 installing an sdist without wheel, distutils, and Debian/Ubuntu apt packages, and the crawler never sees any of them. Every consumer reads that one listing: agent apply/rollback lookups, the hosted redirect_pypi_stale_install guard, the Pipenv vendored stale-install probe, scan -g, and VEX. So all of them miss these packages.

Fix

The site-packages listing (and its test-only equivalence oracle) now reads:

• .egg-info directories: the PKG-INFO headers, falling back to the <name>-<version>[-pyX.Y].egg-info name the way the .dist-info reader falls back to its dir name. setuptools escapes - to _, so the first - ends the name and the second starts the interpreter tag.
• bare .egg-info files (distutils, distro packages such as PyGObject-3.48.2.egg-info). A file only counts when its headers parse, the same guard as the stray *.dist-info file case.

The wrappers under npm/, pypi/ and gem/ only dispatch to the binary, so they need no change.

Behaviour change worth a reviewer's eye

Apt-installed Python packages on Debian/Ubuntu (/usr/lib/python3/dist-packages/*.egg-info) are now visible. That's what #447 asks for under -g. A Python project with no venv already falls back to the global interpreters, so those copies now take part there too, the same way a pip-installed system copy always has. Two CI tests only passed because Ubuntu's apt python3-six 1.16.0 (the fixtures' exact release) was invisible. Each one claims "nothing installed", and both now point VIRTUAL_ENV at an empty venv to make that true:

• e2e_vex_build::hatch hosted pin-basis step: vex correctly refused to attest over the unpatched apt copy.
• vendor_eject_fresh_checkout::pypi_eject_needs_no_virtualenv: the release-variant probe compared the apt copy against the fixture's bytes.

Neither test's assertions changed.

Tests (red → green)

Issue item	Test	Before fix	After
agent mode patches an egg-info install (dir + bare file)	in_process_pypi_apply::pypi_scan_sync_patches_egg_info_install	FAIL (file left unpatched)	pass
hosted stale-install warning fires	scan::hosted::python::tests::egg_info_install_gets_the_stale_install_warning	FAIL (no warning)	pass
discovery / -g / --global-prefix reports egg-info packages	in_process_python_envs::pypi_egg_info_layout_handled (flipped from the pinned "not discovered" contract)	FAIL	pass
reader edge cases (-pyX.Y, headerless dir, bare file, headerless stray file)	python_crawler::tests::egg_info_entries_are_listed, test_parse_egg_info_dir_name	n/a	pass
parallel scan == async oracle (its generator already plants egg-info dirs)	python_crawler::tests::equivalence::randomized_site_packages_match_the_async_oracle	—	pass

Real-world check: pip 23.0.1 in a fresh CPython 3.11 venv without wheel (pip install --no-binary six six==1.16.0) writes exactly six-1.16.0-py3.11.egg-info/ + six.py, the layout the new agent test plants (Windows: Lib\site-packages).

Local runs:

• cargo clippy --workspace --all-features -- -D warnings: clean.
• cargo test --workspace --all-features, batched with --no-fail-fast to fit the sandbox disk: 8324 passed, 10 failed. All 10 are write-failure/permission tests that fail identically on main here because the sandbox runs as root.
• cargo fmt --check: main itself isn't fmt-clean and CI doesn't run it, so I didn't reformat unrelated files. The touched files are rustfmt-clean.

CI on 4cb6673: all 481 check runs green (475 success, 6 conditional skips). Earlier single-cell failures on older heads (Bun 1.1.39, Pipenv 2018.11.26 vendored, PDM 0.12.3 space-unicode agent) didn't recur, which fits the live-API transport flakes noted in the comments. Bugbot reviewed 4cb6673: no issues.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MQdCDEbnNPApU1yUukp19S

---

Note

Medium Risk
Changes shared Python install discovery used by scan, apply, hosted redirects, and VEX; apt/global egg-info packages become visible by design, with broad test coverage but real-environment layout variance.

Overview
Fixes #447 by teaching the Python site-packages scan to treat legacy .egg-info installs like .dist-info, so scan/apply, hosted stale-install checks, and global discovery no longer skip older pip sdist installs, distutils layouts, and distro packages (e.g. Debian python3-*).

Crawler: list_dist_info_packages now reads .egg-info directories via PKG-INFO (with a <name>-<version>[-pyX.Y].egg-info filename fallback) and bare .egg-info files when headers parse; the test oracle matches the blocking scan.

Tests: Discovery expectations flip from “egg-info ignored” to reported PURLs; new coverage for scan --sync patching egg-info layouts, hosted redirect_pypi_stale_install on egg-info, and reader edge cases. Two e2e/vendor tests set VIRTUAL_ENV to an empty venv so Ubuntu’s apt python3-six egg-info does not masquerade as “nothing installed.”

^{Reviewed by Cursor Bugbot for commit 4cb6673. Configure here.}

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] native (ubuntu-latest, 1.1.39) failed on 2801f42 (Bun backtest: 41/44 cells, hosted text / space-unicode / workspace-get-search). I don't think this PR caused it:

• The diff only touches the Python site-packages listing and a Hatch e2e. The Bun cells run npm/Bun code paths that this change doesn't reach.
• The log shows Connection reset by peer from the live https://patches-api.socket.dev/patch/batch in the same leg, which points to a transport flake against that host. The same leg was green on other recent PRs on the same base.

No fix to port. I'll re-run the failed jobs once when the workflow run finishes; a second failure gets treated as real.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] On 35394a9, the Pipenv compatibility leg matrix (ubuntu-latest, 2018.11.26 …) failed in exactly one cell: 2018.11.26 direct vendored, check rescanIdempotent. Every other check in that cell passed (lock-only apply, stale-install warning, lock rewrite, vex), and so did the other 31 cells.

I don't think this PR caused it:

• I re-ran that cell locally with this branch's binary. Its project venv holds only pip-24.0.dist-info, setuptools-69.5.1.dist-info and urllib3-1.26.18.dist-info, with no .egg-info. Because a venv exists, the global-interpreter fallback never runs. So the crawl is identical to main's.
• The step drives the live patches-api.socket.dev. In the same window, the Bun leg logged Connection reset by peer from that host. I couldn't run it end to end here because the sandbox can't reach the API.

No fix to port. I'm re-running the failed job once; a second failure gets treated as real.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] On 35394a9, the PDM compatibility leg native (ubuntu-latest, 0.12.3) failed in exactly one cell: 0.12.3 space-unicode agent, check appliedExactlyOne. The other 31 cells passed, including every other agent shape. The same leg is green on #425 and #383.

I can't yet rule this PR in or out. Agent mode does read the listing this PR changes, but nothing in this change depends on the project path, and the space/unicode path is the only thing different about that cell. The cell's envelope is in the run artifact, and this sandbox can't download artifacts. The harness also drives the live patches-api.socket.dev, so I can't reproduce the cell locally.

I've re-run the failed job once as the diagnostic. If it fails again, I'll treat it as real and dig into PDM 0.12's install layout for that shape.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 4cb6673. Configure here.}

[Mikola Lysenko (mikolalysenko)]

[burn-down agent] Ready for review on 4cb6673: mergeable, 0 commits behind main.

• CI: 476/482 check runs green, 6 skipped by workflow conditions, 0 failing. The earlier Pipenv/PDM leg failures were on 35394a9, not this head.
• Bugbot: reviewed 4cb6673, no issues. No open review threads.
• For the reviewer: the crawler now sees apt-installed .egg-info packages under /usr/lib/python3/dist-packages. Two e2e tests now point VIRTUAL_ENV at an empty venv so their "nothing installed" premise holds. See the description's "Behaviour change" section.

---

Generated by Claude Code