Repository navigation
CI: hermetic spawns in the production e2e suites (#823) - #1528
Merged
Mikola Lysenko (mikolalysenko) merged 3 commits intoOct 14, 2026
Merged
Conversation
Assisted-by: Claude Code:claude-opus-5-5
This was referenced Oct 14, 2026
The hosted and vendored production suites started socket-patch with a bare Command::new and their own SOCKET_* sweeps. The vex legs forced neither the update check nor telemetry off, and the hosted run let an activated venv reach the unscoped scan. All four spawns now go through the shared hermetic builder with the venv scrubbed. Their private hostile seeds (API token, API and proxy URLs, offline) move into the shared HOSTILE_SEEDS. The builder already removed those names, so no other suite's child sees a different environment. With these two files migrated, PENDING_RAW_SPAWNS is empty and goes away: the ratchet now rejects any bare binary spawn in the test tree. Refs #823 Assisted-by: Claude Code:claude-opus-5-5
The shared vendor fixture and the cli_parse_scan JSON-shape test bound the binary path to a local before spawning it, so the ratchet's detector never saw them. Both swept SOCKET_* by hand and kept an ambient SOCKET_NO_CONFIG=0, which let a developer's socket login reach the child. They now spawn through the shared hermetic builder. The ratchet's doc now says which spellings it can see and that PTY spawns keep their own scrubs, and a stale comment about how the vendored pypi leg discovers its venv is corrected. Refs #823 Assisted-by: Claude Code:claude-opus-5-5
Mikola Lysenko (mikolalysenko)
marked this pull request as ready for review
October 14, 2026 09:19
Collaborator
Author
|
BugBot review Generated by Claude Code |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 4b81320. Configure here.
Tanmay Singla (Tanmay182003)
approved these changes
Oct 14, 2026
Mikola Lysenko (mikolalysenko)
deleted the
agent/fix-ci-hermetic-production
branch
October 14, 2026 09:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LLM Description written by Claude Code:claude-opus-5-5
Refs #823
Summary
This is the last bare
Command::newslice of #823. Every test child that spawns thesocket-patchbinary through a path the ratchet can see now goes throughtests/common/hermetic.rs.e2e_hosted_production.rs/e2e_vendored_production.rs: all four binary spawns (run/run_socketand bothbun_manifestless_vex_productionVEX closures) usehermetic::binary_command();run/run_socketalso applyExtra::Venv. Before:run_socketkept an ambientSOCKET_NO_CONFIG/SOCKET_TELEMETRY_DISABLEDvalue instead of forcing them;runlet an activated venv reach the unscoped hosted scan.Hostile seeds: the private seeds those suites planted (
SOCKET_API_TOKEN,SOCKET_API_URL,SOCKET_PROXY_URL,SOCKET_OFFLINE) move into the sharedHOSTILE_SEEDS. The builder already removed all four by name (they are flag-bound), so no other suite's child sees a different environment. The only effect is that the "a dropped scrub turns the suite red" guard now covers them for every child.PENDING_RAW_SPAWNSis removed, sono_new_bare_binary_spawnsis now a hard ban.Found by
/code-review high: two more bare spawns that the detector misses because they bind the binary path to a local first.vendor_ecosystem_fixtures/mod.rs(shared byvendor_group_commit_e2eandvendor_ledger_schema_e2e).cli_parse_scan.rs's empty-scan JSON-shape test.Both swept
SOCKET_*by hand. The fixture also kept an ambientSOCKET_NO_CONFIG=0, so a developer'ssocket logincould reach the child. Both now use the builder. The ratchet's doc now says what the detector can and can't see.Test-only change; no production code.
CI cost
There are no workflow changes, so a lean PR run has the same jobs before and after: 28 jobs and about 8 minutes wall-clock. No cfg(windows)/cfg(unix) code is touched, so no full-scope dispatch.
Test evidence
Run locally on 4b81320 (rust 1.93.1):
cargo fmt --all -- --check: clean.cargo clippy --workspace --all-features -- -D warnings: clean.cargo test -p socket-patch-cli --all-featureson these targets:spawn_env_hygienecli_parse_scanvendor_group_commit_e2evendor_ledger_schema_e2ee2e_hosted_productione2e_vendored_productionscanclirepairThe two
repairfailures are the "lock file unremovable" tests, which can't simulate an unremovable file as root. That is the baseline already noted in Spawn every CLI test child through one hermetic Command builder; 10 test files inherit ambient SOCKET_* today #823 and is unrelated.The same targets under a hostile shell (
SOCKET_DRY_RUN=true SOCKET_OFFLINE=true SOCKET_NO_CONFIG=0 SOCKET_ECOSYSTEMS=cargo SOCKET_JSON=false) give identical results.The live production legs can't run locally: the sandbox proxy refuses
patches-api.socket.dev(403). On this PR's CI,hosted-e2eran the lean live legvlt_pinned_matrix_production_vendored_install_proofthrough the migratedrun_socket, and it passed. The other live legs run in the scheduled full tier.Honest note: on main,
cli_parse_scanand the two vendor-fixture suites also pass under that hostile shell, because their hand-written sweeps covered those vars. What this PR fixes there is the forcedSOCKET_NO_CONFIG/SOCKET_NO_UPDATE_CHECK. That only shows with a realsocket loginor update state on the host, so there's no red→green test for it beyond the existingspawn_env_hygienebuilder contract tests.CI on 4b81320:
ci-ok,clippyand every lean leg are green. Bugbot reviewed 4b81320 and found no issues.Why
Refs, notFixes#823's acceptance criteria still have open items after this PR:
portable_pty::CommandBuilderspawns (covgap_commands_scan_mod,covgap_commands_rollback,cli/interactive_prompts_e2e,cli/covgap_output,remove/covgap_commands_remove,update/covgap_commands_update,update/update_notifier_e2e) keep their own scrubs. They need aCommandBuildercounterpart of the builder.Review findings not taken
These come from
/code-review high:SOCKET_PATCH_SERVER_URL/SOCKET_VENDOR_URL/SOCKET_NPM_REGISTRY: all are flag-bound and already removed by name; the seed list mirrors what the production suites seeded. Left for a follow-up if wanted.runhelpers are now near-identical: each suite is a standalone target with its own docs on why the scrub matters. A shared helper adds a module for two 8-line functions.cmd.env_remove("SOCKET_API_TOKEN")in the builder is now redundant: kept, because its comment documents the public-proxy guarantee.tool()children hand-roll the VIRTUAL_ENV / linker scrubs instead ofscrub_socket_vars+Extra: real but out of scope (they aren't binary spawns). This is a follow-up for Spawn every CLI test child through one hermetic Command builder; 10 test files inherit ambient SOCKET_* today #823's package-manager half.🤖 Generated with Claude Code
https://claude.ai/code/session_01DzsEDJZKqeSa2Vurg5PTq5