Skip to content

CI: hermetic spawns in the production e2e suites (#823) - #1528

Merged
Mikola Lysenko (mikolalysenko) merged 3 commits into
mainfrom
agent/fix-ci-hermetic-production
Oct 14, 2026
Merged

Mikola Lysenko (mikolalysenko) merged 3 commits into
mainfrom
agent/fix-ci-hermetic-production

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

LLM Description written by Claude Code:claude-opus-5-5

Refs #823

Summary

This is the last bare Command::new slice of #823. Every test child that spawns the socket-patch binary through a path the ratchet can see now goes through tests/common/hermetic.rs.

  • e2e_hosted_production.rs / e2e_vendored_production.rs: all four binary spawns (run / run_socket and both bun_manifestless_vex_production VEX closures) use hermetic::binary_command(); run / run_socket also apply Extra::Venv. Before:

    • the VEX closures didn't force the update check off;
    • the vendored run_socket kept an ambient SOCKET_NO_CONFIG/SOCKET_TELEMETRY_DISABLED value instead of forcing them;
    • the hosted run let an activated venv reach the unscoped hosted scan.
  • Hostile seeds: the private seeds those suites planted (SOCKET_API_TOKEN, SOCKET_API_URL, SOCKET_PROXY_URL, SOCKET_OFFLINE) move into the shared HOSTILE_SEEDS. The builder already removed all four by name (they are flag-bound), so no other suite's child sees a different environment. The only effect is that the "a dropped scrub turns the suite red" guard now covers them for every child.

  • PENDING_RAW_SPAWNS is removed, so no_new_bare_binary_spawns is now a hard ban.

  • Found by /code-review high: two more bare spawns that the detector misses because they bind the binary path to a local first.

    • vendor_ecosystem_fixtures/mod.rs (shared by vendor_group_commit_e2e and vendor_ledger_schema_e2e).
    • cli_parse_scan.rs's empty-scan JSON-shape test.

    Both swept SOCKET_* by hand. The fixture also kept an ambient SOCKET_NO_CONFIG=0, so a developer's socket login could reach the child. Both now use the builder. The ratchet's doc now says what the detector can and can't see.

Test-only change; no production code.

CI cost

There are no workflow changes, so a lean PR run has the same jobs before and after: 28 jobs and about 8 minutes wall-clock. No cfg(windows)/cfg(unix) code is touched, so no full-scope dispatch.

Test evidence

Run locally on 4b81320 (rust 1.93.1):

  • cargo fmt --all -- --check: clean.

  • cargo clippy --workspace --all-features -- -D warnings: clean.

  • cargo test -p socket-patch-cli --all-features on these targets:

    target result
    spawn_env_hygiene 16 passed
    cli_parse_scan 48 passed
    vendor_group_commit_e2e 31 passed
    vendor_ledger_schema_e2e 28 passed
    e2e_hosted_production 51 passed, 16 ignored (live)
    e2e_vendored_production 51 passed, 13 ignored (live)
    scan 131 passed
    cli 111 passed
    repair 127 passed, 2 failed

    The two repair failures are the "lock file unremovable" tests, which can't simulate an unremovable file as root. That is the baseline already noted in Spawn every CLI test child through one hermetic Command builder; 10 test files inherit ambient SOCKET_* today #823 and is unrelated.

  • The same targets under a hostile shell (SOCKET_DRY_RUN=true SOCKET_OFFLINE=true SOCKET_NO_CONFIG=0 SOCKET_ECOSYSTEMS=cargo SOCKET_JSON=false) give identical results.

  • The live production legs can't run locally: the sandbox proxy refuses patches-api.socket.dev (403). On this PR's CI, hosted-e2e ran the lean live leg vlt_pinned_matrix_production_vendored_install_proof through the migrated run_socket, and it passed. The other live legs run in the scheduled full tier.

  • Honest note: on main, cli_parse_scan and the two vendor-fixture suites also pass under that hostile shell, because their hand-written sweeps covered those vars. What this PR fixes there is the forced SOCKET_NO_CONFIG / SOCKET_NO_UPDATE_CHECK. That only shows with a real socket login or update state on the host, so there's no red→green test for it beyond the existing spawn_env_hygiene builder contract tests.

CI on 4b81320: ci-ok, clippy and every lean leg are green. Bugbot reviewed 4b81320 and found no issues.

Why Refs, not Fixes

#823's acceptance criteria still have open items after this PR:

  • The 7 portable_pty::CommandBuilder spawns (covgap_commands_scan_mod, covgap_commands_rollback, cli/interactive_prompts_e2e, cli/covgap_output, remove/covgap_commands_remove, update/covgap_commands_update, update/update_notifier_e2e) keep their own scrubs. They need a CommandBuilder counterpart of the builder.
  • The detector doesn't catch a binary path bound to a local first. This PR migrated the two such sites that exist today.

Review findings not taken

These come from /code-review high:

  • Add seeds for SOCKET_PATCH_SERVER_URL / SOCKET_VENDOR_URL / SOCKET_NPM_REGISTRY: all are flag-bound and already removed by name; the seed list mirrors what the production suites seeded. Left for a follow-up if wanted.
  • The two production suites' run helpers are now near-identical: each suite is a standalone target with its own docs on why the scrub matters. A shared helper adds a module for two 8-line functions.
  • cmd.env_remove("SOCKET_API_TOKEN") in the builder is now redundant: kept, because its comment documents the public-proxy guarantee.
  • The production suites' package-manager tool() children hand-roll the VIRTUAL_ENV / linker scrubs instead of scrub_socket_vars + Extra: real but out of scope (they aren't binary spawns). This is a follow-up for Spawn every CLI test child through one hermetic Command builder; 10 test files inherit ambient SOCKET_* today #823's package-manager half.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DzsEDJZKqeSa2Vurg5PTq5

Assisted-by: Claude Code:claude-opus-5-5
The hosted and vendored production suites started socket-patch with a
bare Command::new and their own SOCKET_* sweeps. The vex legs forced
neither the update check nor telemetry off, and the hosted run let an
activated venv reach the unscoped scan. All four spawns now go through
the shared hermetic builder with the venv scrubbed.

Their private hostile seeds (API token, API and proxy URLs, offline)
move into the shared HOSTILE_SEEDS. The builder already removed those
names, so no other suite's child sees a different environment.

With these two files migrated, PENDING_RAW_SPAWNS is empty and goes
away: the ratchet now rejects any bare binary spawn in the test tree.

Refs #823

Assisted-by: Claude Code:claude-opus-5-5
The shared vendor fixture and the cli_parse_scan JSON-shape test bound
the binary path to a local before spawning it, so the ratchet's
detector never saw them. Both swept SOCKET_* by hand and kept an
ambient SOCKET_NO_CONFIG=0, which let a developer's socket login reach
the child. They now spawn through the shared hermetic builder.

The ratchet's doc now says which spellings it can see and that PTY
spawns keep their own scrubs, and a stale comment about how the
vendored pypi leg discovers its venv is corrected.

Refs #823

Assisted-by: Claude Code:claude-opus-5-5
@mikolalysenko
Mikola Lysenko (mikolalysenko) marked this pull request as ready for review October 14, 2026 09:19
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

BugBot review


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 4b81320. Configure here.

Merged via the queue into main with commit a8da202 Oct 14, 2026
86 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the agent/fix-ci-hermetic-production branch October 14, 2026 09:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants