Fix gem pair model ignoring custom lockfile and Bundler 1 twins (#749, #751)

[Mikola Lysenko (mikolalysenko)]

Fixes #749
Fixes #751

Summary

Hosted gem mode wired the wrong files in two Bundler layouts. In both, the scan reported success (and --vex attested the patch) while Bundler installed the unpatched gem or every frozen install failed. Both are now refused before anything is written.

Root cause

formats::gem::manifest::LoadedManifest is the single model of "which manifest/lock pair does Bundler load", and hosted mode (keep_bundler_loaded_gem_files) and vendored mode (gem_manifest_refusal) both rely on it. It modelled BUNDLE_GEMFILE, but:

• Hosted gem redirect ignores Bundler 4's custom lockfile (lockfile setting / BUNDLE_LOCKFILE), so it never pins the lock Bundler uses and frozen installs fail with no warning #749: it derived the lock only from the manifest name, so it never saw Bundler 4's custom lockfile (BUNDLE_LOCKFILE env, or lockfile in .bundle/config). The lock Bundler reads was never pinned. A leftover Gemfile.lock was rewritten instead, even though Bundler ignores it.
• Hosted gem redirect wires gems.rb in a Gemfile/gems.rb twin locked by Bundler 1.17, which loads Gemfile, so the install stays unpatched while the in-run VEX attests it #751: default discovery for a Gemfile + gems.rb twin always followed Bundler ≥ 2 (gems.rb first). Bundler 1.x loads Gemfile first.

Fix

• LoadedManifest::with_lockfile resolves the configured lockfile the way Bundler::CLI does (env first, then app config, then the global ~/.bundle/config that Gem settings resolution skips Bundler's global config (~/.bundle/config / BUNDLE_USER_CONFIG), so a global cache_path or gemfile gets no warning or refusal and VEX attests an unpatched install #577 / Fix Bundler global config being ignored (#577) #621 added; BUNDLE_IGNORE_CONFIG honoured; relative to the project root). If it names the loaded pair's own lock, nothing changes. Anything else is the new UnsupportedLockfile:
  • hosted mode refuses with redirect_gem_bundle_lockfile_unsupported (nothing written, nothing attested);
  • vendored mode refuses with gemfile_not_loaded.
• A Gemfile + gems.rb twin under default discovery is refused with redirect_gem_twin_manifest_ambiguous (manifest::twin_manifest_refusal), whatever its locks' BUNDLED WITH say. Nothing is written or attested. BUNDLED WITH records which Bundler wrote a lock, not which one installs it, and the twin's pair depends only on the running major. Lock inventory and VEX discovery read no lock for a twin (gem_lock_unsupported). A spelling counts as present whenever Bundler's File.file? may see it, which includes one this run couldn't read (symlinked, unreadable or non-UTF-8). BUNDLE_GEMFILE naming either spelling still selects that pair, and a lone Gemfile or gems.rb is unchanged. This matches vendored mode, which already refused twins. An earlier revision picked the pair from BUNDLED WITH; see the security-review follow-up below.
• Docs: CLI_CONTRACT.md (new additive warning codes), docs/ecosystems.md (RubyGems row). The CHANGELOG entry was dropped in 7c84ba8.

Tests (red → green)

Issue	Test	Without fix	With fix
#749	hosted_memory_engine::a_bundler4_custom_lockfile_is_refused	FAILED (rewrote the leftover lock)	ok
#749	e2e_redirect_gem_build::gem_hosted_bundler4_custom_lockfile_redirects_nothing (real Bundler 4.0.17)	FAILED: redirected: 1, rewrittenFiles: [Gemfile, Gemfile.lock], vex statements: 1	ok; frozen bundle install of the untouched project succeeds
#749	ruby_crawler::loaded_manifest_reads_the_lockfile_setting (disk, no leftover lock; env vs config priority; BUNDLE_IGNORE_CONFIG)	new API	ok
#749 × #577	manifest::with_lockfile_accepts_only_the_pairs_own_lock (global tier, shadowed by the app config) and ruby_crawler::loaded_manifest_reads_the_global_config_below_local_and_env (bundle config set --global lockfile)	new (merge)	ok
#749	vendor::gem::a_bundler4_custom_lockfile_is_refused	new	ok
#749	hosted_memory_engine::a_lockfile_setting_naming_the_default_lock_is_wired (control)	ok	ok
#751	hosted_memory_engine::a_gemfile_gems_rb_twin_is_refused (1.x, 2.x and mixed stamps)	FAILED (wired gems.rb)	ok
#751	engine::twin_redirects_nothing_whatever_the_locks_say	FAILED	ok
#751	engine::twin_with_an_unreadable_spelling_redirects_nothing (memory symlink / non-UTF-8 spelling)	FAILED	ok
#751	engine::twin_with_an_unreadable_disk_spelling_redirects_nothing (mode-000 gems.rb, non-root)	FAILED	ok
#751	lock_inventory::tests::gem_inventory_diagnoses_a_lock_it_cannot_read (symlinked twin spellings)	FAILED	ok
#751	engine::default_discovery_wires_a_lone_gems_rb (control)	ok	ok
#751	e2e_redirect_gem_build::gem_hosted_twin_redirects_nothing (every Bundler line; checks the untouched twin still installs frozen)	FAILED (wired gems.rb, attested)	ok (Bundler 4.0.17)
both	manifest.rs unit tests (with_lockfile_*, config_lockfile_*)	new	ok

Local results

• cargo clippy --workspace --all-features -- -D warnings: clean.
• cargo fmt --all -- --check is not clean on main itself (≈500 diffs, and CI has no fmt gate), so I formatted only the hunks I touched.
• cargo test --workspace --all-features --lib --bins: the core lib has 4848 passed and 4 failed. All four (copy_tree::relax_loop_must_not_traverse_symlinked_root, vlt_heal::an_unremovable_hidden_lock_keeps_every_store_entry, pypi_poetry::wire_write_failure_*, pypi_requirements::wire_failure_rolls_back_*) are chmod-based write-failure tests that can't fail writes when run as root (uid 0) in this sandbox. They don't touch gem code.
• Integration tests: hosted_memory_engine 33/33, hosted_memory_parity, in_process_vendor and e2e_redirect_gem_stale_install all ok. in_process_redirect has 104 passed and 3 failed, again only the chmod-based write-failure tests (root sandbox).
• e2e_redirect_gem_build --ignored (real Bundler 4.0.17): new custom-lockfile, dual-boot and gems.rb arms pass.
• I couldn't run the full cargo test --workspace locally because the sandbox's disk allowance runs out building every integration-test binary. CI runs the full suite.

Merge with main (d502a07)

main gained #577 (Bundler's global config, #621) while this PR was open, and the two overlapped in LoadedManifest. The conflicts are resolved by keeping both. Because Bundler 4 reads lockfile through Bundler.settings, which includes the global file, with_lockfile now takes the global tier too. Without that, bundle config set --global lockfile custom.lock would have slipped past the #749 guard. BundlerEnv carries the global config path, and #621's call sites use it. After the merge: cargo clippy --workspace --all-features -- -D warnings is clean; the gem/ruby core lib tests (253), crawler_ruby_e2e (26), hosted_memory_engine (34), hosted_memory_parity, in_process_vendor (106), e2e_redirect_gem_stale_install, and real-Bundler e2e_redirect_gem_build --ignored (17) all pass. The full core lib has 5010 passing and the same 4 chmod-based failures from running as root here.

Bugbot follow-up (9e7af6e)

• Lock-only scans are no longer silent. Since Lock inventory reads only Gemfile.lock, so a gems.rb project's gems.locked is invisible and a stale Gemfile.lock is read instead #736, a gem lock bundler loads but socket-patch can't read (a custom BUNDLE_LOCKFILE, an unsupported BUNDLE_GEMFILE, or a diverging twin) left lock inventory empty with no warning. Inventory now adds a gem_lock_unsupported diagnosis, which scan and the in-memory engine report as a run-level warnings[] entry (documented in CLI_CONTRACT.md). Test: lock_inventory::tests::gem_inventory_diagnoses_a_lock_it_cannot_read (red→green). The hosted_memory_engine refusal tests assert the warning again.
• An empty BUNDLE_LOCKFILE shadows the tiers below it, as Settings#[] does, so an empty env or app-config value now clears a global custom lock.

Security-review follow-ups (a3d31b9, d44c4fb)

• Absolute BUNDLE_LOCKFILE on a memory view (a3d31b9). The in-memory engine compared the lockfile against /, so BUNDLE_LOCKFILE: "/Gemfile.lock" passed as the project's own lock and the pair was rewritten. An absolute value there is now UnsupportedLockfile. Test: engine::absolute_bundle_lockfile_redirects_nothing (red→green).
• Twins no longer trust BUNDLED WITH (d44c4fb). Every default twin is refused, as described under Fix.

Notes / follow-ups

• The Gemfile lockfile "custom.lock" DSL (issue matrix row 4) is Ruby code the model can't read. Bundler 4.0.17 itself fails frozen installs on that shape before any scan.
• A BUNDLE_LOCKFILE setting is refused even on Bundler < 4 (which ignores it). That's a conservative, fail-closed choice.

Takeover follow-up (hourly agent run, 2026-10-08)

• 2cb153f: merges main. Resolved the CLI_CONTRACT.md conflict by keeping main's redirect.patches[] text plus this PR's two gem codes.
• 42657c6: setup-php pin comment # v2 → # 2.37.2 (zizmor ref-version-mismatch, the same as Label the setup-php pin in ci.yml with its real tag #1118) for the red Audit GitHub Actions check.
• 95f1894 (Bugbot): the hosted twin refusal also counts a spelling that's symlinked, unreadable or not UTF-8.
• d00e81f (Bugbot): the same check also asks view.is_file, so an on-disk spelling with a permission-denied read still counts.
• 2ec14fc (security review): lock inventory's twin check (bundler_loaded_lock_diagnosed_in) fails closed on a memory-view symlinked spelling.
• On 2ec14fc: ci-ok is green, Bugbot is clean (completed/success), and every review thread is resolved. The only checks still running are the two non-required Windows gradle hosted cells.

🤖 Generated with Claude Code

https://claude.ai/code/session_018ULgrJMQMWEBAsiuprY449

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

Ready for review at 731f489.

• CI: 402/402 non-skipped checks green on the current head (9/9 workflows, path-filtered).
• Bugbot: reviewed 731f489 and found no issues. There are no unresolved review threads.
• Mergeable: yes, no conflicts with main. The diff is 10 files, all in the gem lane, plus docs and changelog.
• For the reviewer: look at the LoadedManifest pair model in formats/gem/manifest.rs (the configured BUNDLE_LOCKFILE and the BUNDLED WITH major of a Gemfile/gems.rb twin) and the matching ruby_crawler.rs changes.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

coverage (the Linux test gate) failed on the merge commit d502a07. Its one failing target is -p socket-patch-cli --lib. coverage was green on 731f489, before the merge with main.

I can't see which test failed yet. The job log's download URL (productionresultssa5.blob.core.windows.net) is blocked by my sandbox's network policy, and the MCP log tool returns only the last ~5000 of the log's 12022 lines. The CLI lib test section is earlier than that, so the panic isn't in what I can read. The check annotations only carry the exit code.

What I checked locally on d502a07:

• cargo test -p socket-patch-cli --lib with default features and with --all-features: 835/835 pass.
• With CI's SOCKET_PATCH_GO_E2E_REQUIRED=1 / SOCKET_PATCH_GO_E2E_VERSION=1.24 (Go 1.24.7): 835/835 pass.
• The same test binary run as a non-root user: 835/835 pass.
• This PR doesn't touch crates/socket-patch-cli/src. That code is identical to main's.

Next: the macOS and Windows test legs run the same CLI lib tests on this commit, and I'll use them to tell a real failure from an instrumentation or timing one. If anyone can open the job log (coverage → "Run tests with coverage", search for FAILED), the failing test name and panic would let me fix it directly.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Found it, and this PR doesn't cause it: main is red at 4646693. coverage and test-release test the PR merged into the current main. Both fail on the same two CLI lib tests, and they fail identically on main alone:

• commands::vex_consumed::tests::hosted_expands_alias_only_copies (vex_consumed.rs:771, assert!(installed.is_empty()))
• commands::vex_consumed::tests::hosted_reuses_expanded_npm_copies_and_merges_alias_variants (vex_consumed.rs:718, assert_eq!(installed_again, installed))

Cause: the two tests came in with #738 (#356) and assume the name-keyed resolver find_manifest_package_copies_reusing can't see aliased copies. #605 (#601/#603, merged as 4646693) made that resolver probe store entries' bundled trees, so it now returns the aliases, the nested alias and every store peer variant itself. I ran both tests on main with the asserts relaxed:

• calls == []: Fix agent mode skipping npm-aliased copies (#356) #738's alias expansion is no longer needed.
• The final paths set is the complete, correct one: {host/…/lp, left-pad(peer@1.0.0), left-pad(peer@1.0.1)} in the first test, and every root, alias, nested-alias and peer copy in the second.

So the behaviour is right and only the tests' premise is stale.

Proposed patch (for main, not this PR):

• In hosted_expands_alias_only_copies, replace assert!(installed.is_empty()) and assert_eq!(calls, vec[vec[alias.clone()]]) with:
  • installed[&purl] (sorted) equals peers + [alias] (sorted);
  • calls.is_empty();
  • paths == installed[&purl].
• In hosted_reuses_expanded_npm_copies_and_merges_alias_variants:
  • Expect installed_again[&purl] to already contain nm/lp, host/node_modules/lp and the nested peers.
  • Replace calls.len() == 1 / the alias-inputs check with calls.is_empty().
  • Keep the final sorted-set and canonical-uniqueness assertions.
  • The earlier nm/lp step (calls == [[alias]]) needs the same update if the resolver also finds that alias. On main it currently doesn't fail first, so check it when applying.

I'm not adding this to #768. It's in code unrelated to the gem change, and no fix for it is open yet. Once main is green, I'll merge it into this branch and re-run CI.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Burn-down agent: pushed 8c2ba59.

• test (macos-latest), test-release and coverage were red on d502a07. The failures were vex_consumed::tests::hosted_expands_alias_only_copies and hosted_reuses_expanded_npm_copies_and_merges_alias_variants, which come from main and have been red there since Fix npm store copies missed by agent apply and vex (#601, #603) #605. I merged main and cherry-picked Fix vex alias tests broken by store-copy merge #851, a test-only change that becomes a no-op once Fix vex alias tests broken by store-copy merge #851 lands.
• Locally: clippy (CI command) is clean, and socket-patch-core --lib gem (280), socket-patch-cli --lib vex_consumed (10) and --test hosted_memory_engine (34) all pass.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

Burn-down agent: labeled Ready for review.

• Head: 8c2ba59421a2e49711197f1a85f8b7ad7ea879a6
• CI: 458/458 check runs green (success/skipped) on this head; mergeable clean
• Bugbot: reviewed 8c2ba59, no new issues; no open threads
• Reviewer focus: gem pair model honoring a custom lockfile and Bundler 1 twins (Hosted gem redirect ignores Bundler 4's custom lockfile (lockfile setting / BUNDLE_LOCKFILE), so it never pins the lock Bundler uses and frozen installs fail with no warning #749, Hosted gem redirect wires gems.rb in a Gemfile/gems.rb twin locked by Bundler 1.17, which loads Gemfile, so the install stays unpatched while the in-run VEX attests it #751)

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Audit GitHub Actions (zizmor) is red on e029d70, but the cause is outside this PR. The finding is ref-version-mismatch at .github/workflows/ci.yml:1400: shivammathur/setup-php@f3e473d… # v2. Upstream has moved the v2 tag to 7d671baa1068, so the # v2 comment no longer matches the pinned SHA. f3e473d is tag 2.37.2. This PR doesn't touch .github/.

The same failure appeared at the same time on unrelated branches (arch-refactor/905-bom-sites, agent/fix-bun-open-issues). It passed on main 829d0af at 06:14Z, before the tag moved. The check isn't among ci-ok's needs, so it doesn't block the merge.

I found no fix PR. The proposed one-line patch matches the comment composer-compatibility.yml:115 already uses:

-        uses: shivammathur/setup-php@f3e473d116dcccaddc5834248c87452386958240 # v2
+        uses: shivammathur/setup-php@f3e473d116dcccaddc5834248c87452386958240 # 2.37.2

I'm not re-running the check, because it will fail the same way until that line changes on main.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] Took this PR over because its heartbeat was stale (03:28Z), the head conflicted with main, and the Audit GitHub Actions check was red on e029d70. Pushed 42657c6:

• 2cb153f: merges main. The only conflict was the scan --mode hosted paragraph in CLI_CONTRACT.md, resolved by keeping main's redirect.patches[] text plus this PR's redirect_gem_bundle_lockfile_unsupported / redirect_gem_twin_manifest_ambiguous codes.
• 42657c6: relabels the setup-php pin comment # v2 to # 2.37.2 (zizmor ref-version-mismatch, since upstream moved v2). This is the same one-line change as Label the setup-php pin in ci.yml with its real tag #1118, and it no-ops once that lands.

Local checks: cargo clippy --workspace --all-features -- -D warnings is clean; core lib gem/ruby/manifest/lock_inventory tests pass (684); hosted_memory_engine (37) and e2e_redirect_gem_stale_install (33) pass. CI and Bugbot are pending on the new head.

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2ec14fc. Configure here.}