Skip to content

Fix ETXTBSY flake in vlt npx shim self-test - #441

Merged
Mikola Lysenko (mikolalysenko) merged 1 commit into
mainfrom
ci-janitor/vlt-shim-etxtbsy
Oct 1, 2026
Merged

Mikola Lysenko (mikolalysenko) merged 1 commit into
mainfrom
ci-janitor/vlt-shim-etxtbsy

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

vlt_e2e_harness_npx_shim_forwards_the_args_after_the_package fails intermittently on Linux with Text file busy (os error 26). It's a harness self-test in vlt_e2e_common, so it is compiled into 7 test binaries (e2e_vlt, e2e_redirect_vlt_build, e2e_vendor_vlt_build, mode_migration_vlt, e2e_safety_vlt, e2e_hosted_production, e2e_vendored_production). It runs in every test (ubuntu-latest), test-release and coverage job and in every vlt e2e leg, which gives it many chances per PR to fail.

Evidence:

  • run 36801003123: test-release failed on attempt 1 (job 110175102290) and passed on a rerun of the same SHA 18ff786b:
    test vlt_e2e_common::vlt_e2e_harness_npx_shim_forwards_the_args_after_the_package ... FAILED
    panicked at crates/socket-patch-cli/tests/vlt_e2e_common/mod.rs:2538:10:
    called `Result::unwrap()` on an `Err` value: Os { code: 26, kind: ExecutableFileBusy, message: "Text file busy" }
    
  • Over the last 200 CI runs this was the only test-release failure that wasn't either broken on the branch (all four test jobs red on one SHA) or limited to one PR's branch.

Root cause

The test writes two shell scripts with fs::write, chmods them, and execs them right away. Linux refuses execve on a file that any process holds open for writing. If a sibling test thread forks while our write fd is open, its child inherits that fd and holds it until the child execs (the fd is O_CLOEXEC). Under load that window is long enough to overlap our exec.

Locally I reproduced it at both exec sites:

  • our Command::new(npx).output() → ExecutableFileBusy at mod.rs:2538 (the CI failure)
  • the npx shim's own exec of the fake socket binary → npx: 9: exec: …/socket: Text file busy, which surfaces as an assert_ok failure at mod.rs:777

A retry around our spawn, like exec_freshly_written in update_fixture.rs, only covers the first site. The second exec happens inside sh, where we can't retry it.

Fix

Add write_executable(path, body) to vlt_e2e_common. On Unix it writes the bytes to <path>.staged, has a child cp create path, removes the staged file, and chmod 755s the result. The test process never holds a writable fd to the executable, so no sibling fork() can inherit one. This closes the window itself instead of retrying. Windows keeps the plain write.

write_executable is used for:

  • the fake socket script in the self-test
  • the npx/vlt sh shims in write_shims_for. The real-vlt legs (write_shims) also exec these, so they had the same exposure.

Coverage is unchanged: no test is removed, skipped or weakened, and every assertion stays the same.

Proof

Stress harness (a scratch test binary, not committed): it includes vlt_e2e_common and adds 8 sibling tests that spawn /bin/true in a tight loop for 1.5 s. Each iteration runs them alongside the shim test with --test-threads=16.

runs failures (Text file busy)
before (origin/main) 300 (2×150) 9 (7 + 2, at both exec sites)
after (this PR) 300 0

Without the fork churn, e2e_vlt at --test-threads=32 passed 200/200 even before the fix, which is consistent with a load-dependent race.

Also run locally:

  • vlt_e2e_harness* self-tests: pass in all 7 binaries; e2e_vlt non-ignored suite: 24 passed.
  • cargo clippy -p socket-patch-cli --test <each of the 7> -- -D warnings: no findings in vlt_e2e_common. The only findings are pre-existing ones in prebuilt_common/mod.rs; that file is untouched, and CI's clippy job doesn't build test targets.
  • rustfmt --check on the changed file.
  • I couldn't run the real-vlt legs (--include-ignored vlt_pinned_matrix) here because the sandbox can't reach npmjs. The PR's e2e (*, *vlt*, …) CI jobs run them.

Where tests run

Nothing is moved or removed. Every test runs where it did before.

🤖 Generated with Claude Code

https://claude.ai/code/session_013p5RirmgQs47AecZUsbSmd


Generated by Claude Code

vlt_e2e_harness_npx_shim_forwards_the_args_after_the_package wrote
two shell scripts with fs::write and exec'd them right away. When a
sibling test thread forks while that write fd is open, its child keeps
the fd until it execs, and Linux refuses the exec with ETXTBSY ("Text
file busy"). This hit both exec sites: the test's own spawn of the
npx shim and the shim's `exec` of the fake socket binary. The latter
can't be fixed by retrying the spawn.

Add write_executable, which stages the bytes next to the target and
has a child `cp` create the executable, so the test process never
holds a writable fd to it. Use it for the npx/vlt shims (also exec'd
by the real-vlt legs) and the fake binary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013p5RirmgQs47AecZUsbSmd
@mikolalysenko Mikola Lysenko (mikolalysenko) added the ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) label Oct 1, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 30fdbaf. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Oct 1, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[burn-down agent] Ready for review on 30fdbaf: mergeable, 0 commits behind main.

  • CI: 97/97 green (3 skipped by design).
  • Bugbot: reviewed 30fdbaf, no findings.
  • Reviewer focus: test-harness-only change in crates/socket-patch-cli/tests/vlt_e2e_common/mod.rs — the shim self-test no longer execs a script whose write fd could be inherited by a sibling thread's fork (ETXTBSY). No production code touched.

Generated by Claude Code

@mikolalysenko
Mikola Lysenko (mikolalysenko) merged commit e7f2a0b into main Oct 1, 2026
298 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the ci-janitor/vlt-shim-etxtbsy branch October 1, 2026 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants