Skip to content

v5 CI: build e2e binaries once and tier the PM matrix - #291

Merged
Mikola Lysenko (mikolalysenko) merged 11 commits into
release/v5-prereleasefrom
v5/ci-build-once
Sep 28, 2026
Merged

Mikola Lysenko (mikolalysenko) merged 11 commits into
release/v5-prereleasefrom
v5/ci-build-once

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

v5 waste review follow-up W1 (#286). This PR changes workflows and CI scripts only, plus un-ignoring 4 hermetic tests and porting one test fix (cc5f1b6).

Findings

ID Outcome What changed
F51 done A new e2e-build job compiles the CLI and every CLI test target once per OS (--all-features). scripts/ci-e2e-bundle.py bundles the binaries the rows need, and the legs run them directly from the same checkout path with CARGO_MANIFEST_DIR and cargo's [env] set as cargo test would. The legs no longer compile or use rust-cache.
F49 done test and test-release compile one feature set. The feature-gated suites' compile-rot check moves to e2e-build, on 3 OSes.
F52 done 31 e2e rows with no named boundary move to e2e-full: bundler 2.7.2, uv 0.2–0.11 plus 0.5.3/0.5.6, Pipenv 2023–2025, and .NET 7–9 on ubuntu. yarn-berry 4.6.0 and ubuntu 4.12.0 move to yarn-berry-full, and 10 cargo toolchain × lock cells to cargo-vex-matrix-full. These run on main pushes, pushes to release/v5-prerelease, a new nightly schedule (with its own concurrency group) and dispatch. PRs keep every named boundary, the oldest and newest release of each tool, and every vlt row. Every PM version stays.
F41 done e2e-docker runs nightly, on dispatch and on v5 pushes. coverage-docker runs the same 9 suites (plus vendor_gem and vendor_maven) on every PR.
F53 done A new docker-base job builds Dockerfile.base once and hands it to the docker legs with docker save/load.
F46 done The 4 hermetic maven/nuget crawl tests are un-ignored, and test runs them on 3 OSes; they pass on Windows. The e2e_maven, e2e_nuget and e2e_composer rows are gone (none of the three has an #[ignore]d test left). #279's removal of the vacuous e2e_cargo / e2e_golang rows is kept.
F47 done The 3 e2e_safety_cargo_build rows are gone; every cargo-vex leg runs that suite. PRs run a covering subset: every toolchain, every lock, and 1.93.1 with its own lock on 3 OSes. The rest of the cross is in -full. The binaries come from e2e-build.
F55 done pdm-compat builds e2e_vex_build once, and the capstone skips the 7 cells ci.yml runs; test_ci_e2e_tiers.py keeps the two lists equal.
F54 done backtest-pdm.py turns a bootstrap failure into ERROR, and a cell whose rows all skip exits 1. The 16 Windows native rows are dropped because they always skipped (POSIX venv layout). Native Windows PDM stays untested, as it was before; porting the harness would add ~50–60 job-min.
F56 done vlt-compat install-proof skips any suite whose cell ci.yml's e2e runs identically (scripts/ci-vlt-proof-suites.py, tested); a dispatch runs every cell. It is verified in run 36419561398: the ubuntu rc.32 cell logs 3 suites handed to ci.yml.
F57 done (partial) npm- and pnpm-compatibility are path-filtered on PRs to crate sources plus the test files their capstones compile; main pushes stay unfiltered. The watchdog stays disarmed because production still fails the probe (run 36389492734). It runs daily until then.
F40 done by #279 #279 removed setup and the setup-matrix job, and this PR now includes that change through the merge from the base.

Measured savings

Baseline, run 36359489402 Full tier, run 36423647381
e2e 176 legs, 530 job-min 140 legs, 104.5 + e2e-full 30 legs, 15.3 + e2e-build 20.8 = 140.6
cargo-vex 17 legs, ~45 19 legs, 12.1
coverage-docker 89.2 48.4 (+ docker-base 4.9)
e2e-docker 65.1 27.6 (nightly only)
test ×3 74.2 69.1
test-release 29.3 24.8
Run total 945 job-min 399 job-min (−58%), with every tier and e2e-docker
  • Run 36423647381 is a dispatch of the full tier on the throwaway branch v5/ci-build-once-proof, which drops only the needs: test gate. The proxy refused to let me delete that branch; please delete it.
  • On a PR the full tier and e2e-docker are skipped, so a PR run is about 350 job-min against the 945 baseline.

Equivalence

  • Rows:
    • e2e: 176 rows before; 139 on PRs + 31 in e2e-full = 170 after, with no new or changed rows. The other 6 moved: e2e_maven, e2e_nuget and e2e_composer into test, and e2e_safety_cargo_build ×3 into every cargo-vex leg. (Since then, v5: remove setup (WS7) + patch UI streamlining (WS8) #279 also dropped the vacuous e2e_cargo and e2e_golang rows on the base.)
    • cargo-vex: 17 cells before; 9 + 10 = 19 after (2 added for the macOS/Windows pinned-toolchain safety runs).
    • yarn-berry: 7 = 5 + 2. pdm capstone: 30 = 23 + the 7 in ci.yml.
  • Failure modes: every SOCKET_PATCH_*_E2E_REQUIRED/_VERSION gate is unchanged. vlt legs still go through check-vlt-legs.py, with --binary. Exit status reaches the step through pipefail.
  • Fan-out works: in run 36423647381, 127 of 140 e2e legs, 17 of 30 e2e-full legs and all 19 cargo-vex legs pass from the prebuilt binaries on ubuntu, macOS and Windows.
  • Current head 3e1f36d (merges base 14a9cb0, v5 WS4/WS6: one hosted engine for disk + memory; unified Ledgers view #282):
    • CI run 36486018407: 186 jobs. 178 pass, 4 are skipped (tiers that don't run on PRs), and the 4 failures are the yarn-classic releases below.
    • npm, pnpm and PDM compatibility pass.
    • vlt-compat run 36486018148 fails only the 4 install-proof cells below. On its first attempt native (windows-latest, 1.2.0) also failed 5 of 36 production-dependent cases; it passed on the one re-run, as it did on v5 WS4/WS6: one hosted engine for disk + memory; unified Ledgers view #282's own head.
  • Checks: python3 -m unittest discover -s scripts/tests passes, including the new test_ci_e2e_tiers.py, and cargo build --workspace --tests --all-features passes after the v5 WS4/WS6: one hosted engine for disk + memory; unified Ledgers view #282 merge. actionlint reports only "property not defined" on the steps e2e-full shares through the YAML alias; a missing matrix key is null at runtime, as it is for e2e rows today.

Base-inherited failures

Needs an owner check

  • Branch protection: these check names disappear on PRs: the moved e2e (…) rows, e2e-docker (<eco>) ×9 (now one skipped check), 10 cargo cells, 2 yarn-berry cells, and the pdm Windows and excluded cells. If any of them are required checks, PRs will wedge.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HK6zFsizxHRUtyxPvdhM4c

The e2e matrix recompiled the CLI and its test binary in each of its
176 legs. e2e-build now compiles every CLI test target once per OS and
the legs run the downloaded binaries from the same checkout path. That
compile also replaces the --all-features --no-run pass in test and
test-release, so each of those compiles one feature set.

PR runs keep every named version boundary, the oldest and newest
release of each tool and every vlt era. The 31 middle e2e rows, 2
yarn-berry releases and 10 cargo toolchain x lock cells move to *-full
jobs that run on main pushes, a new nightly schedule and dispatch.

- e2e-docker (a subset of coverage-docker) runs nightly only.
- Dockerfile.base is built once per run and loaded by each docker leg.
- The hermetic maven/nuget crawl tests run in `test`; their rows and
  e2e_composer's are gone. e2e_safety_cargo_build rides cargo-vex.
- pdm-compat builds the capstone once, skips the 7 cells ci.yml runs,
  drops the Windows native rows (they never ran), and fails a cell
  whose bootstrap fails or whose rows all skip.
- vlt-compat install-proof leaves ci.yml's identical cells to it.
- npm/pnpm compatibility are path-filtered on PRs; the disarmed vlt
  serve watchdog runs daily instead of every 6 hours.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HK6zFsizxHRUtyxPvdhM4c
- The legs that run test binaries directly set SOCKET_NO_CONFIG and
  SOCKET_NO_UPDATE_CHECK, which cargo's [env] gave `cargo test`.
- cargo 1.93.1 with its own lock (the pinned toolchain the removed
  e2e_safety_cargo_build rows ran) stays on PRs; 1.82.0 own-lock moves
  to the full tier.
- Poetry 2.0.1, the first lock 2.1 writer, stays on PRs.
- e2e-build gets 60 minutes on Windows.
- npm/pnpm filters also watch .cargo/config.toml and cache_env.rs.
- vlt install-proof notes a cell left entirely to ci.yml.
- pdm-compat saves its build cache from main only.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HK6zFsizxHRUtyxPvdhM4c
- ci.yml also runs on pushes to release/v5-prerelease, whose PRs skip
  the full tier and which has no nightly; e2e-docker runs there too.
- cargo 1.93.1 with its own lock runs on macOS and Windows on PRs,
  the cell the old e2e_safety_cargo_build rows ran there.
- e2e-build and pdm-compat print rendered compile errors
  (json-render-diagnostics).
- vlt-compat and pdm-compat also trigger on ci.yml changes, and the
  vlt dedupe only counts ci rows with the same test filter.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HK6zFsizxHRUtyxPvdhM4c
get now defaults to hosted mode (5e5f5ed), so the real-vlt
get_and_remove leg ran a hosted get and found the installed copy
unpatched. It now passes --mode agent, like the other agent-mode
fixtures that commit updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R11RZRvYFL3fzmEnkFAU4A
(cherry picked from commit cc5f1b6)
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] vlt-compatibility install-proof is red, but the failure is not this PR's.


Generated by Claude Code

@mikolalysenko

Mikola Lysenko (mikolalysenko) commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator Author

[agent] vlt-compatibility install-proof is still red on 4d0ced4, now from a second failure that is also not this PR's.


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] test (windows-latest) failed in the full-tier dispatch run 36419098151. The failure is not this PR's.

  • What fails: the single failing target is -p socket-patch-cli --test covgap_commands_scan_mod.
  • Base has it too: the same target fails the same way in v5 fix: unblock the e2e tier on the v5 base #288's CI run 36416569293 (job 108909006900), which changes only concurrency lines on the same base.
  • Tests this PR un-ignored: they pass on Windows in that job. e2e_maven and e2e_nuget now run there for the first time.
  • Fix: none exists on an open v5 branch. v5: remove setup (WS7) + patch UI streamlining (WS8) #279 rewrites this file for WS7/WS8, but I cannot tell from here whether that fixes it.
  • Next step: it is Windows-only test code outside this CI-only PR's scope, so I am leaving it and not re-running it; it is deterministic on both runs.

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

#283 landed on release/v5-prerelease as 06437d2; please merge origin/release/v5-prerelease again, resolve conflicts, get green, and keep it ready.


Generated by Claude Code

@mikolalysenko
Mikola Lysenko (mikolalysenko) marked this pull request as ready for review September 28, 2026 13:22
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] ready to land


Generated by Claude Code

A concurrency group holds one pending run. Sharing main's group let a
queued main push and the nightly cancel each other, and the nightly is
the only automatic run of e2e-docker.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HK6zFsizxHRUtyxPvdhM4c
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] ready to land


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] ready to land

  • Head: 73e4c6b, which merges release/v5-prerelease at 28cebf7. That commit ("Attribute vlt pins and uv overrides in ledger-free hosted rollback") targets the v5 WS1+WS2: ledger-free hosted mode, upstream-restore rollback, vendor ejects hosted projects #280 hosted uv/vlt refusals.
  • Run on the previous head (36a9e82): run 36452450178, 199 jobs: 183 passed, 4 skipped, 12 failed. This is the first PR run where the real needs: test gate let the e2e tier through.
    • 8 of the failures are the hosted uv/vlt refusals that 28cebf7 should fix.
    • The other 4 are yarn-classic 1.0.2 / 1.6.0 / 1.7.0 / 1.9.4. In each, the mode_migration_npm suite fails; every other cell of the script passes, and so do 1.10.1 and 1.22.22.
  • Why the yarn-classic failures are not this PR's: this PR does not change the yarn-classic-matrix job or scripts/yarn-classic-vex-matrix.sh. That job had never run on the v5 base, because the red Windows test job skipped everything behind it until v5 fix: unblock the e2e tier on the v5 base #288. The failing test's details go to the script's temp log, which the job does not upload, so I cannot name the assertion from here.
  • Next step for yarn-classic: no fix exists on any branch yet. It needs a look at mode_migration_npm under yarn classic before 1.10 in the ledger-free hosted base.

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] hosted-e2e failed on 73e4c6b (job 109055280122). The failure is not this PR's: patches-api.socket.dev was unreachable while the job ran.

  • What fails: all 3 attempts. The failing tests differ from one attempt to the next, and nearly every panic is error sending request for url (https://patches-api.socket.dev/patch/...). Affected: preflight_required_patches_are_published, both canaries, and the gem/yarn install proofs, which cannot fetch patch/view.
  • Why it is not this PR's: this PR does not change the hosted-e2e job or its suite. The job passed on the base's own run at 28cebf7 (36454564172), which has the same code as this head, and on v5: remove setup (WS7) + patch UI streamlining (WS8) #279 and v5 WS4/WS6: one hosted engine for disk + memory; unified Ledgers view #282 runs 15–20 minutes earlier.
  • Next step: I'll re-run this job once, after the run finishes.
  • yarn-classic confirmed base-inherited: the base's run on 28cebf7 (36454564172) fails the same 4 yarn-classic releases. Its hosted uv/vlt e2e legs pass there, so 28cebf7 fixed those.

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] Status on 73e4c6b, which merges base 28cebf7:

  • vlt-compat install-proof, ubuntu 0.0.0-1 / ubuntu 0.0.0-11 / windows 0.0.0-11 (run). This is inherited from the base, not caused by this PR.
    • Three e2e_redirect_vlt_build tests fail: vlt_pinned_matrix_hosted_crlf_lock, _idempotence and _rollback_byte_exact. In each one, rollback leaves the lock's tarball-URL field changed instead of restoring it byte-for-byte.
    • The base's own dispatch run on 28cebf7 (36454568301) fails the same cells, plus ubuntu 0.0.0-1/node 22.
    • This PR only changes which install-proof suites run: the ci.yml dedupe. It does not change the rollback code.
    • I don't know of a fix yet. It belongs with whoever owns the hosted rollback path (28cebf7 fixed the hosted_wiring_contested half; this is the remaining LV0 half).
  • uv and mode_migration_vlt legs: they now pass after 28cebf7.
  • yarn-classic 1.0.2 / 1.6.0 / 1.7.0 / 1.9.4 mode_migration_npm: still inherited from the base, as noted earlier (base run 36454564172).
  • hosted-e2e: it could not reach patches-api.socket.dev. I'll re-run it once when the CI run finishes.

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

#281 landed on release/v5-prerelease as 73c0c4f; please merge origin/release/v5-prerelease again, resolve conflicts, get green, and keep it ready.


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] ready to land


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

#279 landed on release/v5-prerelease as f6bdad5; please merge origin/release/v5-prerelease again, resolve conflicts, get green, and keep it ready.


Generated by Claude Code

…i-build-once

# Conflicts:
#	.github/workflows/ci.yml
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] ready to land


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

#282 landed on release/v5-prerelease as 14a9cb0; please merge origin/release/v5-prerelease again, resolve conflicts, get green, and keep it ready.


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] I merged release/v5-prerelease at 14a9cb0 (#282) into the branch as 3e1f36d. The merge had no conflicts.

  • v5 WS4/WS6: one hosted engine for disk + memory; unified Ledgers view #282 changes only crate sources, Cargo.toml/Cargo.lock, the CHANGELOG and CLI_CONTRACT.md. It doesn't touch workflows, CI scripts or test targets, so the e2e-build bundle and the tier lists don't change.
  • The Python CI checks pass (118 tests), and cargo build --workspace --tests --all-features passes.
  • CI on 3e1f36d is running. I'll update the PR description and post [agent] ready to land once it finishes, provided the only failures are the vlt-compat and yarn-classic ones that also fail on the base.

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] native (windows-latest, 1.2.0) in vlt-compat fails on 3e1f36d (job 109145277588). I don't think this PR caused it:

  • Result: 31 of 36 cases are as expected. The five that aren't are scattered and don't share a cause:
    • hosted-alias and hosted-lockfile-only end in safe-refusal, failing serveEncodingIdentity and cleanRefusalWhileEncoded.
    • vendored-optional-mixed and vendored-get-uuid fail repair.
    • agent-workspace fails rollbackByteIdentical.
    • Ten other cases landed on blocked-by-server-encoding, which depends on how the production service is serving right now.
  • It passed with the same code: it passed on v5 WS4/WS6: one hosted engine for disk + memory; unified Ledgers view #282's own head 24a8241 (run 36474482017) and on this PR's previous head 385162a. 3e1f36d builds the same crate sources as v5 WS4/WS6: one hosted engine for disk + memory; unified Ledgers view #282's head, since this PR changes no crate source apart from the two un-ignored test files.
  • This PR doesn't touch the native job: its only changes to vlt-compatibility.yml are path filters and the install-proof suite skip.

No fix exists because nothing points at a code change. I'll re-run the failed jobs once this run finishes. That is the one re-run to confirm it. If it fails the same way again, I'll treat it as real and dig into it.


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] ready to land

  • The v5 WS4/WS6: one hosted engine for disk + memory; unified Ledgers view #282 merge is in as 3e1f36d, and CI on it has finished. The only failures are the ones that also fail on the base (see "Base-inherited failures" in the PR description):
    • CI run 36486018407: 178 of 186 jobs pass, 4 are skipped by tier, and the 4 failures are yarn-classic 1.0.2, 1.6.0, 1.7.0 and 1.9.4.
    • npm, pnpm and PDM compatibility pass.
    • vlt-compat fails only the 4 install-proof cells on vlt 0.0.0-1 and 0.0.0-11.
  • native (windows-latest, 1.2.0) passed on the one re-run (attempt 2), so it goes down as nondeterministic, as I said above. Nothing else is needed there.
  • I updated the PR description's CI numbers to 3e1f36d.

Generated by Claude Code

@mikolalysenko
Mikola Lysenko (mikolalysenko) merged commit f9cb7e1 into release/v5-prerelease Sep 28, 2026
437 of 450 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the v5/ci-build-once branch September 28, 2026 23:18
Mikola Lysenko (mikolalysenko) pushed a commit that referenced this pull request Sep 28, 2026
Brings in #291, which builds the e2e test binaries once and tiers the
package-manager matrix. The only conflict is the `test-release`
timeout: it takes the base's 40 minutes. The 50 minutes here was sized
for the two feature-set builds that #291 removes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BKsyzefGhAnPkYmXCwq3H3
Mikola Lysenko (mikolalysenko) pushed a commit that referenced this pull request Sep 28, 2026
Brings in #291, which builds the e2e binaries once per OS and tiers
the package-manager matrix.

The release test job keeps the base's 40-minute limit. This branch
raised it to 50 because the job used to build the test graph twice;
it now builds once, so the extra time is no longer needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AvGSu5jGg1z3f3sePc2eHC
Mikola Lysenko (mikolalysenko) pushed a commit that referenced this pull request Sep 28, 2026
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N9YdJqaGiT9Jf5LN1hDFhB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants