ci: run CodeSpeed benchmarks on Nx agents#7763
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
View your CI Pipeline Execution ↗ for commit 88c7e40
☁️ Nx Cloud last updated this comment at |
🚀 Changeset Version PreviewNo changeset entries found. Merging this PR will not cause a version bump for any packages. |
Bundle Size Benchmarks
Current gzip tracks all emitted client JS chunks. Initial gzip tracks only the entry/import graph. Trend sparkline is historical current gzip ending with this PR measurement; lower is better. |
![nx-cloud[bot]](https://avatars.githubusercontent.com/in/80458?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Merging this PR will not alter performance
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | client-route-tree-scale navigation loop (react) |
74 ms | 77.3 ms | -4.36% |
| ⚡ | Simulation | client-loaders navigation loop (react) |
55.5 ms | 52.8 ms | +5.1% |
| ⚡ | Simulation | client-async-pipeline navigation loop (vue) |
45.9 ms | 44.4 ms | +3.23% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing codex/codespeed-nx-agents (f82011f) with main (0b178a7)
Footnotes
-
154 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Nx Cloud is proposing a fix for your failed CI:
We identified that setting WITH_INSTRUMENTATION=1 unconditionally activates @codspeed/vitest-plugin across all 15 Vitest sub-projects in the workspace-level config, causing each project to register a global teardown handler that the CodSpeed CLI binary doesn't deduplicate — resulting in Error: teardown called twice on the second teardown call. This fix restricts WITH_INSTRUMENTATION=1 to memory mode only, where valgrind instrumentation genuinely requires it, allowing simulation-mode benchmarks to complete successfully since the CodSpeed CLI detects benchmark results from vitest stdout directly.
Note
⏳ We are verifying this fix by re-running a subset of the 4 failed tasks that were analyzed.
Suggested Fix changes
diff --git a/scripts/nx/run-codspeed-benchmark.mjs b/scripts/nx/run-codspeed-benchmark.mjs
index d0cc888a..67c739c3 100644
--- a/scripts/nx/run-codspeed-benchmark.mjs
+++ b/scripts/nx/run-codspeed-benchmark.mjs
@@ -75,23 +75,35 @@ if (process.env.CODSPEED_TOKEN) {
codspeedArgs.push('--token', process.env.CODSPEED_TOKEN)
}
+// WITH_INSTRUMENTATION=1 activates @codspeed/vitest-plugin's global setup/teardown
+// in every Vitest project. The workspace-level vitest configs use multi-project mode
+// (main + many scenario sub-projects), so each project registers a teardown handler.
+// The CodSpeed action suppresses redundant teardown calls internally, but the CLI
+// binary does not. Restrict WITH_INSTRUMENTATION to memory mode (valgrind) only;
+// simulation mode detects benchmarks from vitest stdout without the plugin.
+const withInstrumentation = benchmarkConfig.mode === 'memory'
+
codspeedArgs.push(
'--',
- `NODE_ENV=production WITH_INSTRUMENTATION=1 vitest bench --config ./${vitestConfig}`,
+ `NODE_ENV=production${withInstrumentation ? ' WITH_INSTRUMENTATION=1' : ''} vitest bench --config ./${vitestConfig}`,
)
await execFile(codspeed, codspeedArgs, {
cwd: join(rootDir, benchmarkConfig.cwd),
env: {
...process.env,
- GH_MATRIX: process.env.GH_MATRIX ?? JSON.stringify({
- benchmark,
- framework,
- }),
- GH_STRATEGY: process.env.GH_STRATEGY ?? JSON.stringify({
- 'fail-fast': false,
- }),
- WITH_INSTRUMENTATION: '1',
+ GH_MATRIX:
+ process.env.GH_MATRIX ??
+ JSON.stringify({
+ benchmark,
+ framework,
+ }),
+ GH_STRATEGY:
+ process.env.GH_STRATEGY ??
+ JSON.stringify({
+ 'fail-fast': false,
+ }),
+ ...(withInstrumentation ? { WITH_INSTRUMENTATION: '1' } : {}),
},
})
@@ -103,7 +115,10 @@ async function ensureGithubEventPath() {
const eventDir = await mkdtemp(join(tmpdir(), 'codspeed-github-event-'))
const eventPath = join(eventDir, 'event.json')
const repository = process.env.GITHUB_REPOSITORY ?? ''
- const repositoryId = Number.parseInt(process.env.GITHUB_REPOSITORY_ID ?? '0', 10)
+ const repositoryId = Number.parseInt(
+ process.env.GITHUB_REPOSITORY_ID ?? '0',
+ 10,
+ )
const pullRequestNumber = getPullRequestNumber()
const event = {
repository: {
Or Apply changes locally with:
npx nx-cloud apply-locally PMPO-6Kdt
Apply fix locally with your editor ↗ View interactive diff ↗
🎓 Learn more about Self-Healing CI on nx.dev
1 participant
Summary
test:codspeed:*targets for client-nav, SSR, memory client, and memory server benchmarks, backed by ascripts/nx/run-codspeed-benchmark.mjsCLI wrapper.Impact
codspeed run -- pnpm nx ...scheduler call.test:codspeed:*targets tolinux-medium-jsagents withparallelism: 1to limit same-agent CPU and memory pollution.id-token: writeand forwards an explicit GitHub OIDC + repository metadata allowlist.--with-env-vars="auto"was tried and rejected because it also forwards runner-private GitHub file-command paths such asGITHUB_ENV,GITHUB_OUTPUT,GITHUB_PATH, andGITHUB_EVENT_PATH, which caused the agent workflow to fail before tasks executed.codspeed run, settingGH_MATRIX/GH_STRATEGYmetadata, and synthesizing the PR event payload with the real PR head/base SHAs instead of the pull-request merge SHA.@benchmarks/client-nav:test:codspeed:reactand@benchmarks/client-nav:test:codspeed:vue. The other CodeSpeed targets remain defined for manual/local iteration, butclient-navSolid currently needs agent build-output follow-up and SSR currently fails in CodeSpeed analysis mode on agents.Testing
node --check scripts/nx/run-codspeed-benchmark.mjspackage.jsonfiles with Node.git diff --checkcodspeedsmoke test confirmed wrapper argv and synthetic PR event payload contain the expected command string, matrix metadata, PR head SHA, and PR base SHA.48dcfcd3fbtried--with-env-vars="auto"; PR CI failed in Nx Cloud before any distributed tasks executed because the auto env set included runner-private GitHub file-command variables.e6ccd73de5restored the explicit env list; GitHub PR CI passed and the CodeSpeed Nx tasks ran successfully, but the external CodeSpeed check reported no detected benchmarks.88c7e404b8aligned wrapper metadata/command shape with the action; CodeSpeed then detected real benchmarks, while broad scheduling exposed agent-only failures in Solid client-nav and SSR.f82011f2b4narrows automatic PR scheduling to the v1 client-nav React/Vue targets and is awaiting CI.