Benchmark tracking: socket-patch scan

[mikolalysenko]

[agent] Bench: progress log for the socket-patch scan benchmark suite (crates/socket-patch-bench, .github/workflows/bench.yml). The daily benchmark steward rewrites this body on every run and posts a short comment for each run.

Last run

• When: 2026-10-02 ~16:50 UTC
• main: 1169ae68 (Fix Bun/vlt bundled copies left unpatched, Fix Bun/vlt bundled copies left unpatched (#469, #471) #472)
• Suite source: PR Add a scan benchmark suite and a CI performance gate #485 (claude/great-heisenberg-buor2s @ 08936bd7), merged locally with origin/main (clean merge)
• Runner: 4 vCPU, Intel(R) Xeon(R) Processor @ 2.10GHz, cloud sandbox. Absolute timings are for trend context only; verdicts come only from same-machine interleaved compare.
• Validation: 39/39 scenarios valid on today's main.
• Daily A/B: base 6e7ef748 (main 24h ago), 15 pairs + 10 confirm. Four confirmed regressions (bun/, vlt/, below), none elsewhere.
• Weekly A/B: WEEK_SHA 1bfe5326 (main 7 days ago) predates the v5 consolidation (v5 prerelease: scan → vex → vendor workflow, hosted by default #277) and fails validation on every scenario (redirect.redirected: got null), so the earliest comparable commit, 2463257a (v5 prerelease: scan → vex → vendor workflow, hosted by default #277 itself), was used. 9 pairs, bun/vlt excluded. No regressions. npm-family and poetry/pdm peak RSS drifted +5–7%, below the 15% gate.
• A/A sanity check (npm/hosted, vlt/hosted, poetry/hosted): clean, so the runner was not too noisy.

Scoreboard

Ratios are the median of paired head/base wall-time ratios. Day: vs 6e7ef748. Week: vs 2463257a (#277); bun/vlt were not run weekly.

scenario	median wall (ms)	ms/pkg	day	week	status
npm/hosted	198.6	0.0662	1.021	1.044	OK
npm/rescan	177.4	0.0591	1.033	1.059	OK
pnpm/hosted	153.6	0.0512	0.984	1.062	OK
pnpm/rescan	143.1	0.0477	0.944	0.964	OK
yarn-classic/hosted	179.9	0.0600	0.975	1.066	OK
yarn-classic/rescan	153.0	0.0510	1.048	1.044	OK
yarn-berry/hosted	219.5	0.0732	0.988	1.038	OK
yarn-berry/rescan	213.9	0.0713	0.998	1.032	OK
bun/hosted	289.7	0.0966	2.166	—	regressed (#578)
bun/rescan	294.5	0.0982	2.132	—	regressed (#578)
vlt/hosted	843.8	0.5625	2.178	—	regressed (#579), slow
vlt/rescan	778.7	0.5192	2.206	—	regressed (#579), slow
pip/hosted	102.2	0.1022	0.999	1.036	OK
pip/rescan	79.1	0.0791	1.001	1.120	OK
uv/hosted	136.0	0.3400	1.046	1.028	slow
uv/rescan	136.9	0.3422	1.003	0.976	slow
pylock/hosted	113.6	0.2840	1.056	1.029	slow
pylock/rescan	103.7	0.2591	1.062	0.986	slow
poetry/hosted	187.6	0.4691	0.998	1.050	slow
poetry/rescan	170.3	0.4259	1.024	1.009	slow
pipenv/hosted	106.2	0.2654	1.010	1.001	slow
pipenv/rescan	94.3	0.2358	1.007	1.041	slow
pdm/hosted	180.4	0.4510	1.067	1.032	slow
pdm/rescan	160.7	0.4018	1.016	1.023	slow
bundler/hosted	136.0	0.1700	1.020	1.001	OK
bundler/rescan	143.9	0.1798	1.000	1.006	OK
composer/hosted	122.2	0.1527	1.047	0.975	OK
composer/rescan	106.7	0.1334	0.942	1.077	OK
cargo/hosted	228.4	0.3806	1.044	1.005	slow
cargo/rescan	229.5	0.3825	0.991	0.997	slow
golang/hosted	68.5	0.0571	0.972	0.971	OK
golang/rescan	69.8	0.0581	1.010	1.003	OK
nuget/hosted	90.9	0.1010	1.001	0.994	OK
nuget/rescan	68.4	0.0760	0.987	0.982	OK
maven/hosted	79.3	0.0793	0.973	1.046	OK
maven/rescan	60.8	0.0608	0.977	1.029	OK
npm/dry-run	189.5	0.0632	1.010	1.007	OK
npm/public-proxy	244.8	0.0816	1.000	1.028	OK
npm/latency	664.9	0.2216	0.988	1.007	OK

Median ms/pkg across */hosted is 0.1275. A package manager is "slow" above 2x that, or with any scenario over 500 ms (npm/latency is excluded: its latency is simulated by design).

Open perf issues

• Perf regression: bun/hosted wall +110% (1169ae68, #472) #578: Perf regression: bun/hosted wall +110% (1169ae6, Fix Bun/vlt bundled copies left unpatched (#469, #471) #472). Confirmed. Hot spot: bun_lock_text::is_bundled_entry JSON-parses each entry once per patch.
• Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579: Perf regression: vlt/hosted wall +127% (1169ae6, Fix Bun/vlt bundled copies left unpatched (#469, #471) #472). Confirmed. Hot spot: the vlt_bundled::store_bundled_copies walk (72k readlink calls, 37k futex calls), run twice per scan.

Open bench refresh PR

None. The suite needed no changes this run.

Standing slow-systems list

Run 1 of 3 toward a slow-system issue. Note that ms/pkg flatters the small fixtures (400 pkgs) less than the large ones, because the roughly 60 ms fixed startup is spread over fewer packages.

pm	ms/pkg (x median)	hot spot (callgrind, head)
vlt	0.563 (4.4x), both scenarios >500 ms	#472's bundled-copy walk (#579), plus redirect::vlt::every_instance_pinned / partition_instances / vlt_preflight::preflight_scope re-splitting DepIds (split_dep_id, registry_identity), ~35% of instructions
poetry	0.469 (3.7x)	utils::poetry_lock::rewrite_poetry_lock_in, 77% of instructions
pdm	0.451 (3.5x)	utils::pdm_lock::rewrite_pdm_lock_in (via redirect::pdm::rewrite), 77%
cargo	0.381 (3.0x)	formats::cargo::CargoLock::parse inside rewrite_cargo, 79%. Looks like a per-patch re-parse of Cargo.lock
uv	0.340 (2.7x)	utils::python_lock::PythonLockSession::rewrite, 39%; vex pypi_locks::extract, 23%
pylock	0.284 (2.2x)	not profiled yet (same Python lock session as uv)
pipenv	0.266 (2.1x)	not profiled yet (borderline)

No scenario's peak RSS is above 2x the median (33.6 MiB).

Coverage-gap backlog

• bun.lockb (binary lockfile): the suite covers only text bun.lock. The hosted bun_binary rewriter and codec, both changed in Fix Bun/vlt bundled copies left unpatched (#469, #471) #472, are unbenchmarked.
• Bundled copies: no fixture has a Bun {"bundled": true} entry or a vlt bundled store copy, so the redirect_bun_bundled_instance_skipped / redirect_vlt_bundled_instance_skipped paths (Fix Bun/vlt bundled copies left unpatched (#469, #471) #472) are never hit. Timing is still exercised: the vlt walk runs regardless.
• hatch: hatch.toml is a HOSTED|PROBE pypi input in formats/registry.rs, but there's no pm:hatch scenario.
• gradle: Maven hosted with Gradle build scripts (presence-only) is not covered; pm:gradle exists.
• pnpm-workspace.yaml edits (Fix pnpm-workspace.yaml key detection and appends (#400, #402) #414): pnpm/hosted rewrites it, but only the fresh-file case is covered, not appending to an existing workspace file.
• Python install layouts: .egg-info installs (Fix Python crawler missing .egg-info installs (#447) #452), Poetry envs.toml venv selection (Fix Poetry venv selection ignoring envs.toml (#476, #526) #527) and lock-only requirements.txt (Fix lock-only requirements.txt discovery (#412, #523) #530) are uncovered.
• Gemfile .bundle/config resolution order (Fix Bundler settings resolution order (#483, #507) #532): the bundler fixture has .bundle/config, but only one layer.
• deno: no hosted rewrite, so no scenario is needed (pm:deno).

Stale / redundant scenarios

None identified on run 1. 14 or more days of history are needed before merging any rescan/hosted pairs.

Machine-readable history

[
 {
  "date": "2026-10-02",
  "main_sha": "1169ae68b9dbb974da4860593b8c66d0d8d75a36",
  "runner": "4 vCPU Intel Xeon @ 2.10GHz (cloud sandbox)",
  "slow": [
   "cargo",
   "pdm",
   "pipenv",
   "poetry",
   "pylock",
   "uv",
   "vlt"
  ],
  "scenarios": {
   "npm/hosted": {
    "median_ms": 198.6,
    "ms_per_pkg": 0.0662,
    "day_ratio": 1.021,
    "week_ratio": 1.044
   },
   "npm/rescan": {
    "median_ms": 177.4,
    "ms_per_pkg": 0.0591,
    "day_ratio": 1.033,
    "week_ratio": 1.059
   },
   "pnpm/hosted": {
    "median_ms": 153.6,
    "ms_per_pkg": 0.0512,
    "day_ratio": 0.984,
    "week_ratio": 1.062
   },
   "pnpm/rescan": {
    "median_ms": 143.1,
    "ms_per_pkg": 0.0477,
    "day_ratio": 0.944,
    "week_ratio": 0.964
   },
   "yarn-classic/hosted": {
    "median_ms": 179.9,
    "ms_per_pkg": 0.06,
    "day_ratio": 0.975,
    "week_ratio": 1.066
   },
   "yarn-classic/rescan": {
    "median_ms": 153.0,
    "ms_per_pkg": 0.051,
    "day_ratio": 1.048,
    "week_ratio": 1.044
   },
   "yarn-berry/hosted": {
    "median_ms": 219.5,
    "ms_per_pkg": 0.0732,
    "day_ratio": 0.988,
    "week_ratio": 1.038
   },
   "yarn-berry/rescan": {
    "median_ms": 213.9,
    "ms_per_pkg": 0.0713,
    "day_ratio": 0.998,
    "week_ratio": 1.032
   },
   "bun/hosted": {
    "median_ms": 289.7,
    "ms_per_pkg": 0.0966,
    "day_ratio": 2.166,
    "week_ratio": null
   },
   "bun/rescan": {
    "median_ms": 294.5,
    "ms_per_pkg": 0.0982,
    "day_ratio": 2.132,
    "week_ratio": null
   },
   "vlt/hosted": {
    "median_ms": 843.8,
    "ms_per_pkg": 0.5625,
    "day_ratio": 2.178,
    "week_ratio": null
   },
   "vlt/rescan": {
    "median_ms": 778.7,
    "ms_per_pkg": 0.5192,
    "day_ratio": 2.206,
    "week_ratio": null
   },
   "pip/hosted": {
    "median_ms": 102.2,
    "ms_per_pkg": 0.1022,
    "day_ratio": 0.999,
    "week_ratio": 1.036
   },
   "pip/rescan": {
    "median_ms": 79.1,
    "ms_per_pkg": 0.0791,
    "day_ratio": 1.001,
    "week_ratio": 1.12
   },
   "uv/hosted": {
    "median_ms": 136.0,
    "ms_per_pkg": 0.34,
    "day_ratio": 1.046,
    "week_ratio": 1.028
   },
   "uv/rescan": {
    "median_ms": 136.9,
    "ms_per_pkg": 0.3422,
    "day_ratio": 1.003,
    "week_ratio": 0.976
   },
   "pylock/hosted": {
    "median_ms": 113.6,
    "ms_per_pkg": 0.284,
    "day_ratio": 1.056,
    "week_ratio": 1.029
   },
   "pylock/rescan": {
    "median_ms": 103.7,
    "ms_per_pkg": 0.2591,
    "day_ratio": 1.062,
    "week_ratio": 0.986
   },
   "poetry/hosted": {
    "median_ms": 187.6,
    "ms_per_pkg": 0.4691,
    "day_ratio": 0.998,
    "week_ratio": 1.05
   },
   "poetry/rescan": {
    "median_ms": 170.3,
    "ms_per_pkg": 0.4259,
    "day_ratio": 1.024,
    "week_ratio": 1.009
   },
   "pipenv/hosted": {
    "median_ms": 106.2,
    "ms_per_pkg": 0.2654,
    "day_ratio": 1.01,
    "week_ratio": 1.001
   },
   "pipenv/rescan": {
    "median_ms": 94.3,
    "ms_per_pkg": 0.2358,
    "day_ratio": 1.007,
    "week_ratio": 1.041
   },
   "pdm/hosted": {
    "median_ms": 180.4,
    "ms_per_pkg": 0.451,
    "day_ratio": 1.067,
    "week_ratio": 1.032
   },
   "pdm/rescan": {
    "median_ms": 160.7,
    "ms_per_pkg": 0.4018,
    "day_ratio": 1.016,
    "week_ratio": 1.023
   },
   "bundler/hosted": {
    "median_ms": 136.0,
    "ms_per_pkg": 0.17,
    "day_ratio": 1.02,
    "week_ratio": 1.001
   },
   "bundler/rescan": {
    "median_ms": 143.9,
    "ms_per_pkg": 0.1798,
    "day_ratio": 1.0,
    "week_ratio": 1.006
   },
   "composer/hosted": {
    "median_ms": 122.2,
    "ms_per_pkg": 0.1527,
    "day_ratio": 1.047,
    "week_ratio": 0.975
   },
   "composer/rescan": {
    "median_ms": 106.7,
    "ms_per_pkg": 0.1334,
    "day_ratio": 0.942,
    "week_ratio": 1.077
   },
   "cargo/hosted": {
    "median_ms": 228.4,
    "ms_per_pkg": 0.3806,
    "day_ratio": 1.044,
    "week_ratio": 1.005
   },
   "cargo/rescan": {
    "median_ms": 229.5,
    "ms_per_pkg": 0.3825,
    "day_ratio": 0.991,
    "week_ratio": 0.997
   },
   "golang/hosted": {
    "median_ms": 68.5,
    "ms_per_pkg": 0.0571,
    "day_ratio": 0.972,
    "week_ratio": 0.971
   },
   "golang/rescan": {
    "median_ms": 69.8,
    "ms_per_pkg": 0.0581,
    "day_ratio": 1.01,
    "week_ratio": 1.003
   },
   "nuget/hosted": {
    "median_ms": 90.9,
    "ms_per_pkg": 0.101,
    "day_ratio": 1.001,
    "week_ratio": 0.994
   },
   "nuget/rescan": {
    "median_ms": 68.4,
    "ms_per_pkg": 0.076,
    "day_ratio": 0.987,
    "week_ratio": 0.982
   },
   "maven/hosted": {
    "median_ms": 79.3,
    "ms_per_pkg": 0.0793,
    "day_ratio": 0.973,
    "week_ratio": 1.046
   },
   "maven/rescan": {
    "median_ms": 60.8,
    "ms_per_pkg": 0.0608,
    "day_ratio": 0.977,
    "week_ratio": 1.029
   },
   "npm/dry-run": {
    "median_ms": 189.5,
    "ms_per_pkg": 0.0632,
    "day_ratio": 1.01,
    "week_ratio": 1.007
   },
   "npm/public-proxy": {
    "median_ms": 244.8,
    "ms_per_pkg": 0.0816,
    "day_ratio": 1.0,
    "week_ratio": 1.028
   },
   "npm/latency": {
    "median_ms": 664.9,
    "ms_per_pkg": 0.2216,
    "day_ratio": 0.988,
    "week_ratio": 1.007
   }
  }
 }
]

---

Generated by Claude Code

[mikolalysenko]

[agent] Bench: run 2026-10-02, main 1169ae68. This is the first run, so this issue was created today.

• All 39 scenarios validated on today's main (suite from Add a scan benchmark suite and a CI performance gate #485, merged locally with main).
• Confirmed regressions from Fix Bun/vlt bundled copies left unpatched (#469, #471) #472 (1169ae6), with a clean A/A check:
  • bun/* is about 2.1x slower: bun/hosted 144 → 302 ms, +117% [+104, +130]. Issue Perf regression: bun/hosted wall +110% (1169ae68, #472) #578.
  • vlt/* is about 2.2–2.4x slower: vlt/hosted 372 → 803 ms, +118% [+110, +126]. Issue Perf regression: vlt/hosted wall +127% (1169ae68, #472) #579.
  • Bisected against the parent, cbf1f748.
• Weekly A/B against v5 prerelease: scan → vex → vendor workflow, hosted by default #277 (2463257a): no other regressions. The original 7-day SHA predates the v5 output format and can't be compared.
• Slow list (run 1 of 3): vlt, poetry, pdm, cargo, uv, pylock, pipenv. The body names each hot spot.
• Suite changes: none, so no PR. The coverage-gap backlog is seeded in the body (bun.lockb, bundled copies, hatch, gradle).

---

Generated by Claude Code