ci: profile Windows Bazel contention with eight test jobs

[anp-oai]

Why

Rerun the Windows Bazel profiling experiment on top of the 8-local-test-job change, while keeping #31339 as the unchanged reference experiment linked from #31352.

What

• copy the commit-scoped test-cache bust, 15-second Windows host telemetry, and full Bazel profile upload from ci: measure Windows Bazel test bottlenecks #31339
• cherry-pick the focused --local_test_jobs=8 change from ci: increase Windows Bazel local test jobs #31352
• use this draft only to measure contention in the long-pole Windows test shards
• latest revision measures 8 GitHub-level Windows shards instead of 4

4 GitHub-shard findings

• forced-cache run passed; the slow shard was 638.0s versus 835.5s in the 4-local-job baseline and 609.3s in the first 8-local-job sample
• shard 1 critical path: core-all-test-windows-cross shard 5 took 504.0s, including 387.7s (77%) in Bazel local-test resource-lock waits; four core-all shards failed once and retried
• shard 2 critical path: utils/image took 110.7s with no resource-lock wait, so it is a real execution-time floor
• shard 3 critical path: app-server-all-test-windows-cross shard 8 took 271.9s, including 161.2s (59%) in resource-lock waits; one 15-second disk burst reached 1.48 GiB/s writes, 34.7ms latency, and queue 371.7
• shard 4 critical path: shell-command took 204.3s, including 117.9s (58%) in resource-lock waits; CPU briefly reached 99.8% with processor queue 268

8 GitHub-shard findings

• forced-cache run passed; the slow shard fell from 638.0s to 580.4s Bazel elapsed time and from 525.2s to 474.8s critical path, about a 9% improvement
• core-all-test-windows-cross remains one selected Bazel target label, so all 16 internal test shards still land on one GitHub runner; the critical shard took 453.0s, including 308.5s (68%) in resource-lock waits
• that target was flaky again, failing 3 of 19 attempts; one internal shard failed twice before passing
• the next-longest GitHub shard was 397.1s; its critical rmcp-client-streamable_http_recovery action took 189.6s, including 132.1s (70%) in resource-lock waits
• average CPU remained 31.7–44.2%; the slow shard briefly hit 99.9% CPU and processor queue 324, but disk, network, and memory did not show sustained saturation
• doubling GitHub runner count only trims the long pole about 9% because it cannot split the dominant Bazel target itself

Manual validation

• yq '.' .github/workflows/bazel.yml
• git diff --check origin/main...HEAD
• just test-github-scripts
• just fmt