Flaky: DefaultServerTests.DeleteQueuedJobPreventsRun / DeleteRunningJobCancelsWorker in GRPC_INTEGRATION_TEST

[ramakrishnap-nv]

Summary

DefaultServerTests.DeleteQueuedJobPreventsRun and DefaultServerTests.DeleteRunningJobCancelsWorker (cpp/tests/linear_programming/grpc/grpc_integration_test.cpp) flaked together on the same CI run, on two different platforms, while unrelated to the PR's changes — a rerun of the identical commit passed both tests cleanly.

Evidence

PR #1809, run 33015105401:

• conda-cpp-tests / 13.3.0, 3.13, arm64, ubuntu26.04, l4, latest-driver, latest-deps — job 98341385500
• conda-cpp-tests / 13.3.0, 3.14, amd64, ubuntu26.04, h100, latest-driver, latest-deps — job 98341385424

Both jobs failed with the identical pair of tests:

grpc_integration_test.cpp:1206: Failure
Expected equality of these values:
  probe_status.status
    Which is: 4-byte object <00-00 00-00>
  job_status_t::COMPLETED
    Which is: 4-byte object <02-00 00-00>
Worker should be free to process a new job after the queued job was deleted
[  FAILED  ] DefaultServerTests.DeleteQueuedJobPreventsRun (62309 ms)

grpc_integration_test.cpp:1242: Failure
Value of: processing
  Actual: false
Expected: true
Job never reached PROCESSING before delete
[  FAILED  ] DefaultServerTests.DeleteRunningJobCancelsWorker (10058 ms)

On the prior CI attempt for the same PR (run 32995142698, same underlying test binary, only a small unrelated header/visibility change between the two commits), both tests passed in ~13s and ~11s respectively. The failing run took 62s and 10s — a >4x duration blowup, consistent with worker startup/claim contention rather than a logic bug.

What the failure means

• DeleteQueuedJobPreventsRun (grpc_integration_test.cpp:1190-1208) submits a probe MIP job after cancelling+deleting a running one, waits up to 60s for it, and expects COMPLETED — it got back a default/zero status, meaning check_status never saw the probe reach COMPLETED in that window.
• DeleteRunningJobCancelsWorker (grpc_integration_test.cpp:1211-1242) polls up to 40×250ms = 10s for a submitted job to reach PROCESSING before proceeding — the worker never claimed it within that window.

Both point at the same underlying sensitivity: worker claim/startup latency under CI load. This looks related in spirit to #1716 (test_mip_incumbent_stream flaking because a worker replacement/claim after a preceding cancel didn't happen fast enough), though these are different (C++ gtest, not Python) tests with their own timeouts.

Suggested next step

Look at whether these two tests, run back-to-back in DefaultServerTests (24 tests in the suite, ~88s total in the failing run), are contending for the same worker pool/GPU init path as tests before them, and whether the polling windows (60s / 10s) are too tight for busy/oversubscribed CI runners — similar to the timeout-tuning history visible in cpp/tests/mip/determinism_test.cu's git log (e44239b2, 64d58f29).

Environment

• Self-hosted CI runners (arm64 l4, amd64 h100), latest-driver, latest-deps.
• Not reproduced locally; only observed on CI so far.