Summary
In periodic.yml, the two jobs
test-models-linux-basic (cmake, dl3, portable, linux.4xlarge.memory, 90)test-models-linux-basic (cmake, edsr, portable, linux.2xlarge, 90)
have hit the 90 minute job limit on every periodic run since at least 2026-08-03. They always report cancelled, never pass and never fail. Periodic runs at 45 0,8,16 * * *, so this is roughly nine runner hours per day with zero signal, and it is also why the whole scheduled run concludes as cancelled.
Evidence
In a recent periodic run, these two were the only non-successes out of 59 jobs. The xnnpack variants of the same two models finished in about ten minutes each. The dl3 log shows an 81 minute silent gap after the runner starts, together with swap pressure warnings, so simply raising the timeout is not the answer.
Proposed short term action
Stop gathering the portable variants of dl3 and edsr so periodic gives a clean result. The xnnpack variants still cover both models.
What still needs an owner
Why portable export of dl3 and edsr became this slow, and whether it is a genuine performance regression in the portable path rather than just a big model. Once that is answered the skip should be removed.
Summary
In
periodic.yml, the two jobstest-models-linux-basic (cmake, dl3, portable, linux.4xlarge.memory, 90)test-models-linux-basic (cmake, edsr, portable, linux.2xlarge, 90)have hit the 90 minute job limit on every periodic run since at least 2026-08-03. They always report
cancelled, never pass and never fail. Periodic runs at45 0,8,16 * * *, so this is roughly nine runner hours per day with zero signal, and it is also why the whole scheduled run concludes ascancelled.Evidence
In a recent periodic run, these two were the only non-successes out of 59 jobs. The
xnnpackvariants of the same two models finished in about ten minutes each. The dl3 log shows an 81 minute silent gap after the runner starts, together with swap pressure warnings, so simply raising the timeout is not the answer.Proposed short term action
Stop gathering the
portablevariants ofdl3andedsrso periodic gives a clean result. Thexnnpackvariants still cover both models.What still needs an owner
Why portable export of
dl3andedsrbecame this slow, and whether it is a genuine performance regression in the portable path rather than just a big model. Once that is answered the skip should be removed.