fix(ci): stop weekly-refresh dying at the 6h ceiling before it dumps - #59
Merged
Conversation
Every run since at least 2026-08-03 has ended at 6:00:2x, always in "Enrich benchmarks (all sources)". GitHub reports a job that hits its 6h ceiling as *cancelled* rather than failed, so nothing ever appeared in the failed-run list while validate, integrity, the static dump and the refresh PR were skipped every single week. The cost is not theoretical: the published dump has drifted away from data/ because the dump step never ran. 565 of 2,030 gpu pages, and 93 more across soc/watch/pda/brand, currently advertise a `verified` value the record no longer has. Give collection a 200-minute budget it checks between sources, so the remainder of the pipeline always gets to run, and rotate the starting point by ISO week so sources at the tail of the list are not starved once the budget starts biting. Add a 330-minute job timeout so this can never again be silently truncated by the platform ceiling. Refs #1
Seungpyo1007
added a commit
that referenced
this pull request
Sep 11, 2026
…vers #59 got the scrape to finish, and the run then died in the dump instead: Enrich 06:13 -> 10:33 260 min (on a 200-minute budget) Dump 10:39 -> 11:36 57 min, cut by the job timeout total 330 min Two things were wrong. The budget is only consulted *between* sources, so one long source overran it by an hour; each source is now capped at 40 minutes by `timeout`, and the budget drops to 150 so the rest of the pipeline keeps real room. But the dump does not belong behind a multi-hour live scrape at all. Whenever the scrape runs long the dump is what gets dropped, which is how the published pages drifted away from the records for six weeks. dump-refresh.yml already regenerates the dump from current data with no scraping — and when it last ran, the dump step succeeded; only the PR creation failed on a transient server error. Give it a weekly schedule (Tuesdays, a day after the Monday jobs) so the published dump is reconciled with data/ every week regardless of how the scrape goes, and a job timeout so it can never be silently truncated either. Refs #1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is broken
Every
weekly-refreshrun ends at 6:00:2x — the GitHub job ceiling — always insideEnrich benchmarks (all sources):A job that hits the ceiling is reported as cancelled, not failed, so it never showed up in the failed-run list. Meanwhile every step after the scrape was skipped:
What it cost
The static dump has been drifting away from
data/for at least six weeks, because the only job that regenerates it never reached that step. Measured against the currentdevelop:verifieddisagrees with the recordThe published API is advertising a verification state the data no longer has — 28% of GPU pages. (Promotion PRs flip
verifiedindata/and do not touchsite/public/v1; the weekly dump is what was supposed to reconcile them.)Change
topcpu-cpu,topcpu-gpu) would be starved forever once the budget bites. Verified: week 37 starts atgeekbench-nbcand wraps.timeout-minutes: 330on the job, so the platform ceiling can never silently truncate it again — it fails loudly instead.bash -non the rendered step body passes; the YAML parses.Refs #1