fix(ci): give the dump its own schedule instead of the scrape's leftovers - #60
Merged
Conversation
…vers #59 got the scrape to finish, and the run then died in the dump instead: Enrich 06:13 -> 10:33 260 min (on a 200-minute budget) Dump 10:39 -> 11:36 57 min, cut by the job timeout total 330 min Two things were wrong. The budget is only consulted *between* sources, so one long source overran it by an hour; each source is now capped at 40 minutes by `timeout`, and the budget drops to 150 so the rest of the pipeline keeps real room. But the dump does not belong behind a multi-hour live scrape at all. Whenever the scrape runs long the dump is what gets dropped, which is how the published pages drifted away from the records for six weeks. dump-refresh.yml already regenerates the dump from current data with no scraping — and when it last ran, the dump step succeeded; only the PR creation failed on a transient server error. Give it a weekly schedule (Tuesdays, a day after the Monday jobs) so the published dump is reconciled with data/ every week regardless of how the scrape goes, and a job timeout so it can never be silently truncated either. Refs #1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #59, which fixed the wrong half of the problem.
#59 got the scrape to finish inside its budget — for the first time in six weeks the run reached
Validate,Integrity checkandGenerate static dump. It then died in the dump:Two separate faults
1. The budget did not bind. It is only consulted between sources, so a source that starts at minute 199 runs as long as it likes — 260 minutes against a 200-minute budget. Each source is now capped with
timeout(40 min), and the budget drops to 150 so the remainder of the pipeline keeps real room.2. The dump should not be queued behind a live scrape at all. Whenever the scrape runs long, the dump is the step that gets dropped — that is exactly how the published pages drifted from the records (565 of 2,030 gpu pages advertise a stale
verified, plus 93 across soc/watch/pda/brand).dump-refresh.ymlalready exists for this: it regenerates the dump from current data with no scraping. When it last ran (2026-06-24) the dump step succeeded — onlyCreate dump-refresh PRfailed, on a transient##[error]Server Error. It has been dispatch-only ever since, so nothing ran it.It now runs Tuesdays 03:00 UTC, a day after the Monday scrape and ingest, so the published dump is reconciled with
data/every week no matter how the scrape goes. Both workflows also carrytimeout-minutes: 330, because a job that hits GitHub's 6h ceiling is reported as cancelled, not failed, and never shows up as broken.YAML parses;
bash -non the rendered enrich step passes.Refs #1