Skip to content

fix(ci): stop weekly-refresh dying at the 6h ceiling before it dumps - #59

Merged
Seungpyo1007 merged 1 commit into
mainfrom
fix/weekly-refresh-budget
Sep 8, 2026
Merged

fix(ci): stop weekly-refresh dying at the 6h ceiling before it dumps#59
Seungpyo1007 merged 1 commit into
mainfrom
fix/weekly-refresh-budget

Conversation

@Seungpyo1007

Copy link
Copy Markdown
Member

What is broken

Every weekly-refresh run ends at 6:00:2x — the GitHub job ceiling — always inside Enrich benchmarks (all sources):

34113867834   6:00:23  cancelled
33389588839   6:00:21  cancelled
32697778694   6:00:21  cancelled
32001853491   6:00:21  cancelled
31364447360   6:00:22  cancelled
30799400591   6:00:23  cancelled

A job that hits the ceiling is reported as cancelled, not failed, so it never showed up in the failed-run list. Meanwhile every step after the scrape was skipped:

cancelled  Enrich benchmarks (all sources)
skipped    Validate (schema / range / slug / FK)
skipped    Integrity check
skipped    Verification band report (Tier 0)
skipped    Generate static dump
skipped    Create refresh PR

What it cost

The static dump has been drifting away from data/ for at least six weeks, because the only job that regenerates it never reached that step. Measured against the current develop:

collection pages verified disagrees with the record
gpu 2,030 565
pda 140 39
brand 207 24
soc 2,104 23
watch 433 7

The published API is advertising a verification state the data no longer has — 28% of GPU pages. (Promotion PRs flip verified in data/ and do not touch site/public/v1; the weekly dump is what was supposed to reconcile them.)

Change

  • Collection gets a 200-minute budget, checked between sources, so validate / integrity / dump / PR always get the remainder. A source is never cut mid-scrape — the budget is only consulted before starting the next one.
  • The source order rotates by ISO week. With a fixed order the tail (topcpu-cpu, topcpu-gpu) would be starved forever once the budget bites. Verified: week 37 starts at geekbench-nbc and wraps.
  • timeout-minutes: 330 on the job, so the platform ceiling can never silently truncate it again — it fails loudly instead.

bash -n on the rendered step body passes; the YAML parses.

Refs #1

Every run since at least 2026-08-03 has ended at 6:00:2x, always in
"Enrich benchmarks (all sources)". GitHub reports a job that hits its
6h ceiling as *cancelled* rather than failed, so nothing ever appeared
in the failed-run list while validate, integrity, the static dump and
the refresh PR were skipped every single week.

The cost is not theoretical: the published dump has drifted away from
data/ because the dump step never ran. 565 of 2,030 gpu pages, and 93
more across soc/watch/pda/brand, currently advertise a `verified` value
the record no longer has.

Give collection a 200-minute budget it checks between sources, so the
remainder of the pipeline always gets to run, and rotate the starting
point by ISO week so sources at the tail of the list are not starved
once the budget starts biting. Add a 330-minute job timeout so this
can never again be silently truncated by the platform ceiling.

Refs #1
@Seungpyo1007 Seungpyo1007 added the bug Something isn't working label Sep 8, 2026
@Seungpyo1007 Seungpyo1007 self-assigned this Sep 8, 2026
@Seungpyo1007
Seungpyo1007 merged commit 0e67ef5 into main Sep 8, 2026
1 check passed
@Seungpyo1007
Seungpyo1007 deleted the fix/weekly-refresh-budget branch September 8, 2026 06:06
Seungpyo1007 added a commit that referenced this pull request Sep 11, 2026
…vers

#59 got the scrape to finish, and the run then died in the dump instead:

    Enrich  06:13 -> 10:33   260 min (on a 200-minute budget)
    Dump    10:39 -> 11:36    57 min, cut by the job timeout
    total                    330 min

Two things were wrong. The budget is only consulted *between* sources,
so one long source overran it by an hour; each source is now capped at
40 minutes by `timeout`, and the budget drops to 150 so the rest of the
pipeline keeps real room.

But the dump does not belong behind a multi-hour live scrape at all.
Whenever the scrape runs long the dump is what gets dropped, which is
how the published pages drifted away from the records for six weeks.
dump-refresh.yml already regenerates the dump from current data with no
scraping — and when it last ran, the dump step succeeded; only the PR
creation failed on a transient server error. Give it a weekly schedule
(Tuesdays, a day after the Monday jobs) so the published dump is
reconciled with data/ every week regardless of how the scrape goes, and
a job timeout so it can never be silently truncated either.

Refs #1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant