Commit e14f21d
committed
fix(ci): give the dump its own schedule instead of the scrape's leftovers
#59 got the scrape to finish, and the run then died in the dump instead:
Enrich 06:13 -> 10:33 260 min (on a 200-minute budget)
Dump 10:39 -> 11:36 57 min, cut by the job timeout
total 330 min
Two things were wrong. The budget is only consulted *between* sources,
so one long source overran it by an hour; each source is now capped at
40 minutes by `timeout`, and the budget drops to 150 so the rest of the
pipeline keeps real room.
But the dump does not belong behind a multi-hour live scrape at all.
Whenever the scrape runs long the dump is what gets dropped, which is
how the published pages drifted away from the records for six weeks.
dump-refresh.yml already regenerates the dump from current data with no
scraping — and when it last ran, the dump step succeeded; only the PR
creation failed on a transient server error. Give it a weekly schedule
(Tuesdays, a day after the Monday jobs) so the published dump is
reconciled with data/ every week regardless of how the scrape goes, and
a job timeout so it can never be silently truncated either.
Refs #11 parent 791950b commit e14f21d
2 files changed
Lines changed: 17 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
8 | 16 | | |
9 | 17 | | |
10 | 18 | | |
| |||
17 | 25 | | |
18 | 26 | | |
19 | 27 | | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
20 | 31 | | |
21 | 32 | | |
22 | 33 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
76 | | - | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
77 | 80 | | |
78 | 81 | | |
79 | 82 | | |
| |||
88 | 91 | | |
89 | 92 | | |
90 | 93 | | |
91 | | - | |
| 94 | + | |
| 95 | + | |
92 | 96 | | |
93 | 97 | | |
94 | 98 | | |
| |||
0 commit comments