Pipeline robustness: stop silent data loss, cache poisoning, and hangs

[LukasWallrich]

Part of a repo audit. Python data-pipeline changes only (no workflow/R/frontend edits).

refresh_data.py

• A failed OpenCitations fetch for a replication no longer substitutes an empty citation list (which silently deflated that original's co-citation counts in a "complete" run) — the affected original is skipped, retried next run, and surfaced in meta.json (fetch_errors, fetch_error_dois, originals_errored) with partial_run set.
• Cache poisoning fixed: non-200 responses no longer cache an empty OMID identifier set for 30 days (which defeated the conflation check), and a 200 with an unparseable body is no longer cached as "zero citations".
• Co-citation matching now extracts a stable key (omid: token, DOI fallback) from OpenCitations' composite citing string instead of comparing the whole space-joined identifier list; applied at read time so existing caches stay valid (verified 41/41 rows on a real cache file).
• 429s get exponential backoff over 4 attempts honoring Retry-After (was: one 8s retry then skip).
• The per-year co-citation timeline now dedupes citing works that co-cite replications of different outcomes, matching the study-level counts.
• Cache TTL is now a fetched_at timestamp stored inside each cache file — file mtimes are reset by every CI checkout, so the 30-day TTL was effectively never expiring. Old-format files are treated as stale and upgrade on first refetch.
• The upstream flora.csv download now goes through the session with a timeout and 3-attempt backoff (was: pd.read_csv(URL) with no timeout, which could hang the 350-minute job).

compute_omc.py

• Transient errors (network, 429, 5xx, parse) are no longer cached as permanent "no venue" results; only a genuine 200-with-no-results caches a negative. Retries with backoff added.
• Aborts nonzero when >50% of new lookups (min 10) fail transiently, so an OpenAlex outage can't produce an incompletely enriched CSV with a fresh timestamp (follow-up to the codex review on Fix workflow push races, stale-data commits, and duplicate refresh #2).
• Atomic cache writes (temp + os.replace); a crash can no longer truncate the committed venue cache.
• The fuzzy venue search's top hit must now pass a token-overlap sanity check before being accepted (was: results[0] unconditionally, risking confidently wrong impact factors).
• The docstring's promised per-run time budget now actually exists (MAX_RUNTIME_SECONDS, mirrors refresh_data.py).

compute_author_overlap.py — outcome breakdown now uses the frontend's classifyOutcome mapping plus an "other" bucket; it previously dropped 90 known-overlap rows with variant labels (verified: by_outcome now sums to 2298 == n_total, was 2208).

refresh_flora.py — download gets a timeout + 3-attempt backoff.

All scripts compile; key helpers (citing-key extraction against a real cache file, TTL round-trip, name matching, outcome classification) unit-tested; the author-overlap script was run end-to-end against real data. The full OC/OpenAlex pipelines were not run (live APIs, hours).

🤖 Generated with Claude Code

[LukasWallrich]

Codex review (gpt, read-only): REQUEST_CHANGES — four findings, all addressed in 95072ff:

1. High — mid-study time-budget exhaustion still published an under-counted original (the inner should_stop() break left rep_fetch_failed false). ✅ Fixed: budget exhaustion now routes through the same skip-and-flag path as a fetch failure and stops the outer loop, so the affected original is deferred to the next run.
2. High — the event-study citation denominator (n_cit) still double-counted works co-citing replications of different outcomes. ✅ Fixed: only + with_any for deduped timeline rows, with the legacy summed-bucket fallback for old rows. Unit-checked (only=2, with_any=3 → 5, not 8).
3. Medium — exponential retries only covered 429; network exceptions and 5xx bailed immediately in both scripts. ✅ Fixed: both now retry under the same attempt budget (other 4xx stay non-retryable).
4. Medium — Retry-After parsing ignored the HTTP-date form. ✅ Fixed: both helpers parse it via parsedate_to_datetime, clamped to 0–120 s. Unit-checked (numeric, clamp, past/future dates, garbage, missing).

Codex verified the rest cleanly: the wrapped cache-format migration is internally consistent (old bare-list files go stale without crashing), citing keys are normalized at every compare/store site, the transient-failure abort correctly excludes cached genuine and weak-match misses, and the Python outcome classifier matches the JS branch ordering.