Corpus-scale scan gaps (catalog scan, novelty accuracy) #413

Gaps found running dokimi over a ~2K-skill corpus sample (GitSkills study). P0 = corrupts reported numbers, P1 = blocks/distorts corpus runs, P2 = ergonomics.

P0 (existing issues)

• B1 novelty: bad LLM replies cached or misreported (judge retry, conflict silent drop, retry warning) — follow-up to #40 #219 bad LLM replies cached or misreported
• B2 novelty: extraction.reached counts the spans of a failed batch #311 (fixed by fix(novelty): a capped run probes atoms spread over the whole input #358) extraction.reached counts spans of a failed batch
• B3 Judge reliability: connect multi-judge pieces and verify evidence #407 phase 0: 1 valid panel vote → agreement 1.0; ground_truth not forwarded; identity hash ignores role→model

P1

• A1 YAML parse error reported as "missing a 'description'" (44.7K of 224K load errors) — covered by fix(novelty): check routing --out before calls, clearer YAML errors, real output estimate, noise shape #403
• A2 _scan_skill_files crashes on ENAMETOOLONG for a sibling path; should be an error row — PR fix(catalog): an OSError on one skill is its error row, not a crash #417
• A3 cross-catalog checks (name-conflict, mirror-drift, near-duplicate) fire between unrelated skills of a corpus root; need per-skill mode — PR feat(catalog): corpus mode for catalog scan (#413 A3, A4, A6, A7) #414
• A4 SKILL.md nested in another skill's bundle counted as separate skill (2,105 rows / 1,979 dirs) — PR feat(catalog): corpus mode for catalog scan (#413 A3, A4, A6, A7) #414
• B4 partial result not written at --max-usd stop — fix(novelty): write the partial routing result when --max-usd stops the run #406
• B5 judge output ceiling 4096 tokens; no cap-partial for analyze/atoms/probe/shorten; absent adjudication silent; no triage per-run estimate
• B6 no built-in test-retest (--repeats 1 noise null)

P2

• A5 catalog eval selects 1,741 skills where scan finds 2,105 on same root; rule not stated
• A6 no public per-skill review API (review_skill_dir(path)) — PR feat(catalog): corpus mode for catalog scan (#413 A3, A4, A6, A7) #414
• A7 no --jsonl / --jobs / resume for 1M+ skills — PR feat(catalog): corpus mode for catalog scan (#413 A3, A4, A6, A7) #414
• A8 no rule filter (--ignore name-mismatch)
• A9 no warning when judge is same model family as measured model (extend feat(eval): warn on self-judging on every judge path that lacked it #408)
• B7 routing output estimate low — fix(novelty): check routing --out before calls, clearer YAML errors, real output estimate, noise shape #403
• B8 replay provenance per answer — record replay provenance per answer in arms artifacts so headroom can exclude cached repeats #370