Harden corpus scraper + first pair-hunting run

[stackedsax]

What

Ran the pairs scrape end-to-end (build-plan stage 2), which exposed and fixed
several scraper bugs, and produced the first real cross-scheduler pair candidates.

Scraper fixes

• Quote directive phrases. GitHub tightened code-search query parsing: a bare
  --array / -l / -W leads with a dash and is now parsed as a NOT operator,
  so every slurm and pbs query returned 422 — the corpus scraper was silently
  broken for both HPC schedulers. Quoting the phrases ("#SBATCH --array") fixes it.
• Survive GitHub's throttling transport failures instead of crashing:
  • honor Retry-After on 403/429 secondary (abuse) rate limits;
  • retry on dropped connections (RemoteDisconnected) and on 408/5xx from the
    search backend;
  • guard each phase-1 query so a single failure can't abort the whole sweep.
• Clear the pairs output dir on start so stale repo dirs from a prior run
  can't misrepresent the current results.

First pairs run

3 candidate pair-repos, all volcano↔kueue (manifest committed; specs stay
gitignored per corpus policy):

Repo	Formats	Notes
NVIDIA/knavigator	volcano, kueue	Genuine equivalent pair — the same benchmark job ({{._NAME_}}, N replicas, ubuntu, 8 GPUs, identical kwok annotations) rendered once per scheduler. Usable as conversion ground truth now.
Project-HAMi/website	volcano, kueue	Co-located tutorial examples; needs review for equivalence.
lasyard/docs	volcano, kueue	Co-located k8s examples; needs review.

Finding

Repo-level co-location surfaces candidates cheaply, but does not prove
equivalence. A false positive from an earlier run (converged-computing/ fractale-experiments, slurm+pbs) turned out to be unrelated jobs a repo merely
collects. The gold seam is benchmarking/harness repos like knavigator that
deliberately express one workload across schedulers. HPC↔HPC and HPC↔K8s real-world
co-located pairs remain scarce, matching the earlier corpus cross-reference.

Testing

• Scrape completed exit 0; --list unchanged; python ast.parse clean.
• On-disk candidate tree reconciled to the committed manifest (3 repos).

🤖 Generated with Claude Code