Skip to content

Corpus: exclude own org + repo-level pair-hunting mode - #19

Merged
stackedsax merged 1 commit into
mainfrom
feat/pair-hunting-scraper
Jul 18, 2026
Merged

Corpus: exclude own org + repo-level pair-hunting mode#19
stackedsax merged 1 commit into
mainfrom
feat/pair-hunting-scraper

Conversation

@stackedsax

Copy link
Copy Markdown
Contributor

What

Two changes to the corpus scraper (scripts/corpus/fetch_corpus.py):

1. Exclude our own org from all scrapes

Every mode now skips repos owned by InsightSoftmax (EXCLUDED_OWNERS), so we
stop re-ingesting BAMMM's own conversions/ and examples back into the corpus
(self-contamination). Applies to both the per-scheduler scrape and the new pairs
mode.

2. New pairs mode — repo-level, multi-format detection

The single-scheduler corpora are ~99% disjoint by repo (only 5 of ~319 repos
appear in 2+ corpora, all docs/example repos), so a PZ-style semantic join across
them for cross-scheduler equivalent pairs would be an expensive near-no-op.

Instead, hunt for repos that themselves contain multiple scheduler formats:

  • Phase 1 — scan every scheduler's existing searches at repo granularity
    (search only, no content fetch): build repo -> {scheduler: [files]}.
  • Phase 2 — keep repos spanning >= --min-schedulers formats (default 2).
  • Phase 3 — fetch those files, run each scheduler's existing accept()
    quality predicate, and save co-located specs under
    testdata/corpus/candidate-pairs/<owner__repo>/<scheduler>/ with a committed
    manifest.json.

Co-located specs from different schedulers are candidate ground-truth conversion
pairs (build-plan stage 2). Specs stay gitignored (third-party licensing); only
the manifest is committed, consistent with the existing corpus policy.

uv run scripts/corpus/fetch_corpus.py pairs
uv run scripts/corpus/fetch_corpus.py pairs --min-schedulers 2 --search-limit 200
make corpus SCHED=pairs

Testing

  • --list shows the new pairs mode.
  • Offline test with stubbed GitHub API verified: multi-format grouping, own-org
    exclusion (a 2-format InsightSoftmax/* repo is dropped), single-format repos
    dropped, and correct on-disk layout + manifest.
  • git check-ignore confirms candidate-pair specs are ignored and the manifest
    is tracked.

🤖 Generated with Claude Code

The single-scheduler corpora are ~99% disjoint by repo, so a semantic
join across them for cross-scheduler equivalent pairs would be a costly
near-no-op. Add a repo-centric "pairs" scrape mode instead:

- Phase 1 scans every scheduler's searches at repo granularity (search
  only, no content fetch) to map repo -> {scheduler: files}.
- Phase 2 keeps repos spanning >= --min-schedulers formats.
- Phase 3 fetches, runs each scheduler's existing accept() predicate, and
  saves co-located specs under testdata/corpus/candidate-pairs/, with a
  committed manifest. These are candidate ground-truth conversion pairs.

Also exclude our own org (InsightSoftmax) from all scrape modes so we
stop re-ingesting BAMMM's own examples back into the corpus.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016PnTz6Zxqa4jHocK8kCbyx
@stackedsax
stackedsax merged commit 6e4afd3 into main Jul 18, 2026
5 checks passed
@stackedsax
stackedsax deleted the feat/pair-hunting-scraper branch July 18, 2026 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant