Corpus: exclude own org + repo-level pair-hunting mode - #19
Merged
Conversation
The single-scheduler corpora are ~99% disjoint by repo, so a semantic
join across them for cross-scheduler equivalent pairs would be a costly
near-no-op. Add a repo-centric "pairs" scrape mode instead:
- Phase 1 scans every scheduler's searches at repo granularity (search
only, no content fetch) to map repo -> {scheduler: files}.
- Phase 2 keeps repos spanning >= --min-schedulers formats.
- Phase 3 fetches, runs each scheduler's existing accept() predicate, and
saves co-located specs under testdata/corpus/candidate-pairs/, with a
committed manifest. These are candidate ground-truth conversion pairs.
Also exclude our own org (InsightSoftmax) from all scrape modes so we
stop re-ingesting BAMMM's own examples back into the corpus.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016PnTz6Zxqa4jHocK8kCbyx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two changes to the corpus scraper (
scripts/corpus/fetch_corpus.py):1. Exclude our own org from all scrapes
Every mode now skips repos owned by
InsightSoftmax(EXCLUDED_OWNERS), so westop re-ingesting BAMMM's own
conversions/and examples back into the corpus(self-contamination). Applies to both the per-scheduler scrape and the new pairs
mode.
2. New
pairsmode — repo-level, multi-format detectionThe single-scheduler corpora are ~99% disjoint by repo (only 5 of ~319 repos
appear in 2+ corpora, all docs/example repos), so a PZ-style semantic join across
them for cross-scheduler equivalent pairs would be an expensive near-no-op.
Instead, hunt for repos that themselves contain multiple scheduler formats:
(search only, no content fetch): build
repo -> {scheduler: [files]}.>= --min-schedulersformats (default 2).accept()quality predicate, and save co-located specs under
testdata/corpus/candidate-pairs/<owner__repo>/<scheduler>/with a committedmanifest.json.Co-located specs from different schedulers are candidate ground-truth conversion
pairs (build-plan stage 2). Specs stay gitignored (third-party licensing); only
the manifest is committed, consistent with the existing corpus policy.
Testing
--listshows the newpairsmode.exclusion (a 2-format
InsightSoftmax/*repo is dropped), single-format reposdropped, and correct on-disk layout + manifest.
git check-ignoreconfirms candidate-pair specs are ignored and the manifestis tracked.
🤖 Generated with Claude Code