Skip to content

bench: add fixed SWE-bench Verified N30 cohort - #716

Merged
unohee merged 1 commit into
mainfrom
heewonoh/agt-3185-swebench-verified-n30
Sep 20, 2026
Merged

unohee merged 1 commit into
mainfrom
heewonoh/agt-3185-swebench-verified-n30

Conversation

@unohee

@unohee unohee commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a reproducible, fixed N=30 pure-logic SWE-bench Verified cohort (10 each: pylint, Sphinx, SymPy)
  • generate complete official records for the existing sweBench.ts harness
  • distinguish low-disk manifest availability from the required sequential amd64 image pull

Validation

  • generated all 30 official records from the Hugging Face dataset
  • verified 30/30 Docker image manifests
  • checked cohort distribution: 10 per repository
  • npm run typecheck
  • openswarm review --path . — APPROVE

Deliberately not run

  • full image pull: local disk had 80GiB free and no SWE image cache; the command is provided but is not claimed as complete
  • paid model solves/grading: excluded by the current no-expensive-API-test instruction

@unohee

unohee commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator Author

🔍 Fresh review of 95772e4 (openswarm pr review --fresh)

✓ Decision: APPROVE
Cohort selection, pagination, validation, and arg parsing are all correct. The data contract (FAIL_TO_PASS as JSON-encoded string) matches sweBench.ts. No production wiring gaps — this is a CLI data generator consumed via JSON file. The three minor observations above are non-blocking.
Suggestions:
- Extract imageFor() into a shared utility if a third consumer appears
- Consider a schema-level test that asserts the output JSON's shape matches sweBench.ts's SweInstance interface

@unohee
unohee merged commit c047983 into main Sep 20, 2026
7 checks passed
@unohee
unohee deleted the heewonoh/agt-3185-swebench-verified-n30 branch September 20, 2026 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant