keygen bench is a benchmark for language models composing tracker music in FastTracker II. a model gets one bash tool in an offline Docker sandbox, writes an XM module, and submits it. the module is then rendered in a fresh FT2 process and scored outside the sandbox.
this repository has the runner, the pinned container build, the evaluator, the results website, every recorded run and the community submission tools. it contains no keygens, cracks or license-bypass code. the name is about the music.
results | scoring | submit a run | benchmark guide | support
- every model runs the same prompt (prompt v2) at the highest reasoning tier its route accepts, with the same frozen limits: 120 minutes per attempt, 120 seconds per command, no step limit.
- each model gets three independent attempts. the leaderboard ranks the best of the three; failures stay visible next to it.
- an attempt that fails for infrastructure reasons (provider outage, quota, network) can be rerun. a rerun never replaces an attempt that produced a score, and the replaced attempt stays listed.
- costs are estimates: recorded tokens times the provider's published API price. models that are free on their maker's API show $0.
the evaluator renders the submitted XM at 44.1 kHz / 16-bit in a fresh FT2 process, then an AI listener rates it: Meta Audiobox Aesthetics, a model trained on human ratings of audio. the score is its content enjoyment rating (1 to 10) times 10. silent tunes score 0 and tunes that are really one long pre-rendered sample are capped at 40. the listener is a model, not a person, and it wasn't trained specifically on tracker music, so listen before reading a score as a preference.
start with contributing runs. the guided CLI builds the environment, configures a provider, checks prerequisites and runs a smoke attempt or the full three-attempt submission. API-key presets include OpenAI and Anthropic; other services can use compatible Chat Completions, Responses or Messages endpoints. OAuth routes use your own authorized local bridge.
model requests can cost money. credentials stay outside the model's sandbox and must never be committed. keep your working runs outside the repository and publish only reviewed bundles.
community submissions stay labelled separately until the maintainer verifies their provenance and re-renders and re-scores the module. the bundle validator can't prove which model produced a file or that no undisclosed attempts happened.
for campaign configuration, evaluation and reporting, see the benchmark guide.
every attempt behind the published results is in runs/, one folder per model:
runs/<model>/
attempt-1/ attempt-2/ attempt-3/ the three ranked attempts
other/ failed, retried, quota and older runs, other services
each attempt keeps the full model conversation, the submitted tune.xm, the trusted render and its scores. videos are left out and audio is stored as lossless FLAC (flac -d --keep-foreign-metadata restores the exact WAV).
| path | purpose |
|---|---|
benchmark/ |
campaign runner, provider adapters, prompts, container build and evaluator |
benchmark/contrib/ |
community-run packaging and validation |
benchmark/tests/, tests/ |
offline regression tests and synthetic fixtures |
scripts/, tools/ |
pinned FT2 builds and native acceptance checks |
runs/ |
every recorded attempt of the maintainer campaigns, by model |
data/ |
scoring calibration metadata, without archived music |
web/classic/ |
results website and publication tools |
submissions/ |
reviewed community bundles |
for code or documentation changes, follow AGENTS.md. it links the task guides and sets the privacy, provenance and verification rules.
use the pinned requirements and build steps from the benchmark guide. the tests make no paid model calls:
python -m unittest discover -s benchmark/tests
python -m unittest discover -s tests
node --test tests/test_classic_player.mjs tests/test_classic_rankings.mjssome integration tests need the pinned dependencies or a native FT2 build. passing offline tests doesn't establish provider access, musical quality or production readiness.
runs are paid out of pocket. if you want more models tested, you can chip in through github sponsors, ko-fi or crypto.
the code and docs are MIT. third-party material keeps its own terms: the FT2 fonts and graphics are CC BY-NC-SA 4.0 and the FT2 build pins 8bitbubsy/ft2-clone via a fork. see NOTICE.md for the full list and the public-launch checklist before publishing a formerly private checkout.
to cite keygen bench, use CITATION.cff with the commit your experiment used.