feat(competition): koth ladder - auto-merge kit PRs scored by vouchbench

[plind-junior]

this is the contributor-onboarding half of the beat-ditto plan: ditto's mining loop (tweak a retrieval starter kit, get scored, dethrone the champion, earn while on top) rebuilt entirely on github primitives. stacks on #561, which supplies the bench.

the ladder surface is one data file, competition/kits/current/kit.yaml - a schema-validated fragment that IS the whole retrieval config of the throwaway bench kb (backend, limit, recency, prompt-gate, rerank, pages-first), every knob bounded. a PR that touches only that file is a ladder entry: the koth-gate workflow validates the kit against a closed-world allowlist, runs vouchbench paired per seed (reigning kit vs challenger, common random numbers), and on a dethrone - mean improvement over max(0.007, 1.96 x paired se) - enables github auto-merge.

security model: pull_request_target, so the workflow definition and every executed line come from the base branch; PR code is never checked out. the only thing read from the PR is the kit file, fetched over the api as data and refused unless it is a small inlined base64 blob (an oversized file would otherwise come back empty and score as champion defaults). the allowlist is closed-world because config can name executables (compile.llm_cmd) - a kit that could reach that key would run commands on a write-token runner, so unknown keys are an error. untrusted event fields reach scripts via env only.

the review gate stays load-bearing in two ways. code never auto-merges: the ladder surface is data, the worst a malicious entry can do is score badly. and the trunk is never auto-written: auto-merge fires only for PRs whose base is a dedicated, non-shipped ladder branch (repo var KOTH_LADDER_BASE). a kit-only PR against main is scored and commented, then a human merges - a beatable benchmark must never be the sole writer to code users install. promoting the reigning kit into the shipped starter defaults is a periodic human PR. this division of labour - humans review capabilities, the ladder tunes coefficients - is how the allowlist grows without ever widening the write path.

the daily ladder is deliberately provisional: seeds derive from public inputs (base sha + utc date) and the scorer is offline-reproducible, so a contributor can grid-search today's seeds and submit a pre-verified winner. that is fine because money is never tied to the daily throne - payout rank is settled by the monthly season's sealed commit-reveal run (docs/vouchbench-seasons.md), whose seeds do not exist until entries freeze. the daily loop is fast feedback; the sealed run is where a kit must generalise.

this branch was hardened against an adversarial review (a fan-out of security/anti-cheat/mechanics agents, each finding verified by an independent refuter). fixes that landed: the empty/oversized-kit bypass, a stray 1-hour real sleep that would have out-slept the ci job, the auto-merge-to-trunk gate-bypass (now ladder-branch-only), band calibration (12 paired seeds, z=1.96), the draft-PR trigger gap, and a branch-protection setup command that 422'd on a stringified boolean.

files: competition/kits/current/kit.yaml (champion, ascii-only - it is concatenated into a config parsed under the machine locale), .github/scripts/validate_kit.py (allowlist) + koth_score.py (paired scoring), .github/workflows/koth-gate.yml (the gate), competition/LEADERBOARD.md (throne ledger), docs/koth-ladder.md (contributor + maintainer walkthrough), tests/test_koth_kit.py (validator regressions). gates green: ruff, mypy (108 files), full pytest.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)

• test
• main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 415a4c72-da3a-4b20-99c9-497a28416cc8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/koth-automerge

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}