Koth ladder - #570
Merged
Merged
Conversation
run() gained a strategy parameter forwarded to build_context_pack, but the strategy module and the context-side parameter never landed in this history — every bench.run call raised TypeError, which would error the koth gate on any ladder PR. remove the dangling coupling; the engine-lane work can reintroduce it together with its context.py half.
no legal kit change can beat the baseline today - measured on the day's 12 paired seeds, every ranking knob (recency, rerank, pages-first, prompt gate) produces exactly zero composite diff because ranking never changes pack membership at bench density; only backend moves the score, and substring moves it down 0.34. so the auto-merge loop gets its integration test the honest way: the reigning kit is pinned to the weak substring backend, and the first challenger restoring backend auto dethrones it with a genuine, reproducible margin. ladder branch only - the shipped defaults are untouched.
pull_request_target checks out the default branch by default, so ladder PRs ran against a tree with no scripts, no engine, and no reigning kit (pr 565's first run). pin actions/checkout to the pull request's base ref - still trusted code, a branch of this repo, never the PR head - and derive the seed sha from the checked-out tree instead of github.sha, which is also the default branch under this trigger.
round-1 ladder entry: the round-0 canary champion pins the substring scan; this kit restores backend auto (fts5 with rrf fusion when embeddings exist). measured locally with the gate's own scorer on the day's derived seeds, the restore clears the dethrone band by a wide margin. kit-only change, as the ladder requires.
merge feat/koth-engine-lane into the ladder branch so the gates score with the shared paired_verdict math and the koth-ledger workflow is live on the branch it triggers from. restores the strategy kwarg on bench.run that the ladder had stripped while the module was uncommitted.
substring backend scores ~0.2-0.4 composite below auto on any seed, so the next kit-only pr restoring auto dethrones decisively. round 1 exercises the full chain live: gate scorecard, auto-merge on the protected ladder branch (gate required, strict), and the new koth-ledger row append.
…566) the reigning round-1 canary runs the substring backend, which scores ~0.2-0.4 composite below auto on any seed. restoring auto clears the dethrone band on the day's paired seeds.
… events round 1 (pr #566) proved the gap live: the gate arms auto-merge with the workflow GITHUB_TOKEN, and github suppresses workflow triggers for pushes caused by that token, so the merge push never reached the ledger. the workflow now sweeps recently merged ladder prs (schedule + manual dispatch + the still-working push path for human merges); the appender stays idempotent per pr, so overlapping sweeps converge.
the ladder's required gate check applies to direct pushes too, so the bot's row append was rejected (round 1 proved it). protection moved from classic branch protection to the koth-ladder-gate ruleset (same gate requirement, strict), and the ledger checkout uses an owner token with admin bypass. the workflow never runs on pr events and never executes untrusted code, so the token stays out of untrusted reach.
with a ranking strategy active, retrieval fetched exactly `limit` hits before the strategy ran, so ordering could never change pack membership - and the bench grades presence, not order, which made every strategy score byte-identical to baseline (measured: 12 seeds, diff 0.0000). retrieval now over-fetches a bounded pool (5x limit, floor 50) when a strategy is active and the top `limit` of the strategy's order survive the cut. de-prioritising below the window excludes the candidate; a strategy still cannot fabricate a result or shrink the pack. contract language in the lane doc and baseline docstring updated to match, plus a regression test that a demoted candidate leaves the pack.
ranks first-hand statements over hearsay (reported speech about a named third party is demoted, hardest when the query is first-person), pushes stored instructions to the tail (a memory that tells the reader what to answer is an injection, not a fact), and boosts change-of-state phrasing so the newest value survives a tight budget. blended with the backend score and plain lexical overlap for everything else. verified locally with the ci-exact loop over generic seeds 1-12: challenger 0.5750 vs baseline 0.5250, diff +0.0500, band 0.0381 - dethroned.
open contributors kept needing the lane rules from three scattered docs. contributing.md now states the two pr shapes the gates enforce: engine lane = one new file under contrib/strategies/ against koth-ladder, sandboxed, scored, never auto-merged, human review for benchmark-keyed logic; kit lane = kit.yaml only, auto-merge on dethrone, low ceiling by construction. includes the local ci-exact practice command and the reproducibility and payout pointers.
workflow lint only runs on prs that touch workflows, so the koth gates' pull_request_target findings never surfaced until the engine-gate registration pr woke the linter. both gates now carry the ignore annotation with the justification inline (base-branch code only, pr content handled as data, read-only tokens), and the scorecard comment script declares its literal backticks for shellcheck.
feat(competition): provenance-rank engine-lane submission
|
Warning Review limit reached
Next review available in: 39 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
docs
github-actions
Bot
added
size: S
pin checkout and setup-python to immutable commit shas (mutable tags in a pull_request_target workflow are a supply-chain surface), and classify a delete-only strategy pr as normal instead of engine - the filename shape matched but there is nothing to fetch at the head sha, so the gate failed messy instead of passing through.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Why
What might break
VEP
Tests
make checkpasses locally (lint + mypy + pytest)CHANGELOG.mdupdated under## [Unreleased]