Skip to content

feat(competition): relevance-guard engine-lane submission - #574

Merged
plind-junior merged 2 commits into
koth-ladderfrom
engine/relevance-guard
Jul 28, 2026
Merged

feat(competition): relevance-guard engine-lane submission#574
plind-junior merged 2 commits into
koth-ladderfrom
engine/relevance-guard

Conversation

@plind-junior

Copy link
Copy Markdown
Member

engine-lane entry for season 1 (#573): relevance-guarded ranking.

one general idea carried through: every class of suspect memory — hearsay, third-person possessive fragments, stored instructions, stale values contradicted by a change-of-state memory — gets a single uniform penalty scaled by how much the suspect matches the query. suspects fall out of a tight candidate pool in damage order, so the item the question is actually about is the one excluded first. past-tense questions ("what was ... before it changed?") suspend the collapse, because there the stale side is the answer. no benchmark-keyed logic: no value lists, no template matching, no category dispatch.

local verification with the ci-exact loop:

vouch bench run --seeds 1,2,3,4,5,6,7,8,9,10,11,12 \
    --strategy contrib/strategies/relevance_guard.py \
    --against contrib/strategies/baseline.py
challenger 0.7083  champion 0.6583  diff +0.0500  band 0.0381
DETHRONED

per the lane rules the gate scores this in a sandbox and it never auto-merges; on a human merge above the band, the ledger sweep promotes it to reigning champion and the threshold ratchets.

one uniform suspect penalty scaled by query relevance, so hearsay,
possessive fragments, stored instructions, and stale conflicting values
fall out of a tight candidate pool in damage order; conflict collapse
marks values contradicted by a change-of-state memory; past-tense
questions suspend collapse because history is the answer there.

verified with the ci-exact loop over seeds 1-12: challenger 0.7083 vs
champion 0.6583, diff +0.0500, band 0.0381 - dethroned.
@plind-junior plind-junior added the season-1 vouchbench season 1 entries and announcements label Jul 28, 2026
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)
  • test
  • main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 660925d3-0ce4-4537-b0fd-784e69406c91

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch engine/relevance-guard

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the size: S 50-199 changed non-doc lines label Jul 28, 2026
plind-junior added a commit that referenced this pull request Jul 28, 2026
pull_request_target checks out the DEFAULT branch when no ref is given,
so the engine gate scored from main - which carries the workflow but not
the scorer or the strategies - and died on a missing file (pr #574's
first run). the kit gate got this exact pin in 683e0a3; the engine gate
now matches, and the seed identity comes from the checked-out base tip
instead of github.sha for the same reason.
plind-junior added a commit that referenced this pull request Jul 28, 2026
pull_request_target checks out the DEFAULT branch when no ref is given,
so the engine gate scored from main - which carries the workflow but not
the scorer or the strategies - and died on a missing file (pr #574's
first run). the kit gate got this exact pin in 683e0a3; the engine gate
now matches, and the seed identity comes from the checked-out base tip
instead of github.sha for the same reason.
plind-junior added a commit that referenced this pull request Jul 28, 2026
pull_request_target checks out the DEFAULT branch when no ref is given,
so the engine gate scored from main - which carries the workflow but not
the scorer or the strategies - and died on a missing file (pr #574's
first run). the kit gate got this exact pin in 683e0a3; the engine gate
now matches, and the seed identity comes from the checked-out base tip
instead of github.sha for the same reason.
@plind-junior plind-junior reopened this Jul 28, 2026
@github-actions github-actions Bot added the ci: passing ci is green label Jul 28, 2026
@github-actions

Copy link
Copy Markdown

koth engine lane - dethroned

{
 "date": "2026-07-28",
 "base_sha": "b11ac52a1fbd1172887eb924a8e170aec15eef5a",
 "seeds": [
  67078370866574,
  116114090607205,
  108999320470477,
  114845009661335,
  248394666013973,
  108843862080367,
  274140852163751,
  57307557147568
 ],
 "lane": "engine",
 "champion": {
  "scores": [
   0.5,
   0.7,
   0.5,
   0.7,
   0.7,
   0.8,
   0.7,
   0.7
  ],
  "mean": 0.6625
 },
 "challenger": {
  "scores": [
   0.5,
   0.7,
   0.6,
   0.7,
   0.7,
   0.9,
   0.8,
   0.7
  ],
  "mean": 0.7
 },
 "mean_diff": 0.037500000000000006,
 "se": 0.018298126367784998,
 "band": 0.03586432768085859,
 "dethroned": true
}

engine code is NOT auto-merged. a winning strategy earns the
leaderboard place; a maintainer reviews the code and merges it
as a new default. the daily result is provisional (public seeds)

  • payout rank is settled by the monthly sealed run.

@plind-junior
plind-junior enabled auto-merge (squash) July 28, 2026 06:45
@github-actions

Copy link
Copy Markdown

koth engine lane - dethroned

{
 "date": "2026-07-28",
 "base_sha": "b11ac52a1fbd1172887eb924a8e170aec15eef5a",
 "seeds": [
  67078370866574,
  116114090607205,
  108999320470477,
  114845009661335,
  248394666013973,
  108843862080367,
  274140852163751,
  57307557147568
 ],
 "lane": "engine",
 "champion": {
  "scores": [
   0.5,
   0.7,
   0.5,
   0.7,
   0.7,
   0.8,
   0.7,
   0.7
  ],
  "mean": 0.6625
 },
 "challenger": {
  "scores": [
   0.5,
   0.7,
   0.6,
   0.7,
   0.7,
   0.9,
   0.8,
   0.7
  ],
  "mean": 0.7
 },
 "mean_diff": 0.037500000000000006,
 "se": 0.018298126367784998,
 "band": 0.03586432768085859,
 "dethroned": true
}

engine code is NOT auto-merged. a winning strategy earns the
leaderboard place; a maintainer reviews the code and merges it
as a new default. the daily result is provisional (public seeds)

  • payout rank is settled by the monthly sealed run.

@plind-junior
plind-junior merged commit 0ce99c2 into koth-ladder Jul 28, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci: passing ci is green season-1 vouchbench season 1 entries and announcements size: S 50-199 changed non-doc lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant