Skip to content

ci(competition): register the engine gate on the default branch #568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
plind-junior merged 4 commits into from
Jul 28, 2026
+160 −1
Merged

ci(competition): register the engine gate on the default branch #568

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
888e334
ci(competition): register the engine gate on the default branch
plind-junior Jul 28, 2026
7bf513c
ci(competition): zizmor annotations and shellcheck note for the gates
plind-junior Jul 28, 2026
9c3c908
ci(competition): bare shellcheck directive directly above the block
plind-junior Jul 28, 2026
d1f73ae
ci(competition): engine gate review fixes - sha pins and delete-only prs
plind-junior Jul 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
157 changes: 157 additions & 0 deletions .github/workflows/koth-engine-gate.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
# koth engine gate - scores strategy (ranking-code) submissions.
#
# the deliberate contrast with koth-gate.yml (the kit lane): a kit is data
# and auto-merges; a STRATEGY is code and NEVER auto-merges. this job has no
# write token, holds no secrets, and merges nothing. it scores the submission
# in a sandbox and posts a scorecard + leaderboard note; a human reviews the
# code and merges it as a new default if it wins. that human review is the
# review gate applied to engine code.
#
# security model:
# - pull_request_target: the workflow, the grader (score_strategy.py), and
# the champion strategy all come from the BASE branch. only the challenger
# .py is read from the PR, and it is executed ONLY inside the sandbox
# child (vouch.strategy.run_sandboxed: rlimits + an audit hook blocking
# network/subprocess/writes).
# - permissions: contents: read only. even a full sandbox escape lands on
# an ephemeral runner with a read-only token and no secrets, and cannot
# merge itself.
name: koth-engine-gate

on:
pull_request_target: # zizmor: ignore[dangerous-triggers] workflow + grader + champion all come from the base branch; the PR's .py runs only inside the sandbox child, and the job token is read-only with no secrets
types: [opened, synchronize, reopened, ready_for_review]
Comment on lines +21 to +23

@coderabbitai coderabbitai Bot Jul 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== workflow file =="
cat -n .github/workflows/koth-engine-gate.yml | sed -n '1,120p'

echo
echo "== workflow lint config / references =="
for f in .github/worklint .github/gitleaks-ignore .zizmor.yml zizmor.toml zizmor.yaml .github/actions.yml .github/actionlint.yaml actionlint.yaml; do
  [ -f "$f" ] && { echo "--- $f"; cat -n "$f"; }
done

echo
echo "== zizmor references =="
rg -n "zizmor|dangerous-triggers|pull_request_target|workflow-lint|actionlint" .github . 2>/dev/null | sed -n '1,200p'

echo
echo "== files maybe relevant =="
git ls-files '.github/workflows/*' '.github/**/*.yml' '.github/**/*.yaml' | sed -n '1,120p'

Repository: vouchdev/vouch

Length of output: 9624

🌐 Web query:

zizmor 1.27.0 dangerous-triggers pull_request_target workflow run zizmor inline ignore

💡 Result:

In zizmor v1.27.0, the dangerous-triggers audit flags workflows using pull_request_target or workflow_run because these triggers are often used insecurely [1][2][3]. However, as of version 1.27, zizmor includes an explicit exception for workflows that use pull_request_target solely with the actions/labeler action [1][4]. Specifically, if a workflow has exactly one job containing only one step, and that step uses actions/labeler, zizmor will suppress the dangerous-triggers finding [2][5]. If you encounter a dangerous-triggers finding that you believe is secure or necessary for your use case, you can suppress it using inline ignore comments or a zizmor.yml configuration file [6][7]: 1. Inline Ignore: You can add an ignore comment to the relevant line in your workflow file [6][8]: # zizmor: ignore[dangerous-triggers] 2. Configuration File (zizmor.yml): You can add a rule to a zizmor.yml file to ignore the finding for a specific file or location [6][9][10]: rules: dangerous-triggers: ignore: - filename.yml Ensure the ignore comment is placed correctly—it must be a valid YAML comment and cannot be placed inside a string or block literal [6][8]. More granular ignores can be achieved in zizmor.yml by specifying line and column numbers (e.g., filename.yml:line:column) [9][10].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== zizmor.yml =="
cat -n .github/zizmor.yml | sed -n '1,160p'

echo
echo "== workflow-lint trigger/run =="
cat -n .github/workflows/workflow-lint.yml | sed -n '1,80p'

echo
echo "== nearby comparable pull_request_target inline suppressions =="
for f in .github/workflows/labeler.yml .github/workflows/coderabbit-gate.yml .github/workflows/ui-screenshot-gate.yml .github/workflows/auto-merge.yml; do
  [ -f "$f" ] && { echo "--- $f"; cat -n "$f" | sed -n '1,70p'; }
done

echo
echo "== full koth-engine-gate =="
cat -n .github/workflows/koth-engine-gate.yml

Repository: vouchdev/vouch

Length of output: 22852

Add a scoped zizmor suppression for the engine gate trigger.

.github/zizmor.yml only disables other rules globally; it does not suppress dangerous-triggers, so zizmor==1.27.0 --offline --persona regular still fails koth-engine-gate.yml. Add a narrowly scoped .github/zizmor.yml override or an inline suppression with the existing “checks out only base + sandbox PR strategy as data” rationale after the action pins are fixed.

🧰 Tools
🪛 GitHub Actions: workflow-lint / 0_zizmor (workflow security).txt

[error] 21-23: zizmor [dangerous-triggers]: use of fundamentally insecure workflow trigger. pull_request_target is almost always used insecurely.

🪛 GitHub Actions: workflow-lint / zizmor (workflow security)

[error] 21-23: zizmor error[dangerous-triggers]: use of fundamentally insecure workflow trigger. 'pull_request_target' is almost always used insecurely.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/koth-engine-gate.yml around lines 21 - 23, Add a narrowly
scoped zizmor suppression for the dangerous-triggers finding on the
koth-engine-gate pull_request_target workflow, using the existing rationale that
it checks out only the base and sandbox PR strategy as data. Place the
suppression in .github/zizmor.yml or inline in the workflow, and apply it only
to this engine gate trigger after action pins are fixed.

Source: Pipeline failures


concurrency:
group: koth-engine-${{ github.event.pull_request.number }}
cancel-in-progress: true

permissions:
contents: read

jobs:
gate:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
pull-requests: write # scorecard comment only - no merge
steps:
- name: checkout base branch (trusted code only)
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4

- name: classify the PR
id: classify
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: |
gh api "repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}/files" \
--paginate --jq '.[].filename' > /tmp/changed.txt
# a delete-only pr matches the filename shape but has nothing to
# fetch at the head sha - classify it normal, not engine.
gh api "repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}/files" \
--paginate --jq '.[] | select(.status != "removed") | .filename' \
> /tmp/present.txt
count=$(wc -l < /tmp/changed.txt)
only=$(head -1 /tmp/present.txt)
case "$only" in
contrib/strategies/baseline.py|contrib/strategies/README.md) only="" ;;
esac
if [ "$count" = "1" ] && \
printf '%s' "$only" | grep -qE '^contrib/strategies/[A-Za-z0-9_]+\.py$'; then
Comment thread
coderabbitai[bot] marked this conversation as resolved.
echo "mode=engine" >> "$GITHUB_OUTPUT"
echo "path=$only" >> "$GITHUB_OUTPUT"
else
echo "mode=normal" >> "$GITHUB_OUTPUT"
fi

- name: pass through (not a strategy PR)
if: steps.classify.outputs.mode == 'normal'
run: echo "not a single-strategy PR - engine gate does not apply."

- name: fetch challenger strategy from the PR (as data)
if: steps.classify.outputs.mode == 'engine'
env:
GH_TOKEN: ${{ github.token }}
HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
KIT_PATH: ${{ steps.classify.outputs.path }}
run: |
gh api "repos/${HEAD_REPO}/contents/${KIT_PATH}?ref=${HEAD_SHA}" \
> /tmp/strat-meta.json
encoding=$(jq -r '.encoding // ""' /tmp/strat-meta.json)
if [ "$encoding" != "base64" ]; then
echo "strategy not returned as an inlined blob (encoding=$encoding)" >&2
exit 1
fi
jq -r '.content' /tmp/strat-meta.json | base64 -d > /tmp/challenger.py
if [ ! -s /tmp/challenger.py ]; then
echo "fetched strategy is empty" >&2
exit 1
fi

- name: set up python
if: steps.classify.outputs.mode == 'engine'
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: '3.12'

- name: install vouch (base branch code)
if: steps.classify.outputs.mode == 'engine'
run: python -m pip install -e .

- name: paired scoring - challenger vs baseline champion (sandboxed)
if: steps.classify.outputs.mode == 'engine'
id: score
env:
BASE_SHA: ${{ github.sha }}
run: |
set +e
python .github/scripts/score_strategy.py \
--champion contrib/strategies/baseline.py \
--challenger /tmp/challenger.py \
--base-sha "$BASE_SHA" \
--out /tmp/engine-report.json
code=$?
set -e
if [ "$code" = "0" ]; then
echo "verdict=dethroned" >> "$GITHUB_OUTPUT"
elif [ "$code" = "3" ]; then
echo "verdict=held" >> "$GITHUB_OUTPUT"
else
exit "$code"
fi

- name: post the scorecard
if: steps.classify.outputs.mode == 'engine'
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.pull_request.number }}
VERDICT: ${{ steps.score.outputs.verdict }}
run: |
{
echo "koth engine lane - ${VERDICT}"
echo
echo '```json'
cat /tmp/engine-report.json
echo '```'
echo
echo "engine code is NOT auto-merged. a winning strategy earns the"
echo "leaderboard place; a maintainer reviews the code and merges it"
echo "as a new default. the daily result is provisional (public seeds)"
echo "- payout rank is settled by the monthly sealed run."
} > /tmp/comment.md
gh pr comment "$PR_NUMBER" --repo "$GITHUB_REPOSITORY" \
--body-file /tmp/comment.md

- name: report the verdict as the check result
if: steps.classify.outputs.mode == 'engine'
env:
VERDICT: ${{ steps.score.outputs.verdict }}
run: |
echo "verdict: ${VERDICT}"
# a held challenger is a green, informational result - nothing merges
# here regardless, so the gate never blocks. a scoring error already
# failed the job above.
exit 0
4 changes: 3 additions & 1 deletion .github/workflows/koth-gate.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
name: koth-gate

on:
pull_request_target:
pull_request_target: # zizmor: ignore[dangerous-triggers] no PR code ever executes: the kit is fetched as data, validated against a closed-world allowlist, and scored by base-branch code with a read-only checkout
types: [opened, synchronize, reopened, ready_for_review]

concurrency:
Expand Down Expand Up @@ -143,6 +143,8 @@ jobs:
else
gate_note="scored only - auto-merge is disabled for base '${BASE_REF}'; a maintainer promotes proven champions to shipped defaults by hand"
fi
# the posted markdown carries literal backticks
# shellcheck disable=SC2016
{
echo "koth ladder - ${VERDICT}"
echo
Expand Down
Loading