DeltaWire: measured performance envelope + first real-model admission run #33
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: DeltaWire v7 no-live validation | |
| on: | |
| push: | |
| branches: [eval/deltawire-v7] | |
| paths: | |
| - "labs/20-deltawire/**" | |
| - ".github/workflows/deltawire-v7.yml" | |
| pull_request: | |
| paths: | |
| - "labs/20-deltawire/**" | |
| - ".github/workflows/deltawire-v7.yml" | |
| permissions: | |
| contents: read | |
| jobs: | |
| validate-v7: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v7 | |
| with: | |
| fetch-depth: 0 | |
| - uses: actions/setup-go@v6 | |
| with: | |
| go-version-file: labs/20-deltawire/go.mod | |
| cache-dependency-path: labs/20-deltawire/go.sum | |
| - uses: actions/setup-python@v6 | |
| with: | |
| python-version: "3.12" | |
| - uses: astral-sh/setup-uv@v7 | |
| - name: Build and stage the frozen Linux binary | |
| run: | | |
| frozen_tree="$RUNNER_TEMP/deltawire-v7-terminal-evidence" | |
| git worktree add --detach "$frozen_tree" deltawire-v7-terminal-evidence | |
| cd "$frozen_tree/labs/20-deltawire" | |
| GOWORK=off CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -buildvcs=false -o "$RUNNER_TEMP/deltawire-linux-amd64" ./cmd/deltawire | |
| cd "$GITHUB_WORKSPACE/labs/20-deltawire" | |
| mkdir -p eval/.generated eval/pairs/range-large-v7-r1/environment/.generated | |
| cp "$RUNNER_TEMP/deltawire-linux-amd64" eval/.generated/deltawire-linux-amd64 | |
| cp eval/.generated/deltawire-linux-amd64 eval/pairs/range-large-v7-r1/environment/.generated/deltawire | |
| test "$(sha256sum eval/.generated/deltawire-linux-amd64 | cut -d' ' -f1)" = e5198d15000e093a2a28e57ad4a093dde8f66bcab21462549ddf9114064c25f4 | |
| - name: Validate skill examples using the actual Linux binary | |
| run: | | |
| python3 labs/20-deltawire/eval/scripts/v7/skill_conformance.py \ | |
| --skill labs/20-deltawire/eval/pairs/range-large-v7-r1/skill/deltawire \ | |
| --binary labs/20-deltawire/eval/.generated/deltawire-linux-amd64 \ | |
| --report /tmp/skill-conformance.json | |
| if test -f labs/20-deltawire/eval/results/preflight-v7/skill-conformance.json; then | |
| cmp /tmp/skill-conformance.json labs/20-deltawire/eval/results/preflight-v7/skill-conformance.json | |
| fi | |
| - name: No-live v7 validation | |
| run: | | |
| python3 labs/20-deltawire/eval/scripts/v6/verify_terminal_evidence_lock.py | |
| python3 labs/20-deltawire/eval/scripts/v7/validate.py | |
| python3 labs/20-deltawire/eval/scripts/v7/verify_terminal_evidence_lock.py | |
| python3 labs/20-deltawire/eval/scripts/v7/test_terminal_evidence_lock.py | |
| for test_file in labs/20-deltawire/eval/scripts/v7/test_*.py; do | |
| DELTAWIRE_TEST_BINARY=labs/20-deltawire/eval/.generated/deltawire-linux-amd64 \ | |
| python3 "$test_file" | |
| done | |
| - name: Prove workflow has no model invocation | |
| run: | | |
| live_runner='runner.py .*pair'"-v7" | |
| harbor_live='harbor '"run" | |
| credential='GEMINI_API'"_KEY" | |
| ! grep -E "$live_runner|$harbor_live|$credential" .github/workflows/deltawire-v7.yml |