hermetic-check: the gate DECLARES suites hermetic by the absence of a marker and never verified it — 44 of 44 - #538
Conversation
…f 44 The `hermetic suites (gating)` job decides membership by the ABSENCE of a marker: a suite is hermetic iff nobody wrote `# SUITE-DEPENDS` in it. Nothing verified that. A suite shelling out to `gh` is declared hermetic by default, passes on every authenticated machine, and fails only on the runner — where it blocks a merge with an assertion naming neither `gh` nor the dependency. Instance that prompted it: #499. `close-condition-scan.py` handles `--states` at line 264, thirty lines after the network fetch at 234. `--states` is a pure DECLARE needing no repository. Locally `gh` answers and the misordering is invisible; the runner sees `AssertionError: 2 != 0`. Each declared-hermetic suite runs twice, varying exactly ONE binary. A suite whose exit code moves is not hermetic. 44 of 44 hermetic at 41daed3 (55 suites; 11 declare SUITE-DEPENDS) Shadowing `gh`, not truncating PATH: replacing PATH wholesale also drops homebrew, so a suite failing because `git` moved would score as a network leak. `git` and `python3` are asserted unmoved before anything is measured. No real `gh` is exit 2, never 0. With nothing to shadow every suite scores hermetic and the clean board is an artefact of the machine. "There was no gh" and "no suite needs gh" are two states the verdict depends on telling apart, so they must not share a code. Mutation-checked: breaking that precondition makes the tool report a clean board with no gh, and the self-test fails. The known-positive is drawn from refs/pr/499, outside the measured population, and asserted as a PAIR with a clean fixture — "the leak is detected" alone passes if everything were flagged, "the clean suite is not flagged" alone passes if nothing were. Two defects found in this tool by testing it rather than reasoning: its own --self-test required a real `gh` and so would have failed on the runner — the exact defect it detects — and `tree_provenance()` reported a clean SHA for a dirty working tree. DEV1
|
Warning Review limit reached
Next review available in: 32 minutes Limit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
⇒ Rebased onto
|
1 participant
hermetic suites (gating)decides membership by the absence of a marker: a suite is hermetic iff nobody wrote# SUITE-DEPENDSin it. Nothing verified the claim. A suite that shells out toghis declared hermetic by default, passes on every authenticated machine, and fails only on the runner — where it blocks a merge with an assertion naming neitherghnor the dependency.The instance that prompted it — #499.
close-condition-scan.pyhandles--statesat line 264, thirty lines after the network fetch at 234.--statesis a pure DECLARE that needs no repository. Locallyghanswers and the misordering is invisible; the runner getsAssertionError: 2 != 0.Design points that are the actual work
⛔ No real
ghis exit 2, never 0. With nothing to shadow, every suite scores hermetic and the clean board is an artefact of the machine. "There was nogh" and "no suite needsgh" are two states the verdict depends on telling apart. Mutation-checked: break that precondition and the tool reports a clean board with nogh— the self-test fails.⚠ Shadowing
gh, not truncatingPATH. My first reproduction on #499 usedPATH=/usr/bin:/bin, which also drops homebrew. Fine for one confirmed case; for a 44-suite survey it manufactures false positives — a suite failing becausegitmoved is not a network leak.gitandpython3are asserted unmoved before anything is measured.Known-positive asserted as a PAIR. The leak is detected alone passes if everything were flagged; the clean suite is not flagged alone passes if nothing were.
⛔ Two defects this tool had, found by running it rather than reasoning about it
--self-testrequired a realgh— so it would have failed on the runner. The exact defect it detects, in the tool that detects it. Caught by running it withghabsent. It now supplies its ownghfixture and passes with and without.tree_provenance()reported a clean SHA for a dirty working tree — letting a reading over uncommitted edits be attributed to a commit that never contained them. Now says so explicitly.⚠ What this does NOT establish
ghONLY. A suite reaching the network bycurl,urllib, or a git remote is not covered and is not thereby clean.ghvs unauthenticated vs rate-limited all produce the same refusal here; I reproduced one.Measured at
41daed3; tool at2fd1182. — DEV1