test(pg-core): assert build.yml's gate wiring so an unapplied patch fails loudly - #316
Conversation
…ails loudly Every gate here lands in two halves: the code in a PR, the workflow YAML applied by hand, because dobby-coder has no `workflows: write`. Nothing detected a missing second half, and it has already happened twice (#265, postguard-js#137) with a green PR both times. pg-pkg/tests/api_gate.rs already closed that hole for api-diff.yml. This is the same instrument pointed at build.yml, which carries the three gates nothing machine-read until now: Wire compat the name the `main: required checks` ruleset pins, and that it still belongs to the job aggregating both halves -- a name moved onto a weaker job satisfies the ruleset just as well (#299) wire-compat-rust filter paths, the `push` override, the seal and open commands, the `sealed` output wire-compat-js that it gates on that output, and opens the same bytes with the published npm readers semver-checks both script calls, and that the breaking-change declaration is still read off the PR title It lives in pg-core because `Test workspace (pg-core)` runs unconditionally: pg-compat's tests run *inside* wire-compat-rust, so a guard beside the gate would be skipped by exactly the deletion it exists to catch. Verified by mutation rather than inspection: 18 drifts applied to build.yml one at a time, all 18 red. COMPATIBILITY.md called wire-compat-rust "a required PR check" and listed #262 and postguard-js#131 as planned; both shipped, and the required context has been `Wire compat` since #296. Corrected.
|
On it — I'll review this PR (part of #272) focusing on whether |
There was a problem hiding this comment.
Rules + review, cycle 1 on 2045c90. 4 findings: 1 blocking, 3 nits.
Ran on the branch: cargo test --manifest-path pg-core/Cargo.toml --features test,rust,stream --test ci_wiring — 5 passed. cargo fmt --check clean.
This is the right instrument in the right place, and the argument for pg-core over pg-compat holds up: Test workspace (pg-core) has no if: and no path filter, so it really does run on every PR, while pg-compat's tests run inside wire-compat-rust and would be skipped by the deletion they'd be guarding against. The prefix guard in every_job_this_test_reads_is_still_found is a real trap avoided — job() matches " wire-compat:" exactly, so wire-compat and wire-compat-rust never alias.
The blocking one is a hole in the assertion the file's own headline case depends on: the aggregator test pins the name, the uniqueness, the needs and the if:, but never the step that reads those results. Four one-line drifts to build.yml leave the suite green while turning the repo's sole required check permanently green — verified locally. Inline, with a suggestion that turns all four red.
Checked and clean, so you don't re-derive it
- Docs cross-references all resolve. #251, #262 and postguard-js#131 are closed, postguard-e2e#21 and #25 are open — so demoting #262/postguard-js#131 out of "still planned, each an open issue" and dropping #251 is accurate, and no link definition is left dangling in
COMPATIBILITY.md. #251 survives inpg-compat/README.mdandpg-compat-js/README.mdas parentage, which is still correct. path-filtered-job-cannot-be-a-required-check— both gates scope with a step-leveldorny/paths-filterand both filters include.github/workflows/build.yml, so a change to a job re-runs it. The test pins both path sets exactly.actions-default-shell-has-no-pipefail—build.ymlhas exactly one shell pipe (echo "$SHA256 …" | sha256sum -c -),sha256sumis last, and that step declaresshell: bashanyway.cargo-semver-checks-wasm-and-overload-traps— the 100-vs-101 mapping andRUSTFLAGS="--cap-lints=warn"both live inscripts/semver-checks.sh(lines 76 and 125), covered bysemver-checks-test.sh, which the workflow calls and this test pins. The workflow's half is right.patch-in-comment-gate-is-not-live-yet— every job the newCLAUDE.mdbullet andCOMPATIBILITY.mddescribe exists inbuild.ymlon this branch. Nothing is claimed present-tense that is still pending. Correctly, this PR touches no workflow file, so there is no second half to apply.
The three gates / four jobs wording in CLAUDE.md reads fine to me — the aggregator isn't a gate, it's what makes two gates into one context, which is exactly how the file's doc comment puts it.
| "the job named {REQUIRED_CHECK:?} must run even when an upstream half failed, or a \ | ||
| red half leaves the required check pending instead of failing it", | ||
| ); | ||
| } |
There was a problem hiding this comment.
Blocking. The aggregator test never asserts that Wire compat acts on the results it needs. It pins the name, the uniqueness, the needs set and if: ${{ !cancelled() }} — but not the one step that turns a red half into a red required check.
All four of these drifts to build.yml leave the suite green (each applied on its own to this branch, suite re-run each time):
exit 1→exit 0in the verdict step- dropping
"${{ needs.wire-compat-js.result }}" != "success"from the condition - dropping the Rust half's clause the same way
- deleting the condition and
exit 1, leaving only the twoechos
Each one yields a Wire compat that keeps its name, needs both halves, runs on !cancelled() — and reports success however those halves ended. That is the repo's sole required check, with bypass_actors: [], permanently green. Strictly worse than the rename in #299 this file was written to catch: a rename at least disarms one ruleset visibly, this certifies a failure as a pass.
The doc comment scopes the threat as "a Wire compat that needs one half, or none". The case it misses is the aggregator that needs both and reads neither.
The suggestion below adds that assertion. Verified on this branch: compiles, cargo fmt --check clean, passes on the unmutated build.yml, and red on all four drifts above.
| } | |
| // `needs` only makes the halves' results *available*; what turns a red half | |
| // into a red required check is the step that reads them. Without this the | |
| // aggregator keeps its name, its `needs` and its `if:` while its script | |
| // stops failing -- a sole required check that is permanently green. | |
| let steps = steps(&aggregator); | |
| let verdict = step_with(&steps, "exit 1"); | |
| for half in AGGREGATED_HALVES { | |
| assert!( | |
| verdict.contains(&format!(r#""${{{{ needs.{half}.result }}}}" != "success""#)), | |
| "the job named {REQUIRED_CHECK:?} does not fail on a red `{half}`, so the repo's \ | |
| sole required check reports success however that half ended", | |
| ); | |
| } | |
| } |
There was a problem hiding this comment.
Right, and it was the one gap that mattered — taken in 933b450.
Reproduced all four drifts myself before fixing, then re-ran them after: exit 0, each clause dropped on its own, and the condition deleted leaving only the echoes. All four now red on the_required_check_name_still_names_the_aggregator.
Took the substance rather than the suggestion block verbatim: asserting against the job block instead of step_with(&steps, "exit 1"), so deleting the verdict step fails with a message about the required check rather than about the reader not finding a step. Same four drifts caught either way.
You put the diagnosis better than the file did, so it is now failure class 3 in the module doc: needs makes the results available; only the step that reads them turns a red half into a red check.
| /// fails here instead of vacuously satisfying every assertion about it. | ||
| fn job(workflow: &str, id: &str) -> String { | ||
| let body = workflow | ||
| .split_once("\njobs:\n") |
There was a problem hiding this comment.
Nit — one line, in a file this PR doesn't touch. split_once("\njobs:\n") here (and at line 159 in job_ids) is a multi-line anchor matched against a file read with fs::read_to_string. rustc normalises CRLF to LF inside string literals; read_to_string does not. So on a Windows checkout (core.autocrlf=true is the Git for Windows default) the file is \r\njobs:\r\n and the anchor misses.
Reproduced on this branch rather than reasoned about — sed -i 's/$/\r/' .github/workflows/build.yml, then the suite:
test result: FAILED. 0 passed; 5 failed
panicked at pg-core/tests/ci_wiring.rs:139:28: no top-level `jobs:` in .../build.yml
All five, on a clean tree, with a message blaming the workflow rather than the line endings. Every other reader in this file is already CRLF-safe (line.trim(), trim_end(), single-line contains) — only this anchor breaks, which is why it hides.
CI here is ubuntu-latest plus one macos-latest, no Windows runner, so the blast radius is local runs only and this can't redden a PR. Non-blocking for that reason. Same class and same fix as the two entries already in root .gitattributes (cryptify#203, #269):
# pg-core/tests/ci_wiring.rs locates the `jobs:` block in this workflow with a
# multi-line anchor. Same reasoning as the specs above.
.github/workflows/build.yml -text
There was a problem hiding this comment.
Confirmed — sed -i "" $'s/$/\r/' on the workflow reds all five on a clean tree, with the panic blaming the workflow rather than the line endings. Fixed in 933b450`, and a CRLF checkout now passes.
Went with normalising at read time (.replace("\r\n", "\n") in workflow()) rather than the .gitattributes entry. The two entries already there have no choice: their anchors are multi-line raw-string content copied out of the spec, which cannot be normalised away without changing what is being matched. Here every reader is line-oriented and line endings carry no meaning in YAML, so normalising holds however the tree was checked out — including a clone that predates the attribute, which the -text route does not. The reasoning is in the workflow() doc comment so the next person does not "fix" it back.
| // be honoured: this repo squash-merges with COMMIT_MESSAGES, so the body | ||
| // never reaches the commit release-plz reads, and a `fix(pg-core):` subject | ||
| // would then cut a patch release of a break the gate had waved through. | ||
| let declared = step_with(&steps, "id: declared"); |
There was a problem hiding this comment.
Nit. The declaration is read off github.event.pull_request.title, and this test pins that — but not the trigger that makes a title edit re-run it. build.yml's on: block carries types: [opened, synchronize, reopened, edited] with a comment naming edited as load bearing, and nothing in this file reads the on: block, so dropping it back to a bare on: pull_request: passes the suite.
The default types are opened, synchronize, reopened — not edited. Two silent passes follow, both in the class this file exists to catch:
- A PR opened as
feat!:runs the gate withrelease_type=majorand goes green. Retitling tofix(pg-core):fires onlyedited, so the green stays attached to the unchanged head sha — and release-plz then cuts a patch release of a break the gate waved through, which is the exact scenario theid: declaredcomment argues against. - A base retarget also fires only
edited, so paths-filter's "nothing relevant changed" verdict survives against a base it never compared.
No suggestion block: where this belongs is a judgement call (it is about on:, not about a job, so it sits outside the job() helper). Something like this in whichever test you'd rather own it:
assert!(
workflow().contains("types: [opened, synchronize, reopened, edited]"),
"`edited` is no longer a pull_request trigger, so retitling a PR (or retargeting it) \
leaves the gate's verdict attached to the title and base it was computed from",
);There was a problem hiding this comment.
Taken in 933b450, as its own test — a_retitled_or_retargeted_pr_still_re_runs_the_gates. It is about on:, not about a job, and it covers both gates rather than only the semver one, so it did not belong inside either.
Asserts the types: list as a set against a pinned PULL_REQUEST_TYPES rather than contains on the literal line, so a reorder does not red it while a drop still does. Both drifts verified red: edited removed, and the whole types: line deleted back to the defaults.
The retitle scenario is the sharper half of this and is now written into the test doc, since it compounds: --release-type major skips every lint, so the green that survives the retitle verified nothing in the first place.
| "wire-compat-js does not run `npm test` in pg-compat-js behind `{SEALED_OUTPUT}`, so \ | ||
| the published npm readers are not being pointed at the sealed set", | ||
| ); | ||
| } |
There was a problem hiding this comment.
Nit. npm ci isn't pinned, only npm test is. build.yml carries an explicit comment on why the install command matters — "ci, not install: the readers are pinned by pg-compat-js/package-lock.json, and a gate that quietly resolved a different reader would be measuring something other than the support window" — but rewriting it to npm install, or dropping its working-directory: pg-compat-js, keeps this suite green. Both verified on this branch.
Same silent-narrowing class as a dropped filter path: the gate still runs, still reports, and is measuring readers COMPATIBILITY.md never declared. Optional, but it is one line and it closes the gap the workflow comment already names.
Verified: compiles, fmt-clean, passes unmutated, red on both drifts.
| } | |
| // `ci`, not `install`: the readers are pinned by pg-compat-js's lockfile, | |
| // so a gate that quietly resolved a different reader would be measuring | |
| // something other than COMPATIBILITY.md's support window. | |
| let install = step_with(&steps, "run: npm ci"); | |
| assert!( | |
| install.contains("working-directory: pg-compat-js"), | |
| "wire-compat-js no longer installs the pinned readers with `npm ci` in pg-compat-js, so \ | |
| the gate measures whatever npm resolves rather than the declared support window", | |
| ); | |
| } |
There was a problem hiding this comment.
Taken in 933b450. Verified both drifts red: npm ci → npm install, and the step losing its working-directory.
Worth the line for the reason you gave — it is the same silent-narrowing class as a dropped filter path, where the gate keeps running and reporting while measuring something other than the declared support window. The workflow comment naming it and nothing enforcing it is exactly what this file exists to replace.
|
Reviewed #316 and posted a REQUEST_CHANGES review with 4 inline comments (1 blocking, 3 nits). The blocking one: I also updated the CRLF detection rule in memory, since its documented tell would have missed this case. |
Review found the hole that mattered: the aggregator test pinned the name, the uniqueness, the `needs` set and `if: !cancelled()` -- but not the step that turns a red half into a red required check. `needs` only makes the halves' results available. An aggregator that needs both and reads neither keeps every property the test asserted and reports success however they ended, which is worse than the #299 rename this file was written to catch: a rename disarms one ruleset visibly, this certifies a failure as a pass. Three smaller gaps in the same class, each verified red: - `npm ci` was unpinned while `npm test` was, so rewriting it to `npm install` left the gate measuring readers COMPATIBILITY.md never declared - nothing read the `on:` block, so dropping `edited` from the pull_request types passed. It is the one type that is not a default, and without it a retitle from `feat!:` to `fix(pg-core):` leaves the semver gate's green attached to the title it was computed from - `split_once("\njobs:\n")` is a multi-line anchor against a file read with `read_to_string`, so all five tests failed on a CRLF checkout with a message blaming the workflow. Normalised at read time rather than via `.gitattributes`: every reader here is line-oriented, so this holds however the tree was checked out 8 further drifts applied to build.yml one at a time, all 8 red, and a CRLF checkout now passes.
Part of #272.
Every gate in this repo lands in two halves — the code and tests in a PR, the workflow YAML applied by hand, because
dobby-coderhas noworkflows: write. Nothing detected a missing second half. It has already happened twice (#265 mergedpg-compatwith no job calling it; postguard-js#137 patched one workflow of two), and the PR was green both times.pg-pkg/tests/api_gate.rsclosed that hole forapi-diff.ymlby reading the workflow back.pg-core/tests/ci_wiring.rsis the same instrument pointed atbuild.yml, which carries the three gates nothing machine-read until now.Wire compatexists, is unique, andneedsboth halvesmain: required checksruleset (bypass_actors: []) pins that literal string. A ruleset requiring a context no job produces blocks nothing, so renaming the aggregator disarms the gate instead of breaking it (#299)wire-compat-rust's 7 filter paths,pushoverride, seal + open commands,sealedoutputpushoverride is what stopped #297's class of silent passwire-compat-jsgates on that output and runsnpm testinpg-compat-jssemver-checkscalls both scripts and reads the declaration off the PR titleBREAKING CHANGE:footer never reaches the squashed commit release-plz readsWhy
pg-coreand not next to each gateTest workspace (pg-core)runs unconditionally.pg-compat's tests run insidewire-compat-rust— a guard living beside the gate would be skipped by exactly the deletion it exists to catch.Verified by mutation, not by inspection
18 drifts applied to
build.ymlone at a time, all 18 red. The two worth naming: deleting the aggregator outright, and moving the nameWire compatontowire-compat-rustso a single-half job satisfies the ruleset — both caught, the second with2 jobs carry that name.This PR touches no workflow file, so there is no second half to apply.
What it deliberately cannot reach
That the ruleset still names
Wire compatwithbypass_actors: []needsgh api repos/encryption4all/postguard/rules/branches/main, which no test runner can call. That half stays manual, as #272's own comment says.Also in here
COMPATIBILITY.mdcalledwire-compat-rust"a required PR check" and listed #262 and postguard-js#131 under "planned consumers, each still an open issue". Both shipped; the required context has beenWire compatsince #296. Corrected, andCLAUDE.mdgains the edit-them-together rule.