gated-caller --stability: does the verdict depend on the stub? It does — and it hid a fourth undercount (dedicated self-tests are 8, not 4)

[jobordu]

This figure has been published as a count three times today and was a property of the stub each time:

1 of 36   the stub died on AttributeError    (#536)
2 of 36   it died on TypeError               (#552)
4 of 36   it survives both                   (#554)

⛔ Chasing stub fidelity is unbounded and gives no signal for when to stop. ⇒ Running the SAME probe at TWO fidelities does: if the answer moves, the answer depends on the instrument rather than on the repository.

⇒ Run once, it found a FOURTH undercount — in a field I had not revisited since this morning

⛔ SELFTEST  MOVED  rich=8 poor=5
             only-rich: estate-provenance.py, pane-binding.py, pretooluse-guard.py
⛔ SWEPT     MOVED  rich=4 poor=2
ok  CALLED   stable at 25
ok  REACHED  stable at 25

⇒ ★ "4 of 34 have a DEDICATED suite running --self-test" — the first line this tool ever printed — was also low. It is 8.

★ This is the error term the exit-code line could not supply

⚠ #554 established that truncation risk sat at 28 of 51 across a stub change that revealed two more real callers — because an exit code says a suite ended badly and never says WHERE.

⇒ Two fidelities disagreeing is a fact about what went unseen. ⛔ It is not a bound either — the sets are what this tool can PROVE it was missing at the poorer fidelity, not the whole of it, because a third fidelity may move it again. The output says exactly that.

⇒ Design notes

• STUB_POOR is kept deliberately. It is not dead code, it is the second fidelity.
• ⚠ STABLE means only that these two stubs agree, never that the count is right. The output refuses the stronger reading in words.
• Opt-in, never the default — it costs 2× and answers a question about the instrument, not about the repository.

Controls

Both directions, because an alarm that always fires is not a check: the late.py fixture — hidden at the poor fidelity, visible at the rich one — must report MOVED; a tree where both fidelities agree must report STABLE.

⚠ NOT established: whether a third, richer fidelity would move it again. ⛔ I expect it would, and nothing here rules it out.

Measured at b20df2d. — DEV1

[coderabbitai]

Warning

Review limit reached

@jobordu, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 22 minutes

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 556437df-fd04-4bcf-81c1-6dfc0a31f3ca

📥 Commits

Reviewing files that changed from the base of the PR and between b43d838 and f157265.

📒 Files selected for processing (2)

• tools/README.md
• tools/gated-caller.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[jobordu]

⇒ Closing my own caveat: a THIRD fidelity does NOT move it. I expected it would

DEV1. ⚠ This PR said "NOT established: whether a third, richer fidelity would move it again. I expect it would, and nothing here rules it out." ⇒ Built one and measured. My expectation was wrong.

third fidelity adds:  __enter__/__exit__ · __next__ · __contains__ · __fspath__

                fidelity 2    fidelity 3
SELFTEST            8      ->      8
SWEPT               4      ->      4
IN-PROCESS         22      ->     22

⇒ ★ Three fields, no movement. That is evidence the sequence 1 → 2 → 4 has converged at this population — not proof, and I will not claim more: a fidelity is only a probe for the operations it adds, and these four may simply not be ones these suites perform. A different third fidelity could still move it.

⚠ An artefact I nearly reported as a finding

The variant also showed RUNNER 28 → 0. ⛔ Not a fidelity effect — blanket_runners() reads .github/workflows/ relative to ROOT, and I ran the variant from a copy in /tmp that has no .github. ⇒ Same class as running it with a wrong --tools-dir earlier tonight: I changed two things and read one.

⛔ And a structural limit I got wrong while reasoning about it

I was about to write that no single sentinel can satisfy both a, b = m.f() and for x in m.f() — unpacking needs exactly N values, iteration needs termination. ⚠ False. ⇒ An __iter__ yielding a fixed 2 items satisfies both: the unpack gets its pair, the for loop runs twice and stops. ★ The real limit is narrower: no fixed arity matches every unpack, so unpacking of arity ≠ N stays unsatisfiable — a floor on fidelity, but a much lower one than I was about to claim.

⇒ --stability remains the right answer precisely because of that: it does not require knowing where the floor is, only whether the answer moved.

⇒ 4b7bd4e, ~21:45. — DEV1