gated-caller: '1 of 36' was an undercount my own stub produced — 2, plus 24 in-process, and 28 of 51 suites bound the residual

[jobordu]

Closes #551, and corrects a figure I published four hours ago that another pane is acting on.

⛔ The cause: my recording stub was killing the callers it measured

test_prevalence.py   :27  imports the subject
                     :41  reads pv.<attr>        <- AttributeError under the old stub
                     :141 runs it BARE           <- THE SWEEP, never reached

⇒ 1 of 36 was a fact about this tool, not about the repository. ★ The stub defined none of the subject's attributes, so any caller that read one died before doing anything else.

⇒ Three changes

1. Record in-process calls. A PEP 562 module __getattr__ in the stub, which fires only for attributes the module does not define — and the stub defines none of the subject's, so every reach lands there. ⚠ Recorded on CALL, never on ACCESS: m.CHANNELS is a read and is not a sweep; m.main() is a call. Writing the record in __getattr__ itself would score the first as the second. Subprocess sweeps: 1 → 2 — prevalence.py was always swept and never visible.

2. In-process invocation is a THIRD STATE, not a wider swept. ⛔ Folding it in gave 26 of 36 — and that number is real but answers a different question: "does a gated caller invoke any function in the module", not "does a gated caller run the tool as a program". test_close_condition_scan.py calls main() with argv patched to --states, a DECLARE path that measures nothing. ⇒ Shipping 26 under the old label would have been a correct reading of the wrong proposition, under a figure #550 is currently acting on. Reported on its own line: 24 of 36 invoked in-process.

3. TRUNCATION RISK as a number, not a worry. ⚠ 28 of 51 gated suites still exit nonzero under the stub. Each may have stopped before a call it would have made ⇒ every count here is a LOWER BOUND, and this line is by how much. The exit code still does not decide caller-hood — that would anti-correlate with rigour — it only bounds the error.

 2 of 36  gated caller runs the SWEEP (subprocess)      <- was 1
24 of 36  INVOKED IN-PROCESS by a gated suite           <- different question
28 of 51  suites exit nonzero under the stub            <- the residual error
 2        instruments in subdirectories, outside all of it

⚠ What this means for anyone holding the old number

⛔ 1 of 36 should not be quoted again. ⇒ It understated subprocess sweeps by at least one, and it was silent about both in-process callers and its own truncation.

★ The gap it named is still real — 34 of 36 have no subprocess sweep caller — but it was never as clean as 1.

Controls

Pairs throughout, because no half fails alone: in-process caller is CALLED and not SWEPT; an attribute read is neither; a nonzero-exiting suite is truncation risk and a clean one is not. Mutation-checked — ignoring the CALL record fails one assertion, a stub that never records fails one.

⚠ NOT established: how much of the 28 is genuine truncation versus suites that legitimately fail under any stub. The line bounds the error; it does not attribute it.

Measured at d2cf3ac. — DEV1

[coderabbitai]

Warning

Review limit reached

@jobordu, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 46 minutes

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: de17a68f-7c7b-4e9d-a534-b834ae3e9cf7

📥 Commits

Reviewing files that changed from the base of the PR and between 8a5434a and 968bf6e.

📒 Files selected for processing (1)

• tools/gated-caller.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[jobordu]

⇒ Attributing the 28, which this PR says it cannot do — measured, and MOST OF IT IS MY STUB

DEV1. ⚠ The PR says "NOT established: how much of the 28 is genuine truncation versus suites that legitimately fail under any stub." ⇒ Measured by classifying the terminating exception:

TypeError                   15     <- the stub returns 0 where a real value is expected
no-exception (assert/exit)   5
AssertionError               5     <- legitimate: the stub has nothing real to assert on
IndexError                   1
FileNotFoundError            1
AttributeError               1
                            ---
                             28 of 51   (matches the tool exactly)

⇒ ⛔ 15 of 28 are TypeError — a suite doing m.f()[0] or iterating m.g() receives the integer 0 and dies. ★ That is my stub being impoverished, not a suite being strict, and it is the same class as the AttributeError this PR already fixes — one layer further along.

⚠ So the lower bound is looser than the honest-sounding split suggests: at most ~10 of the 28 are legitimate strictness (AssertionError + deliberate nonzero exits); the rest is the probe truncating its own subjects.

⇒ The obvious next step, and why it is NOT in this PR

A permissive sentinel — a return value supporting iteration, indexing, len, and comparison instead of a bare 0 — would keep callers alive past those 15 and tighten every count.

⛔ I am not adding it here. ⇒ This PR already corrects a published figure and adds a state; folding a second design change into it would make one review carry two independent risks. ⚠ And the tension is real: a more permissive stub is a LESS FAITHFUL one, and the stub's own header requires faithfulness "in the properties suites select on." More permissive is not automatically better — it trades truncation for divergence, and I have not measured that trade.

⚠ What this does NOT establish

• ⛔ A TypeError suite is not PROVEN to have had a later call. ⇒ It may have died after everything it would ever have recorded. The 15 bounds the truncation; it does not demonstrate it.
• ⚠ Exception attribution reads the LAST traceback in combined stdout+stderr. A suite that catches and re-raises, or prints a traceback it handled, could be misfiled.
• ⚠ I initially measured this as 45 of 51 — I stubbed every non-test .py (~56) instead of the 36 the tool actually stubs. ⛔ Wrong population, in the analysis of a population defect. The 28 above is after aligning it, and it reproduces the tool's number exactly, which is the only reason I trust it.

⇒ d2cf3ac, ~21:30. — DEV1