Skip to content

fix(code-metrics): measure type debt per file and stop reporting an aborted mypy run as 100% typed - #4064

Draft
kyle-sexton wants to merge 2 commits into
mainfrom
claude/code-metrics-audit-type-debt-pw049g
Draft

fix(code-metrics): measure type debt per file and stop reporting an aborted mypy run as 100% typed#4064
kyle-sexton wants to merge 2 commits into
mainfrom
claude/code-metrics-audit-type-debt-pw049g

Conversation

@kyle-sexton

Copy link
Copy Markdown
Contributor

Closes #4002
Closes #4003

Summary

audit-type-debt reported this repository as 100% typed over zero expressions: mypy aborted on a duplicate module name (sanctioned replication), exited 2, still wrote an empty any-exprs.txt, and the collector read its Total 0 0 100.00% row as a measurement. The same collector also left a .mypy_cache/ directory in the consumer's working tree. This PR fixes the collector (#4002) and, in a second commit set still to land, reports type debt per file with a lane total and relays mypy's error count (#4003). Plugin version 0.3.0.

Fix

Landed (#4002):

  • mypy exit 2 is now the adapter contract's exit 4: the Python run row reads unavailable carrying mypy's own message (stdout and stderr, since mypy prints error lines to stdout) and the run continues; never a percentage.
  • A Total row with zero expressions reports type_coverage_pct: null.
  • mypy runs with --explicit-package-bases, so same-named files under identifier-named directories are named by their path and measured once per copy. Two hyphenated directories with a same-named file still collide (mypy's module walk stops at a non-identifier directory) and reach the unavailable row; the Gotchas say so.
  • mypy runs with --cache-dir os.devnull, mypy's documented value for disabling the cache (/dev/null on POSIX, nul on Windows, compared by string equality), so nothing is written into the consumer's tree.
  • Tests: a captured fixture of the empty table mypy really writes on an abort; cases for exit 4 with the relayed message, null on zero expressions, the passed flags, a real-mypy run leaving no cache, and a skill-level aborted-run case. The former exit-2 test now models a report that is missing on a non-fatal exit.

Pending in this PR (#4003): per-file rows plus the lane total for both lanes, mypy's error and missing-stub counts in the run row's reason, schema doc update.

Verification

  • scripts/affected-tests.sh --run: audit-type-debt.test.sh 32 cases, 0 failed; test_mypy_report.py run from its own lane, 14 tests OK.
  • scripts/run-ruff.sh check and format --check on the two Python files: clean.
  • shellcheck and shfmt -d on the edited test script: clean. markdownlint-cli2 on SKILL.md and CHANGELOG.md: 0 issues.
  • scripts/check-changed-skills.sh origin/main: PASS (one pre-existing warning: the description sits 31 codepoints under the field cap). scripts/check-changelog-parity.sh in --check, --check-bump origin/main, --check-order, --check-preserved origin/main: all pass.
  • Real run of the fixed script on this repository, audit-type-debt.sh --json --all: exit 0, Python row ok, 14399 Any expressions over 146827, 90.19%, labelled mypy-reported-errors; no .mypy_cache/ in the tree afterwards. Before the fix the same run reported 0 over 0 at 100%.
  • Load-bearing facts (mypy exit codes, the cache-dir disable value, --explicit-package-bases module derivation, semver and Keep a Changelog rules, type-coverage --detail shape) were verified against mypy's source and docs at v1.19.1 and v2.3.1, semver.org, keepachangelog.com, the type-coverage source, and local runs, with an independent verifier pass; all six graded HIGH.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01Kn6LHhMJfke8gjy8kNdkHk


Generated by Claude Code

…icate-module abort

mypy 1.19.1 exits 2 on "Duplicate module named" before analysing anything
and still writes an any-exprs.txt whose only data row is Total 0 0 100.00%.
The collector fix and its tests replay this capture.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kn6LHhMJfke8gjy8kNdkHk
…an aborted run

mypy exits 2 on a blocking error (a duplicate module name, a usage or
config error) before analysing anything and still writes a report whose
only row is `Total 0 0 100.00%`. The collector read that row as a
measurement labelled mypy-reported-errors, so a repository carrying
sanctioned replication reported 100% typed over zero expressions.

- exit 2 is now the adapter contract's exit 4: the Python row reads
  `unavailable` with mypy's own message (stdout and stderr, since mypy
  prints error lines to stdout) and the run continues;
- a Total row with zero expressions reports type_coverage_pct null;
- mypy runs with --explicit-package-bases, so same-named files under
  identifier-named directories no longer collide (two hyphenated
  directories still do, because mypy's module walk stops at a
  non-identifier directory, and reach the unavailable row);
- mypy runs with --cache-dir os.devnull, mypy's documented value for
  disabling the cache, so no .mypy_cache is written into the tree;
- the exit-2 test fixture now replays the empty table mypy really
  writes, plus cases for null on zero expressions, the passed flags,
  and a real-mypy run leaving no cache; the skill-level suite gains the
  aborted-run case.

Facts verified against mypy's source and docs at v1.19.1 and v2.3.1.
Plugin 0.3.0.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kn6LHhMJfke8gjy8kNdkHk
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants