diff --git a/docs/CATALOG.md b/docs/CATALOG.md index 7826463635..056802c500 100644 --- a/docs/CATALOG.md +++ b/docs/CATALOG.md @@ -88,7 +88,7 @@ plugin manifests and kept in sync by CI. Never hand-edit it; the category vocabu - [`context-guard`](../plugins/context-guard): Per-session context-window observability plus the first shipped consumer: a statusline wrapper tees each session's context_window fields to a per-session snapshot file, a zone resolver classifies usage into smart/acceptable/dumb bands (percentage bands plus window-class token bands, conservative-min combination, zones.json SSOT with shipped defaults), a reader contract fixes how consuming sessions interpret the snapshots, and zone-crossing hooks report once per transition into a worse zone across two channels: the continuation menu to the operator, who owns that choice, and to the model only the zone determination plus the counter-steer that a zone word is not a decay signal (advisory by default; an optional blocking mode gates new mutating work on a fresh dumb-zone snapshot with handoff-writing exempt), with a PostCompact hook persisting an evidence-degraded marker. - [`context-budget`](../plugins/context-budget): Measure a Claude Code session's fixed startup context payload per item, on the consumer's machine at a pinned, version-stamped binary, including per-tool attribution of the built-in tool pools that /context reports only as lump sums, derived live by A/B bare-name-deny differencing with enforced comparability rules (skill-listing signature, one mode, one binary), an SDK-primary exact meter degrading to a version-aware headless /context parser and then to an honest structured error (never a wrong number), and a per-project measure-toggle-remeasure ledger under the plugin data directory recording every lever's real before/after delta. Report-only: prints exact config, applies nothing. - [`plugin-quality`](../plugins/plugin-quality): Post-use behavioral audit of Claude Code plugin components: a six-step audit workflow (evidence capture, grounded mapping in a fresh subagent, blindspot pass, interactive contract lock, presence-gated review seams, work-item emit with draft+confirm) over any skill, agent, hook, command, or config you have actually used, zone-informed by context-guard snapshots when present, conservative when not. -- [`skill-quality`](../plugins/skill-quality): Skill-authoring QA tooling: a static contract checker that runs twenty-six deterministic checks over a Claude Code skill (frontmatter, explicit invocation mode, description/verb-contract polarity, per-skill listing-entry cap, trigger-keyword preservation, line caps, broken internal refs, markdownlint, gotchas surface, evals presence, precompute opportunity, completion-criteria signal, injection shell-declaration, fresh-eyes declaration conformance), a shared skill-listing budget reporter across a set of skills, and a bundled evals.json schema plus a deterministic eval-quality lint (duplicate case identities, missing fixtures, empty or vague grading criteria, set-coverage warnings). Runs against any repo's skills directory via the convention-resolution ladder, with no baked layout. +- [`skill-quality`](../plugins/skill-quality): Skill-authoring QA tooling: a static contract checker that runs twenty-seven deterministic checks over a Claude Code skill (frontmatter, explicit invocation mode, description/verb-contract polarity, per-skill listing-entry cap, trigger-keyword preservation, line caps, broken internal refs, markdownlint, gotchas surface, evals presence, precompute opportunity, completion-criteria signal, injection shell-declaration, fresh-eyes declaration conformance), a shared skill-listing budget reporter across a set of skills, and a bundled evals.json schema plus a deterministic eval-quality lint (duplicate case identities, missing fixtures, empty or vague grading criteria, set-coverage warnings). Runs against any repo's skills directory via the convention-resolution ladder, with no baked layout. - [`computer-use`](../plugins/computer-use): Operating knowledge for Claude Code's built-in computer-use MCP server, the desktop screen-control surface. `/computer-use:diagnose` resolves a symptom to a cause instead of retrying: why every screenshot is downscaled to a fixed pixel budget and why zoom (not a bigger display) is the way back to detail, how to read a capture or input failure, and the per-OS quirks that make a synthesized key or menu behave unlike a human's. `/computer-use:setup` verifies the prerequisites the surface cannot verify for itself and reports the environment settings that end a session mid-run. - [`fleet`](../plugins/fleet): Run a Claude Code agent turn on another machine in a personal fleet, over SSH on the tailnet. Every machine signs into its own Claude account, so the built-in peer tools are same-account and never span machines; this carries the SSH path that does, with target resolution from the rendered fleet manifest, one-shot and multi-turn headless recipes, the Windows-side relay that reaches a target's own sessions, and the permission posture that keeps remote agent launches prompting. - [`instruction-placement`](../plugins/instruction-placement): Routes agent-instruction content to the surface that loads it at the right moment. The audit skill sweeps a repository's instruction layer and its ordinary markdown for content whose scope is narrower than the surface carrying it, meaning conventions keyed to one file type or one subtree sitting in an always-loaded CLAUDE.md or AGENTS.md, and for normative conventions stranded in documentation Claude never loads at all, then classifies each against a routing rubric and proposes a destination whose `paths:` glob is machine-validated before it is ever offered. Safety-class content (irreversible actions, secrets, data integrity, external publication, compliance, agent authority) is hard-denied from demotion and reported as held back rather than proposed, because demotion trades guaranteed presence for conditional presence and deferred surfaces are invisible inside subagents and absent after compaction until re-triggered. Every accepted move regenerates an always-loaded index of deferred surfaces, which is what keeps a demoted rule reachable from a subagent that never receives its injection. The audit is read-only and emits a diffable findings artifact; realignment is a separate skill gated per item with no blanket-approve path; a deterministic check skill gates that every rule glob still resolves and the index is current; and a setup skill verifies the one thing no other gate can see: that the index target is a file Claude Code will actually read, since it reads CLAUDE.md and not AGENTS.md. diff --git a/plugins/code-metrics/.claude-plugin/plugin.json b/plugins/code-metrics/.claude-plugin/plugin.json index 4adbc84074..69dcb6f3a6 100644 --- a/plugins/code-metrics/.claude-plugin/plugin.json +++ b/plugins/code-metrics/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "code-metrics", - "version": "0.2.1", + "version": "0.2.2", "description": "Read-only code measures for a change, with cited references and no verdict: lines per file (audit-size), cyclomatic, cognitive, and Halstead complexity (audit-complexity), duplication with sanctioned-replication exclusions (audit-duplication), coverage per function with CRAP from existing lcov, Cobertura, coverage.py, or Go artifacts (audit-coverage), type debt for TypeScript and Python (audit-type-debt), the literacy router for what each number can and cannot say (principles), and a setup skill for the consumer's .claude/code-metrics.yaml. Runs external collectors only when they already resolve, never installs, never runs tests, never emits a finding.", "author": { "name": "Melodic Software", diff --git a/plugins/code-metrics/CHANGELOG.md b/plugins/code-metrics/CHANGELOG.md index 3589960b03..b55e2e4469 100644 --- a/plugins/code-metrics/CHANGELOG.md +++ b/plugins/code-metrics/CHANGELOG.md @@ -3,6 +3,59 @@ All notable changes to the `code-metrics` plugin are documented here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning. +## [0.2.2] + +### Fixed + +- **`principles`: the Halstead split-file answer overstated what the formula supports.** The + quick guide said difficulty "should not have" moved when a file was split, and eval 1 expected + the same. Difficulty `(n1/2) * (N2/n2)` carries no explicit length term, but both factors change + per half on a split, and a radon run on two functions measured together and apart gave 1.667 for + the whole against 1.000 and 1.800 for the halves. The entry and the eval now say per-file + difficulty legitimately moves on a split, in either direction. +- **`principles`: the §8.2.115 reading is labelled as the plugin's.** thresholds.md, measures.md, + and the configuration reference presented "a function's non-empty lines as a percentage of the + file's" as the clause's words. The clause states `MaxNumberOfNonEmptyLinesOfCode` with a default + of "5%" and names no base; the percentage-of-file base is this plugin's reading, and thresholds.md + now carries the four-part verification record for it (OMG ASCQM v1.1 and the ISO edition, as of + 2026-09-11, recheck on a new revision). The same files now name ISO/IEC 5055:2021 as the ISO + publication of ASCQM v1.0, with v1.1 identical for the cited clauses, and note that the + document's informative CWE summary rows carry different defaults (1000 lines per file, 10%) from + its detection patterns (5%, 90%). +- **`principles`: McCabe's framing and Campbell's switch rule are quoted as written.** McCabe 1976 + frames cyclomatic complexity for modules that are "testable and maintainable", not testability + alone; Campbell v1.7 states "a switch and all its cases combined incurs a single structural + increment". measures.md and literature.md carry both verbatim. + +### Changed + +- **`principles`: the quick guide answers "which measure should I look at" with an intent-keyed + tree**, each branch grounded in its primary: testing burden to cyclomatic (McCabe; NIST SP 500-235 + sets the test count equal to it), readability to cognitive (Campbell), diff size and copying to + lines per file and duplication (ISO/IEC 5055 CWE-1080 and CWE-1041), and whether a suite would + catch a fault to the mutation-testing presence gate, because coverage records execution and the + primary literature disagrees on how well it predicts fault detection (Inozemtseva and Holmes + 2014 against Gopinath, Jensen and Groce 2014 and Kochhar, Thung and Lo 2015). A duplication entry + states that no reference ships and that the percentage moves with `duplication.min_tokens`, and + the routing table names the plugin's report-schema reference for the report vocabulary. +- **`principles`: the no-verdict rule is stated once**, at the top of the skill body, and the + reference files no longer cite the marketplace's ADR by number, which a consumer of the installed + plugin cannot read. +- **`principles`: the reference files state present-tense facts and carry no research narrative.** + The thresholds file's account of how ten candidate values were commissioned from a social post, + scrutinized at an interview, and full-text searched is replaced by a table of popular numbers with + no found source, naming what was checked and what was not; literature.md states each source's + confidence and its basis without narrating the pass that established it. For the record, that + candidate list was 22 (cyclomatic), 22 (cognitive), 80 (Halstead difficulty), 500 (lines per + file), 100 (coverage), 25 (CRAP), and four zeros for count-based concerns; 20 and 1000 survived + as shipped defaults because a citation exists for them, and the rest traced to no source. +- **`principles`: a `## Next` section** names the audit skill for the measure in question and + `/code-metrics:setup` for setting the reader's own reference values, in the mention-only shape + the sibling skills use. +- **`principles`: literature.md gains a coverage-and-test-effectiveness section** citing the five + primaries above with their DOIs, and a duplication row in the thresholds table records that no + duplication reference ships and why. + ## [0.2.1] ### Changed diff --git a/plugins/code-metrics/reference/config.md b/plugins/code-metrics/reference/config.md index fe2f7a4e11..8751fd8654 100644 --- a/plugins/code-metrics/reference/config.md +++ b/plugins/code-metrics/reference/config.md @@ -63,9 +63,9 @@ The third column is written by hand and is not derived from anything. A row whos | `complexity.cyclomatic.reference` | `20` | ISO/IEC 5055:2021 §8.2.117 (normative). Cited alternatives: 10 (McCabe 1976, "reasonable, but not magical") and 15 (NIST SP 500-235, with its six practices) | | `complexity.cognitive.reference` | `null` | Campbell, SonarSource; no standard sets a threshold | | `complexity.halstead.difficulty` | `null` | Halstead 1977; no standard sets a threshold | -| `size.mode` | `file-lines` | `file-lines` compares each file's non-blank lines to `size.file_lines`; `iso-8.2.115` adds each function's non-empty lines as a percentage of the file's, from a collector that reports function ranges | -| `size.file_lines` | `1000` | The plugin's own number. It coincides with an informative figure in ISO/IEC 5055:2021 §6.3 Table 1, which is not normative; 500, the operator-list figure, is selectable | -| `size.function_lines_pct` | `5` | ISO/IEC 5055:2021 §8.2.115 (normative), used in `iso-8.2.115` mode | +| `size.mode` | `file-lines` | `file-lines` compares each file's non-blank lines to `size.file_lines`; `iso-8.2.115` adds each function's non-empty lines as a percentage of the file's (the plugin's reading of the clause, which states 5% with no base), from a collector that reports function ranges | +| `size.file_lines` | `1000` | The plugin's own number. It coincides with an informative figure in ISO/IEC 5055:2021 §6.3 Table 1, which is not normative; 500 is selectable | +| `size.function_lines_pct` | `5` | ISO/IEC 5055:2021 §8.2.115 (normative) states the 5%; the percentage-of-file base is the plugin's reading; used in `iso-8.2.115` mode | | `duplication.min_tokens` | `50` | Passed to the clone collector | | `duplication.min_lines` | `5` | Passed to the clone collector | | `duplication.ignore` | `[]` | Collector ignore globs | diff --git a/plugins/code-metrics/scripts/config-defaults.json b/plugins/code-metrics/scripts/config-defaults.json index bcc08a6206..61e23fd970 100644 --- a/plugins/code-metrics/scripts/config-defaults.json +++ b/plugins/code-metrics/scripts/config-defaults.json @@ -65,14 +65,14 @@ "config_key": "size.file_lines", "value_key": "lines_non_blank", "direction": "at_or_above", - "provenance": "the plugin's own number, not ISO-backed: it coincides with the informative figure in ISO/IEC 5055:2021 §6.3 Table 1, which is not normative, while the normative form (§8.2.115) is a function-level percentage; 500 is the operator-list figure, selectable; size.mode iso-8.2.115 selects the normative function-percentage alternative" + "provenance": "the plugin's own number, not ISO-backed: it coincides with the informative figure in ISO/IEC 5055:2021 §6.3 Table 1, which is not normative, while the normative form (§8.2.115) is a function-level percentage; 500 is selectable; size.mode iso-8.2.115 selects the normative function-percentage alternative" }, { "measure": "function_lines_pct", "config_key": "size.function_lines_pct", "value_key": "function_lines_pct", "direction": "at_or_above", - "provenance": "ISO/IEC 5055:2021 §8.2.115 (normative): a function whose non-empty lines exceed this percentage of the file's; used when size.mode is iso-8.2.115" + "provenance": "ISO/IEC 5055:2021 §8.2.115 (normative) states the 5% with no base; the percentage-of-file reading is the plugin's; used when size.mode is iso-8.2.115" }, { "measure": "coverage", diff --git a/plugins/code-metrics/skills/audit-size/SKILL.md b/plugins/code-metrics/skills/audit-size/SKILL.md index 7a38f03450..2891d340c6 100644 --- a/plugins/code-metrics/skills/audit-size/SKILL.md +++ b/plugins/code-metrics/skills/audit-size/SKILL.md @@ -59,9 +59,8 @@ side as INCONCLUSIVE); otherwise keep the JSON beside your notes and compare by - The reference compares against `lines_non_blank`. The bundled default is 1000, and its provenance is the one sentence the report prints beside it: the plugin's own number, not ISO-backed: it coincides with the informative figure in ISO/IEC 5055:2021 §6.3 Table 1, which is - not normative, while the normative form (§8.2.115) is a function-level percentage; 500 is the - operator-list figure, selectable; size.mode iso-8.2.115 selects the normative - function-percentage alternative. + not normative, while the normative form (§8.2.115) is a function-level percentage; 500 is + selectable; size.mode iso-8.2.115 selects the normative function-percentage alternative. - A `null` value means the collector did not produce it (the bundled counter has no comment count); it is never zero. - `status` is `complete` when every lane in scope was measured, `partial` when one was not, and diff --git a/plugins/code-metrics/skills/principles/SKILL.md b/plugins/code-metrics/skills/principles/SKILL.md index f86270f4c8..c73628dbb4 100644 --- a/plugins/code-metrics/skills/principles/SKILL.md +++ b/plugins/code-metrics/skills/principles/SKILL.md @@ -12,46 +12,59 @@ metadata: The five `audit-*` skills in this plugin print numbers with their provenance and stop. This skill says what each number means, where its reference value came from, and where reading it as a verdict -goes wrong. Reference files in `reference/` are source-attributed; the sections below answer the -common questions without a load. +goes wrong. It renders no verdict of its own, and neither does the plugin: a reference is a value to +count against, never a bar, and no skill here emits a finding, a severity, or an exit-code gate, +because a finding needs a measured corpus sweep behind it and none has been run. Reference files in +`reference/` are source-attributed; the sections below answer the common questions without a load. ## Routing table | Query about... | Load | |---|---| -| What a measure is, its formula, per-lane collector behavior, artifact formats | [measures.md](reference/measures.md) | -| Where a reference value came from, the operator's original list, how to set your own | [thresholds.md](reference/thresholds.md) | +| What a measure is, its formula, per-lane collector behavior, coverage artifact formats | [measures.md](reference/measures.md) | +| Where a reference value came from, which popular numbers have no source, how to set your own | [thresholds.md](reference/thresholds.md) | | The CRAP formula, its two authorial names, the coverage join, what it does not predict | [crap.md](reference/crap.md) | | Who published what, what a paper claims and does not claim, confidence per source | [literature.md](reference/literature.md) | +| The report vocabulary (`cov_source`, `not-applicable`, `partial`) and the JSON document shape | `${CLAUDE_PLUGIN_ROOT}/reference/report-schema.md` | Load the one file the question lands in. Load a second only when the first leaves the question open. ## Quick decision guide (no file load needed) -- **"Which measure should I look at?"**. The one attached to the change in front of you. Complexity - and size are per function or per file and move with an edit; duplication and coverage are - properties of the tree around it. No measure here ranks changes by risk. -- **"Is 22 the cyclomatic threshold?"**. No source found in this plugin's research attributes 22 to - anyone. The bundled reference is 20, from ISO/IEC 5055:2021 §8.2.117, with 10 (McCabe 1976) and - 15 (NIST SP 500-235) as the cited alternatives you can select. +- **"Which measure should I look at?"**. Start from the question, not from the report. How much + testing a function needs is cyclomatic complexity: McCabe 1976 defines it as the size of a basis + set of paths, and NIST SP 500-235 makes that the number of tests a module needs. How hard a + function is to read is cognitive complexity: Campbell's measure penalizes nesting and charges a + `switch` and all its cases once. Whether a diff is too large or copied is lines per file and + duplication over the changed files, the two measures ISO/IEC 5055 files as weaknesses (CWE-1080 + and CWE-1041). Whether the suite would catch a fault is not a coverage question: coverage records + which lines executed, the primary literature disagrees on how well that predicts fault detection, + and the owner of that question is behind the mutation-testing gate below. No measure here ranks + changes by risk. +- **"Is 22 the cyclomatic threshold?"**. No source this plugin cites attributes 22 to anyone. The + bundled reference is 20, from ISO/IEC 5055:2021 §8.2.117, with 10 (McCabe 1976) and 15 (NIST SP + 500-235) as the cited alternatives you can select. - **"What is a good coverage number?"**. No standard sets one, so this plugin ships no coverage reference. A percentage counts executed lines; it says nothing about whether anything was checked. +- **"Is 8% duplication bad?"**. No duplication reference ships, so the report counts nothing as + over. The percentage moves with `duplication.min_tokens`, so two runs compare only at the same + floor, and replication the repository declares in a registry is excluded rather than suppressed. - **"Is CRAP a real metric?"**. It is real and authored (Savoia and Evans, 2007), and it is not a validated change-risk predictor. See [crap.md](reference/crap.md) before quoting a CRAP number. - **"Is this file too long at 1200 lines?"**. The report tells you it is at or above the reference - of 1000, which is this plugin's own number. ISO/IEC 5055's normative file-size clause is a - percentage of a function against its file, not a line count. + of 1000, which is this plugin's own number. ISO/IEC 5055's normative clause for the same weakness + is a percentage on a function, not a line count on a file. - **"Cyclomatic or cognitive?"**. Cyclomatic counts independent paths through a function, which is what makes it a testing measure. Cognitive weights nesting and forgives a flat `switch`, which is what makes it a readability measure. They disagree on the same function by design. -- **"Halstead difficulty rose when I split the file. Why?"**. It should not have. Difficulty has no - length term: it is operator variety times operand reuse. A difficulty that tracks file size is - measuring something else, so check which collector produced it. +- **"Halstead difficulty rose when I split the file. Why?"**. Difficulty is `(n1/2) * (N2/n2)`, + operator variety times operand reuse, with no explicit length term, so it does not track size. A + split changes each half's operator count and reuse ratio, so per-file difficulty legitimately + moves on a split, in either direction. Check which collector produced the row: `multimetric` + reports Halstead per file and `radon` per function. - **"Type coverage is 96%. Is that good?"**. No standard and no CWE anchors the measure, so there is no external answer. The percentage also counts `unknown` as typed, so an `any`-to-`unknown` sweep raises it without adding type information. -- **"Why does the report never fail?"**. A reference here is a value to count against, never a bar. - This version reports measures and emits no finding, no severity, and no gate exit code. ## Cross-metric caveats @@ -114,8 +127,11 @@ This plugin does not measure these. Each one has an owner, and each pointer is g the Crap4j project FAQ. - **Lewis et al.**: "Does Bug Prediction Support Human Developers? Findings from a Google Case Study", ICSE 2013. -- **ISO/IEC 5055:2021** (adopted from OMG ASCQM; current OMG release v1.1) and **ISO/IEC - 25023:2016**, cited by clause and version, never by page number. +- **Coverage and test effectiveness**: Inozemtseva and Holmes, ICSE 2014; Gopinath, Jensen and + Groce, ICSE 2014; Kochhar, Thung and Lo, SANER 2015; Jia and Harman, IEEE TSE 2011; Petrovic and + Ivankovic, ICSE-SEIP 2018; Petrovic, Ivankovic, Fraser and Just, IEEE TSE 2022. +- **ISO/IEC 5055:2021** (the ISO publication of OMG ASCQM v1.0; OMG's current release is v1.1) and + **ISO/IEC 25023:2016**, cited by clause and version, never by page number. Per-source annotation, including what each does not claim and the confidence behind it, is in [literature.md](reference/literature.md). @@ -123,10 +139,15 @@ Per-source annotation, including what each does not claim and the confidence beh ## Scope boundary This skill is **knowledge**, not **workflow**. It measures nothing, runs no collector, and reads no -repository. The `audit-*` skills in this plugin produce the numbers; this one explains them. It also -renders no verdict on your code: a reference value with a citation is the most it offers, and +repository. The `audit-*` skills in this plugin produce the numbers; this one explains them, and deciding what to do about a number is the reader's call. +## Next + +- The number in question still has to be measured: `/code-metrics:audit-complexity`, or the + `audit-*` skill for that measure. +- The bundled reference values are not the ones your codebase can defend: `/code-metrics:setup`. + ## Gotchas - Provenance strength varies inside one report. The cyclomatic reference is normative in a diff --git a/plugins/code-metrics/skills/principles/evals/evals.json b/plugins/code-metrics/skills/principles/evals/evals.json index afd9e5db00..63c3dfdb69 100644 --- a/plugins/code-metrics/skills/principles/evals/evals.json +++ b/plugins/code-metrics/skills/principles/evals/evals.json @@ -5,11 +5,11 @@ "id": 1, "name": "routes-a-definition-question-to-one-reference-file", "prompt": "/code-metrics:principles the report says Halstead difficulty 41 for a file I just split in two. What does that number mean?", - "expected_output": "Answers from the routing table by loading reference/measures.md and no other file: difficulty is (n1/2) * (N2/n2), operator variety times operand reuse, with no length term, so splitting a file should not move it; multimetric reports Halstead per file rather than per function, so the row is file-level. Cites Halstead 1977 and states that no standard sets a difficulty threshold.", + "expected_output": "Answers from the routing table by loading reference/measures.md and no other file: difficulty is (n1/2) * (N2/n2), operator variety times operand reuse, with no explicit length term, so it does not track size; a split changes each half's operator count and reuse ratio, so per-file difficulty legitimately moves on a split, in either direction. multimetric reports Halstead per file rather than per function, so the row is file-level. Cites Halstead 1977 and states that no standard sets a difficulty threshold.", "files": [], "expectations": [ "Loads reference/measures.md first and does not load all four reference files", - "States the difficulty formula and that it carries no length term", + "States the difficulty formula, that it carries no explicit length term, and that a split can move per-file difficulty because each half's operator count and operand-reuse ratio change", "Names the collector behaviour: Halstead from multimetric is per file, not per function", "Says no standard sets a Halstead threshold rather than suggesting a number of its own", "Renders no verdict on whether 41 is acceptable" diff --git a/plugins/code-metrics/skills/principles/reference/crap.md b/plugins/code-metrics/skills/principles/reference/crap.md index 33ea243ba9..bfedcc9174 100644 --- a/plugins/code-metrics/skills/principles/reference/crap.md +++ b/plugins/code-metrics/skills/principles/reference/crap.md @@ -65,8 +65,9 @@ behind it. What the record supports: - The formula, the variable definitions, and the authors' own suggested cutoff of 30 are documented by the authors. The cutoff is presented as a judgement call after "much debate", not as a result. -- No validation study for CRAP surfaced in this plugin's literature pass. The absence is what the - claim rests on, so read it as "no supporting evidence found", not as a refutation. +- No validation study for CRAP is cited by its authors, and none appears among the sources + [literature.md](literature.md) lists. The absence is what the claim rests on, so read it as "no + supporting evidence found", not as a refutation. - The surrounding literature argues against fixed thresholds generally. Nagappan, Ball and Zeller (ICSE 2006) found no single set of complexity metrics that acts as a universally best defect predictor, and Majumder, Mody and Menzies (EMSE 2022, 700 projects) warn that metric-importance diff --git a/plugins/code-metrics/skills/principles/reference/literature.md b/plugins/code-metrics/skills/principles/reference/literature.md index 0b7c9c8af7..0b72c54dc3 100644 --- a/plugins/code-metrics/skills/principles/reference/literature.md +++ b/plugins/code-metrics/skills/principles/reference/literature.md @@ -1,21 +1,22 @@ # Annotated bibliography -Every source this plugin cites, what it claims, what it does not claim, and how strongly it was -verified. Confidence is the level the plugin's own research pass recorded: HIGH where a primary was -downloaded and read, MEDIUM where the terminal source was a preview, a secondary, or an authority -record standing in for a paywalled body. +Every source this plugin cites, what it claims, what it does not claim, and how strongly the +citation is backed. HIGH means the primary text is read directly; MEDIUM means the terminal source +is a preview, a secondary, or an authority record standing in for a paywalled or unavailable body. ## McCabe 1976, cyclomatic complexity T. J. McCabe, "A Complexity Measure", IEEE Transactions on Software Engineering SE-2(4), December -1976, pp. 308-320. Primary downloaded and full-text searched. **HIGH.** +1976, pp. 308-320, DOI 10.1109/TSE.1976.233837. Primary read. **HIGH.** - **Claims.** A control-flow-graph measure, `v(G) = e - n + 2p`, whose value is the number of - linearly independent paths through a module. Reports an operational upper bound of 10 in use at - the author's organization, with an explicit exception for a large case statement. + linearly independent paths through a module, "the size of a basis set". The stated purpose is + modularizing software so the modules are "testable and maintainable", and the paper pairs the + measure with a testing methodology. Reports an operational upper bound of 10 in use at the + author's organization, with an explicit exception for a large case statement. - **Does not claim.** That 10 is derived, optimal, or validated: the paper calls it "a reasonable, - but not magical, upper limit". It sets no other threshold, defines no bands, and says nothing - about maintainability or readability, only testability and module size. + but not magical, upper limit". It sets no other threshold, defines no bands, and offers no model + of reading effort or nesting; that gap is what Campbell's measure addresses. - **Two propagated errors worth avoiding.** Several bibliographic aggregators render the issue as July 1976; the paper's own masthead says December 1976. And the author was at the Department of Defense, National Security Agency, not NIST; the NIST connection is twenty years later. @@ -23,45 +24,52 @@ T. J. McCabe, "A Complexity Measure", IEEE Transactions on Software Engineering ## Watson and McCabe 1996, NIST SP 500-235 Arthur H. Watson and Thomas J. McCabe, *Structured Testing: A Testing Methodology Using the -Cyclomatic Complexity Metric*, NIST Special Publication 500-235, September 1996. Primary downloaded -and full-text searched from two hosts. **HIGH.** - -- **Claims.** Restates the metric and its variants cleanly, reaffirms "the original limit of 10 as - proposed by McCabe", and allows that "limits as high as 15 have been used successfully as well", - reserved for projects with six named advantages: experienced staff, formal design, a modern - programming language, structured programming, code walkthroughs, and a comprehensive test plan. +Cyclomatic Complexity Metric*, NIST Special Publication 500-235, September 1996. Primary read from +NIST's own host. **HIGH.** + +- **Claims.** Restates the metric and its variants, sets the number of tests a module needs equal + to its cyclomatic complexity (basis path testing), reaffirms "the original limit of 10 as proposed + by McCabe", and allows that "limits as high as 15 have been used successfully as well", reserved + for projects with six named advantages: experienced staff, formal design, a modern programming + language, structured programming, code walkthroughs, and a comprehensive test plan. - **Does not claim.** That 15 is unconditional. The document calls the precise limit "somewhat controversial" and frames the relaxation as an organization deciding it "knows what it is doing" - and accepting the extra testing effort. -- **Also does not contain 22.** A full-text search of this document, twice, found no threshold of - 22 anywhere. + and accepting the extra testing effort. It shares an author with McCabe 1976, so the two are one + authorship pool rather than independent corroboration of each other. +- **Contains no threshold of 22.** ## Halstead 1977, software science -Maurice H. Halstead, *Elements of Software Science*, Elsevier North-Holland, 1977. The book itself -was not read; the origin is confirmed through an authority record and the formulas through vendor -implementation documentation. **HIGH for the formulas, HIGH for the origin via the authority -record.** +Maurice H. Halstead, *Elements of Software Science*, Elsevier North-Holland, 1977, ISBN +0-444-00205-7. No online full text exists and the book is not read here; the bibliographic data is +confirmed through an authority record, and the formulas through independent reproductions that all +attribute them to the book: NASA NTRS N90-14803, IBM Rational Asset Analyzer documentation, and the +`radon` documentation, whose implementation computes the same difficulty as the formula on any +input. **MEDIUM for the book as a primary; HIGH for the formulas by consensus of reproductions.** - **Claims.** A family of measures derived from counts of distinct and total operators and operands: vocabulary, length, volume, difficulty `D = (n1/2) * (N2/n2)`, effort `E = D * V`, and estimates of programming time and delivered bugs. -- **Does not claim.** Any threshold, for difficulty or for anything else. Nor does difficulty - measure size: it carries no length term, so an implementation whose difficulty scales with file - size is not computing this. +- **Does not claim.** Any threshold, for difficulty or for anything else. Nor does difficulty carry + an explicit length term: it depends on the distinct-operator count and the operand-reuse ratio, + both of which change per half when a file is split, so per-file difficulty moves on a split even + though it does not track size. ## Campbell, cognitive complexity -G. Ann Campbell, SonarSource S.A., *Cognitive Complexity: a new way of measuring understandability*. -The version read was 1.7, dated 29 August 2023, served from SonarSource's own site; earlier versions -of the same paper exist and were not read. **HIGH for the content of version 1.7.** +G. Ann Campbell, SonarSource S.A., *Cognitive Complexity: a new way of measuring understandability*, +version 1.7, 29 August 2023, served from SonarSource's own site; earlier versions of the same paper +exist and are not the version cited. **HIGH for the content of version 1.7.** - **Claims.** That cyclomatic complexity measures testability well and maintainability poorly, that it predates modern language structures such as `try`/`catch` and lambdas, and that it is "of little use above the method level". Proposes a measure that abandons the graph model, increments - on structures that interrupt linear reading, and penalizes nesting. + on structures that interrupt linear reading, and penalizes nesting. On `switch`, verbatim: "A + switch and all its cases combined incurs a single structural increment." - **Does not claim.** Any threshold. The default of 15 belongs to SonarSource's rule `S3776`, a product decision by the same vendor, and citing the paper for that number misattributes it. + Independent implementations (gocognit, PMD's CognitiveComplexity rule) follow the same increment + rules and cite the paper. ## Savoia and Evans 2007, CRAP @@ -80,8 +88,8 @@ publications read directly. **HIGH for the formula, the rename, and the suggeste Chris Lewis, Zhongpeng Lin, Caitlin Sadowski, Xiaoyan Zhu, Rong Ou and E. James Whitehead Jr., "Does Bug Prediction Support Human Developers? Findings from a Google Case Study", ICSE 2013, pp. 372-381. -The quoted phrase was confirmed verbatim from Google Research's own publication record and the -mechanism from the paper's full text. **HIGH.** +The quoted phrase is confirmed against Google Research's own publication record and the mechanism +against the paper's full text. **HIGH.** - **Claims.** A bug-prediction algorithm was deployed across Google and produced "no identifiable change in developer behavior". The stated reason, from the paper: "unless there was an actionable @@ -94,8 +102,8 @@ mechanism from the paper's full text. **HIGH.** ## Nagappan, Ball and Zeller 2006; Majumder, Mody and Menzies 2022 Two independent studies, sixteen years apart, that together carry the argument against shipping a -fixed default as validated. **MEDIUM-HIGH as sourced from abstracts, each then confirmed from an -open full text.** +fixed default as validated. **MEDIUM-HIGH: sourced from the abstracts and confirmed against open +full texts.** - Nagappan, Ball and Zeller, "Mining Metrics to Predict Component Failures", ICSE 2006, pp. 452-461, across five Microsoft systems: there is no single set of complexity metrics that acts as a @@ -109,25 +117,61 @@ open full text.** rather than strictly project-to-project transfer. What they support is narrower and enough: a threshold shipped as a default is a starting point, not a validated bar. +## Coverage and test effectiveness + +Five primaries, each read from an author-hosted or publisher-hosted full text and each confirmed +against its Crossref record. They disagree with each other, and the disagreement is the finding. +**HIGH for what each paper says; the question itself is contested.** + +- Inozemtseva and Holmes, "Coverage Is Not Strongly Correlated with Test Suite Effectiveness", + ICSE 2014, DOI 10.1145/2568225.2568271. Over 31,000 suites across five Java systems: a low to + moderate correlation between coverage and effectiveness once suite size is controlled for, and the + conclusion that coverage "should not be used as a quality target because it is not a good + indicator of test suite effectiveness". Effectiveness is measured as mutant kill score. +- Gopinath, Jensen and Groce, "Code Coverage for Suite Evaluation by Developers", ICSE 2014, DOI + 10.1145/2568225.2568278. In a per-project prediction setting, statement coverage predicts mutant + kills best of the coverage criteria compared, with high correlation on the original suites. +- Kochhar, Thung and Lo, "Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs + in Large Systems", SANER 2015, DOI 10.1109/SANER.2015.7081877. Against 159 real bugs rather than + mutants: coverage is "moderately to strongly correlated" with effectiveness, and the paper notes + that mutants "do not necessarily represent real bugs". +- Jia and Harman, "An Analysis and Survey of the Development of Mutation Testing", IEEE TSE 37(5), + 2011, DOI 10.1109/TSE.2010.62: the mutation adequacy score "can be used to measure the + effectiveness of a test set in terms of its ability to detect faults". +- Petrovic and Ivankovic, "State of Mutation Testing at Google", ICSE-SEIP 2018, DOI + 10.1145/3183519.3183521, and Petrovic, Ivankovic, Fraser and Just, "Practical Mutation Testing at + Scale: A view from Google", IEEE TSE 48(10), 2022 (early access 2021), DOI 10.1109/TSE.2021.3107634: + mutation analysis is "widely considered one of the strongest test-adequacy criteria". The two + papers share an authorship pool. +- **What this plugin takes from them.** Coverage records which lines executed, not whether an + assertion would catch a fault. Whether it predicts fault detection is contested between primaries, + so no coverage reference ships and no coverage number is read as a quality score. Mutation + analysis is the measure those studies use as ground truth, and it belongs to the mutation-testing + plugin, behind the presence gate `SKILL.md` names. + ## ISO/IEC 5055:2021 and OMG ASCQM ISO/IEC 5055:2021, *Information technology - Software measurement - Software quality measurement - -Automated source code quality measures*, Edition 1, published 2021-03-30, adopted from the OMG -Automated Source Code Quality Measures specification. The OMG PDF (288 pages, the v1.0 text dated -October 2020) was downloaded and searched exhaustively; OMG's current release is v1.1, July 2022, -diffed for the clauses this plugin cites. Cite by version and clause, never by page number. **HIGH.** +Automated source code quality measures*, first edition 2021-03, is the ISO publication of the OMG +Automated Source Code Quality Measures specification v1.0 (October 2020). OMG's current release is +v1.1 (formal/2022-07-01, July 2022); the two carry identical text for every clause this plugin +cites. Both OMG-hosted PDFs are read directly. Cite by version and clause, never by page number. +**HIGH.** - **Claims.** Clause 7 (normative) lists weaknesses, including 7.1.5 dead code (CWE-561), 7.1.10 excessive cyclomatic complexity (CWE-1121), 7.1.18 redundant code (CWE-1041), and 7.1.26 an excessively large source file (CWE-1080). Clause 8 (normative) gives the detection patterns and - their default measurement parameters, among them §8.2.117 cyclomatic complexity 20 and §8.2.115 a - function size of 5% of a file's non-empty lines. Clause 6 is informative and its Table 1 carries - the 1000-line figure. + their default measurement parameters, among them §8.2.117 cyclomatic complexity 20, §8.2.115 a + function's non-empty lines with a default stated as 5% and no base named, and §8.2.116 similar + code between two functions at 90%. Clause 6 is informative, and its §6.3 Table 1 summary rows + carry different defaults from the patterns: 1000 lines per file for CWE-1080 and 10% for CWE-1041. + MITRE's CWE entries corroborate the attachments from the other side through the ASCMM node IDs + (CWE-1080 is ASCMM-MNT-8, CWE-1041 is ASCMM-MNT-19). - **Does not claim.** A 1000-line normative default, a threshold anywhere in clause 7, or any - coverage, mutation, Halstead, or cognitive-complexity measure: a search of the full text returns - zero hits for each of those four. It also contains no weakness for weak or unsound type - declaration, and neither does MITRE's CWE-136 Type Errors category, whose complete membership is - three weaknesses about mishandling types at the point of use. + coverage, mutation, Halstead, or cognitive-complexity measure: the full text contains none of + those four. It also contains no weakness for weak or unsound type declaration, and neither does + MITRE's CWE-136 Type Errors category, whose complete membership is three weaknesses about + mishandling types at the point of use. - **A trap worth naming.** CISQ's public standards page describes the measures in informal prose labels ("High cyclomatic complexity", "Excessive component size") that are not clause titles or usage names. Summarizing that page produces plausible names that do not exist in the standard. @@ -135,23 +179,23 @@ diffed for the clauses this plugin cites. Cite by version and clause, never by p ## ISO/IEC 25023:2016 ISO/IEC 25023:2016, *Systems and software engineering - SQuaRE - Measurement of system and software -product quality*. Only the official free preview was read; the normative body is paywalled. +product quality*. Only the official free preview is available; the normative body is paywalled. **MEDIUM, preview-sourced.** - **Claims, from the preview.** The clause structure, including 8.6 Reliability with 8.6.1 Maturity, and the measure-id grammar from which `RMa-4-S` is a well-formed id for a fourth, Specific-category Maturity measure. -- **Not established.** That `RMa-4-S` is named "Test coverage". The reproduction saying so is an - unauthorized scan reached through a search summary and was never fetched. Even taken at face - value, its measurement function counts capabilities, operational scenarios, or functions performed - against those included in the test suites, which is not line coverage from a coverage tool. +- **Not established.** That `RMa-4-S` is named "Test coverage". Reproductions that say so are not + the standard's text. Even taken at face value, the measurement function they describe counts + capabilities, operational scenarios, or functions performed against those included in the test + suites, which is not line coverage from a coverage tool. - **Recency.** The standard is mid-revision as ISO/IEC DIS 25000-23.2, which replaces it, so any id quoted today may be renumbered on publication. ## Mutation-testing literature -Deliberately not summarized here. Invoke `/mutation-testing:principles` when the mutation-testing -plugin is installed; it owns the primary sources, the vocabulary, and what a surviving mutant means. -Otherwise the concern is out of this plugin's scope and nothing here substitutes for it: coverage -tells you a line executed and cannot tell you it was checked, and no measure in this plugin closes -that gap. +Deliberately not summarized beyond the coverage section above. Invoke `/mutation-testing:principles` +when the mutation-testing plugin is installed; it owns the primary sources, the vocabulary, and what +a surviving mutant means. Otherwise the concern is out of this plugin's scope and nothing here +substitutes for it: coverage tells you a line executed and cannot tell you it was checked, and no +measure in this plugin closes that gap. diff --git a/plugins/code-metrics/skills/principles/reference/measures.md b/plugins/code-metrics/skills/principles/reference/measures.md index ffe20fd469..a5d7db3216 100644 --- a/plugins/code-metrics/skills/principles/reference/measures.md +++ b/plugins/code-metrics/skills/principles/reference/measures.md @@ -18,10 +18,11 @@ reduces to `v(G) = e - n + 2`. Two equivalent statements from NIST SP 500-235 ar hand: when every decision is binary and there are `p` binary decision predicates, `v(G) = p + 1`; and for a planar flow graph, `v(G)` equals the number of regions including the infinite one. -What it counts is the number of linearly independent paths, which is why McCabe framed it as a -testability measure: it is the size of a basis set of paths a test suite would have to exercise. It -says nothing directly about how hard a function is to read. The paper's own caveat about large case -statements is in [thresholds.md](thresholds.md). +What it counts is the number of linearly independent paths, "the size of a basis set" a test suite +would have to exercise. McCabe's own purpose for it was modularizing software so the modules are +"testable and maintainable", and NIST SP 500-235 sets the number of tests a module needs equal to +it. It carries no model of nesting or reading effort, which is the gap Campbell's measure addresses. +The paper's own caveat about large case statements is in [thresholds.md](thresholds.md). ## Cognitive complexity @@ -33,10 +34,10 @@ white paper version 1.7, 29 August 2023. The paper's abstract states the motive > underlying mathematical model is unsatisfactory at producing a value that measures the latter. Cognitive complexity abandons the graph model. It increments on structures that break linear reading -and adds a nesting penalty, so a deeply nested loop costs more than a flat one, while a `switch` -with many cases costs once rather than once per case. Two functions with equal cyclomatic complexity -routinely carry different cognitive complexity, which is the point of the measure rather than a -defect in either. +and adds a nesting penalty, so a deeply nested loop costs more than a flat one, while, in the +paper's words, "a switch and all its cases combined incurs a single structural increment". Two +functions with equal cyclomatic complexity routinely carry different cognitive complexity, which is +the point of the measure rather than a defect in either. The white paper prescribes no threshold. SonarSource's rule `S3776` ships a configurable default of 15, and that is a product decision by the same vendor, not a claim the paper makes. @@ -64,9 +65,11 @@ conflated: Difficulty reads as half the operator vocabulary times the average reuse of each operand. Doubling a program's length without changing its operator set or its operand-reuse ratio leaves difficulty -unchanged while volume and effort both rise. A difficulty figure that scales with file size is -measuring something else. Effort is defined in terms of difficulty, so it is not an independent -alternative to it. +unchanged while volume and effort both rise, so difficulty does not track size. It is not +independent of how a file is cut, though: splitting or merging files changes each part's distinct +operators and its operand reuse, so per-file difficulty legitimately moves on a split, in either +direction. Effort is defined in terms of difficulty, so it is not an independent alternative to +it. ## Lines per file, and the ISO function-percentage form @@ -75,14 +78,16 @@ total plus non-blank from the bundled counter otherwise. The comparison runs aga lines. ISO/IEC 5055:2021 files a large-file weakness at 7.1.26 (CWE-1080, usage name "Excessively large -file"), and its informative clause 6.3 Table 1 carries a 1000-line figure. The normative detection -pattern attached to that weakness, §8.2.115, measures something different: a -`FunctionProcedureOrMethod` whose non-empty lines exceed a percentage of the file's, default 5%. So -the standard's normative form is a function-to-file ratio, and the line count is informative only. -The plugin implements both: `size.mode: file-lines` compares files against `size.file_lines`, and -`size.mode: iso-8.2.115` adds a per-function percentage against `size.function_lines_pct`. The -second needs a collector that reports function end lines, which no Bash collector does; that lane -says so and the run continues. +file"), and its informative §6.3 Table 1 row for that weakness carries a 1000-line figure. The +normative detection pattern attached to the weakness, §8.2.115, measures something different: a +`FunctionProcedureOrMethod` whose `NumberOfNonEmptyLinesOfCode` exceeds a +`MaxNumberOfNonEmptyLinesOfCode` whose stated default is "5%", with no base named in the clause. +This plugin reads the base as the enclosing file's non-empty lines; that reading is the plugin's, +and its verification record is in [thresholds.md](thresholds.md). So the standard's normative form +is a percentage on a function, and the line count is informative only. The plugin implements both: +`size.mode: file-lines` compares files against `size.file_lines`, and `size.mode: iso-8.2.115` adds +a per-function percentage against `size.function_lines_pct`. The second needs a collector that +reports function end lines, which no Bash collector does; that lane says so and the run continues. ## Duplication diff --git a/plugins/code-metrics/skills/principles/reference/thresholds.md b/plugins/code-metrics/skills/principles/reference/thresholds.md index be83fdc22c..7cdab71e93 100644 --- a/plugins/code-metrics/skills/principles/reference/thresholds.md +++ b/plugins/code-metrics/skills/principles/reference/thresholds.md @@ -11,8 +11,9 @@ the plugin found no defensible value to ship, so it reports the measure and coun | Cyclomatic complexity | `20` | ISO/IEC 5055:2021 §8.2.117, the normative detection pattern `ASCQM Limit Algorithmic Complexity via Cyclomatic Complexity Value`, whose `` default is 20 | Normative in the standard | | Cognitive complexity | `null` | Campbell, SonarSource. The white paper prescribes no threshold | No standard sets one | | Halstead difficulty | `null` | Halstead 1977 defines the measure and sets no limit | No standard sets one | -| Lines per file | `1000` | As the report prints it: the plugin's own number, not ISO-backed: it coincides with the informative figure in ISO/IEC 5055:2021 §6.3 Table 1, which is not normative, while the normative form (§8.2.115) is a function-level percentage; 500 is the operator-list figure, selectable; size.mode iso-8.2.115 selects the normative function-percentage alternative | Plugin default, labelled as such | -| Function lines percentage | `5` | ISO/IEC 5055:2021 §8.2.115 (normative): a function whose non-empty lines exceed this percentage of the file's | Normative in the standard | +| Lines per file | `1000` | As the report prints it: the plugin's own number, not ISO-backed: it coincides with the informative figure in ISO/IEC 5055:2021 §6.3 Table 1, which is not normative, while the normative form (§8.2.115) is a function-level percentage; 500 is selectable; size.mode iso-8.2.115 selects the normative function-percentage alternative. The Table 1 row is the CWE-1080 summary row | Plugin default, labelled as such | +| Function lines percentage | `5` | ISO/IEC 5055:2021 §8.2.115 (normative), read by this plugin as a function's non-empty lines against its file's; the clause states the 5% with no base (see the verification record below) | Normative value, plugin's reading of the base | +| Duplication | none | No key ships. `duplication.min_tokens` and `duplication.min_lines` are collector floors, not references; ISO/IEC 5055 §8.2.116 measures element similarity between two functions (default 90%), which is a different quantity from a duplicated-lines percentage | Nothing comparable to cite | | CRAP | `null` | Savoia and Evans 2007. Their own suggested value was 30, offered as a starting point they reserved the right to change | Authors' suggestion, no standard | | Coverage | `null` | No standard states a percentage. ISO/IEC 25023:2016 files test coverage under Reliability and Maturity and sets no value | No standard sets one | | Type coverage | `null` | No standard and no CWE anchors the measure | Nothing to cite | @@ -31,56 +32,57 @@ The two cited alternatives for cyclomatic complexity, selectable through config: and a comprehensive test plan". Those six named practices are part of the citation. Reporting 15 without them presents a conditional figure as an unconditional standard. -Three qualifications the table above cannot hold: +Four qualifications the table above cannot hold: - **The ISO clause map matters more than the ISO name.** Clause 7 (the weakness list) and clause 8 (the detection patterns) are both normative; clause 6 is informative. Clause 7.1.10 names the cyclomatic weakness (CWE-1121) and carries no number at all. The number lives at §8.2.117. The - 1000-line figure lives only in informative clause 6.3 Table 1, while the normative pattern - attached to the same file-size weakness, §8.2.115, states 5% of a function against its file. So - "1000 lines per file, per ISO/IEC 5055" would cite an informative table against a normative clause + 1000-line figure lives only in the informative §6.3 Table 1 row for CWE-1080, while the + normative pattern attached to the same weakness, §8.2.115, is a percentage on a function. So + "1000 lines per file, per ISO/IEC 5055" would cite an informative row against a normative clause that says something different at a different granularity, and this plugin does not say it. -- **ISO/IEC 5055:2021 is the ISO designation of OMG's ASCQM text.** Cite the standard by version and - clause, never by page number. The version ISO adopted is OMG's v1.0, dated 2020; OMG's own current - release is v1.1 (July 2022), verified by diff to keep the §8.2.115 and §8.2.117 defaults and the - clause-7 numbering. The two are not interchangeable if a later revision diverges. -- **The 25023 coverage row is MEDIUM confidence and preview-sourced.** The official ISO preview - confirms the clause structure (8.6 Reliability, 8.6.1 Maturity) and the measure-id grammar, from - which `RMa-4-S` is a structurally valid id for a fourth, Specific-category Maturity measure. The - normative body naming individual measures is paywalled and was not read. Even if the id is exactly - right, the reproduction describes it as counting capabilities, operational scenarios, or functions - performed against those included in the test suites, which is scenario coverage rather than the - line coverage a coverage tool emits. Grouping a line-coverage percentage under that id would - overclaim twice. - -## The operator's starting list, checked - -The list this plugin was commissioned from carried ten numbers: 22, 22, 80, 500, 100, 25, and four -zeros for the count-based concerns. They came from a social post rather than a standard, and the -interview asked for them to be scrutinized. The result: - -| Value | Concern | Verdict | -|---|---|---| -| 22 | Cyclomatic complexity | **No provenance found.** McCabe 1976 and NIST SP 500-235 were downloaded and full-text searched twice, along with the threshold pages of Aivosto, ESLint (20), ReSharper (20), Microsoft CA1502 (25), NDepend (15 and 30), and NASA SWEHB (15). None attributes 22 to anyone. Dropped, not shipped | -| 22 | Cognitive complexity | **No provenance found**, and no standard sets any cognitive threshold. SonarSource's own rule default is 15, which is a vendor product decision | -| 80 | Halstead difficulty | **No provenance found.** Halstead 1977 defines difficulty and sets no limit, and no source in this plugin's research attributes 80 to anyone | -| 500 | Lines per file | Traceable to the social post and to nothing else. The nearest standards figure is the informative 1000, and the normative form is a percentage. Selectable through config, labelled as the operator-list figure | -| 100 | Coverage percentage | A policy, not a standard. No standard sets a coverage percentage, and a coverage number rises whenever a line executes, with or without an assertion | -| 25 | CRAP | **Not the authors' number.** Savoia and Evans suggested 30, and said so as a starting point after "a LOT of opinions". 25 traces to no source found | -| 0 | The count-based concerns | A target of zero is a policy choice. Three of those concerns belong to other plugins, which the routing section of `SKILL.md` names with a presence gate | - -Two of these numbers, 20 for cyclomatic and 1000 for lines per file, survived in the shipped -defaults because a citation exists for them, not because the empirical literature validates them. -No study reviewed in [literature.md](literature.md) supports a fixed threshold that transfers across -projects, and two of them argue against it directly. - -## Why nothing here fires - -The marketplace's ADR 0003 requires a measured corpus sweep before anything emits a finding -default-on. Reporting a number beside a cited reference is a measurement; deciding that the number -is a defect is a finding. This version stays on the measurement side, so no sweep is owed, no -false-positive budget is spent, and there is no `check` gate to argue with. That boundary is stated -in every audit skill's description. A future gate would need the sweep first. +- **The §8.2.115 base is this plugin's reading, not the clause's words.** Verification record. + Claim: §8.2.115 `ASCQM Limit Size of Operations Code` flags a `FunctionProcedureOrMethod` whose + `NumberOfNonEmptyLinesOfCode` exceeds `MaxNumberOfNonEmptyLinesOfCode`, whose stated default is + "5%", and the clause names no base for that percentage; this plugin reads the base as the + enclosing file's non-empty lines. Basis: OMG ASCQM v1.1 (formal/2022-07-01) §8.2.115, and the + OMG-hosted ISO edition (v1.0, October 2020), whose clause text is identical. As of: 2026-09-11. + Recheck when OMG or ISO publishes a new revision of the specification. +- **ISO/IEC 5055:2021 is the ISO publication of OMG ASCQM v1.0.** Cite the standard by version and + clause, never by page number. ISO's first edition (2021-03) carries the v1.0 text dated October + 2020; OMG's own current release is v1.1 (July 2022), which carries identical text for §8.2.115 + and §8.2.117. The two are not interchangeable if a later revision diverges. +- **The 25023 coverage row is MEDIUM confidence.** The official ISO preview confirms the clause + structure (8.6 Reliability, 8.6.1 Maturity) and the measure-id grammar, from which `RMa-4-S` is a + structurally valid id for a fourth, Specific-category Maturity measure. The normative body naming + individual measures is paywalled. Even if the id is exactly right, the measure as reproduced + elsewhere counts capabilities, operational scenarios, or functions performed against those + included in the test suites, which is scenario coverage rather than the line coverage a coverage + tool emits. Grouping a line-coverage percentage under that id would overclaim twice. + +## Popular numbers with no found source + +These values circulate as thresholds. None of them traces to a standard or to the author of the +measure it is applied to, in any source this plugin cites. + +| Value | Applied to | Where it was looked for | What was found | +|---|---|---|---| +| 22 | Cyclomatic complexity | McCabe 1976 and NIST SP 500-235 (full text); the threshold pages of Aivosto, ESLint (20), ReSharper (20), Microsoft CA1502 (25), NDepend (15 and 30), NASA SWEHB (15) | No source attributes 22 to anyone | +| 22 | Cognitive complexity | Campbell's white paper; SonarSource rule S3776 | No standard sets a cognitive threshold; SonarSource's rule default is 15, a product decision | +| 80 | Halstead difficulty | Halstead 1977 as reproduced by NASA NTRS, IBM, and radon; vendor threshold pages | Halstead defines difficulty and sets no limit; nothing attributes 80 to anyone | +| 500 | Lines per file | ISO/IEC 5055 clauses 6 and 8 | The nearest standards figure is the informative 1000, and the normative form is a percentage. Selectable through config | +| 100 | Coverage percentage | ISO/IEC 25023 preview; the coverage-effectiveness studies in literature.md | A policy, not a standard. A coverage number rises whenever a line executes, with or without an assertion | +| 25 | CRAP | The Crap4j FAQ and the 2007 announcement | Not the authors' number. Savoia and Evans suggested 30, as a starting point after "a LOT of opinions" | +| 0 | Count-based concerns (mutants, dead code, lint) | Not applicable | A target of zero is a policy choice; three of those concerns belong to other plugins, which `SKILL.md` names behind a presence gate | + +Sources not checked for any of these rows: vendor products behind a login, and standards bodies +other than ISO, OMG, NIST, and MITRE. A reader who finds an authoritative source for one of these +values should send it; the row changes only on a citation. + +Two of the shipped defaults, 20 for cyclomatic and 1000 for lines per file, exist because a citation +exists for them, not because the empirical literature validates them. No study reviewed in +[literature.md](literature.md) supports a fixed threshold that transfers across projects, and two of +them argue against it directly. ## Setting your own diff --git a/plugins/skill-quality/.claude-plugin/plugin.json b/plugins/skill-quality/.claude-plugin/plugin.json index 3f2caad053..7be1c13426 100644 --- a/plugins/skill-quality/.claude-plugin/plugin.json +++ b/plugins/skill-quality/.claude-plugin/plugin.json @@ -1,8 +1,8 @@ { "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "skill-quality", - "version": "0.22.2", - "description": "Skill-authoring QA tooling: a static contract checker that runs twenty-six deterministic checks over a Claude Code skill (frontmatter, explicit invocation mode, description/verb-contract polarity, per-skill listing-entry cap, trigger-keyword preservation, line caps, broken internal refs, markdownlint, gotchas surface, evals presence, precompute opportunity, completion-criteria signal, injection shell-declaration, fresh-eyes declaration conformance), a shared skill-listing budget reporter across a set of skills, and a bundled evals.json schema plus a deterministic eval-quality lint (duplicate case identities, missing fixtures, empty or vague grading criteria, set-coverage warnings). Runs against any repo's skills directory via the convention-resolution ladder, with no baked layout.", + "version": "0.22.3", + "description": "Skill-authoring QA tooling: a static contract checker that runs twenty-seven deterministic checks over a Claude Code skill (frontmatter, explicit invocation mode, description/verb-contract polarity, per-skill listing-entry cap, trigger-keyword preservation, line caps, broken internal refs, markdownlint, gotchas surface, evals presence, precompute opportunity, completion-criteria signal, injection shell-declaration, fresh-eyes declaration conformance), a shared skill-listing budget reporter across a set of skills, and a bundled evals.json schema plus a deterministic eval-quality lint (duplicate case identities, missing fixtures, empty or vague grading criteria, set-coverage warnings). Runs against any repo's skills directory via the convention-resolution ladder, with no baked layout.", "author": { "name": "Melodic Software", "email": "info@melodicsoftware.com" diff --git a/plugins/skill-quality/CHANGELOG.md b/plugins/skill-quality/CHANGELOG.md index 76dd137105..774e4123c2 100644 --- a/plugins/skill-quality/CHANGELOG.md +++ b/plugins/skill-quality/CHANGELOG.md @@ -3,6 +3,22 @@ All notable changes to the `skill-quality` plugin are documented here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning. +## [0.22.3] + +### Added + +- **`check`: check 27 reads the `## Next` successor section.** The skill-bodies rule places a + skill's successor in a `## Next` section before `## Gotchas`, as one `/plugin:skill` invocation + line or two to four `: /plugin:skill` bullets, and the gate had no check for it: a skill + missing the section passed with 0 warnings while six sibling skills in the same plugin carried + one. Absence is an INFO note rather than a WARN, because whether a skill has a natural successor + is the author's call and most skills in a large fleet have none wired yet; a section that is + present but placed after `## Gotchas`, last in the file, or in neither shape is a WARN. The + single shape has to open with the invocation, not with prose that mentions one, and either + shape is read whole for the phrasing the rule excludes (Skill tool, installed, fallback, + otherwise), so an operative chain dressed as a successor also warns. The check stays advisory, + so the gate's PASS/FAIL verdict is unchanged. + ## [0.22.2] ### Changed diff --git a/plugins/skill-quality/scripts/check-skill.sh b/plugins/skill-quality/scripts/check-skill.sh index 778141fcc0..a30d196625 100755 --- a/plugins/skill-quality/scripts/check-skill.sh +++ b/plugins/skill-quality/scripts/check-skill.sh @@ -109,6 +109,11 @@ # 26. Long spoke files carry a table of contents: a reference|references|context # markdown file over 300 lines whose first 40 lines hold fewer than three # `](#` in-page anchor links WARNs (advisory heuristic) +# 27. `## Next` successor section: absent is INFO (a terminal skill has +# none); present but after `## Gotchas`, last in the file, neither +# the one-invocation nor the two-to-four-outcome-bullet shape, or +# carrying operative-chain phrasing (Skill tool, installed, fallback, +# otherwise) anywhere in the block is WARN # # Notes (static, git-diff-based design): # - Checks 3/8/9 diff the working tree against CHECK_SKILL_BASE_REF (default @@ -1854,6 +1859,71 @@ done < <( done | sort ) +# --- Check 27: `## Next` successor section (advisory) ------------------------- +# The skill-bodies rule says a skill with a natural successor names it in a +# `## Next` section placed before `## Gotchas` (or before the last H2 when the +# file has none), in one of two shapes: one `/plugin:skill` invocation on a +# line, optionally followed by a sentence; or two to four bullets of +# `: /plugin:skill`. Whether a skill HAS a successor is the author's +# call, so absence is an INFO note rather than a WARN: most skills in a large +# fleet are terminal or not yet wired, and a WARN on each would drown the +# gate. A section that is present but misplaced or malformed is a WARN, +# because that is a shape the rule names and the author did not intend. + +NEXT_LINE="$(grep -nE '^## Next[[:space:]]*$' "$SKILL_MD" | head -1 | cut -d: -f1)" +if [[ -z "$NEXT_LINE" ]]; then + note "no '## Next' section: fine for a terminal skill; a skill with a natural successor names it there (skill-bodies rule)" +else + NEXT_GOTCHAS_LINE="$(grep -nEi '^##[[:space:]]+(gotchas|quirks)' "$SKILL_MD" | head -1 | cut -d: -f1)" + NEXT_LAST_H2="$(grep -nE '^## ' "$SKILL_MD" | tail -1 | cut -d: -f1)" + NEXT_HIT="" + if [[ -n "$NEXT_GOTCHAS_LINE" ]] && ((NEXT_LINE > NEXT_GOTCHAS_LINE)); then + NEXT_HIT="placed after '## Gotchas'; the rule puts it before" + elif [[ -z "$NEXT_GOTCHAS_LINE" ]] && ((NEXT_LINE == NEXT_LAST_H2)); then + NEXT_HIT="is the last section; the rule places it before the last H2" + fi + # The section body runs from the heading to the next H2 or end of file. + NEXT_BLOCK="$(awk -v s="$NEXT_LINE" 'NR > s { if ($0 ~ /^## /) exit; print }' "$SKILL_MD")" + NEXT_BULLETS="$(grep -cE '^- ' <<<"$NEXT_BLOCK" || true)" + NEXT_TOKEN='/[a-z0-9-]+:[a-z0-9-]+' + if ((NEXT_BULLETS == 0)); then + # The single shape opens with the invocation itself (a leading backtick + # allowed), not with prose that happens to mention one: prose first is + # how an operative chain reads. + NEXT_FIRST="$(grep -vE '^[[:space:]]*$' <<<"$NEXT_BLOCK" | head -1 || true)" + if [[ -z "$NEXT_FIRST" ]]; then + NEXT_HIT="${NEXT_HIT:+$NEXT_HIT; }body is empty" + elif ! grep -qE "^\`?${NEXT_TOKEN}(\`|[[:space:]]|[.,;:]|$)" <<<"$NEXT_FIRST"; then + NEXT_HIT="${NEXT_HIT:+$NEXT_HIT; }first line does not open with a /plugin:skill invocation" + fi + else + if ((NEXT_BULLETS < 2 || NEXT_BULLETS > 4)); then + NEXT_HIT="${NEXT_HIT:+$NEXT_HIT; }$NEXT_BULLETS bullet(s); the outcome-bullet shape carries two to four" + fi + # A bullet may wrap onto indented continuation lines; judge each bullet + # with its continuation joined. + NEXT_BAD="$(awk -v tok="$NEXT_TOKEN" ' + /^- / { if (b != "" && b !~ tok) n++; b = $0; next } + /^[[:space:]]+[^[:space:]]/ { b = b " " $0; next } + END { if (b != "" && b !~ tok) n++; print n + 0 }' <<<"$NEXT_BLOCK")" + if ((NEXT_BAD > 0)); then + NEXT_HIT="${NEXT_HIT:+$NEXT_HIT; }$NEXT_BAD bullet(s) name no /plugin:skill successor" + fi + fi + # Either shape is a mention for the human, so the whole block, not just the + # line that names the successor, is read for the three things the rule + # excludes: Skill-tool phrasing, an installed-ness gate, a fallback clause. + NEXT_OPERATIVE="$(grep -oiE 'skill tool|installed|fall ?back|otherwise' <<<"$NEXT_BLOCK" | head -1 || true)" + if [[ -n "$NEXT_OPERATIVE" ]]; then + NEXT_HIT="${NEXT_HIT:+$NEXT_HIT; }carries operative-chain phrasing ('$NEXT_OPERATIVE'); a successor is a mention, with no Skill-tool phrasing, installed-ness gate, or fallback clause" + fi + if [[ -n "$NEXT_HIT" ]]; then + warn "'## Next' section $NEXT_HIT. The skill-bodies rule wants one /plugin:skill line, or two to four ': /plugin:skill' bullets, placed before '## Gotchas'" + else + note "'## Next' section present and in the mention-only shape" + fi +fi + # --- Summary --------------------------------------------------------------- printf '\n' diff --git a/plugins/skill-quality/scripts/check-skill.test.sh b/plugins/skill-quality/scripts/check-skill.test.sh index 538f0ca8b8..1fd58e11e6 100755 --- a/plugins/skill-quality/scripts/check-skill.test.sh +++ b/plugins/skill-quality/scripts/check-skill.test.sh @@ -4000,6 +4000,173 @@ else fi fi +# Check 27: the `## Next` successor section. Absence is INFO and never a +# warning; a conforming section is silent; a misplaced or malformed one warns +# and still passes (advisory). +out="$(run good-skill 2>&1)" +rc=$? +if [[ $rc -eq 0 ]] && grep -q "INFO: no '## Next' section" <<<"$out" && ! grep -q "WARN: '## Next'" <<<"$out"; then + pass "a skill with no '## Next' gets an INFO note and no warning" +else + fail "absent '## Next' should be INFO only (rc=$rc): $out" +fi + +make_skill next-ok '--- +name: next-ok +description: "Next fixture. Use when: '"'"'next ok'"'"'." +--- + +## Purpose + +Conforming successor section in the bullet shape. + +## Next + +- The numbers feed a comparison: `/verification:measure metrics`. +- A number is about to be quoted at someone, so the caveats come first: + `/code-metrics:principles`. + +## Gotchas + +None known. +' +out="$(run next-ok 2>&1)" +rc=$? +if [[ $rc -eq 0 ]] && grep -q "INFO: '## Next' section present" <<<"$out" && ! grep -q "WARN: '## Next'" <<<"$out"; then + pass "a conforming bullet-shape '## Next' before '## Gotchas' passes silently" +else + fail "conforming '## Next' should not warn (rc=$rc): $out" +fi + +make_skill next-single '--- +name: next-single +description: "Next fixture. Use when: '"'"'next single'"'"'." +--- + +## Purpose + +Conforming successor section in the single-invocation shape. + +## Next + +`/code-metrics:audit-complexity`. The sibling skills cover the other measures. + +## Gotchas + +None known. +' +out="$(run next-single 2>&1)" +rc=$? +if [[ $rc -eq 0 ]] && ! grep -q "WARN: '## Next'" <<<"$out"; then + pass "a conforming single-invocation '## Next' passes silently" +else + fail "single-invocation '## Next' should not warn (rc=$rc): $out" +fi + +make_skill next-late '--- +name: next-late +description: "Next fixture. Use when: '"'"'next late'"'"'." +--- + +## Purpose + +Successor section placed after Gotchas. + +## Gotchas + +None known. + +## Next + +`/code-metrics:audit-complexity`. +' +out="$(run next-late 2>&1)" +rc=$? +if [[ $rc -eq 0 ]] && grep -q "WARN: '## Next' section placed after '## Gotchas'" <<<"$out"; then + pass "a '## Next' after '## Gotchas' warns and passes" +else + fail "misplaced '## Next' should warn and pass (rc=$rc): $out" +fi + +make_skill next-malformed '--- +name: next-malformed +description: "Next fixture. Use when: '"'"'next malformed'"'"'." +--- + +## Purpose + +Successor section with one bullet that names no skill. + +## Next + +- Go do the next thing. + +## Gotchas + +None known. +' +out="$(run next-malformed 2>&1)" +rc=$? +if [[ $rc -eq 0 ]] && grep -q "WARN: '## Next' section 1 bullet(s); the outcome-bullet shape carries two to four; 1 bullet(s) name no /plugin:skill successor" <<<"$out"; then + pass "a malformed '## Next' warns on both the bullet count and the missing successor" +else + fail "malformed '## Next' should warn on count and token (rc=$rc): $out" +fi + +make_skill next-operative '--- +name: next-operative +description: "Next fixture. Use when: '"'"'next operative'"'"'." +--- + +## Purpose + +Single-shape successor section written as an operative chain: prose first, a +Skill-tool instruction, an installed-ness gate, and a fallback clause. + +## Next + +Ask the Skill tool to invoke /code-metrics:audit-complexity when it is installed. +Otherwise measure by hand. + +## Gotchas + +None known. +' +out="$(run next-operative 2>&1)" +rc=$? +if [[ $rc -eq 0 ]] && grep -q "WARN: '## Next' section first line does not open with a /plugin:skill invocation; carries operative-chain phrasing ('Skill tool')" <<<"$out"; then + pass "a prose-first operative '## Next' warns on the opening and the chain phrasing" +else + fail "operative '## Next' should warn on opening and phrasing (rc=$rc): $out" +fi + +make_skill next-bullets-fallback '--- +name: next-bullets-fallback +description: "Next fixture. Use when: '"'"'next bullets fallback'"'"'." +--- + +## Purpose + +Bullet-shape successor section whose second bullet carries a fallback clause. + +## Next + +- The numbers feed a comparison: `/verification:measure metrics`. +- A number is about to be quoted: `/code-metrics:principles`, or fall back to + the README when that plugin is absent. + +## Gotchas + +None known. +' +out="$(run next-bullets-fallback 2>&1)" +rc=$? +if [[ $rc -eq 0 ]] && grep -q "WARN: '## Next' section carries operative-chain phrasing ('fall back')" <<<"$out"; then + pass "a bullet-shape '## Next' with a fallback clause warns on the chain phrasing" +else + fail "bullet '## Next' with a fallback should warn on phrasing (rc=$rc): $out" +fi + if [[ $fails -ne 0 ]]; then printf '%d assertion(s) failed\n' "$fails" >&2 exit 1 diff --git a/plugins/skill-quality/skills/check/SKILL.md b/plugins/skill-quality/skills/check/SKILL.md index 784d172a12..920727897a 100644 --- a/plugins/skill-quality/skills/check/SKILL.md +++ b/plugins/skill-quality/skills/check/SKILL.md @@ -1,5 +1,5 @@ --- -description: "Skill-authoring QA for Claude Code skills. Use when: 'check this skill', 'skill quality', 'lint my skill', 'is this SKILL.md valid', 'validate skill frontmatter', 'check skill before publishing', 'validate evals.json', 'shared listing budget', 'is the skill listing overflowing', or before shipping a skill or plugin. Actions: `check []` runs a twenty-six-check static contract gate and reports PASS/FAIL with warnings; `validate-evals []` checks a skill's evals/evals.json against the bundled schema, then runs a deterministic eval-quality lint; `listing-budget [ ...]` reports the SHARED aggregate listing-budget estimate across every listing-eligible skill under the resolved root(s). Advisory only, never blocks. Not for: writing new skills, or running model-graded evals." +description: "Skill-authoring QA for Claude Code skills. Use when: 'check this skill', 'skill quality', 'lint my skill', 'is this SKILL.md valid', 'validate skill frontmatter', 'check skill before publishing', 'validate evals.json', 'shared listing budget', 'is the skill listing overflowing', or before shipping a skill or plugin. Actions: `check []` runs a twenty-seven-check static contract gate and reports PASS/FAIL with warnings; `validate-evals []` checks a skill's evals/evals.json against the bundled schema, then runs a deterministic eval-quality lint; `listing-budget [ ...]` reports the SHARED aggregate listing-budget estimate across every listing-eligible skill under the resolved root(s). Advisory only, never blocks. Not for: writing new skills, or running model-graded evals." argument-hint: "[check|validate-evals|listing-budget] [ ...]. Omit the action for check; omit the name/root to run over every skill under the resolved root" user-invocable: true disable-model-invocation: false